Compare commits
3 Commits
3bd167dd49
...
6f1c81c202
Author | SHA1 | Date |
---|---|---|
Wirawan Purwanto | 6f1c81c202 | 2 years ago |
Wirawan Purwanto | b853d19209 | 2 years ago |
Wirawan Purwanto | f63bb3bcc1 | 2 years ago |
1 changed files with 101 additions and 0 deletions
@ -0,0 +1,101 @@ |
||||
# Question: Running Python Code on Jupyter Notebook |
||||
|
||||
<!-- |
||||
Published URL: https://forum.hpc.odu.edu/t/running-a-python-code-on-hpc-through-jupyter/85 |
||||
Note: The version on this doc is slightly different from what published |
||||
there as of 2022-05-19. |
||||
--> |
||||
|
||||
I have Python code that I would like to run on HPC. |
||||
Can I run it through the Jupyter notebook? |
||||
If so, how can I? |
||||
|
||||
<!-- was taken from Wirawan's WORK SCRAP NOTES --> |
||||
|
||||
Yes, but we recommend that you launch the script through the HPC job scheduler, |
||||
called “SLURM”. |
||||
Let's call the script you want to run `SCRIPT.py` |
||||
*(include the full path if the script is not the same directory |
||||
as where you want to run this)*. |
||||
Here is a basic SLURM job script that may work for you: |
||||
|
||||
``` |
||||
#!/bin/bash |
||||
#SBATCH -p gpu |
||||
#SBATCH --gres gpu:1 |
||||
enable_lmod |
||||
module load container_env tensorflow-gpu/2.4.0 |
||||
crun.tensorflow-gpu python3 SCRIPT.py |
||||
``` |
||||
|
||||
**Important notes:** |
||||
|
||||
* Don't worry why we recommend using the `tensorflow` |
||||
[container](https://wiki.hpc.odu.edu/Containers/Container-Intro). |
||||
This container has a installation of Python 3.7 that has fairly |
||||
complete set of basic libraries used by Python users, such as |
||||
`numpy`, `scipy`, `pandas`, `scikit-learn`, and of course, |
||||
TensorFlow. |
||||
* Replace `SCRIPT.py` with your actual script file name. |
||||
* We assume that your workload needs a GPU to run your calculation. |
||||
If you do not, please remove the `#SBATCH -p` and `#SBATCH --gres` lines |
||||
and replace the words `tensorflow-gpu` with `tensorflow-cpu` is everywhere. |
||||
|
||||
Create this script using a text editor, save it to a file (say, `JOB.sh`). |
||||
Then you will submit the script |
||||
from the UNIX shell interface of the cluster by typing: |
||||
|
||||
``` |
||||
$ sbatch JOB.sh |
||||
``` |
||||
|
||||
(The `$` at the beginning of the line represents |
||||
the shell prompt--do not type that.) |
||||
If successful, there will be a message printed |
||||
`Submitted batch job NNNNNN` |
||||
on the terminal, |
||||
where `NNNNNN` is an integer called job ID or job number. |
||||
The output file will be `slurm-NNNNNN.out`. |
||||
|
||||
Here are a few documentations to help understand this process: |
||||
|
||||
* Documentation on Python on ODU HPC: |
||||
<https://wiki.hpc.odu.edu/Software/Python> |
||||
|
||||
* General workflow of using SLURM job scheduler: |
||||
<https://wiki.hpc.odu.edu/slurm#general-workflow-of-using-slurm> |
||||
|
||||
If you prefer video introduction to SLURM, please take a look at this short video to understand what job scheduler is: |
||||
|
||||
**"Working with SLURM"**<br/> |
||||
<https://odumedia.mediaspace.kaltura.com/playlist/dedicated/1_8eqsb16m/1_oy7ls1o0> |
||||
|
||||
(This is part of the |
||||
[Intro to HPC](https://wiki.hpc.odu.edu/Training/HPC-Intro) |
||||
training videos.) |
||||
|
||||
|
||||
----- |
||||
|
||||
*Back to your original question*: |
||||
Yes, you can run the script from a notebook. |
||||
Once you opened a (blank) notebook, you will enter this statement in a new Python cell: |
||||
|
||||
``` |
||||
%load SCRIPT.py |
||||
``` |
||||
|
||||
Press `<Shift+Enter>` once, the code will be loaded into that same cell. |
||||
Press `<Shift+Enter>` once more, the code will execute. |
||||
But please note that this mode of execution can be awkward, i.e. |
||||
we have to open a Jupyter session, |
||||
`%load` the script in order to run it. |
||||
The maximum time to run the script is under 24 hours (the max limit of Jupyter on HPC). |
||||
This mode is useful if you plan to interactively test or troubleshoot a script, |
||||
but less useful when you have a bunch of calculations to do, |
||||
or calculation that you expect to run for a long time. |
||||
|
||||
*If you still have issue or question after reading this response:* |
||||
Can you share the script with us so we may understand the requirements of your script? |
||||
|
||||
|
Loading…
Reference in new issue