Compare commits
3 Commits
3bd167dd49
...
6f1c81c202
Author | SHA1 | Date |
---|---|---|
Wirawan Purwanto | 6f1c81c202 | 2 years ago |
Wirawan Purwanto | b853d19209 | 2 years ago |
Wirawan Purwanto | f63bb3bcc1 | 2 years ago |
1 changed files with 101 additions and 0 deletions
@ -0,0 +1,101 @@ |
|||||||
|
# Question: Running Python Code on Jupyter Notebook |
||||||
|
|
||||||
|
<!-- |
||||||
|
Published URL: https://forum.hpc.odu.edu/t/running-a-python-code-on-hpc-through-jupyter/85 |
||||||
|
Note: The version on this doc is slightly different from what published |
||||||
|
there as of 2022-05-19. |
||||||
|
--> |
||||||
|
|
||||||
|
I have Python code that I would like to run on HPC. |
||||||
|
Can I run it through the Jupyter notebook? |
||||||
|
If so, how can I? |
||||||
|
|
||||||
|
<!-- was taken from Wirawan's WORK SCRAP NOTES --> |
||||||
|
|
||||||
|
Yes, but we recommend that you launch the script through the HPC job scheduler, |
||||||
|
called “SLURM”. |
||||||
|
Let's call the script you want to run `SCRIPT.py` |
||||||
|
*(include the full path if the script is not the same directory |
||||||
|
as where you want to run this)*. |
||||||
|
Here is a basic SLURM job script that may work for you: |
||||||
|
|
||||||
|
``` |
||||||
|
#!/bin/bash |
||||||
|
#SBATCH -p gpu |
||||||
|
#SBATCH --gres gpu:1 |
||||||
|
enable_lmod |
||||||
|
module load container_env tensorflow-gpu/2.4.0 |
||||||
|
crun.tensorflow-gpu python3 SCRIPT.py |
||||||
|
``` |
||||||
|
|
||||||
|
**Important notes:** |
||||||
|
|
||||||
|
* Don't worry why we recommend using the `tensorflow` |
||||||
|
[container](https://wiki.hpc.odu.edu/Containers/Container-Intro). |
||||||
|
This container has a installation of Python 3.7 that has fairly |
||||||
|
complete set of basic libraries used by Python users, such as |
||||||
|
`numpy`, `scipy`, `pandas`, `scikit-learn`, and of course, |
||||||
|
TensorFlow. |
||||||
|
* Replace `SCRIPT.py` with your actual script file name. |
||||||
|
* We assume that your workload needs a GPU to run your calculation. |
||||||
|
If you do not, please remove the `#SBATCH -p` and `#SBATCH --gres` lines |
||||||
|
and replace the words `tensorflow-gpu` with `tensorflow-cpu` is everywhere. |
||||||
|
|
||||||
|
Create this script using a text editor, save it to a file (say, `JOB.sh`). |
||||||
|
Then you will submit the script |
||||||
|
from the UNIX shell interface of the cluster by typing: |
||||||
|
|
||||||
|
``` |
||||||
|
$ sbatch JOB.sh |
||||||
|
``` |
||||||
|
|
||||||
|
(The `$` at the beginning of the line represents |
||||||
|
the shell prompt--do not type that.) |
||||||
|
If successful, there will be a message printed |
||||||
|
`Submitted batch job NNNNNN` |
||||||
|
on the terminal, |
||||||
|
where `NNNNNN` is an integer called job ID or job number. |
||||||
|
The output file will be `slurm-NNNNNN.out`. |
||||||
|
|
||||||
|
Here are a few documentations to help understand this process: |
||||||
|
|
||||||
|
* Documentation on Python on ODU HPC: |
||||||
|
<https://wiki.hpc.odu.edu/Software/Python> |
||||||
|
|
||||||
|
* General workflow of using SLURM job scheduler: |
||||||
|
<https://wiki.hpc.odu.edu/slurm#general-workflow-of-using-slurm> |
||||||
|
|
||||||
|
If you prefer video introduction to SLURM, please take a look at this short video to understand what job scheduler is: |
||||||
|
|
||||||
|
**"Working with SLURM"**<br/> |
||||||
|
<https://odumedia.mediaspace.kaltura.com/playlist/dedicated/1_8eqsb16m/1_oy7ls1o0> |
||||||
|
|
||||||
|
(This is part of the |
||||||
|
[Intro to HPC](https://wiki.hpc.odu.edu/Training/HPC-Intro) |
||||||
|
training videos.) |
||||||
|
|
||||||
|
|
||||||
|
----- |
||||||
|
|
||||||
|
*Back to your original question*: |
||||||
|
Yes, you can run the script from a notebook. |
||||||
|
Once you opened a (blank) notebook, you will enter this statement in a new Python cell: |
||||||
|
|
||||||
|
``` |
||||||
|
%load SCRIPT.py |
||||||
|
``` |
||||||
|
|
||||||
|
Press `<Shift+Enter>` once, the code will be loaded into that same cell. |
||||||
|
Press `<Shift+Enter>` once more, the code will execute. |
||||||
|
But please note that this mode of execution can be awkward, i.e. |
||||||
|
we have to open a Jupyter session, |
||||||
|
`%load` the script in order to run it. |
||||||
|
The maximum time to run the script is under 24 hours (the max limit of Jupyter on HPC). |
||||||
|
This mode is useful if you plan to interactively test or troubleshoot a script, |
||||||
|
but less useful when you have a bunch of calculations to do, |
||||||
|
or calculation that you expect to run for a long time. |
||||||
|
|
||||||
|
*If you still have issue or question after reading this response:* |
||||||
|
Can you share the script with us so we may understand the requirements of your script? |
||||||
|
|
||||||
|
|
Loading…
Reference in new issue