You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
101 lines
3.4 KiB
101 lines
3.4 KiB
# Question: Running Python Code on Jupyter Notebook
|
|
|
|
<!--
|
|
Published URL: https://forum.hpc.odu.edu/t/running-a-python-code-on-hpc-through-jupyter/85
|
|
Note: The version on this doc is slightly different from what published
|
|
there as of 2022-05-19.
|
|
-->
|
|
|
|
I have Python code that I would like to run on HPC.
|
|
Can I run it through the Jupyter notebook?
|
|
If so, how can I?
|
|
|
|
<!-- was taken from Wirawan's WORK SCRAP NOTES -->
|
|
|
|
Yes, but we recommend that you launch the script through the HPC job scheduler,
|
|
called “SLURM”.
|
|
Let's call the script you want to run `SCRIPT.py`
|
|
*(include the full path if the script is not the same directory
|
|
as where you want to run this)*.
|
|
Here is a basic SLURM job script that may work for you:
|
|
|
|
```
|
|
#!/bin/bash
|
|
#SBATCH -p gpu
|
|
#SBATCH --gres gpu:1
|
|
enable_lmod
|
|
module load container_env tensorflow-gpu/2.4.0
|
|
crun.tensorflow-gpu python3 SCRIPT.py
|
|
```
|
|
|
|
**Important notes:**
|
|
|
|
* Don't worry why we recommend using the `tensorflow`
|
|
[container](https://wiki.hpc.odu.edu/Containers/Container-Intro).
|
|
This container has a installation of Python 3.7 that has fairly
|
|
complete set of basic libraries used by Python users, such as
|
|
`numpy`, `scipy`, `pandas`, `scikit-learn`, and of course,
|
|
TensorFlow.
|
|
* Replace `SCRIPT.py` with your actual script file name.
|
|
* We assume that your workload needs a GPU to run your calculation.
|
|
If you do not, please remove the `#SBATCH -p` and `#SBATCH --gres` lines
|
|
and replace the words `tensorflow-gpu` with `tensorflow-cpu` is everywhere.
|
|
|
|
Create this script using a text editor, save it to a file (say, `JOB.sh`).
|
|
Then you will submit the script
|
|
from the UNIX shell interface of the cluster by typing:
|
|
|
|
```
|
|
$ sbatch JOB.sh
|
|
```
|
|
|
|
(The `$` at the beginning of the line represents
|
|
the shell prompt--do not type that.)
|
|
If successful, there will be a message printed
|
|
`Submitted batch job NNNNNN`
|
|
on the terminal,
|
|
where `NNNNNN` is an integer called job ID or job number.
|
|
The output file will be `slurm-NNNNNN.out`.
|
|
|
|
Here are a few documentations to help understand this process:
|
|
|
|
* Documentation on Python on ODU HPC:
|
|
<https://wiki.hpc.odu.edu/Software/Python>
|
|
|
|
* General workflow of using SLURM job scheduler:
|
|
<https://wiki.hpc.odu.edu/slurm#general-workflow-of-using-slurm>
|
|
|
|
If you prefer video introduction to SLURM, please take a look at this short video to understand what job scheduler is:
|
|
|
|
**"Working with SLURM"**<br/>
|
|
<https://odumedia.mediaspace.kaltura.com/playlist/dedicated/1_8eqsb16m/1_oy7ls1o0>
|
|
|
|
(This is part of the
|
|
[Intro to HPC](https://wiki.hpc.odu.edu/Training/HPC-Intro)
|
|
training videos.)
|
|
|
|
|
|
-----
|
|
|
|
*Back to your original question*:
|
|
Yes, you can run the script from a notebook.
|
|
Once you opened a (blank) notebook, you will enter this statement in a new Python cell:
|
|
|
|
```
|
|
%load SCRIPT.py
|
|
```
|
|
|
|
Press `<Shift+Enter>` once, the code will be loaded into that same cell.
|
|
Press `<Shift+Enter>` once more, the code will execute.
|
|
But please note that this mode of execution can be awkward, i.e.
|
|
we have to open a Jupyter session,
|
|
`%load` the script in order to run it.
|
|
The maximum time to run the script is under 24 hours (the max limit of Jupyter on HPC).
|
|
This mode is useful if you plan to interactively test or troubleshoot a script,
|
|
but less useful when you have a bunch of calculations to do,
|
|
or calculation that you expect to run for a long time.
|
|
|
|
*If you still have issue or question after reading this response:*
|
|
Can you share the script with us so we may understand the requirements of your script?
|
|
|
|
|
|
|