* Added documentation on Snakemake. Still WIP; first draft

was published Jan 2023 on ODU HPC wiki.

  Sigs:

    # -rw-r--r-- 1 wirawan wirawan 3569 2023-01-24 12:13 20230123.snakemake-software.md
    # -rw-r--r-- 1 wirawan wirawan 7292 2023-01-24 16:06 20230124.snakemake-intro.md
master^2
Wirawan Purwanto 1 year ago
parent c0ead53c0c
commit b601a7e249
  1. 159
      workflow/20230123.snakemake-software.md
  2. 248
      workflow/20230124.snakemake-intro.md

@ -0,0 +1,159 @@
# Snakemake: A General Introduction
Main-website:
https://snakemake.github.io/
Snakemake Documentation
https://snakemake.readthedocs.io/en/stable/
Tutorials:
* Official tutorial (complete steps)
https://snakemake.readthedocs.io/en/stable/tutorial/tutorial.html#tutorial
- Basic tutorial:
https://snakemake.readthedocs.io/en/stable/tutorial/basics.html#tutorial-basics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
Standalone
This is the easiest way to run it, you will be allocated to a single node, and all resource is allocated to you, there is no cross-node parallelization in this way, you can use job script below:
#!/bin/bash
#SBATCH --exclusive
enable_lmod
module load container_env snakemake
crun snakemake
Break rules to their individual Slurm Jobs
This is my recommended way of running this package, it utilizes resource most effectively, however it is a little bit more complicated, so I have written some wrapper script to do most of legging work, you can just launch it like this:
#!/bin/bash
#SBATCH -c 2
enable_lmod
module load container_env snakemake
snakemake.helper -j 10
Please note that this script should only take 1 or 2 core to run, it is a master script, it does not do the real work, additional jobs will be launched by this job.
Instead of run "crun snakemake" directly, you run "snakemake.helper" instead, it's the helper script I mentioned, it will launch job within the cluster for you.
For cluster resource, I only enforce "threads" in snake rules, any other resource (mem,disk,….) might or might not be enforced by snakemake (I am not sure), at least I will not enforce it on the scheduler level.
rule map_reads:
input:
"data/genome.fa",
"data/samples/{sample}.fastq"
output:
"results/mapped/{sample}.bam"
threads: 2
shell:
"bwa mem {input} | samtools view -b - > {output}"
When setting threads in a rule, I will launch Slurm job with same "--cpus-per-task" configuration to match it. When not explicitly set, it will always be single thread.
My snakemake module also support MPI mode, but from what I observe your tasks usually involving running some code on a lot of input, instead of running a single multiple node code on a single input, so mode 2 should be most useful to you. If you do need to run MPI somehow, please let me know it will require some additional setup, especially when combined with "--use-conda" .
Snakemake conda is supported and tested, you should be able to install any conda package you want to. Unless it requires MPI, it usually works. When using conda, please make sure environment file is given and launch snakemake with "—use-conda":
# Snakefile
rule map_reads:
input:
"data/genome.fa",
"data/samples/{sample}.fastq"
output:
"results/mapped/{sample}.bam"
threads:
2
conda:
"envs/mapping.yaml"
shell:
"bwa mem {input} | samtools view -b - > {output}"
# job_script.sh
crun snakemake --use-conda # for standalone
snakemake.helper -j 10 --use-conda # for running in scheduler mode
Running other container modules is also possible, please just let me know what you need. If you can install what you need with conda, please use conda first since you can do it yourself.
You can find my sample job script in /home/jsun/snakemake , please let me know if you have any questions or issues.

@ -0,0 +1,248 @@
# Snakemake: A General Introduction
Snakemake is a workflow tool for managing and executing
interrelated sets of computation/analysis steps.
For more information, please visit
[Snakemake documentation](https://snakemake.readthedocs.io/en/stable/)
## Snakemake in a Nutshell
Imagine that you have a complex computation that consists of a series
of tasks that must be executed in order
(each taking the output from the previous step as its input):
input preprocessing => simulation => output postprocessing
In the example above, a complete computation consists of:
* step 1: input preprocessing of some sort, such as generating
a set of input files from a few parameters;
* step 2: compute-heavy simulation, potentially taking hours or days
on many CPU cores and/or GPUs;
* step 3: output postprocessing, such as calculating some statistics
from the simulation, determining molecular properties from
the simulation, creating a report with graph panels.
If there is only one computation to do, then it's no brainer to do the
three steps "by hand", i.e. creating up to three SLURM job scripts and
running them in sequence.
What if there are 1000 of such computations to perform?
What if the set of steps are complex (e.g. one step requires a few
inputs from different prior computations), or long?
Is there a better way than sitting in front of terminal submitting
1000+ jobs *in a particular order*?
The answer is, **yes!**
This is where we need *workflow tools* such as Snakemake.
In Snakemake, a complete workflow is stored in a specially formatted text
file named `Snakefile`.
Each processing step is expressed as a *rule* inside the snakefile.
Each rule may depend on one or more other rules; the input and output
file declaration determines this dependency rule.
> A more thorough introduction to Snakemake is beyond the scope of this article.
Readers are referred to some articles and tutorials linked at the end
of this article to learn more.
We assume that you have a basic idea of workflow tools in order to use Snakemake.
{.is-info}
## Snakemake on ODU HPC
On Wahab and Turing, there are two modes to use Snakemake:
Snakemake is installed, there are two modes to run this software:
- Cluster (job-based) execution
- Single-node (standalone) execution
We will describe cluster execution first, which is more scalable and powerful.
### Mode 1: Cluster Execution (recommended)
In the [Cluster (job-based) execution](https://snakemake.readthedocs.io/en/stable/executing/cluster.html),
Snakefile will turn each rule into an individual Slurm job and execute
them via the job scheduler.
This will allow the HPC resources to be used most efficiently,
therefore we recommend using this way to run your workflow, whenever possible.
We created a helper script called `snakemake.helper` to help executing
a snakefile workflow through a job scheduler.
Here is an example invocation (written as a Slurm job script):
```
#!/bin/bash
#SBATCH --cpus-per-task 2
#SBATCH --job-name Example_snakemake_cluster
enable_lmod
module load container_env snakemake
snakemake.helper -j 10
```
This script should only take 1 or 2 core to run, since it is *only* a master script.
It does not do the real work (e.g. it does not run the rule in this script).
When processing the input snakefile, Snakemake will spawn additional Slurm jobs
(where one snakefile rule == one Slurm job)
and monitor their completion in order to push the workflow forward to its completion.
Pros:
- Very scalable, this method can use as many CPU cores and compute nodes as
specified in the snakefile (or by the cluster policy);
- Complete control over resource specification per rule;
Cons:
- May not be efficient for small/simple rules.
> Important: In the cluster execution model, do not run `crun snakemake` directly!
You run `snakemake.helper` helper script instead.
{.is-info}
> For cluster resources, we only enforce `threads` in the snakefile rules.
Any other resource (memory, disk, ..) might or might not be enforced by
Snakemake.
{.is-info}
## Mode 2: Single-node (stand-alone) execution
In the single-node execution mode, you will acquire all the CPU
resources in a single node and can use all the cores to perform as
many tasks as can be parallelized to shorten the execution time.
(Snakemake will figure out what rules can be executed in parallel and
try to execute them concurrently, subject to the CPU core
constraints).
Use the following template to start your own job running Snakemake in
the single-node mode:
```
#!/bin/bash
#SBATCH --exclusive
#SBATCH --job-name Example_snakemake_1node
enable_lmod
module load container_env snakemake
crun snakemake
```
Pros:
- The easiest way to run Snakemake (no need to think about threads, etc.);
- You will have complete access to all compute resources on a single node
(CPU, memory, ...).
Cons:
- No parallelization across multiple nodes,
thus limiting the parallel scalability;
- Rules will be invoked within the same container as the Snakemake program;
if your program requires software in other containers, this will not work.
(Currently a containerized program [those with `crun` in its invocation]
cannot execute another program located inside a different container).
(I am not sure), at least I will not enforce it on the scheduler level.
rule map_reads:
input:
"data/genome.fa",
"data/samples/{sample}.fastq"
output:
"results/mapped/{sample}.bam"
threads: 2
shell:
"bwa mem {input} | samtools view -b - > {output}"
When setting threads in a rule, I will launch Slurm job with same "--cpus-per-task" configuration to match it. When not explicitly set, it will always be single thread.
My snakemake module also support MPI mode, but from what I observe your tasks usually involving running some code on a lot of input, instead of running a single multiple node code on a single input, so mode 2 should be most useful to you. If you do need to run MPI somehow, please let me know it will require some additional setup, especially when combined with "--use-conda" .
-
is installed, there are two modes to run this software:
Standalone
Snakemake conda is supported and tested, you should be able to install any conda package you want to. Unless it requires MPI, it usually works. When using conda, please make sure environment file is given and launch snakemake with "—use-conda":
# Snakefile
rule map_reads:
input:
"data/genome.fa",
"data/samples/{sample}.fastq"
output:
"results/mapped/{sample}.bam"
threads:
2
conda:
"envs/mapping.yaml"
shell:
"bwa mem {input} | samtools view -b - > {output}"
# job_script.sh
crun snakemake --use-conda # for standalone
snakemake.helper -j 10 --use-conda # for running in scheduler mode
Running other container modules is also possible, please just let me know what you need. If you can install what you need with conda, please use conda first since you can do it yourself.
You can find my sample job script in /home/jsun/snakemake , please let me know if you have any questions or issues.
Loading…
Cancel
Save