Snakemake to python package

Project structure 

The first point is to organize the git repository as a python package. To help you, you can run the command:

generate_template -p git_repository -n PKGNAME

which produces the following tree

now I will explain what you have to modify in the files to create your package. You must adapt the highlighted lines on example files, to your project.

Mandatory files 

pyproject.toml file 

This file is used to create a python package as described here in the official documentation.

The init.py file 

This file is the entry point of the python package.

global_variables.py file 

This file allows to group the variables to be used in the wrapper

main.py file 

This is the main script of the workflow. You have to check that its path is in the setup.py file. Normally it is already included with the instruction line 75:

entry_points={
    'console_scripts': [f"{NAME} = {NAME}.main:main"],
},

This is the main script of the workflow. You have to check that its path is in the setup.py file. Normally it is already included with the instruction line 75:

This file is used on snakefile file to add more control of the configuration file and checking user values. The goal is to create a new class that inherits from SnakEcdysis in order to use the attributes in order to have access to, for example, the paths of the scripts, the default/user configuration files, …

snakefile file 

click to open snakefile

#!/usr/bin/env snakemake
# -*- coding: utf-8 -*-

from pathlib import Path
from pprint import pprint as pp
# load own functions
import PKGNAME


PKGNAME_obj = PKGNAME(PKGNAME.dico_tool, workflow=workflow, config=config)
tools_config = PKGNAME_obj.tools_config
cluster_config = PKGNAME_obj.cluster_config

# print(PKGNAME_obj.export_use_yaml)
# print for debug:
# pp(PKGNAME_obj)
# exit()
# print(tools_config)
# exit()

###############################################################################
# dir and suffix
output_dir = config["DATA"]["OUTPUT"]
log_dir = f"{output_dir}LOGS/"

# Change workdir to output path (slurm logs append on outdir)
workdir: output_dir



#############################################
# use threads define in cluster_config rule or rule default or default in snakefile
#############################################
def get_threads(rule, default):
    """
    give threads or 'cpus-per-task from cluster_config rule: threads to SGE and cpus-per-task to SLURM
    """
    if cluster_config:
        if rule in cluster_config and 'threads' in cluster_config[rule]:
            return int(cluster_config[rule]['threads'])
        elif rule in cluster_config and 'cpus-per-task' in cluster_config[rule]:
            return int(cluster_config[rule]['cpus-per-task'])
        elif '__default__' in cluster_config and 'cpus-per-task' in cluster_config['__default__']:
            return int(cluster_config['__default__']['cpus-per-task'])
        elif '__default__' in cluster_config and 'threads' in cluster_config['__default__']:
            return int(cluster_config['__default__']['threads'])
    # if local
    elif workflow.global_resources["_cores"]:
        return workflow.global_resources["_cores"]
    # if cluster not rule and not default or local not _cores return value from call
    return default



################################ ASSEMBLY ####################################
include: f"{PKGNAME_obj.install_path}/snakefiles/assemblers.snake"



rule some_exemple:
    threads: get_threads("some_exemple",1)
    input:
        toto=rules.start.output.machin,
        truc=ref
    output:
        csv_var_per_contig=f"{output_dir}4_STRUCTURAL_VAR/csv_variants/{{samples}}_variants_per_contig.csv"
    log:
        error=f'{log_dir}some_exemple/{{samples}}.e',
        output=f'{log_dir}some_exemple/{{samples}}.o'
    message:
        f"""
        Running {{rule}}
        Input:
            - toto: {{input.toto}}
        Output:
            - csv_file: {{output.csv_var_per_contig}}
        Others
            - Threads: {{threads}}
            - LOG error: {{log.error}}
            - LOG output: {{log.output}}
        """
    singularity:
        tools_config['SINGULARITY']['TOOLS']
    envmodules:
        tools_config["MODULES"]["MINIMAP2"],
        tools_config["MODULES"]["SAMTOOLS"]
    shell:
         f"(python {PKGNAME_obj.snakemake_scripts}/vcf_contigs.py -v {{input.vcf_file}} -r {{input.ref_fasta}} -o {{output.csv_var_per_contig}}) 1>{{log.output}} 2>{{log.error}}"