How does this differ from Snakemake's default behavior?

Standard Snakemake checks for completion at the rule level. If a job is interrupted, it restarts the entire rule. This skill adds logic inside the script to skip individual channels within a single rule execution.

What happens if a channel is only partially finished?

The skill uses an all-or-nothing approach. If any expected file for a channel is missing, the entire channel is reprocessed to prevent data corruption.

Does this require modifications to my Snakefile?

No, the logic is implemented entirely within the Python wrapper scripts, meaning your DAG and Snakemake architecture remain unchanged.

Is this approach safe for network filesystems like HiPerGator?

Yes. It avoids relying on potentially unreliable NFS timestamps, instead using explicit file existence checks and file counts to verify completion.

Snakemake Per-Channel Resume Logic

Name: Snakemake Per-Channel Resume Logic
Author: smith6jt-cop

bysmith6jt-cop

•

Data Science & ML

Implements granular skip-existing checks in Snakemake wrapper scripts to resume interrupted HPC jobs without re-processing completed channels.

This skill provides a specialized implementation pattern for Snakemake workflows running on high-performance computing (HPC) environments like SLURM. While Snakemake typically manages dependencies at the cycle level via sentinel files, this skill introduces fine-grained, per-channel completion checks within Python wrapper scripts (stitching, deconvolution, EDF). This prevents the loss of hours of computation time by allowing interrupted jobs to skip over successfully processed channels while maintaining data integrity through 'all-or-nothing' verification logic.

Key Features

01Custom sentinel file generation including skip/success metadata

02Granular channel-level completion checks inside Python wrapper scripts

031 GitHub stars

04Transparent integration that preserves the existing Snakemake DAG

05Automated resumption of interrupted SLURM jobs without re-processing data

06Integrity verification using expected vs. actual file counts

Use Cases

01Ensuring data consistency on network filesystems (NFS) with unreliable timestamps

02Optimizing long-running Snakemake rules that process independent data subsets

03Resuming multi-channel microscopy pipelines after HPC wall-time interruptions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add smith6jt-cop/skills_registry snakemake-skip-existing

For use in Claude.ai and ChatGPT

Download Skill