How does it handle failed transfers or interrupted sessions?

It utilizes the --sync-level size flag and recursive options, allowing for safe re-runs that only transfer missing or modified files without duplicating existing data.

Can I use this for datasets other than HuBMAP?

While pre-configured for HuBMAP CODEX directory structures, the workflow and script logic can be adapted for any large-scale transfer between two Globus-enabled endpoints.

Why does the skill recommend against using GVFS mounts for staging?

GVFS is desktop-oriented middleware that frequently causes 'end-of-file' errors during sustained high-volume I/O. Proper cifs mounts provide the stability required for stable TB-scale transfers.

What is the main benefit of using this skill over standard Globus commands?

This skill automates complex authentication steps, provides pre-verified HiPerGator endpoint UUIDs, and includes proven workarounds for common transfer failures like GVFS mount timeouts and terminal URL wrapping.

Globus Dataset Staging

Name: Globus Dataset Staging
Author: smith6jt-cop

bysmith6jt-cop

•

Data Science & ML

Manages large-scale scientific dataset transfers between remote Globus endpoints and high-performance computing clusters.

This skill provides a specialized workflow for staging massive HuBMAP CODEX datasets from remote lab SMB shares to HiPerGator storage using the Globus CLI. It encapsulates expert knowledge on authentication, endpoint activation, and handling complex path structures, while providing specific workarounds for common transfer errors like terminal-wrapped consent URLs and unreliable GVFS mounts. It is specifically designed for researchers and data engineers who need to manage terabyte-scale data movements efficiently within academic or high-performance computing (HPC) environments.

Key Features

01Pre-configured Globus CLI workflows for HiPerGator and PATH lab endpoints

02Robust handling of complex directory nesting and special characters in file paths

031 GitHub stars

04Support for recursive transfers with size-based synchronization and timestamp preservation

05Fault-tolerant transfer logic utilizing stable cifs mounts over GVFS

06Automated staging of multi-terabyte HuBMAP CODEX datasets

Use Cases

01Staging large-scale microscopy or genomics datasets for HPC analysis

02Automating data migration from local lab workstations to centralized research clusters

03Troubleshooting and resolving authentication and consent issues in Globus CLI sessions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add smith6jt-cop/skills_registry globus-dataset-staging

For use in Claude.ai and ChatGPT

Download Skill