Standardizes raw academic and medical data files into structured SourceSet DataFrames for research pipelines.
This skill manages the first layer of the 6-layer HAIPipe research data pipeline, focusing on transforming raw formats like CSV, XML, and Parquet into standardized, domain-consistent table structures. It enables researchers to automate data ingestion, maintain schema consistency across different datasets—such as EHR, CGM, or genomic data—and generate new transformation logic using a builder-pattern 'chef' metaphor. By providing a structured interface for data loading and pipeline execution, it ensures that raw research data is perfectly prepared for downstream record alignment and machine learning feature extraction.
Key Features
01Enforces strict schema consistency for seamless cross-dataset integration
02Converts raw CSV, XML, Parquet, and JSON into standardized SourceSet DataFrames
03Provides interactive subcommands for pipeline orchestration and data inspection
04Automates code generation for new data transformation functions via 'Chef' builders
05Supports diverse domains including CGM, EHR, genomics, and wearable device data
060 GitHub stars
Use Cases
01Automating reproducible data preprocessing workflows for academic paper incubation
02Standardizing messy wearable device logs into clean, temporally aligned records
03Ingesting heterogeneous clinical trial data into a unified research format
