Data Pipelines

A comprehensive data engineering skill that teaches production-ready Apache Airflow DAG patterns including TaskFlow API, dynamic DAG generation, sensors, branching logic, and error handling. Also covers dbt transformation patterns, Spark optimization techniques, and data quality frameworks with Great Expectations.

This skill provides comprehensive documentation for integrating with DNAnexus, a cloud platform for biomedical data analysis and genomics. It covers app/applet development, file upload/download operations, job execution, workflow orchestration, and the dxpy Python SDK for building genomics pipelines with FASTQ, BAM, and VCF files.