Spark

A comprehensive reference guide for optimizing Apache Spark jobs. Covers partitioning strategies, join optimization (broadcast, sort-merge, bucket joins), caching patterns, memory configuration, shuffle reduction techniques, and data format optimization with practical PySpark code examples.

A comprehensive data engineering skill that teaches production-ready Apache Airflow DAG patterns including TaskFlow API, dynamic DAG generation, sensors, branching logic, and error handling. Also covers dbt transformation patterns, Spark optimization techniques, and data quality frameworks with Great Expectations.