Big Data

A comprehensive reference guide for optimizing Apache Spark jobs. Covers partitioning strategies, join optimization (broadcast, sort-merge, bucket joins), caching patterns, memory configuration, shuffle reduction techniques, and data format optimization with practical PySpark code examples.

A comprehensive reference skill for Vaex, a high-performance Python library for processing and analyzing large tabular datasets (billions of rows) that exceed available RAM. Covers DataFrame operations, data loading, filtering, aggregations, machine learning pipelines, visualization, and performance optimization strategies.

A comprehensive documentation skill for Dask, a Python library for parallel and distributed computing. It provides detailed reference guides for working with larger-than-memory datasets using DataFrames, Arrays, Bags, and Futures, along with scheduler selection and best practices for performance optimization.

This skill provides AI assistants with deep knowledge about the Addax open-source ETL tool. It covers the project's architecture, plugin system for 20+ data sources (MySQL, PostgreSQL, MongoDB, Kafka, HDFS, etc.), job configuration format, data transformation capabilities, and development guidelines for building custom reader/writer plugins.