Apache Spark

Curated Spark resources: PySpark fundamentals, performance tuning, and Spark for large-scale ML feature engineering.

Best resources to learn Apache Spark

• Learning Spark, 2nd Edition (O'Reilly)

  The textbook. Covers DataFrames, SQL, Structured Streaming, and MLlib. Read chapters 1-7 before anything else.

• PySpark Documentation

  The API reference you will tab-switch to constantly. Bookmark the DataFrame and functions modules.

• Spark Performance Tuning Guide

  Adaptive Query Execution, partition pruning, join strategies. This is where most production Spark pain lives.

• Delta Lake Documentation

  ACID transactions on Spark. Almost every serious Spark data lake uses Delta or Iceberg. Start here.