A curated learning tree for data engineers building AI-ready data systems. Two tiers: Foundations covers the durable data engineering stack; AI Layer covers where data engineering meets modern AI -- with original editorial on each topic, not just link lists.
View as interactive skill graph >>
The spine of the tree. Master these before moving to the AI layer. Each node links to a tight, opinionated curation of the best resources -- no padding.
Curated SQL resources for data engineers: window functions, query optimization, and analytical patterns used in production data pipelines.
The best dbt learning resources: from your first model to production-grade dbt projects with testing, documentation, and CI.
Curated Spark resources: PySpark fundamentals, performance tuning, and Spark for large-scale ML feature engineering.
Curated Airflow resources: DAG patterns, task dependencies, and running reliable data pipelines in production.
Curated resources on data warehouse design: Kimball modeling, modern cloud DWH, and the lake-house pattern.
Curated Python resources for data engineers: core language, data manipulation, scripting patterns, and the libraries that show up in every pipeline.
Curated pandas resources for data engineers: DataFrames, data wrangling, and the core operations that appear in every Python pipeline.
Polars vs pandas for Python data engineering: why Polars is faster (lazy evaluation, columnar execution, true parallelism), when to switch, and the best resources to learn it.
Curated resources on data ingestion patterns: ETL vs. ELT, modern ingestion tools, and choosing the right pattern for your data stack.
Curated resources on Extract-Load tools and data ingestion: Airbyte, dlt, Fivetran, and choosing the right ingestion layer for your data stack.
Curated resources on pipeline orchestration: DAG concepts, Airflow, Dagster, and choosing the right scheduler for your data stack.
Curated resources on modern pipeline orchestrators: Dagster's asset-centric model and Prefect's hybrid execution -- alternatives to Airflow for greenfield data stacks.
Curated Kafka resources: the distributed log, partitioning, consumer groups, Kafka Connect, and building event-driven data pipelines.
Curated stream processing resources: Spark Structured Streaming, Apache Flink, stateful computation, and when streaming beats batch.
The warehouse skills AI agents need from data engineers: query generation, schema exposition, metadata grounding, and semantic interfaces.
Where data engineering meets AI. Each page carries original, opinionated editorial on how data engineers build and maintain these systems -- the decisions, the failure modes, and the organizational realities.
How data engineers build and maintain RAG pipelines: chunking strategies, retrieval quality, and keeping knowledge bases fresh.
A data engineer's guide to vector databases: when they beat traditional search, how to evaluate options, and operational considerations.
What data engineers need to know about embeddings: model selection, generation pipelines, storage, and drift detection.
How data engineers build high-quality training datasets: labeling pipelines, quality metrics, active learning, and data flywheel design.
Building evaluation pipelines for AI systems: what to measure, how to build eval datasets, and how to automate regression detection.
Context engineering is the discipline of designing what information goes into an LLM's context window -- and how. A data engineer's guide.
How data engineers build production AI agent systems: the agent loop, tool use, ReAct, MCP, memory, multi-agent orchestration, and evaluation.
Vendor-managed text-to-SQL and agentic analytics platforms: what Snowflake Cortex Analyst and similar services provide, where they fit, and the trade-offs vs. building your own.
How data catalogs (Dataplex, DataHub, Alation) ground AI agents in warehouse structure: semantic descriptions, lineage, ownership, and the catalog as a knowledge base.
Data contracts define the agreement between data producers and consumers. Why they are essential for AI systems and how to implement them.
How data engineers measure, monitor, and enforce data quality at scale -- and why "data downtime" is the right mental model for the problem.
How to build a reliable dbt test suite: built-in tests, custom generic tests, singular tests, and plugging dbt-utils and dbt-expectations into your CI.
When dbt tests are not enough: how to use Great Expectations and Soda for rich data validation, profiling, and documentation.
Data observability is the ability to understand the health of your data at any point in time. How it differs from testing and monitoring, and how to build it.
Column-level lineage, OpenLineage, and how to use lineage graphs to accelerate root-cause analysis when pipelines break.
How to detect silent data failures -- volume drops, distribution shifts, and freshness gaps -- automatically, without predefining every rule.
How schema changes silently break downstream pipelines, and how data contracts prevent the problem by making schema a versioned, tested interface.
The frontier where LLM-powered agents detect, diagnose, and repair pipeline failures autonomously -- and where the engineering discipline to do it safely is still being built.
How LLM-powered agents work: the perception-reasoning-action loop and the memory and tool infrastructure data engineers build to support it.
The ReAct pattern -- interleaving explicit reasoning traces with tool calls -- and why it is the default loop design for most production agents.
How to design, implement, and operate tools for LLM agents: interface design, safety boundaries, idempotency, and observability.
The Model Context Protocol is the emerging standard for connecting LLMs to tools and data sources. A data engineer's guide to building and operating MCP servers.
When and how to decompose AI tasks across multiple agents: orchestration patterns, coordination strategies, and the data infrastructure that makes multi-agent systems reliable.
A vendor-neutral taxonomy of agent memory: short-term vs. long-term, episodic vs. semantic vs. procedural, working memory vs. retrieval memory, and the failure modes -- memory bloat, stale memory, retrieval drift -- that kill production agents.
A data engineer's taxonomy of LLM hallucinations: where they come from, how to classify them, and what each type implies for system design.
How grounding reduces hallucination by anchoring LLM outputs to retrieved evidence -- the data engineering side of keeping models honest.
How to build AI systems that cite their sources -- the data engineering infrastructure for traceable, auditable LLM outputs.
How to reliably extract structured data from LLMs using schema constraints, JSON mode, and function calling -- the data engineer's guide to type-safe AI outputs.
Runtime validation for LLM outputs: how to build guardrails that catch hallucinations, policy violations, and off-topic responses before they reach users.
How LLMs translate natural language into SQL: the techniques, failure modes, and production safeguards data engineers must own.
How to evaluate text-to-SQL systems: what academic benchmarks measure, why they are insufficient, and how to build your own eval dataset.
How to give LLMs access to large warehouse schemas: embedding table and column metadata, retrieving relevant schema context, and the schema linking problem.
Why the semantic layer (dbt Semantic Layer, Looker LookML, Cube) is the right abstraction for LLM warehouse access -- and how to wire it up.
How the Model Context Protocol (MCP) standardizes agent access to data warehouses: what warehouse MCP servers expose, and the security and cost problems data engineers must solve.
A data engineer's guide to making LLMs reliably factual: grounding, structured outputs, guardrails, self-verification, and factuality evaluation.
New resources and perspective on building AI-ready data systems, a few times a month. No spam.