Curated resources on data ingestion patterns: ETL vs. ELT, modern ingestion tools, and choosing the right pattern for your data stack.
A clear, vendor-neutral explanation of when each pattern applies. Good starting point for understanding the trade-offs.
The best modern Python-native ingestion library. Schema inference, incremental loading, and dozens of built-in sources. Use this before building custom connectors.
300+ pre-built connectors, open-source with a cloud option. The right choice when you need a managed connector catalog without a Fivetran budget.
Chapter 7 covers ingestion in depth, including batch vs. stream, push vs. pull, and change data capture. The best textbook treatment of the topic.
The shift from ETL to ELT reflects the rise of cheap, scalable cloud data warehouses. When compute was expensive, you transformed before loading. Now that Snowflake and BigQuery can transform petabytes in SQL, it is almost always better to load raw data first and transform inside the warehouse with dbt. ETL still makes sense for sensitive data that must be cleaned before storage, or when your target system cannot handle raw data volume.
Modern ingestion is a solved problem for most common sources. Tools like Fivetran, Airbyte, and dlt handle the connectors, schema evolution, incremental loading, and deduplication. The data engineer's job is to evaluate which tool fits the stack, configure it correctly, and build the custom connectors for sources these tools do not cover.
Some course links above are affiliate links. If you enroll, we may earn a small commission at no extra cost to you.
New resources and perspective on building AI-ready data systems, a few times a month. No spam.