Curated pandas resources for data engineers: DataFrames, data wrangling, and the core operations that appear in every Python pipeline.
The official intro. "10 minutes to pandas" is dense but accurate. The comparison with other tools is useful for contextualizing where pandas fits.
A fast reference for the most common operations. Useful to have open while you are writing pipeline code.
Free online. The definitive book on pandas from its creator. Part II on data wrangling is required reading for any data engineer.
The 80/20 rule applies strongly to pandas: read_csv, read_parquet, merge, groupby, pivot_table, and to_parquet cover the vast majority of data engineering use cases. read_parquet with the pyarrow engine is the right default for any pipeline reading columnar data -- it handles schema inference correctly and is significantly faster than CSV parsing for any structured dataset. Using merge with how=left and the validate parameter catches unintended key duplication before it propagates silently through the pipeline, which is safer than the default inner join that silently drops unmatched rows.
groupby followed by agg is the core of aggregation pipelines in data engineering. The named aggregation syntax -- assigning result column names directly in the agg call rather than using a dict of lists -- produces named columns without the multi-level index the older syntax creates, and is more readable in code review and debugging. Use transform instead of agg when you need to add an aggregate value back to the original DataFrame without reducing rows -- this avoids the pattern of groupby-agg followed by a merge, which is both slower and harder to read.
The most common correctness mistake in pandas pipelines is chained assignment: selecting a subset of rows and then assigning into that selection, which modifies a temporary copy rather than the original DataFrame and raises a SettingWithCopyWarning. Use .loc for all in-place modifications. This is not a style preference; chained assignment silently does nothing on some pandas versions while the warning is easy to miss in pipeline logs. In pandas 3.0, copy-on-write semantics are the default, and chained assignment raises a hard error rather than a warning.
The practical memory ceiling for pandas is roughly one-third to one-half of available RAM. pandas builds several intermediate copies during complex operations: a filter creates a copy, a groupby allocates hash tables for unique keys, a merge on string columns creates an index. Operations on large, wide DataFrames with high-cardinality string columns are the most likely to trigger MemoryError or swap activity. When you reach that ceiling, the problem is not the DataFrame itself but the operation overhead.
The migration decision depends on why you are hitting the ceiling. If the bottleneck is memory rather than speed -- the DataFrame fits in RAM but barely -- Polars is the first option to evaluate. Its columnar memory format and string interning typically reduce memory footprint by 30-60 percent versus pandas on the same dataset, often resolving previously problematic workloads without a cluster. If the bottleneck is computation speed on large DataFrames, Polars uses true parallelism across all available CPU cores, versus pandas which is constrained to single-threaded execution by the Python Global Interpreter Lock. If the data does not fit in memory on any single machine, the right options are Dask (which parallelizes pandas-compatible operations across cores or a distributed cluster) or pushing the computation into the warehouse with SQL and dbt, which is usually the highest-leverage move for aggregation-heavy workloads.
Some course links above are affiliate links. If you enroll, we may earn a small commission at no extra cost to you.
New resources and perspective on building AI-ready data systems, a few times a month. No spam.