How data engineers build production AI agent systems: the agent loop, tool use, ReAct, MCP, memory, multi-agent orchestration, and evaluation.
The agent loop is a simple idea that gets complicated fast: an LLM reasons, picks a tool, gets a result, reasons again, and repeats until it has an answer or gives up. Lilian Weng's 2023 post on LLM-powered autonomous agents remains the clearest map of the terrain -- covering the loop itself, the memory components (in-context, external, parametric), and the planning strategies that sit on top. Read it before building anything.
ReAct (Reasoning + Acting) is the pattern that made agentic LLM behavior practical. The key insight: interleave reasoning traces with tool calls so the model can observe tool outputs and adjust its plan. Most production agent frameworks implement ReAct or a close variant, even when they do not name it. The paper is short; read it directly rather than a summary.
Tool use is where data engineers own the most leverage. The model decides which tool to call; the data engineer designs what tools exist, what they do, and how their schemas are written. A well-designed tool schema -- clear parameter names, explicit descriptions, constrained types -- dramatically outperforms a vague one with the same implementation. The quality of your tool definitions is the highest-return investment in agent reliability.
The most consequential architectural decision in AI agents is whether you actually need an agent. Workflows -- fixed sequences of LLM calls and tool invocations -- are cheaper, more predictable, and easier to test than true agents. Use an agent only when the task genuinely requires dynamic routing that cannot be determined in advance. Most systems that call themselves agents should be workflows. Anthropic's "Building Effective Agents" is the clearest treatment of this distinction.
MCP (Model Context Protocol) is the emerging standard for connecting agents to tools and data sources. Think of it as the USB-C of agent connectivity: instead of a custom integration for every LLM-to-tool pair, MCP defines a single protocol that any compliant client and server can speak. Whether it becomes the dominant standard long-term is not settled, but it has enough momentum that any team building agent infrastructure should understand the protocol before evaluating MCP-compatible tooling.
Agent memory is a data engineering problem. In-context memory is fast but bounded by the context window. External memory -- vector stores, key-value stores, structured databases -- gives agents longer recall but adds latency and retrieval error. The decision of what to store, how to index it, and when to retrieve it is exactly the same class of problem as designing a data pipeline. Data engineers are better positioned to solve it than most. See the Anthropic guide on effective context engineering for agents as the practical reference.
Multi-agent orchestration multiplies both capability and complexity. Each agent hop adds latency, cost, and a new failure mode. The coordination tax is real: agents that hand off to other agents need to pass enough context to continue the task, and context compression at handoff points is where quality degrades silently. Start with a single agent. Add orchestration only when you have a specific bottleneck -- parallelism, specialization, context limits -- that a single agent cannot address. LangGraph is the most mature open-source library for building stateful multi-agent graphs.
You cannot evaluate an agent system the same way you evaluate a retrieval pipeline. Agents take variable-length paths to answers, and the intermediate steps matter as much as the final output. Build eval coverage for: final answer correctness, tool selection accuracy, unnecessary tool calls, and context drift across long trajectories. The HF Agents Course is the most complete hands-on introduction that covers building and evaluating agents end to end. Cross-link your agent evals to the Evals branch -- the infrastructure is the same.
The most important architectural decision in AI agent systems is whether you need an agent at all. An agent is justified when the task requires dynamic routing -- when you genuinely cannot determine the next step until you see the output of the previous one. A research task that may require 2 tool calls or may require 20 depending on what it finds is an agent task. A pipeline that always calls tool A, then tool B, then tool C in a fixed sequence is a workflow, regardless of whether it uses an LLM at each step.
Workflows are cheaper, more predictable, and dramatically easier to test. Every major agent framework supports workflow-style execution (a fixed graph of LLM calls and tool calls) and true agent execution (dynamic routing based on model output). For most data engineering automation -- pipeline triage, schema documentation generation, data quality reporting -- the right starting point is a workflow. Build the agent version only after you have measured that the workflow fails because it cannot adapt to unexpected state encountered at runtime.
The cost of premature agentic architecture is concrete: unpredictable execution paths, evaluation difficulty (you cannot write unit tests for a system that routes dynamically), and compounding errors across long tool-call chains where each step can introduce noise. Anthropic's "Building Effective Agents" engineering post is the clearest treatment of this decision: start with augmented LLMs (a single LLM call with context or tools), move to prompt chains only when needed, and reach for true agents only when dynamism is genuinely required.
An agent that fails in production fails in ways that are harder to diagnose than conventional software failures. The failure mode is rarely a crash -- it is a wrong decision made by the model that propagated through a tool-call sequence to produce an incorrect result. To diagnose it, you need the full trace: what was in the context at each step, which tool was selected, what arguments were passed, what the tool returned, and what reasoning the model produced before each decision. Without this trace, post-mortem analysis is guesswork.
Instrument every agent loop iteration with structured logging: a trace ID shared across the entire task, the selected tool name, the arguments passed, the tool output (truncated if large), the latency, and success or failure. Store traces in a queryable store -- even a simple relational table is sufficient to start. Build dashboards on tool error rates, loop iteration counts per completed task, and task failure rates by task type. The patterns in those dashboards -- specific tools that fail at elevated rates, tasks that consistently exhaust their iteration budget without completing -- are the diagnostics you will need when something goes wrong in production. Treating agent observability as a data engineering problem, not an afterthought, is what separates teams that can operate agents reliably from those that cannot.
The definitive map of the agent loop: components, memory types, planning strategies. Required reading before building anything.
The paper that established interleaved reasoning + tool use as the dominant agent pattern. Short and worth reading directly.
The practical guide to tool schema design. The quality of your tool descriptions matters more than you expect.
The clearest treatment of when to use workflows vs. true agents. Read before committing to an agentic architecture.
The emerging standard for connecting agents to tools and data sources. Understand the protocol before evaluating MCP-compatible tooling.
How context design affects agent reliability: what to include, what to prune, and how to structure multi-turn agent memory.
A concrete multi-agent architecture from Anthropic's engineering team, covering orchestration patterns and evaluation.
The most mature open-source library for building stateful multi-agent graphs. Start here for orchestration implementations.
The most complete hands-on introduction to building and evaluating agents. Covers the full stack from loop mechanics to deployment.
New resources and perspective on building AI-ready data systems, a few times a month. No spam.