How data catalogs (Dataplex, DataHub, Alation) ground AI agents in warehouse structure: semantic descriptions, lineage, ownership, and the catalog as a knowledge base.
The data catalog is the knowledge base for warehouse agents. Without it, a text-to-SQL agent knows the table names and column types (from the schema) but not what they mean, why they exist, who owns them, or how they relate to business concepts. The catalog is where that semantic layer lives. Schema RAG is retrieval over your catalog. The quality of your catalog is the ceiling on your agent quality.
Dataplex Knowledge Catalog is Google Cloud's managed approach: automatic schema discovery from BigQuery, Pub/Sub, Cloud Storage, and GCS, combined with manual business metadata enrichment. The key insight from the Dataplex documentation is that catalog metadata is a first-class data asset with its own access controls, lineage tracking, and quality metrics. Treat your catalog entries with the same rigor as your data models.
Business metadata is the hardest part and the most valuable. Technical metadata -- schema, data types, row counts, freshness timestamps -- can be harvested automatically. Business metadata -- what this table represents in the business, what the column values mean, which teams use it for which decisions, which metrics it feeds -- requires human knowledge and cannot be automated. The teams that invest in business metadata get AI-grounded agents; the teams that skip it get hallucinated schema.
Lineage is underused as an agent grounding signal. If a text-to-SQL agent knows not just the schema but the lineage -- this table is derived from these sources, transformed by this dbt model, consumed by these dashboards -- it can generate SQL that respects the data's provenance. "Show me revenue" is safer if the agent knows that the revenue table is the governed, dbt-tested version, not the raw events table that would require custom logic to interpret.
The catalog as a RAG knowledge base is a practical implementation pattern: embed table and column descriptions, index them in a vector store, retrieve the most relevant ones for each incoming question, include them in the schema context. This is schema linking operationalized. Tools like DataHub and Alation expose APIs for this; you build the retrieval pipeline on top. The catalog is the data; the agent is the consumer.
An AI agent querying a warehouse is only as capable as the metadata it retrieves before generating SQL or constructing a retrieval query. If the catalog entry for "orders" says only "order records" with no further context, the agent has no basis for deciding which of the four orders tables to use, what the status column values mean, or which join keys connect it to the customer table. Catalog quality is not a hygiene concern -- it is the primary determinant of whether warehouse agents produce usable results or confidently wrong ones.
This is measurable directly. Build a test set of natural language questions paired with the correct SQL for your warehouse, run the agent on them, and track accuracy as you improve catalog metadata. Teams that instrument this test harness consistently find that improving table and column descriptions raises accuracy more than upgrading the underlying LLM. The catalog is the highest-ROI investment in text-to-SQL quality, and the one most frequently deferred. A warehouse with 1,000 tables and complete, accurate business metadata will outperform a warehouse with 1,000 tables and empty column descriptions, regardless of which model is doing the SQL generation.
DataHub, the open-source metadata platform originally built at LinkedIn, is the right choice for engineering-driven teams that need deep API access and integration flexibility. It supports lineage ingestion from Airflow, Spark, dbt, and Kafka natively via a push-based model, exposes a GraphQL API for programmatic access, and can be self-hosted at no licensing cost. DataHub requires operating Elasticsearch, MySQL or PostgreSQL, and optionally Kafka for real-time metadata events -- the operational overhead is real, but it gives teams the most control over the metadata model and the most extensive ecosystem of open-source integrations.
Alation is the commercial option targeting data governance at enterprise scale, with a focus on analyst-facing features: a search-first UI, crowdsourced trust signals (analysts can flag tables as trusted, deprecated, or under review), and automated query log analysis that surfaces which tables and columns are actually used in practice. Its behavioral intelligence layer -- mining historical query patterns to weight search results -- is the strongest differentiation from DataHub and Dataplex, and it is particularly valuable for large organizations where knowing which tables are actively maintained versus abandoned saves analysts significant investigation time.
Dataplex Knowledge Catalog, Google Cloud's offering, is the natural choice for GCP-native data platforms. Automatic metadata discovery from BigQuery, Cloud Storage, and Pub/Sub eliminates most of the instrumentation burden. Dataplex integrates with Cloud Data Loss Prevention (DLP) for automatic PII classification and tagging, which is valuable for compliance-sensitive warehouses. The access control model inherits GCP IAM directly, which reduces the operational surface area significantly compared to a separately-managed catalog. For teams already operating entirely on Google Cloud, the zero-configuration metadata collection is the primary advantage.
Google Cloud's managed catalog shows what a production metadata layer looks like: automatic technical metadata discovery, business metadata enrichment, fine-grained access controls, and lineage tracking. The architecture applies regardless of platform.
New resources and perspective on building AI-ready data systems, a few times a month. No spam.