SKILL.md
529 lines · 1,711 words
Data Catalog Builder
Overview
A data catalog is the organized inventory of data assets in an organization, making data discoverable, understandable, and trustworthy. This skill covers building and operating data catalogs that serve as the single source of truth for metadata, lineage, ownership, and data governance across the enterprise.
Catalog Architecture
+-------------------+
| Search & UI |
| (Discovery) |
+--------+----------+
|
+--------v----------+
| Metadata Store |
| (Graph DB / ES) |
+--------+----------+
|
+------------------+------------------+
| | |
+-------v------+ +-------v------+ +--------v-----+
| Ingestion | | Lineage | | Governance |
| Connectors | | Tracker | | Engine |
+-------+------+ +-------+------+ +--------+-----+
| | |
+-------v------+ +-------v------+ +--------v-----+
| Data Sources | | Orchestrators| | Policies |
| (DBs, Lakes, | | (Airflow, | | (Access, |
| APIs, Files)| | Spark, dbt) | | PII, Class.)|
+--------------+ +--------------+ +--------------+