01
02
03
04
05
06
Connect Claude to your Airflow REST API and dbt artifacts so that when a pipeline fails, an agent automatically fetches the task logs, parses the failed dbt model SQL, diffs the source table schema against what the model expects, traces the dbt dependency graph backwards from the failure point, and delivers a root-cause brief to Slack before the on-call engineer has opened a single log file. The brief includes the failure class, the exact cause, confidence level, and a specific fix.
1–3 hours per pipeline failure investigation compressed to under 5 minutes
Engineering
Claude
Airflow REST API credentials (read access to DAG runs and task logs), dbt Cloud API token or access to local dbt artifacts (manifest.json, run_results.json, compiled SQL), read-only data warehouse credentials for information_schema queries, Claude API key, Slack webhook URL for the on-call channel
Test on a synthetic Airflow failure in a dev environment before connecting to production. Manually remove a column from a test source table and confirm the schema diff correctly identifies it as the cause. Verify the Slack brief appears within 5 minutes of a triggered failure. Test at least one upstream failure scenario to confirm the dependency tracer correctly identifies the root node rather than the downstream symptom.
Requires read-only access to the production data warehouse — provision a dedicated audit role with your DBA rather than reusing an existing service account. Cannot diagnose failures caused by external API timeouts, network partitions, or cloud provider outages without additional monitoring integration. dbt Cloud API access may require a Team or Enterprise plan; dbt Core users can use local artifact files instead. Schema diff only covers structural changes — data quality failures such as wrong values, nulls, or duplicates require dbt test output to diagnose accurately. This workflow is scoped to dbt task failures in Airflow — non-dbt operators (PythonOperator, BashOperator, SparkSubmitOperator) do not have a dbt model to parse and will only receive log-based diagnosis without schema diff or dependency tracing.
Stack: Airflow REST API (failure detection and log retrieval), dbt artifacts or dbt Cloud API (model SQL and dependency graph), data warehouse information_schema (schema diff), LangGraph (agent orchestration), Anthropic SDK (Claude Sonnet for diagnosis and fix generation), Slack API (brief delivery). The schema diff step is the single highest-value addition over naive log analysis — schema drift is a frequent root cause of production pipeline failures and this detects it in seconds rather than minutes of manual investigation.