Data Engineer Agent for Claude Code: Stop Reinventing Your Pipeline Architecture

Every senior data engineer has lived this moment: a stakeholder needs a new data pipeline, you’re juggling three ongoing incidents, your cloud bill just hit a record high, and someone in a Slack thread is asking why a dashboard metric doesn’t match the source system. The work itself isn’t hard — it’s the volume and the context-switching that kills you.

The Data Engineer agent for Claude Code addresses this directly. It gives you a senior-level collaborator that already knows the tradeoffs between Hudi and Iceberg, understands partition pruning strategies, can design a Snowflake ingestion architecture from scratch, and will implement data quality validation rules without being asked twice. It doesn’t need to be taught what slowly changing dimensions are or why exactly-once processing matters in stream pipelines. You describe the problem; it starts building.

For teams where data engineers are outnumbered by data consumers, or where a single engineer is expected to handle everything from Kafka stream processing to Airflow DAG optimization to Redshift query tuning, this agent compresses hours of architecture design and implementation into a focused technical conversation.

When to Use the Data Engineer Agent

This agent is purpose-built for specific categories of data work. Reach for it when you’re dealing with:

• New pipeline design: You need to ingest from multiple source systems into a warehouse or lake and want a complete architecture — extraction strategy, transformation logic, orchestration pattern, monitoring setup — without starting from a blank document.
• ETL/ELT implementation: You have requirements but need working code: Spark jobs, Airflow DAGs, dbt models, or Flink streaming pipelines with proper error handling and retry logic built in.
• Performance and cost optimization: Your pipeline runtimes are growing, your cloud bill doubled, or queries against your warehouse are timing out. You need someone who can look at the problem systematically across storage, compute, and query layers.
• Data quality frameworks: Dashboards are showing inconsistent numbers, engineers don’t trust the data, and there’s no monitoring in place. You need validation rules, SLA tracking, freshness checks, and anomaly detection stood up properly.
• Data lake architecture: Deciding between Delta Lake, Hudi, and Iceberg; designing partitioning strategies for your S3 data; setting compaction policies — decisions that have long-term cost and performance consequences.
• Streaming pipelines: Real-time event processing with Kafka or Flink where windowing strategies, backpressure handling, and schema evolution need to be considered from the start.
• Dimensional modeling: Designing star schemas, handling slowly changing dimensions, or reviewing whether your current model is causing the performance problems you’re seeing.

Real-World Examples

Scenario 1: Multi-Source ETL into Snowflake

A developer on a growing e-commerce platform needs to consolidate sales data from three systems into a single warehouse for analytics.

Developer: “We need to create an ETL pipeline that ingests daily sales data from our Salesforce, Shopify, and internal databases into Snowflake. It needs to run every 6 hours and maintain data quality.”

Data Engineer Agent: “I’ll design a comprehensive data pipeline architecture for your requirements. Let me analyze your data sources, define the extraction strategies, create transformation logic with quality checks, and set up orchestration for the 6-hour cadence. I’ll start with source connectivity — Salesforce via the REST API using incremental watermark extraction, Shopify via webhook capture plus bulk API fallback, and your internal databases using CDC or timestamp-based polling depending on your schema. Transformation will run through a staging layer with schema enforcement before loading into Snowflake, with completeness, uniqueness, and referential integrity checks at each stage. I’ll scaffold the Airflow DAGs with retry logic, alerting on SLA breach, and a data freshness monitor that flags any table that hasn’t updated within the expected window.”

Scenario 2: Pipeline Performance and Cost Optimization

A data team is watching their cloud bill climb and their pipeline runtimes stretch into hours.

Developer: “Our data pipelines are taking 3 hours to complete, and our cloud bill has doubled. We need to optimize performance and reduce costs without losing data quality.”

Data Engineer Agent: “I’ll analyze your pipeline performance, implement query optimization, optimize storage through intelligent partitioning and compression, right-size compute resources, and set up cost monitoring. I’ll start by profiling your current DAGs to identify the long-tail tasks — typically it’s either a Spark job doing a full scan where partition pruning should be applied, or a Snowflake query missing clustering keys. On the cost side, I’ll audit your storage tiering, check whether you’re running oversized clusters for your actual data volumes, and identify candidates for spot instance usage in non-critical processing windows. I’ll also add cost attribution tags per pipeline so you can see exactly what each workflow is spending.”

Scenario 3: Data Quality and Observability

Analytics consumers are losing confidence in the numbers, and the data team has no visibility into why.

Developer: “We’re getting complaints about data accuracy in our analytics dashboards. Some metrics are inconsistent and we have no visibility into data pipeline health. We need comprehensive data quality checks and monitoring.”

Data Engineer Agent: “I’ll implement a data quality framework with validation rules for completeness, accuracy, and consistency. I’ll set up monitoring for pipeline SLAs, data freshness, and anomalies. I’ll create dashboards that give your team real-time visibility into pipeline health and send alerts before your stakeholders notice problems. The framework will include row count reconciliation between source and destination, null rate tracking for critical fields, cross-table consistency checks for shared metrics, and statistical anomaly detection for volume and value drift. For monitoring, I’ll instrument each pipeline stage with data freshness timestamps and SLA breach alerting, and expose a health dashboard that shows the last successful run, record counts, and quality check pass rates per table.”

What Makes This Agent Powerful

Full-Stack Data Engineering Knowledge

The agent’s system prompt covers the complete data engineering stack without gaps. It reasons across Apache Spark, Kafka, Flink, and Beam for processing; Snowflake, BigQuery, Redshift, and Azure Synapse for warehousing; Airflow, Prefect, Dagster, and Step Functions for orchestration; and Delta Lake, Hudi, and Iceberg for lakehouse table formats. It doesn’t treat these as isolated tools — it understands how they interact and where each fits in a given architecture.

Built-In Quality Gates

The agent operates against a specific data engineering checklist: 99.9% pipeline SLA, data freshness under one hour, zero data loss, and consistent quality check passage. These aren’t aspirational — they’re the defaults the agent designs toward. Cost per TB optimization and monitoring coverage are treated as first-class requirements, not afterthoughts.

Systematic Development Workflow

Rather than jumping straight to code, the agent starts with architecture analysis: source assessment, volume estimation, velocity requirements, SLA definition, and cost targets. This prevents the common failure mode of building a pipeline that works at current scale but breaks at 10x. The implementation phase follows with extraction strategies, incremental processing patterns, error handling, and retry mechanisms already considered.

Cost Awareness as a Design Principle

Storage tiering, compute right-sizing, partition pruning, data compression, spot instance usage, and query optimization are built into how the agent thinks about every solution. It doesn’t optimize for cost only when asked — it treats cost as a dimension of correctness alongside performance and reliability.

Documentation and Governance

The agent includes documentation and governance in its checklist. Solutions come with explanations of design decisions, not just implementation code, which matters when you need to hand off a pipeline or justify architecture choices during an incident post-mortem.

How to Install

Installing the Data Engineer agent takes under a minute. In your project root, create the directory and file at this path:

.claude/agents/data-engineer.md

Paste the full agent system prompt into that file and save it. That’s the complete installation. Claude Code automatically discovers and loads agent files from the .claude/agents/ directory when it starts. No configuration files to update, no commands to run.

Once installed, invoke the agent in Claude Code by referencing it directly in your prompt:

Use the data-engineer agent to design an incremental ingestion pipeline 
from our PostgreSQL transactional database into BigQuery with dbt transformations.

You can commit the .claude/agents/ directory to your repository so the agent is available to every engineer on the team without individual setup.

Conclusion and Next Steps

The Data Engineer agent is most valuable when you treat it as a technical peer rather than a code generator. Give it real constraints — your actual SLA requirements, your current cloud costs, your existing stack — and engage with its architecture proposals critically. The agent will surface tradeoffs you may not have considered, but you still own the final design decisions.

Start with one of the three scenarios that maps most closely to your current work: a new pipeline, a performance problem, or a data quality gap. Run through a complete end-to-end interaction, review the generated architecture and code, and push back where the assumptions don’t fit your environment. You’ll calibrate quickly on where the agent adds the most leverage for your specific context.

If you’re managing multiple data pipelines, consider installing the agent across your data platform repositories so any engineer can invoke it during development without needing to find the right person to review architecture decisions. The agent’s checklist and systematic workflow enforce consistency across pipelines even when different engineers are writing them.

Agent template sourced from the claude-code-templates open source project (MIT License).