Example · Data Engineer
70% of your week as a data engineer is still yours.
AI now writes much of the pipeline code. Owning production when it breaks stays with you.
Make my planA typical week at the middle pace. Your plan uses your own tasks.
Is data engineering safe from AI?
In a typical week at the middle pace, 40% of this work needs your judgment and 30% is work you do using AI. AI could do about 30%. Turning a clear spec into pipeline code, configuration and tests is about 30% of your week, and a frontier model with your repo open does that today. If you have had one write the pipeline and dbt models for a new source, you have seen it get most of the way.
Read more
The two parts of the job many data engineers like least, the 3 a.m. page and the regulated file that goes out the door, are the two most protected things you do. The spec-to-code work you may be fastest at is the work a model does best. A model cannot be the named person when the number is wrong at 3 a.m., own the file that goes to a regulator or partner, or know which of a source system's status codes lie.
That can put you in a better spot than most people early in software. Two outside reasons explain it. In regulated industries a named human is answerable for the data you ship, and many security teams will not let a model near production. The second reason may not last. The plan turns protection that comes from policy into protection that comes from your own work.
The week, task by task
Who does each hour of a typical 40-hour week.
See every task
Needs your judgment.
- Novel incident diagnosis: "why is yesterday different," CDC gaps, silent duplicates, the 3 a.m. page10%
- Regulated and partner outputs: furnishing file, partner reconciliation, finance close feeds8%
- Schema-change negotiation with app teams; impact analysis; saying "not this sprint"6%
- On-call ownership: named responder, incident comms to finance and partners, postmortems6%
- + 2 smaller tasks9%
AI does the slow parts. You still own it.
- Known-failure triage: reruns, connector resets, backfills after a recognized failure8%
- Warehouse modeling for a new domain: grain, keys, slowly changing dimensions, business-rule edge cases8%
- Data-quality monitoring: writing checks, choosing thresholds, tuning noisy alerts6%
- Code review of teammates' PRs5%
- + 1 smaller task4%
AI could do most of it.
- New pipeline code from a clear spec: an ingestion DAG, dbt models for a new source or column16%
- Boilerplate and config: dbt tests and YAML, docs, Terraform for new resources, schema migrations7%
- Routine analyst support: ad-hoc queries, "where does X live," access requests7%
Why some tasks still need a person
Being the person who is paged, and who says what happened afterwards
A runbook can rerun a task. It cannot be the named responder finance escalates to, or write a postmortem a partner will accept. The rotation exists because someone must answer.
Regulated and partner outputs: a credit-bureau furnishing file, a partner-bank reconciliation, finance close feeds
In consumer lending, Reg V (12 CFR 1022.42), the federal rule on furnishing credit data, requires written procedures for the accuracy and integrity of what is furnished to the bureaus, and a partner bank's regulators hold it responsible for your feeds under the 2023 interagency third-party guidance. A wrong file harms consumers and leaves a paper trail, so someone has to sign for it.
Knowing which of the source database's status codes lie
A model reading the schema sees a clean enum. Knowing that CLOSED sometimes means refinanced, and that refinances double-count on the loan key, is in your head, and a wrong answer looks correct until the reconciliation breaks.
"What breaks if we drop this column?"
Lineage tools show edges. They do not show that the finance close, a regulated file and one analyst's forgotten notebook all depend on that column in different ways, or that the app team's PM (product manager) will accept a two-week delay if you ask on Tuesday.
Deciding who may see which columns
The accountability here comes from whoever owns the security program (in financial services, the Safeguards Rule's Qualified Individual). Access reviews and masking policies are a control someone attests to. A model can propose a policy, but it cannot attest to it.
Good to know
What is hard about this field, and what you can do about it.
The hard truth
The plumbing you build is becoming a product vendors sell.
Moving and modeling data used to be custom work. Vendors keep packaging it, and in October 2025 two of the biggest agreed to merge.
Sources (1)
- Fivetran and dbt Labs announced an all-stock merger, with about $600 million in combined annual recurring revenue and 10,000+ customers. Technical.ly, 2025-10
What's next for a data engineer?
Plan A is the best move inside the work you do now.
Grow in the job you have
The main focus. Most weekly sessions build toward it.
- Each week
- 4 sessions
- Ready in
- About 3 to 4 months
Why it fits
Why you. Things break on your watch, so you know what breaks, and usually nobody keeps a page log. If you want to stop being the on-call person forever, this is the way out.
The gap. You need to learn SLO design (written reliability targets), error budgets and postmortem practice. You also need to show finance that "the number by 8" is a service level with a cost. The free Google SRE (site reliability engineering) books cover the mechanics.
The trade-off. Your name goes on the outages, including the ones an agent caused. On a small team this is a half-role; it only works paired with the regulated-output reviewer move. And if the company hires a Head of Platform from outside, "reliability" can read as keeping the lights on.
2 other moves that fit
- Be the reviewer of record for the regulated outputsReady in about 4 to 6 months
- Own the platform for your core source system (the hedge)Ready in about 6 to 9 months
What's a plan B for data engineers?
Plan B is meant to be different from this job. It was picked because skills from this role carry over.
Get ready for a new kind of job
A second option. A newer job AI is creating that these skills already fit.
You already build the checks that catch bad data before anyone sees it, and scoring model outputs runs on the same pipelines and SQL.
- Each week
- 1 session
- Outlook
- Hiring now
- Pay
- $150.4K to $385K a yearTwo postings: Apple (Oct 2026, base pay) and Perplexity (now closed).
2 other options
- AI cost and vendor analystYour warehouse and compute bills are already yours to explain, so tracking model and token spend is a short step with some finance practice.
- Context engineer and knowledge curatorKnowing which source tables are trustworthy and how they join is the core of curating what an AI agent is allowed to know.
In your own plan you can switch to any of the 47 roles.
The 13 weeks
Small steps that add up. One plan for both: most sessions build Plan A, one a week builds Plan B.
By week 13 you will have
AFor Plan A
- A six-month page log with draft SLOs
- A written control document for your regulated file
- A one-page memo asking to own platform reliability
BAnd for Plan B
- A regression table with a significance check
- A published eval write-up for your portfolio
- 1Measure your week
- 2Test an AI tool safely
- 3Decide what should page
- 4Your on-call week
You finish withA one-page list of controls that already need a human sign-off
- 5Map the rules on your file
- 6Add checks with reasons
- 7Propose a safe AI setup
- 8Build a gate and decide
You finish withA written decision on which move to focus on
- 9Draft a data contract
- 10Your second on-call week
- 11Add up your results
- 12Make your ask
- 13Choose your next step
You finish withA one-paragraph decision with a date
Three courses from the plan
- Site Reliability Engineering; The Site Reliability WorkbookGoogle, free online at sre.google/books · Free
Yes, Weeks 2, 3 and 6. It is the highest-value reading in this report, and it is free.
- FTC Safeguards Rule: What Your Business Needs to KnowFederal Trade Commission · Free
Yes, Week 4, if you work in financial services.
- 12 CFR 1022.42 and Appendix E (Reg V furnisher accuracy and integrity)eCFR · Free
Yes, Week 5, if your company furnishes to the credit bureaus.
Checked 2026-09-22.
The fuller picture
Risks beyond AI: moderatePolicy and cycle risk
Two risks here have nothing to do with what models can do. First, part of your protection is your employer's security policy (no model on production data, no agent with credentials), and policy changes in one meeting. Second, headcount in sectors like lending tracks interest rates and funding. An unfilled junior seat is as likely to come from that cycle as from the AI vision doc.
Where this could be wrong
- The shares are guesses until Week 1. If your log shows much more spec-to-code work, the share of your week tools can do passes 40% and junior openings shrink faster.
- A good part of the work that holds up is protected by policy, and policy is a decision. If security approves an in-boundary model next quarter, re-run this report; diagnosis and triage move first.
- We assumed a regulated employer with an output someone must sign for. If your company has no regulated file, the reviewer-of-record move does not exist and owning the source platform is the plan. If your company is public, SOX IT controls, from the federal financial reporting law, add a second anchor and this report is too pessimistic.
- Our estimate of how well AI diagnoses new kinds of failures is uncertain in both directions. With log and lineage access, agents diagnose most known-pattern failures today; on a change-data-capture (CDC) gap in a database whose status codes lie, the evidence is thin.
Also matched: data engineer, analytics engineer, etl developer. Research checked 2026-09-22. Page updated .