What data engineers do, and how AI changed the job
Understand what a data engineer builds, how the role differs from analysts and data scientists, and what AI tools have and have not changed. Start here before the technical lessons.
Every company that uses data to make decisions, run an app or train an AI model needs that data to arrive in the right place, in the right shape, on time. A data engineer builds and runs the systems that make this happen.
Think of a city's water supply. Rain falls in many places (apps, sensors, payment gateways, spreadsheets). Someone has to collect it, clean it, store it and pipe it to every home. Analysts and data scientists are the people who drink the water. Data engineers build the pipes, treatment plants and reservoirs, and get called when the water turns brown.
What the job actually involves
The core of the job is the data pipeline: code that moves data from a source to a destination and changes it on the way. Around that core sit five recurring responsibilities.
The five jobs of a data engineer
| Responsibility | What it means | Example at Brew & Bean |
|---|---|---|
| Ingest | Get data out of source systems | Copy yesterday's orders from the app database and payment gateway exports |
| Transform | Clean, join and reshape the data | Remove test orders, convert paise to rupees, join orders with the menu |
| Store | Choose where and how data lives | Raw files in cloud storage, tidy tables in a warehouse |
| Serve | Make data easy to use | A daily_sales table the owner's dashboard reads |
| Operate | Keep it running and trustworthy | Alerts when a nightly job fails or revenue suddenly drops to zero |
Notice that the last row is not glamorous but is where most of the time goes. A pipeline that works once on your laptop is easy. A pipeline that runs every night for two years, survives bad input and tells you when it breaks is the real job.
A pipeline end to end
Brew & Bean takes pickup orders through an app. The owner wants to know every morning: how much did we sell yesterday, and which items sell best by hour? The data engineer's answer looks like this.
From order to dashboard
- 1Source: the app's database stores each order. A payment provider also sends a daily settlement file.
- 2Ingest: a scheduled job copies new orders and the settlement file into cloud storage, unchanged. Keeping a raw copy means you can always reprocess.
- 3Transform: a job cleans the data (drops test orders, fixes dates), checks it (no negative quantities, every item exists on the menu) and builds tables such as
orders,order_itemsanddaily_sales. - 4Store: the clean tables are loaded into a warehouse, a database built for analytical questions.
- 5Serve: a dashboard, a report or a machine-learning model reads
daily_sales. - 6Monitor: a check confirms yesterday's data arrived and looks sane. If not, someone is told before the owner opens the dashboard.
How data engineers differ from nearby roles
Neighbouring roles
| Role | Main question | Typical tools |
|---|---|---|
| Data engineer | How do we get reliable data to the people and systems that need it? | SQL, Python, Spark, Airflow, dbt, Kafka, a cloud platform |
| Analytics engineer | How do we turn warehouse tables into clean, documented, trusted models? | SQL, dbt, a warehouse, Git |
| Data analyst | What does the data say about the business? | SQL, spreadsheets, Power BI or Looker Studio |
| Data scientist / ML engineer | Can we predict or automate something with this data? | Python, scikit-learn, notebooks, ML frameworks |
| Backend developer | How does the application work and store its own data? | A web framework, an OLTP database |
In small companies one person does several of these. In large companies they are separate teams. The boundaries move, so learn the shared core (SQL and Python) and be clear about which end you enjoy.
How AI changed the job
AI assistants are now part of daily work for data engineers. They have changed what is quick and what is still hard.
AI as an assistant
| Task | What AI does well | What you must still do |
|---|---|---|
| Writing SQL and boilerplate Python | Drafts queries, schemas and pipeline skeletons quickly | Check the logic on real data. A query can run without error and still count wrong |
| Explaining unfamiliar code or errors | Reads a stack trace or a 300-line query and explains it | Confirm the explanation against the code and the docs |
| Writing tests and documentation | Suggests test cases and column descriptions | Decide which rules matter to the business |
| Pipeline design | Lists options and trade-offs | Choose for your data size, budget, team skills and deadlines |
| Debugging data issues | Suggests places to look | Investigate with the real data. The model has not seen your systems |
The skill that has become more valuable is judgement: knowing what correct output looks like so you can spot a plausible but wrong answer. Data engineers who understand joins, grain (what one row means) and failure modes get more from AI than those who paste and hope.
You are a senior data engineer. I will describe a pipeline. Find the 5 most likely ways it could fail or produce wrong data in production, and for each say how I would detect it.
Source: [e.g. orders table in PostgreSQL, about 5,000 new rows a day]
Schedule: [e.g. runs at 2 am daily]
Steps: [list the steps]
Destination: [e.g. BigQuery table daily_sales]
Ask me questions first if something important is missing.What to expect: A good answer covers late or missing data, duplicate rows on re-runs, schema changes in the source, partial failures and silent wrong numbers, each with a concrete check.
Keep personal data out of prompts
Never paste real customer data, passwords, API keys or company secrets into a public AI tool. Use small made-up samples with the same shape. Under India's DPDP Act, personal data of customers needs careful handling (see *Governance, privacy and the DPDP Act* later in this track).
New work that AI created
AI has also created new data engineering work, not just changed old work.
Where demand is growing
- Pipelines for RAG: collecting documents, splitting them into chunks, creating embeddings and keeping a vector index fresh (covered in *Data engineering for AI* later in this track).
- Unstructured data: PDFs, emails, call recordings and images now flow through pipelines, not just rows and columns.
- Feature pipelines for ML: the same customer facts must be computed identically in training and in live predictions.
- Data quality for models: a model trained on bad data fails quietly, so quality checks matter more.
- Cost control: AI and cloud workloads can be expensive, so engineers watch what each pipeline costs.
How this comes up in interviews
Interviewers for junior data engineering roles usually test SQL first (joins, GROUP BY, window functions), then Python, then simple design questions. Expect questions like these.
Common questions
- "What is the difference between a data engineer, a data analyst and a data scientist?"
- "Walk me through a pipeline you built. What happens if it fails halfway?"
- "How would you make a pipeline safe to re-run?" (Idempotency, covered in the pipeline lessons.)
- "How do you know your data is correct?"
- "How do you use AI tools in your work, and how do you check their output?"
Try it: Draw your own pipeline
Pick any data you know: your college attendance sheet, cricket scores, a food-delivery app, UPI transactions. On paper, write the six stages (source, ingest, transform, store, serve, monitor) for it. For each stage write one specific thing that could go wrong.
Key takeaways
- A data engineer builds and operates the pipelines that move, clean, store and serve data.
- The six stages are source, ingest, transform, store, serve and monitor; reliability is most of the work.
- Analytics engineers, analysts, data scientists and backend developers overlap with data engineers but ask different questions.
- AI speeds up SQL, boilerplate, tests and explanations, but you must verify output on real data. Judgement is now the scarce skill.
- AI has created new work: RAG pipelines, unstructured data, feature pipelines and data quality for models.
- SQL and Python are the shared core, and they come first in interviews.
Check yourself
Finishing the quiz marks the lesson as done.
1. A pipeline copies raw orders to storage unchanged before any cleaning. What is the main benefit?
2. Which question best describes the work of a data engineer, compared with a data analyst?
3. An AI assistant writes a SQL query that runs without errors. What should you do before trusting it?
4. Which of these is new work created by AI rather than old work done faster?
5. Why is monitoring considered part of the data engineer's job?
Saved on this device. Create a free account to keep it on every device.
Practise this with Sam
A real mock interview, out loud, with an honest scorecard. Free to start.