Governance, privacy and the DPDP Act

Every data engineer eventually handles personal data: names, phone numbers, email addresses, order histories, locations. How you store, move and share that data is not only a technical matter. It is a legal and ethical one, and in India it is governed by the Digital Personal Data Protection Act, 2023 (DPDP Act).

This lesson covers what data governance is, the practical tools a data engineer uses (catalogs, access control, masking, lineage, retention), and what the DPDP Act means for your pipelines. It is an engineering introduction, not legal advice. For real compliance decisions, involve your company's legal or privacy team and read the official text.

Data governance is the set of rules, roles and tools that decide who can use which data, for what purpose, how long it is kept and how its quality is kept up. Think of a college library: books are catalogued (so you can find them), some are in a reference section you cannot borrow (access rules), old ones are removed after a period (retention), and the librarian knows who is responsible for each section (ownership).

Good governance makes data easier to use safely. Bad governance means people either cannot find data or can see data they should not.

Personal data is any data about an individual who can be identified from it. In Brew & Bean's orders, customer_name and phone are personal data. An order_id alone is not, but it becomes personal data when it can be linked to a person.

Some data needs extra care: payment details, health, biometric data, precise location, and children's data. Whether a specific field counts as sensitive in law depends on the rules that apply, so classify conservatively and ask your privacy team.

The first practical step is an inventory: which tables and columns hold personal data, where did it come from, who uses it and why. You cannot protect what you have not found. Tag columns in your catalog or even in dbt schema.yml with a meta entry, so the information travels with the model.

version: 2

models:
  - name: stg_orders
    description: One row per order, cleaned.
    meta:
      owner: data-team@brewandbean.example
    columns:
      - name: customer_name
        description: Name given at pickup.
        meta:
          classification: personal
      - name: phone
        description: Customer mobile number.
        meta:
          classification: personal
          retention_days: 365
      - name: total
        meta:
          classification: internal

Classification and owner as metadata in dbt. dbt keeps meta values in its generated docs and manifest, so other tools can read them. The retention_days value here is only a label; you still need a job that deletes the data.

The DPDP Act, 2023 is India's law on processing digital personal data. At the time of writing (2026), the Act is in force in stages and its rules set the detailed requirements and timelines. Check the official government sources (MeitY and the Data Protection Board of India) for what applies today. Here are the main ideas that affect data engineers.

Main obligations, in engineering language

- Consent and notice. Personal data is generally processed on the person's consent, which must be free, specific, informed and given by a clear action, with a notice that says what data is collected and why. There are limited "legitimate uses" without consent, such as certain government services or medical emergencies. Your pipeline needs to know which purposes a person agreed to. - Purpose limitation. Use data only for the purpose it was collected for. Pickup orders feeding a delivery tracker is fine. Selling phone numbers to advertisers is not covered by that consent. - Data minimisation. Collect and keep only what you need. If a dashboard needs counts by area, it does not need names. - Accuracy. Keep data complete and correct when it is used for decisions about the person or shared with others. - Storage limitation and erasure. Delete personal data when the purpose is served or consent is withdrawn, unless the law requires you to keep it. - Security safeguards. Take reasonable steps to protect personal data from breaches: encryption, access control, logging, backups. - Breach notification. If a personal data breach occurs, the fiduciary must inform the Board and the affected people in the prescribed way. - Rights of the Data Principal. People can ask for a summary of their data and the processing, request correction and erasure, use a grievance process and nominate someone to exercise rights if they die or cannot act. - Children. For children (under 18 under the Act), verifiable consent of a parent or guardian is required, and tracking, behavioural monitoring and targeted advertising directed at children are restricted. - Cross-border transfers. Transfers outside India are generally allowed except to countries the government restricts by notification. - Penalties. The Act sets financial penalties that can run to hundreds of crores of rupees for serious failures such as inadequate security safeguards. Read the Schedule to the Act for the exact figures.

Least privilege and roles. Give people the minimum access they need, through roles rather than individual grants. An analyst may see order totals but not phone numbers. Review access regularly.

Column masking and row-level security. Many platforms hide or partially show a column depending on the viewer's role, or limit rows to those a user is allowed to see. In Snowflake, for example, you create a masking policy and attach it to a column. Other platforms offer similar features under different names.

CREATE OR REPLACE MASKING POLICY phone_mask AS (val STRING) RETURNS STRING ->
    CASE
        WHEN CURRENT_ROLE() IN ('SUPPORT_LEAD') THEN val
        ELSE CONCAT('******', RIGHT(val, 4))
    END;

ALTER TABLE customers MODIFY COLUMN phone SET MASKING POLICY phone_mask;

Everyone except the SUPPORT_LEAD role sees ******3210 instead of 9876543210. Role names are examples.

Pseudonymisation. Replace a direct identifier with a token so analysts can still count customers or link their orders without knowing who they are. The same input must always give the same token, and the token must not be reversible without a secret key. A plain hash of a phone number is weak: an Indian mobile number has just 10 digits, so there are only a few billion possible values and an attacker can hash them all and look yours up. Use a keyed hash (HMAC) with a secret stored in a secrets manager.

import hashlib
import hmac
import os

SECRET = os.environ.get("PSEUDO_KEY", "change-me").encode()


def pseudonymise(value: str) -> str:
    """Same input always gives the same token, but it cannot be reversed without the key."""
    return hmac.new(SECRET, value.strip().lower().encode(), hashlib.sha256).hexdigest()[:16]


def mask_phone(phone: str) -> str:
    return "******" + phone[-4:]


orders = [
    {"order_id": "A-001", "phone": "9876543210", "total": 330},
    {"order_id": "A-002", "phone": "9876543210", "total": 120},
]
for o in orders:
    print({"order_id": o["order_id"], "customer_token": pseudonymise(o["phone"]),
           "phone_masked": mask_phone(o["phone"]), "total": o["total"]})

Both orders get the same customer_token, so you can count repeat customers without storing the phone number. Set PSEUDO_KEY in the environment; the default is for practice only.

Lineage and catalogs. A catalog lists your tables with descriptions, owners and tags. Lineage shows which sources feed which tables and dashboards. Together they answer "where does the phone number go?" in minutes instead of days. Tools include the lineage in dbt docs, open-source catalogs such as OpenMetadata and DataHub, and the catalogs built into cloud platforms.

Retention and deletion. Write retention into code. A scheduled job deletes raw files and rows older than the retention period. Remember copies: backups, exports, notebooks, log files and analytics tables. Deleting from the main table but not from its copies is a common gap.

Handling an erasure request. The steps below are a typical outline, adapted to your system.

A small team can start simply: a spreadsheet or YAML list of tables with owner and classification, roles instead of shared passwords, masked views for analysts, a retention job for raw data and a short written process for deletion requests. Add a catalog tool when the number of tables makes manual lists unmanageable.

You are a privacy-aware data engineer. I will describe a table. Identify which columns are personal data, which are risky in combination, and suggest for each: keep, mask, pseudonymise, aggregate or drop. Also suggest a retention period to discuss with the privacy team, and what access roles should see raw values.

Table: [name]
Purpose: [why we keep it]
Columns: [list of column names, types and an example value]
Users: [who queries it]

Do not give legal conclusions. Mark anything that needs a lawyer.

What to expect: Check that the answer separates direct identifiers from combinations, and that it defers legal points to a lawyer or privacy team. Use it as a checklist, not as a ruling.

Common questions: "How would you handle personal data in a pipeline?", "How do you delete one user's data from a data lake?", "How do you control who sees what in the warehouse?" and "What do you know about the DPDP Act?" A strong answer names the Data Principal and Data Fiduciary roles, consent and purpose, minimisation, security safeguards, breach notification and erasure, then connects each to practice: classification tags, role-based access, masking, pseudonymisation with a keyed hash, retention jobs, and lineage to find every copy. Honesty helps: say it is not your job to interpret the law, but to build systems that make compliance possible.