File and table formats: Parquet, Delta Lake and Iceberg

Every data pipeline stores data somewhere, and the format decides how fast queries run, how much storage costs and whether you can safely update the data. This lesson explains the three names you will hear constantly: Parquet (a file format), and Delta Lake and Apache Iceberg (table formats built on top of Parquet files).

You should know basic SQL and have read *OLTP vs OLAP: databases, warehouses, lakes and lakehouses*. The Spark lesson in this track already wrote a Parquet dataset.

A CSV file stores data row by row: all the fields of order 1, then all the fields of order 2. This is natural for writing, and for OLTP work such as "fetch order 42". Analytics asks different questions: "total revenue by day". That needs only two columns (created_at, revenue) out of perhaps fifty, but a row format makes you read every byte of every row.

A columnar format stores all values of one column together: all the qty values, then all the price values. A query that touches two columns reads only those two. Values in one column look alike (a column of dates, or of category names), so they also compress very well. Think of a bookshelf sorted by one attribute: to find every book's year of publication you read one shelf, not every page of every book.

Apache Parquet is the standard columnar file format for analytics. Spark, pandas, DuckDB, BigQuery, Snowflake, Athena and almost every other data tool read and write it. Inside a file the data is split into row groups (a block of many rows). Within a row group each column is stored as a column chunk, compressed, with statistics such as the minimum and maximum value. The schema and statistics sit in a footer at the end of the file.

Those statistics give you a big speed-up called predicate pushdown: for WHERE revenue > 1000, the reader looks at each row group's maximum revenue and skips any group where it is below 1000 without reading it. Together with reading only the needed columns (column pruning) this makes Parquet queries far cheaper than CSV.

import pandas as pd
import pyarrow.parquet as pq

# 100,000 fake order lines
df = pd.DataFrame({
    "order_id": range(100_000),
    "item_id": ["espresso", "cappuccino", "masala-chai", "croissant"] * 25_000,
    "qty": [1, 2, 3, 1, 2] * 20_000,
    "created_at": pd.date_range("2026-01-01", periods=100_000, freq="min"),
})

df.to_csv("orders.csv", index=False)
df.to_parquet("orders.parquet", index=False)      # needs: pip install pandas pyarrow

# Read only two columns: Parquet skips the rest
small = pd.read_parquet("orders.parquet", columns=["item_id", "qty"])
print(small.head(3))

# Peek at the file's structure
pf = pq.ParquetFile("orders.parquet")
print(pf.metadata.num_row_groups, "row group(s)")
print(pf.schema_arrow)

Writes the same data as CSV and Parquet, reads two columns back, and prints the schema stored in the Parquet footer. Install with pip install pandas pyarrow.

ls -lh orders.csv orders.parquet

Compare the sizes. The Parquet file is usually clearly smaller than the CSV (about 3 times smaller on a test run of this data); the ratio depends on the data.

Two practical points. First, file size matters: very small files (a few KB each) make reads slow because each file has overhead, while huge files limit parallelism. Aim for files of roughly hundreds of MB in big data systems, which is a common guideline rather than a hard rule. Second, partitioning (folders such as order_date=2026-10-01/) lets engines skip whole folders, as seen in the Spark lesson.

A folder of Parquet files is just files. That causes trouble as soon as more than one job touches it:

No transactions. If a job fails halfway through writing 20 files, readers see 12 new files and 8 missing. A half-finished write corrupts the table.

No updates or deletes. Parquet files are immutable. To correct one order you must rewrite a whole file and hope no query reads it at that moment. Deleting a customer's data (for example for the DPDP Act) means finding and rewriting every file that contains it.

No history. If a bad job overwrites yesterday's data, it is gone.

Slow listing and fragile schema changes. Query engines list thousands of files to plan a query, and changing a column's type can break old files.

Warehouses solved these years ago, but they store data in their own closed format. The goal of a table format is to give a plain pile of Parquet files in cheap object storage (S3, Google Cloud Storage, Azure storage) the same abilities as a warehouse table. This is what makes the lakehouse from the earlier lesson possible.

A table format adds a layer of metadata on top of the data files. The metadata records which files currently make up the table. Readers consult the metadata, not a folder listing. A writer adds new data files first, then commits by atomically publishing a new version of the metadata. Until the commit, readers keep seeing the old version, so a failed job leaves the table unchanged. Every commit is a new snapshot, so old versions can be read too.

The three main open table formats are:

All three give you:

ACID transactions. Writes either fully happen or do not happen. Concurrent writers are handled safely (usually with optimistic concurrency: a commit fails and retries if someone else changed the same data).

Updates, deletes and MERGE. Row-level changes with SQL, for example MERGE INTO to upsert (see *Advanced SQL*). Under the hood the format writes new files and records which old files are no longer part of the table.

Time travel. Query the table as it was at an earlier version or time, and roll back a bad write.

Schema evolution. Add, rename or widen columns without rewriting all the data.

Better query planning. File-level statistics in the metadata let engines skip files without listing folders.

Iceberg has some extra features worth knowing: hidden partitioning (you partition by a transformation such as day(created_at) and users query by created_at without knowing the partition columns) and partition evolution (change the partitioning later without rewriting old data). Iceberg also leans on a catalog (a service that tracks the current metadata for each table) and is designed to be read by many different engines, while Delta Lake is most deeply integrated with Spark and Databricks. In practice the gap between the formats keeps narrowing, and tools exist to read one format as another, so check each product's current documentation before choosing.

The code below runs locally with PySpark and the delta-spark package. Install with pip install delta-spark (it brings a matching PySpark version). It shows an update and time travel on the Brew & Bean menu.

from delta import configure_spark_with_delta_pip, DeltaTable
from pyspark.sql import SparkSession

builder = (
    SparkSession.builder.appName("delta-demo").master("local[*]")
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    .config("spark.sql.catalog.spark_catalog",
            "org.apache.spark.sql.delta.catalog.DeltaCatalog")
)
spark = configure_spark_with_delta_pip(builder).getOrCreate()

path = "tmp/menu_delta"
menu = spark.createDataFrame(
    [("espresso", "Espresso", 120), ("cappuccino", "Cappuccino", 180),
     ("masala-chai", "Masala Chai", 90)],
    ["item_id", "name", "price"],
)

# Version 0: create the table
menu.write.format("delta").mode("overwrite").save(path)

# Version 1: raise the espresso price to 130
table = DeltaTable.forPath(spark, path)
table.update(condition="item_id = 'espresso'", set={"price": "130"})

# Look at the history
table.history().select("version", "operation").show()

# Time travel: read version 0 (price still 120)
old = spark.read.format("delta").option("versionAsOf", 0).load(path)
old.filter("item_id = 'espresso'").show()

# Current data (price now 130)
spark.read.format("delta").load(path).filter("item_id = 'espresso'").show()

Creates a Delta table, updates one row (something plain Parquet cannot do safely), and reads the old version. The first run downloads the Delta jars, so it needs internet.

After running it, look inside tmp/menu_delta. You will find Parquet data files and a _delta_log folder with small JSON files, one per commit (00000000000000000000.json and so on). That log is the table. Notice also that after the update there are more Parquet files than before: Delta wrote a new file with the changed row and marked the old one as removed in the log, but did not delete it yet. That is what makes time travel possible.

For a new project, most teams pick the table format that fits their platform: Delta Lake if they are on Databricks or Microsoft Fabric, Iceberg if they want broad engine support or are on a stack built around it. Both are sound choices. Learn the shared ideas (transaction log or metadata, snapshots, compaction) and you can move between them.

How this comes up in interviews: "Why Parquet over CSV?" (columnar, compressed, schema, pushdown). "What problem do Delta and Iceberg solve?" (ACID, updates, time travel on cheap file storage). "What is the small files problem?" "What does a table format store in its metadata?" "How would you delete one user's data from a lake?" (a DELETE on the table format, then compaction and expiring old snapshots so the old files are physically removed).

You are a data architect. I need to choose a storage format for [describe your data: size, how it arrives, who queries it, which tools and cloud you use]. Compare plain Parquet, Delta Lake and Apache Iceberg for this case. Cover updates and deletes, time travel, tool support, maintenance work and cost. Finish with a recommendation and the main risk of it.

What to expect: A useful answer ties each point to your situation and states trade-offs. Verify tool-support claims in each project's documentation, because support changes quickly.