Apache Spark and PySpark basics

What you will build: a small PySpark job that reads Brew & Bean orders and the menu from CSV files, joins them, calculates revenue per category and per day, and writes the result as partitioned Parquet files.

Prerequisites: Python basics and the SQL from *Advanced SQL* in this track. You also need Python 3.9 or newer and a Java runtime (Spark runs on the JVM; check the Spark install page for the Java versions your PySpark release supports). If installing Java is a hassle, run the same code in a free Google Colab notebook, where you only need pip install pyspark.

A normal Python script (or pandas) runs on one machine and keeps data in that machine's memory. When data reaches hundreds of GB or more, one machine becomes too slow or runs out of memory.

Apache Spark splits the data into pieces called partitions, sends each piece to a different CPU core (or machine) and runs the same operation on all of them at once. You write code as if you were working on one big table. Spark does the splitting, scheduling and recovery from failures.

Think of a kitchen at lunch rush. One cook making 1,000 sandwiches takes all day. Ten cooks each making 100 finish in a tenth of the time, provided someone hands out the work and collects the results. Spark is that manager.

Spark has four main parts you will meet: Spark SQL and DataFrames (what this lesson uses), Structured Streaming (covered in *Stream processing* later), MLlib for machine learning, and the core engine that runs everything. PySpark is simply the Python API.

A Spark application has a driver (your Python program, which plans the work) and executors (workers that process partitions). In local mode, which we use here, the driver and executors are all threads on your laptop. On a cluster they are separate machines, but your code does not change.

Two ideas explain most Spark behaviour:

Lazy evaluation. Operations like filter, select, join and groupBy are transformations. They do not run; they only add a step to a plan. Work starts when you call an action such as show(), count(), collect() or write. Because Spark sees the whole plan first, its optimiser (Catalyst) can reorder and simplify it, for example by reading only the columns you need.

Shuffles. Some steps (groupBy, join, orderBy) need rows with the same key to be on the same machine. Spark must move data across the network to do that, which is called a shuffle. Shuffles are the main cost in most Spark jobs, so fewer shuffles usually means a faster job.

order_id,customer_name,item_id,qty,created_at
1,Asha,cappuccino,2,2026-10-01 09:15:00
2,Ravi,espresso,1,2026-10-01 09:40:00
3,Meera,croissant,2,2026-10-01 18:05:00
4,Asha,masala-chai,1,2026-10-02 08:30:00
5,Kiran,cold-brew,1,2026-10-02 17:45:00
6,Ravi,veg-sandwich,1,2026-10-02 19:10:00
7,Meera,filter-coffee,3,2026-10-03 08:55:00
8,Asha,banana-bread,2,2026-10-03 16:20:00

One row per order line, using the Brew & Bean menu ids. Eight rows over three days.

item_id,name,category,price
espresso,Espresso,coffee,120
cappuccino,Cappuccino,coffee,180
cold-brew,Cold Brew,coffee,220
filter-coffee,Filter Coffee,coffee,80
masala-chai,Masala Chai,tea,90
croissant,Butter Croissant,bakery,150
banana-bread,Banana Bread,bakery,130
veg-sandwich,Veg Grilled Sandwich,food,160

The Brew & Bean menu with prices in rupees.

from pyspark.sql import SparkSession, functions as F

spark = (
    SparkSession.builder
    .appName("brew-and-bean")
    .master("local[*]")          # use all cores on this machine
    .getOrCreate()
)

orders = spark.read.csv("orders.csv", header=True, inferSchema=True)
menu = spark.read.csv("menu.csv", header=True, inferSchema=True)

orders.printSchema()
orders.show(3, truncate=False)

Save this as pipeline.py and keep adding to it. SparkSession is the entry point to everything in Spark.

The DataFrame API reads much like SQL. Each step returns a new DataFrame; DataFrames are never changed in place. Add this to pipeline.py.

lines = (
    orders
    .join(menu, on="item_id", how="inner")
    .withColumn("order_date", F.to_date("created_at"))
    .withColumn("revenue", F.col("qty") * F.col("price"))
)

by_category = (
    lines.groupBy("category")
    .agg(F.sum("revenue").alias("revenue"))
    .orderBy(F.desc("revenue"))
)
by_category.show()

by_day = (
    lines.groupBy("order_date")
    .agg(
        F.countDistinct("order_id").alias("orders"),
        F.sum("revenue").alias("revenue"),
    )
    .orderBy("order_date")
)
by_day.show()

Joins menu prices onto each order line, then totals revenue two ways. Nothing runs until show() is called.

Check one number by hand: Asha's two cappuccinos are 2 x 180 = 360, Ravi's espresso is 120 and Meera's two croissants are 2 x 150 = 300, so 1 October is 780. Always sanity-check a pipeline against a small case you can calculate yourself.

Spark SQL and the DataFrame API compile to the same plan, so use whichever is clearer. Register a DataFrame as a temporary view and query it with SQL.

lines.createOrReplaceTempView("lines")

top_items = spark.sql("""
    SELECT name, SUM(qty) AS units, SUM(revenue) AS revenue
    FROM lines
    GROUP BY name
    ORDER BY revenue DESC
    LIMIT 3
""")
top_items.show()

Window functions, CTEs and everything from the Advanced SQL lesson work here too.

(
    lines
    .select("order_id", "customer_name", "item_id", "name", "category",
            "qty", "price", "revenue", "created_at", "order_date")
    .write
    .mode("overwrite")
    .partitionBy("order_date")
    .parquet("out/order_lines")
)

# Read it back to prove it worked
back = spark.read.parquet("out/order_lines")
print(back.count())              # 8
back.filter(F.col("order_date") == "2026-10-02").show()

spark.stop()

Writes one folder per order_date. Parquet is a compressed, columnar format; see the next lesson on file formats.

Look inside out/order_lines. You will see folders named order_date=2026-10-01, order_date=2026-10-02 and so on, each holding one or more .parquet files. This is partitioning: a query that filters on order_date reads only the matching folders and skips the rest. Choose a partition column that you filter on often and that has a modest number of distinct values (dates are the classic choice; a column like customer_name would create far too many tiny folders).

mode("overwrite") replaces the old output; other modes are append, ignore and error (the default, which fails if the output exists).

by_category.explain()                 # shows the physical plan
print(orders.rdd.getNumPartitions())    # how many pieces Spark split the data into
print(spark.conf.get("spark.sql.shuffle.partitions"))   # default 200

explain() prints the steps Spark will run. Look for Exchange: that word marks a shuffle. Run this before spark.stop().

The default of 200 shuffle partitions is sensible for big data but wasteful for our 8 rows. In a tiny local job you can set spark.conf.set("spark.sql.shuffle.partitions", "4") to make it faster. In recent Spark versions, Adaptive Query Execution (AQE) is on by default and adjusts such settings while the job runs.

lines = orders.join(F.broadcast(menu), on="item_id", how="inner")

A broadcast join: menu has 8 rows, so copy it to every executor instead of shuffling orders.

I am learning PySpark. Explain this physical plan line by line in plain English. Tell me where the shuffles are and one change that could reduce them.

[paste the output of df.explain()]

What to expect: A good answer names the Exchange (shuffle) steps, explains what each scan, filter and aggregate does, and suggests something specific such as filtering earlier, broadcasting a small table or selecting fewer columns.

Common questions: the difference between a transformation and an action; what a shuffle is and why it is expensive; narrow versus wide transformations; how to handle a skewed key (one key has far more rows than the others); when to use repartition versus coalesce (repartition can increase or rebalance partitions with a shuffle, coalesce only reduces them and avoids a full shuffle); and why Parquet is better than CSV for analytics. Be ready to explain lazy evaluation using the plan-then-execute idea above.