Running tests in CI with GitHub Actions

In this tutorial you will set up continuous integration (CI): a robot that runs your tests every time someone pushes code or opens a pull request. You will start with a 15-line workflow for pytest, then add a version matrix, a Postgres database, an end-to-end job and a nightly schedule.

Prerequisites: a GitHub account (free), Git installed, and a Python project with pytest tests. If you don't have one, the project in the *Project: a full test suite for a coffee shop API* lesson is ideal. For the Playwright section, see *End-to-end testing with Playwright*. GitHub Actions is free for public repositories; private repositories get a monthly free allowance that changes over time, so check GitHub's billing page.

A test suite protects you only if it runs. People forget, run only the tests they think matter, or have a different Python version from the rest of the team. CI fixes this: the same commands run on a clean machine, for every change, and the result is visible to everyone on the pull request.

The CI loop is simple. You push a branch. GitHub starts a fresh virtual machine, checks out your code, installs dependencies and runs your test command. If the command exits with an error, the run is red and the pull request shows a failed check. If it exits cleanly, the run is green.

name: tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        run: pytest -v --junitxml=report.xml

      - name: Keep the test report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: junit-report
          path: report.xml

Runs on every push to main and every pull request. Installs Python 3.12 with pip caching, runs pytest and saves a JUnit XML report even if the tests fail.

Read it top to bottom. on says when to run. runs-on: ubuntu-latest picks the machine. actions/checkout@v4 downloads your code, because the machine starts empty. actions/setup-python@v5 installs Python, and cache: pip reuses downloaded packages between runs so it is faster. The run: steps are ordinary shell commands, exactly what you would type in a terminal.

if: always() on the last step means "run this even if an earlier step failed". Without it, a failing test would stop the job before the report is saved, and the report is the thing you want most when tests fail.

Make a test fail on purpose: change an expected value in one test, commit and push. In the Actions tab the run turns red. Click it, click the failed job, then expand the failed step. You see the same output pytest prints locally: the assertion, the values and the line number.

When a test passes locally but fails in CI, the cause is almost always a difference between the two machines. Check these in order.

I have a GitHub Actions job that fails. The test passes on my laptop (Windows, Python [version]). The workflow file and the failing log are below. Explain the most likely cause in plain English, then give me the smallest change to fix it. List two other causes to rule out.

--- workflow ---
[paste .github/workflows/tests.yml]
--- log (last 50 lines of the failed step) ---
[paste log]

What to expect: A good answer points to a specific line in the log, explains the laptop/CI difference and proposes a small change. Verify by re-running; do not accept a fix that only deletes the test.

If your code must work on more than one Python or Node version, a matrix runs the same job once per value, in parallel. fail-fast: false lets the other versions finish even if one fails, so you see the whole picture.

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.11", "3.12"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: pip
      - run: pip install -r requirements.txt
      - run: pytest -v

Creates two parallel jobs, one on Python 3.11 and one on 3.12.

Integration tests (see *Integration tests with a real database*) need a real database. GitHub Actions can start one next to your job as a service container. The health options make the job wait until Postgres accepts connections, so your tests don't start too early.

jobs:
  integration:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: test
          POSTGRES_DB: brew
        ports:
          - 5432:5432
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s
          --health-timeout 5s
          --health-retries 10
    env:
      DATABASE_URL: postgresql://postgres:test@localhost:5432/brew
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt
      - run: pytest tests/integration -v

Starts Postgres 16, waits until it is healthy, then runs the integration tests with DATABASE_URL pointing at it. From the job, the database is on localhost:5432.

End-to-end tests are slow. Run them in a second job that starts only if the fast job passed, using needs. This gives quick feedback first and saves minutes of machine time when a unit test is already red. This job installs Playwright's browsers and uploads the HTML report only when something fails.

jobs:
  e2e:
    runs-on: ubuntu-latest
    needs: test            # only run the slow job if the fast one passed
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 7

Runs after the test job succeeds. npx playwright install --with-deps downloads browsers and their Linux dependencies. The report is kept for 7 days when the job fails.

A healthy pipeline is shaped like the test pyramid: unit tests first (seconds), then API and integration tests (a minute or two), then a small set of end-to-end tests. Put load tests (see *Performance and load testing with k6*) and security scans (see *Security testing basics with OWASP ZAP*) on a schedule rather than on every pull request, because they are slow and need a deployed environment.

on:
  push:
    branches: [main]
  pull_request:
  schedule:
    - cron: "30 20 * * *"   # 20:30 UTC = 02:00 IST, every night
  workflow_dispatch:         # adds a "Run workflow" button

Adds a nightly run and a manual "Run workflow" button. Cron times are in UTC.

When tests call a deployed staging API, you need a URL and a token. Store them in the repository under Settings, Secrets and variables, Actions. Use secrets for sensitive values (they are masked in logs) and variables for plain settings such as a URL. Read them in the workflow like this.

      - name: Run API tests against staging
        env:
          API_BASE_URL: ${{ vars.STAGING_URL }}
          API_TOKEN: ${{ secrets.STAGING_API_TOKEN }}
        run: pytest tests/api -v

Passes a variable and a secret to the test process as environment variables. Your tests read them with os.environ.

A red run is useless if people can merge anyway. In the repository settings on GitHub, add a branch protection rule (or ruleset) for main and require the status check named after your job (here, test) to pass before merging. The exact menu names change from time to time, so look for "Branches" or "Rules" under Settings.

A few habits keep CI trusted:

- Fix or quarantine flaky tests quickly. A test that fails at random teaches the team to ignore red. Don't hide flakiness with automatic retries until you understand the cause (shared state, fixed sleeps, real network calls). - Keep the pipeline fast. Cache dependencies, run jobs in parallel and keep end-to-end tests few. Under ten minutes is a good target for a pull request check. - Fail loudly. Don't add || true or continue-on-error: true to make a job green.