Two Ways to Speed Up E2E Tests: Playwright Workers and a GitHub Actions Matrix

Every engineering team that ships a web product eventually hits the same wall: the end-to-end (E2E) tests become the slowest thing in the pipeline. Unit tests finish in seconds, but the browser-driven suite that clicks through real screens can take tens of minutes — long enough that teams quietly move it out of the pull-request loop and run it once a night, where broken builds sit undiscovered until morning.

The engineering write-up that got us started lays out a complete method for turning that around. This post does three things with it: first it summarises the method and the concrete steps it involves; then it puts the reproducible core of that method on a real machine and measures the results; and finally it offers our own opinion on what is worth adopting and how we will use it.

The method in brief

At heart, the method makes E2E tests run in unit-test time by combining two levels of parallel execution — Playwright workers within a single machine, and a GitHub Actions job matrix across many machines — with the test-structure discipline that makes parallel execution both safe and non-flaky. The framework flag is the easy part; the structure and the CI tuning are what actually make it work. Objectively, the implementation breaks down into these steps:

• Make the tests parallel-safe first. Each test creates its own data through a factory (writing straight to the database) and tears it down afterwards; authentication is generated once and reused via storageState (with several accounts per role so parallel workers never collide); selectors and waits are kept inside a Page Object Model.
• Split the suite into projects by load / release / feature, and set the worker count per profile — lighter UI tests get more workers and full parallelism, heavier backend-bound tests get fewer.
• Turn on Playwright’s parallelism and find the worker count by measuring on the target machine, rather than maximising it blindly.
• Scale out in CI with a GitHub Actions matrix. Shard the suite so each job runs a slice on its own runner (the two-level parallelism), with every job building the full application stack rather than mocks.
• Tune the CI runners: cache the browser binaries, pin tool versions, install the fonts screenshots need, and cancel superseded runs — and expect high parallelism to expose backend limits (connection-pool exhaustion, HTTP keep-alive reuse races) that must be fixed in server configuration.
• Instrument the runners (CPU-pressure and I/O monitors) so parallelism decisions are driven by data, not intuition.
• Enforce anti-flake rules mechanically: ban fixed sleeps in favour of state-based waits, and wait for a positive landmark before asserting that something is absent.
• Optionally, run only the tests affected by a change (a change-to-spec impact map) so the common pull request stays cheap.

That is the method in outline. The rest of this post is us taking the reproducible core of it — the two levels of parallelism and the discipline of measuring — putting it on a real machine, and reporting exactly what happened.

The harness we built

To get clean numbers we needed a suite that behaves like a real one but runs deterministically. We wrote 40 browser tests with Playwright against a small local to-do application, structured the way a healthy E2E suite should be: a Page Object Model to keep selectors out of the test bodies, and — importantly — only state-based waits (wait for an element to appear or a count to change), never fixed sleeps. Hosting the app locally removes network jitter, so any difference in timing comes from the test runner, not the internet.

Lever 1 — more workers on one machine

The first lever is Playwright’s worker count. Two configuration settings do the work: one tells Playwright that every test is independent and may run on its own worker, the other turns the worker count into a knob we can dial from the command line so we can measure each setting.

// playwright.config.ts — the two settings that matter for this experiment
export default defineConfig({
  fullyParallel: true,                 // every test may claim its own worker
  workers: process.env.PW_WORKERS      // make the worker count a tunable knob
    ? Number(process.env.PW_WORKERS)
    : undefined,
  use: { baseURL: 'http://127.0.0.1:4321' },
});

Before trusting any timings, we confirmed the tests genuinely overlap. Running with three workers, the finish order comes out shuffled — test 2 completes before test 1, then 3, then 6 before 5 — which only happens if independent tests are executing at the same time. Every one of the 40 passed.

Then the experiment: we ran the exact same 40 tests seven times, changing only the worker count — from a single worker (fully serial) up to twelve — and recorded the suite time Playwright reports for each.

Plotting the sweep makes the pattern unmistakable. Suite time drops to a clear minimum and then climbs back up. The fastest setting on this machine was two workers — exactly the number of CPU cores available. Push beyond that and the workers simply fight each other for the same two cores; by twelve workers the suite is actually slower than running everything serially.

The lesson from lever 1 is a hard ceiling: worker parallelism is bounded by the number of CPU cores underneath it. A machine can only truly run as many tests at once as it has cores. Ask for more and you get context-switching and memory pressure, not speed. None of this is even safe unless the tests are parallel-ready to begin with — because each of ours owned its own data and waited on real conditions rather than sleeping, every configuration finished green.

Lever 2 — more machines with a GitHub Actions matrix

So one machine tops out at its core count. If you want to go faster than that, you don’t add more workers — you add more machines. This is exactly what a CI matrix is for, and it’s the part of a fast E2E setup that lives entirely in GitHub Actions rather than in Playwright’s config.

The mechanism is Playwright’s –shard flag. Passing –shard=2/4 tells Playwright “run only the second quarter of the tests.” Give each of four CI jobs a different shard and they collectively cover the whole suite, each doing a quarter of the work. We simulated that four-way split locally, giving each shard two workers (as if it were a small two-core runner):

That is the whole point. Because the four jobs run simultaneously on four separate machines, the wall-clock time is not the sum of the shards but the slowest single shard — here about 7.6 seconds, against 19.4 seconds for the best one-machine run. And unlike workers, this keeps scaling: eight shards would roughly halve it again, because you’re adding real cores, not competing for the same ones.

It helps to picture the two levers stacked. GitHub Actions gives you the outer layer — a matrix of jobs, each on its own machine; Playwright gives you the inner layer — workers inside each machine. That is the two-level parallelism that lets a large suite finish in minutes:

Here is a real, working workflow that wires it together. Each shard runs as its own job, caches the browser binaries so it doesn’t re-download them, and writes a machine-readable “blob” report; a final job stitches the four blob reports back into a single HTML report:

name: E2E Tests
on: [pull_request]

# Cancel superseded runs on the same branch so a new push stops the old one.
concurrency:
  group: e2e-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    runs-on: ubuntu-24.04            # one line to pick the machine (2/8/16-core)
    strategy:
      fail-fast: false               # one shard failing shouldn't cancel the rest
      matrix:
        shard: [1, 2, 3, 4]          # 4 jobs, one shard each, on 4 machines
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci

      # Cache the browser binaries so every shard doesn't re-download them.
      - uses: actions/cache@v4
        with:
          path: ~/.cache/ms-playwright
          key: pw-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
      - run: npx playwright install --with-deps chromium

      # Install CJK fonts, or non-Latin text renders as tofu in screenshots.
      - run: sudo apt-get update && sudo apt-get install -y fonts-noto-cjk

      # This job runs ONLY its shard, and writes a machine-readable blob report.
      - run: npx playwright test --shard=${{ matrix.shard }}/4 --reporter=blob
        env:
          PW_WORKERS: 2              # tune to the runner's core count
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report

  merge-report:
    needs: [test]
    if: ${{ !cancelled() }}
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci
      - uses: actions/download-artifact@v4
        with: { path: all-blob-reports, pattern: blob-report-*, merge-multiple: true }
      # Stitch the four shards' blob reports into one HTML report.
      - run: npx playwright merge-reports --reporter=html ./all-blob-reports
      - uses: actions/upload-artifact@v4
        with: { name: html-report, path: playwright-report }

The small CI details that quietly decide whether this works

A matrix that looks simple on paper has a few sharp edges that only appear once ten jobs start at once. These are the ones worth building in from the start:

• Cache the browsers. Without caching ~/.cache/ms-playwright, every shard re-downloads Chromium on every run — minutes of pure waste multiplied by the matrix width.
• Pin the versions of your tooling. When many jobs resolve a “latest” release at the same instant, they can hammer a provider’s API and hit rate limits; a pinned version makes every job deterministic and polite.
• Install the fonts your screenshots need. On a bare runner, non-Latin characters render as empty boxes in screenshot evidence — installing the right font pack (for example Noto CJK) fixes it.
• Cancel superseded runs. A concurrency group keyed on the branch stops an in-flight run the moment a new commit lands, so you don’t pay for stale matrices.
• Merge the reports. Sharding fragments your results across jobs; the blob-report plus merge-reports step gives you one combined HTML report to actually read.

And there is a deeper lesson lurking at higher parallelism that is easy to miss until it bites: the faster your tests hit the backend, the more they expose the backend’s own limits. A suite that was fine at one worker can start returning sporadic 5xx errors at ten, not because the tests are wrong but because a database connection pool is exhausted, or an API gateway is reusing a keep-alive connection the upstream has already closed. The referenced write-up documents chasing exactly this class of failure down to a connection-reuse race — a reminder that at real scale, test parallelism becomes a load test of your own infrastructure, and the fix lives in server configuration, not in the test code.

What we’re taking back to our own work

This started as a small experiment and turned into a mental model we’ll keep using. The headline isn’t “turn on parallelism” — it’s that test speed is something you engineer and measure, choosing the right lever for the job. Here is what we’re carrying into our day-to-day:

• Tune workers to the cores; don’t max them. On a single machine, pin the worker count near the core count — measured on the real runner, not guessed.
• Reach for sharding, not more workers, when you need more speed. More machines keep cutting wall-clock; more workers plateau at the core count. A GitHub Actions matrix is the honest way to go faster.
• Separate the quick tests from the slow ones. Fast UI checks and heavy end-to-end flows have different ideal worker counts and shard well differently; splitting them keeps one slow test from throttling the rest.
• Ban fixed sleeps and isolate each test’s data. These are the preconditions that make any of the above safe — concurrency is only correct when tests never share mutable state and never race a hard-coded timer.
• Build the CI hygiene in from day one: cache browsers, pin versions, cancel superseded runs, and merge shard reports. They’re cheap up front and painful to retrofit.

The most transferable lesson has nothing to do with testing specifically: when a system is slow, identify the actual bottleneck, change one variable, and measure — don’t reason from intuition. We assumed more workers would mean more speed; the numbers said otherwise within a few minutes of looking, and pointed us at the right lever instead. The same discipline pays off on build times, database queries, and API latency just as readily as on a test suite.

At Scuti, this measure-it-yourself habit is close to how we like to build — take an interesting idea, put it on a real machine, and let the evidence decide. Our mission stays the same: to make services people love, by the power of Gen AI.

Reference

This experiment was inspired by an excellent, far more detailed engineering write-up on scaling Playwright test parallelization and tuning GitHub Actions — including the matrix design, the connection-pool and keep-alive fixes, and the instrumentation behind their tuning. It is well worth reading in full: Niwayama Yasuhiro, “E2Eテストをユニットテスト並みの実行時間に—Playwright並列化とGitHub Actionsチューニングの実践” (Zenn).