Our Symfony platform at work has a large, ~3,500, PHPUnit suite. To keep pull request checks fast, we’d ended up with a fairly elaborate GitHub Actions setup.
There was a setup job to build and cache Composer dependencies, followed by a split job that listed every test, ran a 400-line PHP script to divide them into groups, and built a SQLite test database.
Finally, a 40-way matrix of GitHub-hosted runners would download the database artifact and execute its share of the tests.
It worked, and it was fast. The tests completed in around two minutes.
The problem was everything required to make that happen.
Each of those 40 runners needed to start, install PHP, restore Composer dependencies and download the database before running any tests. Even though the pipeline only took about two minutes of wall-clock time, we were consuming roughly 80 runner-minutes per run, before accounting for the setup and split jobs.
Our GitHub Actions bill was starting to exceed $200 per month, and we had several hundred lines of code dedicated to deciding which tests should run where.
There had to be a simpler way.
One big machine instead of 40 small ones
Rather than distributing tests across 40 GitHub-hosted runners, we decided to run everything on one self-hosted EC2 runner and let ParaTest
We chose an AWS c7i.8xlarge, with 32 vCPUs and 64 GiB of RAM.
The idea was straightforward: instead of GitHub Actions managing 40 separate jobs, we’d give ParaTest the resources to run the suite in parallel within a single job.
Our workflow became considerably simpler:
1phpunit: 2 runs-on: [self-hosted, phpunit] 3 timeout-minutes: 30 4 steps: 5 - uses: actions/checkout@v4 6 7 - name: Install Composer packages 8 run: composer install -q --no-ansi --no-interaction --no-scripts --no-progress --prefer-dist 910 - name: Build SQLite database11 run: composer init-test-db1213 - name: Prepare databases for parallel runners14 run: composer prepare-test-db1516 - name: Reset kernel cache17 run: rm -rf var/cache1819 - name: Run ParaTest20 run: |21 vendor/bin/paratest \22 --configuration=phpunit.xml.dist \23 --testsuite=all \24 --functional \25 --processes=$(nproc)
This is the non-coverage path, simplified from the real workflow. Pull requests also run ParaTest with Xdebug coverage enabled, using XDEBUG_MODE=coverage and --coverage-clover.
The SQLite database is still created, but it stays on the same machine rather than being uploaded as an artifact and downloaded by 40 separate jobs.
We also deliberately clear Symfony’s kernel cache after preparing the databases. Otherwise, some code that only executes when Doctrine builds SQL or metadata for the first time could be missed by the coverage report.
The change removed our custom test-splitting script, the SQLite artifact hand-off and the Composer cache job.
529 lines out, 31 in.
Our coverage reporting, coverage ratchet and pull request comments remained in place.
Parallel tests aren’t always independent
Moving everything onto one machine introduced some problems.
We had tests which weren’t properly isolated from each other. For example, one test might upload a file, while another test would delete it as part of its cleanup.
When those tests ran concurrently, the second test could delete the file before the first had finished using it.
The first test would then fail, even though there was nothing wrong with the functionality it was testing.
These sorts of issues are particularly frustrating because the tests can pass individually and fail intermittently when run together.
The old setup had divided tests into separate groups running on isolated GitHub runners. That separation had hidden some of the assumptions in our tests.
With ParaTest running many processes on one machine, we had to think more carefully about shared resources.
Database isolation was part of that, which is why we retain a step to prepare databases for parallel execution.
But databases aren’t the only shared resource. Tests can also interfere with each other through uploaded files, temporary directories and cleanup operations.
The important lesson was that parallel execution requires test isolation, regardless of how many machines are involved.
More CPU doesn’t fix tests that interfere with each other.
We didn’t want a 32-vCPU instance running all day
An EC2 instance of that size isn’t particularly cheap if it’s left running continuously.
Our tests only run when they’re needed, so paying for the machine to sit idle would defeat much of the point of moving away from GitHub-hosted runners.
We needed a way to start the instance automatically when a job was queued, then shut it down once there was nothing left to do.
We built a small system around GitHub webhooks, AWS Lambda and a systemd timer.
Starting EC2 when GitHub needs it
GitHub supports a workflow_job webhook event, which can notify an external service when a workflow job is queued.
We configured an organisation-level webhook to call an AWS Lambda Function URL.
The Lambda validates the webhook’s HMAC signature against a secret stored in AWS Secrets Manager.
Once the request is verified and the runner needs to be started, Lambda calls the EC2 StartInstances API.
The process looks like this:
1GitHub Actions queues a job 2 | 3 v 4GitHub workflow_job webhook 5 | 6 v 7AWS Lambda 8 - Validate webhook signature 9 - Start EC2 instance10 |11 v12EC2 boots13 |14 v15Self-hosted runner connects to GitHub16 |17 v18Runner picks up the queued job19 |20 v21ParaTest executes the test suite
The self-hosted runner starts automatically when the instance boots and connects to GitHub using its configured runner label.
GitHub can then assign the waiting PHPUnit job to it.
In testing, it took approximately 27 seconds from waking the instance to the runner starting the job.
That’s a reasonable trade-off for avoiding the cost of leaving a large EC2 instance running continuously.
Shutting down when the work is finished
Starting the instance was only half of the problem.
We also needed to make sure it stopped when no longer required.
I didn’t want to add a shutdown step to the GitHub Actions workflow itself. A cancelled workflow, failed job or unexpected error could prevent that step from executing.
Instead, the instance manages its own shutdown using a systemd timer.
The timer checks for the GitHub Actions Runner.Worker process, which indicates whether a job is actively executing.
Once no worker process has been running for five minutes, the instance powers itself off.
This gives the machine a little time to pick up another queued job without repeatedly shutting down and starting back up.
It also means that if a workflow fails or is cancelled, the instance should still shut itself down once it becomes idle.
The important distinction is that we’re stopping the EC2 instance, not terminating it. Its configuration and runner installation remain available for the next time GitHub needs it.
Securing the runner
There’s an obvious security consideration when allowing GitHub to trigger an EC2 instance.
The Lambda endpoint is publicly reachable, so it must verify that requests genuinely came from GitHub.
We do that using the webhook signature and a secret stored in Secrets Manager, rather than trusting the incoming request without validation.
The instance itself has no inbound ports open. For administration, we use AWS Systems Manager rather than exposing SSH.
We also need to be careful about which repositories and workflows are allowed to execute on the self-hosted runner.
A self-hosted runner executes code on infrastructure we control, so running untrusted pull request code carries more risk than using an isolated GitHub-hosted runner.
Finally, the runner’s operating system, dependencies and GitHub Actions runner software still need maintaining. That’s work GitHub previously handled for us.
Self-hosting moves those responsibilities to us rather than making them disappear.
The results
The original workflow was already fast, so reducing its runtime wasn’t our primary goal.
What we wanted was to reduce costs and simplify the pipeline without making developers wait longer for their tests.
The results were:
- Over $200 per month in GitHub Actions costs before the change, with a substantial reduction in GitHub-hosted runner usage.
- Approximately two-minute test execution, broadly matching the previous 40-runner approach.
- Around 27 seconds to wake the EC2 instance and start a queued job when the runner is stopped.
- Automatic shutdown after five minutes of inactivity, avoiding the cost of running the instance all day.
- 529 lines removed and 31 added in the workflow simplification.
- No custom test-splitting script or database artifact hand-off.
- Existing coverage reporting and the coverage ratchet retained.
The full cost saving needs to account for EC2 compute, storage and the supporting AWS infrastructure, rather than simply treating the old GitHub Actions bill as money saved.
But we’ve replaced a complicated 40-runner setup with a much simpler test job and an on-demand machine.
When this makes sense
I wouldn’t necessarily recommend self-hosted runners for every project.
For a small PHPUnit suite, GitHub-hosted runners are convenient, and maintaining your own infrastructure probably isn’t worth it.
But if you’re running a large test suite, especially one that can make good use of ParaTest, it’s worth considering whether distributing jobs across dozens of runners is actually the most efficient approach.
There are trade-offs.
You need to maintain the machine, secure it, consider how parallel tests share resources and decide what happens when the runner is unavailable.
For us, the interesting part wasn’t making PHPUnit faster.
We were already running the tests in two minutes. We just didn’t need 40 GitHub runners to do it.