At work, our main application is a Symfony API running in PHP-FPM containers on AWS ECS (Fargate). For a long time, those services scaled the way most tutorials tell you to: add a task when average CPU or memory crosses a threshold.
It didn’t work. When a traffic spike hit, the application would fall over before ECS had even decided to scale. To cope, we kept a minimum of 8 containers running all the time, which is expensive insurance for a problem it didn’t actually solve.
Why CPU and memory were the wrong signal
A PHP-FPM container serves requests with a fixed pool of worker processes. Once all workers are busy, new requests queue, and if they queue for long enough, the load balancer gives up on them.
The trouble is that a busy worker isn’t necessarily using much CPU. Plenty of our requests spend most of their time waiting on the database or another API. So you can have every worker tied up, requests timing out, and CPU sat at a comfortable 40%.
Here is how bad that gets. During load testing, I pulled the per-minute CPU for two very different states of the same service. A healthy fleet averaged 41.1%. A fleet that was actively timing out requests averaged 43.2%.
Two percentage points. That’s the entire signal you get between “fine” and “falling over”.
The average hides it because the work is distributed unevenly. In the same minutes, individual containers were pegged at 100% while others sat near idle, and the mean of those looks perfectly relaxed. Target tracking scales on the mean. So it did nothing.
What we actually cared about was simple: how many FPM workers are busy?
Publishing FPM active processes to CloudWatch
PHP-FPM can expose a status page with exactly that number in it (active processes). The work was getting it into CloudWatch so ECS could scale on it.
We added a small PHP script that runs alongside FPM under supervisord in every container. Every 10 seconds it:
- reads the FPM status page directly over FastCGI (no web server in the way),
- works out which ECS cluster and service it belongs to from the task metadata endpoint,
- publishes
FpmActiveProcessesto CloudWatch as a high-resolution metric.
1$client->putMetricData([ 2 'MetricData' => [[ 3 'Namespace' => 'Example/Application', 4 'MetricName' => 'FpmActiveProcesses', 5 'Dimensions' => [ 6 ['Name' => 'ClusterName', 'Value' => $cluster], 7 ['Name' => 'ServiceName', 'Value' => $service], 8 ], 9 'Value' => (float) $status['active processes'],10 'Unit' => 'Count',11 'StorageResolution' => 1,12 ]],13]);
Three details mattered more than I expected.
It doesn’t boot Symfony. The metric matters most when the application is struggling, so the publisher must keep reporting even when the app can’t serve traffic.
It catches its own failures and carries on. When we first shipped it to production, the task role was missing cloudwatch:PutMetricData, so every call was rejected. Because the script logs the error and keeps looping, the moment the permission was granted, it started publishing on the containers that were already running. No redeploy.
And every container publishes its own datapoint under the same service. CloudWatch’s average across tasks then gives “busy workers per container”, which falls as tasks are added.
One quirk worth knowing: the metric never reads zero. The publisher polls the status page over FastCGI, and that request is itself an active process, so the floor is exactly 1. If you set a scale-in threshold at or below 1, scale-in will never fire, and your fleet will ratchet up forever. I know this because I nearly did it.
Step scaling, not target tracking
My first instinct was a target tracking policy. Pick a target number of busy workers, let AWS do the rest.
For our custom FPM metric, target tracking’s managed alarms couldn’t provide the 10-second evaluation period we wanted. Step scaling gave me control over the alarm configuration and the size of each scaling adjustment.
Step scaling lets you bring your own alarm, and the alarm can be high-resolution. Ours:
1scale out FpmActiveProcesses > 5 2 period 10s, 2 of 3 datapoints 3 +50% / +100% / +200% / +300% 4 at 0 / 2.5 / 7.5 / 17.5 over threshold 5 cooldown 60s 6 7scale in FpmActiveProcesses < 1.5 8 period 60s, 15 evaluation periods 9 -25%10 cooldown 180s
Five active workers might sound low when each container has a pool of 40, but that was deliberate. With new containers taking over a minute to become available in our earlier tests, we wanted to scale long before the existing workers became saturated.
The step adjustments matter as much as the speed. A burst that takes you slightly over threshold gets 50% more containers. One that takes you a long way over gets 300%. You don’t crawl up one task at a time while requests pile up.
On the old CPU policy, the first scaling action after a load test began arrived 5 minutes 20 seconds in. With the high-resolution alarm, I measured the gap between the alarm changing state and Application Auto Scaling acting on it at 0.2 seconds. Capacity was in place 73 seconds later, almost all of which was container boot.
That shifts the whole problem. AWS acts almost immediately once the alarm fires, and most of the remaining delay is waiting for new containers to become available.
Making new containers start faster
So we looked at boot time. Measured on a staging environment:
- Boot time: 56s to 33s
- Image pull: 27s to 4.7s, by publishing a SOCI index
- Image size: 1.59GB to 1.33GB. A recursive
chownin the Dockerfile was rewriting every application file into a new layer, so the image carried the whole application twice - OPcache preload: 600 bytes to 68MB. Our configuration was wrong, so PHP had been silently ignoring it
The deploy that wouldn’t deploy
The first production release of all this failed and rolled back, which was a useful lesson in measuring the right thing.
The containers were fine. They were serving 200s within 52 seconds. But the target group was configured with healthy_threshold = 4 and a 10-second interval, so a container had to pass four consecutive checks before the load balancer would call it healthy. That’s 40 seconds of passing checks on top of the 52 to start.
ECS gives a new task a grace period before it will act on health check failures. Ours was 30 seconds.
The task reached its third passing check at 72 seconds and was killed. The fourth, which would have marked it healthy, would have landed at 80.
It missed by eight seconds.
We raised the grace period to 120 seconds and brought the thresholds in line with our other environments (5-second interval, healthy_threshold = 2). Time to healthy went from 67–82 seconds down to 32–57. The same deploy went out cleanly twenty minutes later.
Worth noting that healthy_threshold and unhealthy_threshold pull in opposite directions here. 4 × 10s and 8 × 5s are both a 40-second failure budget, so switching to the faster interval cost us nothing on that side while halving the time to recognise a good container.
What we got wrong
The scale-in policy, twice.
We started at < 2.5 over 5 evaluation periods. The fleet promptly started shedding containers in the middle of busy periods, because FPM sits near its floor between bursts and five minutes of “quiet” is easy to find at our average requests a minute. We got nine capacity changes in 45 minutes, and one of them removed a container ninety seconds before a burst arrived. p95 went to 21.5 seconds, and we served our only 5xx of the day.
Lowering the threshold to 1.5 and stretching the window to 15 evaluation periods fixed it. The fleet held steady for five hours through the busiest part of the day, then came down cleanly in the evening.
But that created a subtler problem, which took another week to notice. Scale-in triggers when the per-container average drops below the threshold, so the condition is really “total work below 1.5 × containers“. At nine containers, we were averaging around 1.66 active processes each. That’s low utilisation, but still above our 1.5 scale-in threshold. As a result, the fleet couldn’t shrink until traffic dropped further. Because every container also contributes a baseline reading of one, the threshold needs to account for both actual application work and the monitoring request itself.
It self-corrects overnight, so we’ve left it. But if you build this, be aware that a per-container metric with a fixed threshold has a dead zone, and it gets wider the bigger your fleet is.
One more thing that wasn’t scaling at all
While investigating, we found production was using round_robin to distribute requests across containers.
Round robin distributes requests evenly. It does not distribute work evenly, and when some of your requests take 30 seconds, one unlucky container ends up holding several of them while its neighbours idle. On staging, switching to least outstanding requests took p95 from 6.16s to 3.52s and collapsed the spread in per-container CPU from about 48 percentage points to 7.
The results
The floor went from 8 containers to 3, and it’s been there for a month. On a typical weekday, the fleet now sits between 3 and 6, and scales to 9 when something expensive arrives.
The day we dropped the floor, two production releases went out onto a 3-container fleet during the morning ramp, and the FPM listen queue never left zero. Over 1,020 samples of that queue across two days, it was non-zero exactly three times, always with a value of 1. One request waiting, briefly.
A few days later, 6 containers absorbed a substantial requests-per-minute peak with a better p95 than 8 containers had been giving us. That’s the part I’d point at if you’re wondering whether this is worth doing. Not the cost saving. The fact that a smaller fleet, scaled on the right signal, outperformed a larger one scaled on the wrong one.
Scale on the thing that actually runs out. For a PHP-FPM app, that’s workers, not CPU.