At work, our platform sends people reminders before their appointments. These aren’t nice-to-haves. If a reminder doesn’t arrive, people miss appointments.
For a long time, those reminders were sent by a cron job running a console command. It found every reminder that was due, then sent each email there and then, one after another.
The problem
That works fine for ten reminders. It doesn’t work for a thousand.
We send thousands of reminders a day. The command runs every hour from 6am to 10pm, matching the time the reminder is sent to the session start time.
A Symfony command is one long-lived PHP process. Doctrine keeps hold of every entity it loads, and sending a reminder touches several of them. None of it gets let go while the command is running. So memory only went up, and the loop got slower with every reminder it processed. Eventually the process ran out of memory and died part-way through the batch.
Everyone before that point got their reminder. Everyone after it didn’t, and nothing told us. We found out when clients told us their users hadn’t been reminded.
That’s what made it dangerous. The job didn’t fail loudly. It just stopped, so there was nothing to alert on.
The underlying problem was that one process was doing two different jobs: deciding who needed a message, and sending each message. Deciding is quick. Sending is slow, it creates a lot of work for Doctrine, and it depends on email and SMS providers that occasionally have a bad moment.
Splitting deciding from sending
Symfony Messenger
1$this->messageBus->dispatch(new SendReminderMessage(2 Appointment::class,3 $appointment->getId(),4 'oneDayReminder'5));
Before, that line was a direct call that built and sent the email there and then.
Dispatching a message takes milliseconds, so the command finishes quickly. Worker processes consume the queue and do the actual sending, one message at a time, for as long as it takes.
The message carries a class name and an ID, not the entity. A worker loads the one record it needs, sends the reminder and finishes. Nothing builds up between reminders, because no single process ever holds more than one reminder’s worth of work.
Retries and priorities
Two bits of configuration made this robust rather than just faster:
1framework: 2 messenger: 3 transports: 4 async-high: 5 dsn: "%env(MESSENGER_TRANSPORT_DSN)%_high" 6 retry_strategy: 7 max_retries: 5 8 delay: 1000 9 multiplier: 210 async-low:11 dsn: "%env(MESSENGER_TRANSPORT_DSN)%_low"12 # same retry strategy13 routing:14 'App\Message\SendReminderMessage': async-high
Retries with backoff. If an email provider has a blip, a failed message is retried up to five times, waiting 1, 2, 4, 8 and 16 seconds. Before, a blip meant a lost email. Now it means a reminder that arrives a few seconds late.
Separate high and low priority queues. Anything a person is waiting on goes on async-high. Reporting syncs and bulk background jobs go on async-low, so a slow background job can never hold up a reminder.
Both queues are Redis streams, and the consumers are separate processes managed by supervisord inside each container.
What we got wrong
The first version had a single queue, and it got overwhelmed. Reporting messages and reminders shared it, so a burst of one sat in front of the other.
So we rewrote it as two queues, one high priority and one low, each with its own consumers. We also gave the consumers more workers, sized to the container they run in.
We’d also only moved the memory problem, not removed it. A Symfony worker is a long-lived PHP process too, so it has the same problem as the command, just slower. If a handler errors and closes the Doctrine EntityManager, every message after it fails as well.
Restarting the worker after every message fixes that, but cold-booting Symfony took up to 12 seconds each time, which is far too slow for a busy queue.
Our fix is a small master script. It forks a child process for each message, so each message gets a fresh kernel and starts with clean memory, but the children inherit the master’s warm OPcache. It’s one pcntl_fork in a loop, and it’s the least elegant part of the setup. It also works.
Knowing when something fails
Before, finding out that a message hadn’t gone meant opening CloudWatch and scrolling through logs looking for mentions of failures.
Now failed messages are reported to New Relic as errors, and we have dashboards for the queue. Every message is named after its class, like Messenger/SendReminderMessage, so it’s clear what is slow or failing. A small command also reports the size of each queue as a metric.
One catch: Messenger wraps handler failures in a single exception class. That meant every failure landed in New Relic as the same error, split into dozens of groups. We now unwrap the exception first, so errors group by what actually went wrong.
A faster API, as a side effect
Once the queue existed, we moved other communications onto it too, like the emails triggered when someone books or cancels. Those used to be sent while the API request was still open, so the user waited for an email provider before seeing a response. Now the request just dispatches a message and returns. That made our API responses up to 30% faster.
The results
- The reminder run no longer does the sending, so it’s far lighter. It decides and dispatches.
- Memory doesn’t build up across a batch, because each message runs in a fresh process.
- Thousands of reminders a day go out through the queue.
- A provider blip no longer loses a reminder.
- Reporting and bulk jobs can’t hold up a reminder.
- Failures show up in New Relic instead of CloudWatch logs.
- API responses are up to 30% faster.
What I’d do again
If a scheduled job does a lot of slow work in a loop, split it in two: a quick job that decides what needs doing, and a queue that does it. Each message gets a fresh process, so you can’t run out of memory partway through. You get retries and visibility for free, and you stop losing work when the batch gets big.
And keep the queues apart. We got there on the second try.