Setting Up Queue Workers and Cron Jobs

What this is for#

AcelleMail uses Laravel queues + a system cron to do everything beyond rendering web pages — sending campaigns, processing automations, handling bounces, refreshing stats, dispatching scheduled sends, verifying senders, monitoring subscriptions. If the queue workers aren't running or the cron isn't firing, the web UI looks fine but nothing actually happens.

This guide walks the production-correct setup as Acelle's own scaffolding defines it: a two-tier supervisor split with different queue priorities, a system cron line, and the operational discipline to keep it healthy.

Prerequisites: a working AcelleMail install per the Ubuntu 24.04 canonical guide (Steps 1-7 are enough; this article details Step 8 — supervisor — and Step 9 — cron).

Step 1 — System cron#

This is the simpler one. Add a single one-minute cron line, run as the web user:

sudo crontab -u www-data -e
# Add:
* * * * * cd /var/www/acellemail && php artisan schedule:run >> /dev/null 2>&1

Rocky / RHEL: sudo crontab -u nginx -e instead of www-data.

The Laravel scheduler runs once per minute and fans out to AcelleMail's recurring tasks. Per routes/console.php, these are what AcelleMail actually schedules:

Verify cron is wired:

sudo -u www-data crontab -l | grep schedule:run
# Should print the cron line.

# Watch the scheduler tick in real-time (Laravel logs to storage/logs/laravel.log):
sudo tail -f /var/www/acellemail/storage/logs/laravel.log
# After 60 seconds you should see the scheduler entries.

If cron isn't firing, the most common cause is the wrong user — the cron must be in www-data's crontab (Ubuntu/Debian) or nginx's (Rocky), not your sudo user's.

Step 2 — Supervisor (the two-tier split)#

Supervisor keeps the queue worker processes alive — restart on crash, restart on deploy, automatic startup on boot.

sudo apt install -y supervisor
sudo systemctl enable --now supervisor

Rocky: sudo dnf install -y supervisor && sudo systemctl enable --now supervisord — note the trailing d in the service name.

Why two tiers, not one big pool#

Acelle ships two distinct supervisor configs (resources/documents/supervisor_master_config.tmpl + supervisor_worker_config.tmpl) for a reason: import jobs are slow and CPU-heavy (parsing 200k-row CSVs, deduping against existing subscribers), while send jobs are fast and I/O-bound (one HTTP call to SES per email). Mixing them in a single pool means a single import job blocks 5 send jobs behind it — sending throughput collapses during list imports.

The fix: dedicate a small pool (2 workers) to import,default queues, and a larger pool (15 workers) to the send + automation queues.

Master pool — for imports + default#

sudo tee /etc/supervisor/conf.d/acellemail-master.conf <<'CONF'
[program:acellemail-master]
process_name=%(program_name)s_%(process_num)02d
command=/usr/bin/php /var/www/acellemail/artisan queue:work --queue=import,default --tries=1 --max-time=180
autostart=true
autorestart=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/var/log/supervisor/acellemail-master.log
stopwaitsecs=3600
CONF

Worker pool — for sends + automations#

sudo tee /etc/supervisor/conf.d/acellemail-worker.conf <<'CONF'
[program:acellemail-worker]
process_name=%(program_name)s_%(process_num)02d
command=/usr/bin/php /var/www/acellemail/artisan queue:work --queue=high,batch,single,automation,automation-dispatch --tries=1 --max-time=180
autostart=true
autorestart=true
user=www-data
numprocs=15
redirect_stderr=true
stdout_logfile=/var/log/supervisor/acellemail-worker.log
stopwaitsecs=3600
CONF

Apply + start#

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start acellemail-master:* acellemail-worker:*
sudo supervisorctl status
# Expect: 2 master processes + 15 worker processes, all RUNNING

Rocky: drop the configs at /etc/supervisord.d/*.ini (not conf.d/*.conf) and adjust the user= to nginx.

Why --tries=1 --max-time=180?#

These are Acelle's chosen defaults from the shipped templates, and they're right:

• --tries=1 — a single attempt per job. Email-sending jobs aren't safely retryable (you might double-send to a subscriber); Acelle handles retries explicitly at the campaign level for the few cases where it's safe.
• --max-time=180 — each worker exits after 180 seconds. Supervisor immediately restarts it. This bounds memory growth (PHP doesn't always release memory cleanly across requests) and ensures workers pick up any code changes from the most recent deploy within 3 minutes.

If you increase --max-time, you increase the worst-case memory footprint per worker. Don't go above 600s unless you've profiled memory.

Step 3 — Sizing the pools for your workload#

The 2 + 15 default from Acelle's templates is the Medium tier baseline (4 vCPU / 8 GB RAM). Adjust per your hardware:

Memory math: each worker peaks at ~256-512 MB on heavy campaigns. With numprocs=15, worst case is ~7.5 GB just for workers — that's why Medium tier is 8 GB RAM minimum. If your server can't fit the recommended pool, scale down (and accept slower throughput) rather than OOM-killing workers mid-campaign.

CPU math: each worker is mostly I/O-bound (waiting for the sending API), so they share CPU cores well. 15 workers on 4 vCPU is fine; 30 workers on 4 vCPU starts to context-switch-thrash.

Auto-scaling: the queue:adjust scheduled command launches up to 20 extra worker processes when the priority-queue depth crosses a threshold (defined per-campaign via the "Priority" setting in admin). This is opportunistic, not a substitute for sizing the steady-state pool correctly.

Step 4 — Restart on deploy#

Every code deploy (patch upgrade, manual git pull, Docker image rebuild) requires restarting all workers — otherwise they keep running the old code from when they started:

sudo supervisorctl restart acellemail-master:* acellemail-worker:*

Make this the last step of every deploy script. The stopwaitsecs=3600 in the configs gives in-flight jobs up to an hour to finish gracefully before being killed (campaigns in progress aren't interrupted).

Step 5 — Monitoring#

The five things worth watching:

# 1. Workers running?
sudo supervisorctl status
# All entries should be RUNNING; any FATAL or BACKOFF needs investigation

# 2. Queue depth (Redis driver)
redis-cli llen queues:default
redis-cli llen queues:high
redis-cli llen queues:import
redis-cli llen queues:automation
# Steady-state should be 0-100. Sustained > 1000 means workers are
# under-provisioned for the load. Spiking > 10k during a campaign is
# normal; should drain within a few minutes.

# 3. Failed jobs (one-time check + ongoing)
cd /var/www/acellemail && sudo -u www-data php artisan queue:failed
# Should be empty on a healthy install

# 4. Worker logs
sudo tail -f /var/log/supervisor/acellemail-worker.log
# Watch for "Failed:" or "Error:" lines

# 5. Cron last run (via Laravel scheduler log)
sudo tail -50 /var/www/acellemail/storage/logs/laravel.log | grep -E "Scheduled|cronjob"

For production, wire these into your monitoring stack:

• Redis queue depths via a 1-minute Prometheus exporter or DataDog integration
• Worker count via supervisorctl status | grep -c RUNNING — alert if < expected
• Failed-jobs count → daily Slack notification

Step 6 — Diagnose common failures#

FAQ#

Can I use systemd instead of supervisor? Technically yes — systemd has Restart=always and can manage worker pools. Operationally, supervisor's numprocs + per-process logs + supervisorctl restart group:* ergonomics are noticeably better for queue-worker management. Stick with supervisor unless you have a specific reason not to.

Why not use Laravel Horizon? Horizon is a great web UI for Redis-backed queues — auto-balancing, real-time dashboard, failed-job UI. It's not bundled with AcelleMail; you'd install it as a Composer dependency. It works alongside Acelle's supervisor setup but doesn't replace the supervisor configs (Horizon needs supervisor to run its main process). If you're operating multiple AcelleMail installs or you want better visibility, Horizon is worth the setup overhead.

What happens if a worker dies mid-job? Supervisor restarts it within seconds. The job that was in-flight is re-enqueued only if it's marked as such by Laravel — with --tries=1, it'll instead show up in failed_jobs. For email-sending jobs this is correct behaviour (better one missed send than a duplicate).

Can I run workers on a separate server? Yes — point them at the same Redis + MySQL as the web server. This is the standard pattern at Large tier — see Scaling for 100K+ Emails Per Day.

Why does queue:adjust use a custom01 queue I've never heard of? It's an opt-in priority queue for campaigns marked "Priority" in admin. If you never use the priority setting, queue:adjust is a no-op (it sees no jobs on custom01 and exits). Keep the scheduled command — it's cheap and the feature is useful.

What queue driver should I use? Redis (QUEUE_CONNECTION=redis in .env). The database driver works but creates excess load on MySQL — at any reasonable send volume, Redis is significantly faster. See Redis for Queue Processing.

Can I skip supervisor and just run workers in a screen/tmux session? For development, yes. For production, no — they'll die when the session disconnects and won't restart on reboot. Supervisor exists for exactly this.

Related articles#

• Install AcelleMail on Ubuntu 24.04 LTS — sets up the prerequisites (this is Step 8 + 9 of that guide, expanded)
• Redis for Queue Processing — Redis tuning (recommended queue driver)
• Automated Database Backups — companion ops task
• Scaling for 100K+ Emails Per Day — when to add more workers + split off DB
• Post-Install Hardening Checklist — secure the queue + cron infra
• Docker Deployment Guide for AcelleMail — Docker equivalent of supervisor (compose services)
• Server Requirements and Hosting Options — pool sizing reference table