Reliable background jobs let a web app send email, process imports, generate reports, and run scheduled maintenance without keeping a user’s request open. This guide explains how to design a queue-and-worker system, what reliability signals to track, how to review them on a practical cadence, and when to revisit the design as workload or infrastructure changes.
Overview
A background job is work that can run after the web request has returned a response. Common examples include sending transactional email, resizing uploaded images, importing a customer file, synchronizing records with an external API, creating invoices, and deleting expired data.
The basic architecture has four parts:
- Producer: the web application or scheduler that creates a job.
- Queue: durable storage for jobs waiting to be processed.
- Worker: a separate process that retrieves and executes jobs.
- Observability: logs, metrics, alerts, and a way to inspect failed work.
The queue should be treated as a boundary between user-facing traffic and asynchronous work. A request can validate input, save the necessary state, and enqueue a job. The worker then performs the slower operation independently. This separation protects response times and allows worker capacity to be adjusted without changing the web tier.
Choose a queue service or framework integration that matches your deployment model, programming language, durability needs, and operational experience. A managed queue may reduce infrastructure work, while a database-backed queue can be appropriate for a smaller system that already depends on a relational database. The choice matters less than making delivery, failure, retry, and shutdown behavior explicit.
Before implementation, document the job contract. Define its name, required fields, maximum expected runtime, external dependencies, retry policy, and success condition. Do not place large or rapidly changing objects directly in a message when a stable record identifier will do. A job should usually reference application data that the worker can load and validate at execution time.
What to track
Reliability is easier to maintain when each job type has measurable states. At minimum, distinguish queued, running, succeeded, retrying, failed, and dead-lettered. Record timestamps for enqueue, start, completion, and final failure. These fields help separate queue delay from execution time.
Queue health
- Queue depth: how many jobs are waiting. Track it by queue and job type where possible.
- Oldest job age: how long the least recently queued job has waited.
- Enqueue rate: how quickly new work arrives.
- Dequeue or completion rate: how quickly workers remove work successfully.
- Backlog duration: how long the queue remains above its normal operating level.
Queue depth alone can be misleading. A growing backlog indicates that arrivals are exceeding processing capacity, but a short-lived spike may be harmless if workers catch up quickly. Oldest-job age is often more useful for judging user impact, especially when some jobs are time-sensitive.
Worker health
- Job execution duration: measure typical and slow runs, not only the average.
- Success and failure counts: group failures by job type and error category.
- Retry count: identify jobs that eventually succeed only after repeated attempts.
- Worker availability: confirm that expected worker processes are alive and consuming work.
- Capacity utilization: watch CPU, memory, database connections, external API limits, and other bottlenecks.
- Graceful shutdown results: record whether in-progress jobs finish, are safely returned to the queue, or are lost.
Use structured logs with a job ID, job type, attempt number, correlation ID, and relevant resource identifier. Avoid logging secrets, access tokens, or entire customer payloads. Link worker logs to the originating request when a user action created the job; this makes support and debugging substantially easier.
Failure handling
Retries should be limited and intentional. Retry transient failures such as temporary network errors, rate limits, or a short database interruption. Do not repeatedly retry invalid input, missing records, or a consistently rejected request. Use increasing delays between attempts and add randomization when many workers could retry at the same time.
Every job needs a timeout. Without one, a stalled network call or blocked dependency can occupy a worker indefinitely. Set timeouts at the operation level where possible, then enforce an overall job limit. When a job exceeds its limit, record the reason and release the worker for other work.
After the retry limit is reached, move the job to a dead-letter queue or equivalent failure store. Preserve the original payload or reference, error details, timestamps, and attempt history. Provide a controlled replay process that lets an operator correct the underlying issue before retrying. A dead-letter queue is not a substitute for alerting; it is a place to investigate and recover work.
Duplicate prevention and idempotency
Assume that a job can be delivered more than once. A worker may complete an operation but fail before acknowledging the message, causing the queue to deliver it again. Design the handler so repeating it produces the correct final state rather than duplicate side effects.
Useful techniques include an idempotency key, a unique database constraint, a job-result table, or a state transition that can safely be applied more than once. For example, an email task might store a business event ID and refuse to send the same event twice, subject to the guarantees and limitations of the email provider. For payments, account changes, or external writes, use the provider’s supported idempotency mechanism when available and reconcile uncertain outcomes instead of blindly repeating them.
Cadence and checkpoints
Review background job behavior at three levels: during development, immediately after release, and on a recurring operational schedule.
Before production
- Run a successful job through the complete producer, queue, worker, and completion path.
- Force a transient error and verify that the job retries with the intended delay.
- Force a permanent error and confirm that the job reaches the dead-letter state.
- Terminate a worker during execution and check that the job is not silently lost.
- Submit the same logical event twice and verify idempotent behavior.
- Test timeouts, malformed payloads, missing records, and dependency rate limits.
- Confirm that dashboards and alerts identify the queue, job type, and affected resource.
Include these scenarios in automated tests where practical. For a broader release review, use the app release readiness checklist alongside job-specific checks.
After a release
Compare queue age, execution duration, failure categories, and resource consumption with the previous normal period. A code change may not produce more failures, but it could make each job slower and gradually create a backlog. Watch the first full cycle of scheduled jobs, especially if the release changes data models, external integrations, or worker concurrency.
Monthly or quarterly review
On a monthly or quarterly cadence, review trends rather than isolated incidents. Identify job types with rising duration, repeated retries, increasing payload size, or growing dependence on a single external service. Verify that dead-lettered work has an owner and that old failures are either resolved, replayed, or intentionally closed.
Recheck queue retention, maximum message size, visibility or lease duration, concurrency settings, connection pool limits, scheduler behavior, and provider quotas. These settings vary by platform and can change as your application grows, so consult the current documentation for the queue, runtime, database, and deployment platform before making adjustments.
How to interpret changes
A sudden increase in queue depth usually points to one of four conditions: more work is arriving, workers are unavailable, jobs are slower, or a dependency is blocking progress. Check the timeline in that order. If enqueue rate rose with traffic, capacity may need to scale. If enqueue rate is stable but duration increased, inspect recent code, database queries, payloads, and external calls. If workers are healthy but completions stopped, investigate dependency errors and lease or acknowledgment behavior.
Separate user-impacting jobs from maintenance work. Sending a confirmation email may need a shorter queue-age target than rebuilding a search index. Use separate queues or priority policies when a low-priority batch cannot be allowed to delay interactive work. Be cautious with unlimited priority: a constantly busy high-priority queue can starve other tasks.
Concurrency is not automatically capacity. More workers can increase throughput, but they can also exhaust database connections, trigger external API limits, increase lock contention, or raise memory usage. Change one capacity variable at a time and observe queue age, completion rate, error rate, and dependency health together.
When a job fails, classify the cause before changing the retry count. A transient network failure may benefit from backoff. A validation error needs a code or data correction. A timeout may require smaller batches, pagination, a better query, or a longer but bounded limit. A duplicate side effect indicates an idempotency gap, not simply a need for more retries.
For scheduled jobs, track whether each expected run started, completed, and produced the expected result. A scheduler can report that it triggered a task while the actual worker work remains queued or failed. Treat the schedule event and the job outcome as separate checkpoints.
When to revisit
Revisit this design whenever you add a new long-running task, introduce a scheduled workflow, change queue providers, move between serverless hosting and containers, or modify database and external API dependencies. The serverless hosting and containers comparison can help frame worker lifecycle and shutdown decisions during a platform change.
Use a recurring review even when the system appears healthy. At each review, export or record a small set of comparable values: oldest queued job age, queue depth, completion rate, failure rate, median and slow execution duration, retry volume, dead-letter count, and worker resource use. Note deployments, traffic changes, provider incidents, and configuration changes beside the measurements so trends have context.
Then take one concrete action: remove a noisy alert, add a missing timeout, split a congested queue, improve an idempotency key, tune a database query, document replay steps, or adjust worker capacity. Link job failures to the relevant API and release checks; the API testing workflow is useful when background work depends on internal or third-party endpoints.
A reliable background job system is not defined by never failing. It is defined by making failure visible, bounded, recoverable, and safe to repeat. By tracking queue age and worker behavior, reviewing the system on a regular cadence, and testing failure paths before they matter, you can keep asynchronous work dependable as the web app evolves.