A failed production release does not always require an immediate rollback, and rolling back application code does not automatically reverse database changes. This runbook gives you a reusable cloud deployment rollback checklist: identify the faulty release, protect data, control traffic, verify recovery, communicate clearly, and document the follow-up work.
Overview
A rollback is the controlled process of returning an application to a previously known-good release or configuration. It may involve switching traffic to an earlier deployment, restoring a container or build artifact, disabling a feature flag, reverting infrastructure configuration, or moving a database to a compatible state.
The safest response begins with diagnosis rather than reflex. A new release may be healthy while an external dependency, DNS change, expired credential, cache issue, or traffic surge causes the incident. Confirm that the release is the likely cause, define the user impact, and choose the smallest reversible action that can restore service.
Before an incident, record these details for every production release:
- Release identifier, commit or artifact reference, deployment time, and owner.
- The previous known-good version and the exact procedure for selecting it.
- Database migrations, configuration changes, feature flags, and infrastructure changes included in the release.
- Health checks, key user journeys, dashboards, alerts, and log queries used for verification.
- The person authorized to approve a rollback and the channel used for incident communication.
These records turn a stressful production decision into a repeatable release recovery workflow. They are useful whether you deploy through a managed hosting platform, a serverless service, containers, virtual machines, or an internal deployment system.
Checklist by scenario
Before you roll back
- Declare the incident and assign roles. Name an incident lead, a person investigating the fault, and someone responsible for communication. Keep the group focused on recovery rather than debate.
- Identify the release. Capture the deployed version, region, environment, rollout percentage, and time the symptoms began. Compare these details with the first error, latency, or availability signals.
- Assess the impact. Check whether the issue affects all users, a tenant or region, a specific endpoint, a mobile client, background jobs, or only an administrative workflow. Preserve relevant logs and request identifiers.
- Choose rollback, mitigation, or rollforward. A feature flag may be safer than replacing the entire release. A small corrective release may be preferable when the current version includes an irreversible data change.
- Confirm the target version. Select an artifact that was previously deployed and verified, not merely an older commit that has never been tested in the current environment.
When the application code is the problem
- Stop or pause the rollout if the platform supports it.
- Route traffic to the last known-good deployment, or redeploy the verified artifact using the provider's documented process.
- Keep the faulty artifact and logs available for investigation; do not delete evidence during recovery.
- Check that environment variables, secrets, service accounts, permissions, and network settings match the target release.
- Run smoke tests against the public application, authentication flow, primary API paths, and important background processes.
Traffic switching is often safer than rebuilding from memory because it preserves a known release. However, the exact capability varies by deployment platform. Confirm whether the platform supports revisions, release promotion, blue-green traffic changes, canary rollouts, or rapid redeployment before relying on any one method.
When a database migration is involved
Treat application rollback and database rollback as separate decisions. A previous application version may not understand a new column, constraint, index, data format, or required field. Reversing a migration can also destroy data written by the newer release.
- Stop writes or limit affected operations if continuing writes could increase damage.
- Determine whether the migration changed schema only, transformed existing data, removed data, or changed validation rules.
- Check whether the older application version can safely read the current schema.
- Use a tested down-migration or restoration procedure only when its data-loss implications are understood.
- Verify backups, backup age, restore access, and recovery objectives before considering a database restore.
A forward-compatible migration pattern is often easier to recover from: add new fields first, deploy code that can use them, migrate data separately, and remove old structures only after the older release is no longer needed. This is a design practice, not a substitute for testing the actual recovery path.
When a feature flag or configuration change is the cause
- Disable the smallest affected feature or route if the flag system remains available.
- Record the flag state before changing it, including any user, tenant, region, or percentage targeting.
- Check configuration, secrets, certificates, third-party credentials, queues, and connection limits separately from application code.
- Confirm that cached configuration has refreshed across all instances.
When the release cannot be cleanly rolled back
Use a mitigation plan: reduce traffic, isolate the failing endpoint, place a read-only restriction on an affected workflow, queue work for later processing, or deploy a small corrective change. If data integrity or security is at risk, prioritize containment and involve the appropriate security, database, or infrastructure owner. A controlled degraded mode is usually preferable to repeated unverified deployments.
What to double-check
Database compatibility: Confirm that the selected application version supports the current schema and data. Pay particular attention to renamed fields, stricter validation, enum changes, destructive transformations, and code that expects newly introduced values.
Traffic and routing: Verify load balancer rules, service revisions, DNS, regional routing, ingress settings, and health-check behavior. A rollback is incomplete if some users still reach the faulty version.
Cache layers: Identify browser, CDN, reverse-proxy, application, and data caches. Purge only what is necessary. A broad purge can increase load, while a stale cache can make a successful rollback appear broken.
Asynchronous work: Review queues, scheduled jobs, webhooks, event consumers, and retry policies. An older worker may not understand messages produced by the newer release. Pause, drain, redirect, or replay work only after checking message compatibility.
Authentication and permissions: Validate session handling, token claims, API keys, service credentials, and authorization rules. For related troubleshooting guidance, see the comparison of API keys, OAuth 2.0, JWT, and passkeys. Never paste live tokens into logs or online debugging tools.
Monitoring: Compare error rate, latency, availability, resource use, queue depth, failed jobs, and business-critical transactions before and after the change. The application monitoring tools guide can help organize logs, metrics, traces, and alerts into a verification plan.
Client compatibility: Older mobile or browser clients may continue sending requests after the server rollback. Check API versioning, response shapes, minimum supported clients, and cached application assets.
Communication: State what happened, what is affected, what action is underway, and when the next update will arrive. Avoid claiming full resolution until technical and user-facing checks both pass.
Common mistakes
- Rolling back without identifying the change. This can hide the real fault and make later diagnosis harder.
- Assuming code rollback reverses data changes. Schema and data operations require their own compatibility and recovery review.
- Testing only the homepage. Check authentication, core transactions, APIs, background work, and the affected user journey.
- Deleting the failed deployment. Preserve artifacts, logs, configuration snapshots, and timestamps for analysis.
- Ignoring partial rollout states. Confirm the active version in every region, instance group, and worker pool.
- Purging every cache immediately. Coordinate cache changes with capacity checks and confirm which layer contains stale content.
- Making several untracked changes during recovery. Keep an action log with the time, operator, command or change, result, and next decision.
- Failing to rehearse. A rollback plan that exists only in a document may depend on expired permissions, missing artifacts, or undocumented provider behavior.
When to revisit
Review this deployment recovery plan before major releases, seasonal planning cycles, infrastructure migrations, and changes to hosting or deployment tools. Revisit it after introducing a new database migration framework, feature-flag system, queue, CDN, region, authentication flow, or release strategy.
At minimum, schedule a practical rollback test twice a year and after any major change to the production workflow. The test does not need to disrupt customers: use a staging environment, a temporary revision, or a controlled traffic split. Confirm that the team can locate the previous artifact, access the deployment system, switch traffic, handle schema compatibility, clear the right caches, and verify the main user journeys.
After every real incident, update the runbook with the exact release identifier, trigger, decision points, commands or platform actions, verification evidence, and unresolved follow-up items. Before your next production release, run this short final checklist:
- Is the current and previous known-good artifact clearly identified?
- Have database compatibility and backup recovery steps been reviewed?
- Can the team switch traffic or disable the affected feature?
- Are monitoring queries, smoke tests, and communication channels ready?
- Does someone have authority to stop the rollout and approve recovery?
A reliable rollback is part of release design, not an emergency trick added afterward. When the recovery path is tested, observable, and documented, your team can restore service more deliberately while preserving the information needed to prevent the next failed deployment.
