Skip to content

Disaster Recovery Plan

A disaster recovery plan is the document that answers, before the incident, the two questions nobody answers well in the middle of one: how much data can we lose and how fast do we need to be back. Everything else — topology, replication, backup, cost — follows from those two answers.

Disaster timeline showing RPO and RTO, and the three phases: declare, fail over and fail back

RPO measures lost data; RTO measures downtime. They are business numbers, not infrastructure numbers.

RPO and RTO

  • RPO (Recovery Point Objective) — how much data it is acceptable to lose, measured in time. An RPO of 5 minutes means accepting the loss of up to 5 minutes of transactions.
  • RTO (Recovery Time Objective) — how long the service can stay unavailable before it must be serving again.

Both have a price. Zero RPO requires synchronous replication, which charges latency on every write. An RTO of seconds requires idle capacity in the other region. The architecture work is finding the point where the cost of the risk and the cost of the protection cross.

Tier Typical RPO Typical RTO Arrangement
Restorable backup Hours Hours Off-site backup, restore on demand
Asynchronous replica Seconds to minutes Minutes Warm replica in the other region, manual or assisted promotion
Cross-region synchronous replica Near zero Tens of seconds Quorum across br-sp-1 and br-sp-2, automated promotion
Distributed multi-write Zero No write interruption CockroachDB with three failure domains

Covered scenarios

A plan that only anticipates "the datacenter burned down" covers the rarest scenario and ignores the frequent ones.

Scenario Frequency Response
Disk or node failure Common A replica takes over; no human action
Host or rack failure Common Anti-affinity keeps the quorum alive
Logical corruption / human error Common Point-in-time recovery — replication does not help here
Network failure between regions Occasional Fencing and quorum prevent split-brain
Loss of an entire region Rare Failover to the surviving region
Ransomware / malicious deletion Rare, severe Immutable backup with protected retention

The most common disaster is human

In practice, the incident that destroys the most data is not a datacenter outage — it is the wrong command, the buggy deploy and the accidental deletion. That is why backup with PITR is part of the DR plan, not a separate topic.

Structure of the plan

The plan is an operational document, not a presentation. What it contains:

  1. Inventory — systems, databases, dependencies and how critical each one is.
  2. RPO and RTO per system — not everything needs the same tier, and treating everything as critical costs more without protecting more.
  3. Declaration criteria — what counts as a disaster and who has the authority to declare it. Without this, the first half hour is spent deciding whether it is time to act.
  4. Failover procedure — an executable step-by-step, with commands, not with principles.
  5. Communication — who informs customers, the team and, where applicable, the authorities.
  6. Failback procedure — how to come back, which is the forgotten half of most plans.
  7. Drill record — the date and result of the last exercise.

Cross-region failover

InteSys's two regions are Brazilian and sit in the São Paulo metropolitan area — br-sp-1 (Cirion SAO1) and br-sp-2 (Equinix SP3) — interconnected by Lan2Lan links or encrypted tunnels. The latency between them allows synchronous replication, which makes near-zero RPO viable without leaving the country.

Failover sequence:

  1. Detect — more than one observer confirms the loss. A single observer does not decide; a momentary network glitch is not a disaster.
  2. Declare — the person with the authority declares, following the written criteria.
  3. Fence — the old primary is prevented from accepting writes. Without this, two primaries corrupt the data.
  4. Promote — the most advanced replica in the surviving region takes over, after its replication position is verified.
  5. Redirect — the proxy points at the new primary; the application keeps the same endpoint.
  6. Verify — data sanity, latency and error rate before declaring the service restored.
  7. Communicate — status internally and externally, with an estimate for normalization.

Failback

Coming back is a planned operation, never an urgent one:

  1. rebuild the recovered region as a replica of the current one;
  2. sync and wait for zero lag;
  3. pick a low-traffic window;
  4. run the controlled switch, with the same fencing procedure;
  5. confirm that replication, backup and alerting are back to their normal state.

There is no rush to fail back

If the surviving region is serving with acceptable performance, the return can wait until the root cause is fixed. A hasty failback tends to produce the second incident.

Drills

A plan that has never been executed is a hypothesis.

Exercise Frequency What it validates
Failover in a test environment Quarterly The procedure is correct and up to date
Timed full restore Twice a year The agreed RTO is real
Controlled failover in production Yearly, in a window That the plan works with real data and traffic
Review of contacts and authority Twice a year That the people in the plan are still the right ones

Every exercise ends with a report: measured time, deviations from the procedure and fixes applied to the runbook.

Data sovereignty in the DR plan

The whole plan runs inside Brazil. Replicas, logs, backups and the failover target stay in br-sp-1 and br-sp-2, under Brazilian jurisdiction and with no international transfer of personal data.

When a customer asks for a contingency copy in another country — a corporate requirement or geographic isolation of the backup — it is set up on an encrypted volume, with the international transfer documented in the solution design. It is the customer's decision, never a platform default.