tidaldb/docs/runbooks/disaster-recovery.md

# tidalDB Disaster Recovery Runbook

Object-store backup, restore, byte-verification, and the boot-on-restore query-proof
for the **cluster** deployment (ns `tidaldb-cluster`). For single-node/standalone
recovery (corrupt keyspace, stale lock, quiesce-and-copy) see
[`../ops/recovery.md`](../ops/recovery.md). For the live operational API see
[`cluster.md`](cluster.md) and [`kubernetes.md`](kubernetes.md).

> **Status (2026-06-19): PROVEN against real S3.** The full
> export → restore → byte-verify → **query-proof** round-trip is green against a
> real S3 server (self-hosted MinIO on the fleet). Production Cloudflare R2 is the
> *same* `tidalctl` code path — only the endpoint URL and the token differ (see
> [R2](#production-r2)).

---

## What DR protects, and the recovery posture

- **Unit of backup = a per-shard data dir at rest.** The cluster is full-placement
  RF3 (every pod hosts all 3 shard groups under `/data/db/shard-0000N`). A backup
  captures one shard's data dir (fjall keyspaces + `wal/` + `vector/` HNSW graphs +
  the `checkpoint.meta` marker). Back up each of the 3 shards.
- **Integrity = BLAKE3 manifest + per-shard `checkpoint_seq`.** `tidalctl backup`
  writes `BACKUP_MANIFEST.json` (per-file BLAKE3 + `checkpoint_seq`) and uploads it
  **last** as the atomicity marker. `tidalctl restore` verifies every file's BLAKE3
  before writing a byte and refuses a non-empty target or a path-traversal manifest.
- **RPO/RTO posture (honest):** backup operates on a data dir **at rest** (a
  drained/stopped node or a streamed point-in-time copy), so the snapshot is
  trivially consistent. There is **no arbitrary point-in-time recovery yet** — the
  WAL-archive primitive exists but `tidalctl replay --until <seq>` is not shipped
  (see [PITR](#pitr-posture)). Restore is offline (boot a server on the restored
  dir); RTO is dominated by HNSW load (~5 min/100k single-core, or instant if the
  persisted graph restored cleanly).

---

## DR infrastructure (live)

Self-hosted MinIO is the real S3 backend (it is a real S3 server, **not** a mock).
Manifests live in the **orchard9-k3sf** repo:

| Artifact | Path (orchard9-k3sf) | Purpose |
|---|---|---|
| MinIO | `deployments/k8s/base/tidaldb-dr/minio.yaml` | S3 backend, ns `tidaldb-dr`, ClusterIP `minio.tidaldb-dr.svc:9000`, 5Gi PVC |
| MinIO secret | `scripts/dr-minio-secret.sh` | generates `minio-creds` (`MINIO_ROOT_USER`/`MINIO_ROOT_PASSWORD`) out-of-band; never committed |
| Byte-verify drill | `deployments/k8s/base/tidaldb-dr/dr-drill-job.yaml` | export → restore → `sha256` byte-equivalence |
| Query-proof drill | `deployments/k8s/base/tidaldb-dr/dr-queryproof-job.yaml` | seed → backup → restore → **boot a server on the restore → serve** |

The drill image `registry.threesix.ai/tidal/tidalctl:m12-dr-qp` bundles `tidalctl`
+ `tidal-server` + `mc` (built from `docker/release/dr.Dockerfile`).

---

## Procedure 1 — Back up a shard to object storage

`tidalctl` reads `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` from the env and
forces path-style addressing (region is the placeholder `"auto"`). Run it where it
can reach both the data dir and the endpoint (an in-cluster Job for the live shards;
the live RWO PVCs stay attached to the running StatefulSet, so back up a streamed
point-in-time copy, not the attached volume).

```bash
AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD \
tidalctl backup \
  --path /src/db/shard-00000 \
  --out  /work/backup \
  --s3-endpoint http://minio.tidaldb-dr.svc:9000 \
  --s3-bucket   tidaldb-dr \
  --s3-prefix   shard0
```

Writes a local BLAKE3-manifested artifact **and** mirrors every object to
`s3://tidaldb-dr/shard0/`, uploading `BACKUP_MANIFEST.json` last. Verify the marker
landed: `mc ls --recursive dr/tidaldb-dr/shard0 | sort -k4` (manifest timestamp must
be after every data file). Repeat with `--s3-prefix shard1`/`shard2` for the other
two shards.

## Procedure 2 — Restore + byte-verify

```bash
AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD \
tidalctl restore \
  --path /target/restored \
  --s3-endpoint http://minio.tidaldb-dr.svc:9000 \
  --s3-bucket   tidaldb-dr \
  --s3-prefix   shard0
```

Restore downloads the prefix, **BLAKE3-verifies every file**, `safe_join`s each path
(rejects `..`/absolute/backslash), and refuses a non-empty target. Confirm:

```bash
tidalctl status --path /target/restored   # status:ok, checkpoint_seq preserved
# sha256 byte-equivalence vs source (the drill does this for all files)
```

The only expected `diff -r` deltas are empty scaffold dirs (`keyspaces/{2,3}/tables`)
— they carry no data and are correctly excluded from the file manifest.

## Procedure 3 — Query-proof (boot a server on the restore and SERVE)

Byte-equivalence proves the *bytes*; this proves the restore *serves*. A raw
point-in-time stream of a LIVE pod is **not** a standalone-bootable fjall root
(`fjall recovering … No such file or directory`); a consistency-clean source comes
from a **graceful shutdown** (SIGTERM → checkpoint + fsync + persist HNSW graphs —
the only supported way; there is no online checkpoint CLI/HTTP). The
`dr-queryproof-job.yaml` does this end-to-end:

```
seed (100 items via API) → SIGTERM (clean checkpoint) → tidalctl backup → minio
  → tidalctl restore → tidal-server standalone --data-dir <restore> → assert it serves:
      /health items == N · /vector_search non-empty · /search?query=<tok> == entity · /feed non-empty
```

Run + read the verdict:

```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
kubectl -n tidaldb-dr delete job dr-queryproof --ignore-not-found
kubectl -n tidaldb-dr apply -f deployments/k8s/base/tidaldb-dr/dr-queryproof-job.yaml
kubectl -n tidaldb-dr logs job/dr-queryproof   # last line: "DR QUERY-PROOF: PASS"
```

---

## Production R2

Identical `tidalctl` code path — swap the endpoint and supply an R2 token:

```bash
AWS_ACCESS_KEY_ID=<r2-access-key> AWS_SECRET_ACCESS_KEY=<r2-secret> \
tidalctl backup --path <shard> --out /work/backup \
  --s3-endpoint https://<account-id>.r2.cloudflarestorage.com \
  --s3-bucket tidaldb-dr --s3-prefix shard0
```

> **Blocker:** this needs an **account-level** Cloudflare token with
> *Workers R2 Storage: Edit* (account `c75bf009a23e747ef34b776ae8095dc4`). The
> zone/DNS-scoped `THREESIX_CLOUDFLARE_API_TOKEN` in the shell **cannot** mint R2
> S3 keys (auth error 9109) — it is the one external dependency for live-R2 DR.

---

## Full-cluster rebuild from backup

1. Provision the cluster fresh: `kubectl apply -k k8s/cluster/` (creates ns,
   ConfigMaps, certs, Services, StatefulSet). Do **not** seed yet.
2. For each of the 3 shards, `tidalctl restore` the latest backup into the
   corresponding pod's `/data/db/shard-0000N` **before** the server opens it (use an
   init step or restore onto the PVC offline). The restore must land a consistency-
   clean dir (a graceful-shutdown backup, not a live stream).
3. Bring the StatefulSet up; each pod opens its restored shards, the WAL
   `checkpoint.meta` skips replay of already-checkpointed events, and the cluster
   elects a leader and converges. Verify `3/3 Ready`, single leader, and
   `/cluster/status` reports the expected `checkpoint_seq` per shard.
4. If only one pod's data is lost, do **not** restore from object storage — delete
   that pod's PVC + pod and let it reseed fresh from the live quorum (snapshot
   install → converges lag=0). Object-store restore is for total loss.

---

## PITR posture

- **What exists:** the WAL is gap-free and archived; `tidalctl backup` fences a
  consistent `checkpoint_seq`; restore replays the WAL from the checkpoint forward.
- **What does NOT exist yet:** arbitrary point-in-time recovery to a chosen seq
  (`tidalctl replay --until <seq>` is not shipped). Recovery lands you at the
  backup's `checkpoint_seq` + whatever WAL was captured, not an arbitrary instant.
- **Cadence:** run the byte-verify + query-proof drills on every release and on a
  scheduled cadence; record the result in `orchard9-k3sf/deployments/history/tidaldb.md`.

---

## Quick reference

| Need | Command / pointer |
|---|---|
| Back up a shard | `tidalctl backup --path <shard> --out <dir> --s3-endpoint … --s3-bucket … --s3-prefix …` |
| Restore + verify | `tidalctl restore --path <fresh> --s3-endpoint … --s3-bucket … --s3-prefix …` then `tidalctl status --path <fresh>` |
| Prove it serves | apply `dr-queryproof-job.yaml`, read `DR QUERY-PROOF: PASS` |
| One pod lost | delete its PVC+pod → reseeds from quorum (no object-store restore) |
| Total loss | rebuild fresh + `tidalctl restore` each shard before boot |
| Live R2 | same path + account-level R2 token (the one external blocker) |