Bladeren bron

docs(recovery): tiered recovery modes + read-only safety workflow

New Recovery & Read-Only Mode section covers:
- 5 recovery levels (normal, snapshot_fallback, wal_only, best_effort, force_empty)
- Auto-readonly safety net for non-trivial recovery
- Operator workflow for recovering a broken instance
- Client and CLI APIs for lock/unlock/status
- Snapshot durability config
- Exit codes
- --force-readwrite bypass

Also add Key Features bullet describing the feature.
fszontagh 3 maanden geleden
bovenliggende
commit
72ba96c7eb
2 gewijzigde bestanden met toevoegingen van 136 en 0 verwijderingen
  1. 1 0
      CLAUDE.md
  2. 135 0
      docs/integration-guide.md

+ 1 - 0
CLAUDE.md

@@ -33,6 +33,7 @@ cmake --build build -j$(nproc) && ./build/tests/test_vector_storage
 - **Atomic Partial Updates** — `PatchDocument` RPC merges fields into existing documents atomically (server-side, within collection lock). Client `patch()` method. `update()` uses automatic optimistic locking with retry. See `docs/integration-guide.md` for concurrency patterns.
 - **Views** — Read-only named projections over collections. Include/exclude field lists (dot-notation for nested paths), baked-in filters with all operators (EQ/NE/GT/GTE/LT/LTE/IN/CONTAINS/EXISTS/REGEX/SEARCH) AND-merged with caller filters, optional default sort that caller can override. Stored in `_views` system collection, created via `create_view` migration op or `createView` RPC. Writes on view names are rejected.
 - **Per-collection Timestamp Precision** — collection-level `timestamp_precision` config ("ms" default, "ns" for rapid-write collections). Stamps `_created_at`/`_updated_at` at the configured resolution, cached hot-path lookup. `configureCollection` RPC flips the config atomically; `migrateCollectionTimestamps` RPC converts existing data during maintenance windows (idempotent via 10^15 threshold).
+- **Durable Snapshots + Tiered Recovery** — atomic snapshot writer (`.tmp` + fsync + rename + post-write verification), loader fallback chain across snapshots, MySQL-style recovery modes (`normal` / `snapshot_fallback` / `wal_only` / `best_effort` / `force_empty`) selectable via `--recovery-mode` flag or `recovery.mode` config. Non-trivial recovery (fallback, WAL-only, forced empty) automatically enters **read-only mode** — operator must `smartbotic-db-cli unlock` or pass `--force-readwrite` to accept writes. Exit code `10` when recovery is refused.
 
 ## Packaging
 

+ 135 - 0
docs/integration-guide.md

@@ -427,6 +427,141 @@ To set timestamp precision declaratively, call `db.configureCollection()` from y
 }
 ```
 
+### Recovery & Read-Only Mode
+
+smartbotic-database uses WAL + snapshots for durability. When starting up, the server tries to recover state from the newest snapshot and replay WAL entries since that snapshot. If something goes wrong, it uses a tiered recovery system modeled on MySQL's `innodb_force_recovery`.
+
+#### Recovery modes
+
+| Level | Name | Behavior |
+|:-:|------|----------|
+| 0 | `normal` | Load latest snapshot + replay WAL. **Default.** If `auto_escalate=true` (default), falls back to older snapshot automatically when latest is corrupt. |
+| 1 | `snapshot_fallback` | Try snapshots newest→oldest, first that loads wins. |
+| 2 | `wal_only` | Ignore snapshots, replay WAL from sequence 0. Slow on large DBs. |
+| 3 | `best_effort` | Try snapshot_fallback, then fall through to wal_only. |
+| 4 | `force_empty` | Start empty. Snapshots + WAL preserved on disk for forensics. |
+
+Operator escalation path when recovery fails:
+
+```bash
+# After a refuse-to-start, try older snapshots
+smartbotic-database --recovery-mode=snapshot_fallback
+
+# If that still fails, replay WAL from scratch (slow)
+smartbotic-database --recovery-mode=wal_only
+
+# Last-ditch: try everything
+smartbotic-database --recovery-mode=best_effort
+
+# Absolute last resort: start empty (data preserved on disk)
+smartbotic-database --recovery-mode=force_empty
+```
+
+Or set in `config.json`:
+
+```json
+"persistence": {
+    "recovery": {
+        "mode": "normal",
+        "auto_escalate": true,
+        "allow_empty_on_fresh_install": true
+    }
+}
+```
+
+The CLI flag overrides the config for one-shot recoveries.
+
+#### Auto-readonly on non-trivial recovery
+
+**If recovery used anything other than "latest snapshot loaded cleanly," the DB boots in read-only mode.** This is a safety feature — writing onto a possibly-stale state can corrupt your data further. The operator must explicitly acknowledge the state before writes resume.
+
+On startup you'll see (at ERROR level):
+
+```
+[ERROR] Database booted in READ-ONLY mode after non-trivial recovery
+        Reason: fell back to snapshot snapshot-20260419-090719.dat because
+                snapshot-20260419-100747.dat failed: body truncated
+
+Writes will be REJECTED until you acknowledge this state:
+  smartbotic-db-cli unlock                  # live, no restart
+  smartbotic-database --force-readwrite     # on next restart
+```
+
+#### Workflow: recovering a broken instance
+
+1. **Preserve forensics.** Stop the service, snapshot `/var/lib/smartbotic-database/` before anything else.
+2. **Diagnose.** Start the server (default mode = `normal`). It will either recover cleanly or refuse with a clear banner. Use `smartbotic-db-cli status` to see what happened.
+3. **Escalate.** If recovery refused, restart with `--recovery-mode=snapshot_fallback` (and upwards as needed).
+4. **Extract data.** While the DB is in read-only mode, connect with the CLI or client library and read whatever survived. Dump/export facilities are planned for v1.7.0.
+5. **Decide.**
+   - Data looks complete → `smartbotic-db-cli unlock` to resume writes.
+   - Data is missing/stale → don't unlock. Extract to an external backup, then wipe `/var/lib/smartbotic-database/` and start clean.
+
+#### Client API
+
+```cpp
+// Acknowledge auto-readonly state and resume writes
+db.unlock();
+
+// Force read-only for maintenance (regardless of recovery outcome)
+db.lock();
+
+// Inspect state + recovery outcome
+auto s = db.getReadOnlyStatus();
+std::cout << "read-only:        " << s.readOnly << "\n"
+          << "reason:           " << s.reason << "\n"
+          << "recovery outcome: " << s.recoveryOutcome << "\n"
+          << "expected snap:    " << s.expectedSnapshot << "\n"
+          << "snapshot used:    " << s.snapshotUsed << "\n"
+          << "failure reason:   " << s.failureReason << "\n"
+          << "WAL replayed:     " << s.walEntriesReplayed << "\n"
+          << "snapshots tried:  " << s.snapshotsAttempted << "\n";
+```
+
+#### CLI
+
+```bash
+# Inspect read-only + recovery state
+smartbotic-db-cli status
+
+# Acknowledge non-trivial recovery and accept writes
+smartbotic-db-cli unlock
+
+# Manually lock (e.g. before a maintenance window)
+smartbotic-db-cli lock
+```
+
+#### Snapshot durability settings
+
+Under `persistence.snapshots`:
+
+```json
+"snapshots": {
+    "validate_after_write": true,        // verify every snapshot immediately after creation
+    "cleanup_only_if_verified": true     // never evict old snapshots if the new one failed validation
+}
+```
+
+Both default to `true` — turn off only if you really know what you're doing (disables protections against silent writer corruption).
+
+#### Exit codes
+
+| Code | Meaning |
+|:-:|---------|
+| `0` | Normal shutdown |
+| `1` | Generic error |
+| `10` | **Recovery refused.** Snapshot or WAL failure + mode=normal + data exists on disk. Operator must escalate the recovery mode or pass `--force-readwrite`. |
+| `11` | Config invalid (e.g. bad `--recovery-mode` value) |
+
+#### `--force-readwrite`
+
+Bypasses the auto-readonly guard on non-trivial recovery. Use after you've reviewed the recovery state via `smartbotic-db-cli status` and are certain writes onto that state are safe.
+
+```bash
+# After confirming recovery state looks good
+smartbotic-database --force-readwrite
+```
+
 ### Querying
 
 ```cpp