|
@@ -427,6 +427,141 @@ To set timestamp precision declaratively, call `db.configureCollection()` from y
|
|
|
}
|
|
}
|
|
|
```
|
|
```
|
|
|
|
|
|
|
|
|
|
+### Recovery & Read-Only Mode
|
|
|
|
|
+
|
|
|
|
|
+smartbotic-database uses WAL + snapshots for durability. When starting up, the server tries to recover state from the newest snapshot and replay WAL entries since that snapshot. If something goes wrong, it uses a tiered recovery system modeled on MySQL's `innodb_force_recovery`.
|
|
|
|
|
+
|
|
|
|
|
+#### Recovery modes
|
|
|
|
|
+
|
|
|
|
|
+| Level | Name | Behavior |
|
|
|
|
|
+|:-:|------|----------|
|
|
|
|
|
+| 0 | `normal` | Load latest snapshot + replay WAL. **Default.** If `auto_escalate=true` (default), falls back to older snapshot automatically when latest is corrupt. |
|
|
|
|
|
+| 1 | `snapshot_fallback` | Try snapshots newest→oldest, first that loads wins. |
|
|
|
|
|
+| 2 | `wal_only` | Ignore snapshots, replay WAL from sequence 0. Slow on large DBs. |
|
|
|
|
|
+| 3 | `best_effort` | Try snapshot_fallback, then fall through to wal_only. |
|
|
|
|
|
+| 4 | `force_empty` | Start empty. Snapshots + WAL preserved on disk for forensics. |
|
|
|
|
|
+
|
|
|
|
|
+Operator escalation path when recovery fails:
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# After a refuse-to-start, try older snapshots
|
|
|
|
|
+smartbotic-database --recovery-mode=snapshot_fallback
|
|
|
|
|
+
|
|
|
|
|
+# If that still fails, replay WAL from scratch (slow)
|
|
|
|
|
+smartbotic-database --recovery-mode=wal_only
|
|
|
|
|
+
|
|
|
|
|
+# Last-ditch: try everything
|
|
|
|
|
+smartbotic-database --recovery-mode=best_effort
|
|
|
|
|
+
|
|
|
|
|
+# Absolute last resort: start empty (data preserved on disk)
|
|
|
|
|
+smartbotic-database --recovery-mode=force_empty
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Or set in `config.json`:
|
|
|
|
|
+
|
|
|
|
|
+```json
|
|
|
|
|
+"persistence": {
|
|
|
|
|
+ "recovery": {
|
|
|
|
|
+ "mode": "normal",
|
|
|
|
|
+ "auto_escalate": true,
|
|
|
|
|
+ "allow_empty_on_fresh_install": true
|
|
|
|
|
+ }
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+The CLI flag overrides the config for one-shot recoveries.
|
|
|
|
|
+
|
|
|
|
|
+#### Auto-readonly on non-trivial recovery
|
|
|
|
|
+
|
|
|
|
|
+**If recovery used anything other than "latest snapshot loaded cleanly," the DB boots in read-only mode.** This is a safety feature — writing onto a possibly-stale state can corrupt your data further. The operator must explicitly acknowledge the state before writes resume.
|
|
|
|
|
+
|
|
|
|
|
+On startup you'll see (at ERROR level):
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+[ERROR] Database booted in READ-ONLY mode after non-trivial recovery
|
|
|
|
|
+ Reason: fell back to snapshot snapshot-20260419-090719.dat because
|
|
|
|
|
+ snapshot-20260419-100747.dat failed: body truncated
|
|
|
|
|
+
|
|
|
|
|
+Writes will be REJECTED until you acknowledge this state:
|
|
|
|
|
+ smartbotic-db-cli unlock # live, no restart
|
|
|
|
|
+ smartbotic-database --force-readwrite # on next restart
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+#### Workflow: recovering a broken instance
|
|
|
|
|
+
|
|
|
|
|
+1. **Preserve forensics.** Stop the service, snapshot `/var/lib/smartbotic-database/` before anything else.
|
|
|
|
|
+2. **Diagnose.** Start the server (default mode = `normal`). It will either recover cleanly or refuse with a clear banner. Use `smartbotic-db-cli status` to see what happened.
|
|
|
|
|
+3. **Escalate.** If recovery refused, restart with `--recovery-mode=snapshot_fallback` (and upwards as needed).
|
|
|
|
|
+4. **Extract data.** While the DB is in read-only mode, connect with the CLI or client library and read whatever survived. Dump/export facilities are planned for v1.7.0.
|
|
|
|
|
+5. **Decide.**
|
|
|
|
|
+ - Data looks complete → `smartbotic-db-cli unlock` to resume writes.
|
|
|
|
|
+ - Data is missing/stale → don't unlock. Extract to an external backup, then wipe `/var/lib/smartbotic-database/` and start clean.
|
|
|
|
|
+
|
|
|
|
|
+#### Client API
|
|
|
|
|
+
|
|
|
|
|
+```cpp
|
|
|
|
|
+// Acknowledge auto-readonly state and resume writes
|
|
|
|
|
+db.unlock();
|
|
|
|
|
+
|
|
|
|
|
+// Force read-only for maintenance (regardless of recovery outcome)
|
|
|
|
|
+db.lock();
|
|
|
|
|
+
|
|
|
|
|
+// Inspect state + recovery outcome
|
|
|
|
|
+auto s = db.getReadOnlyStatus();
|
|
|
|
|
+std::cout << "read-only: " << s.readOnly << "\n"
|
|
|
|
|
+ << "reason: " << s.reason << "\n"
|
|
|
|
|
+ << "recovery outcome: " << s.recoveryOutcome << "\n"
|
|
|
|
|
+ << "expected snap: " << s.expectedSnapshot << "\n"
|
|
|
|
|
+ << "snapshot used: " << s.snapshotUsed << "\n"
|
|
|
|
|
+ << "failure reason: " << s.failureReason << "\n"
|
|
|
|
|
+ << "WAL replayed: " << s.walEntriesReplayed << "\n"
|
|
|
|
|
+ << "snapshots tried: " << s.snapshotsAttempted << "\n";
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+#### CLI
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# Inspect read-only + recovery state
|
|
|
|
|
+smartbotic-db-cli status
|
|
|
|
|
+
|
|
|
|
|
+# Acknowledge non-trivial recovery and accept writes
|
|
|
|
|
+smartbotic-db-cli unlock
|
|
|
|
|
+
|
|
|
|
|
+# Manually lock (e.g. before a maintenance window)
|
|
|
|
|
+smartbotic-db-cli lock
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+#### Snapshot durability settings
|
|
|
|
|
+
|
|
|
|
|
+Under `persistence.snapshots`:
|
|
|
|
|
+
|
|
|
|
|
+```json
|
|
|
|
|
+"snapshots": {
|
|
|
|
|
+ "validate_after_write": true, // verify every snapshot immediately after creation
|
|
|
|
|
+ "cleanup_only_if_verified": true // never evict old snapshots if the new one failed validation
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Both default to `true` — turn off only if you really know what you're doing (disables protections against silent writer corruption).
|
|
|
|
|
+
|
|
|
|
|
+#### Exit codes
|
|
|
|
|
+
|
|
|
|
|
+| Code | Meaning |
|
|
|
|
|
+|:-:|---------|
|
|
|
|
|
+| `0` | Normal shutdown |
|
|
|
|
|
+| `1` | Generic error |
|
|
|
|
|
+| `10` | **Recovery refused.** Snapshot or WAL failure + mode=normal + data exists on disk. Operator must escalate the recovery mode or pass `--force-readwrite`. |
|
|
|
|
|
+| `11` | Config invalid (e.g. bad `--recovery-mode` value) |
|
|
|
|
|
+
|
|
|
|
|
+#### `--force-readwrite`
|
|
|
|
|
+
|
|
|
|
|
+Bypasses the auto-readonly guard on non-trivial recovery. Use after you've reviewed the recovery state via `smartbotic-db-cli status` and are certain writes onto that state are safe.
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# After confirming recovery state looks good
|
|
|
|
|
+smartbotic-database --force-readwrite
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
### Querying
|
|
### Querying
|
|
|
|
|
|
|
|
```cpp
|
|
```cpp
|