release(v1.11.0): binary-backed Document storage with field-fast-path (phase B)

Phase B of the v2.0 storage engine rewrite. Document::data is now a
smartbotic::db::doc_binary::Doc (yyjson mut_doc wrapper) with a lazy
nlohmann::json view cache. New accessor surface:
  doc.field(name)  — O(field-count) single-field read, no full materialise
  doc.data()       — lazy full-tree, returns cached nlohmann::json ref
  doc.set_data(j)  — re-encode from nlohmann tree, invalidate cache

Wire and on-disk JSON bytes unchanged in v1.11. Same binary reads v1.10
data dirs and vice versa.

HONEST FRAMING (decision-doc amendment in same commit): the format
projection in the decision doc was wrong about RSS. Measured bench
(tests/bench_doc_memory, 100k Zoe-shape docs, separate processes):

  v1.10 nlohmann::json:  1618 bytes/doc
  v1.11 doc_binary::Doc: 1594 bytes/doc
  reduction:             ~1.5%

The per-doc heap footprint is dominated by allocator overhead and
string storage, neither of which differs materially between the two
DOM representations. v1.11 ships as a release that delivers:

  - Field-fast-path API (~60-70% of read callsites get it).
  - Phase C scaffolding (every doc.data["X"] call migrated to the
    accessor surface; Phase C can swap the storage substrate without
    touching the 44+ callsites again).
  - Compounds with v1.10's 2.80× parse-time win.

What v1.11 does NOT deliver: materially smaller per-doc heap or
bounded total RSS. That is the Phase C goal (page-based buffer pool,
fixed buffer_pool_size_mb regardless of dataset size).

Implementation deviation: doc_binary::Doc became copyable (deep clone
via yyjson_mut_val_mut_copy) to support std::vector<Document> return
values from find/getAllDocuments/bulkInsert and replication conflict
resolution. Decision doc amended.

Verified:
  ./packaging/build.sh --local produces
    smartbotic-database_1.11.0-1_amd64.deb
    libsmartbotic-db-client_1.11.0-1_amd64.deb
    libsmartbotic-db-client-dev_1.11.0-1_amd64.deb
    smartbotic-db-cli_1.11.0-1_amd64.deb
  dpkg-deb -I shows libyyjson0 still in Depends.
  ldd on the unpacked binary shows libyyjson.so.0 linked.
  All 8 unit tests pass (snapshot_durability, eviction, vector_storage,
  timestamp_precision, views, config_dropins, json_parse, doc_binary).

CLAUDE.md

@@ -40,6 +40,7 @@ cmake --build build -j$(nproc) && ./build/tests/test_vector_storage
 - **GetMemoryStats RPC + client retry** — `Client::getMemoryStats()` returns per-collection stats + pressure level. Client library auto-retries write RPCs on `DEADLINE_EXCEEDED`/`RESOURCE_EXHAUSTED`/`UNAVAILABLE` with exponential backoff + jitter (`writeRetries=3`, `writeRetryBackoffMs=100`). Reads are not auto-retried.
 - **Auto-backup before deb upgrade** — `apt upgrade smartbotic-database` runs a `preinst` hook that snapshots `/var/lib/smartbotic-database/` to `/var/backups/smartbotic-database/pre-upgrade-<ts>-from-<version>/` via `cp -a` after the old `prerm` stops the service. Retention `3`, 1.2× free-space precheck on the backup volume (aborts the upgrade if short). Opt out with `SMARTBOTIC_DB_SKIP_BACKUP=1 apt upgrade`. Tunables: `SMARTBOTIC_DB_BACKUP_RETENTION`, `SMARTBOTIC_DB_BACKUP_ROOT`.
 - **yyjson hot-path parsing (v1.10.0+)** — every JSON parse on WAL replay, snapshot deserialize, history reads, gRPC request bodies, and replication apply goes through `smartbotic::db::parse_to_nlohmann` (yyjson-backed) instead of `nlohmann::json::parse`. Measured 2.80× speedup on Zoe-shape corpora (136ms→49ms on 10k docs). In-memory representation stays `nlohmann::json` until Phase B. Wire and on-disk JSON bytes unchanged. New runtime dep: `libyyjson0`. Bench: `./build/tests/bench_json_parse <corpus.jsonl>`.
+- **Binary-backed Document storage (v1.11.0+)** — `Document::data` is now a `smartbotic::db::doc_binary::Doc` (yyjson `mut_doc` wrapper) with a lazy `nlohmann::json` view cache. New accessor surface: `doc.field(name)` is the O(field-count) fast-path that reads one top-level field without materialising the full tree (the dominant read pattern), `doc.data()` is the lazy full-tree accessor, `doc.set_data(j)` re-encodes from an nlohmann tree. Wire and on-disk JSON bytes unchanged. **Honest note:** per-doc heap footprint is ~1.5% smaller than v1.10 (1618→1594 bytes on Zoe-shape) — the win is field-access CPU + Phase C scaffolding, not RSS. See `docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md` post-implementation amendment for the bench numbers. Bench: `./build/tests/bench_doc_memory --mode=<nlohmann|binary> [N]`.
 
 ## Packaging
 

VERSION

@@ -1 +1 @@
-1.10.0
+1.11.0

docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md

@@ -166,3 +166,32 @@ WAL and snapshot bytes on disk stay JSON-text in v1.11 — the on-disk format br
 - **Decider:** Operator (fszontagh)
 - **Discussion:** Single-session brainstorm; operator picked yyjson mut_doc on the recommended-option vote when presented with the three candidates.
 - **Re-open if:** yyjson 1.0 ships with a breaking API change OR Phase C settles on a storage engine that has its own native doc format (in which case Phase B's in-memory format may be replaced again).
+
+---
+
+## Post-implementation amendment (2026-05-15, after T8 bench)
+
+The "memory per doc" row in the rationale table claimed ~100-200 bytes for yyjson mut_doc vs ~600-1200 for nlohmann::json — a projected 3-6× reduction. **The actual bench (`tests/bench_doc_memory.cpp`, 100k synthetic Zoe-shape docs, separate processes to avoid allocator carry-over) shows:**
+
+- nlohmann::json: **1618 bytes/doc**
+- yyjson mut_doc: **1594 bytes/doc**
+- **Reduction: ~1.5%**
+
+The projection was wrong: both representations are allocator-backed DOMs with similar node densities (yyjson_mut_val and nlohmann::json's discriminated union are each in the 32-64 byte range). The real per-doc footprint is dominated by the per-doc allocator overhead and string storage, neither of which differs materially between the two formats.
+
+**What Phase B actually delivers (correct framing for v1.11.0 release notes):**
+
+1. **Field-fast-path** — `Document::field(name)` walks immediate object children in O(field-count) without materialising the full tree. The dominant read pattern in `database_grpc_impl.cpp` and `memory_store.cpp` is single-key lookup; this is where the per-request CPU win lives.
+2. **Parser-buffer reuse path** — `Document::fromJson(j)` calls `set_data(j)` which encodes into yyjson. Phase A's `parse_to_nlohmann` still goes through nlohmann as the bridge; Phase C can short-circuit that by going yyjson_doc → yyjson_mut_doc directly, avoiding the intermediate nlohmann tree entirely. Phase B's accessor surface makes that future change a one-file rewrite.
+3. **Scaffolding for Phase C** — every `doc.data["X"]` callsite is now `doc.field("X")` / `doc.data()["X"]` / `doc.set_data(...)`. When Phase C swaps the storage substrate again (page-based pool, RocksDB blobs, whatever the spike picks), the callsite shape doesn't move.
+
+**What Phase B does NOT deliver:**
+
+- Materially smaller per-doc heap footprint. Both formats are tree-allocators.
+- Bounded total RSS independent of dataset size. That is the Phase C goal (page pool of fixed `buffer_pool_size_mb`, not per-doc memory).
+
+**Should Phase B still ship?** Operator confirmed yes, on 2026-05-15 after seeing the bench numbers: the field-fast-path win is real and the Phase C scaffolding burden is now retired. v1.11.0 release notes frame this honestly — no memory reduction is claimed.
+
+**Implementation deviations from this spec:**
+
+- `Doc` is **not move-only**. The `service/src/doc_binary.{hpp,cpp}` implementation added a copy constructor and copy-assignment that deep-copy via `yyjson_mut_val_mut_copy`. The original move-only design assumed every callsite could `std::move`, but `std::vector<Document>` return values from `find`/`getAllDocuments`/`bulkInsert` and replication conflict resolution force copy semantics. See commit `7eac3d6` for the change.