spec: decide v1.11 binary doc format (yyjson mut_doc)

Phase B Task 1 — operator-approved choice of yyjson mut_doc as the
in-memory representation of Document::data.

Rationale (full doc inline):
- Zero new runtime deps (yyjson already shipped in v1.10).
- ~3-6× smaller per-doc footprint vs nlohmann::json AST (~100-200 B
  vs ~600-1200 B on Zoe-shape docs).
- O(field-count) fast-path via yyjson_mut_obj_get — the read pattern
  Phase B exists to enable.
- Compounds with Phase A's parser: WAL replay and snapshot deserialize
  now skip the parse-to-nlohmann materialisation entirely.

Rejected: BSON (new dep, larger footprint), custom packed-JSON tape
(smallest footprint but we own the spec forever).

Encoder/decoder contract laid out for Task 2 TDD. Risks called out
(API lock-in to yyjson, write-amplification on re-encode, mut_doc
allocator behaviour under churn) with mitigations.

docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md

@@ -0,0 +1,168 @@
+# Binary Document Format Decision — v1.11 Phase B
+
+**Status:** Decided. Operator approved 2026-05-15.
+
+**Choice:** **yyjson `mut_doc`** as the in-memory representation of `Document::data`.
+
+Documents are stored in memory as a `std::unique_ptr<yyjson_mut_doc, decltype(&yyjson_mut_doc_free)>` (or equivalent RAII wrapper) plus the original input bytes if a no-copy mode is chosen. The lazy `nlohmann::json` view is materialised on the first full-tree access via the existing Phase A bridge (`smartbotic::db::parse_to_nlohmann`) and cached.
+
+This document is the gate for executing Phase B's Task 2 onward — the encoder/decoder contract, accessor surface, and ~44 callsite migration all key off this choice.
+
+---
+
+## Rationale
+
+Five-axis evaluation laid out in the v2.0 design doc:
+
+| Axis | yyjson mut_doc | Why this wins |
+|------|----------------|---------------|
+| Memory per doc | ~100-200 bytes (Zoe-shape) | Dense allocator-backed tree; ~3-6× smaller than `nlohmann::json` AST (~600-1200 bytes for the same docs). |
+| Field access | O(field-count) without full decode | `yyjson_mut_obj_get(doc, key)` walks the immediate children of the root object — no recursive materialisation. This is the fast-path Phase B exists to enable. |
+| Round-trip fidelity with nlohmann | High (lossy only on number-subtype edges) | yyjson preserves int64 / uint64 / double distinction via subtype; matches what `parse_to_nlohmann` already produces. Tested in Phase A. |
+| Library dependency cost | Zero new deps | yyjson is already a hard runtime dep from Phase A (v1.10.0). No new apt package, no new licence to audit. |
+| Long-term ownership | Tied to yyjson's API | Real lock-in, called out below as the primary risk. Mitigated by yyjson's stable release history (since 0.6.0 the public API has been backward-compatible) and the bridge layer that already abstracts call sites. |
+
+The tiebreaker is the second-order effect: Phase A's parser already produces yyjson DOMs on every parse. Routing that DOM into storage (instead of immediately converting to nlohmann and discarding the yyjson doc) means the hot WAL replay and snapshot deserialize paths skip a full materialisation step they currently pay. The 2.80× parse speedup measured in Phase A compounds with a fewer-allocations win in Phase B.
+
+---
+
+## Rejected alternatives
+
+### BSON (via libbson)
+
+**Rejected.** Adds a new runtime dep (`libbson-1.0-0`) for ~150-300 bytes per Zoe-shape doc — strictly worse than yyjson mut_doc on size, with the same field-access asymptotic. BSON's spec carries types we don't use (Date, ObjectId, Decimal128, JS code) and pays per-field type-tag bytes that don't pull their weight when our docs are JSON-shaped to begin with. The spec stability is nice but doesn't outweigh the size penalty plus the new dependency.
+
+### Custom packed-JSON tape
+
+**Rejected.** Would produce the smallest per-doc footprint (~80-150 bytes) but loses on every other axis: we own the spec forever, every future tool (db inspector, replication wire format if ever changed, future migration tools) needs a parser, encoding bugs are on us with no upstream to fix them. The cost-benefit only justifies a homegrown format if memory is *the* binding constraint — and yyjson mut_doc is already a 3-6× win over the v1.9 baseline. Custom tape is a v2.x optimisation if we later discover yyjson is leaving size on the table.
+
+---
+
+## Encoder / decoder contract
+
+The Phase B helper `service/src/doc_binary.{hpp,cpp}` exposes this surface (subject to refinement during Task 2 TDD):
+
+```cpp
+namespace smartbotic::db::doc_binary {
+
+// Storage holder. Owns the yyjson document. Moveable, not copyable.
+class Doc {
+public:
+    Doc() = default;
+    explicit Doc(yyjson_mut_doc* doc);
+    Doc(Doc&&) noexcept = default;
+    Doc& operator=(Doc&&) noexcept = default;
+    Doc(const Doc&) = delete;
+    Doc& operator=(const Doc&) = delete;
+    ~Doc();  // frees the yyjson_mut_doc via yyjson_mut_doc_free
+    bool empty() const noexcept;
+    yyjson_mut_val* root() const noexcept;
+    yyjson_mut_doc* raw() const noexcept;
+private:
+    yyjson_mut_doc* doc_ = nullptr;
+};
+
+// Encode an nlohmann::json tree into yyjson mut_doc. Used at:
+//   - gRPC ingress (proto bytes → JSON text → parsed → encoded)
+//   - WAL replay (JSON text → parsed → encoded)
+//   - Programmatic doc construction in tests and migrations
+Doc encode(const nlohmann::json& j);
+
+// Decode the full tree to nlohmann::json. The slow path; used only when a
+// caller needs full-tree semantics (e.g., snapshot serialise, dump for log).
+nlohmann::json decode(const Doc& doc);
+
+// Field-fast-path. Returns nullopt if the field is absent. O(top-level
+// field count). The whole point of Phase B.
+std::optional<nlohmann::json> get_field(const Doc& doc, std::string_view field_name);
+
+// Field-fast-path overload returning the raw yyjson value for callers that
+// can stay in yyjson land (avoids the convert-to-nlohmann cost).
+yyjson_mut_val* get_field_raw(const Doc& doc, std::string_view field_name);
+
+// Mutator — returns a new Doc with the field set/replaced. The encoder is
+// not designed for high-rate writes; write paths (Update, Patch) re-encode
+// the whole doc. Phase C is where write optimisation might live.
+Doc with_field(const Doc& doc, std::string_view field_name, const nlohmann::json& value);
+
+// Serialise to JSON text (for WAL/snapshot writes, gRPC egress, log lines).
+// Uses yyjson_mut_write which is fast — comparable to the parser speed.
+std::string to_json_text(const Doc& doc);
+
+}  // namespace
+```
+
+**Error behaviour:**
+- `encode(j)` cannot fail on a valid `nlohmann::json` (yyjson allocator OOM is the only failure mode; treated as `std::bad_alloc`).
+- `decode(doc)` cannot fail on a valid `Doc` (only OOM).
+- `get_field` returns `std::nullopt` for absent fields; throws only on programming errors (null doc).
+- `with_field` is total — it always returns a fresh `Doc`.
+
+**Throws:** `std::bad_alloc` on yyjson allocation failures. Never `nlohmann::json::parse_error` — encoding from a parsed AST cannot produce parse errors. (Callers that have raw JSON text and need to land in a `Doc` go through `parse_to_nlohmann` first and then `encode`, picking up the existing Phase A error path for free.)
+
+---
+
+## Field-fast-path semantics
+
+`get_field(doc, "name")` walks only the immediate children of the root object. Nested-path access (`doc.name.first` etc.) is **not** in scope for v1.11 — callsites that need nested access either materialise the full tree via `decode(doc)` or chain `get_field_raw` calls themselves. The 80/20 split of the actual callsite analysis (from the Phase B plan):
+
+| Pattern | Count | Path |
+|---------|-------|------|
+| `doc.data["_id"]` / similar single top-level field | ~30 sites | Fast-path |
+| `doc.data.dump()` / `doc.data.contains` over multiple keys | ~10 sites | Full-tree (`decode` + cache) |
+| Nested access (`doc.data["meta"]["k"]`) | ~4 sites | Full-tree |
+
+Fast-path hits 60-70% of the callsite mass on the read side. Write paths re-encode.
+
+---
+
+## Memory model
+
+**Ownership.** `Doc` owns the `yyjson_mut_doc*`. RAII destructor frees via `yyjson_mut_doc_free`. Move-only. No reference counting — every `Document` in `MemoryStore` owns its `Doc` outright.
+
+**String lifetime.** Strings copied into yyjson via `yyjson_mut_strncpy` (always copy, never share). This is a deliberate choice: in v1.7's eviction machinery, the source bytes (gRPC protobuf strings, WAL replay buffers) are short-lived and would dangle. The copy cost is paid once at ingress; subsequent reads are free.
+
+**Allocator.** yyjson's default allocator is malloc/free, which goes through jemalloc (linked from v1.9.4). No need to plumb a custom allocator in v1.11 — Phase C might revisit if the storage engine wants pool allocation.
+
+**Cached `nlohmann::json` view.** Optional `std::optional<nlohmann::json>` on `Document` that lazily caches the result of `decode(doc)`. Invalidated on any mutation. Costs memory when populated, so callsites should prefer `get_field` over `data()` where possible.
+
+---
+
+## Library dependency
+
+**No new packages.** yyjson 0.10+ shipped in v1.10.0 (`libyyjson-dev` for build, `libyyjson0` for runtime). yyjson's `mut_doc` API has been stable since 0.6 (2022); the Debian-13-packaged 0.10 has everything we need.
+
+The Dockerfile.base `v4` stamp from Phase A T1 covers this — no further base-image rebuild required for Phase B.
+
+---
+
+## Risks called out
+
+1. **API lock-in.** Our in-memory doc shape becomes a function of yyjson's mut_doc API. If yyjson is ever abandoned upstream, we own the migration to a successor format. Mitigation: yyjson's stable release history (5+ years of backward-compat) and the `doc_binary.{hpp,cpp}` boundary layer that abstracts every call site. If we ever migrate, it's a one-file rewrite.
+
+2. **Write amplification.** Updates re-encode the whole doc. For Zoe-shape docs (~100 bytes) this is negligible; for the rare 10kB+ docs it's a measurable cost. The write paths in `database_grpc_impl.cpp` and `migration_runner.cpp` need to be reviewed during Task 5 (callsite migration) to ensure no hot write loop is doing repeated encode-modify-encode cycles.
+
+3. **mut_doc's allocator behaviour under churn.** yyjson uses pool allocation inside a single doc, but the per-doc allocator is heap-backed. Under sustained write churn (lots of `with_field` calls), allocator pressure may rebuild old patterns. We have jemalloc; we have the `bench_doc_memory` planned in T8 to verify the RSS plateau.
+
+---
+
+## What follows from this decision
+
+Phase B execution proceeds per `docs/superpowers/plans/2026-05-15-v1.11-binary-docs-phase-b.md`:
+
+- **Task 2:** TDD-write the `doc_binary::Doc` / encode / decode / get_field surface against the contract above.
+- **Task 3:** Swap `Document::data` from `nlohmann::json` to a `doc_binary::Doc` (the load-bearing type change; breaks the build temporarily).
+- **Tasks 4-7:** Migrate the ~44 callsites in `service/src/` to the new accessors.
+- **Task 8:** Measure RSS plateau on Zoe-shape corpus.
+- **Task 9:** v1.11.0 release.
+
+WAL and snapshot bytes on disk stay JSON-text in v1.11 — the on-disk format break is Phase C.
+
+---
+
+## Decision metadata
+
+- **Decided:** 2026-05-15
+- **Decider:** Operator (fszontagh)
+- **Discussion:** Single-session brainstorm; operator picked yyjson mut_doc on the recommended-option vote when presented with the three candidates.
+- **Re-open if:** yyjson 1.0 ships with a breaking API change OR Phase C settles on a storage engine that has its own native doc format (in which case Phase B's in-memory format may be replaced again).