# Binary Lazy Document Storage (Phase B → v1.11.0) Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. > > **HARD GATE:** Task 1 is a brainstorming-and-decision task that MUST land first. Tasks 2-9 depend on the format decision Task 1 produces. Do NOT skip ahead — the encoder, the accessor surface, and the callsite migration all key off the chosen format. **Goal:** Change `Document::data` from a heap-allocated `nlohmann::json` AST to a compact binary buffer with a lazy parse-on-demand accessor. This is the phase that actually drops RSS on Zoe-shape workloads. WAL/snapshot bytes on disk stay JSON text — the on-disk format break is deferred to Phase C. **Architecture:** Documents in memory hold a `std::vector` (binary-encoded JSON in the chosen format from Task 1) plus an optional cached `nlohmann::json` view materialised on first field access. Reads that need exactly one field decode just that field (fast). Reads that materialise the whole tree pay a one-shot decode + cache. Writes serialise `nlohmann::json` → binary at the boundary. WAL replay and snapshot deserialize parse JSON text from disk straight into binary (the bridge from Phase A already gets us yyjson; we add a yyjson→binary path here). **Tech Stack:** C++20, yyjson (from Phase A), one of {libbson, custom tape format, yyjson mut_doc} — chosen in Task 1, nlohmann::json (kept for the lazy view), existing CTest + load tests. --- ## Context **Why memory drops here, not in Phase A:** `nlohmann::json` is a discriminated-union AST. Every object node is a `std::map`; every string is a `std::string`; every leaf carries a tag byte and (on libstdc++) an SSO buffer or a heap pointer. A modest 5-field document occupies ~600-1200 bytes on the heap. The same document re-encoded as BSON or a packed tape is ~80-200 bytes. Multiplied by Zoe-scale doc counts, that's where the GB-scale RSS lives. **Why we keep `nlohmann::json` semantics at the boundary:** ~44 callsites in `service/src/` (8 files) touch `doc.data["field"]`, `doc.data.contains("x")`, `doc.data.dump()`, etc. Rewriting all of them as part of Phase B's blast radius is feasible; rewriting them *and* introducing a new accessor pattern *and* swapping the storage format in one step is asking for trouble. The lazy accessor returns an `nlohmann::json` view so callsites that need full-AST semantics keep working; only hot single-field reads get a direct fast-path. **Wire compatibility:** gRPC `Document.data` stays a JSON-text string on the wire. The encode/decode happens at the gRPC boundary (which Phase A already touches). v1.10 clients see no change. **Snapshot/WAL compatibility:** v1.11's snapshot format is identical to v1.10 — documents serialise to JSON text on disk. A v1.10 binary can read v1.11 snapshots, and vice versa. Operator policy ("no intermediate-release upgrades") means we don't *require* this, but it's free and worth keeping. **Files in scope (verified):** | Path | Callsites | Notes | |------|-----------|-------| | `service/src/document.hpp` | 1 | The type definition itself — the load-bearing change | | `service/src/memory_store.cpp` | 17 | Most affected file; reads + writes the store | | `service/src/database_grpc_impl.cpp` | 8 | gRPC boundary — encode/decode happens here | | `service/src/migrations/migration_runner.cpp` | 7 | Migration ops touch doc.data | | `service/src/replication/conflict_resolver.cpp` | 5 | Conflict merge needs full-tree semantics | | `service/src/views/view_manager.cpp` | 2 | Projection / filter evaluation | | `service/src/encryption/encryption_manager.cpp` | 2 | Field-level encrypt/decrypt | | `service/src/config/collection_config_manager.cpp` | 2 | Reads collection options docs | | `tests/` | ~30-40 sites | Most tests construct docs via `Document::fromJson` so the blast radius is smaller than the raw count suggests | --- ## File Structure ### Files to create - `docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md` — the Task 1 deliverable. Produced by brainstorming. Specifies the chosen format, encoder/decoder contract, performance + size projections, and the rejected alternatives with reasons. - `service/src/doc_binary.hpp` — the chosen format's encoder/decoder. Header declares `encode(const nlohmann::json&) -> std::vector`, `decode(span) -> nlohmann::json`, and a field-fast-path `get_field(span, string_view) -> std::optional`. - `service/src/doc_binary.cpp` — implementation. The ONLY file that knows about the chosen format's wire bytes. - `tests/test_doc_binary.cpp` — round-trip tests, field-fast-path tests, malformed-input tests. - `tests/bench_doc_memory.cpp` — measures RSS delta on a fixed corpus before/after the swap. ### Files to modify - `service/src/document.hpp` — `data` field type changes; new accessor methods added. - `service/src/memory_store.{hpp,cpp}` — all 17 callsites + hot-read paths. - `service/src/database_grpc_impl.cpp` — 8 callsites + the proto-↔-internal boundary. - `service/src/migrations/migration_runner.cpp` — 7 callsites. - `service/src/replication/conflict_resolver.cpp` — 5 callsites. - `service/src/views/view_manager.cpp` — 2 callsites. - `service/src/encryption/encryption_manager.cpp` — 2 callsites. - `service/src/config/collection_config_manager.cpp` — 2 callsites. - `service/src/persistence/snapshot.cpp` — serialiser still emits JSON text; deserialiser now goes JSON-text → binary directly (skips intermediate nlohmann tree on the hot path). - `service/src/persistence/wal.cpp` — same pattern: parse-and-encode on the way in, decode-and-emit on the way out. - `service/src/persistence/history_store.cpp` — same. - `VERSION` — 1.10.0 → 1.11.0. - `CLAUDE.md` — Key Features bullet for binary in-memory docs. ### Files NOT to touch - `proto/database.proto` — wire format unchanged. - `client/` — clients see no change. - `tests/` snapshot fixtures — on-disk JSON-text is unchanged. --- ## Tasks ### Task 1 — Brainstorm and decide the binary format **Files:** - Create: `docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md` This is a brainstorm task, not a code task. Use `superpowers:brainstorming` skill. Operator is in the loop. - [ ] **Step 1: Invoke the brainstorming skill** with the specific scope: "Choose the in-memory binary document format for v1.11. Three candidates pre-identified in the v2.0 roadmap: BSON, custom packed-JSON tape, yyjson mut_doc. The decision must address: (a) memory footprint per doc on Zoe-shape data, (b) field-access cost, (c) round-trip fidelity with nlohmann::json, (d) library dependency cost, (e) long-term ownership cost (do we own the spec)." - [ ] **Step 2: Produce the decision document** at `docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md`. Required sections: - **Choice** — one of BSON / custom tape / yyjson mut_doc. - **Rationale** — why this one wins on the five axes above. - **Rejected alternatives** — what the other two are, why they lose. Include real numbers where possible (size benchmark on a 100-doc Zoe-shape sample). - **Encoder/decoder contract** — function signatures, error behaviour on malformed input, what types round-trip exactly and what types lose precision (e.g., does the format preserve int64 vs double distinction?). - **Field-fast-path semantics** — can the format support `get_field(buf, "name")` without decoding the whole doc? If not, the lazy cache is the only fast-path and that's fine. - **Memory model** — does the buffer own its strings, or are they slices into the buffer? Affects `string_view` safety downstream. - **Library dependency** — exact apt package name (if BSON: `libbson-dev`/`libbson-1.0-0`; if yyjson mut_doc: already shipped in Phase A; if custom tape: none). - [ ] **Step 3: Operator review gate.** Per the brainstorming skill's "User Review Gate", do NOT proceed to Task 2 until the operator signs off on the decision document. - [ ] **Step 4: Commit the decision doc.** ```bash git add docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md git commit -m "spec: decide v1.11 binary doc format ($CHOICE)" ``` --- ### Task 2 — Write the encoder/decoder with TDD **Files:** - Create: `service/src/doc_binary.hpp` - Create: `service/src/doc_binary.cpp` - Create: `tests/test_doc_binary.cpp` - Modify: `tests/CMakeLists.txt` - Modify: `service/CMakeLists.txt` The exact byte layout, the `encode`/`decode` internals, and the field-fast-path implementation follow the decision document from Task 1. Below is the *contract*-level test design, which is stable regardless of format choice. - [ ] **Step 1: Write the failing test.** Create `tests/test_doc_binary.cpp`: ```cpp #include #include #include #include "doc_binary.hpp" using nlohmann::json; using smartbotic::db::doc_binary::encode; using smartbotic::db::doc_binary::decode; using smartbotic::db::doc_binary::get_field; namespace { void check(bool cond, const char* msg) { if (!cond) { std::cerr << "FAIL: " << msg << "\n"; std::abort(); } } void roundtrip(const json& j) { auto buf = encode(j); auto back = decode(buf); check(back == j, "roundtrip"); } void test_primitives_roundtrip() { roundtrip(json(nullptr)); roundtrip(json(true)); roundtrip(json(false)); roundtrip(json(0)); roundtrip(json(42)); roundtrip(json(-7)); roundtrip(json(9223372036854775807LL)); roundtrip(json(3.14)); roundtrip(json("hello")); roundtrip(json("")); } void test_collection_roundtrip() { roundtrip(json::array()); roundtrip(json::object()); roundtrip(json::array({1, 2, 3, "four", true, nullptr})); roundtrip(json({{"k", "v"}, {"n", 1}, {"nested", {{"a", 1}}}})); } void test_zoe_shape_roundtrip() { json zoe = { {"_id", "doc-1"}, {"_created_at", 1731628800123LL}, {"name", "Zoe"}, {"tags", json::array({"hot", "pinned"})}, {"_vector", json::array({0.1, -0.2, 0.3, 0.4})}, {"meta", {{"flag", true}, {"score", 0.93}}}, }; roundtrip(zoe); } void test_unicode_roundtrip() { roundtrip(json({{"hu", "Árvíztűrő tükörfúrógép"}, {"emoji", "🚀"}})); } void test_field_fast_path_present() { json j = {{"name", "Zoe"}, {"count", 42}}; auto buf = encode(j); auto name = get_field(buf, "name"); check(name.has_value() && name->get() == "Zoe", "fast-path name"); auto count = get_field(buf, "count"); check(count.has_value() && count->get() == 42, "fast-path count"); } void test_field_fast_path_missing() { json j = {{"name", "Zoe"}}; auto buf = encode(j); auto v = get_field(buf, "missing"); check(!v.has_value(), "fast-path returns nullopt for missing"); } void test_decode_corrupt_input_throws() { std::vector garbage = {0xff, 0xfe, 0xfd, 0xfc}; bool threw = false; try { (void)decode(garbage); } catch (const std::exception&) { threw = true; } check(threw, "garbage decode throws"); } void test_int_vs_double_preserved() { // The decision doc must specify whether int64 vs double are distinguishable // after roundtrip. If yes, this test asserts that. If no, this test asserts // the documented coercion behaviour. Adjust per the decision doc. json i = 42; json d = 42.0; auto bi = encode(i); auto bd = encode(d); check(decode(bi).is_number_integer(), "int stays int"); check(decode(bd).is_number_float(), "double stays double"); } } // namespace int main() { test_primitives_roundtrip(); test_collection_roundtrip(); test_zoe_shape_roundtrip(); test_unicode_roundtrip(); test_field_fast_path_present(); test_field_fast_path_missing(); test_decode_corrupt_input_throws(); test_int_vs_double_preserved(); std::cout << "test_doc_binary: all passed\n"; return 0; } ``` - [ ] **Step 2: Register the test** in `tests/CMakeLists.txt` (mirror the test_json_parse pattern). - [ ] **Step 3: Run the test, watch it fail to link.** Run: `cmake --build build -j$(nproc) --target test_doc_binary` Expected: FAIL — `encode`/`decode`/`get_field` undefined. - [ ] **Step 4: Implement `doc_binary.hpp`** per the Task 1 decision document. Header skeleton (format-agnostic): ```cpp #pragma once #include #include #include #include #include #include namespace smartbotic::db::doc_binary { // Encode an nlohmann::json tree into the binary on-heap format chosen in // docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md. std::vector encode(const nlohmann::json& j); // Decode a binary doc into a fresh nlohmann::json tree. Throws on corrupt // input. Cost is proportional to the doc size — for single-field reads, // prefer get_field. nlohmann::json decode(std::span buf); // Extract a single top-level field without decoding the whole doc. // Returns nullopt if the field is absent. Throws on corrupt input. // Format-specific: if the chosen format does not support O(field-size) // extraction, this still works but internally decodes the whole doc. std::optional get_field(std::span buf, std::string_view field_name); } // namespace smartbotic::db::doc_binary ``` - [ ] **Step 5: Implement `doc_binary.cpp`** per the Task 1 decision document's encoder/decoder contract. The exact byte layout is defined there; the contract is fixed by the tests above. Include only the format-specific library headers (e.g., `` or ``). - [ ] **Step 6: Wire any new lib dependency into CMake / Dockerfile / control.server** if Task 1 chose a format that needs a new package (e.g., BSON). Skip this step if Task 1 chose yyjson mut_doc or custom tape (no new dep). - [ ] **Step 7: Run the test.** Run: `cmake --build build -j$(nproc) --target test_doc_binary && ./build/tests/test_doc_binary` Expected: `test_doc_binary: all passed`. - [ ] **Step 8: Commit.** ```bash git add service/src/doc_binary.hpp service/src/doc_binary.cpp tests/test_doc_binary.cpp tests/CMakeLists.txt service/CMakeLists.txt git commit -m "feat(doc_binary): encode/decode/get_field for in-memory binary docs" ``` --- ### Task 3 — Change `Document::data` type and add lazy view accessor **Files:** - Modify: `service/src/document.hpp` This is the big-blast-radius commit. Every consumer of `Document::data` breaks at compile time — by design. We fix them one file at a time in Tasks 4-7. - [ ] **Step 1: Replace the type and add accessors.** Edit `service/src/document.hpp`: ```cpp struct Document { std::string id; // v1.11.0+ — data lives as a binary buffer; an nlohmann::json view is // materialised lazily on first field access and cached. std::vector data_binary; mutable std::optional data_view; // Lazy accessor — preserves the old `doc.data` callsite ergonomics. // Materialises the full tree on first call; subsequent calls return the // cached view in O(1). const nlohmann::json& data() const; nlohmann::json& mutable_data(); // Replace the binary payload from a nlohmann::json tree. Invalidates the // cached view. void set_data(const nlohmann::json& j); // Fast-path single-field read. Avoids materialising the full tree. std::optional field(std::string_view name) const; // ... existing fields (timestamps, version, _vector, etc.) unchanged ... }; ``` Implementation goes in `document.hpp` if it's header-only currently, or in a new `document.cpp` if not. Match the existing pattern. - [ ] **Step 2: Same treatment for `DocumentVersion`** (line 110+). It also holds `nlohmann::json data` and is used by the history store. - [ ] **Step 3: Build, observe the breakage.** Run: `cmake --build build -j$(nproc)` Expected: FAIL with many `'data' is a private member` or `no member named 'data' in 'Document'` errors. This is the inventory of callsites to migrate in subsequent tasks. - [ ] **Step 4: Commit the type change only.** The compile is broken between Tasks 3 and 4 — that's intentional and short-lived. Use a WIP commit and squash later, or accept a non-bisectable commit here. ```bash git add service/src/document.hpp service/src/document.cpp # if created git commit -m "feat(document): swap Document::data to binary buffer + lazy view (WIP, callsites follow)" ``` --- ### Task 4 — Migrate `memory_store.cpp` callsites **Files:** - Modify: `service/src/memory_store.cpp` — 17 callsites - [ ] **Step 1: Walk the compile errors.** Run: `cmake --build build -j$(nproc) --target smartbotic-database 2>&1 | grep memory_store.cpp | head -40` Expected: enumerates the 17 callsites that need updating. - [ ] **Step 2: Rewrite each callsite** per these patterns: - Read access `doc.data["field"]` → `doc.field("field").value_or(nlohmann::json(nullptr))` for fast-path single-field reads; or `doc.data()["field"]` for paths that already pay full-tree decode. - Write access `doc.data["field"] = v` → load `doc.mutable_data()`, mutate, then call `doc.set_data(doc.data())` to re-encode. Acceptable for the rare write paths; the binary format trades write speed for read speed. - `doc.data.dump()` → `doc.data().dump()` (full-tree materialise, fine for log/error paths). - `doc.data.contains("x")` → `doc.field("x").has_value()` for fast-path; `doc.data().contains("x")` for paths already in a full-tree context. - [ ] **Step 3: Build memory_store object.** Run: `cmake --build build -j$(nproc) --target smartbotic-database` Expected: PASS for memory_store.cpp; subsequent compile errors are in other files (still expected). - [ ] **Step 4: Commit.** ```bash git add service/src/memory_store.cpp service/src/memory_store.hpp git commit -m "feat(memory_store): migrate to Document binary data accessors (phase B)" ``` --- ### Task 5 — Migrate gRPC and migration runner callsites **Files:** - Modify: `service/src/database_grpc_impl.cpp` — 8 callsites - Modify: `service/src/migrations/migration_runner.cpp` — 7 callsites Same migration pattern as Task 4. Done together because these are the next-most-touched files. - [ ] **Step 1: Rewrite `database_grpc_impl.cpp` callsites** using the same patterns from Task 4. Pay extra attention to the proto-↔-internal boundary: incoming `protoDoc.data()` (a JSON string) goes through `parse_to_nlohmann` then `set_data`; outgoing responses do `doc.data().dump()` or use the field-fast-path. - [ ] **Step 2: Rewrite `migration_runner.cpp` callsites.** Migrations are mostly batch mutators — they read the full tree, modify it, and write it back. So most migrate to `doc.mutable_data()` + `doc.set_data(...)` rather than fast-path field reads. - [ ] **Step 3: Build.** Run: `cmake --build build -j$(nproc) --target smartbotic-database` Expected: errors now confined to conflict_resolver / views / encryption / config_manager. - [ ] **Step 4: Commit.** ```bash git add service/src/database_grpc_impl.cpp service/src/migrations/migration_runner.cpp git commit -m "feat(grpc,migrations): migrate Document.data callsites (phase B)" ``` --- ### Task 6 — Migrate remaining service-side callsites **Files:** - Modify: `service/src/replication/conflict_resolver.cpp` — 5 callsites - Modify: `service/src/views/view_manager.cpp` — 2 callsites - Modify: `service/src/encryption/encryption_manager.cpp` — 2 callsites - Modify: `service/src/config/collection_config_manager.cpp` — 2 callsites - [ ] **Step 1: Migrate each file** using the Task 4 patterns. Note for `conflict_resolver.cpp`: conflict merge logic needs full-tree access on both sides, so use `doc.mutable_data()` rather than field-fast-path. - [ ] **Step 2: Migrate view_manager.cpp.** Filter / projection evaluation already walks the tree, so `doc.data()` (full-tree) is fine; field-fast-path is a future optimization. - [ ] **Step 3: Migrate encryption_manager.cpp.** Field-level encrypt/decrypt is a perfect fast-path candidate — only the encrypted fields are touched. Use `doc.field(name)` and `doc.mutable_data()[name] = ciphertext`. - [ ] **Step 4: Migrate collection_config_manager.cpp.** - [ ] **Step 5: Build the full binary.** Run: `cmake --build build -j$(nproc) --target smartbotic-database` Expected: PASS. All callsites migrated. - [ ] **Step 6: Commit.** ```bash git add service/src/replication/conflict_resolver.cpp service/src/views/view_manager.cpp service/src/encryption/encryption_manager.cpp service/src/config/collection_config_manager.cpp git commit -m "feat: migrate remaining Document.data callsites (phase B)" ``` --- ### Task 7 — Migrate persistence layer (WAL / snapshot / history) **Files:** - Modify: `service/src/persistence/wal.cpp` - Modify: `service/src/persistence/snapshot.cpp` - Modify: `service/src/persistence/history_store.cpp` These already use `parse_to_nlohmann` from Phase A. Now they wrap that in `doc_binary::encode` on the way *into* the store, and call `doc.data().dump()` on the way *out* (still JSON-text on disk). - [ ] **Step 1: WAL replay.** After parsing JSON text → `nlohmann::json`, call `doc_binary::encode(j)` and assign to `doc.data_binary` directly (skip the lazy cache). - [ ] **Step 2: Snapshot deserialize.** Same pattern — parse text → encode binary → assign. - [ ] **Step 3: Snapshot serialize.** When writing snapshot bytes, call `doc.data().dump()` for the document body (full-tree materialise is fine on the write path; snapshot writes are infrequent). - [ ] **Step 4: WAL append.** Same as snapshot serialize — `doc.data().dump()` on the write side. - [ ] **Step 5: History store read + write.** Same patterns applied to `DocumentVersion`. - [ ] **Step 6: Build, run all unit tests.** Run: `cmake --build build -j$(nproc) && ctest --test-dir build --output-on-failure` Expected: all tests PASS. - [ ] **Step 7: Run all load tests.** Run: `cd tests/load_test && ./load_test && ./load_test_mixed && ./load_test_integrity && ./load_test_pinned && ./load_test_crash_insert && ./load_test_crash_verify && ./load_test_replication && ./load_test_replica_eviction` Expected: all PASS. - [ ] **Step 8: Commit.** ```bash git add service/src/persistence/wal.cpp service/src/persistence/snapshot.cpp service/src/persistence/history_store.cpp git commit -m "feat(persistence): binary-encode docs after parse, dump on serialize (phase B)" ``` --- ### Task 8 — Memory benchmark + smoke test **Files:** - Create: `tests/bench_doc_memory.cpp` - Modify: `tests/CMakeLists.txt` The whole point of Phase B is RSS reduction. Measure it. - [ ] **Step 1: Write the bench.** Load N synthetic Zoe-shape docs into a `MemoryStore` (or a stand-in that uses `Document` directly), read `/proc/self/status` for `VmRSS`, report. Compare against a baseline run pinned to the v1.10 binary. ```cpp #include #include #include #include #include #include #include "doc_binary.hpp" #include "document.hpp" size_t rss_kb() { std::ifstream f("/proc/self/status"); std::string line; while (std::getline(f, line)) { if (line.rfind("VmRSS:", 0) == 0) { return std::stoul(line.substr(6)); } } return 0; } int main(int argc, char** argv) { size_t n = (argc > 1) ? std::stoul(argv[1]) : 100000; std::cout << "loading " << n << " docs...\n"; std::vector store; store.reserve(n); std::mt19937 rng(42); for (size_t i = 0; i < n; ++i) { nlohmann::json j = { {"_id", "doc-" + std::to_string(i)}, {"name", "name-" + std::to_string(i)}, {"tags", nlohmann::json::array({"a", "b", "c"})}, {"_vector", nlohmann::json::array({0.1, 0.2, 0.3, 0.4})}, {"meta", {{"k", static_cast(rng() % 1000)}}}, }; smartbotic::db::Document d; d.id = "doc-" + std::to_string(i); d.set_data(j); store.push_back(std::move(d)); } std::cout << "RSS after load: " << rss_kb() << " kB\n"; std::cout << "store.size(): " << store.size() << "\n"; // Per-doc bytes: std::cout << "approx per-doc kB: " << (rss_kb() / n) << "\n"; return 0; } ``` - [ ] **Step 2: Register and build.** Run: `cmake --build build -j$(nproc) --target bench_doc_memory` Expected: PASS. - [ ] **Step 3: Run against v1.11 binary.** Run: `./build/tests/bench_doc_memory 100000` Expected: per-doc kB drops noticeably vs v1.10 (target: ≥ 2× reduction on Zoe-shape data; the decision doc from Task 1 will have a more precise target). - [ ] **Step 4: Commit with measured numbers in the commit body.** ```bash git add tests/bench_doc_memory.cpp tests/CMakeLists.txt git commit -m "$(cat <<'EOF' test: add bench_doc_memory for phase B RSS verification Measured at 100k docs on Zoe-shape synthetic corpus: v1.10 (nlohmann::json in-memory): MB total / kB per doc v1.11 ($CHOICE binary): MB total / kB per doc reduction: x EOF )" ``` --- ### Task 9 — Version bump + deb build + soak **Files:** - Modify: `VERSION` - Modify: `CLAUDE.md` - [ ] **Step 1: Bump VERSION** to `1.11.0`. - [ ] **Step 2: CLAUDE.md bullet** under Key Features: > **Binary lazy document storage (v1.11.0+)** — `Document::data` is stored in memory as a $CHOICE-encoded binary buffer with a lazy `nlohmann::json` view materialised on first full-tree access. Single-field reads use `doc.field(name)` and skip the full-tree decode. WAL/snapshot bytes on disk remain JSON text; only in-memory representation changed. - [ ] **Step 3: Local deb build + verify deps.** Run: `./packaging/build.sh --local && dpkg-deb -I dist/smartbotic-database_1.11.0-1_amd64.deb | grep -i depends` Expected: includes any new runtime dep added in Task 2's library-wiring step. - [ ] **Step 4: Soak.** Boot the v1.11 binary against a copy of Zoe's production data dir in a throwaway VM. Watch RSS over a 24-hour soak with synthetic gRPC traffic. Expected: RSS plateau is materially below the v1.10 plateau on the same workload. - [ ] **Step 5: Commit.** ```bash git add VERSION CLAUDE.md git commit -m "release(v1.11.0): binary lazy in-memory documents (phase B)" ``` - [ ] **Step 6: Operator-gated push / sync / Zoe upgrade.** Same gates as Phase A. --- ## Out of scope (Phase B) - **On-disk binary format.** WAL/snapshot still write JSON text. The on-disk break is Phase C. - **Page-based storage.** Phase C. - **Removing nlohmann::json entirely.** The lazy view keeps nlohmann as the public AST surface; this is intentional. Phase C might shrink the nlohmann footprint further, but it stays. - **Wire format change.** gRPC `Document.data` stays JSON-text on the wire forever (or until v2.0+ ships a new RPC). --- ## Success criteria 1. `Document::data_binary` is a `std::vector` and the entire codebase compiles + links against it. 2. All unit tests pass: `test_json_parse`, `test_doc_binary`, `test_snapshot_durability`, `test_eviction`, `test_views`, `test_vector_storage`, `test_timestamp_precision`, `test_config_dropins`. 3. All load tests pass. 4. `bench_doc_memory` reports ≥ 2× per-doc kB reduction vs v1.10 on Zoe-shape data (refine target per Task 1's decision doc). 5. A v1.10 binary can read v1.11's snapshots and WAL (and vice versa) — manual round-trip test in a throwaway VM. 6. 24-hour soak against Zoe-shape traffic shows RSS plateau below v1.10's plateau. --- ## Open questions - **Task 1 is the open question.** Format choice is unresolved at plan-write time; brainstorming session produces the decision document that unblocks Task 2+. - **Migration UX for fast-path adoption** — should we add a clang-tidy / grep gate that forbids new `doc.data()["field"]` callsites in hot files, nudging contributors toward `doc.field(name)`? Defer to post-v1.11 if it becomes a problem.