Binary Lazy Document Storage (Phase B → v1.11.0) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

HARD GATE: Task 1 is a brainstorming-and-decision task that MUST land first. Tasks 2-9 depend on the format decision Task 1 produces. Do NOT skip ahead — the encoder, the accessor surface, and the callsite migration all key off the chosen format.

Goal: Change Document::data from a heap-allocated nlohmann::json AST to a compact binary buffer with a lazy parse-on-demand accessor. This is the phase that actually drops RSS on Zoe-shape workloads. WAL/snapshot bytes on disk stay JSON text — the on-disk format break is deferred to Phase C.

Architecture: Documents in memory hold a std::vector<uint8_t> (binary-encoded JSON in the chosen format from Task 1) plus an optional cached nlohmann::json view materialised on first field access. Reads that need exactly one field decode just that field (fast). Reads that materialise the whole tree pay a one-shot decode + cache. Writes serialise nlohmann::json → binary at the boundary. WAL replay and snapshot deserialize parse JSON text from disk straight into binary (the bridge from Phase A already gets us yyjson; we add a yyjson→binary path here).

Tech Stack: C++20, yyjson (from Phase A), one of {libbson, custom tape format, yyjson mut_doc} — chosen in Task 1, nlohmann::json (kept for the lazy view), existing CTest + load tests.

Context

Why memory drops here, not in Phase A: nlohmann::json is a discriminated-union AST. Every object node is a std::map<std::string, json>; every string is a std::string; every leaf carries a tag byte and (on libstdc++) an SSO buffer or a heap pointer. A modest 5-field document occupies ~600-1200 bytes on the heap. The same document re-encoded as BSON or a packed tape is ~80-200 bytes. Multiplied by Zoe-scale doc counts, that's where the GB-scale RSS lives.

Why we keep nlohmann::json semantics at the boundary: ~44 callsites in service/src/ (8 files) touch doc.data["field"], doc.data.contains("x"), doc.data.dump(), etc. Rewriting all of them as part of Phase B's blast radius is feasible; rewriting them and introducing a new accessor pattern and swapping the storage format in one step is asking for trouble. The lazy accessor returns an nlohmann::json view so callsites that need full-AST semantics keep working; only hot single-field reads get a direct fast-path.

Wire compatibility: gRPC Document.data stays a JSON-text string on the wire. The encode/decode happens at the gRPC boundary (which Phase A already touches). v1.10 clients see no change.

Snapshot/WAL compatibility: v1.11's snapshot format is identical to v1.10 — documents serialise to JSON text on disk. A v1.10 binary can read v1.11 snapshots, and vice versa. Operator policy ("no intermediate-release upgrades") means we don't require this, but it's free and worth keeping.

Files in scope (verified):

Path	Callsites	Notes
`service/src/document.hpp`	1	The type definition itself — the load-bearing change
`service/src/memory_store.cpp`	17	Most affected file; reads + writes the store
`service/src/database_grpc_impl.cpp`	8	gRPC boundary — encode/decode happens here
`service/src/migrations/migration_runner.cpp`	7	Migration ops touch doc.data
`service/src/replication/conflict_resolver.cpp`	5	Conflict merge needs full-tree semantics
`service/src/views/view_manager.cpp`	2	Projection / filter evaluation
`service/src/encryption/encryption_manager.cpp`	2	Field-level encrypt/decrypt
`service/src/config/collection_config_manager.cpp`	2	Reads collection options docs
`tests/`	~30-40 sites	Most tests construct docs via `Document::fromJson` so the blast radius is smaller than the raw count suggests

File Structure

Files to create

docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md — the Task 1 deliverable. Produced by brainstorming. Specifies the chosen format, encoder/decoder contract, performance + size projections, and the rejected alternatives with reasons.
service/src/doc_binary.hpp — the chosen format's encoder/decoder. Header declares encode(const nlohmann::json&) -> std::vector<uint8_t>, decode(span<const uint8_t>) -> nlohmann::json, and a field-fast-path get_field(span<const uint8_t>, string_view) -> std::optional<nlohmann::json>.
service/src/doc_binary.cpp — implementation. The ONLY file that knows about the chosen format's wire bytes.
tests/test_doc_binary.cpp — round-trip tests, field-fast-path tests, malformed-input tests.
tests/bench_doc_memory.cpp — measures RSS delta on a fixed corpus before/after the swap.

Files to modify

service/src/document.hpp — data field type changes; new accessor methods added.
service/src/memory_store.{hpp,cpp} — all 17 callsites + hot-read paths.
service/src/database_grpc_impl.cpp — 8 callsites + the proto-↔-internal boundary.
service/src/migrations/migration_runner.cpp — 7 callsites.
service/src/replication/conflict_resolver.cpp — 5 callsites.
service/src/views/view_manager.cpp — 2 callsites.
service/src/encryption/encryption_manager.cpp — 2 callsites.
service/src/config/collection_config_manager.cpp — 2 callsites.
service/src/persistence/snapshot.cpp — serialiser still emits JSON text; deserialiser now goes JSON-text → binary directly (skips intermediate nlohmann tree on the hot path).
service/src/persistence/wal.cpp — same pattern: parse-and-encode on the way in, decode-and-emit on the way out.
service/src/persistence/history_store.cpp — same.
VERSION — 1.10.0 → 1.11.0.
CLAUDE.md — Key Features bullet for binary in-memory docs.

Files NOT to touch

proto/database.proto — wire format unchanged.
client/ — clients see no change.
tests/ snapshot fixtures — on-disk JSON-text is unchanged.

Tasks

Task 1 — Brainstorm and decide the binary format

Files:

Create: docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md

This is a brainstorm task, not a code task. Use superpowers:brainstorming skill. Operator is in the loop.

[ ] Step 1: Invoke the brainstorming skill with the specific scope: "Choose the in-memory binary document format for v1.11. Three candidates pre-identified in the v2.0 roadmap: BSON, custom packed-JSON tape, yyjson mut_doc. The decision must address: (a) memory footprint per doc on Zoe-shape data, (b) field-access cost, (c) round-trip fidelity with nlohmann::json, (d) library dependency cost, (e) long-term ownership cost (do we own the spec)."
[ ] Step 2: Produce the decision document at docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md. Required sections:
- Choice — one of BSON / custom tape / yyjson mut_doc.
- Rationale — why this one wins on the five axes above.
- Rejected alternatives — what the other two are, why they lose. Include real numbers where possible (size benchmark on a 100-doc Zoe-shape sample).
- Encoder/decoder contract — function signatures, error behaviour on malformed input, what types round-trip exactly and what types lose precision (e.g., does the format preserve int64 vs double distinction?).
- Field-fast-path semantics — can the format support get_field(buf, "name") without decoding the whole doc? If not, the lazy cache is the only fast-path and that's fine.
- Memory model — does the buffer own its strings, or are they slices into the buffer? Affects string_view safety downstream.
- Library dependency — exact apt package name (if BSON: libbson-dev/libbson-1.0-0; if yyjson mut_doc: already shipped in Phase A; if custom tape: none).
[ ] Step 3: Operator review gate. Per the brainstorming skill's "User Review Gate", do NOT proceed to Task 2 until the operator signs off on the decision document.

[ ] Step 4: Commit the decision doc.

git add docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md
git commit -m "spec: decide v1.11 binary doc format ($CHOICE)"

Task 2 — Write the encoder/decoder with TDD

Files:

Create: service/src/doc_binary.hpp
Create: service/src/doc_binary.cpp
Create: tests/test_doc_binary.cpp
Modify: tests/CMakeLists.txt
Modify: service/CMakeLists.txt

The exact byte layout, the encode/decode internals, and the field-fast-path implementation follow the decision document from Task 1. Below is the contract-level test design, which is stable regardless of format choice.

[ ] Step 1: Write the failing test. Create tests/test_doc_binary.cpp:

#include <cassert>
#include <iostream>
#include <nlohmann/json.hpp>
#include "doc_binary.hpp"

using nlohmann::json;
using smartbotic::db::doc_binary::encode;
using smartbotic::db::doc_binary::decode;
using smartbotic::db::doc_binary::get_field;

namespace {

void check(bool cond, const char* msg) {
if (!cond) {
    std::cerr << "FAIL: " << msg << "\n";
    std::abort();
}
}

void roundtrip(const json& j) {
auto buf = encode(j);
auto back = decode(buf);
check(back == j, "roundtrip");
}

void test_primitives_roundtrip() {
roundtrip(json(nullptr));
roundtrip(json(true));
roundtrip(json(false));
roundtrip(json(0));
roundtrip(json(42));
roundtrip(json(-7));
roundtrip(json(9223372036854775807LL));
roundtrip(json(3.14));
roundtrip(json("hello"));
roundtrip(json(""));
}

void test_collection_roundtrip() {
roundtrip(json::array());
roundtrip(json::object());
roundtrip(json::array({1, 2, 3, "four", true, nullptr}));
roundtrip(json({{"k", "v"}, {"n", 1}, {"nested", {{"a", 1}}}}));
}

void test_zoe_shape_roundtrip() {
json zoe = {
    {"_id", "doc-1"},
    {"_created_at", 1731628800123LL},
    {"name", "Zoe"},
    {"tags", json::array({"hot", "pinned"})},
    {"_vector", json::array({0.1, -0.2, 0.3, 0.4})},
    {"meta", {{"flag", true}, {"score", 0.93}}},
};
roundtrip(zoe);
}

void test_unicode_roundtrip() {
roundtrip(json({{"hu", "Árvíztűrő tükörfúrógép"}, {"emoji", "🚀"}}));
}

void test_field_fast_path_present() {
json j = {{"name", "Zoe"}, {"count", 42}};
auto buf = encode(j);
auto name = get_field(buf, "name");
check(name.has_value() && name->get<std::string>() == "Zoe", "fast-path name");
auto count = get_field(buf, "count");
check(count.has_value() && count->get<int>() == 42, "fast-path count");
}

void test_field_fast_path_missing() {
json j = {{"name", "Zoe"}};
auto buf = encode(j);
auto v = get_field(buf, "missing");
check(!v.has_value(), "fast-path returns nullopt for missing");
}

void test_decode_corrupt_input_throws() {
std::vector<uint8_t> garbage = {0xff, 0xfe, 0xfd, 0xfc};
bool threw = false;
try {
    (void)decode(garbage);
} catch (const std::exception&) {
    threw = true;
}
check(threw, "garbage decode throws");
}

void test_int_vs_double_preserved() {
// The decision doc must specify whether int64 vs double are distinguishable
// after roundtrip. If yes, this test asserts that. If no, this test asserts
// the documented coercion behaviour. Adjust per the decision doc.
json i = 42;
json d = 42.0;
auto bi = encode(i);
auto bd = encode(d);
check(decode(bi).is_number_integer(), "int stays int");
check(decode(bd).is_number_float(), "double stays double");
}

}  // namespace

int main() {
test_primitives_roundtrip();
test_collection_roundtrip();
test_zoe_shape_roundtrip();
test_unicode_roundtrip();
test_field_fast_path_present();
test_field_fast_path_missing();
test_decode_corrupt_input_throws();
test_int_vs_double_preserved();
std::cout << "test_doc_binary: all passed\n";
return 0;
}

[ ] Step 2: Register the test in tests/CMakeLists.txt (mirror the test_json_parse pattern).
[ ] Step 3: Run the test, watch it fail to link.

Run: cmake --build build -j$(nproc) --target test_doc_binary Expected: FAIL — encode/decode/get_field undefined.

[ ] Step 4: Implement doc_binary.hpp per the Task 1 decision document. Header skeleton (format-agnostic):

#pragma once

#include <cstdint>
#include <optional>
#include <span>
#include <vector>
#include <string_view>

#include <nlohmann/json.hpp>

namespace smartbotic::db::doc_binary {

// Encode an nlohmann::json tree into the binary on-heap format chosen in
// docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md.
std::vector<uint8_t> encode(const nlohmann::json& j);

// Decode a binary doc into a fresh nlohmann::json tree. Throws on corrupt
// input. Cost is proportional to the doc size — for single-field reads,
// prefer get_field.
nlohmann::json decode(std::span<const uint8_t> buf);

// Extract a single top-level field without decoding the whole doc.
// Returns nullopt if the field is absent. Throws on corrupt input.
// Format-specific: if the chosen format does not support O(field-size)
// extraction, this still works but internally decodes the whole doc.
std::optional<nlohmann::json> get_field(std::span<const uint8_t> buf,
                                    std::string_view field_name);

}  // namespace smartbotic::db::doc_binary

[ ] Step 5: Implement doc_binary.cpp per the Task 1 decision document's encoder/decoder contract. The exact byte layout is defined there; the contract is fixed by the tests above. Include only the format-specific library headers (e.g., <bson.h> or <yyjson.h>).
[ ] Step 6: Wire any new lib dependency into CMake / Dockerfile / control.server if Task 1 chose a format that needs a new package (e.g., BSON). Skip this step if Task 1 chose yyjson mut_doc or custom tape (no new dep).
[ ] Step 7: Run the test.

Run: cmake --build build -j$(nproc) --target test_doc_binary && ./build/tests/test_doc_binary Expected: test_doc_binary: all passed.

[ ] Step 8: Commit.

git add service/src/doc_binary.hpp service/src/doc_binary.cpp tests/test_doc_binary.cpp tests/CMakeLists.txt service/CMakeLists.txt
git commit -m "feat(doc_binary): encode/decode/get_field for in-memory binary docs"

Task 3 — Change `Document::data` type and add lazy view accessor

Files:

Modify: service/src/document.hpp

This is the big-blast-radius commit. Every consumer of Document::data breaks at compile time — by design. We fix them one file at a time in Tasks 4-7.

[ ] Step 1: Replace the type and add accessors. Edit service/src/document.hpp:

struct Document {
std::string id;

// v1.11.0+ — data lives as a binary buffer; an nlohmann::json view is
// materialised lazily on first field access and cached.
std::vector<uint8_t> data_binary;
mutable std::optional<nlohmann::json> data_view;

// Lazy accessor — preserves the old `doc.data` callsite ergonomics.
// Materialises the full tree on first call; subsequent calls return the
// cached view in O(1).
const nlohmann::json& data() const;
nlohmann::json& mutable_data();

// Replace the binary payload from a nlohmann::json tree. Invalidates the
// cached view.
void set_data(const nlohmann::json& j);

// Fast-path single-field read. Avoids materialising the full tree.
std::optional<nlohmann::json> field(std::string_view name) const;

// ... existing fields (timestamps, version, _vector, etc.) unchanged ...
};

Implementation goes in document.hpp if it's header-only currently, or in a new document.cpp if not. Match the existing pattern.

[ ] Step 2: Same treatment for DocumentVersion (line 110+). It also holds nlohmann::json data and is used by the history store.
[ ] Step 3: Build, observe the breakage.

Run: cmake --build build -j$(nproc) Expected: FAIL with many 'data' is a private member or no member named 'data' in 'Document' errors. This is the inventory of callsites to migrate in subsequent tasks.

[ ] Step 4: Commit the type change only. The compile is broken between Tasks 3 and 4 — that's intentional and short-lived. Use a WIP commit and squash later, or accept a non-bisectable commit here.
```
git add service/src/document.hpp service/src/document.cpp  # if created
git commit -m "feat(document): swap Document::data to binary buffer + lazy view (WIP, callsites follow)"
```

Task 4 — Migrate `memory_store.cpp` callsites

Files:

Modify: service/src/memory_store.cpp — 17 callsites
[ ] Step 1: Walk the compile errors.

Run: cmake --build build -j$(nproc) --target smartbotic-database 2>&1 | grep memory_store.cpp | head -40 Expected: enumerates the 17 callsites that need updating.

[ ] Step 2: Rewrite each callsite per these patterns:
- Read access doc.data["field"] → doc.field("field").value_or(nlohmann::json(nullptr)) for fast-path single-field reads; or doc.data()["field"] for paths that already pay full-tree decode.
- Write access doc.data["field"] = v → load doc.mutable_data(), mutate, then call doc.set_data(doc.data()) to re-encode. Acceptable for the rare write paths; the binary format trades write speed for read speed.
- doc.data.dump() → doc.data().dump() (full-tree materialise, fine for log/error paths).
- doc.data.contains("x") → doc.field("x").has_value() for fast-path; doc.data().contains("x") for paths already in a full-tree context.
[ ] Step 3: Build memory_store object.

Run: cmake --build build -j$(nproc) --target smartbotic-database Expected: PASS for memory_store.cpp; subsequent compile errors are in other files (still expected).

[ ] Step 4: Commit.

git add service/src/memory_store.cpp service/src/memory_store.hpp
git commit -m "feat(memory_store): migrate to Document binary data accessors (phase B)"

Task 5 — Migrate gRPC and migration runner callsites

Files:

Modify: service/src/database_grpc_impl.cpp — 8 callsites
Modify: service/src/migrations/migration_runner.cpp — 7 callsites

Same migration pattern as Task 4. Done together because these are the next-most-touched files.

[ ] Step 1: Rewrite database_grpc_impl.cpp callsites using the same patterns from Task 4. Pay extra attention to the proto-↔-internal boundary: incoming protoDoc.data() (a JSON string) goes through parse_to_nlohmann then set_data; outgoing responses do doc.data().dump() or use the field-fast-path.
[ ] Step 2: Rewrite migration_runner.cpp callsites. Migrations are mostly batch mutators — they read the full tree, modify it, and write it back. So most migrate to doc.mutable_data() + doc.set_data(...) rather than fast-path field reads.
[ ] Step 3: Build.

Run: cmake --build build -j$(nproc) --target smartbotic-database Expected: errors now confined to conflict_resolver / views / encryption / config_manager.

[ ] Step 4: Commit.

git add service/src/database_grpc_impl.cpp service/src/migrations/migration_runner.cpp
git commit -m "feat(grpc,migrations): migrate Document.data callsites (phase B)"

Task 6 — Migrate remaining service-side callsites

Files:

Modify: service/src/replication/conflict_resolver.cpp — 5 callsites
Modify: service/src/views/view_manager.cpp — 2 callsites
Modify: service/src/encryption/encryption_manager.cpp — 2 callsites
Modify: service/src/config/collection_config_manager.cpp — 2 callsites
[ ] Step 1: Migrate each file using the Task 4 patterns. Note for conflict_resolver.cpp: conflict merge logic needs full-tree access on both sides, so use doc.mutable_data() rather than field-fast-path.
[ ] Step 2: Migrate view_manager.cpp. Filter / projection evaluation already walks the tree, so doc.data() (full-tree) is fine; field-fast-path is a future optimization.
[ ] Step 3: Migrate encryption_manager.cpp. Field-level encrypt/decrypt is a perfect fast-path candidate — only the encrypted fields are touched. Use doc.field(name) and doc.mutable_data()[name] = ciphertext.
[ ] Step 4: Migrate collection_config_manager.cpp.
[ ] Step 5: Build the full binary.

Run: cmake --build build -j$(nproc) --target smartbotic-database Expected: PASS. All callsites migrated.

[ ] Step 6: Commit.

git add service/src/replication/conflict_resolver.cpp service/src/views/view_manager.cpp service/src/encryption/encryption_manager.cpp service/src/config/collection_config_manager.cpp
git commit -m "feat: migrate remaining Document.data callsites (phase B)"

Task 7 — Migrate persistence layer (WAL / snapshot / history)

Files:

Modify: service/src/persistence/wal.cpp
Modify: service/src/persistence/snapshot.cpp
Modify: service/src/persistence/history_store.cpp

These already use parse_to_nlohmann from Phase A. Now they wrap that in doc_binary::encode on the way into the store, and call doc.data().dump() on the way out (still JSON-text on disk).

[ ] Step 1: WAL replay. After parsing JSON text → nlohmann::json, call doc_binary::encode(j) and assign to doc.data_binary directly (skip the lazy cache).
[ ] Step 2: Snapshot deserialize. Same pattern — parse text → encode binary → assign.
[ ] Step 3: Snapshot serialize. When writing snapshot bytes, call doc.data().dump() for the document body (full-tree materialise is fine on the write path; snapshot writes are infrequent).
[ ] Step 4: WAL append. Same as snapshot serialize — doc.data().dump() on the write side.
[ ] Step 5: History store read + write. Same patterns applied to DocumentVersion.
[ ] Step 6: Build, run all unit tests.

Run: cmake --build build -j$(nproc) && ctest --test-dir build --output-on-failure Expected: all tests PASS.

Step 7: Run all load tests.

Run: cd tests/load_test && ./load_test && ./load_test_mixed && ./load_test_integrity && ./load_test_pinned && ./load_test_crash_insert && ./load_test_crash_verify && ./load_test_replication && ./load_test_replica_eviction Expected: all PASS.

[ ] Step 8: Commit.

git add service/src/persistence/wal.cpp service/src/persistence/snapshot.cpp service/src/persistence/history_store.cpp
git commit -m "feat(persistence): binary-encode docs after parse, dump on serialize (phase B)"

Task 8 — Memory benchmark + smoke test

Files:

Create: tests/bench_doc_memory.cpp
Modify: tests/CMakeLists.txt

The whole point of Phase B is RSS reduction. Measure it.

[ ] Step 1: Write the bench. Load N synthetic Zoe-shape docs into a MemoryStore (or a stand-in that uses Document directly), read /proc/self/status for VmRSS, report. Compare against a baseline run pinned to the v1.10 binary.

#include <fstream>
#include <iostream>
#include <random>
#include <string>
#include <vector>

#include <nlohmann/json.hpp>
#include "doc_binary.hpp"
#include "document.hpp"

size_t rss_kb() {
std::ifstream f("/proc/self/status");
std::string line;
while (std::getline(f, line)) {
    if (line.rfind("VmRSS:", 0) == 0) {
        return std::stoul(line.substr(6));
    }
}
return 0;
}

int main(int argc, char** argv) {
size_t n = (argc > 1) ? std::stoul(argv[1]) : 100000;
std::cout << "loading " << n << " docs...\n";
std::vector<smartbotic::db::Document> store;
store.reserve(n);
std::mt19937 rng(42);
for (size_t i = 0; i < n; ++i) {
    nlohmann::json j = {
        {"_id", "doc-" + std::to_string(i)},
        {"name", "name-" + std::to_string(i)},
        {"tags", nlohmann::json::array({"a", "b", "c"})},
        {"_vector", nlohmann::json::array({0.1, 0.2, 0.3, 0.4})},
        {"meta", {{"k", static_cast<int>(rng() % 1000)}}},
    };
    smartbotic::db::Document d;
    d.id = "doc-" + std::to_string(i);
    d.set_data(j);
    store.push_back(std::move(d));
}
std::cout << "RSS after load: " << rss_kb() << " kB\n";
std::cout << "store.size(): " << store.size() << "\n";
// Per-doc bytes:
std::cout << "approx per-doc kB: " << (rss_kb() / n) << "\n";
return 0;
}

[ ] Step 2: Register and build.

Run: cmake --build build -j$(nproc) --target bench_doc_memory Expected: PASS.

Step 3: Run against v1.11 binary.

Run: ./build/tests/bench_doc_memory 100000 Expected: per-doc kB drops noticeably vs v1.10 (target: ≥ 2× reduction on Zoe-shape data; the decision doc from Task 1 will have a more precise target).

[ ] Step 4: Commit with measured numbers in the commit body.

git add tests/bench_doc_memory.cpp tests/CMakeLists.txt
git commit -m "$(cat <<'EOF'
test: add bench_doc_memory for phase B RSS verification

Measured at 100k docs on Zoe-shape synthetic corpus:
v1.10 (nlohmann::json in-memory): <N> MB total / <N> kB per doc
v1.11 ($CHOICE binary):            <N> MB total / <N> kB per doc
reduction:                         <N>x
EOF
)"

Task 9 — Version bump + deb build + soak

Files:

Modify: VERSION
Modify: CLAUDE.md
[ ] Step 1: Bump VERSION to 1.11.0.
[ ] Step 2: CLAUDE.md bullet under Key Features:

Binary lazy document storage (v1.11.0+) — Document::data is stored in memory as a $CHOICE-encoded binary buffer with a lazy nlohmann::json view materialised on first full-tree access. Single-field reads use doc.field(name) and skip the full-tree decode. WAL/snapshot bytes on disk remain JSON text; only in-memory representation changed.

Step 3: Local deb build + verify deps.

Run: ./packaging/build.sh --local && dpkg-deb -I dist/smartbotic-database_1.11.0-1_amd64.deb | grep -i depends Expected: includes any new runtime dep added in Task 2's library-wiring step.

[ ] Step 4: Soak. Boot the v1.11 binary against a copy of Zoe's production data dir in a throwaway VM. Watch RSS over a 24-hour soak with synthetic gRPC traffic. Expected: RSS plateau is materially below the v1.10 plateau on the same workload.

[ ] Step 5: Commit.

git add VERSION CLAUDE.md
git commit -m "release(v1.11.0): binary lazy in-memory documents (phase B)"

[ ] Step 6: Operator-gated push / sync / Zoe upgrade. Same gates as Phase A.

Out of scope (Phase B)

On-disk binary format. WAL/snapshot still write JSON text. The on-disk break is Phase C.
Page-based storage. Phase C.
Removing nlohmann::json entirely. The lazy view keeps nlohmann as the public AST surface; this is intentional. Phase C might shrink the nlohmann footprint further, but it stays.
Wire format change. gRPC Document.data stays JSON-text on the wire forever (or until v2.0+ ships a new RPC).

Success criteria

Document::data_binary is a std::vector<uint8_t> and the entire codebase compiles + links against it.
All unit tests pass: test_json_parse, test_doc_binary, test_snapshot_durability, test_eviction, test_views, test_vector_storage, test_timestamp_precision, test_config_dropins.
All load tests pass.
bench_doc_memory reports ≥ 2× per-doc kB reduction vs v1.10 on Zoe-shape data (refine target per Task 1's decision doc).
A v1.10 binary can read v1.11's snapshots and WAL (and vice versa) — manual round-trip test in a throwaway VM.
24-hour soak against Zoe-shape traffic shows RSS plateau below v1.10's plateau.

Open questions

Task 1 is the open question. Format choice is unresolved at plan-write time; brainstorming session produces the decision document that unblocks Task 2+.
Migration UX for fast-path adoption — should we add a clang-tidy / grep gate that forbids new doc.data()["field"] callsites in hot files, nudging contributors toward doc.field(name)? Defer to post-v1.11 if it becomes a problem.

2026-05-15-v1.11-binary-docs-phase-b.md 28 KB Istoric Crud

Binary Lazy Document Storage (Phase B → v1.11.0) Implementation Plan

Context

File Structure

Files to create

Files to modify

Files NOT to touch

Tasks

Task 1 — Brainstorm and decide the binary format

Task 2 — Write the encoder/decoder with TDD

Task 3 — Change Document::data type and add lazy view accessor

Task 4 — Migrate memory_store.cpp callsites

Task 5 — Migrate gRPC and migration runner callsites

Task 6 — Migrate remaining service-side callsites

Task 7 — Migrate persistence layer (WAL / snapshot / history)

Task 8 — Memory benchmark + smoke test

Task 9 — Version bump + deb build + soak

Out of scope (Phase B)

Success criteria

Open questions

2026-05-15-v1.11-binary-docs-phase-b.md 28 KB

Istoric Crud

Task 3 — Change `Document::data` type and add lazy view accessor

Task 4 — Migrate `memory_store.cpp` callsites