v2.0 Storage Engine Rewrite — Roadmap

Status: Roadmap stub. Phase A and beyond require their own detailed plan files before subagent dispatch. Do NOT try to execute this file directly — it is an index.

Source of design: /data/smartbotic-database/docs/2026-05-15-v2.0-storage-engine-design.md (316 lines, already merged into main as the canonical design).

Why this exists: The operator asked to "proceed with them all" — all three phases of the v2.0 design. Phases A/B/C span ~4–5 months of engineering and cannot execute in a single session. This file:

Lists the order in which the phases ship
Captures cross-cutting upgrade-safety requirements that span every phase
Identifies where the WAL/snapshot format breaks (Phase C, not A/B)
Points each phase at the plan file it needs before subagents can be dispatched

Cross-cutting upgrade-safety requirements (operator-mandated)

These apply to every phase, not just Phase C:

Backup before upgrade — already shipping in v1.9.5 (2026-05-15-v1.9.5-backup-before-upgrade.md). Every subsequent release inherits this safety net.
WAL/snapshot backward compatibility — each release must boot cleanly on the previous release's data dir. Concretely:
- v1.10 (Phase A) must read v1.9.x WAL + snapshot bytes unchanged (yyjson swap is only the parser; on-disk format is identical).
- v1.11 (Phase B) introduces binary doc storage in memory but keeps the WAL/snapshot doc payload as JSON text on disk so v1.10 readers can still parse v1.11 snapshots. (Migration to a binary on-disk format is deferred to Phase C.)
- v2.0 (Phase C) is the format break: page-based on-disk layout, WAL becomes page-level redo. A one-shot --migrate-from-v1 mode is required (see Phase C plan when written).
Pin-back path — apt install smartbotic-database=<old-version> plus the v1.9.5 backup restore procedure must remain a working rollback for at least one release back.

Phase A → v1.10.0 — yyjson hot-path swap

Status: Plan not yet written. Blocked on: writing 2026-05-15-v1.10-yyjson-phase-a.md.

Goal: Replace nlohmann::json parsing with yyjson at hot paths (WAL replay, snapshot deserialize, gRPC request bodies, history reads). Keep nlohmann::json as the in-memory document representation — Phase A is a parse-speed win, not a memory win.

Honest impact assessment: With jemalloc already shipped in v1.9.4, RSS is bounded. Phase A buys faster boot (parse-heavy paths) and slightly lower gRPC request latency on large bodies, but does not reduce steady-state memory. The operator should know this before Phase A is scheduled — if memory is the priority, Phase B is the work that matters.

Files in scope (parse sites identified, see grep output in the design doc):

service/src/persistence/wal.cpp:233,249 — replay
service/src/persistence/snapshot.cpp:820,929,963 — deserialize
service/src/persistence/history_store.cpp:124 — version read
service/src/database_grpc_impl.cpp — Insert/Update/Patch/Find handlers (~10 call sites)
service/src/database_service.cpp:722 — replication apply
packaging/Dockerfile.base — add libyyjson-dev
packaging/deb/templates/control.server — add libyyjson0 runtime dep
service/CMakeLists.txt — pkg_check_modules(yyjson REQUIRED) or find_package(yyjson)

Task breakdown sketch (to be expanded in the Phase A plan file):

Add yyjson build + runtime deps, bump Dockerfile.base tag
Helper: yyjson_to_nlohmann(const yyjson_val*) -> nlohmann::json — the bridge
Swap WAL replay parse → re-run replica eviction load test
Swap snapshot deserialize parse → re-run snapshot durability test
Swap gRPC handler parses → re-run integration tests
Bench: report parse-time delta on a Zoe-shape snapshot
Bump VERSION to 1.10.0, build deb, sync

Time estimate: 1–1.5 weeks of subagent-driven work across 2–3 sessions.

Phase B → v1.11.0 — binary lazy document storage

Status: Plan not yet written. Blocked on: writing 2026-05-15-v1.11-binary-docs-phase-b.md and settling the format choice.

Goal: Document in-memory representation changes from heap-allocated nlohmann::json AST → std::vector<uint8_t> binary with lazy .data() accessor that parses on demand. This is the phase that actually reduces memory.

Format decision (open): BSON vs custom packed-JSON tape vs yyjson mut_doc-as-storage. Each has trade-offs:

BSON — well-specified, has libbson; but spec includes types we don't use (Date, ObjectId, Decimal128) and field tagging adds overhead.
Custom tape — minimal, fast, but we own the spec forever.
yyjson mut_doc — keep the parser's own buffer as storage; clean but ties us to yyjson semantics.

Pre-execution work: Operator should brainstorm format choice (superpowers:brainstorming skill) before this plan can be written.

Files in scope: service/src/document.hpp, service/src/memory_store.cpp/.hpp, every callsite that does doc.data["field"] (large blast radius — ~80+ call sites across service + tests).

Wire compatibility: gRPC Document.data stays JSON text on the wire. Conversion happens at the boundary.

Snapshot compat: Snapshot format v6 emits documents as JSON text (same as v5) so v1.10 readers still work. Phase C is where the on-disk format breaks.

Time estimate: 3–4 weeks across 5–8 sessions.

Phase C → v2.0 — page-based on-disk storage + buffer pool

Status: Plan not yet written. Blocked on: the build-vs-buy spike (1 week) AND a brainstorm session on the architecture AND writing 2026-05-15-v2.0-storage-engine-phase-c.md.

Goal: Replace the "load everything into memory" architecture with bounded-memory disk-backed page storage. This is the real "act like MySQL/InnoDB" rewrite.

The build-vs-buy spike (must run first):

Option 1: Homegrown 16 KB slotted pages + B+ tree index + LRU buffer pool + page-level WAL redo (~3 months engineering)
Option 2: RocksDB-backed prototype — let LSM do the page management, keep our document/vector/index semantics on top (~3 weeks engineering, ongoing operational dep on RocksDB)
Option 3: LMDB-backed — simpler, single-writer, mmap'd B+ tree (~2 weeks engineering, but write-concurrency limits)

The spike picks one. Without that decision, the Phase C plan cannot be written.

Required cross-cutting work in Phase C:

One-shot --migrate-from-v1 boot mode: read v1.x snapshot + replay v1.x WAL into pages, then write a v2.0 checkpoint and start serving
New WAL format (page-level redo records), new snapshot format (page checkpoints), new on-disk layout under /var/lib/smartbotic-database/pages/
Eviction machinery from v1.7.0 removed — page LRU subsumes it; max_memory_mb → buffer_pool_size_mb; evicted stubs / WAL fallback / docWalSeq_ all deleted

Time estimate: ~13 weeks across many sessions, contingent on spike outcome.

What to do next session

Confirm v1.9.5 backup mechanism is shipped (current session's deliverable).
If continuing into Phase A: write 2026-05-15-v1.10-yyjson-phase-a.md with full task breakdown (use superpowers:writing-plans). Then dispatch implementers per superpowers:subagent-driven-development.
If considering reordering: the honest recommendation is to skip Phase A and go directly to Phase B if memory is the priority. Phase A is foundation work whose primary benefit (parser swap) is also achievable as a Phase B side-effect (yyjson would naturally back the binary doc storage in B). Discuss with operator.

What NOT to do without operator approval

Do not start Phase A without re-confirming memory vs latency priorities with the operator.
Do not start Phase B without settling the binary format choice (BSON vs tape vs yyjson mut_doc).
Do not start Phase C without running and writing up the build-vs-buy spike.
Do not break WAL/snapshot backward compat outside of Phase C.
Do not skip the v1.9.5 backup mechanism — it is the rollback safety net for everything that follows.

2026-05-15-v2.0-roadmap.md 8.1 KB Istoric Crud