2026-05-16-v2.1-backlog.md 6.6 KB
v2.x backlog — shadowman audit follow-up + Stage 5-10 remainder
Origin: shadowman team alignment audit /tmp/sbdbv2.md (2026-05-15) +
the unfinished items from 2026-05-15-v2.0-storage-engine-phase-c-execution.md.
v2.x is the project's terminal arc — no v3 is planned. Every item below targets a v2.x.x slot. Items that don't fit the v2.x roadmap are simply not planned (no v3 backlog to escalate them to).
This document supersedes the loose "Out of scope (deferred to v2.1+)"
section in 2026-05-15-v2.0-roadmap.md (lines 351-359). Each entry has
an explicit drop trigger so the calendar of removals is observable.
Status of shadowman audit items
F-section (open questions) — answered
| F# | Question | Answer (committed v2.1.1) |
|---|---|---|
| F1 | When does the v1.x auto-migration path drop? | Stays in-binary through v2.4. Extracts to separate smartbotic-database-migrate-v1 deb in v2.5 once every production host has booted v2.4 ≥ once. v2.x is the terminal arc — no v3 escalation. |
| F2 | Is setReadOnly operator-only or client-callable? |
Both. Operator via smartbotic-db-cli unlock; client via Client::lock() for self-defensive migrations. |
| F3 | WAL-fallback fields — zero on healthy v2.0? | Yes, guaranteed zero on the LMDB-served path (Find handler explicitly leaves them at their default-constructed zero). Non-zero = real signal, treat as alert. |
| F4 | migrateCollectionTimestamps cost on hot messages? |
Deferred bench until shadowman runs D.1. Back-of-envelope: ~1ms / 1k docs, online-safe up to ~1M docs; >1M docs schedule a maintenance window. Will be re-measured against real traffic post-D.1. |
B-section (back-compat surfaces) — drop tracker
| B# | Surface | Drop trigger | Target |
|---|---|---|---|
| B1 | In-binary v1.x → v2.0 migration (commit 15893b1) |
Every production host has booted v2.4 ≥ 1× | v2.5 (extract to separate deb) |
| B2 | Defensive MemoryStore fallback on LMDB-throw | Shipped — closed v2.1.1 | ✅ done |
| B3 | Stripped QueryOptions in document.hpp |
Shipped — closed v2.1.0 | ✅ done |
| B4 | Plain find(collection) overload |
Shipped — closed v2.1.0 | ✅ done |
| B5 | Default "ms" timestamp precision |
Every rapid-write collection migrated via shadowman D.1 | v2.2 (flip default to "ns") |
| B6 | update() as RMW verb |
Every RMW callsite converted to patch() via shadowman D.2 |
v2.x.next (mark update() as "replace whole doc only" in docs) |
| B7 | 6-value recoveryOutcome enum |
Write-handler migration ships → MemoryStore decommission → LMDB-only recovery semantics | v2.4 (collapse to single lmdb_ready, paired with the write-flip) |
Shadowman D-section (their migration items)
External dependency — not tracked here. The cookbook from their audit:
- D.1 ns-precision timestamps on
messages/metrics_events/tool_call_blobs/agent_task_messages— highest leverage, unblocks their T6.6 timeline refactor - D.2
patch()adoption for 71 RMW callsites - D.3
count(filters)for "any-matching-row" patterns (~10-15 sites) - D.4
findWithMetrics()for slow-query logging - D.5 Set operations for
conversation_tool_state - D.6 Server-side FilterOp on cold paths (their T7.1-T7.3)
- D.7 Diagnostic
createView()(their T7.4)
Unfinished items from the v2.0 Phase C plan
These were architectural endpoints not reached in v2.0:
Stage 5 — sub-db migration remainder
| Sub | Module | Current state in v2.1 | v2.x target |
|---|---|---|---|
| 5.2 | History _history_<coll> |
Still on service/src/persistence/history_store.cpp .hlog files |
LMDB sub-db keyed by <doc_id>:<version> storing DocumentVersion JSON. Delete history_store.cpp. |
| 5.3 | Files _files |
Metadata still in MemoryStore + .file blob path | _files sub-db for metadata; blobs unchanged at files/<id>. |
| 5.4 | Views _views |
View defs stored in MemoryStore _views system collection |
_views LMDB sub-db keyed by view name. Existing createView / getViewInfo handlers ported. |
Estimated effort: 5.2 medium (HistoryStore deletion + test port), 5.3 + 5.4 small. Each is one focused PR.
Stage 4-finish + Stage 6+ — write-handler migration & beyond
This is the v2.4 milestone — the v2.x arc's architectural endpoint. Handlers call doc_store_->put() directly with id-gen / timestamp / version orchestration extracted from MemoryStore into a WriteCoordinator. Once landed, MemoryStore deletes; persistence/wal.cpp / persistence/snapshot.cpp delete; eviction config knobs delete; recovery enum collapses (B7).
Subsequent stages from the original plan, mostly unchanged:
- Stage 6 — replication preservation: validate WAL streaming still works through the write-handler-migration boundary
- Stage 7 — migrations RPC: port
MigrationRunnerto callDocumentStore - Stage 8 — eviction + recovery cleanup audit
- Stage 9 — performance gates + 24h soak (NOT done for v2.0 —
load_test_mixed30s was the proxy) - Stage 10 — v2.4 release (the v2.x arc's architectural endpoint)
Not planned
The v2.0 roadmap's "Out of scope (deferred to v2.1+)" list contained items that, with no v3 in the plan, simply aren't scheduled:
- Secondary user-defined indexes
- Online schema evolution
- Distributed clustering beyond leader+follower
- LMDB writer-pool sharding
- Adaptive
buffer_pool_size_mbauto-tune - Vector index variants (IVF, HNSW)
If any of these become required, they need their own roadmap document
- explicit operator decision to extend the v2.x arc. None of them is blocking the v2.x terminal milestone (v2.4 write-handler migration + MemoryStore decommission).
Sequencing
The earliest valuable sub-PR sequence after v2.1.1:
- 5.4 views (smallest, demonstrates the
_viewssub-db pattern) - 5.3 files (similar shape; metadata-only move)
- Shadowman D.1 (their migration, unblocks B5 in v2.2)
- 5.2 history (bigger refactor; touches history_store.cpp + test_snapshot_durability)
- B5 flip default to ns — v2.2 release
- Then write-handler migration / Stage 4-finish — v2.4 (the v2.x terminal architectural milestone)
Open questions
- Is there a consumer beyond shadowman that we need to coordinate with on B5's default flip? (callerai uses ms today.)
- Does the
smartbotic-database-migrate-v1deb (per F1) need to support v1.x → any v2.x.x, or only v1.x → v2.5 (its target)? Affects how much v1.x code we carry inside the migration deb. - For Stage 9, do we wait for a real Zoe-shape soak before shipping v2.4, or is the
load_test_mixed30s proxy acceptable as the gate to MemoryStore decommission?