2026-05-16-v2.1-backlog.md 6.0 KB
v2.1+ backlog — shadowman audit follow-up + Stage 5-10 remainder
Origin: shadowman team alignment audit /tmp/sbdbv2.md (2026-05-15) +
the unfinished items from 2026-05-15-v2.0-storage-engine-phase-c-execution.md.
This document supersedes the loose "Out of scope (deferred to v2.1+)"
section in 2026-05-15-v2.0-roadmap.md (lines 351-359). Each entry has
an explicit drop trigger so the calendar of removals is observable.
Status of shadowman audit items
F-section (open questions) — answered
| F# | Question | Answer (committed v2.1.1) |
|---|---|---|
| F1 | When does the v1.x auto-migration path drop? | Stays in-binary through v2.x. Extracts to separate smartbotic-database-migrate-v1 deb in v3.0. Trigger: every production host has booted v2.0 at least once + a major version bump. |
| F2 | Is setReadOnly operator-only or client-callable? |
Both. Operator via smartbotic-db-cli unlock; client via Client::lock() for self-defensive migrations. |
| F3 | WAL-fallback fields — zero on healthy v2.0? | Yes, guaranteed zero on the LMDB-served path (Find handler explicitly leaves them at their default-constructed zero). Non-zero = real signal, treat as alert. |
| F4 | migrateCollectionTimestamps cost on hot messages? |
Deferred bench until shadowman runs D.1. Back-of-envelope: ~1ms / 1k docs, online-safe up to ~1M docs; >1M docs schedule a maintenance window. Will be re-measured against real traffic post-D.1. |
B-section (back-compat surfaces) — drop tracker
| B# | Surface | Drop trigger | Target |
|---|---|---|---|
| B1 | In-binary v1.x → v2.0 migration (commit 15893b1) |
Every production host has booted v2.0 ≥ 1× | v3.0 (extract to separate deb) |
| B2 | Defensive MemoryStore fallback on LMDB-throw | Shipped — closed v2.1.1 | ✅ done |
| B3 | Stripped QueryOptions in document.hpp |
Shipped — closed v2.1.0 | ✅ done |
| B4 | Plain find(collection) overload |
Shipped — closed v2.1.0 | ✅ done |
| B5 | Default "ms" timestamp precision |
Every rapid-write collection migrated via shadowman D.1 | v2.2 (flip default to "ns") |
| B6 | update() as RMW verb |
Every RMW callsite converted to patch() via shadowman D.2 |
v2.x.next (mark update() as "replace whole doc only" in docs) |
| B7 | 6-value recoveryOutcome enum |
Write-handler migration ships → MemoryStore decommission → LMDB-only recovery semantics | v3.0 (collapse to single lmdb_ready) |
Shadowman D-section (their migration items)
External dependency — not tracked here. The cookbook from their audit:
- D.1 ns-precision timestamps on
messages/metrics_events/tool_call_blobs/agent_task_messages— highest leverage, unblocks their T6.6 timeline refactor - D.2
patch()adoption for 71 RMW callsites - D.3
count(filters)for "any-matching-row" patterns (~10-15 sites) - D.4
findWithMetrics()for slow-query logging - D.5 Set operations for
conversation_tool_state - D.6 Server-side FilterOp on cold paths (their T7.1-T7.3)
- D.7 Diagnostic
createView()(their T7.4)
Unfinished items from the v2.0 Phase C plan
These were architectural endpoints not reached in v2.0:
Stage 5 — sub-db migration remainder
| Sub | Module | Current state in v2.1 | v2.x target |
|---|---|---|---|
| 5.2 | History _history_<coll> |
Still on service/src/persistence/history_store.cpp .hlog files |
LMDB sub-db keyed by <doc_id>:<version> storing DocumentVersion JSON. Delete history_store.cpp. |
| 5.3 | Files _files |
Metadata still in MemoryStore + .file blob path | _files sub-db for metadata; blobs unchanged at files/<id>. |
| 5.4 | Views _views |
View defs stored in MemoryStore _views system collection |
_views LMDB sub-db keyed by view name. Existing createView / getViewInfo handlers ported. |
Estimated effort: 5.2 medium (HistoryStore deletion + test port), 5.3 + 5.4 small. Each is one focused PR.
Stage 4-finish + Stage 6+ — write-handler migration & beyond
This is the v3.0 milestone: handlers call doc_store_->put() directly with id-gen / timestamp / version orchestration extracted from MemoryStore into a WriteCoordinator. Once landed, MemoryStore deletes; persistence/wal.cpp / persistence/snapshot.cpp delete; eviction config knobs delete; recovery enum collapses (B7).
Subsequent stages from the original plan, mostly unchanged:
- Stage 6 — replication preservation: validate WAL streaming still works through the write-handler-migration boundary
- Stage 7 — migrations RPC: port
MigrationRunnerto callDocumentStore - Stage 8 — eviction + recovery cleanup audit
- Stage 9 — performance gates + 24h soak (NOT done for v2.0 —
load_test_mixed30s was the proxy) - Stage 10 — v3.0 release
Out of session for v2.x — escalated to v3.x backlog
- Secondary user-defined indexes
- Online schema evolution
- Distributed clustering beyond leader+follower
- LMDB writer-pool sharding
- Adaptive
buffer_pool_size_mbauto-tune - Vector index variants (IVF, HNSW)
These are mentioned in the v2.0 roadmap's "Out of scope (deferred to v2.1+)" — they're now formally v3.0+ since the v2.x arc terminates at the write-handler migration.
Sequencing
The earliest valuable sub-PR sequence after v2.1.1:
- 5.4 views (smallest, demonstrates the
_viewssub-db pattern) - 5.3 files (similar shape; metadata-only move)
- Shadowman D.1 (their migration, unblocks B5 in v2.2)
- 5.2 history (bigger refactor; touches history_store.cpp + test_snapshot_durability)
- B5 flip default to ns — v2.2 release
- Then write-handler migration / Stage 4-finish — v3.0 milestone
Open questions
- Is there a consumer beyond shadowman that we need to coordinate with on B5's default flip? (callerai uses ms today.)
- Does the
smartbotic-database-migrate-v1deb (per F1) need to support v1.x → any v2.x, or only v1.x → v3.0 (its target)? Affects how much v1.x code we carry inside the migration deb. - For Stage 9, do we wait for a real Zoe-shape soak before shipping v3.0, or is the load_test_mixed 30s proxy acceptable?