# v2.1+ backlog — shadowman audit follow-up + Stage 5-10 remainder

Origin: shadowman team alignment audit `/tmp/sbdbv2.md` (2026-05-15) +
the unfinished items from `2026-05-15-v2.0-storage-engine-phase-c-execution.md`.

This document supersedes the loose "Out of scope (deferred to v2.1+)"
section in `2026-05-15-v2.0-roadmap.md` (lines 351-359). Each entry has
an explicit drop trigger so the calendar of removals is observable.

## Status of shadowman audit items

### F-section (open questions) — answered

| F# | Question | Answer (committed v2.1.1) |
|----|----------|---------------------------|
| F1 | When does the v1.x auto-migration path drop? | Stays in-binary through v2.x. Extracts to separate `smartbotic-database-migrate-v1` deb in v3.0. Trigger: every production host has booted v2.0 at least once + a major version bump. |
| F2 | Is `setReadOnly` operator-only or client-callable? | Both. Operator via `smartbotic-db-cli unlock`; client via `Client::lock()` for self-defensive migrations. |
| F3 | WAL-fallback fields — zero on healthy v2.0? | Yes, guaranteed zero on the LMDB-served path (Find handler explicitly leaves them at their default-constructed zero). Non-zero = real signal, treat as alert. |
| F4 | `migrateCollectionTimestamps` cost on hot `messages`? | Deferred bench until shadowman runs D.1. Back-of-envelope: ~1ms / 1k docs, online-safe up to ~1M docs; >1M docs schedule a maintenance window. Will be re-measured against real traffic post-D.1. |

### B-section (back-compat surfaces) — drop tracker

| B# | Surface | Drop trigger | Target |
|----|---------|--------------|--------|
| B1 | In-binary v1.x → v2.0 migration (commit `15893b1`) | Every production host has booted v2.0 ≥ 1× | v3.0 (extract to separate deb) |
| B2 | Defensive MemoryStore fallback on LMDB-throw | Shipped — closed v2.1.1 | ✅ done |
| B3 | Stripped `QueryOptions` in document.hpp | Shipped — closed v2.1.0 | ✅ done |
| B4 | Plain `find(collection)` overload | Shipped — closed v2.1.0 | ✅ done |
| B5 | Default `"ms"` timestamp precision | Every rapid-write collection migrated via shadowman D.1 | v2.2 (flip default to `"ns"`) |
| B6 | `update()` as RMW verb | Every RMW callsite converted to `patch()` via shadowman D.2 | v2.x.next (mark `update()` as "replace whole doc only" in docs) |
| B7 | 6-value `recoveryOutcome` enum | Write-handler migration ships → MemoryStore decommission → LMDB-only recovery semantics | v3.0 (collapse to single `lmdb_ready`) |

### Shadowman D-section (their migration items)

External dependency — not tracked here. The cookbook from their audit:

* D.1 ns-precision timestamps on `messages` / `metrics_events` / `tool_call_blobs` / `agent_task_messages` — **highest leverage**, unblocks their T6.6 timeline refactor
* D.2 `patch()` adoption for 71 RMW callsites
* D.3 `count(filters)` for "any-matching-row" patterns (~10-15 sites)
* D.4 `findWithMetrics()` for slow-query logging
* D.5 Set operations for `conversation_tool_state`
* D.6 Server-side FilterOp on cold paths (their T7.1-T7.3)
* D.7 Diagnostic `createView()` (their T7.4)

## Unfinished items from the v2.0 Phase C plan

These were architectural endpoints not reached in v2.0:

### Stage 5 — sub-db migration remainder

| Sub | Module | Current state in v2.1 | v2.x target |
|-----|--------|----------------------|-------------|
| 5.2 | History `_history_<coll>` | Still on `service/src/persistence/history_store.cpp` .hlog files | LMDB sub-db keyed by `<doc_id>:<version>` storing DocumentVersion JSON. Delete `history_store.cpp`. |
| 5.3 | Files `_files` | Metadata still in MemoryStore + .file blob path | `_files` sub-db for metadata; blobs unchanged at `files/<id>`. |
| 5.4 | Views `_views` | View defs stored in MemoryStore `_views` system collection | `_views` LMDB sub-db keyed by view name. Existing `createView` / `getViewInfo` handlers ported. |

Estimated effort: 5.2 medium (HistoryStore deletion + test port), 5.3 + 5.4 small. Each is one focused PR.

### Stage 4-finish + Stage 6+ — write-handler migration & beyond

This is the v3.0 milestone: handlers call `doc_store_->put()` directly with id-gen / timestamp / version orchestration extracted from MemoryStore into a WriteCoordinator. Once landed, MemoryStore deletes; persistence/wal.cpp / persistence/snapshot.cpp delete; eviction config knobs delete; recovery enum collapses (B7).

Subsequent stages from the original plan, mostly unchanged:

* Stage 6 — replication preservation: validate WAL streaming still works through the write-handler-migration boundary
* Stage 7 — migrations RPC: port `MigrationRunner` to call `DocumentStore`
* Stage 8 — eviction + recovery cleanup audit
* Stage 9 — performance gates + 24h soak (NOT done for v2.0 — `load_test_mixed` 30s was the proxy)
* Stage 10 — v3.0 release

## Out of session for v2.x — escalated to v3.x backlog

* Secondary user-defined indexes
* Online schema evolution
* Distributed clustering beyond leader+follower
* LMDB writer-pool sharding
* Adaptive `buffer_pool_size_mb` auto-tune
* Vector index variants (IVF, HNSW)

These are mentioned in the v2.0 roadmap's "Out of scope (deferred to
v2.1+)" — they're now formally v3.0+ since the v2.x arc terminates at
the write-handler migration.

## Sequencing

The earliest valuable sub-PR sequence after v2.1.1:

1. **5.4 views** (smallest, demonstrates the `_views` sub-db pattern)
2. **5.3 files** (similar shape; metadata-only move)
3. **Shadowman D.1** (their migration, unblocks B5 in v2.2)
4. **5.2 history** (bigger refactor; touches history_store.cpp + test_snapshot_durability)
5. **B5 flip default to ns** — v2.2 release
6. Then write-handler migration / Stage 4-finish — v3.0 milestone

## Open questions

* Is there a consumer beyond shadowman that we need to coordinate with on B5's default flip? (callerai uses ms today.)
* Does the `smartbotic-database-migrate-v1` deb (per F1) need to support v1.x → any v2.x, or only v1.x → v3.0 (its target)? Affects how much v1.x code we carry inside the migration deb.
* For Stage 9, do we wait for a real Zoe-shape soak before shipping v3.0, or is the load_test_mixed 30s proxy acceptable?