Преглед изворни кода

release(v2.1.1): drop defensive LMDB-throw fallback + audit answers

Cleanup release following shadowman team's v2.0 alignment audit
(/tmp/sbdbv2.md). Internal-only changes — gRPC wire surface unchanged,
client API unchanged from v2.1.0.

## What changed

- **No defensive fallback on LMDB throw** (Exists / Get / Find /
  SimilaritySearch). A live LMDB exception now returns gRPC INTERNAL
  instead of being silently masked by MemoryStore. Closes audit item
  B2. The gate-closed fallback (mirror_healthy_=false, system collection,
  env unopened) stays — that's correctness, since MemoryStore can
  legitimately be ahead of LMDB when a mirror write has failed.
- **CLAUDE.md** now documents F1-F4 answers:
  - F1: v1.x auto-migration stays in-binary through v2.x, extracts to
    `smartbotic-database-migrate-v1` deb in v3.0
  - F2: setReadOnly is dual-audience (operator + client)
  - F3: WAL-fallback fields are guaranteed zero on LMDB-served path;
    any non-zero value is a real signal
  - F4: migrateCollectionTimestamps bench deferred until shadowman runs
    D.1; estimated online-safe to ~1M docs
- **New: `docs/superpowers/plans/2026-05-16-v2.1-backlog.md`** — the
  canonical successor to the v2.0 roadmap's loose "Out of scope" list.
  Each deferred item now has an explicit drop trigger. Sequencing for
  v2.2 / v3.0 milestones is documented.

## Closes / advances

- audit F1, F2, F3 — answered (F4 partial)
- audit B2 — defensive fallback dropped
- audit B3, B4 — already closed in v2.1.0

## Still in v2.1+ backlog

- audit B1, B5, B6, B7 — pending (each tied to a different drop trigger)
- Stage 5.2/5.3/5.4 — history/files/views sub-db migration
- Stage 4-finish — write-handler migration (v3.0 milestone)

Full suite (12/12) green. No throughput impact — the removed branch was
the exception path, which doesn't fire under normal operation.
fszontagh пре 2 месеци
родитељ
комит
1f6cfa5d39
4 измењених фајлова са 118 додато и 14 уклоњено
  1. 2 0
      CLAUDE.md
  2. 1 1
      VERSION
  3. 99 0
      docs/superpowers/plans/2026-05-16-v2.1-backlog.md
  4. 16 13
      service/src/database_grpc_impl.cpp

Разлика између датотеке није приказан због своје велике величине
+ 2 - 0
CLAUDE.md


+ 1 - 1
VERSION

@@ -1 +1 @@
-2.1.0
+2.1.1

+ 99 - 0
docs/superpowers/plans/2026-05-16-v2.1-backlog.md

@@ -0,0 +1,99 @@
+# v2.1+ backlog — shadowman audit follow-up + Stage 5-10 remainder
+
+Origin: shadowman team alignment audit `/tmp/sbdbv2.md` (2026-05-15) +
+the unfinished items from `2026-05-15-v2.0-storage-engine-phase-c-execution.md`.
+
+This document supersedes the loose "Out of scope (deferred to v2.1+)"
+section in `2026-05-15-v2.0-roadmap.md` (lines 351-359). Each entry has
+an explicit drop trigger so the calendar of removals is observable.
+
+## Status of shadowman audit items
+
+### F-section (open questions) — answered
+
+| F# | Question | Answer (committed v2.1.1) |
+|----|----------|---------------------------|
+| F1 | When does the v1.x auto-migration path drop? | Stays in-binary through v2.x. Extracts to separate `smartbotic-database-migrate-v1` deb in v3.0. Trigger: every production host has booted v2.0 at least once + a major version bump. |
+| F2 | Is `setReadOnly` operator-only or client-callable? | Both. Operator via `smartbotic-db-cli unlock`; client via `Client::lock()` for self-defensive migrations. |
+| F3 | WAL-fallback fields — zero on healthy v2.0? | Yes, guaranteed zero on the LMDB-served path (Find handler explicitly leaves them at their default-constructed zero). Non-zero = real signal, treat as alert. |
+| F4 | `migrateCollectionTimestamps` cost on hot `messages`? | Deferred bench until shadowman runs D.1. Back-of-envelope: ~1ms / 1k docs, online-safe up to ~1M docs; >1M docs schedule a maintenance window. Will be re-measured against real traffic post-D.1. |
+
+### B-section (back-compat surfaces) — drop tracker
+
+| B# | Surface | Drop trigger | Target |
+|----|---------|--------------|--------|
+| B1 | In-binary v1.x → v2.0 migration (commit `15893b1`) | Every production host has booted v2.0 ≥ 1× | v3.0 (extract to separate deb) |
+| B2 | Defensive MemoryStore fallback on LMDB-throw | Shipped — closed v2.1.1 | ✅ done |
+| B3 | Stripped `QueryOptions` in document.hpp | Shipped — closed v2.1.0 | ✅ done |
+| B4 | Plain `find(collection)` overload | Shipped — closed v2.1.0 | ✅ done |
+| B5 | Default `"ms"` timestamp precision | Every rapid-write collection migrated via shadowman D.1 | v2.2 (flip default to `"ns"`) |
+| B6 | `update()` as RMW verb | Every RMW callsite converted to `patch()` via shadowman D.2 | v2.x.next (mark `update()` as "replace whole doc only" in docs) |
+| B7 | 6-value `recoveryOutcome` enum | Write-handler migration ships → MemoryStore decommission → LMDB-only recovery semantics | v3.0 (collapse to single `lmdb_ready`) |
+
+### Shadowman D-section (their migration items)
+
+External dependency — not tracked here. The cookbook from their audit:
+
+* D.1 ns-precision timestamps on `messages` / `metrics_events` / `tool_call_blobs` / `agent_task_messages` — **highest leverage**, unblocks their T6.6 timeline refactor
+* D.2 `patch()` adoption for 71 RMW callsites
+* D.3 `count(filters)` for "any-matching-row" patterns (~10-15 sites)
+* D.4 `findWithMetrics()` for slow-query logging
+* D.5 Set operations for `conversation_tool_state`
+* D.6 Server-side FilterOp on cold paths (their T7.1-T7.3)
+* D.7 Diagnostic `createView()` (their T7.4)
+
+## Unfinished items from the v2.0 Phase C plan
+
+These were architectural endpoints not reached in v2.0:
+
+### Stage 5 — sub-db migration remainder
+
+| Sub | Module | Current state in v2.1 | v2.x target |
+|-----|--------|----------------------|-------------|
+| 5.2 | History `_history_<coll>` | Still on `service/src/persistence/history_store.cpp` .hlog files | LMDB sub-db keyed by `<doc_id>:<version>` storing DocumentVersion JSON. Delete `history_store.cpp`. |
+| 5.3 | Files `_files` | Metadata still in MemoryStore + .file blob path | `_files` sub-db for metadata; blobs unchanged at `files/<id>`. |
+| 5.4 | Views `_views` | View defs stored in MemoryStore `_views` system collection | `_views` LMDB sub-db keyed by view name. Existing `createView` / `getViewInfo` handlers ported. |
+
+Estimated effort: 5.2 medium (HistoryStore deletion + test port), 5.3 + 5.4 small. Each is one focused PR.
+
+### Stage 4-finish + Stage 6+ — write-handler migration & beyond
+
+This is the v3.0 milestone: handlers call `doc_store_->put()` directly with id-gen / timestamp / version orchestration extracted from MemoryStore into a WriteCoordinator. Once landed, MemoryStore deletes; persistence/wal.cpp / persistence/snapshot.cpp delete; eviction config knobs delete; recovery enum collapses (B7).
+
+Subsequent stages from the original plan, mostly unchanged:
+
+* Stage 6 — replication preservation: validate WAL streaming still works through the write-handler-migration boundary
+* Stage 7 — migrations RPC: port `MigrationRunner` to call `DocumentStore`
+* Stage 8 — eviction + recovery cleanup audit
+* Stage 9 — performance gates + 24h soak (NOT done for v2.0 — `load_test_mixed` 30s was the proxy)
+* Stage 10 — v3.0 release
+
+## Out of session for v2.x — escalated to v3.x backlog
+
+* Secondary user-defined indexes
+* Online schema evolution
+* Distributed clustering beyond leader+follower
+* LMDB writer-pool sharding
+* Adaptive `buffer_pool_size_mb` auto-tune
+* Vector index variants (IVF, HNSW)
+
+These are mentioned in the v2.0 roadmap's "Out of scope (deferred to
+v2.1+)" — they're now formally v3.0+ since the v2.x arc terminates at
+the write-handler migration.
+
+## Sequencing
+
+The earliest valuable sub-PR sequence after v2.1.1:
+
+1. **5.4 views** (smallest, demonstrates the `_views` sub-db pattern)
+2. **5.3 files** (similar shape; metadata-only move)
+3. **Shadowman D.1** (their migration, unblocks B5 in v2.2)
+4. **5.2 history** (bigger refactor; touches history_store.cpp + test_snapshot_durability)
+5. **B5 flip default to ns** — v2.2 release
+6. Then write-handler migration / Stage 4-finish — v3.0 milestone
+
+## Open questions
+
+* Is there a consumer beyond shadowman that we need to coordinate with on B5's default flip? (callerai uses ms today.)
+* Does the `smartbotic-database-migrate-v1` deb (per F1) need to support v1.x → any v2.x, or only v1.x → v3.0 (its target)? Affects how much v1.x code we carry inside the migration deb.
+* For Stage 9, do we wait for a real Zoe-shape soak before shipping v3.0, or is the load_test_mixed 30s proxy acceptable?

+ 16 - 13
service/src/database_grpc_impl.cpp

@@ -185,9 +185,10 @@ grpc::Status DatabaseGrpcImpl::Get(
                 doc = ds->get(targetCollection, request->id());
                 used_lmdb = true;
             } catch (const std::exception& e) {
-                spdlog::warn("v2.0 Get LMDB query failed coll={} id={}: {} "
-                             "-- falling back to MemoryStore",
-                             targetCollection, request->id(), e.what());
+                // v2.1.1 — no defensive fallback on throw; surface INTERNAL.
+                spdlog::error("v2.1 Get LMDB query failed coll={} id={}: {}",
+                              targetCollection, request->id(), e.what());
+                return grpc::Status(grpc::StatusCode::INTERNAL, e.what());
             }
         }
     }
@@ -499,9 +500,10 @@ grpc::Status DatabaseGrpcImpl::Exists(
                 exists = ds->exists(request->collection(), request->id());
                 used_lmdb = true;
             } catch (const std::exception& e) {
-                spdlog::warn("v2.0 Exists LMDB query failed coll={} id={}: {} "
-                             "-- falling back to MemoryStore",
-                             request->collection(), request->id(), e.what());
+                // v2.1.1 — no defensive fallback on throw; surface INTERNAL.
+                spdlog::error("v2.1 Exists LMDB query failed coll={} id={}: {}",
+                              request->collection(), request->id(), e.what());
+                return grpc::Status(grpc::StatusCode::INTERNAL, e.what());
             }
         }
     }
@@ -758,9 +760,10 @@ grpc::Status DatabaseGrpcImpl::Find(
                 // WAL-fallback metrics intentionally zero on the LMDB path.
                 used_lmdb = true;
             } catch (const std::exception& e) {
-                spdlog::warn("v2.0 Find LMDB scan failed coll={}: {} "
-                             "-- falling back to MemoryStore",
-                             targetCollection, e.what());
+                // v2.1.1 — no defensive fallback on throw; surface INTERNAL.
+                spdlog::error("v2.1 Find LMDB scan failed coll={}: {}",
+                              targetCollection, e.what());
+                return grpc::Status(grpc::StatusCode::INTERNAL, e.what());
             }
         }
     }
@@ -949,10 +952,10 @@ grpc::Status DatabaseGrpcImpl::SimilaritySearch(
                     // INVALID_ARGUMENT.
                     throw;
                 } catch (const std::exception& e) {
-                    spdlog::warn("v2.0 SimilaritySearch LMDB scan failed "
-                                 "coll={}: {} -- falling back to MemoryStore",
-                                 collection, e.what());
-                    results.clear();  // partial top-K is unsafe to keep
+                    // v2.1.1 — no defensive fallback on throw; surface INTERNAL.
+                    spdlog::error("v2.1 SimilaritySearch LMDB scan failed coll={}: {}",
+                                  collection, e.what());
+                    return grpc::Status(grpc::StatusCode::INTERNAL, e.what());
                 }
             }
         }

Неке датотеке нису приказане због велике количине промена