Эх сурвалжийг харах

release(v2.3.1): two bug fixes from stress + replication tests

Post-2.3.0 stress + replication runs surfaced two real bugs. Both now
fixed; v2.3.1 ships.

## Fix #1 (high) — eviction never fires under ns timestamps

`MemoryStore::getEvictionCandidates` compared `doc.updatedAt` against a
millisecond-scaled `hotWriteFloor`. The v2.2.0 default-precision flip
made `doc.updatedAt` nanoseconds (~10^18) while `hotWriteFloor` stayed
in milliseconds (~10^12). Every doc looked "always within the hot-write
window" → `getEvictionCandidates` returned empty → MemoryStore reached
emergency pressure → admission control rejected all writes.

Pre-existing v2.2.0 regression. load_test_mixed was never re-run
after v2.2.0 shipped, which is why this stayed latent.

Fix: pre-compute both ms- and ns-scaled floors at the top of
getEvictionCandidates; pick per-doc using the 10^15 magnitude
heuristic (same threshold the read path uses for mixed-precision rows).

Verified: load_test_mixed PASSes with 2180 writes/s, 5224 reads/s,
0 hard failures — same shape as pre-2.2.0 baseline.

## Fix #2 (critical) — replicated docs never reach LMDB on followers

`DatabaseService::applyReplicatedEntry` calls
`MemoryStore::loadDocument` which deliberately bypasses the persist
callback (to avoid re-replicating). In v2.0-v2.2 the dual-write
mirror was a side channel and reads still hit MemoryStore, so the
bypass was harmless. In v2.3 LMDB became the primary read path, so
followers ended up with replicated writes in MemoryStore but NOT in
LMDB. Find/Get on the follower returned 0 docs for every replicated
collection.

Fix: in applyReplicatedEntry, after loadDocument, route the entry
through `projects_->getOrCreate(project)` + `applyDualWriteMirror`
directly — the same project-aware mirror the write handlers use.

Verified: load_test_replication PASSes — 100/100 docs replicated in
65ms, zero corruption.

## Other tests

- ctest 14/14 still green
- v2.2→v2.3 stage-C migration (scripted): legacy `env/` atomically
  renamed under `projects/default/env/`, schema_version=2 marker
  preserved across the rename, backfill skips on second boot
- v1.x→v2.0 migration unit test: passes

No client API change. No proto change. Soname stays `.so.2`.
fszontagh 2 сар өмнө
parent
commit
1f35273a45

+ 1 - 0
CLAUDE.md

@@ -41,6 +41,7 @@ cmake --build build -j$(nproc) && ./build/tests/test_vector_storage
 - **Auto-backup before deb upgrade** — `apt upgrade smartbotic-database` runs a `preinst` hook that snapshots `/var/lib/smartbotic-database/` to `/var/backups/smartbotic-database/pre-upgrade-<ts>-from-<version>/` via `cp -a` after the old `prerm` stops the service. Retention `3`, 1.2× free-space precheck on the backup volume (aborts the upgrade if short). Opt out with `SMARTBOTIC_DB_SKIP_BACKUP=1 apt upgrade`. Tunables: `SMARTBOTIC_DB_BACKUP_RETENTION`, `SMARTBOTIC_DB_BACKUP_ROOT`.
 - **yyjson hot-path parsing (v1.10.0+)** — every JSON parse on WAL replay, snapshot deserialize, history reads, gRPC request bodies, and replication apply goes through `smartbotic::db::parse_to_nlohmann` (yyjson-backed) instead of `nlohmann::json::parse`. Measured 2.80× speedup on Zoe-shape corpora (136ms→49ms on 10k docs). In-memory representation stays `nlohmann::json` until Phase B. Wire and on-disk JSON bytes unchanged. New runtime dep: `libyyjson0`. Bench: `./build/tests/bench_json_parse <corpus.jsonl>`.
 - **Binary-backed Document storage (v1.11.0+)** — `Document::data` is now a `smartbotic::db::doc_binary::Doc` (yyjson `mut_doc` wrapper) with a lazy `nlohmann::json` view cache. New accessor surface: `doc.field(name)` is the O(field-count) fast-path that reads one top-level field without materialising the full tree (the dominant read pattern), `doc.data()` is the lazy full-tree accessor, `doc.set_data(j)` re-encodes from an nlohmann tree. Wire and on-disk JSON bytes unchanged. **Honest note:** per-doc heap footprint is ~1.5% smaller than v1.10 (1618→1594 bytes on Zoe-shape) — the win is field-access CPU + Phase C scaffolding, not RSS. See `docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md` post-implementation amendment for the bench numbers. Bench: `./build/tests/bench_doc_memory --mode=<nlohmann|binary> [N]`.
+- **v2.3.1 two bug fixes found by post-2.3.0 stress + replication tests.** (1) `MemoryStore::getEvictionCandidates` compared `doc.updatedAt` against a millisecond-scaled `hotWriteFloor`; v2.2.0's default-precision flip to nanoseconds made every doc look "always within the hot-write window" so eviction never fired. Pre-existing v2.2.0 regression — load_test_mixed was never re-run after v2.2.0 shipped. Fix: pick ms- or ns-scaled floor per-doc using the 10^15 magnitude heuristic. (2) `DatabaseService::applyReplicatedEntry` called `MemoryStore::loadDocument` which bypasses the persist callback, so the v2.3 LMDB mirror never fired for replicated entries. Followers got writes into MemoryStore but the LMDB env stayed empty; v2.3 LMDB-first reads on the follower returned 0 docs. Fix: explicitly drive `applyDualWriteMirror` through the project registry after `loadDocument`. Replication catch-up time on a 100-doc test: 65ms.
 - **v2.3.0 multi-project namespaces.** Each smartbotic-database instance now hosts multiple `project` namespaces, each with its own collections. Wire form: `<project>:<collection>` (e.g. `my_app:users`); bare names like `"users"` auto-qualify to `"default:users"` so every v2.0-v2.2 client keeps working without code changes. Storage: one LMDB env per project at `<dataDir>/projects/<name>/env/`. On first v2.3 boot the legacy `<dataDir>/env/` is atomically renamed under `projects/default/env/` (idempotent; refuses to start if both exist). Server-side: `ProjectStoreRegistry` (`service/src/storage/project_store.{hpp,cpp}`) owns the per-project envs; `DatabaseService::docStore(project)` resolves the LMDB store for a given name; mirror routing is via a resolver callback the service hands to `MemoryStore::setDocumentStoreMirror(resolver, ...)`. Client-side: `Client::Config::project` defaults to `"default"`; every outgoing collection name is auto-prepended (`qualify(coll)`) unless it already contains `:`. Per-call overrides via the qualified form always win. New gRPC RPCs: `ListProjects`, `CreateProject(name)`, `DropProject(name)` — `default` cannot be dropped. CRUD is idempotent and validated against `^[a-zA-Z_][a-zA-Z0-9_-]{0,62}$` (no `:`, no `/`, no `_-prefix). Reserved: `default` (always present); `_-prefix` (system). Tests: `test_project_addressing` (23 parser assertions), `test_project_crud` (29 placeholder assertions). 14/14 ctest green. Stress: load_test_mixed still passes against the multi-project build.
 - **v2.2.1 bug fixes from shadowman code review.** Three issues found by a code-review pass on the v2.0→v2.2.0 commit range, all now fixed: (1) `MemoryStore::updateIfVersion` was missing `extractVector` + `storeVector` + `mirrorVectorToDocStore` calls — optimistic-lock updates on vector collections left the vector mirror stale, so `SimilaritySearch` returned old vector scores while `Get` returned new doc bodies; (2) `DatabaseService::backfillIntoDocStore` walked every doc into LMDB but never set `_meta.schema_version=2`, so every boot re-backfilled the entire dataset (Zoe-scale ~1.85M docs = minutes of pre-READY blocking on every restart); (3) `SimilaritySearch` LMDB handler returned `INTERNAL` instead of `INVALID_ARGUMENT` when an unrelated throw (e.g. bad_alloc) fired after a dimension-mismatch was already observed in the scan. Exposes `smartbotic::db::storage::mark_migration_complete(env)` for the backfill marker. Tests 12/12 green.
 - **v2.2.0 — default timestamp precision flipped to `"ns"`.** BREAKING for fresh collections. `CollectionCfg::timestampPrecision` default went from `"ms"` to `"ns"` so rapid-write collections (chat messages, metrics events, tool-call audits) get nanosecond stamps and strict ordering without needing an explicit `configureCollection()` call. Closes shadowman audit B5. Operators who need ms can still set it explicitly. Existing collections that already have a stored config record keep their value — only collections created post-upgrade WITHOUT an explicit `configureCollection()` inherit the new default. Mixed-precision rows in the same collection are read-correct via the 10^15 magnitude auto-detect; operators wanting clean rows should run `migrateCollectionTimestamps`. **`update()` docstring** updated to clarify: it OVERWRITES the doc; use `patch()` for partial mutations. Reflects shadowman audit B6.

+ 1 - 1
VERSION

@@ -1 +1 @@
-2.3.0
+2.3.1

+ 20 - 0
service/src/database_service.cpp

@@ -980,6 +980,26 @@ void DatabaseService::applyReplicatedEntry(const databasepb::ReplicationEntry& e
                 // the origin-aware overload so eviction/WAL recovery still works.
                 store_->loadDocument(entry.collection(), doc);
 
+                // v2.3.1 — also mirror to LMDB. loadDocument bypasses the
+                // MemoryStore persist callback that normally drives the
+                // dual-write mirror; in v2.0-v2.2 the mirror was a side
+                // channel and reads still hit MemoryStore, so the bypass
+                // was harmless. In v2.3 reads are LMDB-first, so a
+                // follower that only filled MemoryStore would return
+                // empty Find/Get for every replicated doc. Route the
+                // entry through the same project-aware mirror the write
+                // handlers use.
+                if (projects_) {
+                    auto pc = smartbotic::database::parseProjectCollection(
+                                entry.collection());
+                    if (auto* ds = projects_->getOrCreate(pc.project)) {
+                        std::optional<Document> opt_doc(doc);
+                        smartbotic::db::storage::applyDualWriteMirror(
+                            ds, mirror_healthy_, mirror_drift_count_,
+                            pc.collection, doc.id, opt_doc, EventType::INSERT);
+                    }
+                }
+
                 if (persistence_) {
                     uint64_t walSeq = (entry.op() == databasepb::OP_INSERT)
                         ? persistence_->logInsert(entry.collection(), doc, origin)

+ 20 - 4
service/src/memory_store.cpp

@@ -2586,11 +2586,23 @@ uint64_t MemoryStore::evictOneChunk() {
     };
     std::vector<Candidate> candidates;
 
-    const uint64_t now = currentTimeMs();
+    const uint64_t now_ms = currentTimeMs();
     // hotWriteFloorMs may exceed `now` during the first seconds of a run; cap
     // the subtraction so we don't underflow to a huge uint64_t.
-    const uint64_t hotWriteFloor = (config_.hotWriteFloorMs < now)
-        ? (now - config_.hotWriteFloorMs)
+    const uint64_t hotWriteFloor_ms = (config_.hotWriteFloorMs < now_ms)
+        ? (now_ms - config_.hotWriteFloorMs)
+        : 0;
+    // v2.3.1 — per-collection timestamp precision means doc.updatedAt may
+    // be ms or ns. Pre-compute both floors and pick per-doc using the
+    // 10^15 magnitude threshold (same heuristic the read path uses for
+    // mixed-precision rows). Pre-2.3.1 this comparison silently treated
+    // ns timestamps as "always > floor" and eviction never fired.
+    constexpr uint64_t kNsThreshold = 1'000'000'000'000'000ULL;
+    const uint64_t now_ns = now_ms * 1'000'000ULL;
+    const uint64_t hotWriteFloor_ns_offset =
+        static_cast<uint64_t>(config_.hotWriteFloorMs) * 1'000'000ULL;
+    const uint64_t hotWriteFloor_ns = (hotWriteFloor_ns_offset < now_ns)
+        ? (now_ns - hotWriteFloor_ns_offset)
         : 0;
 
     {
@@ -2603,7 +2615,11 @@ uint64_t MemoryStore::evictOneChunk() {
             for (const auto& [docId, doc] : collPtr->documents) {
                 // Hot-write protection: a doc updated within the floor window
                 // is probably still being worked on by a writer. Skip it so
-                // we don't race with a patch()/update() in flight.
+                // we don't race with a patch()/update() in flight. Pick the
+                // ms or ns floor based on the magnitude of this doc's stamp.
+                const uint64_t hotWriteFloor =
+                    (doc.updatedAt >= kNsThreshold) ? hotWriteFloor_ns
+                                                    : hotWriteFloor_ms;
                 if (doc.updatedAt > hotWriteFloor) continue;
 
                 // v1.7.0 T6 quiesce: skip docs with an outstanding write.