Kaynağa Gözat

feat(storage): hard-flip Exists+Get reads to LMDB-first

Stress-validated against load_test_mixed (4 writers / 8 readers / 30s):
240k operations, hard memory pressure (82%), 66k MemoryStore evictions,
zero shadow-read divergence across the full run. With the evidence
banked, both single-doc read handlers now go to doc_store_ first when
the mirror is healthy and drift is zero.

Race-safe fallback: on LMDB not-found, probe MemoryStore once before
returning absent. Closes the microsecond window between MemoryStore
lock release (memory_store.cpp:294) and dual-write commit (line 313)
where a fresh insert is visible to MemoryStore readers but not yet to
LMDB readers. Find stays on MemoryStore for now — its set semantics
don't compose cleanly with the race-fallback pattern.

LMDB-hit path: one mdb_get. Cheap. Only negative results pay the
MemoryStore probe, and those are already cheap (in-memory hash).
fszontagh 2 ay önce
ebeveyn
işleme
ba78aa94bc
1 değiştirilmiş dosya ile 36 ekleme ve 37 silme
  1. 36 37
      service/src/database_grpc_impl.cpp

+ 36 - 37
service/src/database_grpc_impl.cpp

@@ -163,38 +163,33 @@ grpc::Status DatabaseGrpcImpl::Get(
         targetCollection = view->collection;
     }
 
-    auto doc = store_.get(targetCollection, request->id());
-
-    // v2.0 Stage 4 — shadow-read on Get. Same gates as Exists: only when
-    // the mirror is healthy + zero drift + non-system collection. Compares
-    // presence and core fields (id, version, data). Divergence is logged
-    // at WARN. MemoryStore stays authoritative.
+    // v2.0 Stage 4 — LMDB-first read. Same gate as Exists; on LMDB
+    // not-found, fall back to MemoryStore.get() to close the dual-write
+    // race window. On LMDB throw, fall back and log WARN. The fallback
+    // is a single in-memory probe (cheap), so the hot path is one LMDB
+    // get + one MemoryStore probe in the worst case.
+    std::optional<Document> doc;
+    bool used_lmdb = false;
     if (!targetCollection.empty() && targetCollection[0] != '_'
         && service_.mirrorHealthy() && service_.mirrorDriftCount() == 0) {
         if (auto* ds = service_.docStore()) {
             try {
-                auto lmdb_doc = ds->get(targetCollection, request->id());
-                const bool mem_found = doc.has_value();
-                const bool lmdb_found = lmdb_doc.has_value();
-                if (mem_found != lmdb_found) {
-                    spdlog::warn("v2.0 shadow-read Get presence divergence coll={} id={} "
-                                 "memstore={} lmdb={}",
-                                 targetCollection, request->id(), mem_found, lmdb_found);
-                } else if (mem_found && lmdb_found) {
-                    if (doc->id != lmdb_doc->id || doc->version != lmdb_doc->version
-                        || doc->data() != lmdb_doc->data()) {
-                        spdlog::warn("v2.0 shadow-read Get content divergence coll={} id={} "
-                                     "mem_ver={} lmdb_ver={}",
-                                     targetCollection, request->id(),
-                                     doc->version, lmdb_doc->version);
-                    }
-                }
+                doc = ds->get(targetCollection, request->id());
+                used_lmdb = true;
             } catch (const std::exception& e) {
-                spdlog::warn("v2.0 shadow-read Get query failed coll={} id={}: {}",
+                spdlog::warn("v2.0 Get LMDB query failed coll={} id={}: {} "
+                             "-- falling back to MemoryStore",
                              targetCollection, request->id(), e.what());
             }
         }
     }
+    if (!used_lmdb || !doc) {
+        // LMDB blocked / threw / returned absent — try MemoryStore. For
+        // the LMDB-blocked case it's the only path. For the absent case
+        // it catches the mirror-write race for fresh inserts.
+        auto mem_doc = store_.get(targetCollection, request->id());
+        if (mem_doc) doc = std::move(mem_doc);
+    }
 
     if (!doc) {
         response->set_found(false);
@@ -483,29 +478,33 @@ grpc::Status DatabaseGrpcImpl::Exists(
     const pb::ExistsRequest* request,
     pb::ExistsResponse* response
 ) {
-    bool exists = store_.exists(request->collection(), request->id());
-
-    // v2.0 Stage 4 — shadow-read: when the mirror is healthy and has zero
-    // drift, also ask doc_store_ and log any divergence. MemoryStore is
-    // still authoritative; this generates the evidence we need before the
-    // next sub-task hard-flips reads onto doc_store_. System collections
-    // are skipped because they aren't mirrored (Stage 5 owns their format).
+    // v2.0 Stage 4 — LMDB-first read. Gated on mirror_healthy_ + zero
+    // drift + non-system collection + doc_store_ available. On LMDB
+    // not-found, fall back to MemoryStore.exists() to close the small
+    // race window between MemoryStore lock release and dual-write commit
+    // (see memory_store.cpp:294 / 313). On LMDB throw, fall back to
+    // MemoryStore and log WARN.
+    bool exists = false;
+    bool used_lmdb = false;
     if (!request->collection().empty() && request->collection()[0] != '_'
         && service_.mirrorHealthy() && service_.mirrorDriftCount() == 0) {
         if (auto* ds = service_.docStore()) {
             try {
-                const bool lmdb_exists = ds->exists(request->collection(), request->id());
-                if (lmdb_exists != exists) {
-                    spdlog::warn("v2.0 shadow-read Exists divergence coll={} id={} "
-                                 "memstore={} lmdb={}",
-                                 request->collection(), request->id(), exists, lmdb_exists);
-                }
+                exists = ds->exists(request->collection(), request->id());
+                used_lmdb = true;
             } catch (const std::exception& e) {
-                spdlog::warn("v2.0 shadow-read Exists query failed coll={} id={}: {}",
+                spdlog::warn("v2.0 Exists LMDB query failed coll={} id={}: {} "
+                             "-- falling back to MemoryStore",
                              request->collection(), request->id(), e.what());
             }
         }
     }
+    if (!used_lmdb || !exists) {
+        // Either gates blocked LMDB, LMDB threw, or LMDB said no — in the
+        // not-found case we also probe MemoryStore so a freshly-inserted
+        // doc whose mirror hasn't committed yet still reports exists=true.
+        exists = exists || store_.exists(request->collection(), request->id());
+    }
 
     response->set_exists(exists);
     return grpc::Status::OK;