Bladeren bron

fix(eviction): peek-only path for evicted-doc reads under high memory pressure (v1.9.2)

Defensive fix for the cap-bound replica scenario tracked in task #16
(load_test_replica_eviction). Under hard or emergency memory pressure,
`MemoryStore::get` for an evicted doc now uses `peekEvictedDocument`
(read-only, no re-insert) instead of `loadEvictedDocument` (hot-load
back into `coll->documents`). The hot-load path is a warm-cache
optimization that pays off when there's headroom, but under pressure
the just-loaded doc bumps `estimatedMemoryBytes_` over the cap,
eviction immediately removes it again, and concurrent reads queue
behind the per-collection mutex — the thrash cycle that produced
4002/5000 read deadline misses pre-v1.9.0.

Under normal pressure (the steady-state Zoe case) behavior is
unchanged — we still hot-load and get the warm-cache benefit.

In practice on the load_test_replica_eviction harness v1.9.0's
accurate `estimatedMemoryBytes_` already keeps pressure at ~45-49%
(below the soft threshold), so this peek-on-pressure path doesn't
activate. It is purely a safety net for pathological workloads that
do push pressure into the hard/emergency band.

Verification run (10 min cap): 0 "Failed to recover" log lines over
the whole test (was 4002 in v1.8.x). Phase 3 throughput is still
slow — each read does a full WAL fallback scan even with the
walSeq-1 hint, and the test reads 5000 unique IDs sequentially — but
correctness is intact. A real fix for the throughput side needs a
per-doc WAL index, which is v1.10.0+ scope.

No new code paths or files; one `pressure()` check + lambda in `get`.
fszontagh 2 maanden geleden
bovenliggende
commit
cb6c070e8f
2 gewijzigde bestanden met toevoegingen van 23 en 3 verwijderingen
  1. 1 1
      VERSION
  2. 22 2
      service/src/memory_store.cpp

+ 1 - 1
VERSION

@@ -1 +1 @@
-1.9.1
+1.9.2

+ 22 - 2
service/src/memory_store.cpp

@@ -314,11 +314,31 @@ std::string MemoryStore::insert(const std::string& collection, Document doc) {
 std::optional<Document> MemoryStore::get(const std::string& collection, const std::string& id) const {
     auto startTime = std::chrono::steady_clock::now();
 
+    // v1.9.2 — under hard or emergency memory pressure, evicted-doc reads
+    // go through the read-only `peekEvictedDocument` path. Pre-1.9.2 we
+    // always used `loadEvictedDocument`, which hot-loads the doc back
+    // into `coll->documents`. That's a warm-cache optimization that
+    // helps when there's headroom but actively hurts under pressure:
+    // the hot-loaded doc bumps `estimatedMemoryBytes_` over the cap,
+    // eviction immediately evicts it again, the next read hot-loads
+    // again — a thrash cycle that maps onto the replica-eviction
+    // test's 4002/5000 read-deadline misses. The peek path doesn't
+    // touch memory state, so concurrent reads under pressure don't
+    // contend with the eviction loop.
+    auto evictedReadFn = [this](const std::string& c, const std::string& i)
+        -> std::optional<Document> {
+        const MemoryPressure p = pressure();
+        if (p == MemoryPressure::Hard || p == MemoryPressure::Emergency) {
+            return peekEvictedDocument(c, i);
+        }
+        return const_cast<MemoryStore*>(this)->loadEvictedDocument(c, i);
+    };
+
     const CollectionData* coll = getCollection(collection);
     if (!coll) {
         // Check if document was evicted (collection may not exist in memory)
         if (const_cast<MemoryStore*>(this)->isDocumentEvicted(collection, id)) {
-            auto result = const_cast<MemoryStore*>(this)->loadEvictedDocument(collection, id);
+            auto result = evictedReadFn(collection, id);
             // Record timing
             auto elapsed = std::chrono::duration_cast<std::chrono::microseconds>(
                 std::chrono::steady_clock::now() - startTime).count();
@@ -337,7 +357,7 @@ std::optional<Document> MemoryStore::get(const std::string& collection, const st
         lock.unlock();
         // Check if document was evicted
         if (const_cast<MemoryStore*>(this)->isDocumentEvicted(collection, id)) {
-            auto result = const_cast<MemoryStore*>(this)->loadEvictedDocument(collection, id);
+            auto result = evictedReadFn(collection, id);
             // Record timing
             auto elapsed = std::chrono::duration_cast<std::chrono::microseconds>(
                 std::chrono::steady_clock::now() - startTime).count();