Forráskód Böngészése

fix(persistence): track real WAL seq in eviction stubs (v1.7.4)

EvictedDocumentStub.walSequence used to be set to doc.version (a per-doc
1, 2, 3… counter), not a real WAL sequence. The production load callback
at database_service.cpp:591 therefore couldn't use it as a fromSequence
hint and passed 0, forcing a full-WAL replay on every evicted-doc fault.

With many evictions this was O(WAL_size × evicted_count). On Zoe today
(715k docs, 144k evicted, 820k WAL entries) the eviction loop pinned
192% CPU and starved the LLM service to the point that gateway streams
to OpenRouter idled for 60s and the chat hung.

Fix:
- MemoryStore gains a per-collection per-doc map of last-write WAL
  sequences (recordWalSequence / getWalSequence). Memory cost is small
  and grows only with live docs.
- PersistenceManager::log{Insert,Update,Delete,Upsert} now return the
  assigned WAL sequence (were void).
- The persist callback in DatabaseService captures the returned seq and
  calls store_->recordWalSequence so every WAL append anchors the doc's
  real seq.
- Recovery's WAL-replay path also calls recordWalSequence for each
  INSERT/UPDATE/UPSERT entry it applies, so post-recovery state is
  populated even for docs that haven't been re-written since boot.
- evictDocument call sites use the recorded seq instead of doc.version.
- The load callback in DatabaseService passes walSequence-1 as the
  exclusive fromSequence so replay returns entries with seq >= the
  doc's last write — first hit is the right one. Falls back to full
  scan from 0 when the seq is unknown (e.g. cold doc loaded from a
  pre-v1.7.4 snapshot, no re-write since boot) — same as before.

Adds a regression test (test_evict_stub_carries_real_wal_sequence)
that inserts + updates docs, forces eviction, and asserts every stub's
walSequence equals the actual seq returned by wal.append(), not the
per-doc version counter.

Bundles a few v1.8.0 persistence-hardening commits that were in the
working tree but uncommitted: persistence_->start() before migrations
(so view/_migrations writes during startup reach the WAL), final
snapshot on shutdown (so next boot is trivial-success not WAL-only),
nodeId stamped on WAL entries, and originNodeId on log* methods for
replication echo-skip.
fszontagh 2 hónapja
szülő
commit
5ea5ac14a9

+ 1 - 1
VERSION

@@ -1 +1 @@
-1.7.3
+1.7.4

+ 87 - 7
service/src/database_service.cpp

@@ -133,6 +133,20 @@ bool DatabaseService::initialize() {
         // Set replication sequence after recovery (WAL sequence is now known)
         replication_->setSequence(persistence_->currentWalSequence());
 
+        // v1.8.0 — start the persistence manager before loadFromStore /
+        // migrations so any system-collection writes those phases produce
+        // (view docs, _collection_meta entries, _migrations recordings)
+        // reach the WAL like normal mutations. Pre-v1.8, persistence_ was
+        // started later in start(), which silently dropped those writes
+        // (running_=false → logInsert no-op). That meant migrated views
+        // only "survived" because the runner re-created them every boot,
+        // and a manually-created view that landed during the racy startup
+        // window (before full readiness) could be lost.
+        if (!persistence_->start()) {
+            spdlog::error("Failed to start persistence manager before migrations");
+            return false;
+        }
+
         // Load view definitions from the _views system collection (which is now
         // populated by the persistence recovery above).
         view_manager_->loadFromStore();
@@ -221,6 +235,20 @@ void DatabaseService::stop() {
     if (events_) {
         events_->stop();
     }
+    if (persistence_ && store_) {
+        // v1.8.0 — take a final snapshot on graceful shutdown so the next
+        // boot's recovery is "trivial" (snapshot-loaded) instead of "WAL-only
+        // replay" which trips auto-readonly mode. This makes ordinary
+        // restarts pass through cleanly without needing
+        // `smartbotic-db-cli unlock`.
+        try {
+            persistence_->forceSnapshot(*store_);
+            spdlog::info("Final snapshot taken on shutdown");
+        } catch (const std::exception& e) {
+            spdlog::warn("Final snapshot on shutdown failed: {} — recovery on next "
+                         "boot may be WAL-only and trip auto-readonly mode", e.what());
+        }
+    }
     if (persistence_) {
         persistence_->stop();
     }
@@ -475,6 +503,7 @@ void DatabaseService::setupComponents() {
     persistConfig.walSyncIntervalMs = config_.walSyncIntervalMs;
     persistConfig.snapshotIntervalSec = config_.snapshotIntervalSec;
     persistConfig.compressionEnabled = config_.compressionEnabled;
+    persistConfig.nodeId = config_.nodeId;
     // Mirror the resolved recoveryMode back into our Config so downstream code
     // (logging, RPC handlers) can inspect config_.persistenceConfig.recoveryMode
     // without having to reach into the PersistenceManager.
@@ -484,13 +513,17 @@ void DatabaseService::setupComponents() {
     // Connect store callbacks to persistence - uses setPersistCallback for WAL logging
     store_->setPersistCallback([this](const std::string& collection, const std::string& id,
                                        const std::optional<Document>& doc, EventType eventType) {
-        // Log to WAL based on event type
+        // Log to WAL based on event type. v1.7.4: capture the assigned WAL
+        // sequence and record it on the store so eviction stubs can carry
+        // a real seq (instead of doc.version, which was useless as a
+        // `fromSequence` hint and forced a full WAL scan on every fault).
+        uint64_t walSeq = 0;
         switch (eventType) {
             case EventType::INSERT:
-                if (doc) persistence_->logInsert(collection, *doc);
+                if (doc) walSeq = persistence_->logInsert(collection, *doc);
                 break;
             case EventType::UPDATE:
-                if (doc) persistence_->logUpdate(collection, *doc);
+                if (doc) walSeq = persistence_->logUpdate(collection, *doc);
                 break;
             case EventType::DELETE:
                 persistence_->logDelete(collection, id);
@@ -498,12 +531,19 @@ void DatabaseService::setupComponents() {
             default:
                 break;
         }
+        if (walSeq > 0 && (eventType == EventType::INSERT || eventType == EventType::UPDATE)) {
+            store_->recordWalSequence(collection, id, walSeq);
+        }
 
         // Queue for replication broadcast
         if (replication_ && config_.replicationEnabled) {
             databasepb::ReplicationEntry entry;
             entry.set_collection(collection);
             entry.set_document_id(id);
+            // v1.8.0 — stamp our nodeId so peers can attribute the write to
+            // its actual origin. Pre-v1.8 broadcasts left this empty, which
+            // is now treated by receivers as "legacy / unknown origin".
+            entry.set_node_id(config_.nodeId);
             entry.set_global_timestamp(
                 std::chrono::duration_cast<std::chrono::milliseconds>(
                     std::chrono::system_clock::now().time_since_epoch()
@@ -550,12 +590,20 @@ void DatabaseService::setupComponents() {
         }
     });
 
-    // Set up document load callback for LRU eviction recovery
+    // Set up document load callback for LRU eviction recovery.
+    //
+    // v1.7.4: walSequence is now a real WAL sequence (set by the persist
+    // callback below at append time, not the per-doc version counter).
+    // Pass `walSequence - 1` as the exclusive `fromSequence` so replay
+    // returns entries with seq >= walSequence — the doc's last write is
+    // exactly at walSequence, so the very first hit is the one we want.
+    // For docs whose seq is unknown (0 — e.g. loaded from a snapshot
+    // that predates v1.7.4), fall back to the full-WAL scan from 0.
     store_->setDocumentLoadCallback([this](const std::string& collection,
                                            const std::string& id,
                                            uint64_t walSequence) -> std::optional<Document> {
-        // Load document from WAL via persistence manager
-        return persistence_->loadDocument(collection, id, 0);  // Search from beginning
+        uint64_t fromSequence = walSequence > 0 ? walSequence - 1 : 0;
+        return persistence_->loadDocument(collection, id, fromSequence);
     });
 
     // Create encryption manager
@@ -605,6 +653,17 @@ void DatabaseService::setupComponents() {
 
 void DatabaseService::applyReplicatedEntry(const databasepb::ReplicationEntry& entry) {
     try {
+        // v1.8.0 — also append the replicated entry to the local WAL, tagged
+        // with the originating node's id. This is what makes follower-side
+        // recovery (eviction → WAL fallback, restart → WAL replay) preserve
+        // replicated documents. Echo amplification is prevented by:
+        //   - GetEntriesSince emitting walEntry.nodeId (not local node id)
+        //   - SyncProtocol skipping entries whose origin is the peer itself
+        // both implemented in the same v1.8.0 cycle.
+        const std::string origin = entry.node_id().empty()
+            ? config_.nodeId   // legacy peer (pre-1.8) — best-guess local
+            : entry.node_id();
+
         switch (entry.op()) {
             case databasepb::OP_INSERT:
             case databasepb::OP_UPDATE:
@@ -619,8 +678,20 @@ void DatabaseService::applyReplicatedEntry(const databasepb::ReplicationEntry& e
                 doc.id = entry.document_id();
                 doc.nodeId = entry.node_id();
 
-                // Use loadDocument to bypass normal callbacks (avoid re-replication)
+                // loadDocument bypasses the persist-and-broadcast callback to avoid
+                // re-replicating; we explicitly write to the WAL right after with
+                // the origin-aware overload so eviction/WAL recovery still works.
                 store_->loadDocument(entry.collection(), doc);
+
+                if (persistence_) {
+                    uint64_t walSeq = (entry.op() == databasepb::OP_INSERT)
+                        ? persistence_->logInsert(entry.collection(), doc, origin)
+                        : persistence_->logUpdate(entry.collection(), doc, origin);
+                    if (walSeq > 0) {
+                        store_->recordWalSequence(entry.collection(), doc.id, walSeq);
+                    }
+                }
+
                 spdlog::trace("Applied replicated {} to {}/{} from {}",
                              entry.op() == databasepb::OP_INSERT ? "insert" :
                              entry.op() == databasepb::OP_UPDATE ? "update" : "upsert",
@@ -630,6 +701,9 @@ void DatabaseService::applyReplicatedEntry(const databasepb::ReplicationEntry& e
 
             case databasepb::OP_DELETE: {
                 store_->remove(entry.collection(), entry.document_id());
+                if (persistence_) {
+                    persistence_->logDelete(entry.collection(), entry.document_id(), origin);
+                }
                 spdlog::trace("Applied replicated delete to {}/{} from {}",
                              entry.collection(), entry.document_id(), entry.node_id());
                 break;
@@ -638,6 +712,9 @@ void DatabaseService::applyReplicatedEntry(const databasepb::ReplicationEntry& e
             case databasepb::OP_CREATE_COLLECTION: {
                 CollectionOptions options;
                 store_->createCollection(entry.collection(), options);
+                if (persistence_) {
+                    persistence_->logCreateCollection(entry.collection(), options, origin);
+                }
                 spdlog::trace("Applied replicated create collection {} from {}",
                              entry.collection(), entry.node_id());
                 break;
@@ -645,6 +722,9 @@ void DatabaseService::applyReplicatedEntry(const databasepb::ReplicationEntry& e
 
             case databasepb::OP_DROP_COLLECTION: {
                 store_->dropCollection(entry.collection());
+                if (persistence_) {
+                    persistence_->logDropCollection(entry.collection(), origin);
+                }
                 spdlog::trace("Applied replicated drop collection {} from {}",
                              entry.collection(), entry.node_id());
                 break;

+ 27 - 24
service/src/memory_store.cpp

@@ -2490,6 +2490,23 @@ void MemoryStore::setDocumentLoadCallback(DocumentLoadCallback callback) {
     documentLoadCallback_ = std::move(callback);
 }
 
+void MemoryStore::recordWalSequence(const std::string& collection,
+                                     const std::string& id, uint64_t walSeq) {
+    if (walSeq == 0) return;  // 0 is reserved as the "unknown" sentinel
+    std::unique_lock<std::shared_mutex> lock(docWalSeqMutex_);
+    docWalSeq_[collection][id] = walSeq;
+}
+
+uint64_t MemoryStore::getWalSequence(const std::string& collection,
+                                      const std::string& id) const {
+    std::shared_lock<std::shared_mutex> lock(docWalSeqMutex_);
+    auto cIt = docWalSeq_.find(collection);
+    if (cIt == docWalSeq_.end()) return 0;
+    auto dIt = cIt->second.find(id);
+    if (dIt == cIt->second.end()) return 0;
+    return dIt->second;
+}
+
 void MemoryStore::logMemoryCheck() {
     auto stats = getStats();
     spdlog::info("Memory check: {} MB estimated ({}%, pressure={}), {} docs, {} evicted",
@@ -2705,16 +2722,13 @@ uint64_t MemoryStore::evictOneChunk() {
     uint64_t bytesFreedBefore = estimatedMemoryBytes_.load(std::memory_order_relaxed);
     uint64_t evictedCount = 0;
     for (const auto& [coll, id] : toEvict) {
-        uint64_t walSequence = 0;
-        {
-            std::shared_lock<std::shared_mutex> glock(globalMutex_);
-            auto it = collections_.find(coll);
-            if (it == collections_.end()) continue;
-            std::shared_lock<std::shared_mutex> clock(it->second->mutex);
-            auto dIt = it->second->documents.find(id);
-            if (dIt == it->second->documents.end()) continue;
-            walSequence = dIt->second.version;
-        }
+        // Use the recorded real WAL sequence (v1.7.4+). Pre-v1.7.4 this
+        // path stored doc.version (a per-doc 1,2,3… counter) which was
+        // useless as a `fromSequence` hint and forced load callbacks to
+        // scan the full WAL on every evicted-doc fault. Falls back to 0
+        // when unknown — same behaviour as before for cold/snapshot-only
+        // docs that haven't been re-written since boot.
+        uint64_t walSequence = getWalSequence(coll, id);
         if (evictDocument(coll, id, walSequence)) {
             evictedCount++;
         }
@@ -2761,20 +2775,9 @@ uint64_t MemoryStore::evictDocuments() {
 
     uint64_t evicted = 0;
     for (const auto& [collection, id] : candidates) {
-        // Get current WAL sequence (we'll use version as proxy since we don't have direct WAL access)
-        CollectionData* coll = getOrCreateCollection(collection);
-        uint64_t walSequence = 0;
-
-        {
-            std::shared_lock<std::shared_mutex> lock(coll->mutex);
-            auto it = coll->documents.find(id);
-            if (it != coll->documents.end()) {
-                // Use version as a proxy for WAL sequence
-                // The actual WAL sequence will be determined by the persistence layer
-                walSequence = it->second.version;
-            }
-        }
-
+        // Use the recorded real WAL sequence (v1.7.4+) — same fix as
+        // `evictOneChunk`. See `recordWalSequence` doc for rationale.
+        uint64_t walSequence = getWalSequence(collection, id);
         if (evictDocument(collection, id, walSequence)) {
             evicted++;
         }

+ 31 - 0
service/src/memory_store.hpp

@@ -454,6 +454,28 @@ public:
      */
     void setDocumentLoadCallback(DocumentLoadCallback callback);
 
+    /**
+     * Record the WAL sequence assigned to the most recent write of a
+     * document. Called by the persist callback after each successful
+     * `wal_->append` and during WAL replay on recovery.
+     *
+     * Pre-v1.7.4 the eviction stub stored `doc.version` (a per-doc 1,2,3…
+     * counter) under the misleading name `walSequence`, so the load
+     * callback fell back to a full-WAL scan from sequence 0 on every
+     * evicted-doc fault. With this map populated, `evictDocument` can
+     * record the real WAL sequence in the stub, and the load callback
+     * can replay only the tail of the WAL (`replay(walSeq - 1, …)`).
+     */
+    void recordWalSequence(const std::string& collection, const std::string& id, uint64_t walSeq);
+
+    /**
+     * Get the most recently recorded WAL sequence for a doc, or 0 if
+     * unknown (e.g. doc loaded from a snapshot that predates v1.7.4 and
+     * was never re-written since boot). Callers should treat 0 as
+     * "no hint — fall back to full scan."
+     */
+    [[nodiscard]] uint64_t getWalSequence(const std::string& collection, const std::string& id) const;
+
     /**
      * Trigger eviction if memory exceeds threshold.
      * Called automatically by background thread.
@@ -745,6 +767,15 @@ private:
     // Callback for loading evicted documents from WAL
     DocumentLoadCallback documentLoadCallback_;
 
+    // Per-document last-write WAL sequence map. Populated by the persist
+    // callback after each WAL append and by recovery's WAL replay. Used
+    // by `evictDocument` to populate `EvictedDocumentStub.walSequence`
+    // with a real seq instead of the per-doc version counter, so load
+    // callbacks can scan a tight WAL window. Memory cost is small
+    // (~16 bytes per doc id) and only grows with live docs.
+    mutable std::shared_mutex docWalSeqMutex_;
+    std::unordered_map<std::string, std::unordered_map<std::string, uint64_t>> docWalSeq_;
+
     // Eviction helpers
     std::vector<std::pair<std::string, std::string>> selectDocumentsForEviction(uint64_t bytesToFree);
     bool evictDocument(const std::string& collection, const std::string& id, uint64_t walSequence);

+ 20 - 12
service/src/persistence/persistence_manager.cpp

@@ -35,6 +35,9 @@ void applyWalEntry(MemoryStore& store, const WalEntry& entry) {
             if (entry.data) {
                 Document doc = Document::fromJson(*entry.data);
                 store.loadDocument(entry.collection, doc);
+                // v1.7.4: anchor the per-doc WAL sequence so post-recovery
+                // evictions get a real `fromSequence` hint, not 0.
+                store.recordWalSequence(entry.collection, entry.documentId, entry.sequence);
             }
             break;
         case WalOpType::UPDATE:
@@ -42,6 +45,7 @@ void applyWalEntry(MemoryStore& store, const WalEntry& entry) {
             if (entry.data) {
                 Document doc = Document::fromJson(*entry.data);
                 store.loadDocumentWithHistory(entry.collection, doc);
+                store.recordWalSequence(entry.collection, entry.documentId, entry.sequence);
             }
             break;
         case WalOpType::DELETE:
@@ -366,36 +370,40 @@ RecoveryOutcome PersistenceManager::recover(MemoryStore& store) {
     return outcome;
 }
 
-void PersistenceManager::logInsert(const std::string& collection, const Document& doc,
-                                    const std::string& originNodeId) {
-    if (!running_.load()) return;
+uint64_t PersistenceManager::logInsert(const std::string& collection, const Document& doc,
+                                        const std::string& originNodeId) {
+    if (!running_.load()) return 0;
     const std::string& origin = originNodeId.empty() ? config_.nodeId : originNodeId;
     uint64_t seq = wal_->append(WriteAheadLog::makeInsertEntry(collection, doc, origin));
     updateCollectionSequence(collection, seq);
+    return seq;
 }
 
-void PersistenceManager::logUpdate(const std::string& collection, const Document& doc,
-                                    const std::string& originNodeId) {
-    if (!running_.load()) return;
+uint64_t PersistenceManager::logUpdate(const std::string& collection, const Document& doc,
+                                        const std::string& originNodeId) {
+    if (!running_.load()) return 0;
     const std::string& origin = originNodeId.empty() ? config_.nodeId : originNodeId;
     uint64_t seq = wal_->append(WriteAheadLog::makeUpdateEntry(collection, doc, origin));
     updateCollectionSequence(collection, seq);
+    return seq;
 }
 
-void PersistenceManager::logDelete(const std::string& collection, const std::string& id,
-                                    const std::string& originNodeId) {
-    if (!running_.load()) return;
+uint64_t PersistenceManager::logDelete(const std::string& collection, const std::string& id,
+                                        const std::string& originNodeId) {
+    if (!running_.load()) return 0;
     const std::string& origin = originNodeId.empty() ? config_.nodeId : originNodeId;
     uint64_t seq = wal_->append(WriteAheadLog::makeDeleteEntry(collection, id, origin));
     updateCollectionSequence(collection, seq);
+    return seq;
 }
 
-void PersistenceManager::logUpsert(const std::string& collection, const Document& doc,
-                                    const std::string& originNodeId) {
-    if (!running_.load()) return;
+uint64_t PersistenceManager::logUpsert(const std::string& collection, const Document& doc,
+                                        const std::string& originNodeId) {
+    if (!running_.load()) return 0;
     const std::string& origin = originNodeId.empty() ? config_.nodeId : originNodeId;
     uint64_t seq = wal_->append(WriteAheadLog::makeUpsertEntry(collection, doc, origin));
     updateCollectionSequence(collection, seq);
+    return seq;
 }
 
 void PersistenceManager::logCreateCollection(const std::string& collection, const CollectionOptions& options,

+ 31 - 11
service/src/persistence/persistence_manager.hpp

@@ -81,6 +81,10 @@ public:
         RecoveryMode recoveryMode = RecoveryMode::Normal;
         bool autoEscalate = true;
         bool allowEmptyOnFreshInstall = true;
+
+        // v1.8.0 — local node id, stamped onto WAL entries written by this
+        // node so replication can distinguish own writes from peer writes.
+        std::string nodeId;
     };
 
     explicit PersistenceManager(Config config);
@@ -116,44 +120,60 @@ public:
 
     /**
      * Log an insert operation.
+     * `originNodeId` (v1.8.0) is the node that originally produced this
+     * write. Empty defaults to the local node — passing the leader's id
+     * for replicated writes is what makes follower-side eviction-recovery
+     * preserve the leader as origin (and lets replication skip echo).
+     *
+     * v1.7.4: returns the assigned WAL sequence so the caller (the
+     * MemoryStore persist callback in database_service.cpp) can record
+     * it for use by the eviction stub. Pre-v1.7.4 these were void.
      */
-    void logInsert(const std::string& collection, const Document& doc);
+    uint64_t logInsert(const std::string& collection, const Document& doc,
+                       const std::string& originNodeId = "");
 
     /**
-     * Log an update operation.
+     * Log an update operation. Returns the assigned WAL sequence.
      */
-    void logUpdate(const std::string& collection, const Document& doc);
+    uint64_t logUpdate(const std::string& collection, const Document& doc,
+                       const std::string& originNodeId = "");
 
     /**
-     * Log a delete operation.
+     * Log a delete operation. Returns the assigned WAL sequence.
      */
-    void logDelete(const std::string& collection, const std::string& id);
+    uint64_t logDelete(const std::string& collection, const std::string& id,
+                       const std::string& originNodeId = "");
 
     /**
-     * Log an upsert operation.
+     * Log an upsert operation. Returns the assigned WAL sequence.
      */
-    void logUpsert(const std::string& collection, const Document& doc);
+    uint64_t logUpsert(const std::string& collection, const Document& doc,
+                       const std::string& originNodeId = "");
 
     /**
      * Log collection creation.
      */
-    void logCreateCollection(const std::string& collection, const CollectionOptions& options);
+    void logCreateCollection(const std::string& collection, const CollectionOptions& options,
+                             const std::string& originNodeId = "");
 
     /**
      * Log collection drop.
      */
-    void logDropCollection(const std::string& collection);
+    void logDropCollection(const std::string& collection,
+                           const std::string& originNodeId = "");
 
     /**
      * Log a vector put operation.
      */
     uint64_t logVecPut(const std::string& collection, const std::string& docId,
-                       const std::vector<float>& vec);
+                       const std::vector<float>& vec,
+                       const std::string& originNodeId = "");
 
     /**
      * Log a vector delete operation.
      */
-    uint64_t logVecDelete(const std::string& collection, const std::string& docId);
+    uint64_t logVecDelete(const std::string& collection, const std::string& docId,
+                          const std::string& originNodeId = "");
 
     /**
      * Force an immediate snapshot.

+ 93 - 0
tests/test_snapshot_durability.cpp

@@ -523,6 +523,98 @@ void test_snapshot_includes_evicted_documents() {
               << N << " round-tripped)\n";
 }
 
+// ──────────────────────────────────────────────────────────
+// Test: eviction stubs carry the real WAL sequence (v1.7.4 fix)
+// ──────────────────────────────────────────────────────────
+//
+// Pre-v1.7.4 the stub stored doc.version (1, 2, 3…) under the misleading
+// name `walSequence`. Production load callbacks therefore couldn't use
+// it as a `fromSequence` hint and fell back to scanning the entire WAL
+// from sequence 0 on every evicted-doc fault — O(WAL_size) per fault,
+// O(WAL_size × evicted_count) per eviction tick.
+void test_evict_stub_carries_real_wal_sequence() {
+    using smartbotic::database::WriteAheadLog;
+    auto walDir = makeSnapshotDir("stub-walseq");
+
+    WriteAheadLog::Config walCfg;
+    walCfg.walDir = walDir;
+    WriteAheadLog wal(walCfg);
+    if (!wal.open()) { std::cerr << "wal.open failed\n"; std::exit(2); }
+
+    MemoryStore::Config storeCfg;
+    storeCfg.nodeId = "test";
+    storeCfg.maxMemoryBytes = 32ULL * 1024;
+    storeCfg.evictionThresholdPercent = 50;
+    storeCfg.evictionTargetPercent = 30;
+    storeCfg.hotWriteFloorMs = 0;
+    storeCfg.evictionChunkSize = 1000;
+
+    MemoryStore store(storeCfg);
+    store.start();
+    store.createCollection("docs", CollectionOptions{});
+
+    // Insert N docs. Mirror to WAL and record the actual sequence on the
+    // store — this is what database_service.cpp's persist callback does.
+    constexpr int N = 50;
+    std::unordered_map<std::string, uint64_t> expectedSeq;
+    for (int i = 0; i < N; ++i) {
+        Document d = makeDoc("doc-" + std::to_string(i));
+        d.data["padding"] = std::string(300, 'x');
+        uint64_t seq = wal.append(WriteAheadLog::makeInsertEntry("docs", d, "test"));
+        store.insert("docs", d);
+        store.recordWalSequence("docs", d.id, seq);
+        expectedSeq[d.id] = seq;
+    }
+
+    // Re-write a few docs so their `walSequence` should match the LATEST
+    // append, not the first one.
+    for (int i = 0; i < 5; ++i) {
+        Document d = makeDoc("doc-" + std::to_string(i));
+        d.data["padding"] = std::string(400, 'y');
+        d.version = 2;
+        uint64_t seq = wal.append(WriteAheadLog::makeUpdateEntry("docs", d, "test"));
+        store.update("docs", d.id, d);
+        store.recordWalSequence("docs", d.id, seq);
+        expectedSeq[d.id] = seq;
+    }
+
+    store.evictDocuments();
+    auto evictedIds = store.getEvictedDocumentIds("docs");
+    if (evictedIds.empty()) {
+        std::cerr << "FAIL: no docs evicted — eviction config too lax\n";
+        std::exit(1);
+    }
+
+    // Every evicted stub must have the real (latest-append) seq, not 0
+    // and not the per-doc version (1 or 2).
+    for (const auto& id : evictedIds) {
+        auto stub = store.getEvictedStub("docs", id);
+        if (!stub) {
+            std::cerr << "FAIL: no stub for " << id << "\n";
+            std::exit(1);
+        }
+        uint64_t want = expectedSeq[id];
+        if (stub->walSequence != want) {
+            std::cerr << "FAIL: stub.walSequence=" << stub->walSequence
+                      << " for " << id << ", expected " << want
+                      << " (real WAL sequence)\n";
+            std::exit(1);
+        }
+        // Sanity: must not equal a per-doc version counter (≤ 2 here).
+        if (stub->walSequence <= 2) {
+            std::cerr << "FAIL: stub.walSequence=" << stub->walSequence
+                      << " looks like a per-doc version, not a WAL seq\n";
+            std::exit(1);
+        }
+    }
+
+    wal.close();
+    fs::remove_all(walDir);
+    std::cout << "PASS: evict stubs carry real WAL sequence ("
+              << evictedIds.size() << " stubs verified, all match "
+              << "their last wal.append() return value)\n";
+}
+
 int main() {
     test_atomic_write_leaves_no_tmp();
     test_snapshot_round_trip();
@@ -533,6 +625,7 @@ int main() {
     test_v4_format_roundtrip_at_scale();
     test_wal_sequence_floor_anchors_after_truncate();
     test_snapshot_includes_evicted_documents();
+    test_evict_stub_carries_real_wal_sequence();
     std::cout << "\nAll snapshot durability tests PASSED!\n";
     return 0;
 }