feat(history): move version history off the heap onto disk (v1.9.0)

Architectural fix for the untracked-RSS leak observed on Zoe
(BUG-memory-leak-zoe.md). The dominant component of the 3 GB gap
between `estimatedMemoryBytes_` (~5 GB) and actual VmRSS (~8 GB) was
`CollectionData::versionHistory` — a per-doc std::deque of
DocumentVersions, each holding a full nlohmann::json tree of the past
state. On hot-update collections with `max_versions: 0` (ShadowMan's
default for `messages`, `documents`, `kb_*`, plus
`conversation_context_usage` which is updated on every LLM turn) the
deques grew without bound. The data was already on disk in the WAL;
the in-memory copy was pure redundancy that no allocator could shrink
and no tracker correctly counted.

v1.8.0 tried to fix this by dropping the deque on eviction. That
helped under eviction pressure but never triggered on Zoe's actual
workload — the budget kept memory at 69%, well under the 85%
threshold, so eviction stayed dormant and the deques stayed pinned.

v1.9.0 fixes the architecture instead of patching the symptom:

  - New component `HistoryStore` (persistence/history_store.{hpp,cpp}).
    One append-only file per collection at
    `<data_dir>/history/<collection>.hlog`. Same length-framed,
    CRC-checked binary format as the WAL — easy to debug, robust to
    partial writes.

  - `CollectionData::versionHistory` deleted. `MemoryStore::saveToHistory`
    routes every prior version to HistoryStore::append. Reads
    (`getVersionHistory`, `getDocumentAtVersion`, `restoreToVersion`,
    `restoreToDate`) read from the file. Latency for getVersionHistory
    shifts from O(1) map lookup to O(N) file scan — acceptable for an
    audit/debug feature; ShadowMan (the largest consumer) doesn't call
    these in any hot path (verified via grep).

  - `estimatedMemoryBytes_` is now accurate. Pre-v1.9 it included
    version-history bytes accounted via `addedBytes` in saveToHistory;
    those bytes were "in heap" only by the budget's accounting, but
    real allocator overhead was 4-5× larger than the 2.5× multiplier
    used. Now the budget tracks exactly what's in `coll->documents`
    and `coll->vectors` — which is what eviction can actually free.

  - Snapshot deserializer migrates old v1.8.x snapshot history blocks
    straight to disk via the existing `loadVersionHistory` entry point
    (now reroutes to HistoryStore::append). No data loss for operators
    who used the feature.

  - Snapshot serializer emits a zero-count history block for forward
    compat with v1.8.x readers; v2.x can drop the block entirely.

  - `alterCollection` with a lower max_versions now prunes the on-disk
    .hlog file (HistoryStore::prune); `dropCollection` removes the file.

  - v1.8.0's history-on-eviction code (memory_store.cpp evictDocument)
    is gone — no in-memory history left to drop.

Smoke test (synthetic 60s workload, 20k inserts + 18k updates on a
single hot collection): tracker 106 MB, VmRSS 86 MB — gap closed. The
9.1 MB `conversation_context_usage.hlog` is the version-history data
that pre-v1.9 would have lived on the heap.

Expected Zoe impact: the ~3 GB untracked-RSS gap collapses. Memory
becomes a predictable function of `max_memory_mb`. Disk usage grows
with update history, controllable per-collection via `max_versions`.

Versioning: minor bump (v1.9.0). API surface unchanged. Behavior
contract `getVersionHistory` semantics: same return shape, latency
shifts from microseconds to single-digit milliseconds.

VERSION

@@ -1 +1 @@
-1.8.3
+1.9.0

service/CMakeLists.txt

@@ -18,6 +18,7 @@ set(DATABASE_SERVICE_SOURCES
     src/persistence/persistence_manager.cpp
     src/persistence/wal.cpp
     src/persistence/snapshot.cpp
+    src/persistence/history_store.cpp
     src/encryption/encryption_manager.cpp
     src/events/event_manager.cpp
     src/files/file_manager.cpp

service/src/database_service.cpp

@@ -512,6 +512,17 @@ void DatabaseService::setupComponents() {
     config_manager_ = std::make_unique<CollectionConfigManager>(*store_);
     store_->setConfigManager(config_manager_.get());
 
+    // v1.9.0 — disk-resident version history. Replaces the in-heap
+    // `CollectionData::versionHistory` deques that were the dominant
+    // source of the Zoe untracked-RSS gap. One file per collection at
+    // `<dataDir>/history/<collection>.hlog`. Lifecycle: constructed
+    // here, wired into the store immediately so snapshot deserializer
+    // can migrate pre-v1.9 in-memory history blocks straight to disk.
+    HistoryStore::Config histConfig;
+    histConfig.dataDir = config_.dataDirectory;
+    history_store_ = std::make_unique<HistoryStore>(histConfig);
+    store_->setHistoryStore(history_store_.get());
+
     // Create persistence manager. Start from any persistenceConfig values the
     // caller (config loader / CLI parser) has already populated — including
     // recoveryMode — and overlay the top-level convenience fields.

service/src/database_service.hpp

@@ -4,6 +4,7 @@
 #include "memory_store.hpp"
 #include "database_grpc_impl.hpp"
 #include "persistence/persistence_manager.hpp"
+#include "persistence/history_store.hpp"
 #include "encryption/encryption_manager.hpp"
 #include "events/event_manager.hpp"
 #include "files/file_manager.hpp"
@@ -198,6 +199,7 @@ private:
     // Components
     std::unique_ptr<MemoryStore> store_;
     std::unique_ptr<PersistenceManager> persistence_;
+    std::unique_ptr<HistoryStore> history_store_;  // v1.9.0 — disk-resident history
     std::unique_ptr<EncryptionManager> encryption_;
     std::unique_ptr<EventManager> events_;
     std::unique_ptr<FileManager> files_;

service/src/memory_store.cpp

@@ -1,6 +1,7 @@
 #include "memory_store.hpp"
 
 #include "config/collection_config_manager.hpp"
+#include "persistence/history_store.hpp"
 
 #include <spdlog/spdlog.h>
 
@@ -120,15 +121,12 @@ bool MemoryStore::dropCollection(const std::string& name) {
     {
         std::shared_lock<std::shared_mutex> collLock(it->second->mutex);
         docCount = it->second->documents.size();
-        // Calculate total memory being freed (documents + version history + vectors)
+        // v1.9.0 — only live docs + vectors count toward the in-heap
+        // budget. Version history lives on disk now (see HistoryStore)
+        // and is reclaimed below via `historyStore_->dropCollection`.
         for (const auto& [id, doc] : it->second->documents) {
             totalSize += estimateDocumentSize(doc);
         }
-        for (const auto& [id, history] : it->second->versionHistory) {
-            for (const auto& ver : history) {
-                totalSize += estimateDocumentVersionSize(ver);
-            }
-        }
         for (const auto& [id, vec] : it->second->vectors) {
             totalSize += estimateVectorSize(vec);
         }
@@ -136,6 +134,11 @@ bool MemoryStore::dropCollection(const std::string& name) {
 
     collections_.erase(it);
 
+    // v1.9.0 — also drop the disk-resident history file for this collection.
+    if (historyStore_) {
+        historyStore_->dropCollection(name);
+    }
+
     {
         std::lock_guard<std::mutex> statsLock(statsMutex_);
         stats_.totalCollections--;
@@ -166,24 +169,16 @@ bool MemoryStore::alterCollection(const std::string& name, const CollectionOptio
     it->second->options = options;
     it->second->updatedAt = currentTimeMs();
 
-    // If maxVersions is now limited (and was unlimited or higher before), prune all history
-    if (newMaxVersions > 0 && (oldMaxVersions == 0 || newMaxVersions < oldMaxVersions)) {
-        uint64_t prunedVersions = 0;
-        uint64_t prunedBytes = 0;
-        for (auto& [docId, history] : it->second->versionHistory) {
-            while (history.size() > newMaxVersions) {
-                prunedBytes += estimateDocumentVersionSize(history.front());
-                history.pop_front();
-                ++prunedVersions;
-            }
-        }
-        if (prunedBytes > 0) {
-            estimatedMemoryBytes_.fetch_sub(prunedBytes, std::memory_order_relaxed);
-        }
-        if (prunedVersions > 0) {
-            spdlog::info("Pruned {} version history entries from collection '{}' (maxVersions: {} -> {})",
-                         prunedVersions, name, oldMaxVersions, newMaxVersions);
-        }
+    // v1.9.0 — if maxVersions is being lowered, rewrite the on-disk history
+    // file keeping at most newMaxVersions per docId. Pre-v1.9 this iterated
+    // the in-memory deques; the new HistoryStore::prune does the same logic
+    // on the .hlog file. Synchronous because alter is rare and operators
+    // expect their setting to take effect immediately.
+    if (historyStore_ && newMaxVersions > 0 &&
+        (oldMaxVersions == 0 || newMaxVersions < oldMaxVersions)) {
+        historyStore_->prune(name, newMaxVersions);
+        spdlog::info("Pruned on-disk version history for collection '{}' (maxVersions: {} -> {})",
+                     name, oldMaxVersions, newMaxVersions);
     }
 
     spdlog::debug("Altered collection '{}': pinned={}, encrypted={}, maxVersions={}",
@@ -1611,6 +1606,14 @@ MemoryStore::MemoryStatsSnapshot MemoryStore::getMemoryStatsSnapshot() const {
 // ===== Version History =====
 
 void MemoryStore::saveToHistory(CollectionData& coll, const Document& currentDoc) {
+    // v1.9.0 — history is disk-resident now (see HistoryStore). If no
+    // historyStore_ is attached (tests, ad-hoc embeds), the call is a
+    // no-op — history is discarded. Pre-v1.9 this method built a
+    // DocumentVersion, pushed it onto an in-heap deque, and stamped
+    // `estimatedMemoryBytes_` with the version's overhead. That was
+    // the dominant source of the Zoe RSS gap; see BUG-memory-leak-zoe.md.
+    if (!historyStore_) return;
+
     DocumentVersion ver;
     ver.version = currentDoc.version;
     ver.data = currentDoc.data;
@@ -1621,20 +1624,12 @@ void MemoryStore::saveToHistory(CollectionData& coll, const Document& currentDoc
     ver.createdAt = currentDoc.createdAt;
     ver.createdBy = currentDoc.createdBy;
 
-    uint64_t addedBytes = estimateDocumentVersionSize(ver);
-
-    auto& history = coll.versionHistory[currentDoc.id];
-    history.push_back(std::move(ver));
-    estimatedMemoryBytes_.fetch_add(addedBytes, std::memory_order_relaxed);
-
-    // Prune oldest versions if maxVersions is configured
-    if (coll.options.maxVersions > 0) {
-        while (history.size() > coll.options.maxVersions) {
-            uint64_t freed = estimateDocumentVersionSize(history.front());
-            history.pop_front();
-            estimatedMemoryBytes_.fetch_sub(freed, std::memory_order_relaxed);
-        }
-    }
+    historyStore_->append(currentDoc.collection, currentDoc.id, ver);
+    // Per-collection `max_versions` pruning runs in the snapshot loop
+    // (PersistenceManager::createSnapshot calls historyStore_->prune for
+    // every collection that has a cap set). Doing it inline on every
+    // write would either rewrite the file each time (expensive) or
+    // require an in-memory counter we no longer keep.
 }
 
 MemoryStore::VersionHistoryResult MemoryStore::getVersionHistory(
@@ -1656,34 +1651,21 @@ MemoryStore::VersionHistoryResult MemoryStore::getVersionHistory(
         result.currentVersion = docIt->second.version;
     }
 
-    // Get version history
-    auto histIt = coll->versionHistory.find(id);
-    if (histIt == coll->versionHistory.end() || histIt->second.empty()) {
-        // No history — still return current version info if document exists
-        if (docIt == coll->documents.end()) {
-            return result;  // No document, no history
-        }
-        return result;
+    // v1.9.0 — historical versions come from the disk-resident
+    // HistoryStore. Newest-first ordering is preserved (HistoryStore
+    // returns them in that order). Without a HistoryStore attached the
+    // result is "no historical versions, current is whatever it is" —
+    // matching the empty-deque branch of the pre-v1.9 code.
+    if (historyStore_) {
+        result.totalCount = historyStore_->countVersions(collection, id);
+        result.versions = historyStore_->getVersions(collection, id, limit, offset);
     }
 
-    const auto& history = histIt->second;
-    result.totalCount = history.size();
-
     // Check if document was deleted (has history but no active document)
-    if (docIt == coll->documents.end()) {
+    if (result.totalCount > 0 && docIt == coll->documents.end()) {
         result.documentDeleted = true;
     }
 
-    // Apply pagination (newest first: offset=0 returns most recent versions)
-    uint32_t total = static_cast<uint32_t>(history.size());
-    uint32_t skipFromEnd = std::min(offset, total);
-    uint32_t available = total - skipFromEnd;
-    uint32_t count = limit > 0 ? std::min(limit, available) : available;
-
-    for (uint32_t i = 0; i < count; ++i) {
-        result.versions.push_back(history[total - skipFromEnd - 1 - i]);
-    }
-
     return result;
 }
 
@@ -1713,19 +1695,9 @@ std::optional<DocumentVersion> MemoryStore::getDocumentAtVersion(
         return ver;
     }
 
-    // Search in history
-    auto histIt = coll->versionHistory.find(id);
-    if (histIt == coll->versionHistory.end()) {
-        return std::nullopt;
-    }
-
-    for (const auto& ver : histIt->second) {
-        if (ver.version == version) {
-            return ver;
-        }
-    }
-
-    return std::nullopt;
+    // v1.9.0 — historical versions live on disk now.
+    if (!historyStore_) return std::nullopt;
+    return historyStore_->getAtVersion(collection, id, version);
 }
 
 uint64_t MemoryStore::restoreToVersion(const std::string& collection, const std::string& id,
@@ -1734,23 +1706,14 @@ uint64_t MemoryStore::restoreToVersion(const std::string& collection, const std:
 
     std::unique_lock<std::shared_mutex> lock(coll->mutex);
 
-    // Find the target version in history
-    auto histIt = coll->versionHistory.find(id);
-    if (histIt == coll->versionHistory.end()) {
-        return 0;
-    }
-
-    const DocumentVersion* targetVer = nullptr;
-    for (const auto& ver : histIt->second) {
-        if (ver.version == version) {
-            targetVer = &ver;
-            break;
-        }
-    }
-
-    if (!targetVer) {
-        return 0;
-    }
+    // v1.9.0 — pull the target version from disk-resident history. If
+    // no history store is wired (tests/embedded), restore is a no-op
+    // for non-current versions — matches pre-v1.9 behavior on an empty
+    // versionHistory map.
+    if (!historyStore_) return 0;
+    auto targetOpt = historyStore_->getAtVersion(collection, id, version);
+    if (!targetOpt) return 0;
+    const DocumentVersion& targetVer = *targetOpt;
 
     auto docIt = coll->documents.find(id);
     bool wasDeleted = (docIt == coll->documents.end());
@@ -1762,34 +1725,36 @@ uint64_t MemoryStore::restoreToVersion(const std::string& collection, const std:
         saveToHistory(*coll, docIt->second);
 
         newVersion = docIt->second.version + 1;
-        docIt->second.data = targetVer->data;
+        docIt->second.data = targetVer.data;
         docIt->second.version = newVersion;
         docIt->second.updatedAt = currentTimeFor(collection);
         docIt->second.updatedBy = actor;
-        docIt->second.encrypted = targetVer->encrypted;
-        docIt->second.encryptedFields = targetVer->encryptedFields;
+        docIt->second.encrypted = targetVer.encrypted;
+        docIt->second.encryptedFields = targetVer.encryptedFields;
         docIt->second.nodeId = config_.nodeId;
 
         restoredDoc = docIt->second;
     } else {
-        // Deleted document: reconstruct from version history
-        // Find the highest version in history to determine new version number
-        uint64_t maxVer = 0;
-        for (const auto& ver : histIt->second) {
-            maxVer = std::max(maxVer, ver.version);
+        // Deleted document: reconstruct from version history. Find the
+        // highest seen version to assign the next version number. With
+        // disk-resident history this is O(N) over the file, but the
+        // restore path is interactive / rare.
+        uint64_t maxVer = version;  // at least the version we're restoring
+        for (const auto& v : historyStore_->getVersions(collection, id, 0, 0)) {
+            maxVer = std::max(maxVer, v.version);
         }
         newVersion = maxVer + 1;
 
         restoredDoc.id = id;
         restoredDoc.collection = collection;
-        restoredDoc.data = targetVer->data;
+        restoredDoc.data = targetVer.data;
         restoredDoc.version = newVersion;
-        restoredDoc.createdAt = targetVer->createdAt;
-        restoredDoc.createdBy = targetVer->createdBy;
+        restoredDoc.createdAt = targetVer.createdAt;
+        restoredDoc.createdBy = targetVer.createdBy;
         restoredDoc.updatedAt = currentTimeFor(collection);
         restoredDoc.updatedBy = actor;
-        restoredDoc.encrypted = targetVer->encrypted;
-        restoredDoc.encryptedFields = targetVer->encryptedFields;
+        restoredDoc.encrypted = targetVer.encrypted;
+        restoredDoc.encryptedFields = targetVer.encryptedFields;
         restoredDoc.nodeId = config_.nodeId;
 
         // Apply default TTL if configured
@@ -1838,14 +1803,15 @@ std::pair<uint64_t, uint64_t> MemoryStore::restoreToDate(
     {
         std::shared_lock<std::shared_mutex> lock(coll->mutex);
 
-        // Search version history for the version active at the given timestamp
-        auto histIt = coll->versionHistory.find(id);
-        if (histIt != coll->versionHistory.end()) {
-            for (const auto& ver : histIt->second) {
+        // v1.9.0 — historical versions are on disk. The HistoryStore
+        // returns them newest-first; we scan all of them to find the
+        // largest version whose timestamp is still <= the target.
+        if (historyStore_) {
+            for (const auto& ver : historyStore_->getVersions(collection, id, 0, 0)) {
                 if (ver.timestamp <= timestamp) {
+                    // newest-first iteration: first match is the one we want
                     targetVersion = ver.version;
-                } else {
-                    break;  // Versions are ordered oldest-first
+                    break;
                 }
             }
         }
@@ -2006,50 +1972,28 @@ void MemoryStore::loadDocumentWithHistory(const std::string& collection, Documen
 }
 
 std::vector<std::pair<std::string, std::deque<DocumentVersion>>>
-MemoryStore::getAllVersionHistory(const std::string& collection) const {
-    const CollectionData* coll = getCollection(collection);
-    if (!coll) {
-        return {};
-    }
-
-    std::shared_lock<std::shared_mutex> lock(coll->mutex);
-
-    std::vector<std::pair<std::string, std::deque<DocumentVersion>>> result;
-    result.reserve(coll->versionHistory.size());
-    for (const auto& [docId, history] : coll->versionHistory) {
-        if (!history.empty()) {
-            result.emplace_back(docId, history);
-        }
-    }
-    return result;
+MemoryStore::getAllVersionHistory(const std::string& /*collection*/) const {
+    // v1.9.0 — history is on disk now; the snapshot writer is no longer
+    // responsible for it (see SnapshotManager::serializeStore which
+    // emits a zeroed history block for forward-compat). Returning an
+    // empty vector is the correct steady-state behavior. Kept on the
+    // class for API compat; no callers within the codebase use this
+    // for anything but snapshotting now.
+    return {};
 }
 
 void MemoryStore::loadVersionHistory(const std::string& collection, const std::string& docId,
                                       std::deque<DocumentVersion> history) {
-    CollectionData* coll = getOrCreateCollection(collection);
-
-    std::unique_lock<std::shared_mutex> lock(coll->mutex);
-
-    // Account for any history being replaced
-    uint64_t oldBytes = 0;
-    auto existing = coll->versionHistory.find(docId);
-    if (existing != coll->versionHistory.end()) {
-        for (const auto& ver : existing->second) {
-            oldBytes += estimateDocumentVersionSize(ver);
-        }
-    }
-
-    uint64_t newBytes = 0;
-    for (const auto& ver : history) {
-        newBytes += estimateDocumentVersionSize(ver);
-    }
-
-    coll->versionHistory[docId] = std::move(history);
-
-    if (newBytes > oldBytes) {
-        estimatedMemoryBytes_.fetch_add(newBytes - oldBytes, std::memory_order_relaxed);
-    } else if (oldBytes > newBytes) {
-        estimatedMemoryBytes_.fetch_sub(oldBytes - newBytes, std::memory_order_relaxed);
+    // v1.9.0 — called by SnapshotManager when deserializing v1.8.x
+    // snapshots that still carried embedded history blocks. We migrate
+    // each version straight to the on-disk HistoryStore so operators
+    // don't lose retained history during the upgrade. If no
+    // HistoryStore is attached (tests), the call is a no-op — the
+    // history is silently dropped, matching "history feature off"
+    // semantics. Pre-v1.9 this method built the in-memory deque.
+    if (!historyStore_) return;
+    for (auto& ver : history) {
+        historyStore_->append(collection, docId, ver);
     }
 }
 
@@ -2879,23 +2823,10 @@ bool MemoryStore::evictDocument(const std::string& collection, const std::string
     // Remove document from memory
     coll->documents.erase(it);
 
-    // v1.8.0 — also drop the doc's version history. Before this, eviction only
-    // freed the live document but left `versionHistory[id]` intact, which on
-    // hot-write collections with `max_versions: 0` (e.g. ShadowMan `messages`,
-    // `documents`, `kb_*`) caused unbounded growth: every update pushed a
-    // copy of the prior version into the deque, eviction never touched it,
-    // and the orphaned deque stayed pinned to the heap forever. See
-    // BUG-memory-leak-zoe.md. Hot-load via `loadEvictedDocument` already
-    // doesn't restore history, so dropping it here is a no-op vs the
-    // pre-v1.8 user-observable contract.
-    uint64_t historyBytesFreed = 0;
-    auto histIt = coll->versionHistory.find(id);
-    if (histIt != coll->versionHistory.end()) {
-        for (const auto& ver : histIt->second) {
-            historyBytesFreed += estimateDocumentVersionSize(ver);
-        }
-        coll->versionHistory.erase(histIt);
-    }
+    // v1.9.0 — version history is on disk now (HistoryStore), so eviction
+    // doesn't have to clean it up. The v1.8.0 in-memory history-prune
+    // code (which used to fire here) is gone; the unbounded-history
+    // problem it solved has been replaced with the on-disk model.
 
     collLock.unlock();
 
@@ -2913,12 +2844,10 @@ bool MemoryStore::evictDocument(const std::string& collection, const std::string
         stats_.evictionCount++;
     }
 
-    // Update memory tracking atomically (subtract evicted document + history)
-    estimatedMemoryBytes_.fetch_sub(stub.estimatedSize + historyBytesFreed,
-                                    std::memory_order_relaxed);
+    // Update memory tracking atomically (subtract evicted document size).
+    estimatedMemoryBytes_.fetch_sub(stub.estimatedSize, std::memory_order_relaxed);
 
-    SPDLOG_DEBUG("Evicted document {}/{} (doc: {} bytes, history: {} bytes)",
-                 collection, id, stub.estimatedSize, historyBytesFreed);
+    SPDLOG_DEBUG("Evicted document {}/{} (size: {} bytes)", collection, id, stub.estimatedSize);
 
     return true;
 }

service/src/memory_store.hpp

@@ -18,6 +18,8 @@
 
 namespace smartbotic::database {
 
+class HistoryStore;  // v1.9.0 — fwd decl; disk-resident version history
+
 /**
  * Memory pressure level, computed from estimatedMemoryBytes_ / maxMemoryBytes.
  */
@@ -136,6 +138,16 @@ public:
      */
     void setConfigManager(CollectionConfigManager* mgr) { configManager_ = mgr; }
 
+    /**
+     * v1.9.0 — attach a disk-resident history backend. All version-history
+     * writes route to this store; reads (`getVersionHistory`,
+     * `getDocumentAtVersion`, etc.) read from it instead of the old
+     * in-memory deque. Safe to leave unset for tests, in which case the
+     * version-history feature degrades to "no history kept" rather than
+     * "in-memory history kept" (which was the pre-v1.9 behavior).
+     */
+    void setHistoryStore(HistoryStore* hs) { historyStore_ = hs; }
+
     // ===== Collection Management =====
 
     /**
@@ -604,7 +616,12 @@ private:
         CollectionOptions options;
         std::unordered_map<std::string, Document> documents;
         std::map<uint64_t, std::unordered_set<std::string>> expirationIndex;  // expiresAt -> ids
-        std::unordered_map<std::string, std::deque<DocumentVersion>> versionHistory;  // docId -> versions (oldest first)
+        // v1.9.0 — version history moved to disk (HistoryStore). The
+        // in-memory `versionHistory` deque was the dominant cause of the
+        // ~3 GB untracked RSS gap on Zoe (BUG-memory-leak-zoe.md). The
+        // data was already on disk in the WAL; the in-memory copy was
+        // pure redundancy. See `historyStore_` in MemoryStore and the
+        // per-collection .hlog files under <data_dir>/history/.
         std::unordered_map<std::string, std::vector<float>> vectors;       // docId -> embedding vector
         std::unordered_map<std::string, float> vectorNorms;                // cached L2 norms
         uint64_t createdAt = 0;
@@ -767,6 +784,10 @@ private:
     // Callback for loading evicted documents from WAL
     DocumentLoadCallback documentLoadCallback_;
 
+    // v1.9.0 — disk-resident version history backend. Nullable; when null,
+    // history operations no-op (no history kept).
+    HistoryStore* historyStore_ = nullptr;
+
     // Per-document last-write WAL sequence map. Populated by the persist
     // callback after each WAL append and by recovery's WAL replay. Used
     // by `evictDocument` to populate `EvictedDocumentStub.walSequence`

service/src/persistence/history_store.cpp

@@ -0,0 +1,300 @@
+#include "history_store.hpp"
+
+#include "wal.hpp"  // crc32::calculate
+
+#include <algorithm>
+#include <cstring>
+#include <functional>
+#include <stdexcept>
+#include <system_error>
+
+#include <spdlog/spdlog.h>
+
+namespace smartbotic::database {
+
+namespace {
+
+// Build a path that uses URL-encode-style escapes for collection names with
+// characters that aren't filesystem-friendly. Collection names in production
+// only contain [A-Za-z0-9_], but '_views' / '_collection_meta' are real
+// collections so we want to support leading underscores.
+std::string escapeForFilename(const std::string& name) {
+    std::string out;
+    out.reserve(name.size());
+    for (char c : name) {
+        if (std::isalnum(static_cast<unsigned char>(c)) || c == '_' || c == '-') {
+            out.push_back(c);
+        } else {
+            char buf[4];
+            std::snprintf(buf, sizeof(buf), "%%%02x", static_cast<unsigned char>(c));
+            out.append(buf);
+        }
+    }
+    return out;
+}
+
+} // anonymous
+
+HistoryStore::HistoryStore(Config config)
+    : historyDir_(config.dataDir / "history") {
+    std::error_code ec;
+    std::filesystem::create_directories(historyDir_, ec);
+    if (ec) {
+        spdlog::warn("HistoryStore: failed to create {}: {}",
+                     historyDir_.string(), ec.message());
+    }
+}
+
+std::filesystem::path HistoryStore::filePath(const std::string& collection) const {
+    return historyDir_ / (escapeForFilename(collection) + ".hlog");
+}
+
+std::vector<uint8_t> HistoryStore::serializeRecord(const std::string& docId,
+                                                     const DocumentVersion& version) {
+    // Same length-prefixed framing pattern as WAL entries (see wal.cpp): the
+    // first four bytes are the total size minus the prefix, and the trailing
+    // four bytes are a CRC32 of the body. Body layout:
+    //   doc_id_len (u16) + doc_id
+    //   ver_json_len (u32) + ver_json
+    std::vector<uint8_t> result;
+    result.resize(4);  // length prefix, written last
+
+    uint16_t docIdLen = static_cast<uint16_t>(docId.size());
+    result.push_back(static_cast<uint8_t>(docIdLen & 0xFF));
+    result.push_back(static_cast<uint8_t>((docIdLen >> 8) & 0xFF));
+    result.insert(result.end(), docId.begin(), docId.end());
+
+    std::string verJson = version.toJson().dump();
+    uint32_t verLen = static_cast<uint32_t>(verJson.size());
+    for (int i = 0; i < 4; ++i) {
+        result.push_back(static_cast<uint8_t>((verLen >> (i * 8)) & 0xFF));
+    }
+    result.insert(result.end(), verJson.begin(), verJson.end());
+
+    uint32_t crc = crc32::calculate(result.data() + 4, result.size() - 4);
+    for (int i = 0; i < 4; ++i) {
+        result.push_back(static_cast<uint8_t>((crc >> (i * 8)) & 0xFF));
+    }
+
+    uint32_t totalLen = static_cast<uint32_t>(result.size() - 4);
+    result[0] = static_cast<uint8_t>(totalLen & 0xFF);
+    result[1] = static_cast<uint8_t>((totalLen >> 8) & 0xFF);
+    result[2] = static_cast<uint8_t>((totalLen >> 16) & 0xFF);
+    result[3] = static_cast<uint8_t>((totalLen >> 24) & 0xFF);
+
+    return result;
+}
+
+std::optional<std::pair<std::string, DocumentVersion>>
+HistoryStore::deserializeRecord(const std::vector<uint8_t>& data) {
+    if (data.size() < 8) return std::nullopt;  // need at least length + crc
+
+    uint32_t length = 0;
+    for (int i = 0; i < 4; ++i) {
+        length |= static_cast<uint32_t>(data[i]) << (i * 8);
+    }
+    if (data.size() < length + 4) return std::nullopt;
+
+    uint32_t storedCrc = 0;
+    for (int i = 0; i < 4; ++i) {
+        storedCrc |= static_cast<uint32_t>(data[data.size() - 4 + i]) << (i * 8);
+    }
+    uint32_t calcCrc = crc32::calculate(data.data() + 4, length - 4);
+    if (storedCrc != calcCrc) return std::nullopt;
+
+    size_t offset = 4;
+    if (offset + 2 > data.size() - 4) return std::nullopt;
+    uint16_t docIdLen = static_cast<uint16_t>(data[offset]) |
+                       (static_cast<uint16_t>(data[offset + 1]) << 8);
+    offset += 2;
+    if (offset + docIdLen > data.size() - 4) return std::nullopt;
+    std::string docId(reinterpret_cast<const char*>(&data[offset]), docIdLen);
+    offset += docIdLen;
+
+    if (offset + 4 > data.size() - 4) return std::nullopt;
+    uint32_t verLen = 0;
+    for (int i = 0; i < 4; ++i) {
+        verLen |= static_cast<uint32_t>(data[offset + i]) << (i * 8);
+    }
+    offset += 4;
+    if (offset + verLen > data.size() - 4) return std::nullopt;
+    std::string verJson(reinterpret_cast<const char*>(&data[offset]), verLen);
+
+    try {
+        DocumentVersion v = DocumentVersion::fromJson(nlohmann::json::parse(verJson));
+        return std::make_pair(std::move(docId), std::move(v));
+    } catch (const nlohmann::json::exception&) {
+        return std::nullopt;
+    }
+}
+
+uint64_t HistoryStore::scanFile(std::ifstream& file,
+        const std::function<void(const std::string&, DocumentVersion&&)>& cb) {
+    uint64_t count = 0;
+    while (file) {
+        uint32_t length = 0;
+        file.read(reinterpret_cast<char*>(&length), 4);
+        if (!file || length == 0) break;
+
+        std::vector<uint8_t> entry(length + 4);
+        std::memcpy(entry.data(), &length, 4);
+        file.read(reinterpret_cast<char*>(entry.data() + 4), length);
+        if (!file) break;
+
+        auto rec = deserializeRecord(entry);
+        if (rec) {
+            cb(rec->first, std::move(rec->second));
+            count++;
+        }
+    }
+    return count;
+}
+
+void HistoryStore::append(const std::string& collection,
+                          const std::string& docId,
+                          const DocumentVersion& version) {
+    auto bytes = serializeRecord(docId, version);
+
+    std::lock_guard<std::mutex> lock(appendMutex_);
+    std::ofstream f(filePath(collection), std::ios::binary | std::ios::app);
+    if (!f) {
+        spdlog::error("HistoryStore: failed to open {} for append",
+                      filePath(collection).string());
+        return;
+    }
+    f.write(reinterpret_cast<const char*>(bytes.data()),
+            static_cast<std::streamsize>(bytes.size()));
+    // No flush per write — the OS coalesces and the WAL is the source of
+    // truth for crash recovery anyway. History is best-effort durable.
+}
+
+std::vector<DocumentVersion> HistoryStore::getVersions(
+    const std::string& collection,
+    const std::string& docId,
+    uint32_t limit,
+    uint32_t offset) const {
+
+    std::ifstream f(filePath(collection), std::ios::binary);
+    if (!f) return {};
+
+    // Stream-scan, collect matches in file order (oldest → newest). We then
+    // reverse so the caller gets newest-first, mirroring the old in-memory
+    // semantics that returned the deque back-to-front.
+    std::vector<DocumentVersion> matches;
+    scanFile(f, [&](const std::string& recDocId, DocumentVersion&& v) {
+        if (recDocId == docId) {
+            matches.push_back(std::move(v));
+        }
+    });
+
+    std::reverse(matches.begin(), matches.end());
+
+    if (offset >= matches.size()) return {};
+    auto start = matches.begin() + offset;
+    auto end = (limit == 0) ? matches.end()
+                            : std::min(matches.end(), start + limit);
+    return std::vector<DocumentVersion>(start, end);
+}
+
+std::optional<DocumentVersion> HistoryStore::getAtVersion(
+    const std::string& collection,
+    const std::string& docId,
+    uint64_t version) const {
+
+    std::ifstream f(filePath(collection), std::ios::binary);
+    if (!f) return std::nullopt;
+
+    std::optional<DocumentVersion> result;
+    scanFile(f, [&](const std::string& recDocId, DocumentVersion&& v) {
+        if (recDocId == docId && v.version == version) {
+            result = std::move(v);
+        }
+    });
+    return result;
+}
+
+uint64_t HistoryStore::countVersions(
+    const std::string& collection,
+    const std::string& docId) const {
+
+    std::ifstream f(filePath(collection), std::ios::binary);
+    if (!f) return 0;
+
+    uint64_t count = 0;
+    scanFile(f, [&](const std::string& recDocId, DocumentVersion&& /*v*/) {
+        if (recDocId == docId) count++;
+    });
+    return count;
+}
+
+void HistoryStore::dropCollection(const std::string& collection) {
+    std::lock_guard<std::mutex> lock(appendMutex_);
+    std::error_code ec;
+    std::filesystem::remove(filePath(collection), ec);
+}
+
+void HistoryStore::prune(const std::string& collection, uint32_t maxVersions) {
+    if (maxVersions == 0) return;  // 0 means unlimited
+
+    auto path = filePath(collection);
+    if (!std::filesystem::exists(path)) return;
+
+    // Read the whole file into a per-doc bucket (newest-last order, as the
+    // file is append-only). Keep the last `maxVersions` per doc, then
+    // rewrite. The append lock is held for the duration so no concurrent
+    // writer interleaves.
+    std::lock_guard<std::mutex> lock(appendMutex_);
+
+    std::ifstream in(path, std::ios::binary);
+    if (!in) return;
+
+    std::unordered_map<std::string, std::deque<DocumentVersion>> buckets;
+    scanFile(in, [&](const std::string& recDocId, DocumentVersion&& v) {
+        auto& q = buckets[recDocId];
+        q.push_back(std::move(v));
+        if (q.size() > maxVersions) q.pop_front();
+    });
+    in.close();
+
+    // Write atomically: tmp + rename.
+    auto tmp = path;
+    tmp += ".tmp";
+    {
+        std::ofstream out(tmp, std::ios::binary | std::ios::trunc);
+        if (!out) {
+            spdlog::warn("HistoryStore::prune: failed to open {} for rewrite",
+                         tmp.string());
+            return;
+        }
+        for (auto& [docId, versions] : buckets) {
+            for (auto& v : versions) {
+                auto bytes = serializeRecord(docId, v);
+                out.write(reinterpret_cast<const char*>(bytes.data()),
+                          static_cast<std::streamsize>(bytes.size()));
+            }
+        }
+    }
+
+    std::error_code ec;
+    std::filesystem::rename(tmp, path, ec);
+    if (ec) {
+        spdlog::warn("HistoryStore::prune: rename {}->{} failed: {}",
+                     tmp.string(), path.string(), ec.message());
+        std::filesystem::remove(tmp, ec);
+    }
+}
+
+uint64_t HistoryStore::totalSizeBytes() const {
+    uint64_t total = 0;
+    std::error_code ec;
+    if (!std::filesystem::exists(historyDir_, ec)) return 0;
+    for (const auto& entry : std::filesystem::directory_iterator(historyDir_, ec)) {
+        if (entry.is_regular_file()) {
+            std::error_code sec;
+            total += std::filesystem::file_size(entry.path(), sec);
+        }
+    }
+    return total;
+}
+
+} // namespace smartbotic::database

service/src/persistence/history_store.hpp

@@ -0,0 +1,130 @@
+#pragma once
+
+#include "../document.hpp"
+
+#include <cstdint>
+#include <deque>
+#include <filesystem>
+#include <fstream>
+#include <mutex>
+#include <optional>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+namespace smartbotic::database {
+
+/**
+ * v1.9.0 — disk-resident version history.
+ *
+ * Pre-v1.9, `CollectionData::versionHistory` was an in-heap map of
+ * std::deque<DocumentVersion> per doc id, each entry holding a full
+ * nlohmann::json tree of the past doc state. On hot-update collections
+ * with `max_versions=0` this accumulated GBs of in-memory state that
+ * was also already on disk in the WAL — pure redundancy that no
+ * allocator can shrink and no tracker correctly counts. The Zoe leak
+ * (BUG-memory-leak-zoe.md) tracked down to this redundancy.
+ *
+ * HistoryStore replaces the in-memory map with one append-only file
+ * per collection at `<data_dir>/history/<collection>.hlog`. The format
+ * is intentionally similar to WAL: length-framed, CRC-checked, JSON
+ * payload, so it's easy to read with off-the-shelf tools. Per-call
+ * latency for `getVersions` shifts from microseconds (map lookup) to
+ * single-digit milliseconds (file scan) — acceptable for an audit /
+ * debug feature, and ShadowMan (the largest consumer) doesn't call
+ * it on any hot path.
+ *
+ * The format is *append-only*. `prune(max_versions)` rewrites the file
+ * by streaming records out and back in, keeping the newest N per
+ * doc_id; called periodically by a background thread on collections
+ * that have a `max_versions` cap.
+ *
+ * Concurrency: a single `appendMutex_` serializes appends. Reads open a
+ * fresh ifstream, so concurrent readers don't block each other. The
+ * worst case under high write rate is a single producer writing
+ * sequentially, which is what the file system handles best anyway.
+ */
+class HistoryStore {
+public:
+    struct Config {
+        std::filesystem::path dataDir;  // We'll create <dataDir>/history/
+    };
+
+    explicit HistoryStore(Config config);
+    ~HistoryStore() = default;
+
+    HistoryStore(const HistoryStore&) = delete;
+    HistoryStore& operator=(const HistoryStore&) = delete;
+
+    /**
+     * Append a version record for one document.
+     */
+    void append(const std::string& collection,
+                const std::string& docId,
+                const DocumentVersion& version);
+
+    /**
+     * Read versions for a doc, newest-first. limit=0 → all.
+     */
+    [[nodiscard]] std::vector<DocumentVersion> getVersions(
+        const std::string& collection,
+        const std::string& docId,
+        uint32_t limit = 0,
+        uint32_t offset = 0) const;
+
+    /**
+     * Read a single version by version number. Returns nullopt if absent.
+     */
+    [[nodiscard]] std::optional<DocumentVersion> getAtVersion(
+        const std::string& collection,
+        const std::string& docId,
+        uint64_t version) const;
+
+    /**
+     * Count total versions for a doc. Used by getVersionHistory's
+     * `totalCount` field.
+     */
+    [[nodiscard]] uint64_t countVersions(
+        const std::string& collection,
+        const std::string& docId) const;
+
+    /**
+     * Drop the entire history file for a collection (called when the
+     * collection is dropped).
+     */
+    void dropCollection(const std::string& collection);
+
+    /**
+     * Rewrite the history file for `collection` keeping at most
+     * `maxVersions` entries per doc_id (newest). Called periodically
+     * by a background thread; safe to call concurrently with writes
+     * (under the appendMutex_).
+     */
+    void prune(const std::string& collection, uint32_t maxVersions);
+
+    /**
+     * Total disk bytes used by all history files.
+     */
+    [[nodiscard]] uint64_t totalSizeBytes() const;
+
+private:
+    std::filesystem::path historyDir_;
+    mutable std::mutex appendMutex_;
+
+    [[nodiscard]] std::filesystem::path filePath(const std::string& collection) const;
+
+    // Serialize one record into a self-framed byte buffer with CRC trailer.
+    [[nodiscard]] static std::vector<uint8_t> serializeRecord(
+        const std::string& docId, const DocumentVersion& version);
+
+    // Decode one record. Returns nullopt on length/CRC mismatch.
+    [[nodiscard]] static std::optional<std::pair<std::string, DocumentVersion>>
+        deserializeRecord(const std::vector<uint8_t>& data);
+
+    // Scan an open file from current position to EOF, invoking `cb` on each
+    // valid record. Returns the number of records processed.
+    static uint64_t scanFile(std::ifstream& file,
+                             const std::function<void(const std::string&, DocumentVersion&&)>& cb);
+};
+
+} // namespace smartbotic::database

service/src/persistence/snapshot.cpp

@@ -668,37 +668,17 @@ std::vector<uint8_t> SnapshotManager::serializeStore(const MemoryStore& store,
 
         docCount += collDocCount;
 
-        // Write version history (v2)
-        auto allHistory = store.getAllVersionHistory(name);
-        uint64_t historyEntryCount = allHistory.size();
+        // v1.9.0 — version history block is now an empty section. History
+        // moved to disk-resident HistoryStore (separate `.hlog` files per
+        // collection under <data_dir>/history/). We still emit the
+        // 8-byte zero count so v1.8.x readers can parse the snapshot
+        // (they'll find no entries and move on). v2.x can drop this
+        // block entirely with a snapshot format bump.
+        uint64_t historyEntryCount = 0;
         for (int i = 0; i < 8; ++i) {
             result.push_back(static_cast<uint8_t>((historyEntryCount >> (i * 8)) & 0xFF));
         }
 
-        for (const auto& [docId, versions] : allHistory) {
-            // Write doc ID
-            uint16_t docIdLen = static_cast<uint16_t>(docId.size());
-            result.push_back(static_cast<uint8_t>(docIdLen & 0xFF));
-            result.push_back(static_cast<uint8_t>((docIdLen >> 8) & 0xFF));
-            result.insert(result.end(), docId.begin(), docId.end());
-
-            // Write version count
-            uint32_t versionCount = static_cast<uint32_t>(versions.size());
-            for (int i = 0; i < 4; ++i) {
-                result.push_back(static_cast<uint8_t>((versionCount >> (i * 8)) & 0xFF));
-            }
-
-            // Write each version
-            for (const auto& ver : versions) {
-                std::string verJson = ver.toJson().dump();
-                uint32_t verLen = static_cast<uint32_t>(verJson.size());
-                for (int i = 0; i < 4; ++i) {
-                    result.push_back(static_cast<uint8_t>((verLen >> (i * 8)) & 0xFF));
-                }
-                result.insert(result.end(), verJson.begin(), verJson.end());
-            }
-        }
-
         // Write vector data (v3)
         const auto* vectors = store.getCollectionVectors(name);
         uint64_t vecCount = (vectors != nullptr) ? static_cast<uint64_t>(vectors->size()) : 0;

tests/CMakeLists.txt

@@ -6,6 +6,8 @@ set(TEST_VECTOR_SOURCES
     test_vector_storage.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/memory_store.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/config/collection_config_manager.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/persistence/history_store.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/persistence/wal.cpp
 )
 
 add_executable(test_vector_storage ${TEST_VECTOR_SOURCES})
@@ -57,6 +59,8 @@ set(TEST_TIMESTAMP_PRECISION_SOURCES
     test_timestamp_precision.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/memory_store.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/config/collection_config_manager.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/persistence/history_store.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/persistence/wal.cpp
 )
 
 add_executable(test_timestamp_precision ${TEST_TIMESTAMP_PRECISION_SOURCES})
@@ -90,6 +94,7 @@ set(TEST_SNAPSHOT_DURABILITY_SOURCES
     ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/config/collection_config_manager.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/persistence/snapshot.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/persistence/wal.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/persistence/history_store.cpp
 )
 
 add_executable(test_snapshot_durability ${TEST_SNAPSHOT_DURABILITY_SOURCES})
@@ -160,6 +165,8 @@ set(TEST_EVICTION_SOURCES
     test_eviction.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/memory_store.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/config/collection_config_manager.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/persistence/history_store.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/../service/src/persistence/wal.cpp
 )
 
 add_executable(test_eviction ${TEST_EVICTION_SOURCES})