瀏覽代碼

fix(memory): periodic auto malloc_trim + post-recovery trim (v1.9.3)

Per BUG-memory-leak-zoe.md update 2026-05-13 15:05. v1.9.1 added trim
at snapshot boundaries — necessary but not sufficient. Snapshots run
hourly by default; between fresh boot and the first snapshot the
process can accumulate 5-8 GB of glibc arena pages from:

  - Initial snapshot deserialize (handled by v1.9.1)
  - WAL replay after the snapshot loads (NOT handled before this)
  - Migration runner pass (NOT handled before this)
  - First N minutes of normal write traffic (NOT handled before this)

Observed on Zoe at 23 min uptime: VmRSS 9.3 GB while tracker held
flat at 1.4 GB. SIGUSR2 (manual trim from v1.8.3) reclaimed 5.7 GB
in one call — proving the pages are reclaimable, glibc just doesn't
trigger autotrim under the post-boot allocation pattern.

Two trim points added:

  1. DatabaseService::initialize() right after persistence_->recover()
     returns. Catches the WAL replay + initial component setup spike
     in a single trim before gRPC even comes up. The biggest one-shot
     win for operators on tight VMs.

  2. MemoryStore::logMemoryCheck — already runs every 60s. Every 5th
     call (~5 minutes between trims) invokes malloc_trim(0). Cheap
     periodic hygiene that handles ongoing arena retention between
     snapshot boundaries.

Both behind the existing #if defined(__GLIBC__) && !defined(__APPLE__)
guard. The SIGUSR2 manual override from v1.8.3 stays in.

Together with v1.9.1's snapshot-boundary trims, this covers:
  - Boot-time deserialize spike (v1.9.1)
  - Boot-time WAL replay spike (v1.9.3 — point 1)
  - Hourly snapshot peak (v1.9.1)
  - Mid-lifecycle drift (v1.9.3 — point 2)

test_eviction and test_snapshot_durability pass unchanged.

memoryCheckCount_ added to MemoryStore as the cadence counter.
fszontagh 2 月之前
父節點
當前提交
7d8f00f135
共有 4 個文件被更改,包括 41 次插入1 次删除
  1. 1 1
      VERSION
  2. 18 0
      service/src/database_service.cpp
  3. 21 0
      service/src/memory_store.cpp
  4. 1 0
      service/src/memory_store.hpp

+ 1 - 1
VERSION

@@ -1 +1 @@
-1.9.2
+1.9.3

+ 18 - 0
service/src/database_service.cpp

@@ -10,6 +10,10 @@
 #include <systemd/sd-daemon.h>
 #endif
 
+#if defined(__GLIBC__) && !defined(__APPLE__)
+#include <malloc.h>
+#endif
+
 namespace smartbotic::database {
 
 DatabaseService::DatabaseService(Config config)
@@ -66,6 +70,20 @@ bool DatabaseService::initialize() {
         // Recover from persistence
         recovery_outcome_ = persistence_->recover(*store_);
 
+#if defined(__GLIBC__) && !defined(__APPLE__)
+        // v1.9.3 — release freelist pages accumulated during recovery. v1.9.1
+        // added trim at end of loadSnapshot, but the WAL replay phase that
+        // runs afterward (`persistence_->recover`'s `replayWal` step) also
+        // burns through GBs of small Document JSON allocations whose pages
+        // sit on the per-thread freelist with no subsequent allocation to
+        // shake them loose. On Zoe (BUG-memory-leak-zoe.md update 15:05)
+        // this was 5.7 GB at 23 min uptime — fully reclaimable via
+        // malloc_trim, just nothing called it. Paired with the periodic
+        // every-5-minute trim in MemoryStore::logMemoryCheck, so any
+        // post-boot bloat that escapes this trim gets cleaned up shortly.
+        ::malloc_trim(0);
+#endif
+
         if (recovery_outcome_.isFailure()) {
             const std::string modeStr =
                 recoveryModeToString(config_.persistenceConfig.recoveryMode);

+ 21 - 0
service/src/memory_store.cpp

@@ -3,6 +3,10 @@
 #include "config/collection_config_manager.hpp"
 #include "persistence/history_store.hpp"
 
+#if defined(__GLIBC__) && !defined(__APPLE__)
+#include <malloc.h>
+#endif
+
 #include <spdlog/spdlog.h>
 
 #include <algorithm>
@@ -2504,6 +2508,23 @@ void MemoryStore::logMemoryCheck() {
                 memoryPressureToString(pressure()),
                 stats.totalDocuments,
                 stats.evictedCount);
+
+#if defined(__GLIBC__) && !defined(__APPLE__)
+    // v1.9.3 — periodic glibc arena hygiene. v1.9.1's `malloc_trim(0)` at
+    // snapshot boundaries only fires every snapshot_interval_sec (hourly by
+    // default). Between a fresh boot and the first scheduled snapshot the
+    // process accumulates GBs of arena pages from WAL replay, the migration
+    // runner, and the first burst of write traffic — observed on Zoe at
+    // 23 min uptime: VmRSS 9.3 GB while tracker held flat at 1.4 GB, all of
+    // which `malloc_trim(0)` could reclaim (-5.7 GB on manual SIGUSR2).
+    // logMemoryCheck is called once per minute by the eviction loop; trim
+    // every 5th call (~5 minutes between trims) so we don't pay the
+    // walk-the-arena cost too often. SIGUSR2 from v1.8.3 stays in as the
+    // operator override for on-demand trims.
+    if ((memoryCheckCount_++ % 5) == 4) {
+        ::malloc_trim(0);
+    }
+#endif
 }
 
 void MemoryStore::evictionLoop() {

+ 1 - 0
service/src/memory_store.hpp

@@ -822,6 +822,7 @@ private:
 
     // Counter for logMemoryCheck() throttling (every 12th tick ~60s).
     uint64_t evictionCheckCount_{0};
+    uint64_t memoryCheckCount_{0};  // v1.9.3 — ticks per logMemoryCheck call, drives periodic malloc_trim cadence
 };
 
 } // namespace smartbotic::database