Browse Source

fix(memory): auto malloc_trim after snapshot load/write (v1.9.1)

Follow-up to v1.9.0 per BUG-memory-leak-zoe.md update 2026-05-13 12:07.
v1.9.0 closed the tracker gap (5017 MB→1396 MB, growth 0 MB/min) but
left ~2.8 GB of glibc freelist pages parked from the boot-time history
migration. The pages are reusable, but kernel OOM scoring uses RSS,
so the operator-visible memory footprint stayed at v1.8.x levels until
a manual `kill -USR2` (the SIGUSR2 handler shipped in v1.8.3) trimmed
them.

The trigger is the snapshot path's transient-buffer pattern: both
`createSnapshot` (build serialized body → LZ4 compress → discard
uncompressed buffer → write file) and `loadSnapshot` (read file →
decompress to large buffer → parse → discard buffer) burst-allocate
multiple GB that immediately frees back to the freelist. glibc's
threshold-based auto-trim never fires because no subsequent
small-allocation pattern hits the right bin in time.

Two `malloc_trim(0)` calls at the natural quiescent points:
  1. End of `SnapshotManager::loadSnapshot` — after `deserializeStore`
     returns the per-doc Document objects have settled into the
     hot-path collection maps; the multi-GB decompressed buffer is
     gone, the LZ4 staging chunks are gone. Trim drops post-boot RSS
     by the size of those buffers (~2.8 GB on Zoe's 5 GB snapshot).
  2. End of `SnapshotManager::createSnapshot` — after the snapshot has
     been verified and old snapshots cleaned. Keeps the hourly
     snapshot path from drifting RSS up over the day.

Both calls are behind `#if defined(__GLIBC__) && !defined(__APPLE__)`
so non-glibc platforms (musl, etc.) compile cleanly with the trim
becoming a no-op. The SIGUSR2 diagnostic handler from v1.8.3 stays in
as the manual override.

Snapshot durability test passes unchanged.

The bug update also flagged shadowman-cpp's
`/etc/systemd/system/smartbotic-database.service.d/zoe-leak-watchdog.conf`
(the RuntimeMaxSec=5400 watchdog operators added as containment) can
be removed now — v1.9.x doesn't need rolling restarts.
fszontagh 2 months ago
parent
commit
6b2fa54f9e
2 changed files with 40 additions and 1 deletions
  1. 1 1
      VERSION
  2. 39 0
      service/src/persistence/snapshot.cpp

+ 1 - 1
VERSION

@@ -1 +1 @@
-1.9.0
+1.9.1

+ 39 - 0
service/src/persistence/snapshot.cpp

@@ -21,6 +21,29 @@
 #include <lz4.h>
 #endif
 
+#if defined(__GLIBC__) && !defined(__APPLE__)
+#include <malloc.h>
+#endif
+
+namespace {
+// v1.9.1 — `malloc_trim(0)` after a big-buffer phase finishes. The
+// snapshot write and load paths both build multi-GB transient buffers
+// (uncompressed body + LZ4 chunks; per-collection JSON parses). When
+// those buffers free, glibc parks the pages on its arena freelist and
+// only reclaims them lazily during subsequent small allocations. For a
+// quiescent post-boot or post-snapshot process those allocations may
+// not arrive for hours, so the operator-visible VmRSS stays inflated
+// even though `estimatedMemoryBytes_` is correct. Trimming explicitly
+// here turns a 2.8 GB no-op back into a 2.8 GB drop (measured on Zoe
+// 2026-05-13 12:07 via the SIGUSR2 handler from v1.8.3 — same call,
+// just automated). No-op on non-glibc platforms.
+void releaseFreelistPages() {
+#if defined(__GLIBC__) && !defined(__APPLE__)
+    ::malloc_trim(0);
+#endif
+}
+}  // namespace
+
 namespace smartbotic::database {
 
 namespace {
@@ -274,6 +297,14 @@ std::filesystem::path SnapshotManager::createSnapshot(const MemoryStore& store,
         cleanupOldSnapshots();
     }
 
+    // v1.9.1 — release freelist pages after the snapshot write. We just
+    // built and threw away the uncompressed serialized buffer
+    // (potentially several GB) plus LZ4 chunk staging buffers; same
+    // post-burst freelist-retention story as the boot-time load above.
+    // Trimming here keeps the periodic snapshot path from drifting RSS
+    // up over the day. Cheap relative to the snapshot write itself.
+    releaseFreelistPages();
+
     return snapshotPath;
 }
 
@@ -388,6 +419,14 @@ uint64_t SnapshotManager::loadSnapshot(const std::filesystem::path& path, Memory
     store.clear();
     deserializeStore(uncompressedData, store, header.version);
 
+    // v1.9.1 — release allocator freelist pages now that the multi-GB
+    // transient buffers (decompressed body + per-doc JSON parses) are
+    // gone. Without this, glibc keeps those pages parked on the per-
+    // thread freelist; VmRSS stays high until subsequent small allocs
+    // happen to land in the right bin. On Zoe (5 GB snapshot) this
+    // drops post-boot RSS by ~2.8 GB.
+    releaseFreelistPages();
+
     return header.walSequence;
 }