fix(memory): cap glibc malloc arenas to 2 (v1.8.1)

The v1.8.0 version-history-on-eviction fix landed but did not actually
trigger on Zoe — the workload stays under the 85% eviction threshold,
so the leak surface that exploded into the OOM-every-3h pattern wasn't
unreleased history. Zoe-side measurement after the deploy confirmed
the dominant cause is glibc per-thread arena fragmentation:

    Before MALLOC_ARENA_MAX:    After MALLOC_ARENA_MAX=2:
    VmRSS   7.97 GB              VmRSS   5.57 GB
    VmData 11.55 GB              VmData  5.70 GB
    VmSwap  3.27 GB              VmSwap  0 KB
    VmHWM  10.77 GB              VmHWM   8.29 GB
    Tracked 4995 MB              Tracked ~4995 MB (same workload)

The tracker was correct all along; the remaining 6.5 GB was committed
but unreturnable heap. With Type=notify the service runs ~22 threads
(eviction, WAL fsync, snapshot, gRPC pool, replication apply +
connection loop, application code). glibc creates `8 × num_cpus`
arenas by default; small frees go on per-thread free-lists and never
make it back to `madvise(MADV_DONTNEED)`. Two arenas is plenty for
the actual writer + reader split this DB has.

Two layers of fix, so the env-var control survives both deployment
paths:

  1. `Environment=MALLOC_ARENA_MAX=2` in the deb's systemd unit.
     Primary control. Operators can still override per-deployment by
     dropping a 10-override.conf into smartbotic-database.service.d/.

  2. `mallopt(M_ARENA_MAX, 2)` at the very top of `main()`, guarded so
     it skips when MALLOC_ARENA_MAX is already set by the environment.
     Covers docker / ad-hoc / non-systemd invocations.

The boot-time parallel snapshot deserialize from v1.8.0 also landed
cleanly — Zoe's 5 GB snapshot goes from systemd start to READY in
<1 second (sequential pre-v1.8 was ~30 s).

The version-history fix from v1.8.0 stays in: it's still correct, it
just only fires for workloads that hit the eviction threshold. Zoe
historically did during the original incident; with current load it
doesn't.

VERSION

@@ -1 +1 @@
-1.8.0
+1.8.1

packaging/deb/systemd/smartbotic-database.service

@@ -15,6 +15,21 @@ After=network.target
 # the unit active and unblocking dependents.
 Type=notify
 NotifyAccess=main
+
+# v1.8.1: Cap glibc per-thread arenas to two. With Type=notify + threads
+# in eviction / WAL sync / snapshot / gRPC / replication, glibc allocates
+# `8 × num_cpus` arenas by default and small frees go on per-thread
+# free-lists that never `madvise(MADV_DONTNEED)`. On Zoe (22 threads) the
+# committed heap drifted 6.5 GB above the 5 GB the DB itself was tracking
+# — a fragmentation gap large enough to OOM-kill the process every ~3h
+# even though `estimatedMemoryBytes_` was correct. Two arenas is enough
+# for the writer + reader thread split; the small extra contention is
+# negligible vs. the RSS savings (measured 7.97 → 5.57 GB on Zoe).
+# Operators can override per-deployment by setting MALLOC_ARENA_MAX
+# elsewhere; the binary also calls `mallopt(M_ARENA_MAX, 2)` at startup
+# as a fallback for non-systemd deploys.
+Environment="MALLOC_ARENA_MAX=2"
+
 # Recovery on a large dataset (700k+ docs, multi-MB WAL) routinely takes
 # 60–90s. Default systemd start timeout (90s) is too tight; bump to 5 min.
 # The service emits `EXTEND_TIMEOUT_USEC=` during long recovery anyway, but

service/src/main.cpp

@@ -13,6 +13,10 @@
 #include <string>
 #include <vector>
 
+#if defined(__GLIBC__) && !defined(__APPLE__)
+#include <malloc.h>
+#endif
+
 #ifdef HAVE_SYSTEMD
 #include <systemd/sd-daemon.h>
 #endif
@@ -142,7 +146,30 @@ std::filesystem::path findConfigFile(int argc, char* argv[]) {
 
 } // anonymous namespace
 
+// v1.8.1: Cap glibc per-thread malloc arenas. With ~20+ threads
+// (eviction / WAL sync / snapshot / gRPC pool / replication apply +
+// connection loop), the default `8 × num_cpus` arenas fragment heap
+// pages so they're never returned to the OS, drifting RSS far above
+// the DB's tracked working set (Zoe: 6.5 GB gap, OOM every ~3h until
+// MALLOC_ARENA_MAX=2 was set). The systemd unit sets the env var as
+// the primary control; this programmatic call covers non-systemd
+// deployments (docker, ad-hoc launches). Skip if the env var is
+// already set, so operators can override.
+static void capGlibcArenas() {
+#if defined(__GLIBC__) && !defined(__APPLE__)
+    if (const char* v = std::getenv("MALLOC_ARENA_MAX"); v && *v) {
+        return;  // Operator override in effect — respect it.
+    }
+    if (::mallopt(M_ARENA_MAX, 2) == 0) {
+        // mallopt returns nonzero on success; silent — the worst case is
+        // we run with default arenas and observe the historical drift.
+    }
+#endif
+}
+
 int main(int argc, char* argv[]) {
+    capGlibcArenas();
+
     // Parse command line arguments
     for (int i = 1; i < argc; ++i) {
         std::string arg = argv[i];