fix(memory): link jemalloc — bound RSS under sustained gRPC load (v1.9.4)

Investigation on Zoe 2026-05-15 00:11 — v1.9.3 installed, uptime 15h:
  Tracker: 1607 MB (flat, normal pressure, 22%)
  VmRSS:   10.6 GB
  SIGUSR2 (manual malloc_trim): 6.4 GB freed instantly
  Climbed back to 10.6 GB in 90 seconds with near-zero document activity

`malloc_info` SIGUSR1 dump showed arena 0 with 107,175 allocations of
exactly 33 bytes — nlohmann::json RB-tree node fragments from per-RPC
JSON parsing. Under sustained gRPC load these small allocations sit at
the high end of every arena page, defeating `malloc_trim(0)`'s
"trailing region must be entirely free" requirement. v1.9.3's periodic
trim IS firing every 5 min but reclaims ~0 bytes; only an explicit trim
during a brief load lull (which is what SIGUSR2 caught) reclaims
anything significant.

This is exactly the workload shape jemalloc is designed for:

  - Per-thread/per-CPU arenas reduce false sharing on the fragmentation
    pattern: each gRPC worker thread gets its own arena instead of
    contending for two shared ones (the MALLOC_ARENA_MAX=2 cap from
    v1.8.1)
  - A dedicated background decay thread (background_thread:true)
    walks dirty pages on a wallclock timer and madvises them back
    regardless of allocation activity — fixes the "no quiet window
    → no trim" problem at root
  - dirty_decay_ms:1000 / muzzy_decay_ms:1000 says hold idle pages
    for at most 1s — tight enough to keep RSS near working set,
    generous enough to absorb workload bursts

Build changes:
  - service/CMakeLists.txt: pkg-config probe for jemalloc, link with
    -Wl,-no-as-needed so the linker keeps the libjemalloc DT_NEEDED
    even though we don't reference any jemalloc symbol directly (we
    just want its malloc/free to override libc). Graceful fallback to
    glibc when libjemalloc is unavailable at build time.
  - packaging/Dockerfile.base: add libjemalloc-dev to the deb13 build
    base image; bump version tag v2 → v3 to trigger image rebuild.
  - packaging/deb/templates/control.server: add libjemalloc2 to deb
    runtime Depends.
  - packaging/deb/systemd/smartbotic-database.service: add MALLOC_CONF
    env var with the tuning above.

The existing v1.8.3 SIGUSR1/USR2 diagnostics stay in — both work
identically under jemalloc (jemalloc implements malloc_info / mallopt
for compatibility). v1.9.1's snapshot-boundary trims and v1.9.3's
periodic/post-recovery trims also stay in as no-op fallbacks for any
deploy that drops libjemalloc.

Expected on Zoe: VmRSS stays within ~500 MB of tracker continuously
even under sustained gRPC load; no operator SIGUSR2 needed.

Answers the operator's question "what causes this if we have memory
limit?": the `max_memory_mb` config caps `estimatedMemoryBytes_`
(in-memory docs + vectors), not VmRSS. Glibc's arena retention is a
layer below the application's view of memory and the cap can't see it.
jemalloc removes the retention; a v1.10+ follow-up could also surface
RSS into the pressure calculation.

VERSION

@@ -1 +1 @@
-1.9.3
+1.9.4

packaging/Dockerfile.base

@@ -18,6 +18,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     libssl-dev \
     liblz4-dev \
     libsystemd-dev \
+    libjemalloc-dev \
     protobuf-compiler \
     libprotobuf-dev \
     libgrpc++-dev \
@@ -35,4 +36,4 @@ ENV CCACHE_COMPRESSLEVEL=6
 
 WORKDIR /build
 
-RUN echo "smartbotic-db-build-base:debian13:v2" > /etc/smartbotic-db-build-base
+RUN echo "smartbotic-db-build-base:debian13:v3" > /etc/smartbotic-db-build-base

packaging/deb/systemd/smartbotic-database.service

@@ -30,6 +30,18 @@ NotifyAccess=main
 # as a fallback for non-systemd deploys.
 Environment="MALLOC_ARENA_MAX=2"
 
+# v1.9.4: tune jemalloc (which the binary links against, overriding glibc
+# malloc/free). `background_thread:true` starts a dedicated thread that
+# decays dirty pages on a wallclock timer regardless of allocation
+# pressure — fixes the "trim never reclaims because the trailing chunk
+# is always in use" pattern we hit with glibc under sustained gRPC
+# load. `dirty_decay_ms:1000` says hold idle dirty pages for at most
+# one second before madvising them back to the OS — generous enough to
+# absorb workload bursts, tight enough to keep RSS close to working set.
+# `muzzy_decay_ms:1000` same idea for pages glibc-equivalent would call
+# "muzzy" (released but recoverable). Ignored on glibc.
+Environment="MALLOC_CONF=background_thread:true,dirty_decay_ms:1000,muzzy_decay_ms:1000"
+
 # Recovery on a large dataset (700k+ docs, multi-MB WAL) routinely takes
 # 60–90s. Default systemd start timeout (90s) is too tight; bump to 5 min.
 # The service emits `EXTEND_TIMEOUT_USEC=` during long recovery anyway, but

packaging/deb/templates/control.server

@@ -14,5 +14,5 @@ License: proprietary
 Provides: shadowman-database, callerai-storage
 Conflicts: shadowman-database, callerai-storage
 Replaces: shadowman-database, callerai-storage
-Depends: libsmartbotic-db-client (= {{VERSION}}), libssl3t64, liblz4-1, libsystemd0
+Depends: libsmartbotic-db-client (= {{VERSION}}), libssl3t64, liblz4-1, libsystemd0, libjemalloc2
 Installed-Size: {{INSTALLED_SIZE}}

service/CMakeLists.txt

@@ -7,6 +7,16 @@ find_package(OpenSSL REQUIRED)
 find_package(PkgConfig)
 if(PKG_CONFIG_FOUND)
     pkg_check_modules(LZ4 QUIET liblz4)
+    # v1.9.4 — jemalloc replaces glibc malloc for the smartbotic-database
+    # binary. The fragmentation pattern from per-RPC nlohmann::json node
+    # churn (107k 33-byte allocations observed on Zoe) defeats glibc's
+    # malloc_trim because the trailing region of each arena always has a
+    # live chunk; jemalloc's background-thread dirty-page decay sidesteps
+    # the contiguity requirement and reclaims unused pages on a wallclock
+    # timer regardless of allocation activity. Linked as a library so
+    # the binary picks up jemalloc's malloc/free symbols at link time;
+    # no LD_PRELOAD needed.
+    pkg_check_modules(JEMALLOC QUIET jemalloc)
 endif()
 
 # Source files
@@ -71,6 +81,21 @@ if(LZ4_FOUND)
     target_include_directories(smartbotic-database PRIVATE ${LZ4_INCLUDE_DIRS})
 endif()
 
+# v1.9.4 — jemalloc allocator (overrides glibc malloc/free)
+if(JEMALLOC_FOUND)
+    target_compile_definitions(smartbotic-database PRIVATE HAVE_JEMALLOC)
+    target_link_libraries(smartbotic-database PRIVATE ${JEMALLOC_LIBRARIES})
+    target_include_directories(smartbotic-database PRIVATE ${JEMALLOC_INCLUDE_DIRS})
+    # Position-sensitive: jemalloc must be linked early so its symbols win
+    # over libc's. -Wl,-no-as-needed forces the linker to record the
+    # dependency even though we don't reference any jemalloc symbol
+    # directly (we just want its malloc to override).
+    target_link_options(smartbotic-database PRIVATE -Wl,-no-as-needed)
+    message(STATUS "Linking with jemalloc (${JEMALLOC_VERSION})")
+else()
+    message(STATUS "jemalloc not found — falling back to glibc malloc (degraded RSS reclaim under load)")
+endif()
+
 # Systemd support
 if(SYSTEMD_FOUND)
     target_compile_definitions(smartbotic-database PRIVATE HAVE_SYSTEMD)