Переглянути джерело

feat(diag): SIGUSR1 malloc_info, SIGUSR2 malloc_trim (v1.8.3)

Adds two operator-side diagnostic signals to hunt the untracked
~160 MB/min RSS growth observed on Zoe (BUG-memory-leak-zoe.md
"Update 2026-05-13 10:05"). Local heaptrack runs against a synthetic
workload showed only the expected in-memory dataset footprint — the
production-only leak surface (multiple Subscribe streams, encryption-
enabled, real LLM streaming patterns, file uploads, hours of uptime)
isn't reproducible in a 2-minute test. So instead we instrument the
running process on Zoe and probe it directly.

  kill -USR1 $(systemctl show smartbotic-database -p MainPID --value)
    → writes `malloc_info(0)` XML to stderr (caught by the journal).
      Captures per-arena block counts, fastbin cache sizes, mmap'd
      regions. Diff two samples taken N minutes apart to find which
      arena/size-class is growing. Standard glibc allocator
      observability — same XML format `MALLOC_INFO` env writes to.

  kill -USR2 $(systemctl show smartbotic-database -p MainPID --value)
    → calls `malloc_trim(0)` to push freelisted pages back to the OS,
      then logs VmRSS before/after. If RSS drops, the gap is glibc
      retention (allocator caching pages that the application has
      already freed — the fragmentation hypothesis). If RSS doesn't
      drop, the application is still holding the memory (true leak).

Both handlers are async-signal-unsafe by the letter of the spec —
malloc_info / malloc_trim touch allocator internals. In practice both
are known to work from signal handlers on glibc and operators run
them at low frequency. The local smoke test confirmed they write the
expected output and don't disturb in-flight gRPC traffic.

The MALLOC_ARENA_MAX=2 from v1.8.1 stays in place; this just adds the
introspection. v1.8.3 ships intentionally as a small diagnostic-only
release so we can probe Zoe today and decide the actual fix (likely
gRPC arena reset on stream-end, or streaming-write the file
upload/download paths to drop the per-call 500 MB peak) for v1.8.4+.
fszontagh 2 місяців тому
батько
коміт
9b8015bc3b
2 змінених файлів з 57 додано та 1 видалено
  1. 1 1
      VERSION
  2. 56 0
      service/src/main.cpp

+ 1 - 1
VERSION

@@ -1 +1 @@
-1.8.2
+1.8.3

+ 56 - 0
service/src/main.cpp

@@ -6,8 +6,10 @@
 #include <spdlog/sinks/stdout_color_sinks.h>
 
 #include <csignal>
+#include <cstdio>
 #include <cstdlib>
 #include <filesystem>
+#include <fstream>
 #include <iostream>
 #include <optional>
 #include <string>
@@ -32,6 +34,55 @@ void signalHandler(int signal) {
     }
 }
 
+// v1.8.3: diagnostic handlers for hunting the untracked RSS leak observed
+// on Zoe (BUG-memory-leak-zoe.md, "Update 2026-05-13 10:05"). Both are
+// async-signal-unsafe under the spec (they call into glibc allocator
+// state), but `malloc_info` / `malloc_trim` are well-known to work from
+// signal handlers in practice on glibc — and we'd rather have the
+// diagnostic than be pedantic about it.
+//
+//   kill -USR1 $PID   → write malloc_info(0) XML to stderr (and journal)
+//                       — captures arena stats: blocks in use, fastbin
+//                         caches, mmap'd regions. Compare two samples
+//                         over time to find which arena is growing.
+//   kill -USR2 $PID   → call malloc_trim(0), then log before/after RSS
+//                       — if RSS drops sharply, the gap is allocator
+//                         retention (madvise(DONTNEED) not happening
+//                         autonomously). If it doesn't drop, the leak
+//                         is genuinely held by the application.
+void mallocInfoHandler(int) {
+#if defined(__GLIBC__) && !defined(__APPLE__)
+    std::fprintf(stderr, "=== malloc_info dump (SIGUSR1) ===\n");
+    ::malloc_info(0, stderr);
+    std::fprintf(stderr, "=== end malloc_info ===\n");
+    std::fflush(stderr);
+#endif
+}
+
+void mallocTrimHandler(int) {
+#if defined(__GLIBC__) && !defined(__APPLE__)
+    auto readRssKb = []() -> long {
+        std::ifstream s("/proc/self/status");
+        std::string line;
+        while (std::getline(s, line)) {
+            if (line.rfind("VmRSS:", 0) == 0) {
+                long kb = 0;
+                std::sscanf(line.c_str(), "VmRSS: %ld kB", &kb);
+                return kb;
+            }
+        }
+        return 0;
+    };
+    long before = readRssKb();
+    int released = ::malloc_trim(0);
+    long after = readRssKb();
+    std::fprintf(stderr,
+        "=== malloc_trim(0) (SIGUSR2): returned=%d, VmRSS %ld kB -> %ld kB (delta %+ld kB) ===\n",
+        released, before, after, after - before);
+    std::fflush(stderr);
+#endif
+}
+
 void printUsage(const char* programName) {
     std::cout << "Usage: " << programName << " [OPTIONS]\n"
               << "\n"
@@ -249,6 +300,11 @@ int main(int argc, char* argv[]) {
         // Set up signal handlers
         std::signal(SIGINT, signalHandler);
         std::signal(SIGTERM, signalHandler);
+        // v1.8.3: SIGUSR1 / SIGUSR2 for live heap-state probing on Zoe. Both
+        // log to stderr (caught by the systemd journal), do not interrupt
+        // service operation. Documented in CLAUDE.md.
+        std::signal(SIGUSR1, mallocInfoHandler);
+        std::signal(SIGUSR2, mallocTrimHandler);
 
 #ifdef HAVE_SYSTEMD
         // v1.7.5: extend the systemd start timeout while recovery is