fix(shutdown): extend stop watchdog over the final-snapshot path (v1.8.2)

The v1.8.0 final-on-shutdown snapshot (DatabaseService::stop() →
PersistenceManager::forceSnapshot()) was racing the unit's
TimeoutStopSec=30s. On Zoe (1.85 M docs / ~5 GB tracked) the serialize
takes about 70 s and peaks RSS at ~11 GB during the build of the
uncompressed body buffer. Every `systemctl restart` therefore ended
with systemd SIGKILLing the process mid-write — the new snapshot file
never finished, the .tmp got cleaned up on the next boot, and recovery
fell back to (previous-snapshot + WAL replay) anyway. The whole point
of the shutdown-snapshot — fast TrivialSuccess on the next boot — was
defeated.

Two-part fix:

1. `DatabaseService::stop()` now `sd_notify(EXTEND_TIMEOUT_USEC=600s)`
   plus a STATUS=... line right before `forceSnapshot()`. Mirrors the
   recovery path in main.cpp which already does this for the startup
   side. Systemd resets its stop watchdog while the snapshot writes.

2. The deb's systemd unit bumps `TimeoutStopSec` from 30 s to 300 s as
   the static cap. The dynamic EXTEND_TIMEOUT_USEC is the primary
   control; this is the belt-and-braces ceiling.

Verified the trace pattern on shadowman-zoe (10:04:24 — SIGTERM, 10:04:54
— TimeoutStopSec hit, SIGKILL, 11 G peak / 692 M swap, snapshot half-
written). With this change a `systemctl restart` on Zoe should now
complete cleanly, the new snapshot lands, and the next boot's recovery
reports TrivialSuccess instead of "Replayed N WAL entries from sequence
M". Bonus: the unit's drop-in arena-cap.conf and RuntimeMaxSec watchdog
(deployed for the still-open ~160 MB/min steady-state leak) now produce
clean rolling restarts instead of SIGKILLs.

Does not address the new finding from BUG-memory-leak-zoe.md (untracked
~160 MB/min growth even with MALLOC_ARENA_MAX=2 in place). That's a
separate bug; ship target v1.8.3+ after heaptrack.

VERSION

@@ -1 +1 @@
-1.8.1
+1.8.2

packaging/deb/systemd/smartbotic-database.service

@@ -41,7 +41,16 @@ ExecStart=/usr/bin/smartbotic-database --config /etc/smartbotic-database/config.
 WorkingDirectory=/var/lib/smartbotic-database
 Restart=on-failure
 RestartSec=3
-TimeoutStopSec=30
+
+# v1.8.2: stop watchdog must cover the final-on-shutdown snapshot. On Zoe
+# (1.85M docs / ~5 GB tracked) serialize takes ~70 s. Pre-1.8.2 this was
+# 30 s, so `systemctl restart` always SIGKILLed mid-write — peak RSS hit
+# 11 GB during the doomed serialize, the new snapshot never made it to
+# disk, and the next boot fell back to WAL-only replay. The shutdown path
+# also calls `EXTEND_TIMEOUT_USEC` from `DatabaseService::stop()` so this
+# static cap is a safety net, not the primary control.
+TimeoutStopSec=300
+
 LimitNOFILE=65536
 
 [Install]

service/src/database_service.cpp

@@ -6,6 +6,10 @@
 
 #include <fstream>
 
+#ifdef HAVE_SYSTEMD
+#include <systemd/sd-daemon.h>
+#endif
+
 namespace smartbotic::database {
 
 DatabaseService::DatabaseService(Config config)
@@ -241,6 +245,19 @@ void DatabaseService::stop() {
         // replay" which trips auto-readonly mode. This makes ordinary
         // restarts pass through cleanly without needing
         // `smartbotic-db-cli unlock`.
+        //
+        // v1.8.2 — extend systemd's stop watchdog before doing it. On Zoe
+        // (1.85M docs / ~5 GB tracked) the serialize takes ~70 s and the
+        // default TimeoutStopSec=30s SIGKILLed the process mid-write, peak
+        // RSS hit 11 GB (live state + uncompressed serialize buffer), and
+        // the new snapshot never made it to disk anyway. EXTEND_TIMEOUT_USEC
+        // tells systemd we're working — same protocol the recovery path
+        // uses during startup. Paired with TimeoutStopSec=300 in the unit
+        // file as the static cap.
+#ifdef HAVE_SYSTEMD
+        sd_notify(0, "EXTEND_TIMEOUT_USEC=600000000");
+        sd_notify(0, "STATUS=Writing final snapshot");
+#endif
         try {
             persistence_->forceSnapshot(*store_);
             spdlog::info("Final snapshot taken on shutdown");