feat(packaging): backup /var/lib/smartbotic-database before deb upgrades (v1.9.5)

apt upgrade smartbotic-database now runs a preinst hook that snapshots the
data dir to /var/backups/smartbotic-database/pre-upgrade-<ts>-from-<old-version>/
after the old prerm has stopped the service and before new files are unpacked.

Atomic-backup pattern: stage in .tmp, rename on success. A trap wipes the
partial dir on any failure (cp mid-copy, mv, set -e unwinding) so a broken
backup never enters the retention pool. Trap clears AFTER the rename so a
retention failure cannot wipe the final backup.

1.2x disk-space precheck on the backup volume aborts the upgrade with a
restore-instructions error if short. du/df pipeline outputs are validated
(empty or non-numeric -> exit 1) so a silent measurement failure does not
bypass the precheck.

Retention defaults to 3 newest backups. Tunables:
  SMARTBOTIC_DB_SKIP_BACKUP=1    skip backup entirely (logged)
  SMARTBOTIC_DB_BACKUP_RETENTION number of backups to keep
  SMARTBOTIC_DB_BACKUP_ROOT      override the backup location

CLAUDE.md

@@ -38,6 +38,7 @@ cmake --build build -j$(nproc) && ./build/tests/test_vector_storage
 - **Drop-in config (conf.d)** — `/etc/smartbotic-database/conf.d/*.json` deep-merges over `config.json` using RFC 7396 semantics (arrays replace, objects merge). Lets consumer debs ship tuned settings without touching the upstream conffile. Strict on syntax errors (exit 11), lenient on unknown keys.
 - **Chunked pressure-aware eviction** — Eviction spreads work over multiple ticks with a default `eviction_chunk_size=1000`, `eviction_chunk_pause_ms=50`, `hot_write_floor_ms=30000`. Four pressure levels (normal/soft/hard/emergency) gate behavior: soft = trickle, hard = aggressive + `MEMORY_PRESSURE_HIGH` event, emergency = admission control rejects writes with `RESOURCE_EXHAUSTED`. Per-collection `memory_priority` (low/normal/high) biases selection. Quiesce skips docs with in-flight writes. WAL-fallback no longer holds per-collection mutex during disk I/O.
 - **GetMemoryStats RPC + client retry** — `Client::getMemoryStats()` returns per-collection stats + pressure level. Client library auto-retries write RPCs on `DEADLINE_EXCEEDED`/`RESOURCE_EXHAUSTED`/`UNAVAILABLE` with exponential backoff + jitter (`writeRetries=3`, `writeRetryBackoffMs=100`). Reads are not auto-retried.
+- **Auto-backup before deb upgrade** — `apt upgrade smartbotic-database` runs a `preinst` hook that snapshots `/var/lib/smartbotic-database/` to `/var/backups/smartbotic-database/pre-upgrade-<ts>-from-<version>/` via `cp -a` after the old `prerm` stops the service. Retention `3`, 1.2× free-space precheck on the backup volume (aborts the upgrade if short). Opt out with `SMARTBOTIC_DB_SKIP_BACKUP=1 apt upgrade`. Tunables: `SMARTBOTIC_DB_BACKUP_RETENTION`, `SMARTBOTIC_DB_BACKUP_ROOT`.
 
 ## Packaging
 

VERSION

@@ -1 +1 @@
-1.9.4
+1.9.5

packaging/deb/create-debs.sh

@@ -90,6 +90,7 @@ cp "$CONFIG_DIR/config.json" "$PKG1/etc/smartbotic-database/config.json"
 cp "$SYSTEMD_DIR/smartbotic-database.service" "$PKG1/lib/systemd/system/smartbotic-database.service"
 
 cp "$TEMPLATE_DIR/control.server" "$PKG1/DEBIAN/control"
+cp "$SCRIPTS_DIR/server.preinst"  "$PKG1/DEBIAN/preinst"
 cp "$SCRIPTS_DIR/server.postinst" "$PKG1/DEBIAN/postinst"
 cp "$SCRIPTS_DIR/server.prerm"    "$PKG1/DEBIAN/prerm"
 cp "$SCRIPTS_DIR/server.postrm"   "$PKG1/DEBIAN/postrm"

packaging/deb/scripts/server.preinst

@@ -0,0 +1,81 @@
+#!/bin/bash
+# preinst for smartbotic-database (server)
+# v1.9.5: snapshot data dir before deb upgrade (after old prerm stops the service).
+set -e
+
+DATA_DIR=/var/lib/smartbotic-database
+BACKUP_ROOT="${SMARTBOTIC_DB_BACKUP_ROOT:-/var/backups/smartbotic-database}"
+RETENTION="${SMARTBOTIC_DB_BACKUP_RETENTION:-3}"
+
+case "$1" in
+    upgrade)
+        if [ "${SMARTBOTIC_DB_SKIP_BACKUP:-0}" = "1" ]; then
+            echo "smartbotic-database: SMARTBOTIC_DB_SKIP_BACKUP=1, skipping pre-upgrade backup"
+            exit 0
+        fi
+
+        if [ ! -d "$DATA_DIR" ] || [ -z "$(ls -A "$DATA_DIR" 2>/dev/null)" ]; then
+            echo "smartbotic-database: $DATA_DIR is empty or missing, nothing to back up"
+            exit 0
+        fi
+
+        DATA_KB=$(du -sk "$DATA_DIR" | awk '{print $1}')
+        if [ -z "$DATA_KB" ] || ! [ "$DATA_KB" -gt 0 ] 2>/dev/null; then
+            echo "smartbotic-database: failed to measure $DATA_DIR size" >&2
+            exit 1
+        fi
+        REQ_KB=$((DATA_KB * 12 / 10))
+        mkdir -p "$BACKUP_ROOT"
+        FREE_KB=$(df -P "$BACKUP_ROOT" | awk 'NR==2 {print $4}')
+        if [ -z "$FREE_KB" ] || ! [ "$FREE_KB" -gt 0 ] 2>/dev/null; then
+            echo "smartbotic-database: failed to measure free space at $BACKUP_ROOT" >&2
+            exit 1
+        fi
+
+        if [ "$FREE_KB" -lt "$REQ_KB" ]; then
+            cat >&2 <<EOF
+smartbotic-database: insufficient disk space for pre-upgrade backup
+  data size       : ${DATA_KB} KB
+  required (1.2x) : ${REQ_KB} KB
+  free at         : ${BACKUP_ROOT} (${FREE_KB} KB)
+
+Free up space, point SMARTBOTIC_DB_BACKUP_ROOT at a larger volume, or
+override with SMARTBOTIC_DB_SKIP_BACKUP=1 apt upgrade smartbotic-database.
+
+To restore from backup if upgrade fails:
+  systemctl stop smartbotic-database
+  rm -rf /var/lib/smartbotic-database/*
+  cp -a ${BACKUP_ROOT}/<latest>/. /var/lib/smartbotic-database/
+  chown -R smartbotic-db:smartbotic-db /var/lib/smartbotic-database
+  systemctl start smartbotic-database
+EOF
+            exit 1
+        fi
+
+        TS=$(date -u +%Y%m%dT%H%M%SZ)
+        OLD_VERSION="${2:-unknown}"
+        BACKUP_DIR="$BACKUP_ROOT/pre-upgrade-${TS}-from-${OLD_VERSION}"
+        BACKUP_TMP="${BACKUP_DIR}.tmp"
+
+        echo "smartbotic-database: backing up $DATA_DIR to $BACKUP_DIR (${DATA_KB} KB)..."
+        mkdir -p "$BACKUP_TMP"
+        # Atomic-backup pattern: stage in .tmp, rename on success. If anything below
+        # fails (including set -e unwinding), the trap wipes the partial dir so it
+        # never gets mistaken for a complete backup by the retention sweep.
+        trap 'rm -rf "$BACKUP_TMP"' ERR EXIT
+        cp -a "$DATA_DIR/." "$BACKUP_TMP/"
+        echo "$OLD_VERSION" > "$BACKUP_TMP/.from-version"
+        echo "$TS" > "$BACKUP_TMP/.timestamp"
+        mv "$BACKUP_TMP" "$BACKUP_DIR"
+        trap - ERR EXIT
+
+        # Retention: keep the newest $RETENTION backups, delete the rest.
+        # Runs AFTER trap is cleared so a retention failure cannot wipe the final backup.
+        ls -1dt "$BACKUP_ROOT"/pre-upgrade-* 2>/dev/null | tail -n +$((RETENTION + 1)) | xargs -r rm -rf
+
+        echo "smartbotic-database: backup complete"
+        ;;
+    install|abort-upgrade) ;;
+    *) echo "preinst called with unknown argument '$1'" >&2; exit 1 ;;
+esac
+exit 0