deploy: move the database to zeus, in Docker, and point mulan at it

The database and the SD.cpp REST API could not share mulan's memory. The
database now runs on zeus as a container; mulan keeps the webserver and
the runner and reaches it over the network.

The image copies mulan's own binary rather than installing the package.
The .deb in the repo is a later build linked against Abseil 20240722 while
mulan runs one linked against 20260107, so installing it produced a
container that would not start - the same version number, a different
build. Copying the running binary onto the same Ubuntu release keeps the
libraries matched, and matches what was asked for: the version that is
actually running.

Two config changes, both forced. bind_address becomes 0.0.0.0, because
loopback inside a container reaches nothing outside it. And max_memory_mb
goes from 512 to 8192: 512 was what mulan could spare and is the reason
for the move, but in the container it sat at 100% of its budget and
refused every write with "memory pressure emergency" - logins included.
zeus has 79 GB free and the whole dataset is 3 GB.

The data was copied with the database stopped. A live copy of a WAL-backed
store can be torn; recovery on zeus reported TrivialSuccess, 27,288
documents in 93 collections, zero WAL entries to replay.

Verified: 9 workflows, 7 credentials, 10,106 executions and 117 images all
served from zeus; 253 files download byte-exact, which is what proves the
encryption key came across intact - it lives inside the data directory. A
real workflow ran end to end against it, and the node suite passes: 68
passed, 0 failed, 2 skipped for a model this machine is not running.

Not done, and written down in deploy/zeus/README.md: 9004 is published on
all interfaces and speaks plaintext with no authentication. It was on
mulan's loopback before. It holds password hashes and encrypted
credentials.

config/runner.json

@@ -4,7 +4,7 @@
   "webserver_address": "${WEBSERVER_ADDRESS:localhost:8090}",
   "node_sync_address": "${NODE_SYNC_ADDRESS:localhost:9012}",
   "credential_service_address": "${CREDENTIAL_SERVICE_ADDRESS:localhost:9013}",
-  "database_address": "${DATABASE_ADDRESS:localhost:9004}",
+  "database_address": "${DATABASE_ADDRESS:zeus.fsociety.hu:9004}",
   "database_project": "${DATABASE_PROJECT:smartbotic-automation}",
   "max_message_size_mb": 64,
   "node_sync": {

config/webserver.json

@@ -3,7 +3,7 @@
   "node_sync_port": 9012,
   "credential_service_port": 9013,
   "static_files_path": "${WEBUI_PATH:./webui/dist}",
-  "database_address": "${DATABASE_ADDRESS:localhost:9004}",
+  "database_address": "${DATABASE_ADDRESS:zeus.fsociety.hu:9004}",
   "database_project": "${DATABASE_PROJECT:smartbotic-automation}",
   "runners": {
     "load_balancing": "least-connections",

deploy/zeus/Dockerfile

@@ -0,0 +1,32 @@
+# The database exactly as it runs on the host today.
+#
+# Not built from the .deb in the package repo: that one is a later build linked
+# against Abseil 20240722, while the host's binary - the one whose data this is
+# and whose behaviour is known - links 20260107. Installing the .deb produced a
+# container that could not start at all. The host's own files are copied in
+# instead, onto the same Ubuntu release the host runs, so the libraries match.
+FROM ubuntu:26.04
+
+# Runtime dependencies, from the package list the .deb declares plus what the
+# client library pulls in.
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+      libssl3t64 liblz4-1 libsystemd0 libjemalloc2 libyyjson0 liblmdb0 \
+      libprotobuf32t64 libgrpc++1.51t64 libspdlog1.15 libfmt10 ca-certificates \
+ && rm -rf /var/lib/apt/lists/*
+
+COPY hostbin/smartbotic-database /usr/bin/smartbotic-database
+COPY hostbin/libsmartbotic-db-client.so* /usr/lib/x86_64-linux-gnu/
+RUN ldconfig && ldd /usr/bin/smartbotic-database | grep -q "not found" && exit 1 || true
+
+# Matching the host unit, which sets these to stop the allocator holding on to
+# freed memory - the reason this is being moved off that machine at all.
+ENV MALLOC_ARENA_MAX=2
+ENV MALLOC_CONF=background_thread:true,dirty_decay_ms:1000,muzzy_decay_ms:1000
+
+# The encryption key lives inside the data directory, so the data and the key
+# that reads it are never separated.
+VOLUME ["/var/lib/smartbotic-database"]
+EXPOSE 9004
+
+ENTRYPOINT ["/usr/bin/smartbotic-database", "--config", "/etc/smartbotic-database/config.json"]

deploy/zeus/README.md

@@ -0,0 +1,52 @@
+# The database, on zeus, in Docker
+
+The database was moved off mulan because it and the SD.cpp REST API could not
+share that machine's memory. It runs on zeus as a container; mulan keeps the
+webserver and the runner, which reach it over the network.
+
+## What is here
+
+- `Dockerfile` - the image. It copies mulan's own `smartbotic-database` binary
+  and `libsmartbotic-db-client.so` rather than installing the package.
+
+  That is deliberate. The `.deb` in the package repo is a later build linked
+  against Abseil 20240722; mulan runs one linked against 20260107. Installing
+  the package produced a container that could not start at all
+  (`libabsl_log_internal_check_op.so.20240722: cannot open shared object file`).
+  Copying the running binary onto the same Ubuntu release keeps the libraries
+  matched. Rebuild the image from mulan's files whenever the database is
+  upgraded there.
+
+- `config.json` - mulan's config with two changes:
+  - `bind_address` is `0.0.0.0`, because loopback inside a container reaches
+    nothing outside it.
+  - `max_memory_mb` is 8192 rather than 512. The old figure was what mulan could
+    spare and is the reason for the move; at 512 MB the container sat at 100% of
+    its budget and refused writes with "memory pressure emergency".
+
+## Running it
+
+    docker run -d --name smartbotic-db --restart unless-stopped --memory 12g \
+      -v /data/smartbotic-db/data:/var/lib/smartbotic-database \
+      -v /data/smartbotic-db/etc/config.json:/etc/smartbotic-database/config.json:ro \
+      -p 9004:9004 smartbotic-database:2.8.1
+
+Data lives on `/data` (1.6 TB free), not `/` (9 GB free).
+
+**The encryption key is inside the data directory** (`storage.key`). Data and key
+travel together or the data is unreadable - back them up together, and never
+copy the data without it.
+
+## Exposure
+
+Port 9004 is published on all interfaces and the database speaks plaintext with
+no authentication - it was only ever reachable on mulan's loopback before. It
+now holds password hashes and encrypted credentials on a LAN-reachable port. The
+client supports TLS and a bearer token; a host firewall limiting 9004 to mulan
+is the smaller step. Neither is done yet.
+
+## Moving the data
+
+The database was stopped first. A live copy of a WAL-backed store can be torn,
+and recovery reported `TrivialSuccess: 27288 docs in 93 collections` with zero
+WAL entries to replay - which is what a clean copy looks like.

deploy/zeus/config.json

@@ -0,0 +1,61 @@
+{
+  "log_level": "info",
+  "storage": {
+    "bind_address": "0.0.0.0",
+    "rpc_port": 9004,
+    "node_id": "smartbotic-db",
+    "data_directory": "/var/lib/smartbotic-database",
+    "memory": {
+      "max_memory_mb": 8192,
+      "eviction_threshold_percent": 80,
+      "eviction_target_percent": 60,
+      "eviction_check_interval_ms": 5000,
+      "eviction_chunk_size": 1000,
+      "eviction_chunk_pause_ms": 50,
+      "max_eviction_passes_per_trigger": 20,
+      "hot_write_floor_ms": 30000,
+      "memory_soft_percent": 70,
+      "memory_hard_percent": 85,
+      "memory_emergency_percent": 95,
+      "eviction_burst_threshold": 10000
+    },
+    "persistence": {
+      "wal_sync_interval_ms": 100,
+      "snapshot_interval_sec": 3600,
+      "compression": "lz4",
+      "snapshots": {
+        "validate_after_write": true,
+        "cleanup_only_if_verified": true
+      },
+      "recovery": {
+        "mode": "normal",
+        "auto_escalate": true,
+        "allow_empty_on_fresh_install": true
+      }
+    },
+    "grpc": {
+      "max_receive_message_size_mb": 100,
+      "max_send_message_size_mb": 100,
+      "resource_quota_memory_mb": 2048,
+      "max_concurrent_subscribe_streams": 50,
+      "max_concurrent_file_streams": 10
+    },
+    "encryption": {
+      "enabled": true,
+      "key_file": "/var/lib/smartbotic-database/storage.key",
+      "auto_generate_key": true
+    },
+    "migrations": {
+      "enabled": false,
+      "directory": "/etc/smartbotic-database/migrations",
+      "auto_apply": false
+    },
+    "files": {
+      "max_file_size_mb": 500,
+      "cleanup_orphans_interval_sec": 3600
+    },
+    "replication": {
+      "enabled": false
+    }
+  }
+}