smartbotic/deploy/zeus

The database, on zeus, in Docker

The database was moved off mulan because it and the SD.cpp REST API could not share that machine's memory. It runs on zeus as a container; mulan keeps the webserver and the runner, which reach it over the network.

What is here

• Dockerfile - the image. It copies mulan's own smartbotic-database binary and libsmartbotic-db-client.so rather than installing the package.

That is deliberate. The .deb in the package repo is a later build linked against Abseil 20240722; mulan runs one linked against 20260107. Installing the package produced a container that could not start at all (libabsl_log_internal_check_op.so.20240722: cannot open shared object file). Copying the running binary onto the same Ubuntu release keeps the libraries matched. Rebuild the image from mulan's files whenever the database is upgraded there.

• config.json - mulan's config with two changes:
  • bind_address is 0.0.0.0, because loopback inside a container reaches nothing outside it.
  • max_memory_mb is 8192 rather than 512. The old figure was what mulan could spare and is the reason for the move; at 512 MB the container sat at 100% of its budget and refused writes with "memory pressure emergency".

Running it

docker run -d --name smartbotic-db --restart unless-stopped --memory 12g \
  -v /data/smartbotic-db/data:/var/lib/smartbotic-database \
  -v /data/smartbotic-db/etc/config.json:/etc/smartbotic-database/config.json:ro \
  -p 9004:9004 smartbotic-database:2.8.1

Data lives on /data (1.6 TB free), not / (9 GB free).

The encryption key is inside the data directory (storage.key). Data and key travel together or the data is unreadable - back them up together, and never copy the data without it.

Exposure

Port 9004 is published on all interfaces and the database speaks plaintext with no authentication - it was only ever reachable on mulan's loopback before. It now holds password hashes and encrypted credentials on a LAN-reachable port. The client supports TLS and a bearer token; a host firewall limiting 9004 to mulan is the smaller step. Neither is done yet.

Moving the data

The database was stopped first. A live copy of a WAL-backed store can be torn, and recovery reported TrivialSuccess: 27288 docs in 93 collections with zero WAL entries to replay - which is what a clean copy looks like.

Time

zeus was 142 seconds behind mulan. That matters here because the database computes TTL expiries as absolute times, so a clock that disagrees with the machines writing to it shifts every retention deadline by the difference.

chrony is installed and enabled under runit, taking its time from the router at 192.168.2.1 - the same server DHCP advertises. dhcpcd's 50-ntp.conf hook writes the lease's NTP servers into the config, and /etc/dhcpcd.conf sets NTP_CONF=/etc/chrony.conf so it writes them where chrony will read them; by default the hook picks /etc/ntp.conf, which nothing here reads.

Two things learned while setting it up, both visible only in the log:

• Naming the router in chrony.conf by hand and letting DHCP write it in produced Could not add source 192.168.2.1 on every start. DHCP supplies it, so the manual line is gone.

• With pool pool.ntp.org also configured, chrony synced to a stratum-2 pool server in preference to the router at stratum 3. The pool is gone: the router is the source. If it is down the clock free-runs, which is the honest outcome on a LAN rather than quietly taking somebody else's time.

  chronyc tracking # Reference ID should be C0A80201 (192.168.2.1) chronyc sources # ^* marks the selected source

The container takes the host's clock, so nothing is configured inside it.