docs: spec for smartbotic-automation Debian packaging + upstream DB migration

Single .deb (smartbotic-automation) for Debian 13 (trixie), depending on
upstream smartbotic-database / libsmartbotic-db-client. Drops in-repo
src/database/ and rewrites lib/storage/ as an adapter over the upstream
client. Packaging style follows /data/shadowman-cpp/packaging/.

docs/superpowers/specs/2026-05-09-smartbotic-automation-debian-packaging-design.md

@@ -0,0 +1,295 @@
+# smartbotic-automation: Debian 13 packaging + upstream DB migration
+
+**Date:** 2026-05-09
+**Status:** Design — pending user approval
+**Owner:** Ferenc Szontagh
+
+## 1. Goal
+
+Ship the contents of `/data/smartbotic` as a single Debian 13 (`trixie`) `.deb` package named `smartbotic-automation`, distributed through the SmartBotics APT repository at `https://repository.smartbotics.ai`.
+
+As part of this work, drop the in-repo `src/database/` service entirely and have our remaining services (`smartbotic-webserver`, `smartbotic-runner`, `smartbotic-migrate-nodes`) talk to the standalone upstream `smartbotic-database` daemon via its `libsmartbotic-db-client` library (upstream repo: <https://git.smartbotics.ai/fszontagh/smartbotic-database>, current version `1.7.5`).
+
+Packaging style follows `/data/shadowman-cpp/packaging/` and `/tmp/smartbotic-database-upstream/packaging/` — Docker-based two-stage build targeting Debian 13, with `build.sh --sync` integration into `/data/smartbotics-deb-repo/`.
+
+## 2. Identity & metadata
+
+| Field | Value |
+| --- | --- |
+| Package name | `smartbotic-automation` |
+| Version source | `/data/smartbotic/VERSION` (new file, initial value `1.0.0` — matches `project(SmartBotic VERSION 1.0.0 …)` in `CMakeLists.txt`) |
+| Architecture | `amd64` |
+| Maintainer | `Ferenc Szontagh <ferenc.szontagh@smartbotics.ai>` |
+| Section | `contrib/utils` |
+| Priority | `optional` |
+| License | proprietary (`Proprietary software by SmartBotics Kft.`) |
+| Homepage | `https://smartbotics.ai` |
+| Target distro | Debian 13 (`trixie`) |
+
+## 3. Code migration: drop internal DB, adopt upstream client
+
+### 3.1 Removals
+
+- `src/database/` (entire directory: `database_service.{cpp,hpp}`, `memory_store.{cpp,hpp}`, `document.hpp`, `main.cpp`, `persistence/{wal,snapshot}.{cpp,hpp}`)
+- `proto/storage.proto` (replaced by upstream's `database.proto`, which we do not need to ship — upstream client hides gRPC behind PIMPL)
+- The `smartbotic-database` CMake executable target (currently at `CMakeLists.txt:152`)
+- The current `lib/storage/storage_client.{cpp,hpp}` implementations (file paths kept; contents replaced — see §3.2)
+
+### 3.2 `lib/storage/` adapter
+
+Keep the existing public interface so that the 10 caller files do not need to change:
+
+- `src/webserver/webserver_service.{hpp,cpp}`
+- `src/webserver/runners/runner_registry.{hpp,cpp}`
+- `src/webserver/nodes/node_store.{hpp,cpp}`
+- `src/webserver/api/database_controller.cpp`
+- `src/webserver/api/workflow_group_controller.cpp`
+- `src/webserver/api/webhook_controller.{hpp,cpp}`
+
+Internally, `lib/storage/storage_client.cpp` becomes a thin adapter over `smartbotic::database::Client` (`<smartbotic/database/client.hpp>` from `libsmartbotic-db-client-dev`):
+
+| Adapter API (kept stable) | Upstream call |
+| --- | --- |
+| `Result<json> get(coll, id)` | wraps `std::optional<json> Client::get(...)` |
+| `Result<string> insert(coll, data, id, ttl_ms)` | `Client::insert(...)` (note: upstream takes `ttlSeconds` — we convert) |
+| `Result<int64> update(coll, id, data, expected_version, partial)` | dispatch by flags: `partial=false, expected_version=0` → `Client::update`; `partial=false, expected_version!=0` → `Client::updateIfVersion`; `partial=true` → `Client::patch` |
+| `Result<void> remove(coll, id, expected_version)` | `Client::remove` / `Client::removeIfVersion` |
+| `QueryResult query(coll, options)` | `Client::find(...)` (translate filter/sort enums) |
+
+Translation rules:
+- TTL: caller passes ms; upstream takes seconds. Adapter divides by 1000, rounding up; documents the loss-of-precision in a one-line comment.
+- Connection address: adapter `StorageClientConfig::address` default changes from `localhost:9001` to `localhost:9004` (upstream default), and is config-overridable through `webserver.json` / `runner.json`.
+- Versioning options (`VersioningOptions`, `DocumentVersionInfo`, `VersionListResult`) map directly onto upstream's version-history API.
+
+If a method on our existing surface has no upstream equivalent (none expected on a quick read of `client.hpp`), it gets removed and any caller adapted at the same time.
+
+### 3.3 CMake changes
+
+- Remove `src/database/*` source list and the `add_executable(smartbotic-database ...)` block.
+- Replace internal `proto/storage.proto` references in `lib/storage/`'s target with a `find_package(libsmartbotic-db-client REQUIRED)` (or `pkg_check_modules` fallback). `target_link_libraries(smartbotic_storage PUBLIC smartbotic::database::client)` (exact target name to be confirmed when reading the installed CMake config).
+- `proto/credentials.proto`, `proto/runner.proto`, `proto/workflow.proto`, `proto/common.proto` remain — they are between webserver and runner, not the DB.
+
+### 3.4 `smartbotic-migrate-nodes`
+
+Verify (in implementation phase) that this binary only talks to the webserver via HTTP `/api/v1/nodes/migrate` and has no direct DB dependency. If it does have a direct DB dependency, port it to the new client at the same time.
+
+## 4. Installed file layout (system-wide)
+
+```
+/usr/bin/smartbotic-webserver
+/usr/bin/smartbotic-runner
+/usr/bin/smartbotic-migrate-nodes
+
+/usr/share/smartbotic-automation/webui/         # built React app (`webui/dist/`)
+/usr/share/smartbotic-automation/nodes/         # default workflow node JS modules
+
+/etc/smartbotic-automation/webserver.json       # conffile
+/etc/smartbotic-automation/runner.json          # conffile
+/etc/smartbotic-automation/env                  # optional EnvironmentFile (empty default, conffile)
+
+/var/lib/smartbotic-automation/                 # mode 0750, owned smartbotic-automation:smartbotic-automation
+/var/log/smartbotic-automation/                 # mode 0750, owned smartbotic-automation:smartbotic-automation
+
+/lib/systemd/system/smartbotic-automation-webserver.service
+/lib/systemd/system/smartbotic-automation-runner.service
+```
+
+Both default config files point `database.address` at `localhost:9004`. Admin overrides per-deployment.
+
+### 4.1 systemd units
+
+`smartbotic-automation-webserver.service`:
+
+```ini
+[Unit]
+Description=SmartBotic Automation — Webserver (HTTP/WS API)
+After=network.target smartbotic-database.service
+Wants=smartbotic-database.service
+
+[Service]
+Type=simple
+User=smartbotic-automation
+Group=smartbotic-automation
+ExecStart=/usr/bin/smartbotic-webserver --config /etc/smartbotic-automation/webserver.json
+WorkingDirectory=/var/lib/smartbotic-automation
+EnvironmentFile=-/etc/smartbotic-automation/env
+Restart=on-failure
+RestartSec=5
+TimeoutStopSec=15
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target
+```
+
+`smartbotic-automation-runner.service` is structurally identical, with `ExecStart=/usr/bin/smartbotic-runner --config /etc/smartbotic-automation/runner.json` and `After=network.target smartbotic-automation-webserver.service`.
+
+The unit *Wants* `smartbotic-database.service` — that upstream daemon is a `Depends:` of our package. The admin is responsible for ensuring it listens on the port `webserver.json` points at (default `9004`; in test environments where the host already has another `smartbotic-database` instance, run ours on an alternate port and override).
+
+## 5. Dependencies
+
+```
+Depends:
+  smartbotic-database (>= 1.7.5),
+  libsmartbotic-db-client (>= 1.7.5),
+  libssl3t64,
+  libprotobuf32t64,
+  libgrpc++1.51t64,
+  libspdlog1.15,
+  libfmt10,
+  libuuid1,
+  zlib1g,
+  nodejs
+
+Build-Depends:
+  libsmartbotic-db-client-dev (>= 1.7.5),
+  libgrpc++-dev,
+  libprotobuf-dev,
+  protobuf-compiler,
+  protobuf-compiler-grpc,
+  libssl-dev,
+  libspdlog-dev,
+  libfmt-dev,
+  libuuid-dev,
+  nlohmann-json3-dev,
+  nodejs,
+  npm,
+  cmake (>= 3.20),
+  ninja-build,
+  g++ (>= 12)
+```
+
+(The exact versioned names match Debian 13 / `trixie` package names — the `t64` suffix is the time64 transition. We mirror ShadowMan's list and trim to what we actually link.)
+
+## 6. Maintainer scripts
+
+### 6.1 `preinst`
+
+No-op for `install` and `upgrade` (mirrors ShadowMan).
+
+### 6.2 `postinst configure`
+
+1. Create system group + user (idempotent), home `/home/smartbotic-automation`, shell `/usr/sbin/nologin`.
+2. `install -d -o smartbotic-automation -g smartbotic-automation -m 0750` on:
+   - `/home/smartbotic-automation`
+   - `/var/lib/smartbotic-automation`
+   - `/var/log/smartbotic-automation`
+   - `/etc/smartbotic-automation` (only if missing)
+3. `systemctl daemon-reload`
+4. **First install:** `deb-systemd-helper enable smartbotic-automation-webserver.service smartbotic-automation-runner.service` — **but do not start them.** Reason: the operator must first verify that `smartbotic-database` is reachable at the configured port (default `9004`) before starting our services. A `NEWS.Debian` or echo at install time will tell the admin to run `systemctl start smartbotic-automation-webserver smartbotic-automation-runner` once the DB is ready.
+5. **Upgrade:** `deb-systemd-invoke try-restart` for any unit that was already active.
+
+### 6.3 `prerm remove` / `prerm upgrade`
+
+`deb-systemd-invoke stop` both units. On `prerm remove`: also `deb-systemd-helper disable` them.
+
+### 6.4 `postrm purge`
+
+- `deluser --system smartbotic-automation` (idempotent)
+- `delgroup --system smartbotic-automation` (idempotent)
+- `rm -rf /var/lib/smartbotic-automation /var/log/smartbotic-automation`
+- Leave `/etc/smartbotic-automation/` to dpkg's normal conffile handling.
+
+## 7. `packaging/` directory layout (added to `/data/smartbotic`)
+
+```
+packaging/
+  Dockerfile.base                 # Debian 13 + build deps + libsmartbotic-db-client-dev
+  Dockerfile.build                # COPY source, build C++ + npm run build, dpkg-deb assembly
+  build.sh                        # orchestrator
+  smartbotics-repo.gpg            # public APT key (copied from /data/smartbotics-deb-repo/)
+  deb/
+    templates/
+      control.smartbotic-automation
+    scripts/
+      preinst
+      postinst
+      prerm
+      postrm
+    systemd/
+      smartbotic-automation-webserver.service
+      smartbotic-automation-runner.service
+    config/
+      webserver.json
+      runner.json
+      env
+```
+
+### 7.1 `Dockerfile.base`
+
+Mirrors ShadowMan's `Dockerfile.base`:
+
+- `FROM debian:trixie`
+- Configure SmartBotics APT repo with `REPO_USER` / `REPO_PASS` build args (used to install `libsmartbotic-db-client-dev`)
+- Install build toolchain: `build-essential cmake ninja-build git pkg-config ccache libssl-dev libprotobuf-dev libgrpc++-dev protobuf-compiler-grpc nlohmann-json3-dev libspdlog-dev libfmt-dev libuuid-dev zlib1g-dev nodejs npm libsmartbotic-db-client-dev`
+
+Image tag: `smartbotic-automation-build-base:debian13`.
+
+### 7.2 `Dockerfile.build`
+
+- `FROM smartbotic-automation-build-base:debian13`
+- Mount source as `/src`, build dir `/build`, output dir `/out`
+- `cmake -G Ninja -B /build -S /src -DCMAKE_BUILD_TYPE=Release`
+- `cmake --build /build -j${BUILD_JOBS}`
+- `npm --prefix /src/webui ci && npm --prefix /src/webui run build`
+- `bash /src/packaging/scripts/assemble-deb.sh /build /src/webui/dist /out` — writes the `DEBIAN/` payload tree, runs `dpkg-deb --build`
+- Final `.deb` lands in `/out/`
+
+### 7.3 `build.sh` flag surface (matches ShadowMan + upstream)
+
+```
+[VERSION]
+--rebuild-base
+--no-cache
+--deb-revision N        (default 1)
+--repo                  (run /data/smartbotics-deb-repo/scripts/{add-packages,create-repo --suite})
+--suite NAME            (default trixie)
+--sync                  (--repo + sync-repo.sh)
+--help / -h
+```
+
+Env: `BUILD_JOBS` (default `nproc`), `DEB_REPO_DIR` (default `/data/smartbotics-deb-repo`).
+
+Outputs land in `/data/smartbotic/dist/debian13/smartbotic-automation_<version>-<rev>_amd64.deb`.
+
+## 8. APT repository deployment
+
+`build.sh --sync --suite trixie` chains:
+
+1. `${DEB_REPO_DIR}/scripts/add-packages.sh dist/debian13/`
+2. `${DEB_REPO_DIR}/scripts/create-repo.sh --suite trixie`
+3. `${DEB_REPO_DIR}/scripts/sync-repo.sh`
+
+End state: `apt install smartbotic-automation` works on a `trixie` host that has the SmartBotics repo configured.
+
+## 9. Local-instance cleanup (this host)
+
+Confirmed during exploration:
+
+- Nothing built from `/data/smartbotic` is currently running as a service or process.
+- The system service `smartbotic-database.service` running `/usr/bin/smartbotic-database` (installed package `smartbotic-database 1.7.1-1`) belongs to a different project — left intact per user direction.
+
+No cleanup action needed on this host. Testing of the new package will happen on a separate VM or with a side-by-side smartbotic-database instance running on an alternate port.
+
+## 10. Decisions explicitly approved by user
+
+1. **DB strategy:** delete `src/database/`, depend on upstream `smartbotic-database` package, talk to it via `libsmartbotic-db-client`.
+2. **Package layout:** single `smartbotic-automation` package (no split).
+3. **Install prefix:** system-wide (`/usr/bin/`, `/usr/share/smartbotic-automation/`, `/etc/smartbotic-automation/`).
+4. **Local instance:** leave the unrelated `smartbotic-database 1.7.1-1` system service alone.
+5. **Auto-start policy on first install:** `enable` units, but **do not start** — admin must point config at a reachable `smartbotic-database` and start manually (the upstream DB port is deployment-specific).
+
+## 11. Out of scope
+
+- Repackaging the upstream `smartbotic-database` itself.
+- Any frontend / `webui/` feature work beyond `npm run build`.
+- Cross-distro builds (only `trixie` is targeted).
+- Migrating the existing `/data/smartbotic` `smartbotic-database` *data* (WAL, snapshots) — operators run a fresh upstream instance for their deployment.
+
+## 12. Implementation phases (for the plan)
+
+1. **Phase A — code migration**: remove `src/database/`, rewrite `lib/storage/`, fix CMake, ensure native build (`cmake --build build`) succeeds.
+2. **Phase B — packaging skeleton**: add `packaging/` directory, write `Dockerfile.base`, `Dockerfile.build`, `build.sh`, control template, maintainer scripts, systemd units, default configs, `VERSION` file. Confirm Docker build produces a `.deb`.
+3. **Phase C — install / smoke test**: install the `.deb` in a throwaway container or VM, start the upstream `smartbotic-database` on the configured port, start `smartbotic-automation-webserver`, hit `/api/v1/healthz`, confirm node migration and basic workflow works.
+4. **Phase D — repo publish**: `build.sh --sync --suite trixie`. Verify `apt update && apt install smartbotic-automation` works from a clean `trixie` host.