|
|
@@ -0,0 +1,1754 @@
|
|
|
+# smartbotic-automation Debian Packaging Implementation Plan
|
|
|
+
|
|
|
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
+
|
|
|
+**Goal:** Ship `/data/smartbotic` as a Debian 13 `.deb` package named `smartbotic-automation`, replacing the in-repo `src/database/` with the upstream `smartbotic-database` daemon (talked to via `libsmartbotic-db-client`).
|
|
|
+
|
|
|
+**Architecture:** Three phases: (A) code migration — drop `src/database/`, rewrite `lib/storage/` as a thin adapter over `smartbotic::database::Client`; (B) packaging — Docker-based two-stage Debian 13 build mirroring `/data/shadowman-cpp/packaging/`; (C) smoke test + (D) repo publish via `/data/smartbotics-deb-repo/`.
|
|
|
+
|
|
|
+**Tech Stack:** C++20, gRPC (via upstream client only), `libsmartbotic-db-client`, CMake + Ninja, Docker BuildKit, Debian 13 (`trixie`), systemd, `dpkg-deb`.
|
|
|
+
|
|
|
+**Spec:** [`docs/superpowers/specs/2026-05-09-smartbotic-automation-debian-packaging-design.md`](../specs/2026-05-09-smartbotic-automation-debian-packaging-design.md)
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## File Structure Map
|
|
|
+
|
|
|
+### Created
|
|
|
+
|
|
|
+| Path | Purpose |
|
|
|
+| --- | --- |
|
|
|
+| `VERSION` | Single-line version string (`1.0.0`) |
|
|
|
+| `packaging/Dockerfile.base` | Debian 13 + build deps + `libsmartbotic-db-client-dev` |
|
|
|
+| `packaging/Dockerfile.build` | Compile C++ + WebUI, assemble `.deb` payload, run `dpkg-deb` |
|
|
|
+| `packaging/build.sh` | Orchestrator (mirrors `/data/shadowman-cpp/packaging/build.sh`) |
|
|
|
+| `packaging/smartbotics-repo.gpg` | Public key for SmartBotics APT repo (copied from deb repo) |
|
|
|
+| `packaging/scripts/create-deb.sh` | Inside-container packager: stages files, runs `dpkg-deb --build` |
|
|
|
+| `packaging/deb/templates/control.smartbotic-automation` | Debian control file (with `{{VERSION}}` / `{{INSTALLED_SIZE}}` placeholders) |
|
|
|
+| `packaging/deb/scripts/preinst` | No-op pre-install hook |
|
|
|
+| `packaging/deb/scripts/postinst` | Create user/group, dirs, daemon-reload, enable units |
|
|
|
+| `packaging/deb/scripts/prerm` | Stop + (on remove) disable units |
|
|
|
+| `packaging/deb/scripts/postrm` | Purge user/group + data/log dirs |
|
|
|
+| `packaging/deb/systemd/smartbotic-automation-webserver.service` | systemd unit |
|
|
|
+| `packaging/deb/systemd/smartbotic-automation-runner.service` | systemd unit |
|
|
|
+| `packaging/deb/config/webserver.json` | Default `/etc/smartbotic-automation/webserver.json` |
|
|
|
+| `packaging/deb/config/runner.json` | Default `/etc/smartbotic-automation/runner.json` |
|
|
|
+| `packaging/deb/config/env` | Default empty `/etc/smartbotic-automation/env` |
|
|
|
+
|
|
|
+### Modified
|
|
|
+
|
|
|
+| Path | Change |
|
|
|
+| --- | --- |
|
|
|
+| `CMakeLists.txt` | Drop `smartbotic-database` target + `src/database/` sources; add `find_package(smartbotic-db-client)`; link `lib/storage` against upstream client; drop `proto/storage.proto` from generated set |
|
|
|
+| `lib/storage/storage_client.hpp` | Drop gRPC + internal proto includes; keep public API stable |
|
|
|
+| `lib/storage/storage_client.cpp` | Replace gRPC stub calls with `smartbotic::database::Client` calls |
|
|
|
+
|
|
|
+### Deleted
|
|
|
+
|
|
|
+| Path | Reason |
|
|
|
+| --- | --- |
|
|
|
+| `src/database/` (entire directory) | Replaced by upstream `smartbotic-database` daemon |
|
|
|
+| `proto/storage.proto` | Replaced by upstream's `database.proto` (hidden inside client lib) |
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## Phase A — Code Migration
|
|
|
+
|
|
|
+### Task A1: Confirm `smartbotic-migrate-nodes` has no direct DB dependency
|
|
|
+
|
|
|
+**Files:** Read-only — `src/tools/migrate_nodes.cpp`
|
|
|
+
|
|
|
+- [ ] **Step 1: Grep for storage/proto usage**
|
|
|
+
|
|
|
+```bash
|
|
|
+grep -nE 'storage_client|StorageClient|smartbotic::storage|proto::Storage|<smartbotic/database' \
|
|
|
+ /data/smartbotic/src/tools/migrate_nodes.cpp
|
|
|
+```
|
|
|
+
|
|
|
+Expected: no matches. The tool is a pure HTTP client (uses `<curl/curl.h>` and `<nlohmann/json.hpp>` only). If matches appear, STOP and add the migration of this tool to Phase A before continuing.
|
|
|
+
|
|
|
+- [ ] **Step 2: Verify CMake target deps**
|
|
|
+
|
|
|
+```bash
|
|
|
+grep -n 'smartbotic-migrate-nodes\|migrate_nodes' /data/smartbotic/CMakeLists.txt
|
|
|
+```
|
|
|
+
|
|
|
+Expected: target `smartbotic-migrate-nodes` links only against `CURL::libcurl`, `nlohmann_json::nlohmann_json`, `smartbotic_logging`, `smartbotic_common`. No `smartbotic_storage` or `smartbotic_proto` link. If it links to those, document the additional refactor needed and add a Task A6½ to fix.
|
|
|
+
|
|
|
+### Task A2: Install `libsmartbotic-db-client-dev` for native dev build
|
|
|
+
|
|
|
+**Files:** None (host setup)
|
|
|
+
|
|
|
+- [ ] **Step 1: Check if already installed**
|
|
|
+
|
|
|
+```bash
|
|
|
+dpkg -l libsmartbotic-db-client-dev 2>&1 | grep '^ii' || echo "not installed"
|
|
|
+```
|
|
|
+
|
|
|
+If already installed, skip to Task A3.
|
|
|
+
|
|
|
+- [ ] **Step 2: Install if missing**
|
|
|
+
|
|
|
+```bash
|
|
|
+sudo apt-get update && sudo apt-get install -y libsmartbotic-db-client-dev
|
|
|
+```
|
|
|
+
|
|
|
+Expected: pulls in the dev headers under `/usr/include/smartbotic/database/` and CMake config files under `/usr/lib/x86_64-linux-gnu/cmake/` (or pkg-config under `/usr/lib/x86_64-linux-gnu/pkgconfig/`).
|
|
|
+
|
|
|
+- [ ] **Step 3: Locate the install path of the client**
|
|
|
+
|
|
|
+```bash
|
|
|
+dpkg -L libsmartbotic-db-client-dev | grep -E '(client\.hpp|smartbotic-db-client.*\.cmake|smartbotic-db-client\.pc)$'
|
|
|
+```
|
|
|
+
|
|
|
+Record the CMake target name or pkg-config name in your shell history — Task A3 will need it. (Likely either `smartbotic::database::client` CMake target or `smartbotic-db-client` pkg-config name.)
|
|
|
+
|
|
|
+### Task A3: Add `libsmartbotic-db-client` to `CMakeLists.txt`
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Modify: `/data/smartbotic/CMakeLists.txt:80-100` (around the `smartbotic_storage` target)
|
|
|
+
|
|
|
+- [ ] **Step 1: Add a `find_package` block**
|
|
|
+
|
|
|
+Insert near the other `find_package` calls (top of CMakeLists, before `smartbotic_storage` definition). Use a fallback chain so we work whether the upstream ships CMake config or only pkg-config:
|
|
|
+
|
|
|
+```cmake
|
|
|
+# Upstream smartbotic-database client library
|
|
|
+find_package(smartbotic-db-client CONFIG QUIET)
|
|
|
+if(NOT smartbotic-db-client_FOUND)
|
|
|
+ find_package(PkgConfig REQUIRED)
|
|
|
+ pkg_check_modules(SMARTBOTIC_DB_CLIENT REQUIRED IMPORTED_TARGET smartbotic-db-client)
|
|
|
+ add_library(smartbotic::database::client ALIAS PkgConfig::SMARTBOTIC_DB_CLIENT)
|
|
|
+endif()
|
|
|
+```
|
|
|
+
|
|
|
+If Task A2 Step 3 revealed a different target name, substitute it everywhere in this plan.
|
|
|
+
|
|
|
+- [ ] **Step 2: Configure & verify**
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /data/smartbotic
|
|
|
+rm -rf build
|
|
|
+cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug
|
|
|
+```
|
|
|
+
|
|
|
+Expected: configuration succeeds. The `smartbotic-db-client` package is found. (Build will still pass at this stage because nothing links against it yet.)
|
|
|
+
|
|
|
+- [ ] **Step 3: Commit**
|
|
|
+
|
|
|
+```bash
|
|
|
+git add CMakeLists.txt
|
|
|
+git commit -m "build: discover libsmartbotic-db-client via CMake/pkg-config"
|
|
|
+```
|
|
|
+
|
|
|
+### Task A4: Rewrite `lib/storage/storage_client.hpp`
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Rewrite: `/data/smartbotic/lib/storage/storage_client.hpp` (entire file, 115 lines → ~110 lines)
|
|
|
+
|
|
|
+- [ ] **Step 1: Replace the file contents**
|
|
|
+
|
|
|
+```cpp
|
|
|
+#pragma once
|
|
|
+
|
|
|
+#include <memory>
|
|
|
+#include <string>
|
|
|
+#include <vector>
|
|
|
+
|
|
|
+#include <nlohmann/json.hpp>
|
|
|
+
|
|
|
+#include "common/error.hpp"
|
|
|
+
|
|
|
+namespace smartbotic::storage {
|
|
|
+
|
|
|
+// Storage client configuration
|
|
|
+struct StorageClientConfig {
|
|
|
+ std::string address = "localhost:9010"; // upstream smartbotic-database default in our deployments
|
|
|
+ int timeout_ms = 5000;
|
|
|
+ int max_message_size_mb = 64; // kept for back-compat; ignored by adapter
|
|
|
+};
|
|
|
+
|
|
|
+// Query options - field projection and multi-sort kept for source compatibility,
|
|
|
+// but the upstream client supports only a single sort field. The adapter will
|
|
|
+// log a warning and use only the first entry of `sorts`. Field projection is
|
|
|
+// silently ignored by upstream; callers must filter in-process if needed.
|
|
|
+struct QueryOptions {
|
|
|
+ std::vector<std::pair<std::string, nlohmann::json>> filters;
|
|
|
+ std::vector<std::pair<std::string, bool>> sorts; // field, ascending
|
|
|
+ std::vector<std::string> fields;
|
|
|
+ int32_t page = 1;
|
|
|
+ int32_t page_size = 100;
|
|
|
+};
|
|
|
+
|
|
|
+struct QueryResult {
|
|
|
+ std::vector<nlohmann::json> documents;
|
|
|
+ int64_t total_count = 0;
|
|
|
+ bool has_more = false;
|
|
|
+};
|
|
|
+
|
|
|
+struct VersioningOptions {
|
|
|
+ bool enabled = false;
|
|
|
+ int32_t max_versions = 0;
|
|
|
+ int64_t version_ttl_ms = 0;
|
|
|
+ bool keep_on_delete = false;
|
|
|
+};
|
|
|
+
|
|
|
+struct DocumentVersionInfo {
|
|
|
+ std::string id;
|
|
|
+ std::string document_id;
|
|
|
+ int64_t version = 0;
|
|
|
+ nlohmann::json data;
|
|
|
+ int64_t created_at = 0;
|
|
|
+};
|
|
|
+
|
|
|
+struct VersionListResult {
|
|
|
+ std::vector<DocumentVersionInfo> versions;
|
|
|
+ int64_t total_count = 0;
|
|
|
+ bool has_more = false;
|
|
|
+};
|
|
|
+
|
|
|
+// Adapter over smartbotic::database::Client. Public surface unchanged from the
|
|
|
+// pre-1.0 internal-DB version; PIMPL hides the upstream client.
|
|
|
+class StorageClient {
|
|
|
+public:
|
|
|
+ explicit StorageClient(const StorageClientConfig& config);
|
|
|
+ ~StorageClient();
|
|
|
+
|
|
|
+ StorageClient(const StorageClient&) = delete;
|
|
|
+ StorageClient& operator=(const StorageClient&) = delete;
|
|
|
+ StorageClient(StorageClient&&) noexcept;
|
|
|
+ StorageClient& operator=(StorageClient&&) noexcept;
|
|
|
+
|
|
|
+ common::Result<nlohmann::json> get(const std::string& collection, const std::string& id);
|
|
|
+
|
|
|
+ common::Result<std::string> insert(const std::string& collection,
|
|
|
+ const nlohmann::json& data,
|
|
|
+ const std::string& id = "",
|
|
|
+ int64_t ttl_ms = 0);
|
|
|
+
|
|
|
+ common::Result<int64_t> update(const std::string& collection,
|
|
|
+ const std::string& id,
|
|
|
+ const nlohmann::json& data,
|
|
|
+ int64_t expected_version = 0,
|
|
|
+ bool partial = false);
|
|
|
+
|
|
|
+ common::Result<void> remove(const std::string& collection,
|
|
|
+ const std::string& id,
|
|
|
+ int64_t expected_version = 0);
|
|
|
+
|
|
|
+ common::Result<QueryResult> query(const std::string& collection);
|
|
|
+ common::Result<QueryResult> query(const std::string& collection, const QueryOptions& options);
|
|
|
+
|
|
|
+ common::Result<void> createCollection(const std::string& name,
|
|
|
+ const nlohmann::json& schema = nlohmann::json{},
|
|
|
+ int64_t default_ttl_ms = 0,
|
|
|
+ const VersioningOptions* versioning = nullptr);
|
|
|
+
|
|
|
+ common::Result<void> dropCollection(const std::string& name);
|
|
|
+
|
|
|
+ std::vector<std::string> listCollections();
|
|
|
+
|
|
|
+ common::Result<nlohmann::json> getVersion(const std::string& collection,
|
|
|
+ const std::string& id,
|
|
|
+ int64_t version);
|
|
|
+
|
|
|
+ common::Result<VersionListResult> listVersions(const std::string& collection,
|
|
|
+ const std::string& id,
|
|
|
+ int32_t limit = 100,
|
|
|
+ int32_t offset = 0);
|
|
|
+
|
|
|
+ bool isConnected() const;
|
|
|
+
|
|
|
+private:
|
|
|
+ class Impl;
|
|
|
+ std::unique_ptr<Impl> impl_;
|
|
|
+};
|
|
|
+
|
|
|
+} // namespace smartbotic::storage
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 2: Sanity-check with grep**
|
|
|
+
|
|
|
+```bash
|
|
|
+grep -nE 'grpcpp|proto::Storage|storage\.grpc\.pb' /data/smartbotic/lib/storage/storage_client.hpp
|
|
|
+```
|
|
|
+
|
|
|
+Expected: zero matches (header is now grpc/proto-free).
|
|
|
+
|
|
|
+### Task A5: Rewrite `lib/storage/storage_client.cpp`
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Rewrite: `/data/smartbotic/lib/storage/storage_client.cpp` (entire file, 347 lines → ~280 lines)
|
|
|
+
|
|
|
+- [ ] **Step 1: Replace the file contents**
|
|
|
+
|
|
|
+```cpp
|
|
|
+#include "storage/storage_client.hpp"
|
|
|
+
|
|
|
+#include <chrono>
|
|
|
+#include <smartbotic/database/client.hpp>
|
|
|
+
|
|
|
+#include "logging/logger.hpp"
|
|
|
+
|
|
|
+namespace smartbotic::storage {
|
|
|
+
|
|
|
+namespace dbc = smartbotic::database;
|
|
|
+using common::Error;
|
|
|
+using common::Result;
|
|
|
+
|
|
|
+namespace {
|
|
|
+
|
|
|
+// Round milliseconds up to whole seconds. 0 -> 0 (no expiry).
|
|
|
+uint32_t msToSec(int64_t ms) {
|
|
|
+ if (ms <= 0) return 0;
|
|
|
+ return static_cast<uint32_t>((ms + 999) / 1000);
|
|
|
+}
|
|
|
+
|
|
|
+// Translate our string-encoded operator to upstream FilterOp.
|
|
|
+// Returns FilterOp::EQ for unknown ops (with a warning log).
|
|
|
+dbc::Client::FilterOp translateOp(const std::string& op) {
|
|
|
+ if (op == "eq" || op.empty()) return dbc::Client::FilterOp::EQ;
|
|
|
+ if (op == "ne") return dbc::Client::FilterOp::NE;
|
|
|
+ if (op == "gt") return dbc::Client::FilterOp::GT;
|
|
|
+ if (op == "gte") return dbc::Client::FilterOp::GTE;
|
|
|
+ if (op == "lt") return dbc::Client::FilterOp::LT;
|
|
|
+ if (op == "lte") return dbc::Client::FilterOp::LTE;
|
|
|
+ if (op == "in") return dbc::Client::FilterOp::IN;
|
|
|
+ if (op == "contains") return dbc::Client::FilterOp::CONTAINS;
|
|
|
+ if (op == "exists") return dbc::Client::FilterOp::EXISTS;
|
|
|
+ if (op == "regex") return dbc::Client::FilterOp::REGEX;
|
|
|
+ if (op == "search") return dbc::Client::FilterOp::SEARCH;
|
|
|
+ LOG_WARN("Storage: unknown filter op '{}', defaulting to EQ", op);
|
|
|
+ return dbc::Client::FilterOp::EQ;
|
|
|
+}
|
|
|
+
|
|
|
+// Decode a {field, value} pair into an upstream Filter. The value can be a
|
|
|
+// raw JSON value (implicit EQ) or {"op": "...", "value": ...}.
|
|
|
+dbc::Client::Filter buildFilter(const std::string& field, const nlohmann::json& v) {
|
|
|
+ dbc::Client::Filter f;
|
|
|
+ f.field = field;
|
|
|
+ if (v.is_object() && v.contains("op")) {
|
|
|
+ f.op = translateOp(v.value("op", "eq"));
|
|
|
+ f.value = v.value("value", nlohmann::json{});
|
|
|
+ } else {
|
|
|
+ f.op = dbc::Client::FilterOp::EQ;
|
|
|
+ f.value = v;
|
|
|
+ }
|
|
|
+ return f;
|
|
|
+}
|
|
|
+
|
|
|
+} // namespace
|
|
|
+
|
|
|
+class StorageClient::Impl {
|
|
|
+public:
|
|
|
+ explicit Impl(const StorageClientConfig& cfg) {
|
|
|
+ dbc::Client::Config config;
|
|
|
+ config.address = cfg.address;
|
|
|
+ config.timeoutMs = static_cast<uint32_t>(cfg.timeout_ms);
|
|
|
+ client_ = std::make_unique<dbc::Client>(std::move(config));
|
|
|
+ connected_ = client_->connect();
|
|
|
+ if (!connected_) {
|
|
|
+ LOG_WARN("StorageClient: connect() to {} returned false", cfg.address);
|
|
|
+ }
|
|
|
+ }
|
|
|
+
|
|
|
+ std::unique_ptr<dbc::Client> client_;
|
|
|
+ bool connected_ = false;
|
|
|
+};
|
|
|
+
|
|
|
+StorageClient::StorageClient(const StorageClientConfig& config)
|
|
|
+ : impl_(std::make_unique<Impl>(config)) {}
|
|
|
+
|
|
|
+StorageClient::~StorageClient() = default;
|
|
|
+StorageClient::StorageClient(StorageClient&&) noexcept = default;
|
|
|
+StorageClient& StorageClient::operator=(StorageClient&&) noexcept = default;
|
|
|
+
|
|
|
+bool StorageClient::isConnected() const {
|
|
|
+ return impl_ && impl_->client_ && impl_->client_->isConnected();
|
|
|
+}
|
|
|
+
|
|
|
+Result<nlohmann::json> StorageClient::get(const std::string& collection, const std::string& id) {
|
|
|
+ auto opt = impl_->client_->get(collection, id);
|
|
|
+ if (!opt) {
|
|
|
+ return Error{6002, "Document not found: " + collection + "/" + id};
|
|
|
+ }
|
|
|
+ return *opt;
|
|
|
+}
|
|
|
+
|
|
|
+Result<std::string> StorageClient::insert(const std::string& collection,
|
|
|
+ const nlohmann::json& data,
|
|
|
+ const std::string& id,
|
|
|
+ int64_t ttl_ms) {
|
|
|
+ auto new_id = impl_->client_->insert(collection, data, id, msToSec(ttl_ms));
|
|
|
+ if (new_id.empty()) {
|
|
|
+ return Error{6000, "Insert failed: " + collection};
|
|
|
+ }
|
|
|
+ return new_id;
|
|
|
+}
|
|
|
+
|
|
|
+Result<int64_t> StorageClient::update(const std::string& collection,
|
|
|
+ const std::string& id,
|
|
|
+ const nlohmann::json& data,
|
|
|
+ int64_t expected_version,
|
|
|
+ bool partial) {
|
|
|
+ if (partial) {
|
|
|
+ uint64_t v = impl_->client_->patch(collection, id, data);
|
|
|
+ if (v == 0) return Error{6000, "Patch failed: " + collection + "/" + id};
|
|
|
+ return static_cast<int64_t>(v);
|
|
|
+ }
|
|
|
+ if (expected_version != 0) {
|
|
|
+ bool ok = impl_->client_->updateIfVersion(
|
|
|
+ collection, id, data, static_cast<uint64_t>(expected_version));
|
|
|
+ if (!ok) return Error{6004, "Version conflict: " + collection + "/" + id};
|
|
|
+ return expected_version + 1;
|
|
|
+ }
|
|
|
+ bool ok = impl_->client_->update(collection, id, data);
|
|
|
+ if (!ok) return Error{6000, "Update failed: " + collection + "/" + id};
|
|
|
+ return 0; // unknown new version when version not requested
|
|
|
+}
|
|
|
+
|
|
|
+Result<void> StorageClient::remove(const std::string& collection,
|
|
|
+ const std::string& id,
|
|
|
+ int64_t expected_version) {
|
|
|
+ (void)expected_version; // upstream remove() is unconditional; document this
|
|
|
+ bool ok = impl_->client_->remove(collection, id);
|
|
|
+ if (!ok) return Error{6002, "Remove failed: " + collection + "/" + id};
|
|
|
+ return {};
|
|
|
+}
|
|
|
+
|
|
|
+Result<QueryResult> StorageClient::query(const std::string& collection) {
|
|
|
+ QueryOptions opts;
|
|
|
+ return query(collection, opts);
|
|
|
+}
|
|
|
+
|
|
|
+Result<QueryResult> StorageClient::query(const std::string& collection, const QueryOptions& opts) {
|
|
|
+ dbc::Client::QueryOptions up;
|
|
|
+ for (const auto& [field, val] : opts.filters) {
|
|
|
+ up.filters.push_back(buildFilter(field, val));
|
|
|
+ }
|
|
|
+ if (!opts.sorts.empty()) {
|
|
|
+ if (opts.sorts.size() > 1) {
|
|
|
+ LOG_WARN("Storage: upstream client supports only one sort field; using first");
|
|
|
+ }
|
|
|
+ up.sortField = opts.sorts.front().first;
|
|
|
+ up.sortDescending = !opts.sorts.front().second;
|
|
|
+ }
|
|
|
+ if (!opts.fields.empty()) {
|
|
|
+ LOG_WARN("Storage: field projection not supported by upstream client; returning all fields");
|
|
|
+ }
|
|
|
+ up.limit = static_cast<uint32_t>(opts.page_size);
|
|
|
+ up.offset = static_cast<uint32_t>(std::max(0, (opts.page - 1) * opts.page_size));
|
|
|
+
|
|
|
+ auto fr = impl_->client_->findWithMetrics(collection, up);
|
|
|
+ QueryResult out;
|
|
|
+ out.documents = std::move(fr.documents);
|
|
|
+ out.total_count = static_cast<int64_t>(fr.totalCount);
|
|
|
+ out.has_more = fr.hasMore;
|
|
|
+ return out;
|
|
|
+}
|
|
|
+
|
|
|
+Result<void> StorageClient::createCollection(const std::string& name,
|
|
|
+ const nlohmann::json& schema,
|
|
|
+ int64_t default_ttl_ms,
|
|
|
+ const VersioningOptions* versioning) {
|
|
|
+ dbc::Client::CollectionOptions co;
|
|
|
+ co.schema = schema;
|
|
|
+ co.defaultTtlSeconds = msToSec(default_ttl_ms);
|
|
|
+ if (versioning) {
|
|
|
+ co.versioning.enabled = versioning->enabled;
|
|
|
+ co.versioning.maxVersions = static_cast<uint32_t>(versioning->max_versions);
|
|
|
+ co.versioning.versionTtlMs = static_cast<uint64_t>(versioning->version_ttl_ms);
|
|
|
+ co.versioning.keepOnDelete = versioning->keep_on_delete;
|
|
|
+ }
|
|
|
+ bool ok = impl_->client_->createCollection(name, co);
|
|
|
+ if (!ok) return Error{6001, "createCollection failed: " + name};
|
|
|
+ return {};
|
|
|
+}
|
|
|
+
|
|
|
+Result<void> StorageClient::dropCollection(const std::string& name) {
|
|
|
+ bool ok = impl_->client_->dropCollection(name);
|
|
|
+ if (!ok) return Error{6001, "dropCollection failed: " + name};
|
|
|
+ return {};
|
|
|
+}
|
|
|
+
|
|
|
+std::vector<std::string> StorageClient::listCollections() {
|
|
|
+ return impl_->client_->listCollections();
|
|
|
+}
|
|
|
+
|
|
|
+Result<nlohmann::json> StorageClient::getVersion(const std::string& collection,
|
|
|
+ const std::string& id,
|
|
|
+ int64_t version) {
|
|
|
+ auto entry = impl_->client_->getDocumentVersion(collection, id, static_cast<uint64_t>(version));
|
|
|
+ if (!entry) return Error{6010, "Version not found"};
|
|
|
+ return entry->data;
|
|
|
+}
|
|
|
+
|
|
|
+Result<VersionListResult> StorageClient::listVersions(const std::string& collection,
|
|
|
+ const std::string& id,
|
|
|
+ int32_t limit,
|
|
|
+ int32_t offset) {
|
|
|
+ auto h = impl_->client_->getVersionHistory(collection, id,
|
|
|
+ static_cast<uint32_t>(limit),
|
|
|
+ static_cast<uint32_t>(offset));
|
|
|
+ VersionListResult out;
|
|
|
+ out.total_count = static_cast<int64_t>(h.totalCount);
|
|
|
+ out.has_more = (offset + static_cast<int32_t>(h.versions.size())) <
|
|
|
+ static_cast<int32_t>(h.totalCount);
|
|
|
+ for (auto& v : h.versions) {
|
|
|
+ DocumentVersionInfo info;
|
|
|
+ info.id = id;
|
|
|
+ info.document_id = id;
|
|
|
+ info.version = static_cast<int64_t>(v.version);
|
|
|
+ info.data = std::move(v.data);
|
|
|
+ info.created_at = static_cast<int64_t>(v.timestamp);
|
|
|
+ out.versions.push_back(std::move(info));
|
|
|
+ }
|
|
|
+ return out;
|
|
|
+}
|
|
|
+
|
|
|
+} // namespace smartbotic::storage
|
|
|
+```
|
|
|
+
|
|
|
+> NOTE: the exact field names on `dbc::Client::CollectionOptions` (`schema`, `defaultTtlSeconds`, `versioning.maxVersions`, etc.) come from `<smartbotic/database/client.hpp>`. If they differ, fix at compile time — the structure is the same, only the spelling may shift. Look at `/usr/include/smartbotic/database/client.hpp` after Task A2 to confirm.
|
|
|
+
|
|
|
+### Task A6: Update `lib/storage` CMake target to link upstream client
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Modify: `/data/smartbotic/CMakeLists.txt:57-75` (the `smartbotic_storage` target block)
|
|
|
+
|
|
|
+- [ ] **Step 1: Replace the storage target block**
|
|
|
+
|
|
|
+Find the current block (around lines 57-75). Replace it with:
|
|
|
+
|
|
|
+```cmake
|
|
|
+# Storage client library — adapter over upstream smartbotic-database client
|
|
|
+add_library(smartbotic_storage STATIC
|
|
|
+ lib/storage/storage_client.cpp
|
|
|
+)
|
|
|
+target_include_directories(smartbotic_storage PUBLIC
|
|
|
+ ${CMAKE_CURRENT_SOURCE_DIR}/lib
|
|
|
+)
|
|
|
+target_link_libraries(smartbotic_storage PUBLIC
|
|
|
+ smartbotic_common
|
|
|
+ smartbotic_logging
|
|
|
+ nlohmann_json::nlohmann_json
|
|
|
+ smartbotic::database::client
|
|
|
+)
|
|
|
+```
|
|
|
+
|
|
|
+Note: `smartbotic_proto` is no longer linked here. The storage client no longer needs the local proto files.
|
|
|
+
|
|
|
+- [ ] **Step 2: Verify build**
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /data/smartbotic
|
|
|
+cmake --build build -j$(nproc) --target smartbotic_storage
|
|
|
+```
|
|
|
+
|
|
|
+Expected: target compiles. If linker complains about a different target name (e.g., `smartbotic-db-client::smartbotic-db-client`), update the link line and Task A3's `find_package` block to match.
|
|
|
+
|
|
|
+### Task A7: Build webserver + runner against the new adapter
|
|
|
+
|
|
|
+**Files:** None (build-only)
|
|
|
+
|
|
|
+- [ ] **Step 1: Build dependent binaries**
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /data/smartbotic
|
|
|
+cmake --build build -j$(nproc) --target smartbotic-webserver smartbotic-runner smartbotic-migrate-nodes
|
|
|
+```
|
|
|
+
|
|
|
+Expected: all three targets build clean. The 10 caller files in `src/webserver/` should compile without changes because the public adapter API is preserved.
|
|
|
+
|
|
|
+If a caller file breaks: it's relying on a method, return type, or struct field that we changed. Fix at the call site (do not re-add the old internal proto). Most likely culprits:
|
|
|
+- Code expecting a specific gRPC `Status` value — handle the `Result<T>` `Error.code` instead.
|
|
|
+- Code reading the now-unused `max_message_size_mb` — leave it; it's still present.
|
|
|
+
|
|
|
+- [ ] **Step 2: Commit Phase A part 1**
|
|
|
+
|
|
|
+```bash
|
|
|
+git add CMakeLists.txt lib/storage/storage_client.hpp lib/storage/storage_client.cpp
|
|
|
+git commit -m "feat(storage): adapt lib/storage to upstream smartbotic-db-client
|
|
|
+
|
|
|
+Replace gRPC stubs over our internal proto::StorageService with calls into
|
|
|
+smartbotic::database::Client (libsmartbotic-db-client). Public API of
|
|
|
+StorageClient is unchanged; PIMPL hides the upstream client. Field projection
|
|
|
+and multi-sort warnings emitted; TTL converted ms->s with round-up."
|
|
|
+```
|
|
|
+
|
|
|
+### Task A8: Remove the `smartbotic-database` executable target
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Modify: `/data/smartbotic/CMakeLists.txt:140-180` (the `add_executable(smartbotic-database ...)` block and any associated target_link_libraries / install rules)
|
|
|
+
|
|
|
+- [ ] **Step 1: Delete the target block**
|
|
|
+
|
|
|
+Locate the section beginning with `add_executable(smartbotic-database`. Delete it entirely, including its `target_link_libraries`, `target_include_directories`, and any `install(TARGETS smartbotic-database ...)` lines.
|
|
|
+
|
|
|
+- [ ] **Step 2: Verify config**
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /data/smartbotic
|
|
|
+cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug
|
|
|
+```
|
|
|
+
|
|
|
+Expected: `smartbotic-database` no longer appears in the target list. (Build dir keeps a stale binary until cleaned — fine.)
|
|
|
+
|
|
|
+- [ ] **Step 3: Remove `src/database/`**
|
|
|
+
|
|
|
+```bash
|
|
|
+git rm -r src/database/
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 4: Build remaining targets**
|
|
|
+
|
|
|
+```bash
|
|
|
+rm -rf build
|
|
|
+cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug
|
|
|
+cmake --build build -j$(nproc)
|
|
|
+```
|
|
|
+
|
|
|
+Expected: `build/smartbotic-webserver`, `build/smartbotic-runner`, `build/smartbotic-migrate-nodes` produced. No `smartbotic-database`.
|
|
|
+
|
|
|
+### Task A9: Remove `proto/storage.proto` and its CMake reference
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Delete: `/data/smartbotic/proto/storage.proto`
|
|
|
+- Modify: `/data/smartbotic/CMakeLists.txt` (the proto file list, likely under a `set(PROTO_FILES ...)` or similar)
|
|
|
+
|
|
|
+- [ ] **Step 1: Find proto list**
|
|
|
+
|
|
|
+```bash
|
|
|
+grep -n 'storage\.proto\|PROTO_FILES\|protobuf_generate' /data/smartbotic/CMakeLists.txt
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 2: Remove `proto/storage.proto` from the proto sources**
|
|
|
+
|
|
|
+Edit the matching line(s) to drop `storage.proto`. The other protos (`common`, `credentials`, `runner`, `workflow`) remain — they are not DB-related.
|
|
|
+
|
|
|
+- [ ] **Step 3: Delete the file**
|
|
|
+
|
|
|
+```bash
|
|
|
+git rm proto/storage.proto
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 4: Reconfigure & rebuild**
|
|
|
+
|
|
|
+```bash
|
|
|
+rm -rf build
|
|
|
+cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug
|
|
|
+cmake --build build -j$(nproc)
|
|
|
+```
|
|
|
+
|
|
|
+Expected: clean build. No references to `storage.grpc.pb.h` or `storage.pb.h` anywhere.
|
|
|
+
|
|
|
+- [ ] **Step 5: Verify**
|
|
|
+
|
|
|
+```bash
|
|
|
+grep -rn 'storage\.grpc\.pb\|storage\.pb\.h\|proto::StorageService\|proto::Document' \
|
|
|
+ /data/smartbotic --include='*.cpp' --include='*.hpp' \
|
|
|
+ --exclude-dir=build --exclude-dir=build-asan
|
|
|
+```
|
|
|
+
|
|
|
+Expected: zero matches.
|
|
|
+
|
|
|
+- [ ] **Step 6: Commit Phase A part 2**
|
|
|
+
|
|
|
+```bash
|
|
|
+git add -A CMakeLists.txt proto src
|
|
|
+git commit -m "feat: drop in-repo smartbotic-database service and storage.proto
|
|
|
+
|
|
|
+Remove the entire src/database/ tree and proto/storage.proto. Going forward
|
|
|
+all storage goes through libsmartbotic-db-client against the upstream
|
|
|
+smartbotic-database daemon, configured via webserver.json / runner.json
|
|
|
+'database_address'."
|
|
|
+```
|
|
|
+
|
|
|
+### Task A10: Native smoke test against a live upstream `smartbotic-database`
|
|
|
+
|
|
|
+**Files:** None (runtime test)
|
|
|
+
|
|
|
+- [ ] **Step 1: Start a temporary upstream DB on an alternate port**
|
|
|
+
|
|
|
+The host already has `smartbotic-database 1.7.1-1` listening on its default port (in use by another project). Run a second instance from the just-installed binary on port `9099`:
|
|
|
+
|
|
|
+```bash
|
|
|
+mkdir -p /tmp/sb-test-db
|
|
|
+/usr/bin/smartbotic-database --grpc-port 9099 --data-dir /tmp/sb-test-db &
|
|
|
+SB_DB_PID=$!
|
|
|
+sleep 2
|
|
|
+```
|
|
|
+
|
|
|
+If the binary's flags differ, point a config file at `--config`. Adjust based on what `/usr/bin/smartbotic-database --help` shows.
|
|
|
+
|
|
|
+- [ ] **Step 2: Run our webserver against it**
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /data/smartbotic
|
|
|
+DATABASE_ADDRESS=localhost:9099 \
|
|
|
+ ./build/smartbotic-webserver config/webserver.json &
|
|
|
+WS_PID=$!
|
|
|
+sleep 3
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 3: Hit health + auth endpoints**
|
|
|
+
|
|
|
+```bash
|
|
|
+curl -s http://localhost:8090/api/v1/healthz
|
|
|
+TOKEN=$(curl -s http://localhost:8090/api/v1/auth/login \
|
|
|
+ -H 'Content-Type: application/json' \
|
|
|
+ -d '{"username":"admin","password":"admin"}' | jq -r '.accessToken')
|
|
|
+test -n "$TOKEN" && echo "auth OK" || echo "auth FAILED"
|
|
|
+```
|
|
|
+
|
|
|
+Expected: `200 OK` from healthz and a non-empty `$TOKEN`.
|
|
|
+
|
|
|
+- [ ] **Step 4: Migrate nodes and list them**
|
|
|
+
|
|
|
+```bash
|
|
|
+curl -s -X POST http://localhost:8090/api/v1/nodes/migrate \
|
|
|
+ -H "Authorization: Bearer $TOKEN" \
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
+ -d '{"nodesPath":"./nodes"}'
|
|
|
+curl -s http://localhost:8090/api/v1/nodes \
|
|
|
+ -H "Authorization: Bearer $TOKEN" | jq '.nodes | length'
|
|
|
+```
|
|
|
+
|
|
|
+Expected: a positive node count (matches what `nodes/` contains).
|
|
|
+
|
|
|
+- [ ] **Step 5: Tear down**
|
|
|
+
|
|
|
+```bash
|
|
|
+kill $WS_PID || true
|
|
|
+kill $SB_DB_PID || true
|
|
|
+rm -rf /tmp/sb-test-db
|
|
|
+```
|
|
|
+
|
|
|
+If any step failed, fix and rerun before continuing to Phase B.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## Phase B — Packaging
|
|
|
+
|
|
|
+### Task B1: Create `VERSION` file
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Create: `/data/smartbotic/VERSION`
|
|
|
+
|
|
|
+- [ ] **Step 1: Write the file**
|
|
|
+
|
|
|
+```bash
|
|
|
+echo "1.0.0" > /data/smartbotic/VERSION
|
|
|
+git add VERSION
|
|
|
+```
|
|
|
+
|
|
|
+(Plain text, single line, no trailing newline issue — `printf` if you want strict.)
|
|
|
+
|
|
|
+### Task B2: Create `packaging/` directory tree
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Create: directory layout under `/data/smartbotic/packaging/`
|
|
|
+
|
|
|
+- [ ] **Step 1: Create directories**
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /data/smartbotic
|
|
|
+mkdir -p packaging/deb/templates \
|
|
|
+ packaging/deb/scripts \
|
|
|
+ packaging/deb/systemd \
|
|
|
+ packaging/deb/config \
|
|
|
+ packaging/scripts
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 2: Copy the SmartBotics APT public key**
|
|
|
+
|
|
|
+```bash
|
|
|
+cp /data/smartbotics-deb-repo/smartbotics-repo.gpg packaging/smartbotics-repo.gpg
|
|
|
+```
|
|
|
+
|
|
|
+### Task B3: Write `packaging/Dockerfile.base`
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Create: `/data/smartbotic/packaging/Dockerfile.base`
|
|
|
+
|
|
|
+- [ ] **Step 1: Write the file**
|
|
|
+
|
|
|
+```dockerfile
|
|
|
+# smartbotic-automation Build Base Image
|
|
|
+# Pre-installed Debian 13 build dependencies for fast iterative builds.
|
|
|
+#
|
|
|
+# Build:
|
|
|
+# docker buildx build -f packaging/Dockerfile.base \
|
|
|
+# --build-arg REPO_PASS=<password> \
|
|
|
+# -t smartbotic-automation-build-base:debian13 .
|
|
|
+
|
|
|
+FROM debian:trixie
|
|
|
+
|
|
|
+ENV DEBIAN_FRONTEND=noninteractive
|
|
|
+
|
|
|
+RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates \
|
|
|
+ && rm -rf /var/lib/apt/lists/*
|
|
|
+
|
|
|
+# SmartBotics APT repository (for libsmartbotic-db-client-dev)
|
|
|
+ARG REPO_USER=callerai
|
|
|
+ARG REPO_PASS
|
|
|
+COPY packaging/smartbotics-repo.gpg /usr/share/keyrings/smartbotics-repo.gpg
|
|
|
+RUN echo "deb [signed-by=/usr/share/keyrings/smartbotics-repo.gpg] https://repository.smartbotics.ai trixie main" \
|
|
|
+ > /etc/apt/sources.list.d/smartbotics.list \
|
|
|
+ && printf "machine repository.smartbotics.ai\nlogin %s\npassword %s\n" \
|
|
|
+ "$REPO_USER" "$REPO_PASS" > /etc/apt/auth.conf.d/smartbotics.conf \
|
|
|
+ && chmod 600 /etc/apt/auth.conf.d/smartbotics.conf
|
|
|
+
|
|
|
+RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
|
+ # Build tools
|
|
|
+ build-essential cmake ninja-build git pkg-config ccache dpkg-dev \
|
|
|
+ # Compression / crypto / uuid
|
|
|
+ libssl-dev zlib1g-dev libbrotli-dev uuid-dev \
|
|
|
+ # gRPC + protobuf
|
|
|
+ protobuf-compiler libprotobuf-dev libgrpc++-dev protobuf-compiler-grpc \
|
|
|
+ # JSON + logging + fmt
|
|
|
+ nlohmann-json3-dev libspdlog-dev libfmt-dev \
|
|
|
+ # bcrypt / curl
|
|
|
+ libbcrypt-dev libcurl4-openssl-dev \
|
|
|
+ # WebUI build (Node.js + npm)
|
|
|
+ nodejs npm \
|
|
|
+ # Upstream smartbotic-database client (the whole point)
|
|
|
+ libsmartbotic-db-client-dev \
|
|
|
+ && rm -rf /var/lib/apt/lists/* \
|
|
|
+ && echo "smartbotic-automation:debian13:$(date -u +%Y-%m-%dT%H:%M:%SZ)" > /etc/smartbotic-automation-build-base
|
|
|
+```
|
|
|
+
|
|
|
+> NOTE: if `libbcrypt-dev` is not packaged in trixie, drop that line and rely on the in-tree `libbcrypt_lib.a` that the project already vendors (visible in your local `build/`).
|
|
|
+
|
|
|
+### Task B4: Write `packaging/Dockerfile.build`
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Create: `/data/smartbotic/packaging/Dockerfile.build`
|
|
|
+
|
|
|
+- [ ] **Step 1: Write the file**
|
|
|
+
|
|
|
+```dockerfile
|
|
|
+# syntax=docker/dockerfile:1.4
|
|
|
+# smartbotic-automation Package Builder
|
|
|
+# Compiles the C++ binaries + WebUI bundle, then assembles a single .deb.
|
|
|
+#
|
|
|
+# Build:
|
|
|
+# docker buildx build -f packaging/Dockerfile.build \
|
|
|
+# --build-arg BASE_IMAGE=smartbotic-automation-build-base:debian13 \
|
|
|
+# --build-arg BUILD_VERSION=1.0.0 \
|
|
|
+# --target packages \
|
|
|
+# --output "type=local,dest=dist/debian13/" .
|
|
|
+
|
|
|
+ARG BASE_IMAGE=debian:trixie
|
|
|
+FROM ${BASE_IMAGE} AS builder
|
|
|
+
|
|
|
+ARG BUILD_VERSION=0.0.0
|
|
|
+ARG BUILD_DEB_REVISION=1
|
|
|
+ARG BUILD_JOBS=8
|
|
|
+ARG BUILD_GIT_COMMIT=unknown
|
|
|
+ENV DEBIAN_FRONTEND=noninteractive
|
|
|
+ENV BUILD_VERSION=${BUILD_VERSION}
|
|
|
+ENV BUILD_DEB_REVISION=${BUILD_DEB_REVISION}
|
|
|
+ENV BUILD_JOBS=${BUILD_JOBS}
|
|
|
+
|
|
|
+ENV CCACHE_DIR=/ccache
|
|
|
+ENV CCACHE_MAXSIZE=5G
|
|
|
+ENV PATH="/usr/lib/ccache:${PATH}"
|
|
|
+
|
|
|
+# Fall back to fresh dep install if the base image isn't our prebuilt one
|
|
|
+RUN if [ ! -f /etc/smartbotic-automation-build-base ]; then \
|
|
|
+ apt-get update && apt-get install -y --no-install-recommends \
|
|
|
+ build-essential cmake ninja-build git pkg-config ccache dpkg-dev \
|
|
|
+ libssl-dev zlib1g-dev libbrotli-dev uuid-dev \
|
|
|
+ protobuf-compiler libprotobuf-dev libgrpc++-dev protobuf-compiler-grpc \
|
|
|
+ nlohmann-json3-dev libspdlog-dev libfmt-dev libcurl4-openssl-dev \
|
|
|
+ nodejs npm libsmartbotic-db-client-dev \
|
|
|
+ && rm -rf /var/lib/apt/lists/*; \
|
|
|
+ else echo "Using pre-built base: $(cat /etc/smartbotic-automation-build-base)"; fi
|
|
|
+
|
|
|
+WORKDIR /build
|
|
|
+
|
|
|
+# WebUI build (cached separately from C++)
|
|
|
+COPY webui/package.json webui/package-lock.json ./webui/
|
|
|
+RUN --mount=type=cache,target=/npm-cache,id=smartbotic-automation-npm-cache \
|
|
|
+ cd webui && npm ci --cache /npm-cache
|
|
|
+COPY VERSION ./
|
|
|
+COPY webui/ ./webui/
|
|
|
+RUN cd webui && npm run build
|
|
|
+
|
|
|
+# C++ build
|
|
|
+COPY CMakeLists.txt ./
|
|
|
+COPY cmake/ ./cmake/
|
|
|
+COPY proto/ ./proto/
|
|
|
+COPY lib/ ./lib/
|
|
|
+COPY src/ ./src/
|
|
|
+COPY config/ ./config/
|
|
|
+COPY nodes/ ./nodes/
|
|
|
+COPY packaging/ ./packaging/
|
|
|
+
|
|
|
+RUN --mount=type=cache,target=/ccache,id=smartbotic-automation-ccache \
|
|
|
+ cmake -B build -G Ninja \
|
|
|
+ -DCMAKE_BUILD_TYPE=Release \
|
|
|
+ -DCMAKE_C_COMPILER_LAUNCHER=ccache \
|
|
|
+ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
|
|
|
+ && cmake --build build --parallel ${BUILD_JOBS} \
|
|
|
+ && (ccache --show-stats || true)
|
|
|
+
|
|
|
+# ----- Package assembly stage -----
|
|
|
+FROM builder AS packager
|
|
|
+RUN mkdir -p /packages
|
|
|
+RUN chmod +x /build/packaging/scripts/create-deb.sh \
|
|
|
+ && OUTPUT_DIR=/packages /build/packaging/scripts/create-deb.sh
|
|
|
+
|
|
|
+# ----- Final export stage -----
|
|
|
+FROM scratch AS packages
|
|
|
+COPY --from=packager /packages/*.deb /
|
|
|
+```
|
|
|
+
|
|
|
+### Task B5: Write `packaging/scripts/create-deb.sh`
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Create: `/data/smartbotic/packaging/scripts/create-deb.sh`
|
|
|
+
|
|
|
+- [ ] **Step 1: Write the file**
|
|
|
+
|
|
|
+```bash
|
|
|
+#!/usr/bin/env bash
|
|
|
+# Assemble the smartbotic-automation .deb tree and run dpkg-deb.
|
|
|
+# Runs INSIDE the Docker build container after `cmake --build`.
|
|
|
+set -euo pipefail
|
|
|
+
|
|
|
+PROJECT_DIR="${PROJECT_DIR:-/build}"
|
|
|
+OUTPUT_DIR="${OUTPUT_DIR:-/packages}"
|
|
|
+VERSION="${BUILD_VERSION:-$(cat "$PROJECT_DIR/VERSION")}"
|
|
|
+REVISION="${BUILD_DEB_REVISION:-1}"
|
|
|
+ARCH="amd64"
|
|
|
+NAME="smartbotic-automation"
|
|
|
+PKG_VERSION="${VERSION}-${REVISION}"
|
|
|
+
|
|
|
+PKG_ROOT="$(mktemp -d)"
|
|
|
+trap "rm -rf $PKG_ROOT" EXIT
|
|
|
+
|
|
|
+# Directory layout inside the .deb
|
|
|
+install -d "$PKG_ROOT/DEBIAN"
|
|
|
+install -d "$PKG_ROOT/usr/bin"
|
|
|
+install -d "$PKG_ROOT/usr/share/$NAME/webui"
|
|
|
+install -d "$PKG_ROOT/usr/share/$NAME/nodes"
|
|
|
+install -d "$PKG_ROOT/etc/$NAME"
|
|
|
+install -d "$PKG_ROOT/lib/systemd/system"
|
|
|
+install -d "$PKG_ROOT/var/lib/$NAME"
|
|
|
+install -d "$PKG_ROOT/var/log/$NAME"
|
|
|
+
|
|
|
+# Binaries
|
|
|
+install -m 0755 "$PROJECT_DIR/build/smartbotic-webserver" "$PKG_ROOT/usr/bin/"
|
|
|
+install -m 0755 "$PROJECT_DIR/build/smartbotic-runner" "$PKG_ROOT/usr/bin/"
|
|
|
+install -m 0755 "$PROJECT_DIR/build/smartbotic-migrate-nodes" "$PKG_ROOT/usr/bin/"
|
|
|
+
|
|
|
+# WebUI bundle
|
|
|
+cp -a "$PROJECT_DIR/webui/dist/." "$PKG_ROOT/usr/share/$NAME/webui/"
|
|
|
+
|
|
|
+# Default workflow nodes
|
|
|
+cp -a "$PROJECT_DIR/nodes/." "$PKG_ROOT/usr/share/$NAME/nodes/"
|
|
|
+
|
|
|
+# Default configs (will become conffiles via DEBIAN/conffiles)
|
|
|
+install -m 0644 "$PROJECT_DIR/packaging/deb/config/webserver.json" "$PKG_ROOT/etc/$NAME/webserver.json"
|
|
|
+install -m 0644 "$PROJECT_DIR/packaging/deb/config/runner.json" "$PKG_ROOT/etc/$NAME/runner.json"
|
|
|
+install -m 0644 "$PROJECT_DIR/packaging/deb/config/env" "$PKG_ROOT/etc/$NAME/env"
|
|
|
+
|
|
|
+# Systemd units
|
|
|
+install -m 0644 "$PROJECT_DIR/packaging/deb/systemd/$NAME-webserver.service" \
|
|
|
+ "$PKG_ROOT/lib/systemd/system/"
|
|
|
+install -m 0644 "$PROJECT_DIR/packaging/deb/systemd/$NAME-runner.service" \
|
|
|
+ "$PKG_ROOT/lib/systemd/system/"
|
|
|
+
|
|
|
+# Maintainer scripts
|
|
|
+for script in preinst postinst prerm postrm; do
|
|
|
+ install -m 0755 "$PROJECT_DIR/packaging/deb/scripts/$script" "$PKG_ROOT/DEBIAN/$script"
|
|
|
+done
|
|
|
+
|
|
|
+# Conffiles
|
|
|
+cat > "$PKG_ROOT/DEBIAN/conffiles" <<EOF
|
|
|
+/etc/$NAME/webserver.json
|
|
|
+/etc/$NAME/runner.json
|
|
|
+/etc/$NAME/env
|
|
|
+EOF
|
|
|
+
|
|
|
+# Render control file from template
|
|
|
+INSTALLED_SIZE=$(du -sk "$PKG_ROOT" --exclude=DEBIAN | cut -f1)
|
|
|
+sed -e "s/{{VERSION}}/${PKG_VERSION}/g" \
|
|
|
+ -e "s/{{INSTALLED_SIZE}}/${INSTALLED_SIZE}/g" \
|
|
|
+ "$PROJECT_DIR/packaging/deb/templates/control.smartbotic-automation" \
|
|
|
+ > "$PKG_ROOT/DEBIAN/control"
|
|
|
+
|
|
|
+# Build the .deb
|
|
|
+mkdir -p "$OUTPUT_DIR"
|
|
|
+DEB_FILE="$OUTPUT_DIR/${NAME}_${PKG_VERSION}_${ARCH}.deb"
|
|
|
+dpkg-deb --build --root-owner-group "$PKG_ROOT" "$DEB_FILE"
|
|
|
+echo "Built: $DEB_FILE"
|
|
|
+ls -lh "$DEB_FILE"
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 2: Make executable**
|
|
|
+
|
|
|
+```bash
|
|
|
+chmod +x /data/smartbotic/packaging/scripts/create-deb.sh
|
|
|
+```
|
|
|
+
|
|
|
+### Task B6: Write `packaging/deb/templates/control.smartbotic-automation`
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Create: `/data/smartbotic/packaging/deb/templates/control.smartbotic-automation`
|
|
|
+
|
|
|
+- [ ] **Step 1: Write the file**
|
|
|
+
|
|
|
+```
|
|
|
+Package: smartbotic-automation
|
|
|
+Version: {{VERSION}}
|
|
|
+Architecture: amd64
|
|
|
+Maintainer: Ferenc Szontagh <ferenc.szontagh@smartbotics.ai>
|
|
|
+Description: SmartBotic Automation — workflow execution platform
|
|
|
+ HTTP/WebSocket API server, workflow runner, and built-in JavaScript node
|
|
|
+ set for the SmartBotic automation platform. Persists workflows, executions,
|
|
|
+ credentials and runtime state in the upstream smartbotic-database daemon
|
|
|
+ (via libsmartbotic-db-client). Includes a React WebUI bundle.
|
|
|
+ Proprietary software by SmartBotics Kft.
|
|
|
+Homepage: https://smartbotics.ai
|
|
|
+Section: contrib/utils
|
|
|
+Priority: optional
|
|
|
+License: proprietary
|
|
|
+Depends: smartbotic-database (>= 1.7.5), libsmartbotic-db-client (>= 1.7.5), libssl3t64, libprotobuf32t64, libgrpc++1.51t64, libspdlog1.15, libfmt10, libuuid1, libcurl4t64, zlib1g, libbrotli1, nodejs
|
|
|
+Installed-Size: {{INSTALLED_SIZE}}
|
|
|
+```
|
|
|
+
|
|
|
+> NOTE: the exact `tXX` suffix on Debian library packages depends on the time64 transition state of trixie at build time. Verify with `apt-cache search libssl3` inside the base image; adjust if the suffix is just `libssl3` or `libssl3t64`. The `libcurl4t64` line may need to drop to `libcurl4` if your trixie snapshot predates the transition.
|
|
|
+
|
|
|
+### Task B7: Write systemd units
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Create: `/data/smartbotic/packaging/deb/systemd/smartbotic-automation-webserver.service`
|
|
|
+- Create: `/data/smartbotic/packaging/deb/systemd/smartbotic-automation-runner.service`
|
|
|
+
|
|
|
+- [ ] **Step 1: Write the webserver unit**
|
|
|
+
|
|
|
+```ini
|
|
|
+[Unit]
|
|
|
+Description=SmartBotic Automation — Webserver (HTTP/WS API)
|
|
|
+Documentation=https://smartbotics.ai
|
|
|
+After=network-online.target smartbotic-database.service
|
|
|
+Wants=network-online.target smartbotic-database.service
|
|
|
+
|
|
|
+[Service]
|
|
|
+Type=simple
|
|
|
+User=smartbotic-automation
|
|
|
+Group=smartbotic-automation
|
|
|
+ExecStart=/usr/bin/smartbotic-webserver --config /etc/smartbotic-automation/webserver.json
|
|
|
+WorkingDirectory=/var/lib/smartbotic-automation
|
|
|
+EnvironmentFile=-/etc/smartbotic-automation/env
|
|
|
+Restart=on-failure
|
|
|
+RestartSec=5
|
|
|
+TimeoutStopSec=15
|
|
|
+StandardOutput=journal
|
|
|
+StandardError=journal
|
|
|
+
|
|
|
+[Install]
|
|
|
+WantedBy=multi-user.target
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 2: Write the runner unit**
|
|
|
+
|
|
|
+```ini
|
|
|
+[Unit]
|
|
|
+Description=SmartBotic Automation — Runner (workflow execution)
|
|
|
+Documentation=https://smartbotics.ai
|
|
|
+After=network-online.target smartbotic-automation-webserver.service
|
|
|
+Wants=network-online.target smartbotic-automation-webserver.service
|
|
|
+
|
|
|
+[Service]
|
|
|
+Type=simple
|
|
|
+User=smartbotic-automation
|
|
|
+Group=smartbotic-automation
|
|
|
+ExecStart=/usr/bin/smartbotic-runner --config /etc/smartbotic-automation/runner.json
|
|
|
+WorkingDirectory=/var/lib/smartbotic-automation
|
|
|
+EnvironmentFile=-/etc/smartbotic-automation/env
|
|
|
+Restart=on-failure
|
|
|
+RestartSec=5
|
|
|
+TimeoutStopSec=15
|
|
|
+StandardOutput=journal
|
|
|
+StandardError=journal
|
|
|
+
|
|
|
+[Install]
|
|
|
+WantedBy=multi-user.target
|
|
|
+```
|
|
|
+
|
|
|
+### Task B8: Write default config files
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Create: `/data/smartbotic/packaging/deb/config/webserver.json`
|
|
|
+- Create: `/data/smartbotic/packaging/deb/config/runner.json`
|
|
|
+- Create: `/data/smartbotic/packaging/deb/config/env`
|
|
|
+
|
|
|
+- [ ] **Step 1: Write `webserver.json`**
|
|
|
+
|
|
|
+```json
|
|
|
+{
|
|
|
+ "http_port": 8090,
|
|
|
+ "node_sync_port": 9012,
|
|
|
+ "credential_service_port": 9013,
|
|
|
+ "static_files_path": "${WEBUI_PATH:/usr/share/smartbotic-automation/webui}",
|
|
|
+ "database_address": "${DATABASE_ADDRESS:localhost:9010}",
|
|
|
+ "runners": {
|
|
|
+ "load_balancing": "least-connections",
|
|
|
+ "heartbeat_timeout_sec": 30,
|
|
|
+ "offline_removal_sec": 60
|
|
|
+ },
|
|
|
+ "auth": {
|
|
|
+ "jwt_secret": "${JWT_SECRET:CHANGE_ME_BEFORE_PRODUCTION}",
|
|
|
+ "access_token_lifetime_sec": 900,
|
|
|
+ "refresh_token_lifetime_sec": 604800
|
|
|
+ },
|
|
|
+ "credentials": {
|
|
|
+ "master_key": "${CREDENTIALS_MASTER_KEY:CHANGE_ME_BEFORE_PRODUCTION}",
|
|
|
+ "pbkdf2_iterations": 100000
|
|
|
+ },
|
|
|
+ "cors": {
|
|
|
+ "enabled": true,
|
|
|
+ "origins": ["*"]
|
|
|
+ },
|
|
|
+ "logging": {
|
|
|
+ "level": "${LOG_LEVEL:info}",
|
|
|
+ "file": "${LOG_FILE:/var/log/smartbotic-automation/webserver.log}"
|
|
|
+ }
|
|
|
+}
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 2: Write `runner.json`**
|
|
|
+
|
|
|
+```json
|
|
|
+{
|
|
|
+ "grpc_port": 9011,
|
|
|
+ "runner_id": "${RUNNER_ID:runner-1}",
|
|
|
+ "webserver_address": "${WEBSERVER_ADDRESS:localhost:8090}",
|
|
|
+ "node_sync_address": "${NODE_SYNC_ADDRESS:localhost:9012}",
|
|
|
+ "credential_service_address": "${CREDENTIAL_SERVICE_ADDRESS:localhost:9013}",
|
|
|
+ "database_address": "${DATABASE_ADDRESS:localhost:9010}",
|
|
|
+ "max_message_size_mb": 64,
|
|
|
+ "node_sync": {
|
|
|
+ "enabled": true,
|
|
|
+ "reconnect_interval_ms": 5000
|
|
|
+ },
|
|
|
+ "registration": {
|
|
|
+ "heartbeat_interval_sec": 10,
|
|
|
+ "max_concurrent_executions": 10
|
|
|
+ },
|
|
|
+ "execution": {
|
|
|
+ "default_timeout_ms": 60000,
|
|
|
+ "max_memory_per_script_mb": 64
|
|
|
+ },
|
|
|
+ "logging": {
|
|
|
+ "level": "${LOG_LEVEL:info}",
|
|
|
+ "file": "${LOG_FILE:/var/log/smartbotic-automation/runner.log}"
|
|
|
+ }
|
|
|
+}
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 3: Write `env` (empty by default; admins set `DATABASE_ADDRESS`, secrets here)**
|
|
|
+
|
|
|
+```
|
|
|
+# /etc/smartbotic-automation/env
|
|
|
+# Override config values with environment variables here.
|
|
|
+# Example:
|
|
|
+# DATABASE_ADDRESS=localhost:9010
|
|
|
+# JWT_SECRET=...
|
|
|
+# CREDENTIALS_MASTER_KEY=...
|
|
|
+# LOG_LEVEL=info
|
|
|
+```
|
|
|
+
|
|
|
+### Task B9: Write maintainer scripts
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Create: `/data/smartbotic/packaging/deb/scripts/preinst`
|
|
|
+- Create: `/data/smartbotic/packaging/deb/scripts/postinst`
|
|
|
+- Create: `/data/smartbotic/packaging/deb/scripts/prerm`
|
|
|
+- Create: `/data/smartbotic/packaging/deb/scripts/postrm`
|
|
|
+
|
|
|
+- [ ] **Step 1: Write `preinst`**
|
|
|
+
|
|
|
+```bash
|
|
|
+#!/bin/bash
|
|
|
+# preinst for smartbotic-automation — minimal pre-install checks
|
|
|
+set -e
|
|
|
+
|
|
|
+case "$1" in
|
|
|
+ install|upgrade)
|
|
|
+ # No-op; postinst handles user/group + dirs
|
|
|
+ ;;
|
|
|
+esac
|
|
|
+
|
|
|
+exit 0
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 2: Write `postinst`**
|
|
|
+
|
|
|
+```bash
|
|
|
+#!/bin/bash
|
|
|
+# postinst for smartbotic-automation
|
|
|
+set -e
|
|
|
+
|
|
|
+SM_USER="smartbotic-automation"
|
|
|
+SM_GROUP="smartbotic-automation"
|
|
|
+SM_HOME="/home/smartbotic-automation"
|
|
|
+SERVICES="smartbotic-automation-webserver smartbotic-automation-runner"
|
|
|
+
|
|
|
+case "$1" in
|
|
|
+ configure)
|
|
|
+ # System group + user (idempotent)
|
|
|
+ if ! getent group "$SM_GROUP" >/dev/null 2>&1; then
|
|
|
+ addgroup --system "$SM_GROUP"
|
|
|
+ fi
|
|
|
+ if ! getent passwd "$SM_USER" >/dev/null 2>&1; then
|
|
|
+ adduser --system --ingroup "$SM_GROUP" --home "$SM_HOME" \
|
|
|
+ --shell /usr/sbin/nologin \
|
|
|
+ --gecos "SmartBotic Automation Service User" "$SM_USER"
|
|
|
+ fi
|
|
|
+
|
|
|
+ install -d -o "$SM_USER" -g "$SM_GROUP" -m 0750 "$SM_HOME"
|
|
|
+ install -d -o "$SM_USER" -g "$SM_GROUP" -m 0750 /var/lib/smartbotic-automation
|
|
|
+ install -d -o "$SM_USER" -g "$SM_GROUP" -m 0750 /var/log/smartbotic-automation
|
|
|
+ install -d -o root -g "$SM_GROUP" -m 0750 /etc/smartbotic-automation
|
|
|
+
|
|
|
+ # systemd
|
|
|
+ if [ -d /run/systemd/system ]; then
|
|
|
+ systemctl daemon-reload
|
|
|
+ fi
|
|
|
+
|
|
|
+ if [ -z "$2" ]; then
|
|
|
+ # First install: enable but do NOT start (admin must configure
|
|
|
+ # database_address first; the upstream DB port is deployment-specific).
|
|
|
+ for svc in $SERVICES; do
|
|
|
+ deb-systemd-helper enable "$svc.service" >/dev/null || true
|
|
|
+ done
|
|
|
+ echo
|
|
|
+ echo "smartbotic-automation: services are enabled but not started."
|
|
|
+ echo " 1. Edit /etc/smartbotic-automation/webserver.json (set database_address)"
|
|
|
+ echo " 2. Edit /etc/smartbotic-automation/runner.json (set database_address)"
|
|
|
+ echo " 3. systemctl start smartbotic-automation-webserver smartbotic-automation-runner"
|
|
|
+ echo
|
|
|
+ else
|
|
|
+ # Upgrade path: try-restart any active units
|
|
|
+ for svc in $SERVICES; do
|
|
|
+ deb-systemd-invoke try-restart "$svc.service" >/dev/null 2>&1 || true
|
|
|
+ done
|
|
|
+ fi
|
|
|
+ ;;
|
|
|
+esac
|
|
|
+
|
|
|
+#DEBHELPER#
|
|
|
+exit 0
|
|
|
+```
|
|
|
+
|
|
|
+> NOTE: the `#DEBHELPER#` marker is harmless if we're not using debhelper — `dpkg-deb --build` ignores it. Leaving it in keeps the script compatible if we later switch to a `debian/` directory layout.
|
|
|
+
|
|
|
+- [ ] **Step 3: Write `prerm`**
|
|
|
+
|
|
|
+```bash
|
|
|
+#!/bin/bash
|
|
|
+# prerm for smartbotic-automation
|
|
|
+set -e
|
|
|
+
|
|
|
+SERVICES="smartbotic-automation-webserver smartbotic-automation-runner"
|
|
|
+
|
|
|
+case "$1" in
|
|
|
+ remove|deconfigure)
|
|
|
+ if [ -d /run/systemd/system ]; then
|
|
|
+ for svc in $SERVICES; do
|
|
|
+ deb-systemd-invoke stop "$svc.service" >/dev/null 2>&1 || true
|
|
|
+ deb-systemd-helper disable "$svc.service" >/dev/null 2>&1 || true
|
|
|
+ done
|
|
|
+ fi
|
|
|
+ ;;
|
|
|
+ upgrade|failed-upgrade)
|
|
|
+ if [ -d /run/systemd/system ]; then
|
|
|
+ for svc in $SERVICES; do
|
|
|
+ deb-systemd-invoke stop "$svc.service" >/dev/null 2>&1 || true
|
|
|
+ done
|
|
|
+ fi
|
|
|
+ ;;
|
|
|
+esac
|
|
|
+
|
|
|
+exit 0
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 4: Write `postrm`**
|
|
|
+
|
|
|
+```bash
|
|
|
+#!/bin/bash
|
|
|
+# postrm for smartbotic-automation
|
|
|
+set -e
|
|
|
+
|
|
|
+SM_USER="smartbotic-automation"
|
|
|
+SM_GROUP="smartbotic-automation"
|
|
|
+
|
|
|
+case "$1" in
|
|
|
+ purge)
|
|
|
+ rm -rf /var/lib/smartbotic-automation /var/log/smartbotic-automation
|
|
|
+ if getent passwd "$SM_USER" >/dev/null 2>&1; then
|
|
|
+ deluser --system "$SM_USER" >/dev/null 2>&1 || true
|
|
|
+ fi
|
|
|
+ if getent group "$SM_GROUP" >/dev/null 2>&1; then
|
|
|
+ delgroup --system "$SM_GROUP" >/dev/null 2>&1 || true
|
|
|
+ fi
|
|
|
+ ;;
|
|
|
+ remove|upgrade|failed-upgrade|abort-install|abort-upgrade|disappear)
|
|
|
+ if [ -d /run/systemd/system ]; then
|
|
|
+ systemctl daemon-reload >/dev/null 2>&1 || true
|
|
|
+ fi
|
|
|
+ ;;
|
|
|
+esac
|
|
|
+
|
|
|
+exit 0
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 5: Make all scripts executable**
|
|
|
+
|
|
|
+```bash
|
|
|
+chmod +x /data/smartbotic/packaging/deb/scripts/{preinst,postinst,prerm,postrm}
|
|
|
+```
|
|
|
+
|
|
|
+### Task B10: Write `packaging/build.sh`
|
|
|
+
|
|
|
+**Files:**
|
|
|
+- Create: `/data/smartbotic/packaging/build.sh`
|
|
|
+
|
|
|
+- [ ] **Step 1: Write the file**
|
|
|
+
|
|
|
+```bash
|
|
|
+#!/usr/bin/env bash
|
|
|
+# smartbotic-automation production build orchestrator.
|
|
|
+# Modeled on /data/shadowman-cpp/packaging/build.sh.
|
|
|
+#
|
|
|
+# Usage:
|
|
|
+# packaging/build.sh # Build with version from VERSION
|
|
|
+# packaging/build.sh 1.2.0 # Override version
|
|
|
+# packaging/build.sh --rebuild-base # Force rebuild of base image
|
|
|
+# packaging/build.sh --sync --suite trixie # Build + publish to APT repo
|
|
|
+set -euo pipefail
|
|
|
+
|
|
|
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
|
+PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
|
+cd "$PROJECT_DIR"
|
|
|
+
|
|
|
+BASE_IMAGE_NAME="smartbotic-automation-build-base:debian13"
|
|
|
+OUTPUT_DIR="${PROJECT_DIR}/dist/debian13"
|
|
|
+BUILD_JOBS="${BUILD_JOBS:-$(nproc)}"
|
|
|
+DEB_REPO_DIR="${DEB_REPO_DIR:-/data/smartbotics-deb-repo}"
|
|
|
+
|
|
|
+RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; BLUE='\033[0;34m'; NC='\033[0m'
|
|
|
+log_info() { echo -e "${BLUE}[INFO]${NC} $1"; }
|
|
|
+log_success() { echo -e "${GREEN}[OK]${NC} $1"; }
|
|
|
+log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
|
|
|
+log_error() { echo -e "${RED}[ERROR]${NC} $1"; }
|
|
|
+
|
|
|
+REBUILD_BASE=false
|
|
|
+NO_CACHE=false
|
|
|
+BUILD_REPO=false
|
|
|
+SYNC_REPO=false
|
|
|
+REPO_SUITE=""
|
|
|
+VERSION=""
|
|
|
+DEB_REVISION="1"
|
|
|
+
|
|
|
+while [[ $# -gt 0 ]]; do
|
|
|
+ case $1 in
|
|
|
+ --rebuild-base) REBUILD_BASE=true; shift ;;
|
|
|
+ --no-cache) NO_CACHE=true; shift ;;
|
|
|
+ --deb-revision) DEB_REVISION="$2"; shift 2 ;;
|
|
|
+ --repo) BUILD_REPO=true; shift ;;
|
|
|
+ --suite) REPO_SUITE="$2"; shift 2 ;;
|
|
|
+ --sync) SYNC_REPO=true; BUILD_REPO=true; shift ;;
|
|
|
+ --help|-h)
|
|
|
+ cat <<EOF
|
|
|
+smartbotic-automation Production Build Script
|
|
|
+
|
|
|
+Usage: $0 [OPTIONS] [VERSION]
|
|
|
+
|
|
|
+Options:
|
|
|
+ --rebuild-base Force rebuild the base image
|
|
|
+ --no-cache Build without using BuildKit caches
|
|
|
+ --deb-revision N Debian revision suffix (default: 1)
|
|
|
+ --repo Generate APT repo from built packages
|
|
|
+ --suite NAME Distribution suite for repo (e.g., trixie)
|
|
|
+ --sync Generate repo and sync to repository.smartbotics.ai
|
|
|
+
|
|
|
+Environment:
|
|
|
+ BUILD_JOBS Parallel build jobs (default: nproc)
|
|
|
+ DEB_REPO_DIR Path to smartbotics-deb-repo (default: /data/smartbotics-deb-repo)
|
|
|
+EOF
|
|
|
+ exit 0 ;;
|
|
|
+ *) VERSION="$1"; shift ;;
|
|
|
+ esac
|
|
|
+done
|
|
|
+
|
|
|
+if [ -z "$VERSION" ]; then
|
|
|
+ VERSION="$(tr -d '[:space:]' < VERSION 2>/dev/null || true)"
|
|
|
+ if [ -z "$VERSION" ]; then
|
|
|
+ log_error "Could not read version from VERSION file"; exit 1
|
|
|
+ fi
|
|
|
+fi
|
|
|
+
|
|
|
+if ! command -v docker &>/dev/null; then
|
|
|
+ log_error "Docker not found in PATH"; exit 1
|
|
|
+fi
|
|
|
+
|
|
|
+base_image_exists() { docker image inspect "$BASE_IMAGE_NAME" &>/dev/null; }
|
|
|
+build_base_image() {
|
|
|
+ log_info "Building base image: $BASE_IMAGE_NAME"
|
|
|
+ local repo_pass=""
|
|
|
+ if [ -f "${DEB_REPO_DIR}/.env" ]; then
|
|
|
+ repo_pass=$(grep -oP 'RepoPass:\s*\K.*' "${DEB_REPO_DIR}/.env" | tr -d '[:space:]')
|
|
|
+ fi
|
|
|
+ if [ -z "$repo_pass" ]; then
|
|
|
+ log_error "Could not read RepoPass from ${DEB_REPO_DIR}/.env"; exit 1
|
|
|
+ fi
|
|
|
+ docker buildx build \
|
|
|
+ -f packaging/Dockerfile.base \
|
|
|
+ --build-arg REPO_PASS="$repo_pass" \
|
|
|
+ -t "$BASE_IMAGE_NAME" \
|
|
|
+ .
|
|
|
+ log_success "Base image built"
|
|
|
+}
|
|
|
+
|
|
|
+if $REBUILD_BASE; then
|
|
|
+ build_base_image
|
|
|
+elif base_image_exists; then
|
|
|
+ log_success "Base image cached: $BASE_IMAGE_NAME"
|
|
|
+else
|
|
|
+ build_base_image
|
|
|
+fi
|
|
|
+
|
|
|
+GIT_COMMIT=$(git rev-parse --short HEAD 2>/dev/null || echo unknown)
|
|
|
+
|
|
|
+rm -rf "$OUTPUT_DIR"; mkdir -p "$OUTPUT_DIR"
|
|
|
+
|
|
|
+BUILD_ARGS=(
|
|
|
+ -f packaging/Dockerfile.build
|
|
|
+ --build-arg BASE_IMAGE="$BASE_IMAGE_NAME"
|
|
|
+ --build-arg BUILD_VERSION="$VERSION"
|
|
|
+ --build-arg BUILD_DEB_REVISION="$DEB_REVISION"
|
|
|
+ --build-arg BUILD_GIT_COMMIT="$GIT_COMMIT"
|
|
|
+ --build-arg BUILD_JOBS="$BUILD_JOBS"
|
|
|
+ --target packages
|
|
|
+ --output "type=local,dest=$OUTPUT_DIR"
|
|
|
+)
|
|
|
+$NO_CACHE && BUILD_ARGS+=(--no-cache)
|
|
|
+
|
|
|
+log_info "Building smartbotic-automation ${VERSION}-${DEB_REVISION} (commit ${GIT_COMMIT})"
|
|
|
+docker buildx build "${BUILD_ARGS[@]}" .
|
|
|
+
|
|
|
+log_success "Built packages:"
|
|
|
+ls -lh "$OUTPUT_DIR"/*.deb
|
|
|
+
|
|
|
+if $BUILD_REPO; then
|
|
|
+ if [ -z "$REPO_SUITE" ]; then
|
|
|
+ log_error "--suite is required with --repo"; exit 1
|
|
|
+ fi
|
|
|
+ if [ ! -d "$DEB_REPO_DIR" ]; then
|
|
|
+ log_error "Repository dir not found: $DEB_REPO_DIR"; exit 1
|
|
|
+ fi
|
|
|
+ "$DEB_REPO_DIR/scripts/add-packages.sh" "$OUTPUT_DIR"
|
|
|
+ "$DEB_REPO_DIR/scripts/create-repo.sh" --suite "$REPO_SUITE"
|
|
|
+fi
|
|
|
+
|
|
|
+if $SYNC_REPO; then
|
|
|
+ "$DEB_REPO_DIR/scripts/sync-repo.sh"
|
|
|
+fi
|
|
|
+
|
|
|
+log_success "Done."
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 2: Make executable**
|
|
|
+
|
|
|
+```bash
|
|
|
+chmod +x /data/smartbotic/packaging/build.sh
|
|
|
+```
|
|
|
+
|
|
|
+### Task B11: Build the base image
|
|
|
+
|
|
|
+**Files:** None (Docker)
|
|
|
+
|
|
|
+- [ ] **Step 1: Run base build**
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /data/smartbotic
|
|
|
+./packaging/build.sh --rebuild-base 2>&1 | tail -40
|
|
|
+```
|
|
|
+
|
|
|
+Expected: ends with `[OK] Base image built`. Verify with:
|
|
|
+
|
|
|
+```bash
|
|
|
+docker image inspect smartbotic-automation-build-base:debian13 | jq '.[0].Size' \
|
|
|
+ | awk '{ printf "Base image size: %.0f MB\n", $1/1024/1024 }'
|
|
|
+```
|
|
|
+
|
|
|
+If apt fails to find `libsmartbotic-db-client-dev`, check that:
|
|
|
+- The repo password from `.env` resolved (`RepoPass:` line is present)
|
|
|
+- The `trixie` suite has been published in `/data/smartbotics-deb-repo/repo/dists/trixie/`
|
|
|
+
|
|
|
+### Task B12: Build the `.deb`
|
|
|
+
|
|
|
+**Files:** None (Docker output to `dist/debian13/`)
|
|
|
+
|
|
|
+- [ ] **Step 1: Build**
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /data/smartbotic
|
|
|
+./packaging/build.sh 2>&1 | tail -30
|
|
|
+```
|
|
|
+
|
|
|
+Expected: a single `smartbotic-automation_1.0.0-1_amd64.deb` in `dist/debian13/`.
|
|
|
+
|
|
|
+- [ ] **Step 2: Inspect the package metadata**
|
|
|
+
|
|
|
+```bash
|
|
|
+dpkg-deb -I dist/debian13/smartbotic-automation_1.0.0-1_amd64.deb
|
|
|
+```
|
|
|
+
|
|
|
+Expected: lists Maintainer, Depends including `smartbotic-database (>= 1.7.5)` and `libsmartbotic-db-client (>= 1.7.5)`, sane Installed-Size.
|
|
|
+
|
|
|
+- [ ] **Step 3: Inspect file list**
|
|
|
+
|
|
|
+```bash
|
|
|
+dpkg-deb -c dist/debian13/smartbotic-automation_1.0.0-1_amd64.deb | head -40
|
|
|
+```
|
|
|
+
|
|
|
+Expected: `/usr/bin/smartbotic-{webserver,runner,migrate-nodes}`, `/usr/share/smartbotic-automation/webui/...`, `/etc/smartbotic-automation/...`, systemd units, owned `root:root` mode-correct.
|
|
|
+
|
|
|
+- [ ] **Step 4: Lint with `lintian` (best effort)**
|
|
|
+
|
|
|
+```bash
|
|
|
+lintian dist/debian13/smartbotic-automation_1.0.0-1_amd64.deb 2>&1 | head -20 || true
|
|
|
+```
|
|
|
+
|
|
|
+`lintian` warnings are not fatal — fix any errors (E:) and any warnings about systemd units, postinst exit codes, or missing depends. Skip purely cosmetic warnings (no manpage, etc.).
|
|
|
+
|
|
|
+### Task B13: Commit Phase B
|
|
|
+
|
|
|
+**Files:** None (git)
|
|
|
+
|
|
|
+- [ ] **Step 1: Stage and commit**
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /data/smartbotic
|
|
|
+git add VERSION packaging/
|
|
|
+git commit -m "build: add Debian 13 packaging for smartbotic-automation
|
|
|
+
|
|
|
+Single .deb mirroring shadowman-cpp packaging style. Two-stage Docker build
|
|
|
+(Dockerfile.base + Dockerfile.build) targeting trixie. build.sh wraps
|
|
|
+buildx and integrates with /data/smartbotics-deb-repo for --repo / --sync.
|
|
|
+Installs binaries to /usr/bin, configs to /etc/smartbotic-automation,
|
|
|
+data to /var/lib/smartbotic-automation, with systemd units for webserver
|
|
|
+and runner."
|
|
|
+```
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## Phase C — Smoke Test
|
|
|
+
|
|
|
+### Task C1: Spin up a fresh Debian 13 container
|
|
|
+
|
|
|
+**Files:** None (host)
|
|
|
+
|
|
|
+- [ ] **Step 1: Start container with the .deb mounted**
|
|
|
+
|
|
|
+```bash
|
|
|
+docker run -it --rm \
|
|
|
+ -v /data/smartbotic/dist/debian13:/debs:ro \
|
|
|
+ -v /data/smartbotics-deb-repo/smartbotics-repo.gpg:/etc/apt/keyrings/sb.gpg:ro \
|
|
|
+ --name sb-auto-test \
|
|
|
+ debian:trixie bash
|
|
|
+```
|
|
|
+
|
|
|
+Inside the container, the next steps run.
|
|
|
+
|
|
|
+- [ ] **Step 2: Configure the SmartBotics APT repo (for the `smartbotic-database` dep)**
|
|
|
+
|
|
|
+```bash
|
|
|
+apt-get update && apt-get install -y curl ca-certificates
|
|
|
+echo "deb [signed-by=/etc/apt/keyrings/sb.gpg] https://repository.smartbotics.ai trixie main" \
|
|
|
+ > /etc/apt/sources.list.d/smartbotics.list
|
|
|
+# Use the repo creds from /data/smartbotics-deb-repo/.env
|
|
|
+printf "machine repository.smartbotics.ai\nlogin %s\npassword %s\n" \
|
|
|
+ callerai "$(grep -oP 'RepoPass:\s*\K.*' /etc/apt/auth.conf.d/smartbotics.conf || echo)" \
|
|
|
+ > /etc/apt/auth.conf.d/smartbotics.conf
|
|
|
+apt-get update
|
|
|
+```
|
|
|
+
|
|
|
+(Adjust if you mount the password file differently.)
|
|
|
+
|
|
|
+- [ ] **Step 3: Install the upstream DB then our package**
|
|
|
+
|
|
|
+```bash
|
|
|
+apt-get install -y /debs/smartbotic-automation_1.0.0-1_amd64.deb
|
|
|
+```
|
|
|
+
|
|
|
+Expected: APT pulls `smartbotic-database`, `libsmartbotic-db-client`, `nodejs`, etc. as dependencies. Postinst runs, prints the "configure database_address" message, exits 0.
|
|
|
+
|
|
|
+### Task C2: Configure and start
|
|
|
+
|
|
|
+**Files:** Inside container — `/etc/smartbotic-automation/{webserver,runner}.json`
|
|
|
+
|
|
|
+- [ ] **Step 1: Start upstream `smartbotic-database`**
|
|
|
+
|
|
|
+```bash
|
|
|
+systemctl start smartbotic-database.service
|
|
|
+systemctl status smartbotic-database --no-pager | head -10
|
|
|
+ss -tlnp | grep smartbotic-database # confirm a port
|
|
|
+```
|
|
|
+
|
|
|
+Note the port (likely `9010`). If different, update both config files' `database_address`.
|
|
|
+
|
|
|
+- [ ] **Step 2: Start our services**
|
|
|
+
|
|
|
+```bash
|
|
|
+systemctl start smartbotic-automation-webserver smartbotic-automation-runner
|
|
|
+systemctl status smartbotic-automation-webserver --no-pager | head -15
|
|
|
+systemctl status smartbotic-automation-runner --no-pager | head -15
|
|
|
+```
|
|
|
+
|
|
|
+Expected: both `active (running)`. If the runner exits because it can't reach the webserver, restart in order with a `sleep 2` between.
|
|
|
+
|
|
|
+### Task C3: Functional smoke check
|
|
|
+
|
|
|
+**Files:** None
|
|
|
+
|
|
|
+- [ ] **Step 1: Health + auth**
|
|
|
+
|
|
|
+```bash
|
|
|
+curl -s http://localhost:8090/api/v1/healthz
|
|
|
+TOKEN=$(curl -s http://localhost:8090/api/v1/auth/login \
|
|
|
+ -H 'Content-Type: application/json' \
|
|
|
+ -d '{"username":"admin","password":"admin"}' | jq -r '.accessToken')
|
|
|
+echo "TOKEN length: ${#TOKEN}"
|
|
|
+```
|
|
|
+
|
|
|
+Expected: healthz returns 200; token is non-empty (>50 chars).
|
|
|
+
|
|
|
+- [ ] **Step 2: Migrate nodes**
|
|
|
+
|
|
|
+```bash
|
|
|
+curl -s -X POST http://localhost:8090/api/v1/nodes/migrate \
|
|
|
+ -H "Authorization: Bearer $TOKEN" \
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
+ -d '{"nodesPath":"/usr/share/smartbotic-automation/nodes"}' | jq
|
|
|
+curl -s http://localhost:8090/api/v1/nodes \
|
|
|
+ -H "Authorization: Bearer $TOKEN" | jq '.nodes | length'
|
|
|
+```
|
|
|
+
|
|
|
+Expected: a positive node count.
|
|
|
+
|
|
|
+- [ ] **Step 3: Verify WebUI is served**
|
|
|
+
|
|
|
+```bash
|
|
|
+curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8090/
|
|
|
+curl -s http://localhost:8090/ | grep -oE '<title>[^<]*</title>' | head -1
|
|
|
+```
|
|
|
+
|
|
|
+Expected: `200` and a `<title>` containing "SmartBotic" or similar.
|
|
|
+
|
|
|
+- [ ] **Step 4: Stop, then test purge**
|
|
|
+
|
|
|
+```bash
|
|
|
+systemctl stop smartbotic-automation-webserver smartbotic-automation-runner
|
|
|
+apt-get purge -y smartbotic-automation
|
|
|
+test -d /var/lib/smartbotic-automation && echo "FAIL: data dir still present" || echo "OK: data dir purged"
|
|
|
+test -d /etc/smartbotic-automation && echo "INFO: config dir kept (conffile dpkg behavior)" || true
|
|
|
+getent passwd smartbotic-automation && echo "FAIL: user still present" || echo "OK: user purged"
|
|
|
+```
|
|
|
+
|
|
|
+Expected: data dir gone, user gone, config dir may remain (dpkg conffile semantics).
|
|
|
+
|
|
|
+- [ ] **Step 5: Exit container**
|
|
|
+
|
|
|
+```bash
|
|
|
+exit
|
|
|
+```
|
|
|
+
|
|
|
+If any step failed, fix in source, rebuild via Phase B Task B12, retry Phase C from scratch.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## Phase D — Repository Publish
|
|
|
+
|
|
|
+### Task D1: Build with `--sync`
|
|
|
+
|
|
|
+**Files:** None
|
|
|
+
|
|
|
+- [ ] **Step 1: Confirm repo-side state is clean**
|
|
|
+
|
|
|
+```bash
|
|
|
+ls -lh /data/smartbotic/dist/debian13/*.deb
|
|
|
+ls /data/smartbotics-deb-repo/dist/ 2>/dev/null
|
|
|
+```
|
|
|
+
|
|
|
+If `/data/smartbotics-deb-repo/dist/` already has a stale `smartbotic-automation_*.deb`, remove it first:
|
|
|
+
|
|
|
+```bash
|
|
|
+rm -f /data/smartbotics-deb-repo/dist/smartbotic-automation_*.deb
|
|
|
+```
|
|
|
+
|
|
|
+- [ ] **Step 2: Run sync**
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /data/smartbotic
|
|
|
+./packaging/build.sh --sync --suite trixie 2>&1 | tail -40
|
|
|
+```
|
|
|
+
|
|
|
+Expected: build completes, `add-packages.sh` copies into `dist/`, `create-repo.sh --suite trixie` regenerates `repo/dists/trixie/`, `sync-repo.sh` uploads to repository.smartbotics.ai.
|
|
|
+
|
|
|
+### Task D2: Verify from a clean trixie container
|
|
|
+
|
|
|
+**Files:** None
|
|
|
+
|
|
|
+- [ ] **Step 1: Pull and install over the network**
|
|
|
+
|
|
|
+```bash
|
|
|
+docker run --rm debian:trixie bash -c '
|
|
|
+ apt-get update && apt-get install -y curl ca-certificates gnupg
|
|
|
+ curl -fsSL https://repository.smartbotics.ai/smartbotics-repo.gpg \
|
|
|
+ | gpg --dearmor -o /usr/share/keyrings/smartbotics-repo.gpg
|
|
|
+ echo "deb [signed-by=/usr/share/keyrings/smartbotics-repo.gpg] https://repository.smartbotics.ai trixie main" \
|
|
|
+ > /etc/apt/sources.list.d/smartbotics.list
|
|
|
+ apt-get update 2>&1 | grep -E "smartbotic|Err|^E:" || true
|
|
|
+ apt-cache policy smartbotic-automation
|
|
|
+'
|
|
|
+```
|
|
|
+
|
|
|
+(If the repo is auth-protected, supply credentials via `auth.conf.d`.)
|
|
|
+
|
|
|
+Expected: `apt-cache policy smartbotic-automation` shows the version `1.0.0-1` with `https://repository.smartbotics.ai trixie/main` as candidate source.
|
|
|
+
|
|
|
+- [ ] **Step 2: Final commit (if any tweaks were needed)**
|
|
|
+
|
|
|
+If Phase C or D revealed issues that required source edits:
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /data/smartbotic
|
|
|
+git add -A
|
|
|
+git commit -m "fix(packaging): <whatever was tweaked>"
|
|
|
+./packaging/build.sh --sync --suite trixie # re-publish
|
|
|
+```
|
|
|
+
|
|
|
+Otherwise nothing to commit.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## Self-Review Checklist (run before handoff)
|
|
|
+
|
|
|
+- [ ] Spec §3 (code migration) — covered by tasks A1, A4, A5, A6, A7, A8, A9
|
|
|
+- [ ] Spec §4 (file layout) — covered by Task B5 (`create-deb.sh` stages exactly the spec'd paths)
|
|
|
+- [ ] Spec §4.1 (systemd) — covered by Task B7
|
|
|
+- [ ] Spec §5 (deps) — covered by Task B6
|
|
|
+- [ ] Spec §6 (maintainer scripts) — covered by Task B9
|
|
|
+- [ ] Spec §7 (`packaging/` layout) — covered by Tasks B2-B10
|
|
|
+- [ ] Spec §8 (APT repo deploy) — covered by Tasks D1, D2
|
|
|
+- [ ] Spec §9 (no local cleanup) — confirmed in design; Phase C explicitly uses a separate container
|
|
|
+- [ ] Spec §10 decisions 1-5 — all reflected: drop `src/database/` (A8-9), single `smartbotic-automation` package (B6), `/usr/bin` install (B5), unrelated 1.7.1 instance left alone (test in container), enable-but-don't-start (B9 postinst)
|
|
|
+- [ ] Spec §11 (out of scope) — no tasks for repackaging upstream DB, no cross-distro builds, no data migration; correct
|