# Smartbotic Database Packaging Consolidation > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Centralize Debian packaging of smartbotic-database into this repo. Two build modes: Docker (targeting Debian 13 for production) and local (native build for the current system, for development/testing). Produces four `.deb` packages: server, shared runtime library, development headers, and CLI tool. **Architecture:** The client library becomes a shared `.so` with soversioning. Four packages follow Debian conventions: `libfoo` (runtime `.so`), `libfoo-dev` (headers + linker symlink), plus the server and CLI. Docker builds target Debian 13 and publish to the apt repo. Local builds produce identical packages for the current system so consumer projects can be built locally with `find_package()` against installed dev packages. **Tech Stack:** CMake + Ninja, dpkg-deb, Docker (for production builds), smartbotics-deb-repo tooling --- ## Two Build Modes ```bash # Production: Docker build targeting Debian 13 ./packaging/build.sh # Build 4 .debs in dist/ ./packaging/build.sh --sync --suite trixie # Build + publish to repo # Local development: native build for current system ./packaging/build.sh --local # Build 4 .debs in dist/ sudo dpkg -i dist/libsmartbotic-db-client_*.deb dist/libsmartbotic-db-client-dev_*.deb # → consumer projects can now: find_package(smartbotic-db-client) ``` ## Four Packages ### 1. `smartbotic-database` (server) | Field | Value | |-------|-------| | Binary | `/usr/bin/smartbotic-database` | | Config | `/etc/smartbotic-database/config.json` (conffile) | | Data dir | `/var/lib/smartbotic-database/` | | Systemd | `smartbotic-database.service` | | User/Group | `smartbotic-db:smartbotic-db` | | Depends | `libsmartbotic-db-client (= {{VERSION}}), libssl3t64, liblz4-1` | | Provides | `shadowman-database, callerai-storage` | | Conflicts/Replaces | `shadowman-database, callerai-storage` | ### 2. `libsmartbotic-db-client` (shared runtime library) | Field | Value | |-------|-------| | Shared lib | `/usr/lib/x86_64-linux-gnu/libsmartbotic-db-client.so.1.2.0` | | Soname link | `/usr/lib/x86_64-linux-gnu/libsmartbotic-db-client.so.1` | | Depends | `libprotobuf32t64, libgrpc++1.51t64, libspdlog1.15, libfmt10` | Triggers `ldconfig` in postinst/postrm. ### 3. `libsmartbotic-db-client-dev` (development headers) | Field | Value | |-------|-------| | Headers | `/usr/include/smartbotic/database/client.hpp`, `document.hpp` | | Proto headers | `/usr/include/smartbotic/database/proto/database.pb.h`, `database.grpc.pb.h` | | Linker symlink | `/usr/lib/x86_64-linux-gnu/libsmartbotic-db-client.so` → `.so.1` | | Proto source | `/usr/share/smartbotic-database/proto/database.proto` | | CMake config | `/usr/lib/x86_64-linux-gnu/cmake/smartbotic-db-client/smartbotic-db-clientConfig.cmake` | | Depends | `libsmartbotic-db-client (= {{VERSION}}), libprotobuf-dev, libgrpc++-dev, libspdlog-dev, nlohmann-json3-dev` | ### 4. `smartbotic-db-cli` (CLI admin tool) | Field | Value | |-------|-------| | Binary | `/usr/bin/smartbotic-db-cli` | | Depends | `libsmartbotic-db-client (>= {{VERSION}})` | --- ## CMake Changes Required ### Root `CMakeLists.txt` — shared/static option ```cmake option(BUILD_SHARED_LIBS "Build shared libraries" ${SMARTBOTIC_DB_STANDALONE}) ``` Standalone (packaging): shared `.so`. Submodule (embedded in consumer project): static `.a`. ### `proto/CMakeLists.txt` — STATIC → OBJECT (when shared) ```cmake if(BUILD_SHARED_LIBS) add_library(smartbotic_db_proto OBJECT ${PROTO_SRCS} ${GRPC_SRCS}) set_target_properties(smartbotic_db_proto PROPERTIES POSITION_INDEPENDENT_CODE ON) else() add_library(smartbotic_db_proto STATIC ${PROTO_SRCS} ${GRPC_SRCS}) endif() ``` ### `client/CMakeLists.txt` — SHARED with soversion + install rules ```cmake if(BUILD_SHARED_LIBS) add_library(smartbotic-db-client SHARED src/client.cpp $ ) set_target_properties(smartbotic-db-client PROPERTIES VERSION ${SMARTBOTIC_DB_VERSION} SOVERSION 1 ) else() add_library(smartbotic-db-client STATIC src/client.cpp) endif() ``` ### `service/CMakeLists.txt` — link shared client lib When shared, link `smartbotic-db-client` (which bundles proto). When static, link `smartbotic_db_proto` directly. --- ## File Structure ``` packaging/ ├── Dockerfile.base # Debian 13 base image with build deps ├── Dockerfile.build # Multi-stage: compile + package → 4 .debs ├── build.sh # Orchestrator: --local or Docker, optional repo publish ├── deb/ │ ├── create-debs.sh # dpkg-deb staging for all 4 packages │ ├── config/ │ │ └── config.json # Default production config │ ├── templates/ │ │ ├── control.server # smartbotic-database │ │ ├── control.lib # libsmartbotic-db-client │ │ ├── control.dev # libsmartbotic-db-client-dev │ │ └── control.cli # smartbotic-db-cli │ ├── scripts/ │ │ ├── server.postinst # User/dirs/migration/enable service │ │ ├── server.prerm # Stop service │ │ ├── server.postrm # Reload systemd, purge cleanup │ │ ├── lib.postinst # ldconfig │ │ └── lib.postrm # ldconfig │ └── systemd/ │ └── smartbotic-database.service ``` --- ### Task 1: Modify CMakeLists.txt for shared library support **Files:** - Modify: `CMakeLists.txt` (root) — add `BUILD_SHARED_LIBS` option - Modify: `proto/CMakeLists.txt` — STATIC → OBJECT when shared - Modify: `client/CMakeLists.txt` — SHARED with soversion, bundle proto objects, install rules - Modify: `service/CMakeLists.txt` — link shared client lib when shared - [ ] **Step 1: Root CMakeLists.txt — add shared lib option** After standalone detection block, add: ```cmake # Shared libraries for standalone (packaging), static for embedded (submodule) option(BUILD_SHARED_LIBS "Build shared libraries" ${SMARTBOTIC_DB_STANDALONE}) ``` - [ ] **Step 2: proto/CMakeLists.txt — conditional OBJECT/STATIC** Replace `add_library(smartbotic_db_proto STATIC ...)` with: ```cmake if(BUILD_SHARED_LIBS) add_library(smartbotic_db_proto OBJECT ${PROTO_SRCS} ${GRPC_SRCS}) set_target_properties(smartbotic_db_proto PROPERTIES POSITION_INDEPENDENT_CODE ON) else() add_library(smartbotic_db_proto STATIC ${PROTO_SRCS} ${GRPC_SRCS}) endif() ``` Keep all existing target properties, compile options, and link libraries unchanged. - [ ] **Step 3: client/CMakeLists.txt — SHARED with soversion** Replace the `add_library` call: ```cmake if(BUILD_SHARED_LIBS) add_library(smartbotic-db-client SHARED src/client.cpp $ ) set_target_properties(smartbotic-db-client PROPERTIES VERSION ${SMARTBOTIC_DB_VERSION} SOVERSION 1 ) else() add_library(smartbotic-db-client STATIC src/client.cpp) endif() ``` Keep all existing `target_include_directories`, `target_link_libraries`, alias. Add install rules at bottom: ```cmake if(SMARTBOTIC_DB_STANDALONE) include(GNUInstallDirs) install(TARGETS smartbotic-db-client EXPORT smartbotic-db-client-targets LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} ) install(DIRECTORY include/smartbotic DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} ) install(EXPORT smartbotic-db-client-targets FILE smartbotic-db-clientConfig.cmake NAMESPACE smartbotic:: DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/smartbotic-db-client ) endif() ``` - [ ] **Step 4: service/CMakeLists.txt — link shared client lib** Replace the existing `smartbotic_db_proto` link in `target_link_libraries`: ```cmake if(BUILD_SHARED_LIBS) target_link_libraries(smartbotic-database PRIVATE smartbotic-db-client) else() target_link_libraries(smartbotic-database PRIVATE smartbotic_db_proto) endif() ``` - [ ] **Step 5: Build and test both modes** Shared (standalone): ```bash cmake -B build -G Ninja cmake --build build -j$(nproc) file build/client/libsmartbotic-db-client.so.1.2.0 # Expected: ELF 64-bit LSB shared object ldd build/cli/smartbotic-db-cli | grep smartbotic # Expected: libsmartbotic-db-client.so.1 LD_LIBRARY_PATH=build/client ./build/tests/test_vector_storage ``` Static (submodule mode): ```bash cmake -B build-static -G Ninja -DBUILD_SHARED_LIBS=OFF cmake --build build-static -j$(nproc) file build-static/client/libsmartbotic-db-client.a # Expected: current ar archive ``` - [ ] **Step 6: Commit** ```bash git add CMakeLists.txt proto/CMakeLists.txt client/CMakeLists.txt service/CMakeLists.txt git commit -m "feat: shared library support (BUILD_SHARED_LIBS) Standalone builds produce libsmartbotic-db-client.so with soversion 1 and proto objects bundled in. Server and CLI link the shared lib. Submodule builds produce static .a as before — no changes needed in consumer projects." ``` --- ### Task 2: Create Dockerfiles for Debian 13 build **Files:** - Create: `packaging/Dockerfile.base` - Create: `packaging/Dockerfile.build` - [ ] **Step 1: Create Dockerfile.base** Create `packaging/Dockerfile.base`: ```dockerfile # Smartbotic Database Build Base Image — Debian 13 (trixie) # # Build: # docker buildx build -f packaging/Dockerfile.base -t smartbotic-db-build-base:debian13 . FROM debian:trixie ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt-get install -y --no-install-recommends \ build-essential \ cmake \ ninja-build \ git \ pkg-config \ ca-certificates \ ccache \ libssl-dev \ liblz4-dev \ protobuf-compiler \ libprotobuf-dev \ libgrpc++-dev \ protobuf-compiler-grpc \ nlohmann-json3-dev \ libspdlog-dev \ libfmt-dev \ dpkg-dev \ && rm -rf /var/lib/apt/lists/* ENV CCACHE_DIR=/ccache ENV CCACHE_MAXSIZE=2G ENV CCACHE_COMPRESS=1 ENV CCACHE_COMPRESSLEVEL=6 WORKDIR /build RUN echo "smartbotic-db-build-base:debian13:v1" > /etc/smartbotic-db-build-base ``` - [ ] **Step 2: Create Dockerfile.build** Create `packaging/Dockerfile.build`: ```dockerfile # syntax=docker/dockerfile:1.4 # Smartbotic Database Package Builder — Debian 13 (trixie) # Produces 4 .deb packages: server, libclient, libclient-dev, cli # # Usage: # docker buildx build -f packaging/Dockerfile.build \ # --build-arg BASE_IMAGE=smartbotic-db-build-base:debian13 \ # --target packages --output "type=local,dest=dist/" . ARG BASE_IMAGE=debian:trixie FROM ${BASE_IMAGE} AS builder ARG BUILD_VERSION=0.0.0 ARG BUILD_DEB_REVISION=1 ARG BUILD_JOBS=16 ARG BUILD_GIT_COMMIT=unknown ENV DEBIAN_FRONTEND=noninteractive ENV BUILD_VERSION=${BUILD_VERSION} ENV BUILD_DEB_REVISION=${BUILD_DEB_REVISION} ENV BUILD_JOBS=${BUILD_JOBS} ENV CCACHE_DIR=/ccache ENV CCACHE_MAXSIZE=2G ENV CCACHE_COMPRESS=1 ENV CCACHE_COMPRESSLEVEL=6 ENV PATH="/usr/lib/ccache:${PATH}" # Install deps if not using pre-built base RUN if [ ! -f /etc/smartbotic-db-build-base ]; then \ apt-get update && apt-get install -y --no-install-recommends \ build-essential cmake ninja-build git pkg-config ca-certificates ccache \ libssl-dev liblz4-dev \ protobuf-compiler libprotobuf-dev libgrpc++-dev protobuf-compiler-grpc \ nlohmann-json3-dev libspdlog-dev libfmt-dev dpkg-dev \ && rm -rf /var/lib/apt/lists/*; \ else echo "Using pre-built base: $(cat /etc/smartbotic-db-build-base)"; fi WORKDIR /build COPY CMakeLists.txt VERSION ./ COPY cmake/ ./cmake/ COPY proto/ ./proto/ COPY client/ ./client/ COPY service/ ./service/ COPY cli/ ./cli/ COPY tests/ ./tests/ COPY packaging/ ./packaging/ # Build with shared libs + Release RUN --mount=type=cache,target=/ccache,id=smartbotic-db-ccache \ cmake -B build -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_C_COMPILER_LAUNCHER=ccache \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ -DSMARTBOTIC_DB_BUILD_SERVICE=ON \ -DSMARTBOTIC_DB_BUILD_CLIENT=ON \ -DSMARTBOTIC_DB_BUILD_CLI=ON \ -DSMARTBOTIC_DB_BUILD_TESTS=ON \ -DBUILD_SHARED_LIBS=ON \ && cmake --build build --parallel ${BUILD_JOBS} \ && ccache --show-stats || true # Run tests RUN cd build && LD_LIBRARY_PATH=./client:$LD_LIBRARY_PATH ./tests/test_vector_storage # Package FROM builder AS packager RUN mkdir -p /packages ENV OUTPUT_DIR=/packages RUN chmod +x /build/packaging/deb/create-debs.sh && \ /build/packaging/deb/create-debs.sh # Output FROM scratch AS packages COPY --from=packager /packages/*.deb / ``` - [ ] **Step 3: Commit** ```bash git add packaging/Dockerfile.base packaging/Dockerfile.build git commit -m "packaging: add Docker build files targeting Debian 13 (trixie)" ``` --- ### Task 3: Create control templates for all four packages **Files:** - Create: `packaging/deb/templates/control.server` - Create: `packaging/deb/templates/control.lib` - Create: `packaging/deb/templates/control.dev` - Create: `packaging/deb/templates/control.cli` - [ ] **Step 1: Create control.server** ``` Package: smartbotic-database Version: {{VERSION}} Architecture: amd64 Maintainer: Ferenc Szontagh Description: Smartbotic Database — document store with gRPC API Standalone document database with gRPC API, vector similarity search, WAL persistence, LZ4 compression, AES-256-GCM field-level encryption, file storage with blob deduplication, and JSON-based migrations. Proprietary software by SmartBotics Kft. Homepage: https://smartbotics.ai Section: contrib/utils Priority: optional License: proprietary Provides: shadowman-database, callerai-storage Conflicts: shadowman-database, callerai-storage Replaces: shadowman-database, callerai-storage Depends: libsmartbotic-db-client (= {{VERSION}}), libssl3t64, liblz4-1 Installed-Size: {{INSTALLED_SIZE}} ``` - [ ] **Step 2: Create control.lib** ``` Package: libsmartbotic-db-client Version: {{VERSION}} Architecture: amd64 Maintainer: Ferenc Szontagh Description: Smartbotic Database — client shared library Shared library for applications connecting to smartbotic-database via gRPC. Includes compiled protobuf/gRPC stubs. Homepage: https://smartbotics.ai Section: contrib/libs Priority: optional License: proprietary Depends: libprotobuf32t64, libgrpc++1.51t64, libspdlog1.15, libfmt10 Installed-Size: {{INSTALLED_SIZE}} ``` - [ ] **Step 3: Create control.dev** ``` Package: libsmartbotic-db-client-dev Version: {{VERSION}} Architecture: amd64 Maintainer: Ferenc Szontagh Description: Smartbotic Database — client development files C++ headers, linker symlink, proto source, and CMake config for building applications against the smartbotic-database client API. Homepage: https://smartbotics.ai Section: contrib/libdevel Priority: optional License: proprietary Depends: libsmartbotic-db-client (= {{VERSION}}), libprotobuf-dev, libgrpc++-dev, libspdlog-dev, nlohmann-json3-dev Installed-Size: {{INSTALLED_SIZE}} ``` - [ ] **Step 4: Create control.cli** ``` Package: smartbotic-db-cli Version: {{VERSION}} Architecture: amd64 Maintainer: Ferenc Szontagh Description: Smartbotic Database — CLI administration tool Command-line tool for managing a smartbotic-database instance. Connects via gRPC — does not require the server installed locally. Homepage: https://smartbotics.ai Section: contrib/utils Priority: optional License: proprietary Depends: libsmartbotic-db-client (>= {{VERSION}}) Installed-Size: {{INSTALLED_SIZE}} ``` - [ ] **Step 5: Commit** ```bash git add packaging/deb/templates/ git commit -m "packaging: add control templates for server, lib, dev, and cli packages" ``` --- ### Task 4: Create default config, systemd unit, and maintainer scripts **Files:** - Create: `packaging/deb/config/config.json` - Create: `packaging/deb/systemd/smartbotic-database.service` - Create: `packaging/deb/scripts/server.postinst` - Create: `packaging/deb/scripts/server.prerm` - Create: `packaging/deb/scripts/server.postrm` - Create: `packaging/deb/scripts/lib.postinst` - Create: `packaging/deb/scripts/lib.postrm` - [ ] **Step 1: Create default production config** Create `packaging/deb/config/config.json`: ```json { "log_level": "info", "storage": { "bind_address": "localhost", "rpc_port": 9004, "node_id": "smartbotic-db", "data_directory": "/var/lib/smartbotic-database", "memory": { "max_memory_mb": 512, "eviction_threshold_percent": 80, "eviction_target_percent": 60, "eviction_check_interval_ms": 5000 }, "persistence": { "wal_sync_interval_ms": 100, "snapshot_interval_sec": 3600, "compression": "lz4" }, "encryption": { "enabled": true, "key_file": "/var/lib/smartbotic-database/storage.key", "auto_generate_key": true }, "migrations": { "enabled": false, "directory": "/etc/smartbotic-database/migrations", "auto_apply": false }, "files": { "max_file_size_mb": 500, "cleanup_orphans_interval_sec": 3600 }, "replication": { "enabled": false } } } ``` - [ ] **Step 2: Create systemd unit** Create `packaging/deb/systemd/smartbotic-database.service`: ```ini [Unit] Description=Smartbotic Database Service After=network.target [Service] Type=simple User=smartbotic-db Group=smartbotic-db ExecStart=/usr/bin/smartbotic-database --config /etc/smartbotic-database/config.json WorkingDirectory=/var/lib/smartbotic-database Restart=on-failure RestartSec=3 TimeoutStopSec=30 LimitNOFILE=65536 [Install] WantedBy=multi-user.target ``` - [ ] **Step 3: Create server.postinst** Create `packaging/deb/scripts/server.postinst`: ```bash #!/bin/bash # postinst for smartbotic-database (server) set -e CONFIG_FILE="/etc/smartbotic-database/config.json" DATA_DIR="/var/lib/smartbotic-database" case "$1" in configure) if ! getent group smartbotic-db >/dev/null 2>&1; then addgroup --system smartbotic-db fi if ! getent passwd smartbotic-db >/dev/null 2>&1; then adduser --system --ingroup smartbotic-db --home "$DATA_DIR" \ --shell /usr/sbin/nologin --gecos "Smartbotic Database Service" smartbotic-db fi install -d -o smartbotic-db -g smartbotic-db -m 0750 "$DATA_DIR" install -d -o root -g smartbotic-db -m 0750 /etc/smartbotic-database # ── Migration from shadowman-database ── OLD_SM_DATA="/var/lib/shadowman/data/database" OLD_SM_CONFIG="/etc/shadowman/database.json" if [ -d "$OLD_SM_DATA" ] && [ "$(ls -A "$OLD_SM_DATA" 2>/dev/null)" ] && \ [ ! "$(ls -A "$DATA_DIR" 2>/dev/null)" ]; then echo "smartbotic-database: migrating data from shadowman-database..." cp -a "$OLD_SM_DATA"/. "$DATA_DIR"/ chown -R smartbotic-db:smartbotic-db "$DATA_DIR" fi if [ -f "$OLD_SM_CONFIG" ] && [ ! -f "$CONFIG_FILE" ]; then cp "$OLD_SM_CONFIG" "$CONFIG_FILE" sed -i 's|/var/lib/shadowman/data/database|/var/lib/smartbotic-database|g' "$CONFIG_FILE" sed -i 's|/var/lib/shadowman/data/storage\.key|/var/lib/smartbotic-database/storage.key|g' "$CONFIG_FILE" fi # ── Migration from callerai-storage ── OLD_CA_DATA="/var/lib/callerai/storage" OLD_CA_CONFIG="/etc/callerai/storage.json" if [ -d "$OLD_CA_DATA" ] && [ "$(ls -A "$OLD_CA_DATA" 2>/dev/null)" ] && \ [ ! "$(ls -A "$DATA_DIR" 2>/dev/null)" ]; then echo "smartbotic-database: migrating data from callerai-storage..." cp -a "$OLD_CA_DATA"/. "$DATA_DIR"/ chown -R smartbotic-db:smartbotic-db "$DATA_DIR" fi if [ -f "$OLD_CA_CONFIG" ] && [ ! -f "$CONFIG_FILE" ]; then cp "$OLD_CA_CONFIG" "$CONFIG_FILE" sed -i 's|/var/lib/callerai/storage|/var/lib/smartbotic-database|g' "$CONFIG_FILE" fi if [ -f "$CONFIG_FILE" ]; then chown root:smartbotic-db "$CONFIG_FILE" chmod 640 "$CONFIG_FILE" fi systemctl daemon-reload || true if [ -z "$2" ]; then systemctl enable smartbotic-database.service || true systemctl start smartbotic-database.service || true else systemctl reset-failed smartbotic-database.service 2>/dev/null || true systemctl restart smartbotic-database.service || true fi ;; abort-upgrade|abort-remove|abort-deconfigure) ;; *) echo "postinst called with unknown argument '$1'" >&2; exit 1 ;; esac exit 0 ``` - [ ] **Step 4: Create server.prerm** ```bash #!/bin/bash set -e case "$1" in remove|deconfigure) systemctl stop smartbotic-database.service || true systemctl disable smartbotic-database.service || true ;; upgrade) systemctl stop smartbotic-database.service || true ;; failed-upgrade) ;; *) echo "prerm called with unknown argument '$1'" >&2; exit 1 ;; esac exit 0 ``` - [ ] **Step 5: Create server.postrm** ```bash #!/bin/bash set -e case "$1" in remove) systemctl daemon-reload || true ;; purge) systemctl daemon-reload || true rm -f /etc/smartbotic-database/config.json rm -f /etc/smartbotic-database/config.json.dpkg-* rmdir /etc/smartbotic-database 2>/dev/null || true ;; upgrade|failed-upgrade|abort-install|abort-upgrade|disappear) ;; *) echo "postrm called with unknown argument '$1'" >&2; exit 1 ;; esac exit 0 ``` - [ ] **Step 6: Create lib.postinst and lib.postrm** `packaging/deb/scripts/lib.postinst`: ```bash #!/bin/bash set -e ldconfig exit 0 ``` `packaging/deb/scripts/lib.postrm`: ```bash #!/bin/bash set -e ldconfig exit 0 ``` - [ ] **Step 7: Commit** ```bash git add packaging/deb/ git commit -m "packaging: add server config, systemd, maintainer scripts, ldconfig triggers" ``` --- ### Task 5: Create the create-debs.sh multi-package script **Files:** - Create: `packaging/deb/create-debs.sh` - [ ] **Step 1: Create create-debs.sh** Create `packaging/deb/create-debs.sh`: ```bash #!/bin/bash # Smartbotic Database .deb Package Creation Script # Creates four Debian packages from built artifacts: # - smartbotic-database (server binary + systemd + config) # - libsmartbotic-db-client (shared library .so) # - libsmartbotic-db-client-dev (headers + linker symlink + proto + cmake config) # - smartbotic-db-cli (CLI admin tool binary) # # Environment variables: # BUILD_DIR - Build output directory (default: ./build) # SOURCE_DIR - Source directory (default: .) # OUTPUT_DIR - Output for .deb files (default: ./dist) # BUILD_DEB_REVISION - Debian revision suffix (default: 1) set -e SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" SOURCE_DIR="${SOURCE_DIR:-$(cd "$SCRIPT_DIR/../.." && pwd)}" BUILD_DIR="${BUILD_DIR:-$SOURCE_DIR/build}" OUTPUT_DIR="${OUTPUT_DIR:-$SOURCE_DIR/dist}" BUILD_DEB_REVISION="${BUILD_DEB_REVISION:-1}" if [ ! -f "$SOURCE_DIR/VERSION" ]; then echo "ERROR: VERSION file not found"; exit 1 fi VERSION=$(cat "$SOURCE_DIR/VERSION" | tr -d '[:space:]') DEB_VERSION="${VERSION}-${BUILD_DEB_REVISION}" TEMPLATES_DIR="$SCRIPT_DIR/templates" SCRIPTS_DIR="$SCRIPT_DIR/scripts" SYSTEMD_DIR="$SCRIPT_DIR/systemd" CONFIG_DIR="$SCRIPT_DIR/config" GREEN='\033[0;32m' BLUE='\033[0;34m' RED='\033[0;31m' NC='\033[0m' print_info() { echo -e "${GREEN}[INFO]${NC} $1"; } print_step() { echo -e "${BLUE}[STEP]${NC} $1"; } print_error() { echo -e "${RED}[ERROR]${NC} $1"; } calc_installed_size() { local size=0 for d in "$1"/*/; do [ "$(basename "$d")" = "DEBIAN" ] && continue size=$((size + $(du -sk "$d" | cut -f1))) done echo "$size" } build_deb() { local pkg_name="$1" staging="$2" local INSTALLED_SIZE=$(calc_installed_size "$staging") sed -i "s/{{VERSION}}/${DEB_VERSION}/g" "$staging/DEBIAN/control" sed -i "s/{{INSTALLED_SIZE}}/${INSTALLED_SIZE}/g" "$staging/DEBIAN/control" for s in postinst prerm postrm preinst; do [ -f "$staging/DEBIAN/$s" ] && chmod 755 "$staging/DEBIAN/$s" done chmod 644 "$staging/DEBIAN/control" [ -f "$staging/DEBIAN/conffiles" ] && chmod 644 "$staging/DEBIAN/conffiles" dpkg-deb --build --root-owner-group "$staging" "$OUTPUT_DIR/${pkg_name}_${DEB_VERSION}_amd64.deb" print_info " Created: ${pkg_name}_${DEB_VERSION}_amd64.deb" } # ── 1. smartbotic-database (server) ────────────────────────── create_server_package() { print_step "Creating smartbotic-database (server)..." local S="/tmp/staging-smartbotic-database" rm -rf "$S" mkdir -p "$S/DEBIAN" "$S/usr/bin" "$S/etc/smartbotic-database" "$S/lib/systemd/system" cp "$TEMPLATES_DIR/control.server" "$S/DEBIAN/control" cp "$SCRIPTS_DIR/server.postinst" "$S/DEBIAN/postinst" cp "$SCRIPTS_DIR/server.prerm" "$S/DEBIAN/prerm" cp "$SCRIPTS_DIR/server.postrm" "$S/DEBIAN/postrm" local bin="$BUILD_DIR/service/smartbotic-database" [ ! -f "$bin" ] && print_error "Server binary not found: $bin" && exit 1 cp "$bin" "$S/usr/bin/smartbotic-database" && chmod 755 "$S/usr/bin/smartbotic-database" cp "$CONFIG_DIR/config.json" "$S/etc/smartbotic-database/config.json" echo "/etc/smartbotic-database/config.json" > "$S/DEBIAN/conffiles" cp "$SYSTEMD_DIR/smartbotic-database.service" "$S/lib/systemd/system/" build_deb "smartbotic-database" "$S" rm -rf "$S" } # ── 2. libsmartbotic-db-client (shared library) ───────────── create_lib_package() { print_step "Creating libsmartbotic-db-client (shared library)..." local S="/tmp/staging-libsmartbotic-db-client" local LIBDIR="$S/usr/lib/x86_64-linux-gnu" rm -rf "$S" mkdir -p "$S/DEBIAN" "$LIBDIR" cp "$TEMPLATES_DIR/control.lib" "$S/DEBIAN/control" cp "$SCRIPTS_DIR/lib.postinst" "$S/DEBIAN/postinst" cp "$SCRIPTS_DIR/lib.postrm" "$S/DEBIAN/postrm" local SO_FILE=$(find "$BUILD_DIR/client" -name "libsmartbotic-db-client.so.$VERSION" 2>/dev/null | head -1) [ -z "$SO_FILE" ] && SO_FILE=$(find "$BUILD_DIR/client" -name "libsmartbotic-db-client.so.*.*.*" 2>/dev/null | head -1) [ -z "$SO_FILE" ] && print_error "Shared library not found in $BUILD_DIR/client/" && exit 1 cp "$SO_FILE" "$LIBDIR/libsmartbotic-db-client.so.$VERSION" ln -s "libsmartbotic-db-client.so.$VERSION" "$LIBDIR/libsmartbotic-db-client.so.1" build_deb "libsmartbotic-db-client" "$S" rm -rf "$S" } # ── 3. libsmartbotic-db-client-dev (headers + linker symlink) ─ create_dev_package() { print_step "Creating libsmartbotic-db-client-dev (development)..." local S="/tmp/staging-libsmartbotic-db-client-dev" local INCDIR="$S/usr/include/smartbotic/database" local PROTO_INCDIR="$S/usr/include/smartbotic/database/proto" local LIBDIR="$S/usr/lib/x86_64-linux-gnu" local CMAKEDIR="$LIBDIR/cmake/smartbotic-db-client" local PROTODIR="$S/usr/share/smartbotic-database/proto" rm -rf "$S" mkdir -p "$S/DEBIAN" "$INCDIR" "$PROTO_INCDIR" "$LIBDIR" "$CMAKEDIR" "$PROTODIR" cp "$TEMPLATES_DIR/control.dev" "$S/DEBIAN/control" # Client headers cp "$SOURCE_DIR/client/include/smartbotic/database/client.hpp" "$INCDIR/" cp "$SOURCE_DIR/client/include/smartbotic/database/document.hpp" "$INCDIR/" # Generated proto headers cp "$BUILD_DIR/proto/generated/database.pb.h" "$PROTO_INCDIR/" cp "$BUILD_DIR/proto/generated/database.grpc.pb.h" "$PROTO_INCDIR/" # Linker symlink ln -s "libsmartbotic-db-client.so.1" "$LIBDIR/libsmartbotic-db-client.so" # Proto source cp "$SOURCE_DIR/proto/database.proto" "$PROTODIR/" # CMake config cat > "$CMAKEDIR/smartbotic-db-clientConfig.cmake" <<'CMAKEOF' # smartbotic-db-client CMake config # Usage: find_package(smartbotic-db-client REQUIRED) # Provides: smartbotic::db-client target include(CMakeFindDependencyMacro) find_dependency(Protobuf REQUIRED) find_dependency(gRPC CONFIG QUIET) if(NOT gRPC_FOUND) find_package(PkgConfig REQUIRED) pkg_check_modules(GRPC REQUIRED IMPORTED_TARGET grpc++) endif() if(NOT TARGET smartbotic::db-client) add_library(smartbotic::db-client SHARED IMPORTED) set_target_properties(smartbotic::db-client PROPERTIES IMPORTED_LOCATION "/usr/lib/x86_64-linux-gnu/libsmartbotic-db-client.so" INTERFACE_INCLUDE_DIRECTORIES "/usr/include" ) if(TARGET gRPC::grpc++) set_property(TARGET smartbotic::db-client APPEND PROPERTY INTERFACE_LINK_LIBRARIES protobuf::libprotobuf gRPC::grpc++) else() set_property(TARGET smartbotic::db-client APPEND PROPERTY INTERFACE_LINK_LIBRARIES protobuf::libprotobuf PkgConfig::GRPC) endif() endif() CMAKEOF build_deb "libsmartbotic-db-client-dev" "$S" rm -rf "$S" } # ── 4. smartbotic-db-cli (CLI tool) ───────────────────────── create_cli_package() { print_step "Creating smartbotic-db-cli (CLI)..." local S="/tmp/staging-smartbotic-db-cli" rm -rf "$S" mkdir -p "$S/DEBIAN" "$S/usr/bin" cp "$TEMPLATES_DIR/control.cli" "$S/DEBIAN/control" local bin="$BUILD_DIR/cli/smartbotic-db-cli" [ ! -f "$bin" ] && print_error "CLI binary not found: $bin" && exit 1 cp "$bin" "$S/usr/bin/smartbotic-db-cli" && chmod 755 "$S/usr/bin/smartbotic-db-cli" build_deb "smartbotic-db-cli" "$S" rm -rf "$S" } # ── Main ───────────────────────────────────────────────────── main() { print_info "Smartbotic Database v${VERSION} (deb: ${DEB_VERSION})" print_info "Build dir: $BUILD_DIR" print_info "Output dir: $OUTPUT_DIR" echo "" mkdir -p "$OUTPUT_DIR" create_server_package create_lib_package create_dev_package create_cli_package echo "" print_info "All packages created:" ls -lh "$OUTPUT_DIR"/*.deb 2>/dev/null || true } main ``` - [ ] **Step 2: Commit** ```bash chmod +x packaging/deb/create-debs.sh git add packaging/deb/create-debs.sh git commit -m "packaging: add create-debs.sh for 4-package build" ``` --- ### Task 6: Create build.sh with --local and Docker modes **Files:** - Create: `packaging/build.sh` - [ ] **Step 1: Create build.sh** Create `packaging/build.sh`: ```bash #!/bin/bash # Smartbotic Database Build & Package Script # # Two modes: # Default (Docker): Builds inside Debian 13 container for production .debs # --local: Builds natively on the current system for dev/testing # # Usage: # ./packaging/build.sh # Docker build → dist/ # ./packaging/build.sh --local # Native build → dist/ # ./packaging/build.sh --sync --suite trixie # Docker build + publish to repo # ./packaging/build.sh --local --install # Native build + sudo dpkg -i set -e SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PROJECT_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" cd "$PROJECT_DIR" BASE_IMAGE_NAME="smartbotic-db-build-base:debian13" OUTPUT_DIR="${PROJECT_DIR}/dist" BUILD_JOBS="${BUILD_JOBS:-$(nproc)}" DEB_REPO_DIR="${DEB_REPO_DIR:-/data/smartbotics-deb-repo}" RED='\033[0;31m' GREEN='\033[0;32m' YELLOW='\033[1;33m' BLUE='\033[0;34m' NC='\033[0m' log_info() { echo -e "${BLUE}[INFO]${NC} $1"; } log_success() { echo -e "${GREEN}[OK]${NC} $1"; } log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; } log_error() { echo -e "${RED}[ERROR]${NC} $1"; } # Parse arguments LOCAL_BUILD=false REBUILD_BASE=false NO_CACHE=false BUILD_REPO=false SYNC_REPO=false INSTALL_LOCAL=false SKIP_TESTS=false REPO_SUITE="" DEB_REVISION="1" while [[ $# -gt 0 ]]; do case $1 in --local) LOCAL_BUILD=true; shift ;; --install) INSTALL_LOCAL=true; shift ;; --rebuild-base) REBUILD_BASE=true; shift ;; --no-cache) NO_CACHE=true; shift ;; --skip-tests) SKIP_TESTS=true; shift ;; --deb-revision) DEB_REVISION="$2"; shift 2 ;; --repo) BUILD_REPO=true; shift ;; --suite) REPO_SUITE="$2"; shift 2 ;; --sync) SYNC_REPO=true; BUILD_REPO=true; shift ;; --help|-h) echo "Smartbotic Database Build & Package Script" echo "" echo "Usage: $0 [OPTIONS]" echo "" echo "Build modes:" echo " (default) Docker build targeting Debian 13 (production)" echo " --local Native build for current system (development)" echo "" echo "Options:" echo " --install Install .debs locally after build (--local only)" echo " --skip-tests Skip running tests" echo " --deb-revision N Debian revision suffix (default: 1)" echo " --rebuild-base Force rebuild Docker base image" echo " --no-cache Build without caches" echo " --repo Add packages to local APT repository" echo " --suite NAME Distribution suite (e.g., trixie)" echo " --sync Add to repo and upload to repository.smartbotics.ai" echo "" echo "Examples:" echo " $0 # Docker → dist/*.deb (Debian 13)" echo " $0 --local # Native → dist/*.deb (current system)" echo " $0 --local --install # Native → install locally" echo " $0 --sync --suite trixie # Docker → publish to repo" exit 0 ;; *) log_error "Unknown option: $1"; exit 1 ;; esac done VERSION=$(cat VERSION 2>/dev/null | tr -d '[:space:]') if [ -z "$VERSION" ]; then log_error "Could not read VERSION file"; exit 1 fi echo "==================================================" echo " Smartbotic Database v${VERSION}-${DEB_REVISION}" if [ "$LOCAL_BUILD" = true ]; then echo " Mode: Local ($(lsb_release -ds 2>/dev/null || uname -sr))" else echo " Mode: Docker (Debian 13 trixie)" fi echo "==================================================" echo "" rm -rf "$OUTPUT_DIR" mkdir -p "$OUTPUT_DIR" if [ "$LOCAL_BUILD" = true ]; then # ── Local native build ────────────────────────────────── log_info "Building natively..." cmake -B build -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ -DSMARTBOTIC_DB_BUILD_SERVICE=ON \ -DSMARTBOTIC_DB_BUILD_CLIENT=ON \ -DSMARTBOTIC_DB_BUILD_CLI=ON \ -DSMARTBOTIC_DB_BUILD_TESTS=ON \ -DBUILD_SHARED_LIBS=ON cmake --build build -j"$BUILD_JOBS" log_success "Build complete" if [ "$SKIP_TESTS" = false ]; then log_info "Running tests..." LD_LIBRARY_PATH=build/client:$LD_LIBRARY_PATH ./build/tests/test_vector_storage log_success "Tests passed" fi log_info "Creating .deb packages..." BUILD_DEB_REVISION="$DEB_REVISION" ./packaging/deb/create-debs.sh if [ "$INSTALL_LOCAL" = true ]; then echo "" log_info "Installing packages locally..." sudo dpkg -i \ "$OUTPUT_DIR/libsmartbotic-db-client_${VERSION}-${DEB_REVISION}_amd64.deb" \ "$OUTPUT_DIR/libsmartbotic-db-client-dev_${VERSION}-${DEB_REVISION}_amd64.deb" \ "$OUTPUT_DIR/smartbotic-db-cli_${VERSION}-${DEB_REVISION}_amd64.deb" \ "$OUTPUT_DIR/smartbotic-database_${VERSION}-${DEB_REVISION}_amd64.deb" \ 2>/dev/null || sudo apt-get install -f -y log_success "Packages installed" echo "" log_info "Verify: pkg-config or cmake can find the client library" echo " cmake: find_package(smartbotic-db-client REQUIRED)" echo " ldconfig: ldconfig -p | grep smartbotic" fi else # ── Docker build (Debian 13) ──────────────────────────── if ! command -v docker &>/dev/null; then log_error "Docker not found. Use --local for native build."; exit 1 fi # Base image base_image_exists() { docker image inspect "$BASE_IMAGE_NAME" &>/dev/null; } build_base_image() { log_info "Building base image: $BASE_IMAGE_NAME" docker buildx build -f packaging/Dockerfile.base -t "$BASE_IMAGE_NAME" . log_success "Base image built" echo "" } if [ "$REBUILD_BASE" = true ]; then build_base_image elif base_image_exists; then log_success "Base image found: $BASE_IMAGE_NAME" else log_warn "Base image not found, building..." build_base_image fi log_info "Building packages in Docker..." GIT_COMMIT=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown") BUILD_ARGS=( -f packaging/Dockerfile.build --build-arg BASE_IMAGE="$BASE_IMAGE_NAME" --build-arg BUILD_VERSION="$VERSION" --build-arg BUILD_DEB_REVISION="$DEB_REVISION" --build-arg BUILD_GIT_COMMIT="$GIT_COMMIT" --build-arg BUILD_JOBS="$BUILD_JOBS" --target packages --output "type=local,dest=$OUTPUT_DIR" ) [ "$NO_CACHE" = true ] && BUILD_ARGS+=(--no-cache) docker buildx build "${BUILD_ARGS[@]}" . fi echo "" echo "==================================================" echo " Build Complete" echo "==================================================" echo "" log_info "Packages:" ls -lh "$OUTPUT_DIR"/*.deb 2>/dev/null | while read line; do echo " $line"; done echo "" # Repository if [ "$BUILD_REPO" = true ]; then if [ -z "$REPO_SUITE" ]; then log_error "--suite required with --repo"; exit 1 fi [ ! -d "$DEB_REPO_DIR" ] && log_error "Repo not found: $DEB_REPO_DIR" && exit 1 log_info "Adding to repository..." "$DEB_REPO_DIR/scripts/add-packages.sh" "$OUTPUT_DIR" echo "" "$DEB_REPO_DIR/scripts/create-repo.sh" --suite "$REPO_SUITE" fi if [ "$SYNC_REPO" = true ]; then echo "" "$DEB_REPO_DIR/scripts/sync-repo.sh" fi echo "" log_success "Done!" if [ "$LOCAL_BUILD" = true ] && [ "$INSTALL_LOCAL" = false ]; then echo "" echo "To install locally:" echo " sudo dpkg -i dist/*.deb" echo " sudo apt-get install -f -y # fix deps if needed" fi if [ "$LOCAL_BUILD" = false ]; then echo "" echo "To publish:" echo " $0 --repo --suite trixie" echo " $0 --sync --suite trixie" fi ``` - [ ] **Step 2: Commit** ```bash chmod +x packaging/build.sh git add packaging/build.sh git commit -m "packaging: add build.sh with --local and Docker modes" ``` --- ### Task 7: Test local build + install - [ ] **Step 1: Build locally** ```bash ./packaging/build.sh --local ``` Expected: four `.deb` files in `dist/`. - [ ] **Step 2: Inspect packages** ```bash for pkg in smartbotic-database libsmartbotic-db-client libsmartbotic-db-client-dev smartbotic-db-cli; do echo "=== $pkg ===" dpkg-deb --info dist/${pkg}_1.2.0-1_amd64.deb dpkg-deb --contents dist/${pkg}_1.2.0-1_amd64.deb echo "" done ``` - [ ] **Step 3: Verify .so is shared and soversion is correct** ```bash dpkg-deb -x dist/libsmartbotic-db-client_1.2.0-1_amd64.deb /tmp/test-lib file /tmp/test-lib/usr/lib/x86_64-linux-gnu/libsmartbotic-db-client.so.1.2.0 readelf -d /tmp/test-lib/usr/lib/x86_64-linux-gnu/libsmartbotic-db-client.so.1.2.0 | grep SONAME rm -rf /tmp/test-lib ``` - [ ] **Step 4: Install locally and verify** ```bash ./packaging/build.sh --local --install # Verify shared lib is registered ldconfig -p | grep smartbotic # Verify headers are in place ls /usr/include/smartbotic/database/client.hpp # Verify cmake config works cmake --find-package -DNAME=smartbotic-db-client -DCOMPILER_ID=GNU -DLANGUAGE=CXX -DMODE=EXIST 2>&1 ``` - [ ] **Step 5: Verify submodule (static) mode still works** ```bash cmake -B build-static -G Ninja -DBUILD_SHARED_LIBS=OFF cmake --build build-static -j$(nproc) file build-static/client/libsmartbotic-db-client.a ``` --- ### Task 8: Test Docker build pipeline - [ ] **Step 1: Build base image** ```bash docker buildx build -f packaging/Dockerfile.base -t smartbotic-db-build-base:debian13 . ``` - [ ] **Step 2: Build packages** ```bash ./packaging/build.sh ``` - [ ] **Step 3: Compare local vs Docker outputs** Both should produce identical package names with the same structure (different binaries since different target system). --- ### Task 9: Publish to repository - [ ] **Step 1: Build and publish** ```bash ./packaging/build.sh --sync --suite trixie ``` - [ ] **Step 2: Verify in repo** ```bash ls /data/smartbotics-deb-repo/repo/pool/trixie/main/smartbotic-database_* ls /data/smartbotics-deb-repo/repo/pool/trixie/main/libsmartbotic-db-client_* ls /data/smartbotics-deb-repo/repo/pool/trixie/main/libsmartbotic-db-client-dev_* ls /data/smartbotics-deb-repo/repo/pool/trixie/main/smartbotic-db-cli_* ``` --- ### Task 10: Update CLAUDE.md - [ ] **Step 1: Add packaging section to CLAUDE.md** ```markdown ## Packaging Canonical source for all smartbotic-database Debian packages. Replaces the old `shadowman-database` and `callerai-storage` packages. ### Build modes ```bash # Production (Docker, Debian 13) ./packaging/build.sh # Build 4 .debs ./packaging/build.sh --sync --suite trixie # Build + publish to repo # Local development (native, current system) ./packaging/build.sh --local # Build 4 .debs ./packaging/build.sh --local --install # Build + install locally ``` ### Packages | Package | Contents | Deployed where | |---------|----------|---------------| | `smartbotic-database` | Server binary + systemd + config | Production DB servers | | `libsmartbotic-db-client` | Shared `.so` (runtime) | All production machines | | `libsmartbotic-db-client-dev` | Headers + linker symlink + `.proto` + cmake config | Docker build images, dev workstations | | `smartbotic-db-cli` | CLI tool | Anywhere (optional) | ### Local dev with installed packages After `./packaging/build.sh --local --install`, consumer projects can use: ```cmake find_package(smartbotic-db-client REQUIRED) target_link_libraries(myapp PRIVATE smartbotic::db-client) ``` The submodule fallback still works for projects that haven't switched. ### Version policy Bump `VERSION` for API/protocol changes. Deb revision (`-N`) for packaging-only changes. ``` - [ ] **Step 2: Commit** ```bash git add CLAUDE.md git commit -m "docs: add packaging documentation to CLAUDE.md" ``` --- ### Task 11: End-to-end verification - [ ] **Step 1: Local build + install + verify find_package** - [ ] **Step 2: Docker build + verify package contents** - [ ] **Step 3: Verify server Provides/Conflicts/Replaces** - [ ] **Step 4: Verify submodule static build still works** - [ ] **Step 5: Verify old packages remain in repo for rollback** --- ## Execution Order ``` Task 1 : CMake shared library changes Task 2 : Dockerfiles Task 3 : Control templates Task 4 : Config, systemd, maintainer scripts Task 5 : create-debs.sh Task 6 : build.sh (--local + Docker) Task 7 : Test local build + install Task 8 : Test Docker build Task 9 : Publish to repo Task 10 : CLAUDE.md Task 11 : End-to-end verification ``` All tasks are in this repo only. Consumer projects (shadowman-cpp, callerai) are handled separately by the user. ## Rollback Plan - `BUILD_SHARED_LIBS=OFF` restores static library for submodule builds - Old `shadowman-database` and `callerai-storage` packages stay in the repo - `sudo dpkg -r libsmartbotic-db-client-dev libsmartbotic-db-client` removes local install