docs: add smartbotic-vectorapi design spec

Approved design for a C++20 REST API fronting smartbotic-database for n8n
workflows (RAG vectors + JSON), bearer auth, React admin UI, minimal config
with DB-backed hot-reloaded settings, OpenAI embeddings, and Debian 13 .debs.

.gitignore

@@ -0,0 +1,14 @@
+# Build output
+/build/
+/build-*/
+/dist/
+
+# WebUI
+webui/node_modules/
+webui/dist/
+*.tsbuildinfo
+
+# Editor / OS
+.DS_Store
+*.swp
+compile_commands.json

VERSION

@@ -0,0 +1 @@
+0.1.0

docs/superpowers/specs/2026-05-31-smartbotic-vectorapi-design.md

@@ -0,0 +1,306 @@
+# smartbotic-vectorapi — Design Spec
+
+- **Date:** 2026-05-31
+- **Status:** Approved (pre-implementation)
+- **Repo:** `ssh://git@git.smartbotics.ai:10022/fszontagh/smartbotic-vectorapi.git`
+- **Version:** 0.1.0
+
+## 1. Purpose
+
+A C++20 REST API service that fronts `smartbotic-database` (a gRPC document +
+vector store) for **n8n workflows**. It stores:
+
+- **Vectors for RAG** — embeddings + payload, with cosine similarity search.
+- **JSON for everything else** — arbitrary documents with full CRUD.
+
+It adds bearer authentication (a single key is sufficient) and a small React
+admin web UI (login, DB stats, full CRUD on JSON content, RAG search tester,
+settings editor). It produces the `smartbotic-vectorapi` binary and Debian 13
+`.deb` packages for production deployment.
+
+## 2. Context & constraints
+
+- The database is reached over gRPC through the installed **`libsmartbotic-db-client`**
+  (v2.2.1; headers + cmake config from `libsmartbotic-db-client-dev`). Link via
+  `find_package(smartbotic-db-client REQUIRED)` → `smartbotic::db-client`.
+- HTTP layer: **cpp-httplib** (latest release), pulled via CMake `FetchContent`,
+  compiled with `CPPHTTPLIB_OPENSSL_SUPPORT` (needed both for the server and for
+  the outbound HTTPS call to OpenAI).
+- Web UI: **React 19 + Vite + TypeScript + TailwindCSS + Monaco editor +
+  react-query + zustand + react-router** — the same stack as `/data/callerai/webui`.
+- Packaging mirrors `/data/callerai` and `smartbotic-database`: a Docker
+  Debian-13 build plus a `--local` native build, published through
+  `/data/smartbotics-deb-repo`.
+- Filesystem layout follows **smartbotic-database (`/usr`)**, with a dedicated
+  `smartbotic-vectorapi` system user.
+- Toolchain present: g++ 15.2, cmake 4.2.3, C++20.
+
+### Key product decisions (from brainstorming)
+
+| Decision | Choice |
+|---|---|
+| Embeddings source | **Both, configurable** — store a supplied vector, else generate from text |
+| Embedding backend | **OpenAI cloud** (`/v1/embeddings`) |
+| Embedding model config | **Per-collection** — model stored with each collection; output dim validated against the collection's fixed dimension |
+| Collection management | **Admin-managed** — created via UI or management endpoint before n8n writes |
+| Web UI login | **With the API key** → HttpOnly session cookie |
+| Edit scope | **Full CRUD on JSON docs** |
+| Configuration | **Minimal `config.json`** (bootstrap only); everything else lives in the DB with **hot reload** |
+| API key bootstrap | **Auto-generate on first run + log once** (env var seeds/overrides) |
+| WebUI packaging | **Separate `smartbotic-vectorapi-webui` .deb** |
+| Install layout | **`/usr` (smartbotic-database style)** |
+| Agent discoverability | Ship + serve **`openapi.json`** and **`llms.txt`** |
+
+## 3. Architecture
+
+```
+  n8n workflows ──HTTP/JSON (Authorization: Bearer)──┐
+                                                     ▼
+  admin browser ──HTTP + session cookie──▶  smartbotic-vectorapi  ──gRPC──▶ smartbotic-database
+                                                     │  (libsmartbotic-db-client)
+                                                     └──HTTPS──▶ api.openai.com /v1/embeddings
+```
+
+cpp-httplib is a synchronous, thread-per-connection server; it pairs cleanly
+with the blocking gRPC client — no async runtime required.
+
+### Components (each independently testable)
+
+| Module | Responsibility | Depends on |
+|---|---|---|
+| `config` | Load/validate the minimal bootstrap JSON + env | nlohmann/json |
+| `db_gateway` | Owns one `smartbotic::database::Client`; connect/reconnect; exposes typed helpers | client lib |
+| `settings_store` | DB-backed runtime settings (`_vectorapi_settings`); hot reload via `subscribe()` + atomic snapshot swap; first-run key generation | db_gateway |
+| `collection_registry` | Per-collection metadata (`_vectorapi_collections`): kind, vector_dimension, embedding_model | db_gateway |
+| `embeddings` | Call OpenAI `/v1/embeddings`, parse `data[0].embedding`, validate dimension | httplib(SSL) |
+| `auth` | Validate bearer token (== settings.api_key) or session cookie; mint/expire cookies | settings_store |
+| `server` + `handlers/*` | Register routes, parse/validate requests, map errors → HTTP, serve static webui + meta files | all of the above |
+
+## 4. Configuration
+
+### 4.1 Bootstrap `config.json` (the only file on disk)
+
+Minimal — only the chicken-and-egg essentials needed before the DB is reachable:
+
+```json
+{
+  "log_level": "info",
+  "http": { "bind_address": "0.0.0.0", "port": 8080 },
+  "database": { "address": "localhost:9004" }
+}
+```
+
+- `webui` asset directory is a **compiled default** (`/usr/share/smartbotic-vectorapi/webui`),
+  overridable only via `--webui-dir` for local dev — not part of `config.json`.
+- Registered as a dpkg conffile.
+
+### 4.2 Runtime settings (DB-backed, hot-reloaded)
+
+Stored in the **encrypted** `_vectorapi_settings` collection, single document
+`id = "current"`:
+
+```jsonc
+{
+  "api_key": "<random>",            // sensitive (encrypted at rest)
+  "openai_api_base": "https://api.openai.com",
+  "openai_api_key": "",             // sensitive (encrypted at rest)
+  "default_embedding_model": "text-embedding-3-small",
+  "cors_origins": ["*"],
+  "session_ttl_minutes": 720,
+  "webui_enabled": true
+}
+```
+
+- Created on first start via `createCollection("_vectorapi_settings", 0, /*encrypted*/true, …)`.
+  At-rest protection of `api_key`/`openai_api_key` relies on the DB's field-level
+  encryption being enabled for the collection (the client `createCollection`
+  exposes only the `encrypted` flag; marking specific sensitive fields is a DB
+  collection-option detail to confirm during implementation). If encryption is
+  unavailable, the secrets are stored plaintext in a `_`-prefixed system
+  collection not exposed by the UI — acceptable for v1, noted as a risk.
+- Env overrides at startup (applied to the stored doc on first run, never logged):
+  `SMARTBOTIC_VECTORAPI_KEY`, `OPENAI_API_KEY`.
+- **Hot reload:** `Client::subscribe({"_vectorapi_settings","_vectorapi_collections"}, cb)`.
+  On any event the service rebuilds an immutable `Settings`/registry object and
+  atomically swaps a `std::shared_ptr<const Settings>`; readers always see a
+  consistent snapshot. A low-frequency poll (e.g. 30 s) backstops missed events.
+
+### 4.3 First-run key bootstrap
+
+On startup, after ensuring `_vectorapi_settings` exists:
+1. If `SMARTBOTIC_VECTORAPI_KEY` is set and no key stored → store it.
+2. Else if no key stored → generate a cryptographically random key, store it,
+   and **log it exactly once** at WARN: `Generated initial API key: <key> — store it now; this is shown only once.`
+3. Else → use the stored key.
+
+The key is never logged again after first generation.
+
+## 5. Data model
+
+- **`_vectorapi_collections`** (registry): one doc per managed collection —
+  `{ name, kind: "vector"|"json", vector_dimension, embedding_model, created_at }`.
+  The DB's `createCollection` only carries a vector dimension, so this registry
+  holds the extra metadata the API needs.
+- **Vector collections:** documents store the embedding in the DB's conventional
+  `_vector` field; remaining JSON is payload/metadata. Generated-vector dimension
+  is validated against the collection's fixed dimension (mismatch → HTTP 422).
+- **JSON collections:** arbitrary documents, full CRUD.
+- System collections are prefixed `_` and hidden from the UI's browse list.
+
+## 6. REST API
+
+Base path `/api/v1`. All `/api/v1/*` routes require auth (bearer **or** session
+cookie) **except** where noted. JSON request/response bodies.
+
+### 6.1 Collections (admin-managed)
+
+| Method | Path | Body / Notes |
+|---|---|---|
+| `POST` | `/collections` | `{name, kind:"vector"\|"json", vector_dimension?, embedding_model?}` → creates in DB + registry. `vector_dimension` required when `kind=="vector"`. |
+| `GET` | `/collections` | List managed collections (registry ⨝ `getCollectionInfo` stats). |
+| `GET` | `/collections/{name}` | Collection info + doc count + size. |
+| `DELETE` | `/collections/{name}` | Drop collection + registry entry. |
+
+### 6.2 JSON documents (full CRUD)
+
+| Method | Path | Notes |
+|---|---|---|
+| `POST` | `/collections/{name}/documents` | Insert (optional `id`); returns `{id}`. |
+| `GET` | `/collections/{name}/documents/{id}` | Get one. |
+| `GET` | `/collections/{name}/documents` | Find: query params `filter` (repeatable `field:op:value`), `limit`, `offset`, `sort`, `desc`. |
+| `PATCH` | `/collections/{name}/documents/{id}` | Server-side field merge (`patch`). |
+| `PUT` | `/collections/{name}/documents/{id}` | Replace whole doc (`update`). |
+| `DELETE` | `/collections/{name}/documents/{id}` | Remove. |
+
+### 6.3 Vectors / RAG
+
+| Method | Path | Body / Notes |
+|---|---|---|
+| `POST` | `/collections/{name}/vectors` | `{id?, text?, vector?, metadata?}`. If `vector` present → store as-is (dim-checked). Else if `text` → embed via the collection's model, then store `_vector` + payload. Returns `{id}`. |
+| `POST` | `/collections/{name}/search` | `{query_text? \| query_vector?, top_k=5, min_score=0.0, filters?}`. If `query_text` → embed first. Returns `[{id, score, data}]`. (RAG retrieval.) |
+
+### 6.4 Ops & meta
+
+| Method | Path | Auth | Notes |
+|---|---|---|---|
+| `GET` | `/healthz` | none | Liveness (process up). |
+| `GET` | `/readyz` | none | Readiness: DB connected. |
+| `GET` | `/openapi.json` | none | OpenAPI 3.1 spec. |
+| `GET` | `/llms.txt` | none | Agent-facing API summary (llmstxt.org format). |
+| `GET` | `/api/v1/stats` | bearer/cookie | `getStats` + `getMemoryStats` + collection list. |
+
+### 6.5 Web UI auth
+
+| Method | Path | Notes |
+|---|---|---|
+| `GET` | `/` (+ static assets) | Serves the React app (public assets; the app itself gates on session). |
+| `POST` | `/ui/login` | `{key}`; if `== settings.api_key`, set HttpOnly/SameSite=Strict cookie (TTL from settings). |
+| `POST` | `/ui/logout` | Clear the session cookie. |
+
+### 6.6 Error mapping
+
+| Condition | HTTP |
+|---|---|
+| Missing/invalid auth | 401 |
+| Unknown collection / document | 404 |
+| Validation error (bad body, dim mismatch, missing required field) | 422 |
+| DB unavailable / gRPC error | 503 |
+| Unexpected | 500 |
+
+Error body: `{ "error": { "code": "<slug>", "message": "<human readable>" } }`.
+
+## 7. Web UI (React)
+
+Stack mirrors `/data/callerai/webui`. Dev server (`vite`) proxies `/api` → the
+backend (`:8080`). Pages:
+
+- **Login** — enter the API key → cookie session.
+- **Dashboard** — DB stats (docs, collections, memory pressure, op latencies).
+- **Collections** — list; create (name, kind, dimension, embedding model); drop.
+- **Documents** — pick a collection, browse/search, view/create/edit/delete with
+  a **Monaco JSON editor**. Vector fields shown read-only.
+- **RAG Tester** — pick a vector collection, enter query text or vector, see
+  ranked results with scores.
+- **Settings** — edit the DB-backed settings (OpenAI base/key, default model,
+  CORS, session TTL, rotate API key) → saved to DB → hot-reloaded.
+
+## 8. Build system
+
+- Root `CMakeLists.txt`: C++20; `FetchContent` cpp-httplib (pinned release tag);
+  `find_package(smartbotic-db-client REQUIRED)`; OpenSSL; spdlog; nlohmann/json;
+  optional `find_package(systemd)` for `sd_notify`.
+- `cmake/WebUI.cmake`: option `BUILD_WEBUI=ON`. A custom target runs `npm ci`
+  (when `node_modules` is absent) + `npm run build -- --outDir ${CMAKE_BINARY_DIR}/webui/dist`,
+  wired into the default (`ALL`) build so a normal `cmake --build` produces the
+  UI in the build tree. Skippable for backend-only CI.
+- `CPPHTTPLIB_OPENSSL_SUPPORT` defined on the server target.
+- Install rules: binary → `bin`; `config/config.json` → `/etc/smartbotic-vectorapi`;
+  `systemd/*.service` → `/lib/systemd/system`; `api/{openapi.json,llms.txt}` →
+  `/usr/share/smartbotic-vectorapi`; webui dist → `/usr/share/smartbotic-vectorapi/webui`.
+
+## 9. Packaging & deployment
+
+Mirror `/data/callerai/packaging` and `smartbotic-database/packaging`.
+
+- **`smartbotic-vectorapi`** (`amd64`):
+  - `/usr/bin/smartbotic-vectorapi`
+  - `/etc/smartbotic-vectorapi/config.json` (conffile)
+  - `/lib/systemd/system/smartbotic-vectorapi.service`
+  - `/usr/share/smartbotic-vectorapi/{openapi.json,llms.txt}`
+  - **Depends:** `libsmartbotic-db-client (>= 2.2.0)`, `libssl3t64`, `libsystemd0`
+    (gRPC/protobuf pulled transitively via the client lib's dependencies).
+  - **Recommends:** `smartbotic-vectorapi-webui`.
+  - **postinst:** create `smartbotic-vectorapi` system user/group, `/var/lib/smartbotic-vectorapi`
+    (0750), `daemon-reload`, enable + start (restart on upgrade).
+  - **prerm/postrm:** stop/disable; purge removes the data dir + user.
+- **`smartbotic-vectorapi-webui`** (`all`):
+  - `/usr/share/smartbotic-vectorapi/webui/*` (Vite build output).
+- **systemd unit:** `Type=notify` + `sd_notify(READY=1)` + `WATCHDOG`,
+  `After=smartbotic-database.service`, dedicated user, callerai-style hardening
+  (`NoNewPrivileges`, `ProtectSystem=strict`, `ReadWritePaths=/var/lib/smartbotic-vectorapi`,
+  `SystemCallFilter=@system-service`, etc.; outbound network allowed for OpenAI).
+- **`packaging/build.sh`:** `--local` (native cmake build + `create-debs.sh`) and
+  default Docker Debian-13 (`Dockerfile.base` + `Dockerfile.build`). `--repo`/`--sync`
+  hand off to `/data/smartbotics-deb-repo` (`add-packages.sh` →
+  `create-repo.sh --suite trixie` → `sync-repo.sh`).
+
+## 10. `openapi.json` & `llms.txt`
+
+- `api/openapi.json` — OpenAPI 3.1, the source of truth for the REST surface
+  (served at `/openapi.json`).
+- `api/llms.txt` — llmstxt.org-style markdown (H1 title, summary blockquote,
+  endpoint sections with one-line descriptions) so agentic coding tools can
+  understand the API (served at `/llms.txt`).
+- Both are maintained in-repo and kept in sync with the handlers; the surface is
+  small enough that manual sync is acceptable. Shipped in the server `.deb`.
+
+## 11. Testing
+
+TDD where logic lives:
+
+- `config` parse + env override + defaults.
+- `auth`: bearer match, cookie validity/expiry, public-route bypass.
+- `settings_store`: build snapshot from a doc, atomic swap, env seeding, first-run
+  key generation (generate-once semantics).
+- `collection_registry`: create/list/drop round-trip.
+- `embeddings`: parse a well-formed OpenAI response; dimension mismatch → error.
+- request validation: bad bodies, missing required fields, dim mismatch → 422.
+
+Integration (against the locally-installed `smartbotic-database`): start the
+server on a test port → create a JSON collection → insert/get/patch/find/delete;
+create a vector collection → store a supplied vector and a generated one (OpenAI
+mocked with a local httplib server) → similarity search returns ranked results.
+
+## 12. Out of scope (YAGNI for v1)
+
+- Multiple API keys / per-key scopes / roles (single key is sufficient).
+- Non-OpenAI embedding backends.
+- Auto-creating collections on write (collections are admin-managed).
+- WebSocket / realtime UI updates.
+- Rate limiting (left to the reverse proxy / firewall layer).
+
+## 13. Open setup items
+
+- Local dev/test uses the already-installed `smartbotic-database` (v2.2.1) on
+  `localhost:9004`.
+- Git remote configured but no push until the user requests it.