docs: add README, CLAUDE.md, and docs/api.md API reference

CLAUDE.md

@@ -0,0 +1,63 @@
+# CLAUDE.md — smartbotic-vectorapi
+
+C++20 REST API fronting `smartbotic-database` (gRPC) for n8n + other clients. Stores RAG vectors + JSON, organized into **projects** with **multi-key per-project-grant auth**. React admin UI. Ships as Debian `.deb`s.
+
+## Build / test / run
+
+```bash
+# Backend + web UI (Release)
+cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_WEBUI=ON -S .
+cmake --build build -j"$(nproc)"
+
+# Tests (GoogleTest). Integration tests need a running smartbotic-database on localhost:9004;
+# they GTEST_SKIP() cleanly if it's unreachable. BUILD_WEBUI=OFF to skip the npm build during test cycles.
+cmake -B build -G Ninja -DBUILD_TESTS=ON -DBUILD_WEBUI=OFF -S . && cmake --build build -j"$(nproc)"
+ctest --test-dir build --output-on-failure
+
+# Run locally (use the repo's api/ for /openapi.json,/llms.txt,/docs and the built UI)
+SMARTBOTIC_VECTORAPI_KEY=dev ./build/src/smartbotic-vectorapi \
+  --config config/config.json --share-dir "$PWD/api" --webui-dir "$PWD/build/webui/dist"
+
+# Web UI only
+cd webui && npm install && npm run build   # → webui/dist ; `npm run dev` for the :3000 proxy dev server
+```
+
+## Architecture (where things live)
+
+- `src/` — the server. `vectorapi_core` static lib + `main.cpp`. Namespace `svapi`. Flat headers, quoted includes.
+  - `db_gateway` (`qualify(project,coll)` + `DbGateway` over `smartbotic::database::Client`), `settings_store` (global settings, hot-reload), `key_store` (API keys, hot-reload, bootstrap admin key), `collection_registry` (per-project metadata), `embeddings` (OpenAI), `auth` (sessions, bearer/cookie), `server` (ApiServer + auth helpers), `handlers/*` (one file per route group).
+  - Auth model: pre-routing **authenticates** `/api/*` (401); each handler **authorizes** via `requireKey` → `requireAdmin` / `requireProjectAccess` (403). Keys are referenced by a stable **`id`** (not the secret).
+- `webui/` — React 19 + Vite + TS (strict, `erasableSyntaxOnly`) + Tailwind + Monaco. Cookie-session auth (`credentials:'include'`). Pages read `currentProject`/`admin` from `useAuthStore`; all calls go through `src/api/client.ts`.
+- `api/` — `openapi.json` (3.1), `llms.txt` (llmstxt.org), `docs/` (Redoc page). Served at `/openapi.json`, `/llms.txt`, `/docs`. Shipped in the server `.deb`. **Keep these in sync when changing routes.**
+- `packaging/` — `build.sh` (`--local` + Docker trixie), `deb/create-debs.sh`, control templates, systemd unit, maintainer scripts. `docs/superpowers/` — design spec (esp. **Addendum A**) + the 3 implementation plans.
+
+## Conventions
+
+- C++20, `nlohmann/json`, `spdlog`, cpp-httplib (FetchContent, OpenSSL on). Handlers `throw svapi::ApiError`; a global exception handler maps to `{error:{code,message}}` + HTTP status (`errors.hpp`: 400/401/403/404/422/503/500).
+- Filters: `field:op:value` (`op` ∈ eq/ne/gt/gte/lt/lte/in/contains/exists/regex/search).
+- TS: no `any`, no TS `enum`, no constructor parameter-property shorthand (tsconfig `erasableSyntaxOnly`). Iterating parsed JSON: bind to a named var first (range-for over `nlohmann::json::parse(...)[...]` dangles).
+- Commits: conventional prefixes (`feat`/`fix`/`build`/`docs`/`test`). Branch for feature work; `main` is pushed to `ssh://git.smartbotics.ai:10022/fszontagh/smartbotic-vectorapi.git`.
+
+## Data model & runtime config
+
+- Bootstrap `config.json` is minimal: `log_level`, `http.{bind_address,port}`, `database.address`. Everything else lives in the DB and **hot-reloads** (subscribe + atomic snapshot swap).
+- Global service collections (DB `default` project, **no leading underscore** — see gotcha): `vectorapi_keys`, `vectorapi_settings`. Per-project registry: `<project>:vectorapi_collections`. The `vectorapi_` prefix is reserved (collection-create rejects `_` and `vectorapi_`).
+- First run with no keys generates an admin key (`projects:["*"], admin:true`) and logs it once; `SMARTBOTIC_VECTORAPI_KEY` seeds it, `OPENAI_API_KEY` seeds embeddings.
+
+## ⚠️ Gotchas (hard-won — don't relearn these)
+
+- **DB v2.3.1: `_`-prefixed collections misroute when project-qualified.** Writes land in MemoryStore (mirror skips `_`-names) but reads of `default:_foo` route to LMDB → null. → service collections use **no leading underscore** (`vectorapi_*`). Suspected DB bug.
+- **DB v2.3.1: `dropProject` does not physically purge collection data.** A recycled project name is otherwise blocked by stale collections/registry → the project-delete handler drops the registry + collections first, and collection-create drops invisible orphans (after the registry "exists" check).
+- **DB client strips an `id` field from stored doc bodies.** So API keys store their stable id as `key_id` in the body (exposed as `id` externally); document handlers strip `id` before insert and use the separate id param.
+- **Debian 13 (trixie) gRPC cmake config is broken** unless `protobuf-compiler-grpc` + `libgrpc++-dev` + `libprotobuf-dev` are installed — `find_package(gRPC CONFIG)` (used transitively by `smartbotic-db-clientConfig.cmake`) fails on a `gRPCTargets.cmake` referencing missing plugin files. The Docker base installs them.
+- **Abseil ABI:** `smartbotic-database`/client debs must be built against the system's current Abseil; a stale build crash-loops on `libabsl_*.so.<old>`. Rebuild the DB if it won't start.
+- **Monaco** is bundled locally (`webui/src/monacoSetup.ts`) so the editor works offline — do not revert to the CDN loader.
+- systemd unit is `Type=notify`; the binary must be built with `libsystemd` (it is, when `libsystemd-dev` is present) or change to `Type=exec`.
+
+## Deploy / publish
+
+```bash
+./packaging/build.sh                                # Debian-13 .debs → dist/debian13/
+./packaging/build.sh --repo --suite trixie --sync   # + publish to repository.smartbotics.ai (outward-facing!)
+```
+Publishing uploads to the live apt repo — only with explicit confirmation. The deb-repo tooling is `/data/smartbotics-deb-repo/scripts/`.

README.md

@@ -0,0 +1,119 @@
+# smartbotic-vectorapi
+
+A general-purpose, client-agnostic **REST API in front of [`smartbotic-database`](https://git.smartbotics.ai/fszontagh/smartbotic-database)** for storing data from automation workflows. It stores **RAG embedding vectors** (with cosine similarity search) and **arbitrary JSON documents**, organized into **projects** (isolated namespaces) and protected by **API keys with per-project grants**.
+
+n8n workflows are the primary client, but any HTTP client can use it. A React admin console ships alongside.
+
+- **Multi-project** — collections live in isolated project namespaces, addressed as a URL path segment (`/api/v1/projects/{project}/...`), like MySQL databases.
+- **Multi-key auth** — each API key is granted access to one or more projects (or all, `*`); `admin` keys manage projects and keys. Bearer-token for the API; cookie session for the web UI.
+- **RAG** — store a precomputed vector, or send text and have it embedded via OpenAI; then run cosine similarity search.
+- **JSON CRUD** — full create/read/update/patch/delete with filtered find.
+- **Dynamic config** — only a minimal bootstrap `config.json` on disk; everything else (keys, OpenAI settings, CORS, etc.) lives in the database and **hot-reloads** without a restart.
+- **Self-describing** — `GET /openapi.json` (OpenAPI 3.1), `GET /llms.txt` (agent-friendly summary), and a rendered docs page at `/docs`.
+
+## Architecture
+
+```
+  n8n / clients ──HTTP+Bearer──┐
+                               ▼
+  admin browser ──cookie──▶ smartbotic-vectorapi ──gRPC──▶ smartbotic-database
+                               │  (libsmartbotic-db-client)
+                               └──HTTPS──▶ api.openai.com/v1/embeddings
+```
+
+The server is C++20 (cpp-httplib, OpenSSL). Project multi-tenancy is threaded at the gateway boundary via `qualify(project, collection)`; authorization (key → project grants) is the API's own layer. The web UI is React 19 + Vite + TypeScript + TailwindCSS + Monaco, served by the same binary (same-origin) or via the Vite dev proxy.
+
+| Component | Path |
+|---|---|
+| Server | `src/` → `smartbotic-vectorapi` binary |
+| Web UI | `webui/` (React) → built assets |
+| API spec | `api/openapi.json`, `api/llms.txt` |
+| Packaging | `packaging/` (Debian `.deb`, systemd, Docker) |
+| Design + plans | `docs/superpowers/specs/`, `docs/superpowers/plans/` |
+| API reference | `docs/api.md` |
+
+## Install (Debian 13 / trixie)
+
+The packages are published to the smartbotics apt repository.
+
+```bash
+# one-time repo setup (see /data/smartbotics-deb-repo for credentials)
+curl -fsSL https://repository.smartbotics.ai/add-repo.sh | sudo bash -s -- --user <user> --password <pass>
+
+sudo apt update
+sudo apt install smartbotic-vectorapi smartbotic-vectorapi-webui
+```
+
+On first start the service generates an **admin API key** and logs it **once**:
+
+```bash
+journalctl -u smartbotic-vectorapi | grep "Generated initial admin API key"
+```
+
+Optionally seed secrets before first boot in `/etc/smartbotic-vectorapi/vectorapi.env`:
+
+```ini
+SMARTBOTIC_VECTORAPI_KEY=<your-admin-key>     # seed the admin key instead of auto-generating
+OPENAI_API_KEY=sk-...                          # enables text→embedding generation for RAG
+```
+
+The service listens on `:8080` (configurable). The web console is at `http://<host>:8080/`; the rendered API docs are at `/docs`.
+
+## Build from source
+
+Requires: `cmake`, `ninja`, `g++` (C++20), `pkg-config`, `libsmartbotic-db-client-dev (>= 2.3.0)`, `nlohmann-json3-dev`, `libspdlog-dev`, `libssl-dev`, `libsystemd-dev`, and `nodejs`/`npm` (for the web UI). cpp-httplib is fetched via CMake.
+
+```bash
+# backend + web UI
+cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_WEBUI=ON -S .
+cmake --build build -j"$(nproc)"
+
+# run the tests (needs a running smartbotic-database on localhost:9004 for the integration tests)
+cmake -B build -G Ninja -DBUILD_TESTS=ON -DBUILD_WEBUI=OFF -S . && cmake --build build && ctest --test-dir build
+
+# run locally (point --webui-dir at the built assets)
+SMARTBOTIC_VECTORAPI_KEY=dev ./build/src/smartbotic-vectorapi \
+  --config config/config.json --share-dir "$PWD/api" --webui-dir "$PWD/build/webui/dist"
+```
+
+The web UI alone:
+
+```bash
+cd webui && npm install && npm run dev      # dev server on :3000, proxies /api + /ui to :8080
+cd webui && npm run build                   # production build → dist/
+```
+
+## Packaging
+
+```bash
+./packaging/build.sh --local                 # native .debs for the current system → dist/local/
+./packaging/build.sh                          # Debian-13 .debs via Docker → dist/debian13/
+./packaging/build.sh --repo --suite trixie --sync   # build + publish to repository.smartbotics.ai
+```
+
+Produces two packages: `smartbotic-vectorapi` (server + systemd unit + config + `openapi.json`/`llms.txt` + rendered `/docs`) and `smartbotic-vectorapi-webui` (the built admin console).
+
+## Quick API tour
+
+```bash
+KEY=<admin-key>; BASE=http://localhost:8080
+
+# create a project + a vector collection (OpenAI text-embedding-3-small = 1536 dims)
+curl -s -X POST $BASE/api/v1/projects -H "Authorization: Bearer $KEY" \
+  -H 'Content-Type: application/json' -d '{"name":"acme"}'
+curl -s -X POST $BASE/api/v1/projects/acme/collections -H "Authorization: Bearer $KEY" \
+  -H 'Content-Type: application/json' \
+  -d '{"name":"memories","kind":"vector","vector_dimension":1536,"embedding_model":"text-embedding-3-small"}'
+
+# store a RAG item from text (embedded server-side) and search it
+curl -s -X POST $BASE/api/v1/projects/acme/collections/memories/vectors -H "Authorization: Bearer $KEY" \
+  -H 'Content-Type: application/json' -d '{"text":"the user prefers dark mode","metadata":{"src":"n8n"}}'
+curl -s -X POST $BASE/api/v1/projects/acme/collections/memories/search -H "Authorization: Bearer $KEY" \
+  -H 'Content-Type: application/json' -d '{"query_text":"dark mode","top_k":5}'
+```
+
+Full reference: [`docs/api.md`](docs/api.md), the live `/docs` page, and `GET /openapi.json`.
+
+## License
+
+Proprietary — Smartbotics AI.

docs/api.md

@@ -0,0 +1,175 @@
+# smartbotic-vectorapi — API Reference
+
+Version 0.1.0. This is the human-readable reference. Machine-readable equivalents
+are served live by the running service and shipped in the package:
+
+- **`GET /openapi.json`** — OpenAPI 3.1 spec (source: `api/openapi.json`).
+- **`GET /llms.txt`** — a compact, agent-friendly summary following the
+  [llmstxt.org](https://llmstxt.org) convention (source: `api/llms.txt`).
+- **`GET /docs`** — this reference rendered as an interactive page (Redoc).
+
+## Base & authentication
+
+- Base path: **`/api/v1`**. All `/api/v1/*` routes require authentication.
+- **API clients** send an API key as a bearer token: `Authorization: Bearer <KEY>`.
+- **The web UI** logs in with a key (`POST /ui/login`) and rides a cookie session
+  thereafter (`credentials: include`).
+- Public (no auth): `GET /healthz`, `GET /readyz`, `GET /openapi.json`,
+  `GET /llms.txt`, `GET /docs`, and the static UI.
+
+### Authorization model (MySQL-like grants)
+
+Each API key carries grants: a set of project names it may access (or `"*"` for
+all) plus an `admin` flag.
+
+| Route class | Requirement | On failure |
+|---|---|---|
+| `/api/v1/projects/{project}/…` (collections, documents, vectors, search, project stats) | key is `admin`, or holds `"*"`, or lists `{project}` | **403** |
+| `POST/DELETE /api/v1/projects`, all `/api/v1/keys`, `GET /api/v1/stats` | `admin` | **403** |
+| `GET /api/v1/projects` | any valid key (result filtered to granted projects) | — |
+| any `/api/v1/*` with a missing/unknown key | — | **401** |
+
+On first run with no keys, the service generates one **admin** key
+(`projects:["*"], admin:true`) and logs it once.
+
+### Errors
+
+All errors return JSON `{"error": {"code": "<slug>", "message": "<text>"}}` with a
+standard HTTP status: `400` malformed request, `401` unauthenticated, `403`
+forbidden (valid key lacking the grant), `404` not found, `422` validation
+(bad body, dimension mismatch, reserved name), `503` database/embedding
+unavailable, `500` unexpected.
+
+---
+
+## Projects
+
+A project is an isolated namespace (collections in one project are invisible to
+others). `default` always exists and cannot be deleted.
+
+| Method | Path | Auth | Body / notes |
+|---|---|---|---|
+| `GET` | `/api/v1/projects` | any key | → `{"projects": ["default", ...]}` (only granted projects) |
+| `POST` | `/api/v1/projects` | admin | `{"name":"acme"}` — name `^[a-zA-Z_][a-zA-Z0-9_-]{0,62}$` → `201 {"name":"acme"}` |
+| `GET` | `/api/v1/projects/{project}` | project access | → `{"name","collections"}` |
+| `DELETE` | `/api/v1/projects/{project}` | admin | drops the project + its collections (not `default`) |
+
+```bash
+curl -s -X POST $BASE/api/v1/projects -H "Authorization: Bearer $KEY" \
+  -H 'Content-Type: application/json' -d '{"name":"acme"}'
+```
+
+## API keys (admin only)
+
+Keys are referenced by a stable, non-secret **`id`** (from the list); the secret
+value is only shown once, at creation.
+
+| Method | Path | Body / notes |
+|---|---|---|
+| `GET` | `/api/v1/keys` | → `{"keys":[{id,label,projects,admin,created_at,key_prefix}]}` (secret masked) |
+| `POST` | `/api/v1/keys` | `{"label":"n8n","projects":["acme"],"admin":false}` → `201` with the full `key` **once** |
+| `PATCH` | `/api/v1/keys/{id}` | `{label?,projects?,admin?}` — update grants |
+| `DELETE` | `/api/v1/keys/{id}` | revoke (rejected with `422` if it's the last admin) |
+
+```bash
+# create a scoped key for n8n, limited to project "acme"
+curl -s -X POST $BASE/api/v1/keys -H "Authorization: Bearer $ADMIN" \
+  -H 'Content-Type: application/json' \
+  -d '{"label":"n8n-acme","projects":["acme"],"admin":false}'
+# → {"id":"...","label":"n8n-acme","projects":["acme"],"admin":false,"key":"<shown once>"}
+```
+
+## Collections
+
+Collections are **admin-managed**: create one before writing. `kind` is `json`
+(documents) or `vector` (RAG; fixed `vector_dimension`). Names may not start with
+`_` or the reserved `vectorapi_` prefix.
+
+| Method | Path | Body / notes |
+|---|---|---|
+| `POST` | `/api/v1/projects/{p}/collections` | `{"name","kind":"json"\|"vector","vector_dimension"?,"embedding_model"?}` |
+| `GET` | `/api/v1/projects/{p}/collections` | → `{"collections":[{name,kind,vector_dimension,embedding_model,document_count,size_bytes}]}` |
+| `GET` | `/api/v1/projects/{p}/collections/{name}` | one collection's info |
+| `DELETE` | `/api/v1/projects/{p}/collections/{name}` | drop it |
+
+`embedding_model` (vector collections) is the OpenAI model used when storing/
+searching by text; its output dimension must equal `vector_dimension`
+(`text-embedding-3-small` = 1536, `text-embedding-3-large` = 3072).
+
+```bash
+curl -s -X POST $BASE/api/v1/projects/acme/collections -H "Authorization: Bearer $KEY" \
+  -H 'Content-Type: application/json' \
+  -d '{"name":"memories","kind":"vector","vector_dimension":1536,"embedding_model":"text-embedding-3-small"}'
+```
+
+## JSON documents
+
+Under `/api/v1/projects/{p}/collections/{name}/documents`.
+
+| Method | Path | Body / notes |
+|---|---|---|
+| `POST` | `.../documents` | `{"id"?,"data":{...}}` (or the raw doc) → `201 {"id"}` |
+| `GET` | `.../documents/{id}` | the document |
+| `GET` | `.../documents` | find — query params below → `{"documents":[...],"count"}` |
+| `PATCH` | `.../documents/{id}` | merge fields → `{"id","version"}` |
+| `PUT` | `.../documents/{id}` | replace the whole document |
+| `DELETE` | `.../documents/{id}` | delete |
+
+**Find query params:** `filter` (repeatable, `field:op:value`, AND-combined),
+`limit`, `offset`, `sort`, `desc` (`true`/`false`). Operators (`op`): `eq`, `ne`,
+`gt`, `gte`, `lt`, `lte`, `in`, `contains`, `exists`, `regex`, `search`. The value
+is parsed as JSON when possible, else as a string.
+
+```bash
+curl -s -X POST $BASE/api/v1/projects/acme/collections/users/documents -H "Authorization: Bearer $KEY" \
+  -H 'Content-Type: application/json' -d '{"data":{"name":"Alice","age":30}}'
+
+curl -s "$BASE/api/v1/projects/acme/collections/users/documents?filter=age:gte:18&filter=name:search:Ali&limit=20&sort=age&desc=true" \
+  -H "Authorization: Bearer $KEY"
+```
+
+## RAG: vectors & similarity search
+
+Under `/api/v1/projects/{p}/collections/{name}` (vector collections only).
+
+| Method | Path | Body |
+|---|---|---|
+| `POST` | `.../vectors` | `{"id"?,"text"?,"vector"?,"metadata"?}` — provide `vector` to store as-is, or `text` to embed (OpenAI) → `201 {"id"}` |
+| `POST` | `.../search` | `{"query_text"?\|"query_vector"?,"top_k":5,"min_score":0.0}` → `{"results":[{id,score,data}]}` ordered by descending cosine similarity |
+
+A provided/generated vector's length must equal the collection's
+`vector_dimension` (else `422`). Storing by `text` or searching by `query_text`
+requires `OPENAI_API_KEY` to be configured (else `422`/`503`).
+
+```bash
+curl -s -X POST $BASE/api/v1/projects/acme/collections/memories/vectors -H "Authorization: Bearer $KEY" \
+  -H 'Content-Type: application/json' -d '{"text":"user prefers dark mode","metadata":{"src":"n8n"}}'
+
+curl -s -X POST $BASE/api/v1/projects/acme/collections/memories/search -H "Authorization: Bearer $KEY" \
+  -H 'Content-Type: application/json' -d '{"query_text":"dark mode","top_k":5,"min_score":0.7}'
+```
+
+## Stats & ops
+
+| Method | Path | Auth | Returns |
+|---|---|---|---|
+| `GET` | `/api/v1/projects/{p}/stats` | project access | `{project, collections, documents}` |
+| `GET` | `/api/v1/stats` | admin | server-wide: documents, collections, memory, pressure level, project count |
+| `GET` | `/healthz` | public | `{"status":"ok"}` (liveness) |
+| `GET` | `/readyz` | public | `{"ready":bool}` — `200` if the database is reachable, else `503` |
+
+## Web UI session
+
+| Method | Path | Body / notes |
+|---|---|---|
+| `POST` | `/ui/login` | `{"key":"<API key>"}` → sets an HttpOnly `svapi_session` cookie; body `{ok,admin,projects}` |
+| `POST` | `/ui/logout` | clears the session cookie |
+
+---
+
+## n8n usage notes
+
+- Use an **HTTP Request** node with **Header Auth** / a generic `Authorization: Bearer <key>` header.
+- Give each workflow (or tenant) its own **project** and a **scoped key** granted only that project — so a leaked key can't reach other tenants' data.
+- For RAG: create a `vector` collection with the embedding model matching your dimension, `POST .../vectors` with `text` to ingest, and `POST .../search` with `query_text` to retrieve context for the LLM node.
+- For everything else: a `json` collection with `POST .../documents` (store) and `GET .../documents?filter=...` (query).