# Smartbotic Database

## Build

```bash
cmake -B build -G Ninja
cmake --build build -j$(nproc)
```

## Test

```bash
cmake --build build -j$(nproc) && ./build/tests/test_vector_storage
```

## Architecture

- `proto/database.proto` — gRPC service and message definitions
- `service/src/` — Database server implementation
  - `memory_store.hpp/.cpp` — In-memory document + vector storage
  - `document.hpp` — Document and CollectionOptions types
  - `database_grpc_impl.cpp` — gRPC request handlers
  - `persistence/` — WAL, snapshots, persistence manager
  - `migrations/` — Migration runner
- `client/` — C++ client library (`smartbotic-db-client`)
- `tests/` — Integration tests

## Key Features

- JSON document store with collections, version history, field-level encryption
- gRPC API with replication, events, file storage, migrations
- **Vector Storage** — Embedding vectors stored alongside documents with SIMD-accelerated (AVX2/SSE4.1) cosine similarity search via `SimilaritySearch` RPC. Vectors use `_vector` document field, stored in parallel float arrays, persisted through WAL (`VEC_PUT`/`VEC_DELETE`) and snapshots (v3 format). Collection `vector_dimension` option is immutable after creation.
- **Atomic Partial Updates** — `PatchDocument` RPC merges fields into existing documents atomically (server-side, within collection lock). Client `patch()` method. `update()` uses automatic optimistic locking with retry. See `docs/integration-guide.md` for concurrency patterns.
- **Views** — Read-only named projections over collections. Include/exclude field lists (dot-notation for nested paths), baked-in filters with all operators (EQ/NE/GT/GTE/LT/LTE/IN/CONTAINS/EXISTS/REGEX/SEARCH) AND-merged with caller filters, optional default sort that caller can override. Stored in `_views` system collection, created via `create_view` migration op or `createView` RPC. Writes on view names are rejected.
- **Per-collection Timestamp Precision** — collection-level `timestamp_precision` config ("ms" default, "ns" for rapid-write collections). Stamps `_created_at`/`_updated_at` at the configured resolution, cached hot-path lookup. `configureCollection` RPC flips the config atomically; `migrateCollectionTimestamps` RPC converts existing data during maintenance windows (idempotent via 10^15 threshold).
- **Durable Snapshots + Tiered Recovery** — atomic snapshot writer (`.tmp` + fsync + rename + post-write verification), loader fallback chain across snapshots, MySQL-style recovery modes (`normal` / `snapshot_fallback` / `wal_only` / `best_effort` / `force_empty`) selectable via `--recovery-mode` flag or `recovery.mode` config. Non-trivial recovery (fallback, WAL-only, forced empty) automatically enters **read-only mode** — operator must `smartbotic-db-cli unlock` or pass `--force-readwrite` to accept writes. Exit code `10` when recovery is refused.
- **gRPC Concurrency Cap** — bounded inbound memory via gRPC `ResourceQuota` + per-RPC-type stream limits (`Subscribe`, `UploadFile`, `DownloadFile`). Configurable under `storage.grpc`. Excess streams get `RESOURCE_EXHAUSTED` with operator log.
- **Drop-in config (conf.d)** — `/etc/smartbotic-database/conf.d/*.json` deep-merges over `config.json` using RFC 7396 semantics (arrays replace, objects merge). Lets consumer debs ship tuned settings without touching the upstream conffile. Strict on syntax errors (exit 11), lenient on unknown keys.
- **Chunked pressure-aware eviction** — Eviction spreads work over multiple ticks with a default `eviction_chunk_size=1000`, `eviction_chunk_pause_ms=50`, `hot_write_floor_ms=30000`. Four pressure levels (normal/soft/hard/emergency) gate behavior: soft = trickle, hard = aggressive + `MEMORY_PRESSURE_HIGH` event, emergency = admission control rejects writes with `RESOURCE_EXHAUSTED`. Per-collection `memory_priority` (low/normal/high) biases selection. Quiesce skips docs with in-flight writes. WAL-fallback no longer holds per-collection mutex during disk I/O.
- **GetMemoryStats RPC + client retry** — `Client::getMemoryStats()` returns per-collection stats + pressure level. Client library auto-retries write RPCs on `DEADLINE_EXCEEDED`/`RESOURCE_EXHAUSTED`/`UNAVAILABLE` with exponential backoff + jitter (`writeRetries=3`, `writeRetryBackoffMs=100`). Reads are not auto-retried.
- **Auto-backup before deb upgrade** — `apt upgrade smartbotic-database` runs a `preinst` hook that snapshots `/var/lib/smartbotic-database/` to `/var/backups/smartbotic-database/pre-upgrade-<ts>-from-<version>/` via `cp -a` after the old `prerm` stops the service. Retention `3`, 1.2× free-space precheck on the backup volume (aborts the upgrade if short). Opt out with `SMARTBOTIC_DB_SKIP_BACKUP=1 apt upgrade`. Tunables: `SMARTBOTIC_DB_BACKUP_RETENTION`, `SMARTBOTIC_DB_BACKUP_ROOT`.
- **yyjson hot-path parsing (v1.10.0+)** — every JSON parse on WAL replay, snapshot deserialize, history reads, gRPC request bodies, and replication apply goes through `smartbotic::db::parse_to_nlohmann` (yyjson-backed) instead of `nlohmann::json::parse`. Measured 2.80× speedup on Zoe-shape corpora (136ms→49ms on 10k docs). In-memory representation stays `nlohmann::json` until Phase B. Wire and on-disk JSON bytes unchanged. New runtime dep: `libyyjson0`. Bench: `./build/tests/bench_json_parse <corpus.jsonl>`.
- **Binary-backed Document storage (v1.11.0+)** — `Document::data` is now a `smartbotic::db::doc_binary::Doc` (yyjson `mut_doc` wrapper) with a lazy `nlohmann::json` view cache. New accessor surface: `doc.field(name)` is the O(field-count) fast-path that reads one top-level field without materialising the full tree (the dominant read pattern), `doc.data()` is the lazy full-tree accessor, `doc.set_data(j)` re-encodes from an nlohmann tree. Wire and on-disk JSON bytes unchanged. **Honest note:** per-doc heap footprint is ~1.5% smaller than v1.10 (1618→1594 bytes on Zoe-shape) — the win is field-access CPU + Phase C scaffolding, not RSS. See `docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md` post-implementation amendment for the bench numbers. Bench: `./build/tests/bench_doc_memory --mode=<nlohmann|binary> [N]`.
- **v2.3.1 two bug fixes found by post-2.3.0 stress + replication tests.** (1) `MemoryStore::getEvictionCandidates` compared `doc.updatedAt` against a millisecond-scaled `hotWriteFloor`; v2.2.0's default-precision flip to nanoseconds made every doc look "always within the hot-write window" so eviction never fired. Pre-existing v2.2.0 regression — load_test_mixed was never re-run after v2.2.0 shipped. Fix: pick ms- or ns-scaled floor per-doc using the 10^15 magnitude heuristic. (2) `DatabaseService::applyReplicatedEntry` called `MemoryStore::loadDocument` which bypasses the persist callback, so the v2.3 LMDB mirror never fired for replicated entries. Followers got writes into MemoryStore but the LMDB env stayed empty; v2.3 LMDB-first reads on the follower returned 0 docs. Fix: explicitly drive `applyDualWriteMirror` through the project registry after `loadDocument`. Replication catch-up time on a 100-doc test: 65ms.
- **v2.3.0 multi-project namespaces.** Each smartbotic-database instance now hosts multiple `project` namespaces, each with its own collections. Wire form: `<project>:<collection>` (e.g. `my_app:users`); bare names like `"users"` auto-qualify to `"default:users"` so every v2.0-v2.2 client keeps working without code changes. Storage: one LMDB env per project at `<dataDir>/projects/<name>/env/`. On first v2.3 boot the legacy `<dataDir>/env/` is atomically renamed under `projects/default/env/` (idempotent; refuses to start if both exist). Server-side: `ProjectStoreRegistry` (`service/src/storage/project_store.{hpp,cpp}`) owns the per-project envs; `DatabaseService::docStore(project)` resolves the LMDB store for a given name; mirror routing is via a resolver callback the service hands to `MemoryStore::setDocumentStoreMirror(resolver, ...)`. Client-side: `Client::Config::project` defaults to `"default"`; every outgoing collection name is auto-prepended (`qualify(coll)`) unless it already contains `:`. Per-call overrides via the qualified form always win. New gRPC RPCs: `ListProjects`, `CreateProject(name)`, `DropProject(name)` — `default` cannot be dropped. CRUD is idempotent and validated against `^[a-zA-Z_][a-zA-Z0-9_-]{0,62}$` (no `:`, no `/`, no `_-prefix). Reserved: `default` (always present); `_-prefix` (system). Tests: `test_project_addressing` (23 parser assertions), `test_project_crud` (29 placeholder assertions). 14/14 ctest green. Stress: load_test_mixed still passes against the multi-project build.
- **v2.2.1 bug fixes from shadowman code review.** Three issues found by a code-review pass on the v2.0→v2.2.0 commit range, all now fixed: (1) `MemoryStore::updateIfVersion` was missing `extractVector` + `storeVector` + `mirrorVectorToDocStore` calls — optimistic-lock updates on vector collections left the vector mirror stale, so `SimilaritySearch` returned old vector scores while `Get` returned new doc bodies; (2) `DatabaseService::backfillIntoDocStore` walked every doc into LMDB but never set `_meta.schema_version=2`, so every boot re-backfilled the entire dataset (Zoe-scale ~1.85M docs = minutes of pre-READY blocking on every restart); (3) `SimilaritySearch` LMDB handler returned `INTERNAL` instead of `INVALID_ARGUMENT` when an unrelated throw (e.g. bad_alloc) fired after a dimension-mismatch was already observed in the scan. Exposes `smartbotic::db::storage::mark_migration_complete(env)` for the backfill marker. Tests 12/12 green.
- **v2.2.0 — default timestamp precision flipped to `"ns"`.** BREAKING for fresh collections. `CollectionCfg::timestampPrecision` default went from `"ms"` to `"ns"` so rapid-write collections (chat messages, metrics events, tool-call audits) get nanosecond stamps and strict ordering without needing an explicit `configureCollection()` call. Closes shadowman audit B5. Operators who need ms can still set it explicitly. Existing collections that already have a stored config record keep their value — only collections created post-upgrade WITHOUT an explicit `configureCollection()` inherit the new default. Mixed-precision rows in the same collection are read-correct via the 10^15 magnitude auto-detect; operators wanting clean rows should run `migrateCollectionTimestamps`. **`update()` docstring** updated to clarify: it OVERWRITES the doc; use `patch()` for partial mutations. Reflects shadowman audit B6.
- **v2.1.1 cleanup — no defensive fallback on LMDB throw.** Removed the `catch → fall back to MemoryStore` paths in Exists / Get / Find / SimilaritySearch. A live LMDB exception now returns gRPC `INTERNAL` to the client instead of silently being masked by MemoryStore. The gate-closed fallback (`mirror_healthy_=false`, system collection, env unopened) stays — that's correctness, since MemoryStore can legitimately be ahead of LMDB when a mirror write has failed. **Answers shadowman audit F3:** `usedWalFallback` / `walMatchCount` / `memoryMatchCount` / `memorySearchMicros` / `walSearchMicros` are guaranteed zero on the LMDB-served path. Any non-zero value is a real signal — it means the read fell back to MemoryStore (mirror unhealthy). Operators can treat `walMatchCount > 0` as a paging/SLA alert.
- **v2.1 audit answers (shadowman alignment).** F1 — v1.x auto-migration: stays in-binary through v2.4. Extracts to separate `smartbotic-database-migrate-v1` deb in v2.5 once every production host has booted v2.4 ≥ once. v2.x is the project's terminal arc, no v3 planned. F2 — `setReadOnly` / `lock()` / `unlock()` are dual-audience: operator-side via `smartbotic-db-cli unlock` and client-side via `Client::lock()` for self-defensive migrations. F3 — see above. F4 — `migrateCollectionTimestamps` cost benchmark deferred until shadowman starts D.1 (we haven't seen real traffic on the path yet); back-of-envelope ~1ms per 1k docs on Zoe-shape data, online-safe up to ~1M docs.
- **Client API cleanup (v2.1.0)** — BREAKING. Removed the pre-v1.6 `smartbotic::database::QueryOptions` struct from `client/include/smartbotic/database/document.hpp` (dead code; the EQ-only `vector<pair<string, json>>` shape that predated FilterOp). Removed the no-options `Client::find(collection)` overload — pass `QueryOptions{}` explicitly. Added ergonomic factories on `Client::Filter`: `Filter::eq` / `ne` / `gt` / `gte` / `lt` / `lte` / `in` / `contains` / `exists` / `regex` / `search`. Callsites now read like the predicate they're asking: `find("users", {.filters = {Filter::gt("age", 18), Filter::in("role", {"admin", "owner"})}})`. SOVERSION stays at 2 (matches the v2.0.0 deb; no soname bump needed) — consumers must rebuild against v2.1 headers, but the soname is identical. `libsmartbotic-db-client (>= 2.1.0)` should be the floor in consumer deb deps.
- **LMDB storage substrate (v2.0.0+)** — Documents and vectors live in LMDB sub-databases under `<dataDir>/env/`. Each user collection becomes one sub-db `<collection>` for documents; collections with `vector_dimension > 0` get a sidecar `_vectors_<collection>` storing raw float32 bytes keyed by docId. `LmdbEnv` (RAII over `MDB_env*`) is opened in `DatabaseService::setupComponents()`; `LmdbDocumentStore` provides `put`/`get`/`del`/`scan` + `put_vector`/`del_vector`/`get_vector`. Dual-write mirror runs under `MemoryStore`'s per-collection lock (`mirrorWriteToDocStore` + `mirrorVectorToDocStore` invoked before `lock.unlock()` at every write site), so concurrent LMDB readers observe the same state as MemoryStore readers — no race window. On failure: ERROR log, `mirror_drift_count_` bumps, `mirror_healthy_` flips to false; read handlers (Exists/Get/Find) fall back to MemoryStore. Filter eval lifted to `service/src/storage/filter_eval.hpp` shared between substrates so divergence is attributable to data state, not code paths. **v2.0 read coverage:** Exists, Get, Find, SimilaritySearch all LMDB-first when `mirror_healthy_` + zero drift. SimilaritySearch iterates the `_vectors_<collection>` sub-db via `DocumentStore::scan_vectors` and scores with the AVX2/SSE4.1/scalar SIMD kernel extracted to `service/src/storage/cosine_simd.hpp`. **Deferred to v2.1:** history (`_history_<coll>`) / files (`_files`) / views (`_views`) sub-db migration, MemoryStore decommission, write-handler migration so handlers call `doc_store_->put()` directly. **Migration:** v1.x→v2.0 auto-migrates on boot (`--migrate-from-v1` opt-in / `--no-auto-migrate` opt-out) via `migrate_v1_to_v2`; backfill of any docs added before mirror wiring runs sync in `initialize()`. **Eviction config knobs** (`max_memory_mb`, `eviction_chunk_size`, etc.) are kept for back-compat with v1.x snapshots-still-loaded-into-MemoryStore but emit a deprecation WARN at boot. RSS is bounded by the LMDB mapsize + OS page cache, not by MemoryStore eviction. **Stress numbers** (`load_test_mixed` 30s / 4 writers / 8 readers / 32MB MemoryStore budget): 2385 writes/s, 5769 reads/s, p99 < 100ms, 0 hard failures, 240k ops total, 282MB persisted in LMDB. New runtime dep: `liblmdb0`.

## Packaging

Canonical source for all smartbotic-database Debian packages.
Replaces the old `shadowman-database` and `callerai-storage` packages.

### Build modes

```bash
# Production (Docker, Debian 13)
./packaging/build.sh                          # Build 4 .debs
./packaging/build.sh --rebuild-base           # Rebuild Docker base image
./packaging/build.sh --sync --suite trixie    # Build + publish to repository.smartbotics.ai

# Local development (native, current system)
./packaging/build.sh --local                  # Build 4 .debs
./packaging/build.sh --local --install        # Build + install locally
```

### Packages

| Package | Contents | Deployed where |
|---------|----------|---------------|
| `smartbotic-database` | Server binary + systemd + config | Production DB servers |
| `libsmartbotic-db-client` | Shared `.so` (runtime) | All production machines |
| `libsmartbotic-db-client-dev` | Headers + linker symlink + `.proto` + cmake config | Docker build images, dev workstations |
| `smartbotic-db-cli` | CLI tool | Anywhere (optional) |

### Local dev with installed packages

After `./packaging/build.sh --local --install`, consumer projects can use:

```cmake
find_package(smartbotic-db-client REQUIRED)
target_link_libraries(myapp PRIVATE smartbotic::db-client)
```

The submodule fallback still works for projects that haven't switched (`BUILD_SHARED_LIBS=OFF`).

### Consumers

| Project | Depends on | Min version | Migrations dir |
|---------|-----------|-------------|----------------|
| shadowman-cpp | `smartbotic-database (>= 1.2.0)` | 1.2.0 | `/opt/shadowman/share/shadowman/migrations/json` |
| callerai | `smartbotic-database (>= 1.2.0)` | 1.2.0 | `/etc/callerai/migrations` |

### Version policy

Bump `VERSION` for API/protocol changes. Deb revision (`-N`) for packaging-only changes.

## systemd integration (Type=notify, v1.7.5+)

The service uses **`Type=notify`** in its systemd unit and signals readiness
via `sd_notify(READY=1)` only after recovery + migrations complete and the
gRPC listener is up (see `service/src/main.cpp` around the `service.start()`
call). This is the lifecycle contract for dependents:

- `After=smartbotic-database.service` + `Type=notify` on this unit means
  systemd blocks the dependent's startup until the DB sends `READY=1`. A
  dependent doing `db.get(...)` immediately on launch sees a fully-recovered
  DB, never an in-recovery one. Pre-1.7.5 the unit was `Type=simple`, so
  `sd_notify` was decoration and consumers could observe an in-recovery DB
  and silently fall back to defaults — surfaced on the shadowman side as
  `instance_type` reading null and `Dev mode enabled` never logging until
  manual restart.
- `TimeoutStartSec=300` covers slow recoveries on large datasets. The
  service emits `EXTEND_TIMEOUT_USEC=600000000` (10 min) at each phase
  boundary as belt-and-braces against systemd's start watchdog.
- `STATUS=...` notifications are emitted at each phase ("Initializing —
  running recovery and migrations" → "Starting gRPC server" → "Ready") so
  `systemctl status smartbotic-database` shows where the service is during
  a slow boot.
- `sd_notify(STOPPING=1)` is emitted on graceful shutdown so dependents
  observe a draining state distinct from a crash. The graceful-shutdown
  path also takes a final snapshot (v1.8.0 hardening) so the next boot's
  recovery is `TrivialSuccess` rather than `WalOnlyReplay`.

**Dependents (shadowman-* services and any other consumer)** should:
- Set `After=smartbotic-database.service` and `Wants=smartbotic-database.service`.
- Set their own `Type=notify` if they have a meaningful "ready" boundary.
- Treat critical-config DB reads as fail-loud: a null result on a setting
  the service depends on is a startup-ordering bug, not a default fallback.

**Operator-facing tooling** (deb postinst, ansible, etc.) should not layer
its own port-poll or sleep loops on top of the unit. `systemctl restart
smartbotic-database` (or `systemctl start ... && systemctl start dependent`)
already blocks correctly until READY.

## Conventions

- C++20
- nlohmann/json for JSON handling
- spdlog for logging
- LZ4 for compression
- gRPC/Protobuf for API