# Smartbotic Database

## Build

```bash
cmake -B build -G Ninja
cmake --build build -j$(nproc)
```

## Test

```bash
cmake --build build -j$(nproc) && ./build/tests/test_vector_storage
```

## Architecture

- `proto/database.proto` — gRPC service and message definitions
- `service/src/` — Database server implementation
  - `memory_store.hpp/.cpp` — In-memory document + vector storage
  - `document.hpp` — Document and CollectionOptions types
  - `database_grpc_impl.cpp` — gRPC request handlers
  - `persistence/` — WAL, snapshots, persistence manager
  - `migrations/` — Migration runner
- `client/` — C++ client library (`smartbotic-db-client`)
- `tests/` — Integration tests

## Key Features

- JSON document store with collections, version history, field-level encryption
- gRPC API with replication, events, file storage, migrations
- **Vector Storage** — Embedding vectors stored alongside documents with SIMD-accelerated (AVX2/SSE4.1) cosine similarity search via `SimilaritySearch` RPC. Vectors use `_vector` document field, stored in parallel float arrays, persisted through WAL (`VEC_PUT`/`VEC_DELETE`) and snapshots (v3 format). Collection `vector_dimension` option is immutable after creation.
- **Atomic Partial Updates** — `PatchDocument` RPC merges fields into existing documents atomically (server-side, within collection lock). Client `patch()` method. `update()` uses automatic optimistic locking with retry. See `docs/integration-guide.md` for concurrency patterns.
- **Views** — Read-only named projections over collections. Include/exclude field lists (dot-notation for nested paths), baked-in filters with all operators (EQ/NE/GT/GTE/LT/LTE/IN/CONTAINS/EXISTS/REGEX/SEARCH) AND-merged with caller filters, optional default sort that caller can override. Stored in `_views` system collection, created via `create_view` migration op or `createView` RPC. Writes on view names are rejected.
- **Per-collection Timestamp Precision** — collection-level `timestamp_precision` config ("ms" default, "ns" for rapid-write collections). Stamps `_created_at`/`_updated_at` at the configured resolution, cached hot-path lookup. `configureCollection` RPC flips the config atomically; `migrateCollectionTimestamps` RPC converts existing data during maintenance windows (idempotent via 10^15 threshold).
- **Durable Snapshots + Tiered Recovery** — atomic snapshot writer (`.tmp` + fsync + rename + post-write verification), loader fallback chain across snapshots, MySQL-style recovery modes (`normal` / `snapshot_fallback` / `wal_only` / `best_effort` / `force_empty`) selectable via `--recovery-mode` flag or `recovery.mode` config. Non-trivial recovery (fallback, WAL-only, forced empty) automatically enters **read-only mode** — operator must `smartbotic-db-cli unlock` or pass `--force-readwrite` to accept writes. Exit code `10` when recovery is refused.
- **gRPC Concurrency Cap** — bounded inbound memory via gRPC `ResourceQuota` + per-RPC-type stream limits (`Subscribe`, `UploadFile`, `DownloadFile`). Configurable under `storage.grpc`. Excess streams get `RESOURCE_EXHAUSTED` with operator log.
- **Drop-in config (conf.d)** — `/etc/smartbotic-database/conf.d/*.json` deep-merges over `config.json` using RFC 7396 semantics (arrays replace, objects merge). Lets consumer debs ship tuned settings without touching the upstream conffile. Strict on syntax errors (exit 11), lenient on unknown keys.
- **Chunked pressure-aware eviction** — Eviction spreads work over multiple ticks with a default `eviction_chunk_size=1000`, `eviction_chunk_pause_ms=50`, `hot_write_floor_ms=30000`. Four pressure levels (normal/soft/hard/emergency) gate behavior: soft = trickle, hard = aggressive + `MEMORY_PRESSURE_HIGH` event, emergency = admission control rejects writes with `RESOURCE_EXHAUSTED`. Per-collection `memory_priority` (low/normal/high) biases selection. Quiesce skips docs with in-flight writes. WAL-fallback no longer holds per-collection mutex during disk I/O.
- **GetMemoryStats RPC + client retry** — `Client::getMemoryStats()` returns per-collection stats + pressure level. Client library auto-retries write RPCs on `DEADLINE_EXCEEDED`/`RESOURCE_EXHAUSTED`/`UNAVAILABLE` with exponential backoff + jitter (`writeRetries=3`, `writeRetryBackoffMs=100`). Reads are not auto-retried.
- **Auto-backup before deb upgrade** — `apt upgrade smartbotic-database` runs a `preinst` hook that snapshots `/var/lib/smartbotic-database/` to `/var/backups/smartbotic-database/pre-upgrade-<ts>-from-<version>/` via `cp -a` after the old `prerm` stops the service. Retention `3`, 1.2× free-space precheck on the backup volume (aborts the upgrade if short). Opt out with `SMARTBOTIC_DB_SKIP_BACKUP=1 apt upgrade`. Tunables: `SMARTBOTIC_DB_BACKUP_RETENTION`, `SMARTBOTIC_DB_BACKUP_ROOT`.
- **yyjson hot-path parsing (v1.10.0+)** — every JSON parse on WAL replay, snapshot deserialize, history reads, gRPC request bodies, and replication apply goes through `smartbotic::db::parse_to_nlohmann` (yyjson-backed) instead of `nlohmann::json::parse`. Measured 2.80× speedup on Zoe-shape corpora (136ms→49ms on 10k docs). In-memory representation stays `nlohmann::json` until Phase B. Wire and on-disk JSON bytes unchanged. New runtime dep: `libyyjson0`. Bench: `./build/tests/bench_json_parse <corpus.jsonl>`.
- **Binary-backed Document storage (v1.11.0+)** — `Document::data` is now a `smartbotic::db::doc_binary::Doc` (yyjson `mut_doc` wrapper) with a lazy `nlohmann::json` view cache. New accessor surface: `doc.field(name)` is the O(field-count) fast-path that reads one top-level field without materialising the full tree (the dominant read pattern), `doc.data()` is the lazy full-tree accessor, `doc.set_data(j)` re-encodes from an nlohmann tree. Wire and on-disk JSON bytes unchanged. **Honest note:** per-doc heap footprint is ~1.5% smaller than v1.10 (1618→1594 bytes on Zoe-shape) — the win is field-access CPU + Phase C scaffolding, not RSS. See `docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md` post-implementation amendment for the bench numbers. Bench: `./build/tests/bench_doc_memory --mode=<nlohmann|binary> [N]`.
- **Client API cleanup (v2.1.0)** — BREAKING. Removed the pre-v1.6 `smartbotic::database::QueryOptions` struct from `client/include/smartbotic/database/document.hpp` (dead code; the EQ-only `vector<pair<string, json>>` shape that predated FilterOp). Removed the no-options `Client::find(collection)` overload — pass `QueryOptions{}` explicitly. Added ergonomic factories on `Client::Filter`: `Filter::eq` / `ne` / `gt` / `gte` / `lt` / `lte` / `in` / `contains` / `exists` / `regex` / `search`. Callsites now read like the predicate they're asking: `find("users", {.filters = {Filter::gt("age", 18), Filter::in("role", {"admin", "owner"})}})`. SOVERSION stays at 2 (matches the v2.0.0 deb; no soname bump needed) — consumers must rebuild against v2.1 headers, but the soname is identical. `libsmartbotic-db-client (>= 2.1.0)` should be the floor in consumer deb deps.
- **LMDB storage substrate (v2.0.0+)** — Documents and vectors live in LMDB sub-databases under `<dataDir>/env/`. Each user collection becomes one sub-db `<collection>` for documents; collections with `vector_dimension > 0` get a sidecar `_vectors_<collection>` storing raw float32 bytes keyed by docId. `LmdbEnv` (RAII over `MDB_env*`) is opened in `DatabaseService::setupComponents()`; `LmdbDocumentStore` provides `put`/`get`/`del`/`scan` + `put_vector`/`del_vector`/`get_vector`. Dual-write mirror runs under `MemoryStore`'s per-collection lock (`mirrorWriteToDocStore` + `mirrorVectorToDocStore` invoked before `lock.unlock()` at every write site), so concurrent LMDB readers observe the same state as MemoryStore readers — no race window. On failure: ERROR log, `mirror_drift_count_` bumps, `mirror_healthy_` flips to false; read handlers (Exists/Get/Find) fall back to MemoryStore. Filter eval lifted to `service/src/storage/filter_eval.hpp` shared between substrates so divergence is attributable to data state, not code paths. **v2.0 read coverage:** Exists, Get, Find, SimilaritySearch all LMDB-first when `mirror_healthy_` + zero drift. SimilaritySearch iterates the `_vectors_<collection>` sub-db via `DocumentStore::scan_vectors` and scores with the AVX2/SSE4.1/scalar SIMD kernel extracted to `service/src/storage/cosine_simd.hpp`. **Deferred to v2.1:** history (`_history_<coll>`) / files (`_files`) / views (`_views`) sub-db migration, MemoryStore decommission, write-handler migration so handlers call `doc_store_->put()` directly. **Migration:** v1.x→v2.0 auto-migrates on boot (`--migrate-from-v1` opt-in / `--no-auto-migrate` opt-out) via `migrate_v1_to_v2`; backfill of any docs added before mirror wiring runs sync in `initialize()`. **Eviction config knobs** (`max_memory_mb`, `eviction_chunk_size`, etc.) are kept for back-compat with v1.x snapshots-still-loaded-into-MemoryStore but emit a deprecation WARN at boot. RSS is bounded by the LMDB mapsize + OS page cache, not by MemoryStore eviction. **Stress numbers** (`load_test_mixed` 30s / 4 writers / 8 readers / 32MB MemoryStore budget): 2385 writes/s, 5769 reads/s, p99 < 100ms, 0 hard failures, 240k ops total, 282MB persisted in LMDB. New runtime dep: `liblmdb0`.

## Packaging

Canonical source for all smartbotic-database Debian packages.
Replaces the old `shadowman-database` and `callerai-storage` packages.

### Build modes

```bash
# Production (Docker, Debian 13)
./packaging/build.sh                          # Build 4 .debs
./packaging/build.sh --rebuild-base           # Rebuild Docker base image
./packaging/build.sh --sync --suite trixie    # Build + publish to repository.smartbotics.ai

# Local development (native, current system)
./packaging/build.sh --local                  # Build 4 .debs
./packaging/build.sh --local --install        # Build + install locally
```

### Packages

| Package | Contents | Deployed where |
|---------|----------|---------------|
| `smartbotic-database` | Server binary + systemd + config | Production DB servers |
| `libsmartbotic-db-client` | Shared `.so` (runtime) | All production machines |
| `libsmartbotic-db-client-dev` | Headers + linker symlink + `.proto` + cmake config | Docker build images, dev workstations |
| `smartbotic-db-cli` | CLI tool | Anywhere (optional) |

### Local dev with installed packages

After `./packaging/build.sh --local --install`, consumer projects can use:

```cmake
find_package(smartbotic-db-client REQUIRED)
target_link_libraries(myapp PRIVATE smartbotic::db-client)
```

The submodule fallback still works for projects that haven't switched (`BUILD_SHARED_LIBS=OFF`).

### Consumers

| Project | Depends on | Min version | Migrations dir |
|---------|-----------|-------------|----------------|
| shadowman-cpp | `smartbotic-database (>= 1.2.0)` | 1.2.0 | `/opt/shadowman/share/shadowman/migrations/json` |
| callerai | `smartbotic-database (>= 1.2.0)` | 1.2.0 | `/etc/callerai/migrations` |

### Version policy

Bump `VERSION` for API/protocol changes. Deb revision (`-N`) for packaging-only changes.

## systemd integration (Type=notify, v1.7.5+)

The service uses **`Type=notify`** in its systemd unit and signals readiness
via `sd_notify(READY=1)` only after recovery + migrations complete and the
gRPC listener is up (see `service/src/main.cpp` around the `service.start()`
call). This is the lifecycle contract for dependents:

- `After=smartbotic-database.service` + `Type=notify` on this unit means
  systemd blocks the dependent's startup until the DB sends `READY=1`. A
  dependent doing `db.get(...)` immediately on launch sees a fully-recovered
  DB, never an in-recovery one. Pre-1.7.5 the unit was `Type=simple`, so
  `sd_notify` was decoration and consumers could observe an in-recovery DB
  and silently fall back to defaults — surfaced on the shadowman side as
  `instance_type` reading null and `Dev mode enabled` never logging until
  manual restart.
- `TimeoutStartSec=300` covers slow recoveries on large datasets. The
  service emits `EXTEND_TIMEOUT_USEC=600000000` (10 min) at each phase
  boundary as belt-and-braces against systemd's start watchdog.
- `STATUS=...` notifications are emitted at each phase ("Initializing —
  running recovery and migrations" → "Starting gRPC server" → "Ready") so
  `systemctl status smartbotic-database` shows where the service is during
  a slow boot.
- `sd_notify(STOPPING=1)` is emitted on graceful shutdown so dependents
  observe a draining state distinct from a crash. The graceful-shutdown
  path also takes a final snapshot (v1.8.0 hardening) so the next boot's
  recovery is `TrivialSuccess` rather than `WalOnlyReplay`.

**Dependents (shadowman-* services and any other consumer)** should:
- Set `After=smartbotic-database.service` and `Wants=smartbotic-database.service`.
- Set their own `Type=notify` if they have a meaningful "ready" boundary.
- Treat critical-config DB reads as fail-loud: a null result on a setting
  the service depends on is a startup-ordering bug, not a default fallback.

**Operator-facing tooling** (deb postinst, ansible, etc.) should not layer
its own port-poll or sleep loops on top of the unit. `systemctl restart
smartbotic-database` (or `systemctl start ... && systemctl start dependent`)
already blocks correctly until READY.

## Conventions

- C++20
- nlohmann/json for JSON handling
- spdlog for logging
- LZ4 for compression
- gRPC/Protobuf for API