# Smartbotic Database

## Build

```bash
cmake -B build -G Ninja
cmake --build build -j$(nproc)
```

## Test

```bash
cmake --build build -j$(nproc) && ./build/tests/test_vector_storage
```

## Architecture

- `proto/database.proto` — gRPC service and message definitions
- `service/src/` — Database server implementation
  - `memory_store.hpp/.cpp` — In-memory document + vector storage
  - `document.hpp` — Document and CollectionOptions types
  - `database_grpc_impl.cpp` — gRPC request handlers
  - `persistence/` — WAL, snapshots, persistence manager
  - `migrations/` — Migration runner
- `client/` — C++ client library (`smartbotic-db-client`)
- `tests/` — Integration tests

## Key Features

- JSON document store with collections, version history, field-level encryption
- gRPC API with replication, events, file storage, migrations
- **Vector Storage** — Embedding vectors stored alongside documents with SIMD-accelerated (AVX2/SSE4.1) cosine similarity search via `SimilaritySearch` RPC. Vectors use `_vector` document field, stored in parallel float arrays, persisted through WAL (`VEC_PUT`/`VEC_DELETE`) and snapshots (v3 format). Collection `vector_dimension` option is immutable after creation.
- **Atomic Partial Updates** — `PatchDocument` RPC merges fields into existing documents atomically (server-side, within collection lock). Client `patch()` method. `update()` uses automatic optimistic locking with retry. See `docs/integration-guide.md` for concurrency patterns.
- **Views** — Read-only named projections over collections. Include/exclude field lists (dot-notation for nested paths), baked-in filters with all operators (EQ/NE/GT/GTE/LT/LTE/IN/CONTAINS/EXISTS/REGEX/SEARCH) AND-merged with caller filters, optional default sort that caller can override. Stored in `_views` system collection, created via `create_view` migration op or `createView` RPC. Writes on view names are rejected.
- **Per-collection Timestamp Precision** — collection-level `timestamp_precision` config ("ms" default, "ns" for rapid-write collections). Stamps `_created_at`/`_updated_at` at the configured resolution, cached hot-path lookup. `configureCollection` RPC flips the config atomically; `migrateCollectionTimestamps` RPC converts existing data during maintenance windows (idempotent via 10^15 threshold).
- **Durable Snapshots + Tiered Recovery** — atomic snapshot writer (`.tmp` + fsync + rename + post-write verification), loader fallback chain across snapshots, MySQL-style recovery modes (`normal` / `snapshot_fallback` / `wal_only` / `best_effort` / `force_empty`) selectable via `--recovery-mode` flag or `recovery.mode` config. Non-trivial recovery (fallback, WAL-only, forced empty) automatically enters **read-only mode** — operator must `smartbotic-db-cli unlock` or pass `--force-readwrite` to accept writes. Exit code `10` when recovery is refused.
- **gRPC Concurrency Cap** — bounded inbound memory via gRPC `ResourceQuota` + per-RPC-type stream limits (`Subscribe`, `UploadFile`, `DownloadFile`). Configurable under `storage.grpc`. Excess streams get `RESOURCE_EXHAUSTED` with operator log.
- **Drop-in config (conf.d)** — `/etc/smartbotic-database/conf.d/*.json` deep-merges over `config.json` using RFC 7396 semantics (arrays replace, objects merge). Lets consumer debs ship tuned settings without touching the upstream conffile. Strict on syntax errors (exit 11), lenient on unknown keys.
- **Chunked pressure-aware eviction** — Eviction spreads work over multiple ticks with a default `eviction_chunk_size=1000`, `eviction_chunk_pause_ms=50`, `hot_write_floor_ms=30000`. Four pressure levels (normal/soft/hard/emergency) gate behavior: soft = trickle, hard = aggressive + `MEMORY_PRESSURE_HIGH` event, emergency = admission control rejects writes with `RESOURCE_EXHAUSTED`. Per-collection `memory_priority` (low/normal/high) biases selection. Quiesce skips docs with in-flight writes. WAL-fallback no longer holds per-collection mutex during disk I/O.
- **GetMemoryStats RPC + client retry** — `Client::getMemoryStats()` returns per-collection stats + pressure level. Client library auto-retries write RPCs on `DEADLINE_EXCEEDED`/`RESOURCE_EXHAUSTED`/`UNAVAILABLE` with exponential backoff + jitter (`writeRetries=3`, `writeRetryBackoffMs=100`). Reads are not auto-retried.
- **Auto-backup before deb upgrade** — `apt upgrade smartbotic-database` runs a `preinst` hook that snapshots `/var/lib/smartbotic-database/` to `/var/backups/smartbotic-database/pre-upgrade-<ts>-from-<version>/` via `cp -a` after the old `prerm` stops the service. Retention `3`, 1.2× free-space precheck on the backup volume (aborts the upgrade if short). Opt out with `SMARTBOTIC_DB_SKIP_BACKUP=1 apt upgrade`. Tunables: `SMARTBOTIC_DB_BACKUP_RETENTION`, `SMARTBOTIC_DB_BACKUP_ROOT`.
- **yyjson hot-path parsing (v1.10.0+)** — every JSON parse on WAL replay, snapshot deserialize, history reads, gRPC request bodies, and replication apply goes through `smartbotic::db::parse_to_nlohmann` (yyjson-backed) instead of `nlohmann::json::parse`. Measured 2.80× speedup on Zoe-shape corpora (136ms→49ms on 10k docs). In-memory representation stays `nlohmann::json` until Phase B. Wire and on-disk JSON bytes unchanged. New runtime dep: `libyyjson0`. Bench: `./build/tests/bench_json_parse <corpus.jsonl>`.

## Packaging

Canonical source for all smartbotic-database Debian packages.
Replaces the old `shadowman-database` and `callerai-storage` packages.

### Build modes

```bash
# Production (Docker, Debian 13)
./packaging/build.sh                          # Build 4 .debs
./packaging/build.sh --rebuild-base           # Rebuild Docker base image
./packaging/build.sh --sync --suite trixie    # Build + publish to repository.smartbotics.ai

# Local development (native, current system)
./packaging/build.sh --local                  # Build 4 .debs
./packaging/build.sh --local --install        # Build + install locally
```

### Packages

| Package | Contents | Deployed where |
|---------|----------|---------------|
| `smartbotic-database` | Server binary + systemd + config | Production DB servers |
| `libsmartbotic-db-client` | Shared `.so` (runtime) | All production machines |
| `libsmartbotic-db-client-dev` | Headers + linker symlink + `.proto` + cmake config | Docker build images, dev workstations |
| `smartbotic-db-cli` | CLI tool | Anywhere (optional) |

### Local dev with installed packages

After `./packaging/build.sh --local --install`, consumer projects can use:

```cmake
find_package(smartbotic-db-client REQUIRED)
target_link_libraries(myapp PRIVATE smartbotic::db-client)
```

The submodule fallback still works for projects that haven't switched (`BUILD_SHARED_LIBS=OFF`).

### Consumers

| Project | Depends on | Min version | Migrations dir |
|---------|-----------|-------------|----------------|
| shadowman-cpp | `smartbotic-database (>= 1.2.0)` | 1.2.0 | `/opt/shadowman/share/shadowman/migrations/json` |
| callerai | `smartbotic-database (>= 1.2.0)` | 1.2.0 | `/etc/callerai/migrations` |

### Version policy

Bump `VERSION` for API/protocol changes. Deb revision (`-N`) for packaging-only changes.

## systemd integration (Type=notify, v1.7.5+)

The service uses **`Type=notify`** in its systemd unit and signals readiness
via `sd_notify(READY=1)` only after recovery + migrations complete and the
gRPC listener is up (see `service/src/main.cpp` around the `service.start()`
call). This is the lifecycle contract for dependents:

- `After=smartbotic-database.service` + `Type=notify` on this unit means
  systemd blocks the dependent's startup until the DB sends `READY=1`. A
  dependent doing `db.get(...)` immediately on launch sees a fully-recovered
  DB, never an in-recovery one. Pre-1.7.5 the unit was `Type=simple`, so
  `sd_notify` was decoration and consumers could observe an in-recovery DB
  and silently fall back to defaults — surfaced on the shadowman side as
  `instance_type` reading null and `Dev mode enabled` never logging until
  manual restart.
- `TimeoutStartSec=300` covers slow recoveries on large datasets. The
  service emits `EXTEND_TIMEOUT_USEC=600000000` (10 min) at each phase
  boundary as belt-and-braces against systemd's start watchdog.
- `STATUS=...` notifications are emitted at each phase ("Initializing —
  running recovery and migrations" → "Starting gRPC server" → "Ready") so
  `systemctl status smartbotic-database` shows where the service is during
  a slow boot.
- `sd_notify(STOPPING=1)` is emitted on graceful shutdown so dependents
  observe a draining state distinct from a crash. The graceful-shutdown
  path also takes a final snapshot (v1.8.0 hardening) so the next boot's
  recovery is `TrivialSuccess` rather than `WalOnlyReplay`.

**Dependents (shadowman-* services and any other consumer)** should:
- Set `After=smartbotic-database.service` and `Wants=smartbotic-database.service`.
- Set their own `Type=notify` if they have a meaningful "ready" boundary.
- Treat critical-config DB reads as fail-loud: a null result on a setting
  the service depends on is a startup-ordering bug, not a default fallback.

**Operator-facing tooling** (deb postinst, ansible, etc.) should not layer
its own port-poll or sleep loops on top of the unit. `systemctl restart
smartbotic-database` (or `systemctl start ... && systemctl start dependent`)
already blocks correctly until READY.

## Conventions

- C++20
- nlohmann/json for JSON handling
- spdlog for logging
- LZ4 for compression
- gRPC/Protobuf for API