smartbotic-database/CLAUDE.md

Smartbotic Database

Build

cmake -B build -G Ninja
cmake --build build -j$(nproc)

Test

cmake --build build -j$(nproc) && ./build/tests/test_vector_storage

Architecture

• proto/database.proto — gRPC service and message definitions
• service/src/ — Database server implementation
  • memory_store.hpp/.cpp — In-memory document + vector storage
  • document.hpp — Document and CollectionOptions types
  • database_grpc_impl.cpp — gRPC request handlers
  • persistence/ — WAL, snapshots, persistence manager
  • migrations/ — Migration runner
• client/ — C++ client library (smartbotic-db-client)
• tests/ — Integration tests

Key Features

• JSON document store with collections, version history, field-level encryption
• gRPC API with replication, events, file storage, migrations
• Vector Storage — Embedding vectors stored alongside documents with SIMD-accelerated (AVX2/SSE4.1) cosine similarity search via SimilaritySearch RPC. Vectors use _vector document field, stored in parallel float arrays, persisted through WAL (VEC_PUT/VEC_DELETE) and snapshots (v3 format). Collection vector_dimension option is immutable after creation.
• Atomic Partial Updates — PatchDocument RPC merges fields into existing documents atomically (server-side, within collection lock). Client patch() method. update() uses automatic optimistic locking with retry. See docs/integration-guide.md for concurrency patterns.
• Views — Read-only named projections over collections. Include/exclude field lists (dot-notation for nested paths), baked-in filters with all operators (EQ/NE/GT/GTE/LT/LTE/IN/CONTAINS/EXISTS/REGEX/SEARCH) AND-merged with caller filters, optional default sort that caller can override. Stored in _views system collection, created via create_view migration op or createView RPC. Writes on view names are rejected.
• Per-collection Timestamp Precision — collection-level timestamp_precision config ("ms" default, "ns" for rapid-write collections). Stamps _created_at/_updated_at at the configured resolution, cached hot-path lookup. configureCollection RPC flips the config atomically; migrateCollectionTimestamps RPC converts existing data during maintenance windows (idempotent via 10^15 threshold).
• Durable Snapshots + Tiered Recovery — atomic snapshot writer (.tmp + fsync + rename + post-write verification), loader fallback chain across snapshots, MySQL-style recovery modes (normal / snapshot_fallback / wal_only / best_effort / force_empty) selectable via --recovery-mode flag or recovery.mode config. Non-trivial recovery (fallback, WAL-only, forced empty) automatically enters read-only mode — operator must smartbotic-db-cli unlock or pass --force-readwrite to accept writes. Exit code 10 when recovery is refused.
• gRPC Concurrency Cap — bounded inbound memory via gRPC ResourceQuota + per-RPC-type stream limits (Subscribe, UploadFile, DownloadFile). Configurable under storage.grpc. Excess streams get RESOURCE_EXHAUSTED with operator log.
• Drop-in config (conf.d) — /etc/smartbotic-database/conf.d/*.json deep-merges over config.json using RFC 7396 semantics (arrays replace, objects merge). Lets consumer debs ship tuned settings without touching the upstream conffile. Strict on syntax errors (exit 11), lenient on unknown keys.
• Chunked pressure-aware eviction — Eviction spreads work over multiple ticks with a default eviction_chunk_size=1000, eviction_chunk_pause_ms=50, hot_write_floor_ms=30000. Four pressure levels (normal/soft/hard/emergency) gate behavior: soft = trickle, hard = aggressive + MEMORY_PRESSURE_HIGH event, emergency = admission control rejects writes with RESOURCE_EXHAUSTED. Per-collection memory_priority (low/normal/high) biases selection. Quiesce skips docs with in-flight writes. WAL-fallback no longer holds per-collection mutex during disk I/O.
• GetMemoryStats RPC + client retry — Client::getMemoryStats() returns per-collection stats + pressure level. Client library auto-retries write RPCs on DEADLINE_EXCEEDED/RESOURCE_EXHAUSTED/UNAVAILABLE with exponential backoff + jitter (writeRetries=3, writeRetryBackoffMs=100). Reads are not auto-retried.
• Auto-backup before deb upgrade — apt upgrade smartbotic-database runs a preinst hook that snapshots /var/lib/smartbotic-database/ to /var/backups/smartbotic-database/pre-upgrade-<ts>-from-<version>/ via cp -a after the old prerm stops the service. Retention 3, 1.2× free-space precheck on the backup volume (aborts the upgrade if short). Opt out with SMARTBOTIC_DB_SKIP_BACKUP=1 apt upgrade. Tunables: SMARTBOTIC_DB_BACKUP_RETENTION, SMARTBOTIC_DB_BACKUP_ROOT.
• yyjson hot-path parsing (v1.10.0+) — every JSON parse on WAL replay, snapshot deserialize, history reads, gRPC request bodies, and replication apply goes through smartbotic::db::parse_to_nlohmann (yyjson-backed) instead of nlohmann::json::parse. Measured 2.80× speedup on Zoe-shape corpora (136ms→49ms on 10k docs). In-memory representation stays nlohmann::json until Phase B. Wire and on-disk JSON bytes unchanged. New runtime dep: libyyjson0. Bench: ./build/tests/bench_json_parse <corpus.jsonl>.
• Binary-backed Document storage (v1.11.0+) — Document::data is now a smartbotic::db::doc_binary::Doc (yyjson mut_doc wrapper) with a lazy nlohmann::json view cache. New accessor surface: doc.field(name) is the O(field-count) fast-path that reads one top-level field without materialising the full tree (the dominant read pattern), doc.data() is the lazy full-tree accessor, doc.set_data(j) re-encodes from an nlohmann tree. Wire and on-disk JSON bytes unchanged. Honest note: per-doc heap footprint is ~1.5% smaller than v1.10 (1618→1594 bytes on Zoe-shape) — the win is field-access CPU + Phase C scaffolding, not RSS. See docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md post-implementation amendment for the bench numbers. Bench: ./build/tests/bench_doc_memory --mode=<nlohmann|binary> [N].
• v2.3.1 two bug fixes found by post-2.3.0 stress + replication tests. (1) MemoryStore::getEvictionCandidates compared doc.updatedAt against a millisecond-scaled hotWriteFloor; v2.2.0's default-precision flip to nanoseconds made every doc look "always within the hot-write window" so eviction never fired. Pre-existing v2.2.0 regression — load_test_mixed was never re-run after v2.2.0 shipped. Fix: pick ms- or ns-scaled floor per-doc using the 10^15 magnitude heuristic. (2) DatabaseService::applyReplicatedEntry called MemoryStore::loadDocument which bypasses the persist callback, so the v2.3 LMDB mirror never fired for replicated entries. Followers got writes into MemoryStore but the LMDB env stayed empty; v2.3 LMDB-first reads on the follower returned 0 docs. Fix: explicitly drive applyDualWriteMirror through the project registry after loadDocument. Replication catch-up time on a 100-doc test: 65ms.
• v2.3.0 multi-project namespaces. Each smartbotic-database instance now hosts multiple project namespaces, each with its own collections. Wire form: <project>:<collection> (e.g. my_app:users); bare names like "users" auto-qualify to "default:users" so every v2.0-v2.2 client keeps working without code changes. Storage: one LMDB env per project at <dataDir>/projects/<name>/env/. On first v2.3 boot the legacy <dataDir>/env/ is atomically renamed under projects/default/env/ (idempotent; refuses to start if both exist). Server-side: ProjectStoreRegistry (service/src/storage/project_store.{hpp,cpp}) owns the per-project envs; DatabaseService::docStore(project) resolves the LMDB store for a given name; mirror routing is via a resolver callback the service hands to MemoryStore::setDocumentStoreMirror(resolver, ...). Client-side: Client::Config::project defaults to "default"; every outgoing collection name is auto-prepended (qualify(coll)) unless it already contains :. Per-call overrides via the qualified form always win. New gRPC RPCs: ListProjects, CreateProject(name), DropProject(name) — default cannot be dropped. CRUD is idempotent and validated against ^[a-zA-Z_][a-zA-Z0-9_-]{0,62}$ (no :, no /, no _-prefix). Reserved:default(always present);_-prefix(system). Tests:test_project_addressing(23 parser assertions),test_project_crud` (29 placeholder assertions). 14/14 ctest green. Stress: load_test_mixed still passes against the multi-project build.
• v2.2.1 bug fixes from shadowman code review. Three issues found by a code-review pass on the v2.0→v2.2.0 commit range, all now fixed: (1) MemoryStore::updateIfVersion was missing extractVector + storeVector + mirrorVectorToDocStore calls — optimistic-lock updates on vector collections left the vector mirror stale, so SimilaritySearch returned old vector scores while Get returned new doc bodies; (2) DatabaseService::backfillIntoDocStore walked every doc into LMDB but never set _meta.schema_version=2, so every boot re-backfilled the entire dataset (Zoe-scale ~1.85M docs = minutes of pre-READY blocking on every restart); (3) SimilaritySearch LMDB handler returned INTERNAL instead of INVALID_ARGUMENT when an unrelated throw (e.g. bad_alloc) fired after a dimension-mismatch was already observed in the scan. Exposes smartbotic::db::storage::mark_migration_complete(env) for the backfill marker. Tests 12/12 green.
• v2.2.0 — default timestamp precision flipped to "ns". BREAKING for fresh collections. CollectionCfg::timestampPrecision default went from "ms" to "ns" so rapid-write collections (chat messages, metrics events, tool-call audits) get nanosecond stamps and strict ordering without needing an explicit configureCollection() call. Closes shadowman audit B5. Operators who need ms can still set it explicitly. Existing collections that already have a stored config record keep their value — only collections created post-upgrade WITHOUT an explicit configureCollection() inherit the new default. Mixed-precision rows in the same collection are read-correct via the 10^15 magnitude auto-detect; operators wanting clean rows should run migrateCollectionTimestamps. update() docstring updated to clarify: it OVERWRITES the doc; use patch() for partial mutations. Reflects shadowman audit B6.
• v2.1.1 cleanup — no defensive fallback on LMDB throw. Removed the catch → fall back to MemoryStore paths in Exists / Get / Find / SimilaritySearch. A live LMDB exception now returns gRPC INTERNAL to the client instead of silently being masked by MemoryStore. The gate-closed fallback (mirror_healthy_=false, system collection, env unopened) stays — that's correctness, since MemoryStore can legitimately be ahead of LMDB when a mirror write has failed. Answers shadowman audit F3: usedWalFallback / walMatchCount / memoryMatchCount / memorySearchMicros / walSearchMicros are guaranteed zero on the LMDB-served path. Any non-zero value is a real signal — it means the read fell back to MemoryStore (mirror unhealthy). Operators can treat walMatchCount > 0 as a paging/SLA alert.
• v2.1 audit answers (shadowman alignment). F1 — v1.x auto-migration: stays in-binary through v2.4. Extracts to separate smartbotic-database-migrate-v1 deb in v2.5 once every production host has booted v2.4 ≥ once. v2.x is the project's terminal arc, no v3 planned. F2 — setReadOnly / lock() / unlock() are dual-audience: operator-side via smartbotic-db-cli unlock and client-side via Client::lock() for self-defensive migrations. F3 — see above. F4 — migrateCollectionTimestamps cost benchmark deferred until shadowman starts D.1 (we haven't seen real traffic on the path yet); back-of-envelope ~1ms per 1k docs on Zoe-shape data, online-safe up to ~1M docs.
• Client API cleanup (v2.1.0) — BREAKING. Removed the pre-v1.6 smartbotic::database::QueryOptions struct from client/include/smartbotic/database/document.hpp (dead code; the EQ-only vector<pair<string, json>> shape that predated FilterOp). Removed the no-options Client::find(collection) overload — pass QueryOptions{} explicitly. Added ergonomic factories on Client::Filter: Filter::eq / ne / gt / gte / lt / lte / in / contains / exists / regex / search. Callsites now read like the predicate they're asking: find("users", {.filters = {Filter::gt("age", 18), Filter::in("role", {"admin", "owner"})}}). SOVERSION stays at 2 (matches the v2.0.0 deb; no soname bump needed) — consumers must rebuild against v2.1 headers, but the soname is identical. libsmartbotic-db-client (>= 2.1.0) should be the floor in consumer deb deps.
• LMDB storage substrate (v2.0.0+) — Documents and vectors live in LMDB sub-databases under <dataDir>/env/. Each user collection becomes one sub-db <collection> for documents; collections with vector_dimension > 0 get a sidecar _vectors_<collection> storing raw float32 bytes keyed by docId. LmdbEnv (RAII over MDB_env*) is opened in DatabaseService::setupComponents(); LmdbDocumentStore provides put/get/del/scan + put_vector/del_vector/get_vector. Dual-write mirror runs under MemoryStore's per-collection lock (mirrorWriteToDocStore + mirrorVectorToDocStore invoked before lock.unlock() at every write site), so concurrent LMDB readers observe the same state as MemoryStore readers — no race window. On failure: ERROR log, mirror_drift_count_ bumps, mirror_healthy_ flips to false; read handlers (Exists/Get/Find) fall back to MemoryStore. Filter eval lifted to service/src/storage/filter_eval.hpp shared between substrates so divergence is attributable to data state, not code paths. v2.0 read coverage: Exists, Get, Find, SimilaritySearch all LMDB-first when mirror_healthy_ + zero drift. SimilaritySearch iterates the _vectors_<collection> sub-db via DocumentStore::scan_vectors and scores with the AVX2/SSE4.1/scalar SIMD kernel extracted to service/src/storage/cosine_simd.hpp. Deferred to v2.1: history (_history_<coll>) / files (_files) / views (_views) sub-db migration, MemoryStore decommission, write-handler migration so handlers call doc_store_->put() directly. Migration: v1.x→v2.0 auto-migrates on boot (--migrate-from-v1 opt-in / --no-auto-migrate opt-out) via migrate_v1_to_v2; backfill of any docs added before mirror wiring runs sync in initialize(). Eviction config knobs (max_memory_mb, eviction_chunk_size, etc.) are kept for back-compat with v1.x snapshots-still-loaded-into-MemoryStore but emit a deprecation WARN at boot. RSS is bounded by the LMDB mapsize + OS page cache, not by MemoryStore eviction. Stress numbers (load_test_mixed 30s / 4 writers / 8 readers / 32MB MemoryStore budget): 2385 writes/s, 5769 reads/s, p99 < 100ms, 0 hard failures, 240k ops total, 282MB persisted in LMDB. New runtime dep: liblmdb0.

Packaging

Canonical source for all smartbotic-database Debian packages. Replaces the old shadowman-database and callerai-storage packages.

Build modes

# Production (Docker, Debian 13)
./packaging/build.sh                          # Build 4 .debs
./packaging/build.sh --rebuild-base           # Rebuild Docker base image
./packaging/build.sh --sync --suite trixie    # Build + publish to repository.smartbotics.ai

# Local development (native, current system)
./packaging/build.sh --local                  # Build 4 .debs
./packaging/build.sh --local --install        # Build + install locally

Packages

Package	Contents	Deployed where
smartbotic-database	Server binary + systemd + config	Production DB servers
libsmartbotic-db-client	Shared .so (runtime)	All production machines
libsmartbotic-db-client-dev	Headers + linker symlink + .proto + cmake config	Docker build images, dev workstations
smartbotic-db-cli	CLI tool	Anywhere (optional)

Local dev with installed packages

After ./packaging/build.sh --local --install, consumer projects can use:

find_package(smartbotic-db-client REQUIRED)
target_link_libraries(myapp PRIVATE smartbotic::db-client)

The submodule fallback still works for projects that haven't switched (BUILD_SHARED_LIBS=OFF).

Consumers

Project	Depends on	Min version	Migrations dir
shadowman-cpp	smartbotic-database (>= 1.2.0)	1.2.0	/opt/shadowman/share/shadowman/migrations/json
callerai	smartbotic-database (>= 1.2.0)	1.2.0	/etc/callerai/migrations

Version policy

Bump VERSION for API/protocol changes. Deb revision (-N) for packaging-only changes.

systemd integration (Type=notify, v1.7.5+)

The service uses Type=notify in its systemd unit and signals readiness via sd_notify(READY=1) only after recovery + migrations complete and the gRPC listener is up (see service/src/main.cpp around the service.start() call). This is the lifecycle contract for dependents:

• After=smartbotic-database.service + Type=notify on this unit means systemd blocks the dependent's startup until the DB sends READY=1. A dependent doing db.get(...) immediately on launch sees a fully-recovered DB, never an in-recovery one. Pre-1.7.5 the unit was Type=simple, so sd_notify was decoration and consumers could observe an in-recovery DB and silently fall back to defaults — surfaced on the shadowman side as instance_type reading null and Dev mode enabled never logging until manual restart.
• TimeoutStartSec=300 covers slow recoveries on large datasets. The service emits EXTEND_TIMEOUT_USEC=600000000 (10 min) at each phase boundary as belt-and-braces against systemd's start watchdog.
• STATUS=... notifications are emitted at each phase ("Initializing — running recovery and migrations" → "Starting gRPC server" → "Ready") so systemctl status smartbotic-database shows where the service is during a slow boot.
• sd_notify(STOPPING=1) is emitted on graceful shutdown so dependents observe a draining state distinct from a crash. The graceful-shutdown path also takes a final snapshot (v1.8.0 hardening) so the next boot's recovery is TrivialSuccess rather than WalOnlyReplay.

Dependents (shadowman-* services and any other consumer) should:

• Set After=smartbotic-database.service and Wants=smartbotic-database.service.
• Set their own Type=notify if they have a meaningful "ready" boundary.
• Treat critical-config DB reads as fail-loud: a null result on a setting the service depends on is a startup-ordering bug, not a default fallback.

Operator-facing tooling (deb postinst, ansible, etc.) should not layer its own port-poll or sleep loops on top of the unit. systemctl restart smartbotic-database (or systemctl start ... && systemctl start dependent) already blocks correctly until READY.

Conventions

• C++20
• nlohmann/json for JSON handling
• spdlog for logging
• LZ4 for compression
• gRPC/Protobuf for API