smartbotic-database/CLAUDE.md

Smartbotic Database

Build

cmake -B build -G Ninja
cmake --build build -j$(nproc)

Test

cmake --build build -j$(nproc) && ./build/tests/test_vector_storage

Architecture

• proto/database.proto — gRPC service and message definitions
• service/src/ — Database server implementation
  • memory_store.hpp/.cpp — In-memory document + vector storage
  • document.hpp — Document and CollectionOptions types
  • database_grpc_impl.cpp — gRPC request handlers
  • persistence/ — WAL, snapshots, persistence manager
  • migrations/ — Migration runner
• client/ — C++ client library (smartbotic-db-client)
• tests/ — Integration tests

Key Features

• JSON document store with collections, version history, field-level encryption
• gRPC API with replication, events, file storage, migrations
• Vector Storage — Embedding vectors stored alongside documents with SIMD-accelerated (AVX2/SSE4.1) cosine similarity search via SimilaritySearch RPC. Vectors use _vector document field, stored in parallel float arrays, persisted through WAL (VEC_PUT/VEC_DELETE) and snapshots (v3 format). Collection vector_dimension option is immutable after creation.
• Atomic Partial Updates — PatchDocument RPC merges fields into existing documents atomically (server-side, within collection lock). Client patch() method. update() uses automatic optimistic locking with retry. See docs/integration-guide.md for concurrency patterns.
• Views — Read-only named projections over collections. Include/exclude field lists (dot-notation for nested paths), baked-in filters with all operators (EQ/NE/GT/GTE/LT/LTE/IN/CONTAINS/EXISTS/REGEX/SEARCH) AND-merged with caller filters, optional default sort that caller can override. Stored in _views system collection, created via create_view migration op or createView RPC. Writes on view names are rejected.
• Per-collection Timestamp Precision — collection-level timestamp_precision config ("ms" default, "ns" for rapid-write collections). Stamps _created_at/_updated_at at the configured resolution, cached hot-path lookup. configureCollection RPC flips the config atomically; migrateCollectionTimestamps RPC converts existing data during maintenance windows (idempotent via 10^15 threshold).
• Durable Snapshots + Tiered Recovery — atomic snapshot writer (.tmp + fsync + rename + post-write verification), loader fallback chain across snapshots, MySQL-style recovery modes (normal / snapshot_fallback / wal_only / best_effort / force_empty) selectable via --recovery-mode flag or recovery.mode config. Non-trivial recovery (fallback, WAL-only, forced empty) automatically enters read-only mode — operator must smartbotic-db-cli unlock or pass --force-readwrite to accept writes. Exit code 10 when recovery is refused.
• gRPC Concurrency Cap — bounded inbound memory via gRPC ResourceQuota + per-RPC-type stream limits (Subscribe, UploadFile, DownloadFile). Configurable under storage.grpc. Excess streams get RESOURCE_EXHAUSTED with operator log.

Packaging

Canonical source for all smartbotic-database Debian packages. Replaces the old shadowman-database and callerai-storage packages.

Build modes

# Production (Docker, Debian 13)
./packaging/build.sh                          # Build 4 .debs
./packaging/build.sh --rebuild-base           # Rebuild Docker base image
./packaging/build.sh --sync --suite trixie    # Build + publish to repository.smartbotics.ai

# Local development (native, current system)
./packaging/build.sh --local                  # Build 4 .debs
./packaging/build.sh --local --install        # Build + install locally

Packages

Package	Contents	Deployed where
smartbotic-database	Server binary + systemd + config	Production DB servers
libsmartbotic-db-client	Shared .so (runtime)	All production machines
libsmartbotic-db-client-dev	Headers + linker symlink + .proto + cmake config	Docker build images, dev workstations
smartbotic-db-cli	CLI tool	Anywhere (optional)

Local dev with installed packages

After ./packaging/build.sh --local --install, consumer projects can use:

find_package(smartbotic-db-client REQUIRED)
target_link_libraries(myapp PRIVATE smartbotic::db-client)

The submodule fallback still works for projects that haven't switched (BUILD_SHARED_LIBS=OFF).

Consumers

Project	Depends on	Min version	Migrations dir
shadowman-cpp	smartbotic-database (>= 1.2.0)	1.2.0	/opt/shadowman/share/shadowman/migrations/json
callerai	smartbotic-database (>= 1.2.0)	1.2.0	/etc/callerai/migrations

Version policy

Bump VERSION for API/protocol changes. Deb revision (-N) for packaging-only changes.

Conventions

• C++20
• nlohmann/json for JSON handling
• spdlog for logging
• LZ4 for compression
• gRPC/Protobuf for API