Home Esplora Aiuto
Accedi
fszontagh
/
smartbotic-database
1
0
Forka 0
File Problemi 0 Pull Requests 0 Wiki
Albero (Tree): 5ea5ac14a9
Rami (Branch) Tag
smartbotic-data...
/
CLAUDE.md

CLAUDE.md 6.1 KB
Cronologia Originale

Smartbotic Database

Build

cmake -B build -G Ninja
cmake --build build -j$(nproc)

Test

cmake --build build -j$(nproc) && ./build/tests/test_vector_storage

Architecture

  • proto/database.proto — gRPC service and message definitions
  • service/src/ — Database server implementation
    • memory_store.hpp/.cpp — In-memory document + vector storage
    • document.hpp — Document and CollectionOptions types
    • database_grpc_impl.cpp — gRPC request handlers
    • persistence/ — WAL, snapshots, persistence manager
    • migrations/ — Migration runner
  • client/ — C++ client library (smartbotic-db-client)
  • tests/ — Integration tests

Key Features

  • JSON document store with collections, version history, field-level encryption
  • gRPC API with replication, events, file storage, migrations
  • Vector Storage — Embedding vectors stored alongside documents with SIMD-accelerated (AVX2/SSE4.1) cosine similarity search via SimilaritySearch RPC. Vectors use _vector document field, stored in parallel float arrays, persisted through WAL (VEC_PUT/VEC_DELETE) and snapshots (v3 format). Collection vector_dimension option is immutable after creation.
  • Atomic Partial Updates — PatchDocument RPC merges fields into existing documents atomically (server-side, within collection lock). Client patch() method. update() uses automatic optimistic locking with retry. See docs/integration-guide.md for concurrency patterns.
  • Views — Read-only named projections over collections. Include/exclude field lists (dot-notation for nested paths), baked-in filters with all operators (EQ/NE/GT/GTE/LT/LTE/IN/CONTAINS/EXISTS/REGEX/SEARCH) AND-merged with caller filters, optional default sort that caller can override. Stored in _views system collection, created via create_view migration op or createView RPC. Writes on view names are rejected.
  • Per-collection Timestamp Precision — collection-level timestamp_precision config ("ms" default, "ns" for rapid-write collections). Stamps _created_at/_updated_at at the configured resolution, cached hot-path lookup. configureCollection RPC flips the config atomically; migrateCollectionTimestamps RPC converts existing data during maintenance windows (idempotent via 10^15 threshold).
  • Durable Snapshots + Tiered Recovery — atomic snapshot writer (.tmp + fsync + rename + post-write verification), loader fallback chain across snapshots, MySQL-style recovery modes (normal / snapshot_fallback / wal_only / best_effort / force_empty) selectable via --recovery-mode flag or recovery.mode config. Non-trivial recovery (fallback, WAL-only, forced empty) automatically enters read-only mode — operator must smartbotic-db-cli unlock or pass --force-readwrite to accept writes. Exit code 10 when recovery is refused.
  • gRPC Concurrency Cap — bounded inbound memory via gRPC ResourceQuota + per-RPC-type stream limits (Subscribe, UploadFile, DownloadFile). Configurable under storage.grpc. Excess streams get RESOURCE_EXHAUSTED with operator log.
  • Drop-in config (conf.d) — /etc/smartbotic-database/conf.d/*.json deep-merges over config.json using RFC 7396 semantics (arrays replace, objects merge). Lets consumer debs ship tuned settings without touching the upstream conffile. Strict on syntax errors (exit 11), lenient on unknown keys.
  • Chunked pressure-aware eviction — Eviction spreads work over multiple ticks with a default eviction_chunk_size=1000, eviction_chunk_pause_ms=50, hot_write_floor_ms=30000. Four pressure levels (normal/soft/hard/emergency) gate behavior: soft = trickle, hard = aggressive + MEMORY_PRESSURE_HIGH event, emergency = admission control rejects writes with RESOURCE_EXHAUSTED. Per-collection memory_priority (low/normal/high) biases selection. Quiesce skips docs with in-flight writes. WAL-fallback no longer holds per-collection mutex during disk I/O.
  • GetMemoryStats RPC + client retry — Client::getMemoryStats() returns per-collection stats + pressure level. Client library auto-retries write RPCs on DEADLINE_EXCEEDED/RESOURCE_EXHAUSTED/UNAVAILABLE with exponential backoff + jitter (writeRetries=3, writeRetryBackoffMs=100). Reads are not auto-retried.

Packaging

Canonical source for all smartbotic-database Debian packages. Replaces the old shadowman-database and callerai-storage packages.

Build modes

# Production (Docker, Debian 13)
./packaging/build.sh                          # Build 4 .debs
./packaging/build.sh --rebuild-base           # Rebuild Docker base image
./packaging/build.sh --sync --suite trixie    # Build + publish to repository.smartbotics.ai

# Local development (native, current system)
./packaging/build.sh --local                  # Build 4 .debs
./packaging/build.sh --local --install        # Build + install locally

Packages

Package Contents Deployed where
smartbotic-database Server binary + systemd + config Production DB servers
libsmartbotic-db-client Shared .so (runtime) All production machines
libsmartbotic-db-client-dev Headers + linker symlink + .proto + cmake config Docker build images, dev workstations
smartbotic-db-cli CLI tool Anywhere (optional)

Local dev with installed packages

After ./packaging/build.sh --local --install, consumer projects can use:

find_package(smartbotic-db-client REQUIRED)
target_link_libraries(myapp PRIVATE smartbotic::db-client)

The submodule fallback still works for projects that haven't switched (BUILD_SHARED_LIBS=OFF).

Consumers

Project Depends on Min version Migrations dir
shadowman-cpp smartbotic-database (>= 1.2.0) 1.2.0 /opt/shadowman/share/shadowman/migrations/json
callerai smartbotic-database (>= 1.2.0) 1.2.0 /etc/callerai/migrations

Version policy

Bump VERSION for API/protocol changes. Deb revision (-N) for packaging-only changes.

Conventions

  • C++20
  • nlohmann/json for JSON handling
  • spdlog for logging
  • LZ4 for compression
  • gRPC/Protobuf for API