Smartbotic Database Integration Guide

How to install, configure, and integrate smartbotic-database into your C++ project.

Installing the Database
Installing the Client Library
Integrating into a C++ Project
Client API Reference
Configuration
Migrations
Docker Build Integration
CLI Administration
Upgrading from Legacy Packages
Drop-in configuration (conf.d)
Eviction & Memory Pressure

Installing the Database

From APT Repository

Add the Smartbotics repository to your system:

# Download GPG key
curl -fsSL https://repository.smartbotics.ai/smartbotics-repo.gpg | \
    sudo gpg --dearmor -o /usr/share/keyrings/smartbotics-repo.gpg

# Configure authentication
sudo tee /etc/apt/auth.conf.d/smartbotics.conf << EOF
machine repository.smartbotics.ai
login callerai
password <repo-password>
EOF
sudo chmod 600 /etc/apt/auth.conf.d/smartbotics.conf

# Add repository
echo "deb [signed-by=/usr/share/keyrings/smartbotics-repo.gpg] https://repository.smartbotics.ai trixie main" | \
    sudo tee /etc/apt/sources.list.d/smartbotics.list

# Install
sudo apt update
sudo apt install smartbotic-database

This installs:

/usr/bin/smartbotic-database — the server binary
/etc/smartbotic-database/config.json — configuration file
smartbotic-database.service — systemd service (starts automatically)

The service listens on localhost:9004 by default.

Verify Installation

# Check service status
sudo systemctl status smartbotic-database

# Check health via CLI (if installed)
smartbotic-db-cli health

Available Packages

Package	What it provides	Install when...
`smartbotic-database`	Server + systemd service	You need to run the database on this machine
`libsmartbotic-db-client`	Shared library (`.so`)	Your application links the client at runtime
`libsmartbotic-db-client-dev`	Headers + cmake config	You're building a C++ project against the client API
`smartbotic-db-cli`	CLI admin tool	You want to administer the database from the command line

Installing the Client Library

On a Development Machine (for local builds)

sudo apt install libsmartbotic-db-client-dev

This pulls in libsmartbotic-db-client (the runtime .so) as a dependency, plus all development headers needed to compile.

On a Production Machine (runtime only)

sudo apt install libsmartbotic-db-client

Only the shared library — no headers, no cmake config, no dev tooling.

Integrating into a C++ Project

Option A: System-installed package (recommended for Docker builds)

If libsmartbotic-db-client-dev is installed (e.g., in your Docker build image):

# CMakeLists.txt
find_package(smartbotic-db-client REQUIRED)

add_executable(myapp main.cpp)
target_link_libraries(myapp PRIVATE smartbotic::db-client)

The smartbotic::db-client target automatically sets up include paths and links the shared library.

Option B: Git submodule (for local development on non-Debian systems)

git submodule add ssh://git@git.smartbotics.ai:10022/fszontagh/smartbotic-database.git \
    external/smartbotic-database

# CMakeLists.txt
add_subdirectory(external/smartbotic-database)

add_executable(myapp main.cpp)
target_link_libraries(myapp PRIVATE smartbotic-db-client)

When built as a submodule, the library is static (.a) and linked directly into your binary.

Option C: Both (recommended)

Use the system package when available, fall back to submodule:

# Try system-installed first (Docker builds, CI)
find_package(smartbotic-db-client QUIET)
if(smartbotic-db-client_FOUND)
    message(STATUS "Using system smartbotic-db-client")
    set(DB_CLIENT_TARGET smartbotic::db-client)
else()
    # Fall back to submodule (local development)
    if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/external/smartbotic-database/CMakeLists.txt")
        add_subdirectory(external/smartbotic-database)
        set(DB_CLIENT_TARGET smartbotic-db-client)
    else()
        message(FATAL_ERROR
            "smartbotic-database not found. Either:\n"
            "  apt install libsmartbotic-db-client-dev\n"
            "  git submodule update --init external/smartbotic-database")
    endif()
endif()

add_executable(myapp main.cpp)
target_link_libraries(myapp PRIVATE ${DB_CLIENT_TARGET})

Client API Reference

Connecting

#include <smartbotic/database/client.hpp>

smartbotic::database::Client::Config config;
config.address = "localhost:9004";   // gRPC endpoint
config.timeoutMs = 5000;            // per-RPC timeout
config.maxRetries = 3;              // auto-retry on transient failures

smartbotic::database::Client db(config);
if (!db.connect()) {
    // handle connection failure
}

Document Operations

// Insert
nlohmann::json doc = {{"name", "Alice"}, {"email", "alice@example.com"}};
std::string id = db.insert("users", doc);
// Or with explicit ID:
std::string id = db.insert("users", doc, "user-123");

// Get
auto result = db.get("users", "user-123");
if (result) {
    std::cout << (*result)["name"] << std::endl;
}

// Update (full document replacement, with automatic optimistic locking)
db.update("users", "user-123", {{"name", "Alice Smith"}, {"email", "alice@example.com"}});

// Update with explicit optimistic locking (fails if version changed)
db.updateIfVersion("users", "user-123", {{"name", "Alice"}}, /*expectedVersion=*/2);

// Patch — atomic partial update (only specified fields are modified)
uint64_t newVer = db.patch("users", "user-123", {{"name", "Alice Smith"}});

// Upsert (insert or update)
auto [uid, isNew] = db.upsert("users", doc, "user-123");

// Delete
db.remove("users", "user-123");

// Check existence
bool found = db.exists("users", "user-123");

Concurrency: update() vs patch()

When multiple clients modify the same document concurrently, choose the right method:

patch() — use for most updates (recommended). Merges only the specified fields on the server, atomically. Two clients can safely modify different fields simultaneously:

// Client A:                                    // Client B:
db.patch("users", "user-123",                   db.patch("users", "user-123",
    {{"name", "Alice Smith"}});                      {{"email", "alice@new.com"}});
// Result: both fields updated, nothing lost

update() — full document replacement with automatic retry. Uses optimistic locking internally (reads current version, writes with version check, retries on conflict). Use when you need to replace the entire document:

auto doc = db.get("users", "user-123");
(*doc)["name"] = "Alice Smith";
(*doc)["role"] = "admin";
db.update("users", "user-123", *doc);  // replaces entire document

Note: update() is "last writer wins" for complete replacement — if two clients modify different fields via update(), the last one overwrites the other's changes. Use patch() to avoid this.

updateIfVersion() — explicit optimistic locking. Use when you need to detect conflicts and handle them yourself:

auto doc = db.get("users", "user-123");
uint64_t version = (*doc)["_version"];
(*doc)["balance"] = (*doc)["balance"].get<int>() + 100;
if (!db.updateIfVersion("users", "user-123", *doc, version)) {
    // Version conflict — someone else modified the document
    // Re-read and retry, or report error to user
}

Views

Views are named, read-only projections over collections. They filter which fields are returned and optionally enforce baked-in filters and default sort order. Views are ideal for:

Schema evolution — add fields to a collection without affecting consumers that query through a view
Security boundary — hide sensitive fields (passwords, tokens) and enforce row-level constraints (status=active, tenant=X)
API clarity — name and document a stable contract ("this view returns these fields with these filters")

Creating a view

using Op = smartbotic::database::Client::FilterOp;

// Simple: just project fields
db.createView("users_public",
              /*collection=*/"users",
              /*include=*/{"id", "name", "avatar_url", "profile.public_bio"});

// Full-featured: fields + filters + default sort
db.createView("active_admins",
              /*collection=*/"users",
              /*include=*/{"id", "name", "role"},
              /*exclude=*/{},
              /*where=*/{
                  {"status", Op::EQ, "active"},
                  {"role",   Op::EQ, "admin"}
              },
              /*defaultSort=*/smartbotic::database::Client::Sort{"name", false});

Querying a view

Views work like collections — use the view name in get(), find(), count():

auto user = db.get("users_public", "u1");         // projection applied
auto results = db.find("active_admins", {
    .filters = {{"department", Op::EQ, "eng"}},   // AND-merged with view's where
    .limit = 50
});
auto total = db.count("active_admins", {{"department", Op::EQ, "eng"}});

Inspecting and dropping

auto views = db.listViews();
auto info = db.getViewInfo("users_public");
db.dropView("users_public");

Semantics

Dimension	Behavior
Filters	View AND caller — view's `where` is always applied. Caller's filters are merged with AND. Consumers cannot escape the view's constraints.
Sort	Caller overrides view — if the caller specifies a sort, it wins. Otherwise the view's `defaultSort` is used.
Include	Whitelist — when non-empty, only those paths are returned.
Exclude	Blacklist — ignored if include is non-empty; otherwise removes listed paths.
Metadata	`_id`, `_version`, `_created_at`, `_updated_at`, `_created_by`, `_updated_by` are always returned regardless of include/exclude.
Nested paths	Dot-notation supported: `user.profile.name`. Stops at arrays.
Writes	Rejected — `insert()`, `update()`, `patch()`, `upsert()`, `remove()` on a view name return an error.
View-of-view	Not allowed — views must target real collections.

Via migrations (recommended)

{
  "version": "020",
  "name": "create_active_admins_view",
  "operations": [
    {
      "type": "create_view",
      "name": "active_admins",
      "collection": "users",
      "include": ["id", "name", "role"],
      "where": [
        {"field": "status", "op": "EQ", "value": "active"},
        {"field": "role",   "op": "EQ", "value": "admin"}
      ],
      "default_sort": {"field": "name", "descending": false}
    }
  ]
}

The op field in where filters accepts either a string name (EQ, NE, GT, GTE, LT, LTE, IN, CONTAINS, EXISTS, REGEX, SEARCH) or the matching integer. Migrations are idempotent — re-running a migration that creates an existing view is a no-op.

Why AND-merge filters?

The view's filters act as a security enforcement boundary. A consumer querying active_admins literally cannot see inactive users — the filter is applied server-side, after authentication, before projection. This is how SQL views with WHERE clauses work, and how row-level security works in most databases. Any consumer filter is ANDed on top, narrowing further.

Collection Configuration

Per-collection runtime settings. Currently supports timestamp precision; extensible for future knobs.

Timestamp precision

Every document carries _created_at and _updated_at — int64 timestamps since epoch. By default these are milliseconds. Collections that receive rapid-fire writes (LLM streaming chunks, agent tool loops) often see ties at ms resolution that make ORDER BY _created_at ambiguous. Configure those collections for nanoseconds to get unique, strictly ordered timestamps.

using Cfg = smartbotic::database::Client::CollectionConfig;

// Default: ms (no configuration needed)
auto id = db.insert("logs", {{"msg", "..."}});
// doc["_created_at"] is ~13 digits, milliseconds since epoch

// Switch a collection to nanoseconds
db.configureCollection("llm_tokens", Cfg{"ns"});

// Future inserts into llm_tokens get ns-precision timestamps
auto id2 = db.insert("llm_tokens", {{"token", "hello"}});
// doc["_created_at"] is ~19 digits, nanoseconds since epoch

Reading the config

auto cfg = db.getCollectionConfig("llm_tokens");
std::cout << cfg.timestampPrecision << std::endl;  // "ns"

bool explicit_set = db.hasCollectionConfig("llm_tokens");  // true
bool default_only = db.hasCollectionConfig("logs");        // false (still ms default)

What configureCollection does NOT do

It does not convert existing timestamps. Flipping a collection from ms to ns only affects future inserts. Pre-existing docs keep whatever precision they had. This keeps the config flip atomic and reversible.

Consequences of mixed precision within one collection:

int64 ordering within the collection is still consistent (ms values are always < 10^15, ns values are always > 10^15, so ns docs sort after ms docs). Queries don't break.
But the ns-labeled docs appear to be in the future relative to the ms-labeled docs unless you convert.

Migrating existing timestamps

When you're ready to convert legacy data, run the migration helper during a planned maintenance window:

auto result = db.migrateCollectionTimestamps("llm_tokens",
                                              /*fromPrecision=*/"ms",
                                              /*toPrecision=*/"ns");
if (result.success) {
    std::cout << "migrated " << result.rowsMigrated
              << " rows, skipped " << result.rowsSkipped << std::endl;
} else {
    std::cerr << "failed: " << result.error << std::endl;
}

Semantics:

ms → ns: multiply _created_at / _updated_at by 10^6
ns → ms: divide by 10^6 (truncation — loses sub-ms precision)
Idempotent and resumable. Rows that are already in the target range are detected via a 10^15 threshold and skipped. If the migration is interrupted (network drop, deadline), rerun it — already-converted rows will count as rows_skipped.
from == to is a no-op success.
Metadata-only — document version numbers are NOT incremented, WAL is not appended, history is not updated. This is an infrastructure rewrite, not a user-visible change.
Deadline: the client's RPC timeout for this call is 10 minutes — large collections may take a while.

Recommended workflow for switching precision on production data

Schedule a maintenance window (writes against the collection may pause briefly if needed).
Call configureCollection(name, {ns}) — atomic config flip; future writes immediately use ns.
Call migrateCollectionTimestamps(name, "ms", "ns") — converts existing docs.
Verify with a sample read that _created_at values are now in the ns range.
Resume normal traffic.

Between steps 2 and 3, new writes are ns and old docs are ms — this is temporary but normal. Ordering still works because the 10^15 threshold cleanly separates the two ranges.

Via migrations

To set timestamp precision declaratively, call db.configureCollection() from your consumer project's startup code. The configure_collection migration operation will be available in a future version.

{
  "version": "021",
  "name": "llm_tokens_ns_precision",
  "operations": [
    {
      "type": "configure_collection",
      "collection": "llm_tokens",
      "config": { "timestamp_precision": "ns" }
    }
  ]
}

Recovery & Read-Only Mode

smartbotic-database uses WAL + snapshots for durability. When starting up, the server tries to recover state from the newest snapshot and replay WAL entries since that snapshot. If something goes wrong, it uses a tiered recovery system modeled on MySQL's innodb_force_recovery.

Recovery modes

Level	Name	Behavior
0	`normal`	Load latest snapshot + replay WAL. Default. If `auto_escalate=true` (default), falls back to older snapshot automatically when latest is corrupt.
1	`snapshot_fallback`	Try snapshots newest→oldest, first that loads wins.
2	`wal_only`	Ignore snapshots, replay WAL from sequence 0. Slow on large DBs.
3	`best_effort`	Try snapshot_fallback, then fall through to wal_only.
4	`force_empty`	Start empty. Snapshots + WAL preserved on disk for forensics.

Operator escalation path when recovery fails:

# After a refuse-to-start, try older snapshots
smartbotic-database --recovery-mode=snapshot_fallback

# If that still fails, replay WAL from scratch (slow)
smartbotic-database --recovery-mode=wal_only

# Last-ditch: try everything
smartbotic-database --recovery-mode=best_effort

# Absolute last resort: start empty (data preserved on disk)
smartbotic-database --recovery-mode=force_empty

Or set in config.json:

"persistence": {
    "recovery": {
        "mode": "normal",
        "auto_escalate": true,
        "allow_empty_on_fresh_install": true
    }
}

The CLI flag overrides the config for one-shot recoveries.

Auto-readonly on non-trivial recovery

If recovery used anything other than "latest snapshot loaded cleanly," the DB boots in read-only mode. This is a safety feature — writing onto a possibly-stale state can corrupt your data further. The operator must explicitly acknowledge the state before writes resume.

On startup you'll see (at ERROR level):

[ERROR] Database booted in READ-ONLY mode after non-trivial recovery
        Reason: fell back to snapshot snapshot-20260419-090719.dat because
                snapshot-20260419-100747.dat failed: body truncated

Writes will be REJECTED until you acknowledge this state:
  smartbotic-db-cli unlock                  # live, no restart
  smartbotic-database --force-readwrite     # on next restart

Workflow: recovering a broken instance

Preserve forensics. Stop the service, snapshot /var/lib/smartbotic-database/ before anything else.
Diagnose. Start the server (default mode = normal). It will either recover cleanly or refuse with a clear banner. Use smartbotic-db-cli status to see what happened.
Escalate. If recovery refused, restart with --recovery-mode=snapshot_fallback (and upwards as needed).
Extract data. While the DB is in read-only mode, connect with the CLI or client library and read whatever survived. Dump/export facilities are planned for v1.7.0.
Decide.
- Data looks complete → smartbotic-db-cli unlock to resume writes.
- Data is missing/stale → don't unlock. Extract to an external backup, then wipe /var/lib/smartbotic-database/ and start clean.

Client API

// Acknowledge auto-readonly state and resume writes
db.unlock();

// Force read-only for maintenance (regardless of recovery outcome)
db.lock();

// Inspect state + recovery outcome
auto s = db.getReadOnlyStatus();
std::cout << "read-only:        " << s.readOnly << "\n"
          << "reason:           " << s.reason << "\n"
          << "recovery outcome: " << s.recoveryOutcome << "\n"
          << "expected snap:    " << s.expectedSnapshot << "\n"
          << "snapshot used:    " << s.snapshotUsed << "\n"
          << "failure reason:   " << s.failureReason << "\n"
          << "WAL replayed:     " << s.walEntriesReplayed << "\n"
          << "snapshots tried:  " << s.snapshotsAttempted << "\n";

CLI

# Inspect read-only + recovery state
smartbotic-db-cli status

# Acknowledge non-trivial recovery and accept writes
smartbotic-db-cli unlock

# Manually lock (e.g. before a maintenance window)
smartbotic-db-cli lock

Snapshot durability settings

Under persistence.snapshots:

"snapshots": {
    "validate_after_write": true,        // verify every snapshot immediately after creation
    "cleanup_only_if_verified": true     // never evict old snapshots if the new one failed validation
}

Both default to true — turn off only if you really know what you're doing (disables protections against silent writer corruption).

Exit codes

Code	Meaning
`0`	Normal shutdown
`1`	Generic error
`10`	Recovery refused. Snapshot or WAL failure + mode=normal + data exists on disk. Operator must escalate the recovery mode or pass `--force-readwrite`.
`11`	Config invalid (e.g. bad `--recovery-mode` value)

`--force-readwrite`

Bypasses the auto-readonly guard on non-trivial recovery. Use after you've reviewed the recovery state via smartbotic-db-cli status and are certain writes onto that state are safe.

# After confirming recovery state looks good
smartbotic-database --force-readwrite

Querying

// Find with filters (equality is default)
smartbotic::database::Client::QueryOptions opts;
opts.filters = {
    {"status", "active"},   // EQ (default) — 2-arg Filter constructor
    {"role", "admin"},      // EQ
};
opts.sortField = "name";
opts.sortDescending = false;
opts.limit = 50;
opts.offset = 0;

auto docs = db.find("users", opts);
for (const auto& doc : docs) {
    std::cout << doc["name"] << std::endl;
}

// Find all documents in a collection
auto allDocs = db.find("users");

// Count
uint64_t total = db.count("users");
uint64_t admins = db.count("users", {{"role", "admin"}});

Filter operators

Filters support a range of operators via Client::FilterOp. The 2-argument Filter(field, value) constructor defaults to EQ — existing brace-init code continues to compile and behave the same. To use other operators, pass the operator explicitly with the 3-argument constructor:

using Op = smartbotic::database::Client::FilterOp;

smartbotic::database::Client::QueryOptions opts;
opts.filters = {
    {"age", Op::GTE, 18},                                             // age >= 18
    {"status", Op::NE, "deleted"},                                    // status != "deleted"
    {"tags", Op::CONTAINS, "admin"},                                  // tags array contains "admin"
    {"role", Op::IN, nlohmann::json::array({"admin", "moderator"})},  // role IN [...]
    {"email", Op::REGEX, ".*@example\\.com$"},                        // regex match
    {"name", Op::SEARCH, "alice"},                                    // full-text search
    {"deleted_at", Op::EXISTS, false},                                // field does NOT exist
};
auto docs = db.find("users", opts);

Operator	Semantics	Value type
`FilterOp::EQ`	field equals value	any JSON
`FilterOp::NE`	field does not equal value	any JSON
`FilterOp::GT`	field > value	number or string
`FilterOp::GTE`	field >= value	number or string
`FilterOp::LT`	field < value	number or string
`FilterOp::LTE`	field <= value	number or string
`FilterOp::IN`	field value appears in the supplied array	JSON array
`FilterOp::CONTAINS`	field (array) contains the supplied value	any JSON
`FilterOp::EXISTS`	field presence check (`true` = must exist, `false` = must not)	bool
`FilterOp::REGEX`	field matches the supplied regex pattern	string
`FilterOp::SEARCH`	full-text search across document ID and string fields	string

Note: the old _search magic field name that auto-selected SEARCH is no longer special — use FilterOp::SEARCH explicitly instead.

Collection Management

// Create a basic collection
db.createCollection("users");

// Create with options
db.createCollection("sessions",
    /*defaultTtlSeconds=*/3600,    // auto-expire after 1 hour
    /*encrypted=*/false,
    /*maxVersions=*/5,             // keep last 5 versions
    /*vectorDimension=*/0);        // no vector support

// Create vector-enabled collection
db.createCollection("memories",
    /*defaultTtlSeconds=*/0,
    /*encrypted=*/false,
    /*maxVersions=*/0,
    /*vectorDimension=*/384);      // 384-dim embeddings

// List and inspect
auto collections = db.listCollections();
auto info = db.getCollectionInfo("users");
if (info) {
    std::cout << "Documents: " << info->documentCount << std::endl;
}

// Drop
db.dropCollection("old_data");

Vector Similarity Search

Requires a collection created with vectorDimension > 0.

// Insert documents with embeddings
nlohmann::json doc = {
    {"id", "mem-1"},
    {"content", "The user prefers dark mode"},
    {"_vector", embedding}   // std::vector<float>, size must match vectorDimension
};
db.insert("memories", doc, "mem-1");

// Search
std::vector<float> queryVec = getEmbedding("user preferences");
auto results = db.similaritySearch("memories", queryVec, /*topK=*/5, /*minScore=*/0.7f);

for (const auto& r : results) {
    std::cout << r.id << " (score=" << r.score << "): "
              << r.data["content"] << std::endl;
}

Notes:

_vector is extracted and stored separately — not returned in get() or find() responses
Dimension is validated on every insert — mismatches are rejected
SIMD-accelerated (AVX2/SSE4.1) brute-force cosine similarity

Version History

// Get history
auto history = db.getVersionHistory("users", "user-123", /*limit=*/10);
for (const auto& v : history.versions) {
    std::cout << "v" << v.version << " by " << v.updatedBy
              << " at " << v.timestamp << std::endl;
}

// Get a specific version
auto oldVersion = db.getDocumentVersion("users", "user-123", /*version=*/2);

// Restore to a previous version
uint64_t newVer = db.restoreVersion("users", "user-123", /*version=*/2);

// Restore to a point in time
auto [fromVer, newVersion] = db.restoreToDate("users", "user-123", timestamp);

File Storage

Two-level storage with content-addressed blob deduplication. Identical files are stored once.

// Upload
std::vector<uint8_t> data = readFile("photo.jpg");
smartbotic::database::Client::FileUploadMeta meta;
meta.name = "photo.jpg";
meta.mime_type = "image/jpeg";
meta.file_type = "document";        // "plugin", "document", "generated"
meta.related_id = "conversation-1";
meta.is_public = false;

auto result = db.uploadFile(data, meta);
std::cout << "File ID: " << result.id
          << " Deduplicated: " << result.deduplicated << std::endl;

// Download
auto fileData = db.downloadFile(result.id);

// Get file info
auto fileInfo = db.getFileInfo(result.id);

// List files
auto files = db.listFiles(
    /*file_type=*/"document",
    /*related_id=*/"conversation-1",
    /*limit=*/100, /*offset=*/0,
    /*checksum=*/"",
    /*name=*/"photo.jpg");

// Delete
db.deleteFile(result.id);

Event Subscription

auto subscription = db.subscribe({"users", "sessions"}, [](
    const std::string& collection,
    const std::string& id,
    const std::string& eventType,
    const std::optional<nlohmann::json>& data) {

    std::cout << eventType << " on " << collection << "/" << id << std::endl;
});

// Unsubscribe by releasing the handle
subscription.reset();

Set Operations

Redis-compatible set operations on top of documents.

db.setAdd("tags", "user-123", "admin");
db.setAdd("tags", "user-123", "active");

auto members = db.setMembers("tags", "user-123");
bool isAdmin = db.setIsMember("tags", "user-123", "admin");

db.setRemove("tags", "user-123", "active");

Health & Statistics

// Quick health check
if (db.healthCheck()) {
    std::cout << "Database is healthy" << std::endl;
}

// Detailed health
auto health = db.getHealthInfo();
if (health) {
    std::cout << "Uptime: " << health->uptimeMs << "ms"
              << " Docs: " << health->documentCount << std::endl;
}

// Full statistics
auto stats = db.getStats();
if (stats) {
    std::cout << "Memory: " << stats->memoryUsedBytes << " bytes"
              << " Insert avg: " << stats->insertAvgMicros() << " us" << std::endl;
}

Configuration

The server configuration file is at /etc/smartbotic-database/config.json.

Default Configuration

{
  "log_level": "info",
  "storage": {
    "bind_address": "localhost",
    "rpc_port": 9004,
    "node_id": "smartbotic-db",
    "data_directory": "/var/lib/smartbotic-database",
    "memory": {
      "max_memory_mb": 512,
      "eviction_threshold_percent": 80,
      "eviction_target_percent": 60,
      "eviction_check_interval_ms": 5000
    },
    "persistence": {
      "wal_sync_interval_ms": 100,
      "snapshot_interval_sec": 3600,
      "compression": "lz4"
    },
    "encryption": {
      "enabled": true,
      "key_file": "/var/lib/smartbotic-database/storage.key",
      "auto_generate_key": true
    },
    "migrations": {
      "enabled": false,
      "directory": "/etc/smartbotic-database/migrations",
      "auto_apply": false
    },
    "files": {
      "max_file_size_mb": 500,
      "cleanup_orphans_interval_sec": 3600
    },
    "replication": {
      "enabled": false
    }
  }
}

Key Settings

Setting	Default	Description
`bind_address`	`localhost`	Change to `0.0.0.0` to accept remote connections
`rpc_port`	`9004`	gRPC listen port
`data_directory`	`/var/lib/smartbotic-database`	WAL, snapshots, files stored here
`memory.max_memory_mb`	`512`	Maximum memory for document cache. Evicts LRU when exceeded
`persistence.snapshot_interval_sec`	`3600`	Full snapshot every N seconds (WAL truncated after)
`encryption.enabled`	`true`	Field-level AES-256-GCM encryption. Key auto-generated on first start
`migrations.enabled`	`false`	Enable to auto-apply JSON migrations on startup
`migrations.directory`	`/etc/smartbotic-database/migrations`	Path to migration JSON files

gRPC Configuration

Under storage.grpc (added in v1.6.2):

Setting	Default	Purpose
`max_receive_message_size_mb`	`100`	Max inbound gRPC message size. Raise for larger file uploads.
`max_send_message_size_mb`	`100`	Max outbound gRPC message size.
`resource_quota_memory_mb`	`256`	Global memory budget for gRPC inbound buffers across all RPCs combined.
`max_concurrent_subscribe_streams`	`50`	Max simultaneous `Subscribe` event-stream clients. Excess gets `RESOURCE_EXHAUSTED`.
`max_concurrent_file_streams`	`10`	Max simultaneous `UploadFile` + `DownloadFile` streams (shared pool).

When a concurrency limit is hit, the server returns RESOURCE_EXHAUSTED immediately and logs WARN with the current / max counts. Clients should retry with backoff.

Drop-in configuration (conf.d)

In addition to /etc/smartbotic-database/config.json, the server loads any *.json files from /etc/smartbotic-database/conf.d/ and deep-merges them into the running config in lexicographic order.

This lets consumer projects (shadowman-cpp, callerai, your own apps) ship their own tuning without touching the upstream config.json that dpkg --configure guards as a conffile.

Merge semantics (RFC 7396)

Objects merge recursively: sibling fields not mentioned in a drop-in are preserved from the lower-priority layer.
Arrays and scalars are replaced entirely by a higher-priority layer.

Example:

// /etc/smartbotic-database/config.json
{
  "log_level": "info",
  "storage": {
    "rpc_port": 9004,
    "memory": { "max_memory_mb": 512 }
  }
}

// /etc/smartbotic-database/conf.d/50-shadowman.json  (shipped by shadowman-cpp's deb)
{
  "storage": {
    "memory": { "max_memory_mb": 2048 }
  }
}

// Effective merged config — sibling rpc_port preserved, max_memory_mb overridden
{
  "log_level": "info",
  "storage": {
    "rpc_port": 9004,
    "memory": { "max_memory_mb": 2048 }
  }
}

File naming conventions

Use numeric prefixes for deterministic merge order. Last writer wins.

File	Owner	Purpose
`00-defaults.json`	smartbotic-database (optional)	Baseline conservative tunings
`50-<project>.json`	consumer project's deb postinst	Workload-tuned values
`99-local.json`	operator	Site-specific overrides

Files are merged in filename-sort order, so 00- applies before 50- applies before 99-.

Failure modes

Missing base config.json — OK, treated as {}
Missing conf.d/ directory — OK
Syntax error in any file — server refuses to start, exit code 11, log identifies the file + parser location
Unknown keys — lenient. Individual config consumers may log WARN but don't block startup. This lets v1.x consumers drop a conf.d/ file that mentions a key only v1.(x+1) knows about

What gets logged

At startup, the server prints each file as it merges it:

[INFO] Loading base config: /etc/smartbotic-database/config.json
[INFO] Merging drop-in: /etc/smartbotic-database/conf.d/50-shadowman.json
[INFO] Merging drop-in: /etc/smartbotic-database/conf.d/99-local.json

If a consumer wonders which tuning values are actually in effect, this is where to look.

Eviction & Memory Pressure

smartbotic-database holds documents in memory for fast reads. When usage approaches the configured budget (storage.memory.max_memory_mb), eviction removes the least-useful docs to free space. Eviction is chunked, throttled, and hot-write-aware so it doesn't block foreground writes — a design requirement after the 2026-04-22 Zoe incident where one-shot eviction stalled LLM streaming writes long enough to miss their gRPC deadline.

Pressure levels

The server tracks four pressure levels based on current usage vs. max_memory_mb:

Level	Default %	Behavior
`normal`	< 70%	No eviction
`soft`	70–85%	Trickle eviction: one chunk per tick
`hard`	85–95%	Aggressive eviction: up to `max_eviction_passes_per_trigger / 2` chunks per tick. Fires `MEMORY_PRESSURE_HIGH` event (edge-triggered)
`emergency`	≥ 95%	All writes rejected with `RESOURCE_EXHAUSTED` until pressure drops below hard. Eviction runs at full rate

Query current state via smartbotic-db-cli status or the GetMemoryStats RPC.

Chunked eviction

Eviction never stops the world. Each tick (default every eviction_check_interval_ms = 5000ms, but reactive to pressure):

Collects candidates — docs whose updatedAt is older than hot_write_floor_ms (30 s default) ago, in non-pinned collections
Sorts by (memory_priority, lastAccessedAt) — low-priority + cold goes first
Applies per-collection budget: no one collection contributes more than (share × 1.5) of a single chunk
Evicts eviction_chunk_size docs (default 1000) in one pass
Sleeps eviction_chunk_pause_ms (default 50 ms)
Repeats up to max_eviction_passes_per_trigger times, stopping early if target reached

Docs being actively written (quiesce) are skipped via an in-flight-write counter — eviction can't land mid-upsert.

Per-collection priority

Use CollectionOptions.memory_priority to bias eviction:

Value	Meaning
`low`	Evicted preferentially — cold archive collections
`normal` (default)	Standard LRU weight
`high`	Evicted last — hot operational collections
(pinned=true on CollectionOptions)	Never evicted

Set via createCollection or migration:

{
  "type": "create_collection",
  "collection": "messages",
  "options": {
    "memory_priority": "high"
  }
}

Config reference (`storage.memory`)

Setting	Default	Meaning
`max_memory_mb`	512	Memory budget before eviction kicks in
`eviction_chunk_size`	1000	Docs evicted per chunk
`eviction_chunk_pause_ms`	50	Sleep between chunks within one tick
`max_eviction_passes_per_trigger`	20	Safety cap on chunks per tick
`hot_write_floor_ms`	30000	Docs updated within this window are unevictable
`memory_soft_percent`	70	Enter `soft` pressure
`memory_hard_percent`	85	Enter `hard` pressure — fires PRESSURE_HIGH event
`memory_emergency_percent`	95	Enter `emergency` — admission control rejects writes
`eviction_burst_threshold`	10000	Fire `MEMORY_EVICTION_BURST` event if chunk ≥ this
`eviction_target_percent`	60	Stop evicting once usage drops below this
`eviction_check_interval_ms`	5000	Wake eviction loop this often

Events

Subscribe to receive memory-related notifications:

Event type	When	Payload
`MEMORY_PRESSURE_HIGH`	Edge-triggered: pressure enters `hard` or `emergency`	`{pressure_level, pressure_percent, estimated_bytes, max_bytes}`
`MEMORY_EVICTION_BURST`	An eviction pass touched ≥ `eviction_burst_threshold` docs	`{evicted_docs, bytes_freed, pressure_level}`

Both are system-level events (empty collection / id). Subscribers filter by event type.

Client-side retry

The client library (v1.7.0+) automatically retries write RPCs that return transient errors:

Status	Retry?
`DEADLINE_EXCEEDED`	✅
`RESOURCE_EXHAUSTED` (admission control / concurrency cap)	✅
`UNAVAILABLE`	✅
All others (INVALID_ARGUMENT, FAILED_PRECONDITION, etc.)	❌ — final

Config under Client::Config:

Field	Default	Meaning
`writeRetries`	3	Max retry attempts
`writeRetryBackoffMs`	100	Initial backoff (exponential: 100 → 200 → 400 …)
`writeRetryMaxBackoffMs`	5000	Cap on any single backoff
`writeRetryJitter`	0.25	± 25% jitter

Writes wrapped: insert, updateIfVersion, upsert, remove, patch. Reads are NOT auto-retried — callers retry at their discretion.

Observability

GetMemoryStats RPC (and Client::getMemoryStats()) returns:

Total / max memory bytes, pressure percent + level
Per-collection: document_count, estimated_bytes, evicted_stub_count, priority

Last eviction: timestamp, docs evicted, bytes freed

auto s = db.getMemoryStats();
std::cout << "Memory: " << s.totalMemoryBytes << "/" << s.maxMemoryBytes
      << " (" << s.pressurePercent << "% — " << s.pressureLevel << ")" << std::endl;
for (const auto& c : s.collections) {
std::cout << "  " << c.collection << ": " << c.documentCount << " docs, "
          << c.estimatedBytes << " bytes, priority=" << c.priority << std::endl;
}

Also shown in smartbotic-db-cli status output (if the CLI exposes it; otherwise subscribers can consume the events).

Recommended tunings

Default (512 MB) — development or small services with ~100 K docs

2 GB — production shadowman-cpp profile:

// /etc/smartbotic-database/conf.d/50-shadowman.json
{
"storage": {
"memory": {
  "max_memory_mb": 2048,
  "memory_soft_percent": 65,
  "memory_hard_percent": 80,
  "eviction_chunk_size": 500,
  "hot_write_floor_ms": 60000
}
}
}

8+ GB large — consider raising eviction_burst_threshold to 50000 so you don't get event spam, and consider hot_write_floor_ms: 120000 if your write paths are slow (e.g. STT-heavy callerai)

Troubleshooting

"Client::insert failed: RESOURCE_EXHAUSTED (memory pressure emergency)" — server is in emergency pressure, client is retrying. If this persists, raise max_memory_mb in a conf.d drop-in or lower memory_emergency_percent.

"Eviction: nothing evictable this pass" — every candidate is either pinned, hot-written, or in-flight. Usually benign; server will re-try next tick. If persistent, consider reducing hot_write_floor_ms or flagging some collections as unpinned/low-priority.

Pressure stays at hard despite quiet writes — check eviction_target_percent; eviction stops when usage drops below this. If you want eviction to drain further during idle, raise max_memory_mb or lower the target.

Configuring for Your Project

Consumer projects typically enable migrations by configuring the database via their own postinst script:

# Example: in your project's postinst
DB_CONFIG="/etc/smartbotic-database/config.json"
python3 -c "
import json
with open('$DB_CONFIG') as f:
    cfg = json.load(f)
s = cfg['storage']
s['migrations']['enabled'] = True
s['migrations']['directory'] = '/opt/myproject/migrations'
s['migrations']['auto_apply'] = True
with open('$DB_CONFIG', 'w') as f:
    json.dump(cfg, f, indent=2)
"
systemctl restart smartbotic-database

Migrations

Declarative JSON-based schema migrations. Create numbered JSON files in the migrations directory.

Migration File Format

{
  "version": "001",
  "name": "create_users",
  "description": "Create users collection with encryption",
  "operations": [
    {
      "type": "create_collection",
      "collection": "users",
      "options": {
        "encrypted": true,
        "max_versions": 10
      }
    }
  ]
}

Available Operations

Operation	Fields	Description
`create_collection`	`collection`, `options`	Create a new collection. Options: `encrypted`, `max_versions`, `default_ttl_seconds`, `vector_dimension`
`put`	`collection`, `document`	Insert/upsert a document
`delete`	`collection`, `id`	Delete a document
`delete_where_id_prefix`	`collection`, `prefix`	Bulk-delete documents whose ID starts with prefix

Vector-Enabled Collection Migration

{
  "version": "002",
  "name": "create_memories",
  "description": "Create vector-enabled memories collection",
  "operations": [
    {
      "type": "create_collection",
      "collection": "memories",
      "options": {
        "vector_dimension": 384
      }
    }
  ]
}

Note: vector_dimension is immutable after collection creation.

Docker Build Integration

When building your project in Docker for production, install the dev package in your base image instead of compiling the submodule from source.

Dockerfile.base Example

FROM debian:trixie

# Add Smartbotics APT repository
COPY smartbotics-repo.gpg /usr/share/keyrings/
RUN echo "deb [signed-by=/usr/share/keyrings/smartbotics-repo.gpg] \
    https://repository.smartbotics.ai trixie main" \
    > /etc/apt/sources.list.d/smartbotics.list

# Configure authentication
ARG REPO_USER=callerai
ARG REPO_PASS
RUN printf "machine repository.smartbotics.ai\nlogin %s\npassword %s\n" \
    "$REPO_USER" "$REPO_PASS" > /etc/apt/auth.conf.d/smartbotics.conf && \
    chmod 600 /etc/apt/auth.conf.d/smartbotics.conf

# Install client dev package (pulls shared lib as dependency)
RUN apt-get update && apt-get install -y \
    libsmartbotic-db-client-dev \
    && rm -rf /var/lib/apt/lists/*

Then your project's CMake uses find_package(smartbotic-db-client REQUIRED) and links against the pre-built shared library.

Production Dockerfile

Your project's production .deb should declare:

Depends: smartbotic-database (>= 1.2.0), libsmartbotic-db-client (>= 1.2.0), ...

This ensures both the server and runtime library are installed on the target machine.

CLI Administration

Install the CLI tool:

sudo apt install smartbotic-db-cli

The CLI connects to the database via gRPC. It does not require the server installed on the same machine — it can administer a remote instance.

# Connect to default (localhost:9004)
smartbotic-db-cli health

# Connect to a remote instance
smartbotic-db-cli --address 192.168.1.50:9004 health

Upgrading from Legacy Packages

If your machine previously had shadowman-database or callerai-storage installed, the new smartbotic-database package handles the transition automatically:

sudo apt install smartbotic-database

What happens:

dpkg sees Conflicts: shadowman-database / callerai-storage
The old package is removed (its prerm stops the old service)
smartbotic-database is installed
The postinst script detects old data/config and migrates:
- /var/lib/shadowman/data/database/ → /var/lib/smartbotic-database/
- /etc/shadowman/database.json → /etc/smartbotic-database/config.json (paths updated)
- Same for callerai paths

The old data directory is preserved (copied, not moved) so rollback is safe.

Rollback

If something goes wrong:

sudo apt remove smartbotic-database
sudo apt install shadowman-database=1.2.0-14   # or callerai-storage=0.5.18-1

Old packages remain in the repository.

Proto File

The raw .proto file is available at:

Package: /usr/share/smartbotic-database/proto/database.proto
Source: proto/database.proto

This can be used to generate client stubs in other languages (Python, Go, etc.) if needed.

integration-guide.md 45 KB تاريخچه خام