smartbotic-database/docs/integration-guide.md

Smartbotic Database Integration Guide

How to install, configure, and integrate smartbotic-database into your C++ project.

Table of Contents

• Installing the Database
• Installing the Client Library
• Integrating into a C++ Project
• Client API Reference
• Configuration
• Migrations
• Docker Build Integration
• CLI Administration
• Upgrading from Legacy Packages
• Drop-in configuration (conf.d)
• Eviction & Memory Pressure

---

Installing the Database

From APT Repository

Add the Smartbotics repository to your system:

# Download GPG key
curl -fsSL https://repository.smartbotics.ai/smartbotics-repo.gpg | \
    sudo gpg --dearmor -o /usr/share/keyrings/smartbotics-repo.gpg

# Configure authentication
sudo tee /etc/apt/auth.conf.d/smartbotics.conf << EOF
machine repository.smartbotics.ai
login callerai
password <repo-password>
EOF
sudo chmod 600 /etc/apt/auth.conf.d/smartbotics.conf

# Add repository
echo "deb [signed-by=/usr/share/keyrings/smartbotics-repo.gpg] https://repository.smartbotics.ai trixie main" | \
    sudo tee /etc/apt/sources.list.d/smartbotics.list

# Install
sudo apt update
sudo apt install smartbotic-database

This installs:

• /usr/bin/smartbotic-database — the server binary
• /etc/smartbotic-database/config.json — configuration file
• smartbotic-database.service — systemd service (starts automatically)

The service listens on localhost:9004 by default.

Verify Installation

# Check service status
sudo systemctl status smartbotic-database

# Check health via CLI (if installed)
smartbotic-db-cli health

Available Packages

Package	What it provides	Install when...
smartbotic-database	Server + systemd service	You need to run the database on this machine
libsmartbotic-db-client	Shared library (.so)	Your application links the client at runtime
libsmartbotic-db-client-dev	Headers + cmake config	You're building a C++ project against the client API
smartbotic-db-cli	CLI admin tool	You want to administer the database from the command line

---

Installing the Client Library

On a Development Machine (for local builds)

sudo apt install libsmartbotic-db-client-dev

This pulls in libsmartbotic-db-client (the runtime .so) as a dependency, plus all development headers needed to compile.

On a Production Machine (runtime only)

sudo apt install libsmartbotic-db-client

Only the shared library — no headers, no cmake config, no dev tooling.

---

Integrating into a C++ Project

Option A: System-installed package (recommended for Docker builds)

If libsmartbotic-db-client-dev is installed (e.g., in your Docker build image):

# CMakeLists.txt
find_package(smartbotic-db-client REQUIRED)

add_executable(myapp main.cpp)
target_link_libraries(myapp PRIVATE smartbotic::db-client)

The smartbotic::db-client target automatically sets up include paths and links the shared library.

Option B: Git submodule (for local development on non-Debian systems)

git submodule add ssh://git@git.smartbotics.ai:10022/fszontagh/smartbotic-database.git \
    external/smartbotic-database

# CMakeLists.txt
add_subdirectory(external/smartbotic-database)

add_executable(myapp main.cpp)
target_link_libraries(myapp PRIVATE smartbotic-db-client)

When built as a submodule, the library is static (.a) and linked directly into your binary.

Option C: Both (recommended)

Use the system package when available, fall back to submodule:

# Try system-installed first (Docker builds, CI)
find_package(smartbotic-db-client QUIET)
if(smartbotic-db-client_FOUND)
    message(STATUS "Using system smartbotic-db-client")
    set(DB_CLIENT_TARGET smartbotic::db-client)
else()
    # Fall back to submodule (local development)
    if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/external/smartbotic-database/CMakeLists.txt")
        add_subdirectory(external/smartbotic-database)
        set(DB_CLIENT_TARGET smartbotic-db-client)
    else()
        message(FATAL_ERROR
            "smartbotic-database not found. Either:\n"
            "  apt install libsmartbotic-db-client-dev\n"
            "  git submodule update --init external/smartbotic-database")
    endif()
endif()

add_executable(myapp main.cpp)
target_link_libraries(myapp PRIVATE ${DB_CLIENT_TARGET})

---

Client API Reference

Connecting

#include <smartbotic/database/client.hpp>

smartbotic::database::Client::Config config;
config.address = "localhost:9004";   // gRPC endpoint
config.timeoutMs = 5000;            // per-RPC timeout
config.maxRetries = 3;              // auto-retry on transient failures

smartbotic::database::Client db(config);
if (!db.connect()) {
    // handle connection failure
}

Document Operations

// Insert
nlohmann::json doc = {{"name", "Alice"}, {"email", "alice@example.com"}};
std::string id = db.insert("users", doc);
// Or with explicit ID:
std::string id = db.insert("users", doc, "user-123");

// Get
auto result = db.get("users", "user-123");
if (result) {
    std::cout << (*result)["name"] << std::endl;
}

// Update (full document replacement, with automatic optimistic locking)
db.update("users", "user-123", {{"name", "Alice Smith"}, {"email", "alice@example.com"}});

// Update with explicit optimistic locking (fails if version changed)
db.updateIfVersion("users", "user-123", {{"name", "Alice"}}, /*expectedVersion=*/2);

// Patch — atomic partial update (only specified fields are modified)
uint64_t newVer = db.patch("users", "user-123", {{"name", "Alice Smith"}});

// Upsert (insert or update)
auto [uid, isNew] = db.upsert("users", doc, "user-123");

// Delete
db.remove("users", "user-123");

// Check existence
bool found = db.exists("users", "user-123");

Concurrency: update() vs patch()

When multiple clients modify the same document concurrently, choose the right method:

patch() — use for most updates (recommended). Merges only the specified fields on the server, atomically. Two clients can safely modify different fields simultaneously:

// Client A:                                    // Client B:
db.patch("users", "user-123",                   db.patch("users", "user-123",
    {{"name", "Alice Smith"}});                      {{"email", "alice@new.com"}});
// Result: both fields updated, nothing lost

update() — full document replacement with automatic retry. Uses optimistic locking internally (reads current version, writes with version check, retries on conflict). Use when you need to replace the entire document:

auto doc = db.get("users", "user-123");
(*doc)["name"] = "Alice Smith";
(*doc)["role"] = "admin";
db.update("users", "user-123", *doc);  // replaces entire document

Note: update() is "last writer wins" for complete replacement — if two clients modify different fields via update(), the last one overwrites the other's changes. Use patch() to avoid this.

updateIfVersion() — explicit optimistic locking. Use when you need to detect conflicts and handle them yourself:

auto doc = db.get("users", "user-123");
uint64_t version = (*doc)["_version"];
(*doc)["balance"] = (*doc)["balance"].get<int>() + 100;
if (!db.updateIfVersion("users", "user-123", *doc, version)) {
    // Version conflict — someone else modified the document
    // Re-read and retry, or report error to user
}

Views

Views are named, read-only projections over collections. They filter which fields are returned and optionally enforce baked-in filters and default sort order. Views are ideal for:

• Schema evolution — add fields to a collection without affecting consumers that query through a view
• Security boundary — hide sensitive fields (passwords, tokens) and enforce row-level constraints (status=active, tenant=X)
• API clarity — name and document a stable contract ("this view returns these fields with these filters")

Creating a view

using Op = smartbotic::database::Client::FilterOp;

// Simple: just project fields
db.createView("users_public",
              /*collection=*/"users",
              /*include=*/{"id", "name", "avatar_url", "profile.public_bio"});

// Full-featured: fields + filters + default sort
db.createView("active_admins",
              /*collection=*/"users",
              /*include=*/{"id", "name", "role"},
              /*exclude=*/{},
              /*where=*/{
                  {"status", Op::EQ, "active"},
                  {"role",   Op::EQ, "admin"}
              },
              /*defaultSort=*/smartbotic::database::Client::Sort{"name", false});

Querying a view

Views work like collections — use the view name in get(), find(), count():

auto user = db.get("users_public", "u1");         // projection applied
auto results = db.find("active_admins", {
    .filters = {{"department", Op::EQ, "eng"}},   // AND-merged with view's where
    .limit = 50
});
auto total = db.count("active_admins", {{"department", Op::EQ, "eng"}});

Inspecting and dropping

auto views = db.listViews();
auto info = db.getViewInfo("users_public");
db.dropView("users_public");

Semantics

Dimension	Behavior
Filters	View AND caller — view's where is always applied. Caller's filters are merged with AND. Consumers cannot escape the view's constraints.
Sort	Caller overrides view — if the caller specifies a sort, it wins. Otherwise the view's defaultSort is used.
Include	Whitelist — when non-empty, only those paths are returned.
Exclude	Blacklist — ignored if include is non-empty; otherwise removes listed paths.
Metadata	_id, _version, _created_at, _updated_at, _created_by, _updated_by are always returned regardless of include/exclude.
Nested paths	Dot-notation supported: user.profile.name. Stops at arrays.
Writes	Rejected — insert(), update(), patch(), upsert(), remove() on a view name return an error.
View-of-view	Not allowed — views must target real collections.
Project scope (2.4.2+)	Views belong to a project, exactly like collections. A view created by a client with project = "acme" is addressable only from that project, and two projects may each own a view of the same name. listViews() returns only the calling project's views.
Unknown name (2.4.2+)	A name matching neither a view nor a collection makes find() throw std::runtime_error (gRPC NOT_FOUND). Before 2.4.2 it returned an empty result, which was indistinguishable from an empty collection.

Views and projects (2.4.2+)

View names are namespaced with the client's Config::project, so you keep using the bare name and the client qualifies it:

Client::Config cfg;
cfg.project = "acme";
Client db(cfg);

db.createView("adults", "users", {"name", "age"});  // -> acme:adults over acme:users
auto rows = db.find("adults", {});                  // resolves within acme only

A different workspace may define its own adults over its own users without colliding, and neither can read through the other's view.

Upgrading from <= 2.4.1. Views were unreachable through the client on 2.3.0-2.4.1: createView registered the bare name while every read path sent the project-qualified one, so lookups always missed and returned an empty result with no error. Existing view definitions are re-keyed into the default project automatically on first boot of 2.4.2 (idempotent, logged as ViewManager: re-keyed N legacy view(s)). No operator action required. Callers that treated the empty result as "view has no rows" will start seeing real rows.

Via migrations (recommended)

{
  "version": "020",
  "name": "create_active_admins_view",
  "operations": [
    {
      "type": "create_view",
      "name": "active_admins",
      "collection": "users",
      "include": ["id", "name", "role"],
      "where": [
        {"field": "status", "op": "EQ", "value": "active"},
        {"field": "role",   "op": "EQ", "value": "admin"}
      ],
      "default_sort": {"field": "name", "descending": false}
    }
  ]
}

The op field in where filters accepts either a string name (EQ, NE, GT, GTE, LT, LTE, IN, CONTAINS, EXISTS, REGEX, SEARCH) or the matching integer. Migrations are idempotent — re-running a migration that creates an existing view is a no-op.

Why AND-merge filters?

A view's filters are applied server-side, before projection, and a consumer filter is ANDed on top rather than replacing them — so a caller querying active_admins cannot widen the result set beyond what the view defines. That makes a view a reliable schema and shape contract: what comes back through this name is always a subset of what the view allows.

A view is NOT a security boundary. Earlier revisions of this guide described view filters as "a security enforcement boundary" and compared them to row-level security. That was wrong, and should not be used as the basis of a threat model. Three reasons:

1. There is no caller identity. Authentication (2.4+) compares a bearer token against a flat list of shared keys per listener. It attaches no principal to the call, so the server cannot know who is asking — and every key holder has identical, complete access.
2. The underlying collection stays readable. Views only reject writes on a view name. find("active_admins") narrows; find("users") does not. Nothing stops a client bypassing the view entirely.
3. Projects are not a boundary either. Config::project is chosen by the client, and a fully-qualified other_project:collection name is accepted from any client, so namespaces isolate naming, not access.

Views constrain what a cooperating client sees. Treat them as ergonomics and API hygiene. Real row- and column-level security requires a per-principal identity and a policy evaluated at every read handler; that is planned but not present today.

Related: field-level encryption (sensitiveFields) is encryption at rest, not access control. Sensitive fields are decrypted for every authenticated reader; the protection is against someone reading data files, snapshots, or backups, not against a client.

Collection Configuration

Per-collection runtime settings. Currently supports timestamp precision; extensible for future knobs.

Timestamp precision

Every document carries _created_at and _updated_at — int64 timestamps since epoch.

Since 2.2.0 the default is nanoseconds (ns). It was milliseconds up to 2.1.x. The flip was made because collections that receive rapid-fire writes (LLM streaming chunks, agent tool loops) saw ties at ms resolution that made ORDER BY _created_at ambiguous; ns gives unique, strictly ordered timestamps out of the box. Collections that already had a stored config keep their value — only collections created after the upgrade without an explicit configureCollection() inherit the new default.

using Cfg = smartbotic::database::Client::CollectionConfig;

// Default since 2.2.0: ns (no configuration needed)
auto id = db.insert("llm_tokens", {{"token", "hello"}});
// doc["_created_at"] is ~19 digits, nanoseconds since epoch

// Opt a collection back to milliseconds
db.configureCollection("logs", Cfg{"ms"});

// Future inserts into logs get ms-precision timestamps
auto id2 = db.insert("logs", {{"msg", "..."}});
// doc["_created_at"] is ~13 digits, milliseconds since epoch

Note: Client::CollectionConfig::timestampPrecision still defaults to "ms" in the header, so Cfg{} sends an explicit "ms" rather than meaning "server default". Pass the precision you want explicitly, or leave the string empty to mean "leave unchanged" (2.4.5+, see partial updates below).

Reading the config

auto cfg = db.getCollectionConfig("logs");
std::cout << cfg.timestampPrecision << std::endl;  // "ms"

bool explicit_set = db.hasCollectionConfig("logs");        // true
bool default_only = db.hasCollectionConfig("llm_tokens");  // false (still ns default)

Version history on/off (2.4.5+)

Version history can be switched per collection, on a collection that already exists and already holds data:

Cfg off;
off.timestampPrecision = "";      // empty = leave precision unchanged
off.versioningEnabled  = false;   // stop recording new versions
db.configureCollection("metrics_events", off);

// ... later, turn it back on
Cfg on;
on.timestampPrecision = "";
on.versioningEnabled  = true;
db.configureCollection("metrics_events", on);

Semantics:

• Reversible. Disabling stops new versions being recorded. History already on disk stays readable via getVersionHistory() / getDocumentVersion(), so re-enabling simply resumes appending.
• Not the same as maxVersions = 0, which means "keep unlimited history". versioningEnabled = false means "keep none".
• Durable. The setting lives in the _collection_meta system collection, so it survives restarts through the normal WAL + snapshot path.
• Takes effect on the next write.

Partial updates (2.4.5+)

configureCollection() is a partial update. Fields you leave unset are preserved, not reset:

• empty timestampPrecision → precision unchanged

• unset versioningEnabled (std::nullopt) → versioning unchanged

  Cfg justPrecision;
  justPrecision.timestampPrecision = "ms";   // versioningEnabled stays nullopt
  db.configureCollection("logs", justPrecision);
  // -> precision becomes ms; versioning is left exactly as it was
  

On getCollectionConfig(), versioningEnabled is always populated with the collection's effective value rather than the "leave unchanged" sentinel.

Before 2.4.5 this RPC replaced the whole config. That was harmless while precision was the only field, but it is why versioning_enabled is declared optional in the proto: a plain proto3 bool defaults to false, so any precision-only call would otherwise have silently switched versioning off.

What configureCollection does NOT do

It does not convert existing timestamps. Flipping a collection from ms to ns only affects future inserts. Pre-existing docs keep whatever precision they had. This keeps the config flip atomic and reversible.

Consequences of mixed precision within one collection:

• int64 ordering within the collection is still consistent (ms values are always < 10^15, ns values are always > 10^15, so ns docs sort after ms docs). Queries don't break.
• But the ns-labeled docs appear to be in the future relative to the ms-labeled docs unless you convert.

Migrating existing timestamps

When you're ready to convert legacy data, run the migration helper during a planned maintenance window:

auto result = db.migrateCollectionTimestamps("llm_tokens",
                                              /*fromPrecision=*/"ms",
                                              /*toPrecision=*/"ns");
if (result.success) {
    std::cout << "migrated " << result.rowsMigrated
              << " rows, skipped " << result.rowsSkipped << std::endl;
} else {
    std::cerr << "failed: " << result.error << std::endl;
}

Semantics:

• ms → ns: multiply _created_at / _updated_at by 10^6
• ns → ms: divide by 10^6 (truncation — loses sub-ms precision)
• Idempotent and resumable. Rows that are already in the target range are detected via a 10^15 threshold and skipped. If the migration is interrupted (network drop, deadline), rerun it — already-converted rows will count as rows_skipped.
• from == to is a no-op success.
• Metadata-only — document version numbers are NOT incremented, WAL is not appended, history is not updated. This is an infrastructure rewrite, not a user-visible change.
• Deadline: the client's RPC timeout for this call is 10 minutes — large collections may take a while.

Recommended workflow for switching precision on production data

1. Schedule a maintenance window (writes against the collection may pause briefly if needed).
2. Call configureCollection(name, {ns}) — atomic config flip; future writes immediately use ns.
3. Call migrateCollectionTimestamps(name, "ms", "ns") — converts existing docs.
4. Verify with a sample read that _created_at values are now in the ns range.
5. Resume normal traffic.

Between steps 2 and 3, new writes are ns and old docs are ms — this is temporary but normal. Ordering still works because the 10^15 threshold cleanly separates the two ranges.

Via migrations

To set timestamp precision declaratively, call db.configureCollection() from your consumer project's startup code. The configure_collection migration operation will be available in a future version.

{
  "version": "021",
  "name": "llm_tokens_ns_precision",
  "operations": [
    {
      "type": "configure_collection",
      "collection": "llm_tokens",
      "config": { "timestamp_precision": "ns" }
    }
  ]
}

Recovery & Read-Only Mode

smartbotic-database uses WAL + snapshots for durability. When starting up, the server tries to recover state from the newest snapshot and replay WAL entries since that snapshot. If something goes wrong, it uses a tiered recovery system modeled on MySQL's innodb_force_recovery.

Recovery modes

Level	Name	Behavior
0	normal	Load latest snapshot + replay WAL. Default. If auto_escalate=true (default), falls back to older snapshot automatically when latest is corrupt.
1	snapshot_fallback	Try snapshots newest→oldest, first that loads wins.
2	wal_only	Ignore snapshots, replay WAL from sequence 0. Slow on large DBs.
3	best_effort	Try snapshot_fallback, then fall through to wal_only.
4	force_empty	Start empty. Snapshots + WAL preserved on disk for forensics.

Operator escalation path when recovery fails:

# After a refuse-to-start, try older snapshots
smartbotic-database --recovery-mode=snapshot_fallback

# If that still fails, replay WAL from scratch (slow)
smartbotic-database --recovery-mode=wal_only

# Last-ditch: try everything
smartbotic-database --recovery-mode=best_effort

# Absolute last resort: start empty (data preserved on disk)
smartbotic-database --recovery-mode=force_empty

Or set in config.json:

"persistence": {
    "recovery": {
        "mode": "normal",
        "auto_escalate": true,
        "allow_empty_on_fresh_install": true
    }
}

The CLI flag overrides the config for one-shot recoveries.

Auto-readonly on non-trivial recovery

If recovery used anything other than "latest snapshot loaded cleanly," the DB boots in read-only mode. This is a safety feature — writing onto a possibly-stale state can corrupt your data further. The operator must explicitly acknowledge the state before writes resume.

On startup you'll see (at ERROR level):

[ERROR] Database booted in READ-ONLY mode after non-trivial recovery
        Reason: fell back to snapshot snapshot-20260419-090719.dat because
                snapshot-20260419-100747.dat failed: body truncated

Writes will be REJECTED until you acknowledge this state:
  smartbotic-db-cli unlock                  # live, no restart
  smartbotic-database --force-readwrite     # on next restart

Workflow: recovering a broken instance

1. Preserve forensics. Stop the service, snapshot /var/lib/smartbotic-database/ before anything else.
2. Diagnose. Start the server (default mode = normal). It will either recover cleanly or refuse with a clear banner. Use smartbotic-db-cli status to see what happened.
3. Escalate. If recovery refused, restart with --recovery-mode=snapshot_fallback (and upwards as needed).
4. Extract data. While the DB is in read-only mode, connect with the CLI or client library and read whatever survived. Dump/export facilities are planned for v1.7.0.
5. Decide.
   • Data looks complete → smartbotic-db-cli unlock to resume writes.
   • Data is missing/stale → don't unlock. Extract to an external backup, then wipe /var/lib/smartbotic-database/ and start clean.

Client API

// Acknowledge auto-readonly state and resume writes
db.unlock();

// Force read-only for maintenance (regardless of recovery outcome)
db.lock();

// Inspect state + recovery outcome
auto s = db.getReadOnlyStatus();
std::cout << "read-only:        " << s.readOnly << "\n"
          << "reason:           " << s.reason << "\n"
          << "recovery outcome: " << s.recoveryOutcome << "\n"
          << "expected snap:    " << s.expectedSnapshot << "\n"
          << "snapshot used:    " << s.snapshotUsed << "\n"
          << "failure reason:   " << s.failureReason << "\n"
          << "WAL replayed:     " << s.walEntriesReplayed << "\n"
          << "snapshots tried:  " << s.snapshotsAttempted << "\n";

CLI

# Inspect read-only + recovery state
smartbotic-db-cli status

# Acknowledge non-trivial recovery and accept writes
smartbotic-db-cli unlock

# Manually lock (e.g. before a maintenance window)
smartbotic-db-cli lock

Snapshot durability settings

Under persistence.snapshots:

"snapshots": {
    "validate_after_write": true,        // verify every snapshot immediately after creation
    "cleanup_only_if_verified": true     // never evict old snapshots if the new one failed validation
}

Both default to true — turn off only if you really know what you're doing (disables protections against silent writer corruption).

Exit codes

Code	Meaning
0	Normal shutdown
1	Generic error
10	Recovery refused. Snapshot or WAL failure + mode=normal + data exists on disk. Operator must escalate the recovery mode or pass --force-readwrite.
11	Config invalid (e.g. bad --recovery-mode value)

--force-readwrite

Bypasses the auto-readonly guard on non-trivial recovery. Use after you've reviewed the recovery state via smartbotic-db-cli status and are certain writes onto that state are safe.

# After confirming recovery state looks good
smartbotic-database --force-readwrite

Querying

// Find with filters (equality is default)
smartbotic::database::Client::QueryOptions opts;
opts.filters = {
    {"status", "active"},   // EQ (default) — 2-arg Filter constructor
    {"role", "admin"},      // EQ
};
opts.sortField = "name";
opts.sortDescending = false;
opts.limit = 50;
opts.offset = 0;

auto docs = db.find("users", opts);
for (const auto& doc : docs) {
    std::cout << doc["name"] << std::endl;
}

// Find all documents in a collection
auto allDocs = db.find("users");

// Count
uint64_t total = db.count("users");
uint64_t admins = db.count("users", {{"role", "admin"}});

Filter operators

Filters support a range of operators via Client::FilterOp. The 2-argument Filter(field, value) constructor defaults to EQ — existing brace-init code continues to compile and behave the same. To use other operators, pass the operator explicitly with the 3-argument constructor:

using Op = smartbotic::database::Client::FilterOp;

smartbotic::database::Client::QueryOptions opts;
opts.filters = {
    {"age", Op::GTE, 18},                                             // age >= 18
    {"status", Op::NE, "deleted"},                                    // status != "deleted"
    {"tags", Op::CONTAINS, "admin"},                                  // tags array contains "admin"
    {"role", Op::IN, nlohmann::json::array({"admin", "moderator"})},  // role IN [...]
    {"email", Op::REGEX, ".*@example\\.com$"},                        // regex match
    {"name", Op::SEARCH, "alice"},                                    // full-text search
    {"deleted_at", Op::EXISTS, false},                                // field does NOT exist
};
auto docs = db.find("users", opts);

Operator	Semantics	Value type
FilterOp::EQ	field equals value	any JSON
FilterOp::NE	field does not equal value	any JSON
FilterOp::GT	field > value	number or string
FilterOp::GTE	field >= value	number or string
FilterOp::LT	field < value	number or string
FilterOp::LTE	field <= value	number or string
FilterOp::IN	field value appears in the supplied array	JSON array
FilterOp::CONTAINS	field (array) contains the supplied value	any JSON
FilterOp::EXISTS	field presence check (true = must exist, false = must not)	bool
FilterOp::REGEX	field matches the supplied regex pattern	string
FilterOp::SEARCH	full-text search across document ID and string fields	string

Note: the old _search magic field name that auto-selected SEARCH is no longer special — use FilterOp::SEARCH explicitly instead.

Collection Management

// Create a basic collection
db.createCollection("users");

// Create with options
db.createCollection("sessions",
    /*defaultTtlSeconds=*/3600,    // auto-expire after 1 hour
    /*encrypted=*/false,
    /*maxVersions=*/5,             // keep last 5 versions
    /*vectorDimension=*/0);        // no vector support

// Create vector-enabled collection
db.createCollection("memories",
    /*defaultTtlSeconds=*/0,
    /*encrypted=*/false,
    /*maxVersions=*/0,
    /*vectorDimension=*/384);      // 384-dim embeddings

// List and inspect
auto collections = db.listCollections();
auto info = db.getCollectionInfo("users");
if (info) {
    std::cout << "Documents: " << info->documentCount << std::endl;
}

// Drop
db.dropCollection("old_data");

Collections and projects (2.4.5+)

Collection names are namespaced with the client's Config::project, exactly like documents. You keep passing bare names; the client qualifies them on the wire and un-qualifies them coming back, so listCollections() round-trips:

Client::Config cfg;
cfg.project = "acme";
Client db(cfg);

db.createCollection("users");              // -> acme:users
db.insert("users", {{"name", "Ada"}});     // -> acme:users
auto info = db.getCollectionInfo("users"); // -> acme:users
db.listCollections();                      // -> { "users", ... }

Fixed in 2.4.5 — upgrade if you call createCollection. On 2.3.0-2.4.4 createCollection, dropCollection and getCollectionInfo sent the bare name while insert/get/find sent the project-qualified one. Because the client qualifies against Config::project unconditionally — including default — every project was affected. One createCollection("things") plus one insert("things") created two collections: the bare one received the options and stayed empty, while <project>:things was created implicitly by the insert with defaults. No error was raised.

Concretely, on affected versions:

• vectorDimension never reached the collection holding the data, so similaritySearch() returned no hits.
• encrypted = true silently did not apply.
• dropCollection() returned true while every document remained readable — it dropped the empty phantom.

After upgrading: leftover empty phantom collections are harmless litter, but options that were swallowed do not retroactively apply. createCollection on a now-correctly-resolved existing collection returns false and only updates maxVersions. Since vectorDimension is immutable after creation, a collection that needs vector search must be recreated: create a new collection with the right vectorDimension, copy the documents across, then drop the old one.

Note that listCollections() currently returns every project's collections — names outside your own project stay visibly qualified rather than being flattened into yours. Unlike ListViewsRequest, ListCollectionsRequest has no project filter yet.

Vector Similarity Search

Requires a collection created with vectorDimension > 0.

// Insert documents with embeddings
nlohmann::json doc = {
    {"id", "mem-1"},
    {"content", "The user prefers dark mode"},
    {"_vector", embedding}   // std::vector<float>, size must match vectorDimension
};
db.insert("memories", doc, "mem-1");

// Search
std::vector<float> queryVec = getEmbedding("user preferences");
auto results = db.similaritySearch("memories", queryVec, /*topK=*/5, /*minScore=*/0.7f);

for (const auto& r : results) {
    std::cout << r.id << " (score=" << r.score << "): "
              << r.data["content"] << std::endl;
}

Notes:

• _vector is extracted and stored separately — not returned in get() or find() responses
• Dimension is validated on every insert — mismatches are rejected
• SIMD-accelerated (AVX2/SSE4.1) brute-force cosine similarity

Version History

// Get history
auto history = db.getVersionHistory("users", "user-123", /*limit=*/10);
for (const auto& v : history.versions) {
    std::cout << "v" << v.version << " by " << v.updatedBy
              << " at " << v.timestamp << std::endl;
}

// Get a specific version
auto oldVersion = db.getDocumentVersion("users", "user-123", /*version=*/2);

// Restore to a previous version
uint64_t newVer = db.restoreVersion("users", "user-123", /*version=*/2);

// Restore to a point in time
auto [fromVer, newVersion] = db.restoreToDate("users", "user-123", timestamp);

File Storage

Two-level storage with content-addressed blob deduplication. Identical files are stored once.

Files and projects (2.6.0+)

Files are namespaced by project, like collections. The client fills the project from Config::project on every file call, so you keep passing bare ids:

Client::Config cfg;
cfg.project = "acme";
Client db(cfg);

auto up = db.uploadFile(data, meta);   // owned by project "acme"
db.getFileInfo(up.id);                 // resolves
db.listFiles("document");              // only acme's files

An id belonging to another project is reported as NOT_FOUND, not as "forbidden" — getFileInfo returns nullopt, downloadFile throws, deleteFile returns false, and listFiles omits it. The indistinguishability is deliberate: a distinct "exists but not yours" answer would confirm that a peer project holds that file.

Before 2.6.0, files had no project at all — one shared blob tree, one flat record tree, and no project field on any file RPC. Any client could read, list, or delete any file by id. On upgrade, existing records are stamped with project: "default" automatically (idempotent, logged at INFO with a count). If your consumer used a non-default project and uploaded files, those files are now owned by default and will not be visible to that project — move them by re-uploading under the intended project.

FileRecord gained a project member, which changes the struct size, so consumers must rebuild against 2.6.0 headers.

Deduplication is per project

result.deduplicated means this project already referenced those bytes. Blob storage still dedups globally, so the disk saving is unchanged, but the flag is scoped deliberately: reporting the global hit would let you upload a candidate file and learn from the flag whether some other project holds it.

// Upload
std::vector<uint8_t> data = readFile("photo.jpg");
smartbotic::database::Client::FileUploadMeta meta;
meta.name = "photo.jpg";
meta.mime_type = "image/jpeg";
meta.file_type = "document";        // "plugin", "document", "generated"
meta.related_id = "conversation-1";
meta.is_public = false;

auto result = db.uploadFile(data, meta);
std::cout << "File ID: " << result.id
          << " Deduplicated: " << result.deduplicated << std::endl;

// Download
auto fileData = db.downloadFile(result.id);

// Get file info
auto fileInfo = db.getFileInfo(result.id);

// List files
auto files = db.listFiles(
    /*file_type=*/"document",
    /*related_id=*/"conversation-1",
    /*limit=*/100, /*offset=*/0,
    /*checksum=*/"",
    /*name=*/"photo.jpg");

// Delete
db.deleteFile(result.id);

Event Subscription

auto subscription = db.subscribe({"users", "sessions"}, [](
    const std::string& collection,
    const std::string& id,
    const std::string& eventType,
    const std::optional<nlohmann::json>& data) {

    std::cout << eventType << " on " << collection << "/" << id << std::endl;
});

// Unsubscribe by releasing the handle
subscription.reset();

Set Operations

Redis-compatible set operations on top of documents.

db.setAdd("tags", "user-123", "admin");
db.setAdd("tags", "user-123", "active");

auto members = db.setMembers("tags", "user-123");
bool isAdmin = db.setIsMember("tags", "user-123", "admin");

db.setRemove("tags", "user-123", "active");

Health & Statistics

// Quick health check
if (db.healthCheck()) {
    std::cout << "Database is healthy" << std::endl;
}

// Detailed health
auto health = db.getHealthInfo();
if (health) {
    std::cout << "Uptime: " << health->uptimeMs << "ms"
              << " Docs: " << health->documentCount << std::endl;
}

// Full statistics
auto stats = db.getStats();
if (stats) {
    std::cout << "Memory: " << stats->memoryUsedBytes << " bytes"
              << " Insert avg: " << stats->insertAvgMicros() << " us" << std::endl;
}

---

Configuration

The server configuration file is at /etc/smartbotic-database/config.json.

Default Configuration

{
  "log_level": "info",
  "storage": {
    "bind_address": "localhost",
    "rpc_port": 9004,
    "node_id": "smartbotic-db",
    "data_directory": "/var/lib/smartbotic-database",
    "memory": {
      "max_memory_mb": 512,
      "eviction_threshold_percent": 80,
      "eviction_target_percent": 60,
      "eviction_check_interval_ms": 5000
    },
    "persistence": {
      "wal_sync_interval_ms": 100,
      "snapshot_interval_sec": 3600,
      "compression": "lz4"
    },
    "encryption": {
      "enabled": true,
      "key_file": "/var/lib/smartbotic-database/storage.key",
      "auto_generate_key": true
    },
    "migrations": {
      "enabled": false,
      "directory": "/etc/smartbotic-database/migrations",
      "auto_apply": false
    },
    "files": {
      "max_file_size_mb": 500,
      "cleanup_orphans_interval_sec": 3600
    },
    "replication": {
      "enabled": false
    }
  }
}

Key Settings

Setting	Default	Description
bind_address	localhost	Change to 0.0.0.0 to accept remote connections
rpc_port	9004	gRPC listen port
data_directory	/var/lib/smartbotic-database	WAL, snapshots, files stored here
memory.max_memory_mb	512	Maximum memory for document cache. Evicts LRU when exceeded
memory.eviction_max_episode_percent	50	(2.4.3+) Ceiling on the share of the resident set one pressure episode may evict. 0 disables. See the drain-cap note below
persistence.snapshot_interval_sec	3600	Full snapshot every N seconds (WAL truncated after)
encryption.enabled	true	Field-level AES-256-GCM encryption. Key auto-generated on first start
migrations.enabled	false	Enable to auto-apply JSON migrations on startup
migrations.directory	/etc/smartbotic-database/migrations	Path to migration JSON files

gRPC Configuration

Under storage.grpc (added in v1.6.2):

Setting	Default	Purpose
max_receive_message_size_mb	100	Max inbound gRPC message size. Raise for larger file uploads.
max_send_message_size_mb	100	Max outbound gRPC message size.
resource_quota_memory_mb	256	Global memory budget for gRPC inbound buffers across all RPCs combined.
max_concurrent_subscribe_streams	50	Max simultaneous Subscribe event-stream clients. Excess gets RESOURCE_EXHAUSTED.
max_concurrent_file_streams	10	Max simultaneous UploadFile + DownloadFile streams (shared pool).

When a concurrency limit is hit, the server returns RESOURCE_EXHAUSTED immediately and logs WARN with the current / max counts. Clients should retry with backoff.

Drop-in configuration (conf.d)

In addition to /etc/smartbotic-database/config.json, the server loads any *.json files from /etc/smartbotic-database/conf.d/ and deep-merges them into the running config in lexicographic order.

This lets consumer projects (shadowman-cpp, callerai, your own apps) ship their own tuning without touching the upstream config.json that dpkg --configure guards as a conffile.

Merge semantics (RFC 7396)

• Objects merge recursively: sibling fields not mentioned in a drop-in are preserved from the lower-priority layer.
• Arrays and scalars are replaced entirely by a higher-priority layer.

Example:

// /etc/smartbotic-database/config.json
{
  "log_level": "info",
  "storage": {
    "rpc_port": 9004,
    "memory": { "max_memory_mb": 512 }
  }
}

// /etc/smartbotic-database/conf.d/50-shadowman.json  (shipped by shadowman-cpp's deb)
{
  "storage": {
    "memory": { "max_memory_mb": 2048 }
  }
}

// Effective merged config — sibling rpc_port preserved, max_memory_mb overridden
{
  "log_level": "info",
  "storage": {
    "rpc_port": 9004,
    "memory": { "max_memory_mb": 2048 }
  }
}

File naming conventions

Use numeric prefixes for deterministic merge order. Last writer wins.

File	Owner	Purpose
00-defaults.json	smartbotic-database (optional)	Baseline conservative tunings
50-<project>.json	consumer project's deb postinst	Workload-tuned values
99-local.json	operator	Site-specific overrides

Files are merged in filename-sort order, so 00- applies before 50- applies before 99-.

Failure modes

• Missing base config.json — OK, treated as {}
• Missing conf.d/ directory — OK
• Syntax error in any file — server refuses to start, exit code 11, log identifies the file + parser location
• Unknown keys — lenient. Individual config consumers may log WARN but don't block startup. This lets v1.x consumers drop a conf.d/ file that mentions a key only v1.(x+1) knows about

What gets logged

At startup, the server prints each file as it merges it:

[INFO] Loading base config: /etc/smartbotic-database/config.json
[INFO] Merging drop-in: /etc/smartbotic-database/conf.d/50-shadowman.json
[INFO] Merging drop-in: /etc/smartbotic-database/conf.d/99-local.json

If a consumer wonders which tuning values are actually in effect, this is where to look.

Eviction & Memory Pressure

smartbotic-database holds documents in memory for fast reads. When usage approaches the configured budget (storage.memory.max_memory_mb), eviction removes the least-useful docs to free space. Eviction is chunked, throttled, and hot-write-aware so it doesn't block foreground writes — a design requirement after the 2026-04-22 Zoe incident where one-shot eviction stalled LLM streaming writes long enough to miss their gRPC deadline.

Pressure levels

The server tracks four pressure levels based on current usage vs. max_memory_mb:

Level	Default %	Behavior
normal	< 70%	No eviction
soft	70–85%	Trickle eviction: one chunk per tick
hard	85–95%	Aggressive eviction: up to max_eviction_passes_per_trigger / 2 chunks per tick. Fires MEMORY_PRESSURE_HIGH event (edge-triggered)
emergency	≥ 95%	All writes rejected with RESOURCE_EXHAUSTED until pressure drops below hard. Eviction runs at full rate

Query current state via smartbotic-db-cli status or the GetMemoryStats RPC.

Drain cap (2.4.3+)

Eviction assumes the memory estimate falls as documents leave. When memory is held by something eviction cannot free, that assumption breaks: the target is never reachable, and eviction keeps trimming until the cache is empty. On 2026-08-03 a production instance did exactly this — 5996 of ~5990 documents evicted over 30 minutes while the estimate rose (347 → 358 MB). Every collection then reported 0 docs, which looks indistinguishable from data loss. No data was actually lost; it was all still in LMDB.

Two guards now bound this:

• Episode cap — one pressure episode may evict at most memory.eviction_max_episode_percent (default 50%) of the resident set it started with, enforced per chunk. An episode re-arms only after pressure has been normal for 3 consecutive ticks.
• No-progress detector — after 3 consecutive ticks that evict documents without lowering the estimate, eviction pauses until pressure clears.

Both log at ERROR. Hitting either means the memory estimate is wrong, not that the store is oversized — treat it as an alert, raise storage.memory.max_memory_mb, and check what is holding memory. Set eviction_max_episode_percent: 0 to disable the cap (not recommended).

Note for v2.0+. MemoryStore is a cache in front of LMDB; the durable copy lives in <dataDir>/projects/<name>/env/. Eviction draining the cache only becomes visible if the LMDB mirror is also unhealthy, which is what made the above incident look total. Watch for v2.0 mirror failed in the log.

Chunked eviction

Eviction never stops the world. Each tick (default every eviction_check_interval_ms = 5000ms, but reactive to pressure):

1. Collects candidates — docs whose updatedAt is older than hot_write_floor_ms (30 s default) ago, in non-pinned collections
2. Sorts by (memory_priority, lastAccessedAt) — low-priority + cold goes first
3. Applies per-collection budget: no one collection contributes more than (share × 1.5) of a single chunk
4. Evicts eviction_chunk_size docs (default 1000) in one pass
5. Sleeps eviction_chunk_pause_ms (default 50 ms)
6. Repeats up to max_eviction_passes_per_trigger times, stopping early if target reached

Docs being actively written (quiesce) are skipped via an in-flight-write counter — eviction can't land mid-upsert.

Per-collection priority

Use CollectionOptions.memory_priority to bias eviction:

Value	Meaning
low	Evicted preferentially — cold archive collections
normal (default)	Standard LRU weight
high	Evicted last — hot operational collections
(pinned=true on CollectionOptions)	Never evicted

Set via createCollection or migration:

{
  "type": "create_collection",
  "collection": "messages",
  "options": {
    "memory_priority": "high"
  }
}

Config reference (storage.memory)

Setting	Default	Meaning
max_memory_mb	512	Memory budget before eviction kicks in
eviction_chunk_size	1000	Docs evicted per chunk
eviction_chunk_pause_ms	50	Sleep between chunks within one tick
max_eviction_passes_per_trigger	20	Safety cap on chunks per tick
hot_write_floor_ms	30000	Docs updated within this window are unevictable
memory_soft_percent	70	Enter soft pressure
memory_hard_percent	85	Enter hard pressure — fires PRESSURE_HIGH event
memory_emergency_percent	95	Enter emergency — admission control rejects writes
eviction_burst_threshold	10000	Fire MEMORY_EVICTION_BURST event if chunk ≥ this
eviction_target_percent	60	Stop evicting once usage drops below this
eviction_check_interval_ms	5000	Wake eviction loop this often

Events

Subscribe to receive memory-related notifications:

Event type	When	Payload
MEMORY_PRESSURE_HIGH	Edge-triggered: pressure enters hard or emergency	{pressure_level, pressure_percent, estimated_bytes, max_bytes}
MEMORY_EVICTION_BURST	An eviction pass touched ≥ eviction_burst_threshold docs	{evicted_docs, bytes_freed, pressure_level}

Both are system-level events (empty collection / id). Subscribers filter by event type.

Client-side retry

The client library (v1.7.0+) automatically retries write RPCs that return transient errors:

Status	Retry?
DEADLINE_EXCEEDED	✅
RESOURCE_EXHAUSTED (admission control / concurrency cap)	✅
UNAVAILABLE	✅
All others (INVALID_ARGUMENT, FAILED_PRECONDITION, etc.)	❌ — final

Config under Client::Config:

Field	Default	Meaning
writeRetries	3	Max retry attempts
writeRetryBackoffMs	100	Initial backoff (exponential: 100 → 200 → 400 …)
writeRetryMaxBackoffMs	5000	Cap on any single backoff
writeRetryJitter	0.25	± 25% jitter

Writes wrapped: insert, updateIfVersion, upsert, remove, patch. Reads are NOT auto-retried — callers retry at their discretion.

Observability

GetMemoryStats RPC (and Client::getMemoryStats()) returns:

• Total / max memory bytes, pressure percent + level
• Per-collection: document_count, estimated_bytes, evicted_stub_count, priority

• Last eviction: timestamp, docs evicted, bytes freed

  auto s = db.getMemoryStats();
  std::cout << "Memory: " << s.totalMemoryBytes << "/" << s.maxMemoryBytes
        << " (" << s.pressurePercent << "% — " << s.pressureLevel << ")" << std::endl;
  for (const auto& c : s.collections) {
  std::cout << "  " << c.collection << ": " << c.documentCount << " docs, "
            << c.estimatedBytes << " bytes, priority=" << c.priority << std::endl;
  }
  

Also shown in smartbotic-db-cli status output (if the CLI exposes it; otherwise subscribers can consume the events).

Recommended tunings

• Default (512 MB) — development or small services with ~100 K docs

• 2 GB — production shadowman-cpp profile:

  // /etc/smartbotic-database/conf.d/50-shadowman.json
  {
  "storage": {
  "memory": {
    "max_memory_mb": 2048,
    "memory_soft_percent": 65,
    "memory_hard_percent": 80,
    "eviction_chunk_size": 500,
    "hot_write_floor_ms": 60000
  }
  }
  }
  

• 8+ GB large — consider raising eviction_burst_threshold to 50000 so you don't get event spam, and consider hot_write_floor_ms: 120000 if your write paths are slow (e.g. STT-heavy callerai)

Troubleshooting

"Client::insert failed: RESOURCE_EXHAUSTED (memory pressure emergency)" — server is in emergency pressure, client is retrying. If this persists, raise max_memory_mb in a conf.d drop-in or lower memory_emergency_percent.

"Eviction: nothing evictable this pass" — every candidate is either pinned, hot-written, or in-flight. Usually benign; server will re-try next tick. If persistent, consider reducing hot_write_floor_ms or flagging some collections as unpinned/low-priority.

Pressure stays at hard despite quiet writes — check eviction_target_percent; eviction stops when usage drops below this. If you want eviction to drain further during idle, raise max_memory_mb or lower the target.

Configuring for Your Project

Consumer projects typically enable migrations by configuring the database via their own postinst script:

# Example: in your project's postinst
DB_CONFIG="/etc/smartbotic-database/config.json"
python3 -c "
import json
with open('$DB_CONFIG') as f:
    cfg = json.load(f)
s = cfg['storage']
s['migrations']['enabled'] = True
s['migrations']['directory'] = '/opt/myproject/migrations'
s['migrations']['auto_apply'] = True
with open('$DB_CONFIG', 'w') as f:
    json.dump(cfg, f, indent=2)
"
systemctl restart smartbotic-database

---

Migrations

Declarative JSON-based schema migrations. Create numbered JSON files in the migrations directory.

Migration File Format

{
  "version": "001",
  "name": "create_users",
  "description": "Create users collection with encryption",
  "operations": [
    {
      "type": "create_collection",
      "collection": "users",
      "options": {
        "encrypted": true,
        "max_versions": 10
      }
    }
  ]
}

Available Operations

Operation	Fields	Description
create_collection	collection, options	Create a new collection. Options: encrypted, max_versions, default_ttl_seconds, vector_dimension
put	collection, document	Insert/upsert a document
delete	collection, id	Delete a document
delete_where_id_prefix	collection, prefix	Bulk-delete documents whose ID starts with prefix

Vector-Enabled Collection Migration

{
  "version": "002",
  "name": "create_memories",
  "description": "Create vector-enabled memories collection",
  "operations": [
    {
      "type": "create_collection",
      "collection": "memories",
      "options": {
        "vector_dimension": 384
      }
    }
  ]
}

Note: vector_dimension is immutable after collection creation.

---

Docker Build Integration

When building your project in Docker for production, install the dev package in your base image instead of compiling the submodule from source.

Dockerfile.base Example

FROM debian:trixie

# Add Smartbotics APT repository
COPY smartbotics-repo.gpg /usr/share/keyrings/
RUN echo "deb [signed-by=/usr/share/keyrings/smartbotics-repo.gpg] \
    https://repository.smartbotics.ai trixie main" \
    > /etc/apt/sources.list.d/smartbotics.list

# Configure authentication
ARG REPO_USER=callerai
ARG REPO_PASS
RUN printf "machine repository.smartbotics.ai\nlogin %s\npassword %s\n" \
    "$REPO_USER" "$REPO_PASS" > /etc/apt/auth.conf.d/smartbotics.conf && \
    chmod 600 /etc/apt/auth.conf.d/smartbotics.conf

# Install client dev package (pulls shared lib as dependency)
RUN apt-get update && apt-get install -y \
    libsmartbotic-db-client-dev \
    && rm -rf /var/lib/apt/lists/*

Then your project's CMake uses find_package(smartbotic-db-client REQUIRED) and links against the pre-built shared library.

Production Dockerfile

Your project's production .deb should declare:

Depends: smartbotic-database (>= 1.2.0), libsmartbotic-db-client (>= 1.2.0), ...

This ensures both the server and runtime library are installed on the target machine.

---

CLI Administration

Install the CLI tool:

sudo apt install smartbotic-db-cli

The CLI connects to the database via gRPC. It does not require the server installed on the same machine — it can administer a remote instance.

# Connect to default (localhost:9004)
smartbotic-db-cli health

# Connect to a remote instance
smartbotic-db-cli --address 192.168.1.50:9004 health

---

Upgrading from Legacy Packages

If your machine previously had shadowman-database or callerai-storage installed, the new smartbotic-database package handles the transition automatically:

sudo apt install smartbotic-database

What happens:

1. dpkg sees Conflicts: shadowman-database / callerai-storage
2. The old package is removed (its prerm stops the old service)
3. smartbotic-database is installed
4. The postinst script detects old data/config and migrates:
   • /var/lib/shadowman/data/database/ → /var/lib/smartbotic-database/
   • /etc/shadowman/database.json → /etc/smartbotic-database/config.json (paths updated)
   • Same for callerai paths

The old data directory is preserved (copied, not moved) so rollback is safe.

Rollback

If something goes wrong:

sudo apt remove smartbotic-database
sudo apt install shadowman-database=1.2.0-14   # or callerai-storage=0.5.18-1

Old packages remain in the repository.

---

Proto File

The raw .proto file is available at:

• Package: /usr/share/smartbotic-database/proto/database.proto
• Source: proto/database.proto

This can be used to generate client stubs in other languages (Python, Go, etc.) if needed.