# Smartbotic Database Integration Guide

How to install, configure, and integrate smartbotic-database into your C++ project.

## Table of Contents

- [Installing the Database](#installing-the-database)
- [Installing the Client Library](#installing-the-client-library)
- [Integrating into a C++ Project](#integrating-into-a-c-project)
- [Client API Reference](#client-api-reference)
- [Configuration](#configuration)
- [Migrations](#migrations)
- [Docker Build Integration](#docker-build-integration)
- [CLI Administration](#cli-administration)
- [Upgrading from Legacy Packages](#upgrading-from-legacy-packages)
- [Drop-in configuration (conf.d)](#drop-in-configuration-confd)
- [Eviction & Memory Pressure](#eviction--memory-pressure)
- [Access Policy — row and column level security](#access-policy--row-and-column-level-security-270)

---

## Installing the Database

### From APT Repository

Add the Smartbotics repository to your system:

```bash
# Download GPG key
curl -fsSL https://repository.smartbotics.ai/smartbotics-repo.gpg | \
    sudo gpg --dearmor -o /usr/share/keyrings/smartbotics-repo.gpg

# Configure authentication
sudo tee /etc/apt/auth.conf.d/smartbotics.conf << EOF
machine repository.smartbotics.ai
login callerai
password <repo-password>
EOF
sudo chmod 600 /etc/apt/auth.conf.d/smartbotics.conf

# Add repository
echo "deb [signed-by=/usr/share/keyrings/smartbotics-repo.gpg] https://repository.smartbotics.ai trixie main" | \
    sudo tee /etc/apt/sources.list.d/smartbotics.list

# Install
sudo apt update
sudo apt install smartbotic-database
```

This installs:
- `/usr/bin/smartbotic-database` — the server binary
- `/etc/smartbotic-database/config.json` — configuration file
- `smartbotic-database.service` — systemd service (starts automatically)

The service listens on `localhost:9004` by default.

### Verify Installation

```bash
# Check service status
sudo systemctl status smartbotic-database

# Check health via CLI (if installed)
smartbotic-db-cli health
```

### Available Packages

| Package | What it provides | Install when... |
|---------|-----------------|-----------------|
| `smartbotic-database` | Server + systemd service | You need to run the database on this machine |
| `libsmartbotic-db-client` | Shared library (`.so`) | Your application links the client at runtime |
| `libsmartbotic-db-client-dev` | Headers + cmake config | You're building a C++ project against the client API |
| `smartbotic-db-cli` | CLI admin tool | You want to administer the database from the command line |

---

## Installing the Client Library

### On a Development Machine (for local builds)

```bash
sudo apt install libsmartbotic-db-client-dev
```

This pulls in `libsmartbotic-db-client` (the runtime `.so`) as a dependency, plus all development headers needed to compile.

### On a Production Machine (runtime only)

```bash
sudo apt install libsmartbotic-db-client
```

Only the shared library — no headers, no cmake config, no dev tooling.

---

## Integrating into a C++ Project

### Option A: System-installed package (recommended for Docker builds)

If `libsmartbotic-db-client-dev` is installed (e.g., in your Docker build image):

```cmake
# CMakeLists.txt
find_package(smartbotic-db-client REQUIRED)

add_executable(myapp main.cpp)
target_link_libraries(myapp PRIVATE smartbotic::db-client)
```

The `smartbotic::db-client` target automatically sets up include paths and links the shared library.

### Option B: Git submodule (for local development on non-Debian systems)

```bash
git submodule add ssh://git@git.smartbotics.ai:10022/fszontagh/smartbotic-database.git \
    external/smartbotic-database
```

```cmake
# CMakeLists.txt
add_subdirectory(external/smartbotic-database)

add_executable(myapp main.cpp)
target_link_libraries(myapp PRIVATE smartbotic-db-client)
```

When built as a submodule, the library is static (`.a`) and linked directly into your binary.

### Option C: Both (recommended)

Use the system package when available, fall back to submodule:

```cmake
# Try system-installed first (Docker builds, CI)
find_package(smartbotic-db-client QUIET)
if(smartbotic-db-client_FOUND)
    message(STATUS "Using system smartbotic-db-client")
    set(DB_CLIENT_TARGET smartbotic::db-client)
else()
    # Fall back to submodule (local development)
    if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/external/smartbotic-database/CMakeLists.txt")
        add_subdirectory(external/smartbotic-database)
        set(DB_CLIENT_TARGET smartbotic-db-client)
    else()
        message(FATAL_ERROR
            "smartbotic-database not found. Either:\n"
            "  apt install libsmartbotic-db-client-dev\n"
            "  git submodule update --init external/smartbotic-database")
    endif()
endif()

add_executable(myapp main.cpp)
target_link_libraries(myapp PRIVATE ${DB_CLIENT_TARGET})
```

---

## Client API Reference

### Connecting

```cpp
#include <smartbotic/database/client.hpp>

smartbotic::database::Client::Config config;
config.address = "localhost:9004";   // gRPC endpoint
config.timeoutMs = 5000;            // per-RPC timeout
config.maxRetries = 3;              // auto-retry on transient failures

smartbotic::database::Client db(config);
if (!db.connect()) {
    // handle connection failure
}
```

### Document Operations

```cpp
// Insert
nlohmann::json doc = {{"name", "Alice"}, {"email", "alice@example.com"}};
std::string id = db.insert("users", doc);
// Or with explicit ID:
std::string id = db.insert("users", doc, "user-123");

// Get
auto result = db.get("users", "user-123");
if (result) {
    std::cout << (*result)["name"] << std::endl;
}

// Update (full document replacement, with automatic optimistic locking)
db.update("users", "user-123", {{"name", "Alice Smith"}, {"email", "alice@example.com"}});

// Update with explicit optimistic locking (fails if version changed)
db.updateIfVersion("users", "user-123", {{"name", "Alice"}}, /*expectedVersion=*/2);

// Patch — atomic partial update (only specified fields are modified)
uint64_t newVer = db.patch("users", "user-123", {{"name", "Alice Smith"}});

// Upsert (insert or update)
auto [uid, isNew] = db.upsert("users", doc, "user-123");

// Delete
db.remove("users", "user-123");

// Check existence
bool found = db.exists("users", "user-123");
```

### Concurrency: update() vs patch()

When multiple clients modify the same document concurrently, choose the right method:

**`patch()`** — use for most updates (recommended). Merges only the specified fields on the server, atomically. Two clients can safely modify different fields simultaneously:

```cpp
// Client A:                                    // Client B:
db.patch("users", "user-123",                   db.patch("users", "user-123",
    {{"name", "Alice Smith"}});                      {{"email", "alice@new.com"}});
// Result: both fields updated, nothing lost
```

**`update()`** — full document replacement with automatic retry. Uses optimistic locking internally (reads current version, writes with version check, retries on conflict). Use when you need to replace the entire document:

```cpp
auto doc = db.get("users", "user-123");
(*doc)["name"] = "Alice Smith";
(*doc)["role"] = "admin";
db.update("users", "user-123", *doc);  // replaces entire document
```

Note: `update()` is "last writer wins" for complete replacement — if two clients modify different fields via `update()`, the last one overwrites the other's changes. Use `patch()` to avoid this.

**`updateIfVersion()`** — explicit optimistic locking. Use when you need to detect conflicts and handle them yourself:

```cpp
auto doc = db.get("users", "user-123");
uint64_t version = (*doc)["_version"];
(*doc)["balance"] = (*doc)["balance"].get<int>() + 100;
if (!db.updateIfVersion("users", "user-123", *doc, version)) {
    // Version conflict — someone else modified the document
    // Re-read and retry, or report error to user
}
```

### Views

Views are named, read-only projections over collections. They filter which fields are returned and optionally enforce baked-in filters and default sort order. Views are ideal for:

- **Schema evolution** — add fields to a collection without affecting consumers that query through a view
- **Security boundary** — hide sensitive fields (passwords, tokens) and enforce row-level constraints (status=active, tenant=X)
- **API clarity** — name and document a stable contract ("this view returns these fields with these filters")

#### Creating a view

```cpp
using Op = smartbotic::database::Client::FilterOp;

// Simple: just project fields
db.createView("users_public",
              /*collection=*/"users",
              /*include=*/{"id", "name", "avatar_url", "profile.public_bio"});

// Full-featured: fields + filters + default sort
db.createView("active_admins",
              /*collection=*/"users",
              /*include=*/{"id", "name", "role"},
              /*exclude=*/{},
              /*where=*/{
                  {"status", Op::EQ, "active"},
                  {"role",   Op::EQ, "admin"}
              },
              /*defaultSort=*/smartbotic::database::Client::Sort{"name", false});
```

#### Querying a view

Views work like collections — use the view name in `get()`, `find()`, `count()`:

```cpp
auto user = db.get("users_public", "u1");         // projection applied
auto results = db.find("active_admins", {
    .filters = {{"department", Op::EQ, "eng"}},   // AND-merged with view's where
    .limit = 50
});
auto total = db.count("active_admins", {{"department", Op::EQ, "eng"}});
```

#### Inspecting and dropping

```cpp
auto views = db.listViews();
auto info = db.getViewInfo("users_public");
db.dropView("users_public");
```

#### Semantics

| Dimension | Behavior |
|-----------|----------|
| **Filters** | **View AND caller** — view's `where` is always applied. Caller's filters are merged with AND. Consumers cannot escape the view's constraints. |
| **Sort** | **Caller overrides view** — if the caller specifies a sort, it wins. Otherwise the view's `defaultSort` is used. |
| **Include** | Whitelist — when non-empty, only those paths are returned. |
| **Exclude** | Blacklist — ignored if include is non-empty; otherwise removes listed paths. |
| **Metadata** | `_id`, `_version`, `_created_at`, `_updated_at`, `_created_by`, `_updated_by` are always returned regardless of include/exclude. |
| **Nested paths** | Dot-notation supported: `user.profile.name`. Stops at arrays. |
| **Writes** | Rejected — `insert()`, `update()`, `patch()`, `upsert()`, `remove()` on a view name return an error. |
| **View-of-view** | Not allowed — views must target real collections. |
| **Project scope** (2.4.2+) | Views belong to a project, exactly like collections. A view created by a client with `project = "acme"` is addressable only from that project, and two projects may each own a view of the same name. `listViews()` returns only the calling project's views. |
| **Unknown name** (2.4.2+) | A name matching neither a view nor a collection makes `find()` throw `std::runtime_error` (gRPC `NOT_FOUND`). Before 2.4.2 it returned an empty result, which was indistinguishable from an empty collection. |

#### Views and projects (2.4.2+)

View names are namespaced with the client's `Config::project`, so you keep
using the bare name and the client qualifies it:

```cpp
Client::Config cfg;
cfg.project = "acme";
Client db(cfg);

db.createView("adults", "users", {"name", "age"});  // -> acme:adults over acme:users
auto rows = db.find("adults", {});                  // resolves within acme only
```

A different workspace may define its own `adults` over its own `users`
without colliding, and neither can read through the other's view.

> **Upgrading from <= 2.4.1.** Views were unreachable through the client on
> 2.3.0-2.4.1: `createView` registered the bare name while every read path
> sent the project-qualified one, so lookups always missed and returned an
> empty result with no error. Existing view definitions are re-keyed into the
> `default` project automatically on first boot of 2.4.2 (idempotent, logged
> as `ViewManager: re-keyed N legacy view(s)`). No operator action required.
> Callers that treated the empty result as "view has no rows" will start
> seeing real rows.

#### Via migrations (recommended)

```json
{
  "version": "020",
  "name": "create_active_admins_view",
  "operations": [
    {
      "type": "create_view",
      "name": "active_admins",
      "collection": "users",
      "include": ["id", "name", "role"],
      "where": [
        {"field": "status", "op": "EQ", "value": "active"},
        {"field": "role",   "op": "EQ", "value": "admin"}
      ],
      "default_sort": {"field": "name", "descending": false}
    }
  ]
}
```

The `op` field in `where` filters accepts either a string name (`EQ`, `NE`, `GT`, `GTE`, `LT`, `LTE`, `IN`, `CONTAINS`, `EXISTS`, `REGEX`, `SEARCH`) or the matching integer. Migrations are idempotent — re-running a migration that creates an existing view is a no-op.

#### Why AND-merge filters?

A view's filters are applied **server-side**, before projection, and a consumer
filter is ANDed on top rather than replacing them — so a caller querying
`active_admins` cannot widen the result set beyond what the view defines. That
makes a view a reliable **schema and shape contract**: what comes back through
this name is always a subset of what the view allows.

> **A view is NOT a security boundary.** Earlier revisions of this guide
> described view filters as "a security enforcement boundary" and compared them
> to row-level security. That was wrong, and should not be used as the basis of
> a threat model. Three reasons:
>
> 1. **There is no caller identity.** Authentication (2.4+) compares a bearer
>    token against a flat list of shared keys per listener. It attaches no
>    principal to the call, so the server cannot know *who* is asking — and
>    every key holder has identical, complete access.
> 2. **The underlying collection stays readable.** Views only reject *writes* on
>    a view name. `find("active_admins")` narrows; `find("users")` does not.
>    Nothing stops a client bypassing the view entirely.
> 3. **Projects are not a boundary either.** `Config::project` is chosen by the
>    client, and a fully-qualified `other_project:collection` name is accepted
>    from any client, so namespaces isolate naming, not access.
>
> Views constrain what a **cooperating** client sees. Treat them as ergonomics
> and API hygiene. Real row- and column-level security requires a per-principal
> identity and a policy evaluated at every read handler; that is planned but not
> present today.
>
> Related: field-level encryption (`sensitiveFields`) is encryption **at rest**,
> not access control. Sensitive fields are decrypted for every authenticated
> reader; the protection is against someone reading data files, snapshots, or
> backups, not against a client.

#### ⚠ Field-level encryption is ON by default and keyed off FIELD NAMES

This surprises people, so it is spelled out here rather than left to be
discovered. `encryption.enabled` defaults to **`true`**, and the daemon encrypts
any field whose path matches one of these patterns, in **every** collection -
independent of the collection-level `encrypted` flag and of
`CollectionOptions.sensitiveFields`:

```
*.api_key   *.apiKey   *.password   *.secret   *.token   *.credentials
```

So a document with `config.password` is stored as `"$ENC$..."` whether or not you
asked for encryption. Consequences worth knowing:

- **Reads are symmetric.** `Get`, `Find`, `SimilaritySearch`, view reads,
  `GetVersionHistory` and `GetDocumentVersion` all return plaintext. (Before
  2.11.1 the two version handlers did **not** decrypt, so a live read returned
  plaintext while every version of the same document returned `$ENC$...` - which
  made "does this document differ from its published version?" answer *yes*
  forever. If you compare documents against versions, require >= 2.11.1.)
- **The same ciphertext in consecutive versions is correct**, not a fixed IV.
  When a sensitive field does not change between writes, the stored ciphertext is
  carried over verbatim instead of being re-encrypted, precisely so version
  history does not show a spurious difference on every save.
- **An encrypted field cannot be indexed or filtered on usefully**, since the
  stored bytes are ciphertext.
- To turn it off, set `storage.encryption.enabled: false`. Existing `$ENC$`
  values then read back as the literal ciphertext string - decryption is gated on
  the manager being enabled - so switch it off before storing such fields, not
  after.

### Access Policy — row and column level security (2.7.0+)

Per-project row- and column-level access control. **Off by default**: a project
with no policy behaves exactly as it did before 2.7.0, so upgrading changes
nothing until you turn it on.

> Before 2.7.0 there was no access control beyond authentication. Every API-key
> holder had identical, complete access. Views were never a boundary - see the
> note under "Why AND-merge filters".

#### The model

- **Principal** = the *name* of the API key the request arrived with. Keys became
  named in 2.7.0:
  ```json
  "auth": { "required": true, "keys": [
      { "name": "shadowman", "key": "<base64>" },
      { "name": "callerai",  "key": "<base64>" } ] }
  ```
  A bare string key still works and maps to the reserved principal `unnamed`. A
  request with no usable token is the reserved principal `anonymous` - which is
  grantable, so the local CLI keeps working by explicit policy rather than by an
  implicit hole. `anonymous` and `unnamed` are refused as key names.
- **Policy** = one record per (project, principal) granting `read`/`write` per
  collection or file type, with an optional column `mask` and row predicate.
- **admin** grants everything within a project plus the right to edit policy.

#### Turning it on safely

Enabling is **refused** unless some policy in the project has `admin: true` -
lockout is prevented structurally, not by care. Removing the last admin of a
secured project is refused for the same reason.

Always arm a live project in **audit** mode first. Audit evaluates every policy,
logs what it *would* deny, and allows the request:

```bash
smartbotic-db-cli policy-set acme ops '{"admin":true}'
smartbotic-db-cli policy-set acme svc \
  '{"collections":{"users":{"read":true,"mask":["ssn"],
     "row":[{"field":"tenant","op":0,"value":"acme"}]}}}'

smartbotic-db-cli security-set acme on audit     # watch the log
smartbotic-db-cli security-set acme on enforce   # then commit
smartbotic-db-cli security acme                  # show current state
smartbotic-db-cli policies acme                  # list principals
```

Once enforcing, the project is **deny-by-default**: a principal with no matching
rule gets nothing.

#### What enforcement covers

Every RPC that reads or writes user data. Notably including the ones that are
easy to forget:

- **`Subscribe`** is filtered per event, since an empty collection list means
  "everything" and there is no single name to authorise up front. Masked columns
  are stripped from event payloads.
- **Version history** (`GetVersionHistory`, `GetDocumentVersion`,
  `RestoreVersion`, `RestoreToDate`) - it returns previous document bodies, so
  leaving it open would bypass a column mask entirely.
- **Enumeration and counts** - `ListCollections`, `ListViews`, `ListFiles`,
  `Count`, `GetCollectionInfo`, `GetMemoryStats` filter or refuse. A name or a
  document count is information about data you may not read. `ListFiles` also
  recomputes `total_count`, which would otherwise report how many files exist in
  types you have no grant for.
- **Batch and set operations**, which are ordinary reads and writes.

Semantics worth knowing:

- A **column mask** removes fields from every document returned, and **filtering
  on a masked field is refused** - `salary > 100000` would otherwise leak a value
  you cannot read.
- A **write touching a masked field is denied**, not silently dropped. Silent
  field loss is worse than an error.
- A **row predicate** hides documents outright: `Get` reports not-found rather
  than "forbidden", because the difference is itself information.
- **System collections** (`_policies`, `_views`, …) require admin. `_policies`
  describes the whole access model.

#### Denials are visible to your code

`get`, `exists`, `find`, `findWithMetrics` and `count` **throw** on a denial
rather than returning empty. This is deliberate: a denial that looked like "no
data" would leave your application unable to tell "you may not see this" from
"there is nothing to see", and it would silently take the wrong branch.

#### Limits

- **Service-wide operations cannot be expressed per project.** `GetStats`,
  `SetReadOnly`, `CreateProject` and `DropProject` require admin of *some*
  secured project; `ListProjects` filters. The real boundary for operator
  surfaces is **listener separation** - do not expose an admin listener publicly.
- `HealthCheck` and `GetReadOnlyStatus` are intentionally open; they expose no
  user data.
- **Auth requires TLS.** gRPC aborts if an auth processor is attached to insecure
  credentials, so `auth.required: true` with `tls.enabled: false` is refused at
  startup.

### Collection Configuration

Per-collection runtime settings. Currently supports timestamp precision; extensible for future knobs.

#### Timestamp precision

Every document carries `_created_at` and `_updated_at` — int64 timestamps since epoch.

**Since 2.2.0 the default is nanoseconds (`ns`).** It was milliseconds up to
2.1.x. The flip was made because collections that receive rapid-fire writes
(LLM streaming chunks, agent tool loops) saw ties at ms resolution that made
`ORDER BY _created_at` ambiguous; ns gives unique, strictly ordered timestamps
out of the box. Collections that already had a stored config keep their value —
only collections created after the upgrade *without* an explicit
`configureCollection()` inherit the new default.

```cpp
using Cfg = smartbotic::database::Client::CollectionConfig;

// Default since 2.2.0: ns (no configuration needed)
auto id = db.insert("llm_tokens", {{"token", "hello"}});
// doc["_created_at"] is ~19 digits, nanoseconds since epoch

// Opt a collection back to milliseconds
db.configureCollection("logs", Cfg{"ms"});

// Future inserts into logs get ms-precision timestamps
auto id2 = db.insert("logs", {{"msg", "..."}});
// doc["_created_at"] is ~13 digits, milliseconds since epoch
```

> **Note:** `Client::CollectionConfig::timestampPrecision` still defaults to
> `"ms"` in the header, so `Cfg{}` sends an explicit `"ms"` rather than meaning
> "server default". Pass the precision you want explicitly, or leave the string
> empty to mean "leave unchanged" (2.4.5+, see partial updates below).

#### Reading the config

```cpp
auto cfg = db.getCollectionConfig("logs");
std::cout << cfg.timestampPrecision << std::endl;  // "ms"

bool explicit_set = db.hasCollectionConfig("logs");        // true
bool default_only = db.hasCollectionConfig("llm_tokens");  // false (still ns default)
```

#### Version history on/off (2.4.5+)

Version history can be switched per collection, **on a collection that already
exists and already holds data**:

```cpp
Cfg off;
off.timestampPrecision = "";      // empty = leave precision unchanged
off.versioningEnabled  = false;   // stop recording new versions
db.configureCollection("metrics_events", off);

// ... later, turn it back on
Cfg on;
on.timestampPrecision = "";
on.versioningEnabled  = true;
db.configureCollection("metrics_events", on);
```

Semantics:

- **Reversible.** Disabling stops *new* versions being recorded. History already
  on disk stays readable via `getVersionHistory()` / `getDocumentVersion()`, so
  re-enabling simply resumes appending.
- **Not the same as `maxVersions = 0`,** which means "keep unlimited history".
  `versioningEnabled = false` means "keep none".
- **Durable.** The setting lives in the `_collection_meta` system collection, so
  it survives restarts through the normal WAL + snapshot path.
- Takes effect on the next write.

#### Partial updates (2.4.5+)

`configureCollection()` is a **partial update**. Fields you leave unset are
preserved, not reset:

- empty `timestampPrecision` → precision unchanged
- unset `versioningEnabled` (`std::nullopt`) → versioning unchanged

```cpp
Cfg justPrecision;
justPrecision.timestampPrecision = "ms";   // versioningEnabled stays nullopt
db.configureCollection("logs", justPrecision);
// -> precision becomes ms; versioning is left exactly as it was
```

On `getCollectionConfig()`, `versioningEnabled` is always populated with the
collection's effective value rather than the "leave unchanged" sentinel.

> Before 2.4.5 this RPC replaced the whole config. That was harmless while
> precision was the only field, but it is why `versioning_enabled` is declared
> `optional` in the proto: a plain proto3 bool defaults to `false`, so any
> precision-only call would otherwise have silently switched versioning **off**.

#### What configureCollection does NOT do

**It does not convert existing timestamps.** Flipping a collection from `ms` to `ns` only affects *future* inserts. Pre-existing docs keep whatever precision they had. This keeps the config flip atomic and reversible.

Consequences of mixed precision within one collection:
- `int64` ordering within the collection is still consistent (ms values are always < 10^15, ns values are always > 10^15, so ns docs sort after ms docs). Queries don't break.
- But the ns-labeled docs appear to be in the future relative to the ms-labeled docs unless you convert.

#### Migrating existing timestamps

When you're ready to convert legacy data, run the migration helper during a planned maintenance window:

```cpp
auto result = db.migrateCollectionTimestamps("llm_tokens",
                                              /*fromPrecision=*/"ms",
                                              /*toPrecision=*/"ns");
if (result.success) {
    std::cout << "migrated " << result.rowsMigrated
              << " rows, skipped " << result.rowsSkipped << std::endl;
} else {
    std::cerr << "failed: " << result.error << std::endl;
}
```

Semantics:
- `ms → ns`: multiply `_created_at` / `_updated_at` by 10^6
- `ns → ms`: divide by 10^6 (truncation — loses sub-ms precision)
- **Idempotent and resumable.** Rows that are already in the target range are detected via a 10^15 threshold and skipped. If the migration is interrupted (network drop, deadline), rerun it — already-converted rows will count as `rows_skipped`.
- `from == to` is a no-op success.
- **Metadata-only** — document version numbers are NOT incremented, WAL is not appended, history is not updated. This is an infrastructure rewrite, not a user-visible change.
- Deadline: the client's RPC timeout for this call is 10 minutes — large collections may take a while.

#### Recommended workflow for switching precision on production data

1. Schedule a maintenance window (writes against the collection may pause briefly if needed).
2. Call `configureCollection(name, {ns})` — atomic config flip; future writes immediately use ns.
3. Call `migrateCollectionTimestamps(name, "ms", "ns")` — converts existing docs.
4. Verify with a sample read that `_created_at` values are now in the ns range.
5. Resume normal traffic.

Between steps 2 and 3, new writes are ns and old docs are ms — this is temporary but normal. Ordering still works because the 10^15 threshold cleanly separates the two ranges.

#### Via migrations

To set timestamp precision declaratively, call `db.configureCollection()` from your consumer project's startup code. The `configure_collection` migration operation will be available in a future version.

```json
{
  "version": "021",
  "name": "llm_tokens_ns_precision",
  "operations": [
    {
      "type": "configure_collection",
      "collection": "llm_tokens",
      "config": { "timestamp_precision": "ns" }
    }
  ]
}
```

### Recovery & Read-Only Mode

smartbotic-database uses WAL + snapshots for durability. When starting up, the server tries to recover state from the newest snapshot and replay WAL entries since that snapshot. If something goes wrong, it uses a tiered recovery system modeled on MySQL's `innodb_force_recovery`.

#### Recovery modes

| Level | Name | Behavior |
|:-:|------|----------|
| 0 | `normal` | Load latest snapshot + replay WAL. **Default.** If `auto_escalate=true` (default), falls back to older snapshot automatically when latest is corrupt. |
| 1 | `snapshot_fallback` | Try snapshots newest→oldest, first that loads wins. |
| 2 | `wal_only` | Ignore snapshots, replay WAL from sequence 0. Slow on large DBs. |
| 3 | `best_effort` | Try snapshot_fallback, then fall through to wal_only. |
| 4 | `force_empty` | Start empty. Snapshots + WAL preserved on disk for forensics. |

Operator escalation path when recovery fails:

```bash
# After a refuse-to-start, try older snapshots
smartbotic-database --recovery-mode=snapshot_fallback

# If that still fails, replay WAL from scratch (slow)
smartbotic-database --recovery-mode=wal_only

# Last-ditch: try everything
smartbotic-database --recovery-mode=best_effort

# Absolute last resort: start empty (data preserved on disk)
smartbotic-database --recovery-mode=force_empty
```

Or set in `config.json`:

```json
"persistence": {
    "recovery": {
        "mode": "normal",
        "auto_escalate": true,
        "allow_empty_on_fresh_install": true
    }
}
```

The CLI flag overrides the config for one-shot recoveries.

#### Auto-readonly on non-trivial recovery

**If recovery used anything other than "latest snapshot loaded cleanly," the DB boots in read-only mode.** This is a safety feature — writing onto a possibly-stale state can corrupt your data further. The operator must explicitly acknowledge the state before writes resume.

On startup you'll see (at ERROR level):

```
[ERROR] Database booted in READ-ONLY mode after non-trivial recovery
        Reason: fell back to snapshot snapshot-20260419-090719.dat because
                snapshot-20260419-100747.dat failed: body truncated

Writes will be REJECTED until you acknowledge this state:
  smartbotic-db-cli unlock                  # live, no restart
  smartbotic-database --force-readwrite     # on next restart
```

#### Workflow: recovering a broken instance

1. **Preserve forensics.** Stop the service, snapshot `/var/lib/smartbotic-database/` before anything else.
2. **Diagnose.** Start the server (default mode = `normal`). It will either recover cleanly or refuse with a clear banner. Use `smartbotic-db-cli status` to see what happened.
3. **Escalate.** If recovery refused, restart with `--recovery-mode=snapshot_fallback` (and upwards as needed).
4. **Extract data.** While the DB is in read-only mode, connect with the CLI or client library and read whatever survived. Dump/export facilities are planned for v1.7.0.
5. **Decide.**
   - Data looks complete → `smartbotic-db-cli unlock` to resume writes.
   - Data is missing/stale → don't unlock. Extract to an external backup, then wipe `/var/lib/smartbotic-database/` and start clean.

#### Client API

```cpp
// Acknowledge auto-readonly state and resume writes
db.unlock();

// Force read-only for maintenance (regardless of recovery outcome)
db.lock();

// Inspect state + recovery outcome
auto s = db.getReadOnlyStatus();
std::cout << "read-only:        " << s.readOnly << "\n"
          << "reason:           " << s.reason << "\n"
          << "recovery outcome: " << s.recoveryOutcome << "\n"
          << "expected snap:    " << s.expectedSnapshot << "\n"
          << "snapshot used:    " << s.snapshotUsed << "\n"
          << "failure reason:   " << s.failureReason << "\n"
          << "WAL replayed:     " << s.walEntriesReplayed << "\n"
          << "snapshots tried:  " << s.snapshotsAttempted << "\n";
```

#### CLI

```bash
# Inspect read-only + recovery state
smartbotic-db-cli status

# Acknowledge non-trivial recovery and accept writes
smartbotic-db-cli unlock

# Manually lock (e.g. before a maintenance window)
smartbotic-db-cli lock
```

#### Snapshot durability settings

Under `persistence.snapshots`:

```json
"snapshots": {
    "validate_after_write": true,        // verify every snapshot immediately after creation
    "cleanup_only_if_verified": true     // never evict old snapshots if the new one failed validation
}
```

Both default to `true` — turn off only if you really know what you're doing (disables protections against silent writer corruption).

#### Exit codes

| Code | Meaning |
|:-:|---------|
| `0` | Normal shutdown |
| `1` | Generic error |
| `10` | **Recovery refused.** Snapshot or WAL failure + mode=normal + data exists on disk. Operator must escalate the recovery mode or pass `--force-readwrite`. |
| `11` | Config invalid (e.g. bad `--recovery-mode` value) |

#### `--force-readwrite`

Bypasses the auto-readonly guard on non-trivial recovery. Use after you've reviewed the recovery state via `smartbotic-db-cli status` and are certain writes onto that state are safe.

```bash
# After confirming recovery state looks good
smartbotic-database --force-readwrite
```

### Querying

```cpp
// Find with filters (equality is default)
smartbotic::database::Client::QueryOptions opts;
opts.filters = {
    {"status", "active"},   // EQ (default) — 2-arg Filter constructor
    {"role", "admin"},      // EQ
};
opts.sortField = "name";
opts.sortDescending = false;
opts.limit = 50;
opts.offset = 0;

auto docs = db.find("users", opts);
for (const auto& doc : docs) {
    std::cout << doc["name"] << std::endl;
}

// Find all documents in a collection
auto allDocs = db.find("users");

// Count
uint64_t total = db.count("users");
uint64_t admins = db.count("users", {{"role", "admin"}});
```

#### Filter operators

Filters support a range of operators via `Client::FilterOp`. The 2-argument
`Filter(field, value)` constructor defaults to `EQ` — existing brace-init code
continues to compile and behave the same. To use other operators, pass the
operator explicitly with the 3-argument constructor:

```cpp
using Op = smartbotic::database::Client::FilterOp;

smartbotic::database::Client::QueryOptions opts;
opts.filters = {
    {"age", Op::GTE, 18},                                             // age >= 18
    {"status", Op::NE, "deleted"},                                    // status != "deleted"
    {"tags", Op::CONTAINS, "admin"},                                  // tags array contains "admin"
    {"role", Op::IN, nlohmann::json::array({"admin", "moderator"})},  // role IN [...]
    {"email", Op::REGEX, ".*@example\\.com$"},                        // regex match
    {"name", Op::SEARCH, "alice"},                                    // full-text search
    {"deleted_at", Op::EXISTS, false},                                // field does NOT exist
};
auto docs = db.find("users", opts);
```

| Operator              | Semantics                                                      | Value type                  |
|-----------------------|----------------------------------------------------------------|-----------------------------|
| `FilterOp::EQ`        | field equals value                                             | any JSON                    |
| `FilterOp::NE`        | field does not equal value                                     | any JSON                    |
| `FilterOp::GT`        | field > value                                                  | number or string            |
| `FilterOp::GTE`       | field >= value                                                 | number or string            |
| `FilterOp::LT`        | field < value                                                  | number or string            |
| `FilterOp::LTE`       | field <= value                                                 | number or string            |
| `FilterOp::IN`        | field value appears in the supplied array                      | JSON array                  |
| `FilterOp::CONTAINS`  | field (array) contains the supplied value                      | any JSON                    |
| `FilterOp::EXISTS`    | field presence check (`true` = must exist, `false` = must not) | bool                        |
| `FilterOp::REGEX`     | field matches the supplied regex pattern                       | string                      |
| `FilterOp::SEARCH`    | full-text search across document ID and string fields          | string                      |

Note: the old `_search` magic field name that auto-selected SEARCH is no
longer special — use `FilterOp::SEARCH` explicitly instead.

### Collection Management

```cpp
// Create a basic collection
db.createCollection("users");

// Create with options
db.createCollection("sessions",
    /*defaultTtlSeconds=*/3600,    // auto-expire after 1 hour
    /*encrypted=*/false,
    /*maxVersions=*/5,             // keep last 5 versions
    /*vectorDimension=*/0);        // no vector support

// Create vector-enabled collection
db.createCollection("memories",
    /*defaultTtlSeconds=*/0,
    /*encrypted=*/false,
    /*maxVersions=*/0,
    /*vectorDimension=*/384);      // 384-dim embeddings

// List and inspect
auto collections = db.listCollections();
auto info = db.getCollectionInfo("users");
if (info) {
    std::cout << "Documents: " << info->documentCount << std::endl;
}

// Drop
db.dropCollection("old_data");
```

#### Collections and projects (2.4.5+)

Collection names are namespaced with the client's `Config::project`, exactly
like documents. You keep passing bare names; the client qualifies them on the
wire and un-qualifies them coming back, so `listCollections()` round-trips:

```cpp
Client::Config cfg;
cfg.project = "acme";
Client db(cfg);

db.createCollection("users");              // -> acme:users
db.insert("users", {{"name", "Ada"}});     // -> acme:users
auto info = db.getCollectionInfo("users"); // -> acme:users
db.listCollections();                      // -> { "users", ... }
```

> **Fixed in 2.4.5 — upgrade if you call `createCollection`.** On 2.3.0-2.4.4
> `createCollection`, `dropCollection` and `getCollectionInfo` sent the **bare**
> name while `insert`/`get`/`find` sent the project-qualified one. Because the
> client qualifies against `Config::project` unconditionally — including
> `default` — **every** project was affected. One `createCollection("things")`
> plus one `insert("things")` created *two* collections: the bare one received
> the options and stayed empty, while `<project>:things` was created implicitly
> by the insert with **defaults**. No error was raised.
>
> Concretely, on affected versions:
> - `vectorDimension` never reached the collection holding the data, so
>   `similaritySearch()` returned no hits.
> - `encrypted = true` silently did not apply.
> - `dropCollection()` returned `true` while every document remained readable —
>   it dropped the empty phantom.
>
> **After upgrading:** leftover empty phantom collections are harmless litter,
> but options that were swallowed do **not** retroactively apply.
> `createCollection` on a now-correctly-resolved existing collection returns
> `false` and only updates `maxVersions`. Since `vectorDimension` is **immutable
> after creation**, a collection that needs vector search must be recreated:
> create a new collection with the right `vectorDimension`, copy the documents
> across, then drop the old one.

Note that `listCollections()` currently returns every project's collections —
names outside your own project stay visibly qualified rather than being
flattened into yours. Unlike `ListViewsRequest`, `ListCollectionsRequest` has no
project filter yet.

### Vector Similarity Search

Requires a collection created with `vectorDimension > 0`.

```cpp
// Insert documents with embeddings
nlohmann::json doc = {
    {"id", "mem-1"},
    {"content", "The user prefers dark mode"},
    {"_vector", embedding}   // std::vector<float>, size must match vectorDimension
};
db.insert("memories", doc, "mem-1");

// Search
std::vector<float> queryVec = getEmbedding("user preferences");
auto results = db.similaritySearch("memories", queryVec, /*topK=*/5, /*minScore=*/0.7f);

for (const auto& r : results) {
    std::cout << r.id << " (score=" << r.score << "): "
              << r.data["content"] << std::endl;
}
```

Notes:
- `_vector` is extracted and stored separately — not returned in `get()` or `find()` responses
- Dimension is validated on every insert — mismatches are rejected
- SIMD-accelerated (AVX2/SSE4.1) brute-force cosine similarity

### Version History

```cpp
// Get history
auto history = db.getVersionHistory("users", "user-123", /*limit=*/10);
for (const auto& v : history.versions) {
    std::cout << "v" << v.version << " by " << v.updatedBy
              << " at " << v.timestamp << std::endl;
}

// Get a specific version
auto oldVersion = db.getDocumentVersion("users", "user-123", /*version=*/2);

// Restore to a previous version
uint64_t newVer = db.restoreVersion("users", "user-123", /*version=*/2);

// Restore to a point in time
auto [fromVer, newVersion] = db.restoreToDate("users", "user-123", timestamp);
```

### File Storage

Two-level storage with content-addressed blob deduplication. Identical files are stored once.

#### Files and projects (2.6.0+)

Files are namespaced by project, like collections. The client fills the project
from `Config::project` on every file call, so you keep passing bare ids:

```cpp
Client::Config cfg;
cfg.project = "acme";
Client db(cfg);

auto up = db.uploadFile(data, meta);   // owned by project "acme"
db.getFileInfo(up.id);                 // resolves
db.listFiles("document");              // only acme's files
```

An id belonging to another project is reported as **`NOT_FOUND`**, not as
"forbidden" — `getFileInfo` returns `nullopt`, `downloadFile` throws,
`deleteFile` returns `false`, and `listFiles` omits it. The indistinguishability
is deliberate: a distinct "exists but not yours" answer would confirm that a peer
project holds that file.

> **Before 2.6.0, files had no project at all** — one shared blob tree, one flat
> record tree, and no `project` field on any file RPC. Any client could read,
> list, or delete any file by id. On upgrade, existing records are stamped with
> `project: "default"` automatically (idempotent, logged at INFO with a count).
> If your consumer used a non-default project *and* uploaded files, those files
> are now owned by `default` and will not be visible to that project — move them
> by re-uploading under the intended project.

`FileRecord` gained a `project` member, which changes the struct size, so
**consumers must rebuild** against 2.6.0 headers.

#### Deduplication is per project

`result.deduplicated` means **this project** already referenced those bytes. Blob
storage still dedups globally, so the disk saving is unchanged, but the flag is
scoped deliberately: reporting the global hit would let you upload a candidate
file and learn from the flag whether some *other* project holds it.

```cpp
// Upload
std::vector<uint8_t> data = readFile("photo.jpg");
smartbotic::database::Client::FileUploadMeta meta;
meta.name = "photo.jpg";
meta.mime_type = "image/jpeg";
meta.file_type = "document";        // "plugin", "document", "generated"
meta.related_id = "conversation-1";
meta.is_public = false;

auto result = db.uploadFile(data, meta);
std::cout << "File ID: " << result.id
          << " Deduplicated: " << result.deduplicated << std::endl;

// Download
auto fileData = db.downloadFile(result.id);

// Get file info
auto fileInfo = db.getFileInfo(result.id);

// List files
auto files = db.listFiles(
    /*file_type=*/"document",
    /*related_id=*/"conversation-1",
    /*limit=*/100, /*offset=*/0,
    /*checksum=*/"",
    /*name=*/"photo.jpg");

// Delete
db.deleteFile(result.id);
```

### Event Subscription

```cpp
auto subscription = db.subscribe({"users", "sessions"}, [](
    const std::string& collection,
    const std::string& id,
    const std::string& eventType,
    const std::optional<nlohmann::json>& data) {

    std::cout << eventType << " on " << collection << "/" << id << std::endl;
});

// Unsubscribe by releasing the handle
subscription.reset();
```

### Set Operations

Redis-compatible set operations on top of documents.

```cpp
db.setAdd("tags", "user-123", "admin");
db.setAdd("tags", "user-123", "active");

auto members = db.setMembers("tags", "user-123");
bool isAdmin = db.setIsMember("tags", "user-123", "admin");

db.setRemove("tags", "user-123", "active");
```

### Health & Statistics

```cpp
// Quick health check
if (db.healthCheck()) {
    std::cout << "Database is healthy" << std::endl;
}

// Detailed health
auto health = db.getHealthInfo();
if (health) {
    std::cout << "Uptime: " << health->uptimeMs << "ms"
              << " Docs: " << health->documentCount << std::endl;
}

// Full statistics
auto stats = db.getStats();
if (stats) {
    std::cout << "Memory: " << stats->memoryUsedBytes << " bytes"
              << " Insert avg: " << stats->insertAvgMicros() << " us" << std::endl;
}
```

---

## Configuration

The server configuration file is at `/etc/smartbotic-database/config.json`.

### Default Configuration

```json
{
  "log_level": "info",
  "storage": {
    "bind_address": "localhost",
    "rpc_port": 9004,
    "node_id": "smartbotic-db",
    "data_directory": "/var/lib/smartbotic-database",
    "memory": {
      "max_memory_mb": 512,
      "eviction_threshold_percent": 80,
      "eviction_target_percent": 60,
      "eviction_check_interval_ms": 5000
    },
    "persistence": {
      "wal_sync_interval_ms": 100,
      "snapshot_interval_sec": 3600,
      "compression": "lz4"
    },
    "encryption": {
      "enabled": true,
      "key_file": "/var/lib/smartbotic-database/storage.key",
      "auto_generate_key": true
    },
    "migrations": {
      "enabled": false,
      "directory": "/etc/smartbotic-database/migrations",
      "auto_apply": false
    },
    "files": {
      "max_file_size_mb": 500,
      "cleanup_orphans_interval_sec": 3600
    },
    "replication": {
      "enabled": false
    }
  }
}
```

### Key Settings

| Setting | Default | Description |
|---------|---------|-------------|
| `bind_address` | `localhost` | Change to `0.0.0.0` to accept remote connections |
| `rpc_port` | `9004` | gRPC listen port |
| `data_directory` | `/var/lib/smartbotic-database` | WAL, snapshots, files stored here |
| `memory.max_memory_mb` | `512` | Maximum memory for document cache. Evicts LRU when exceeded |
| `memory.eviction_max_episode_percent` | `50` | (2.4.3+) Ceiling on the share of the resident set one pressure episode may evict. `0` disables. See the drain-cap note below |
| `persistence.snapshot_interval_sec` | `3600` | Full snapshot every N seconds (WAL truncated after) |
| `encryption.enabled` | `true` | Field-level AES-256-GCM encryption. Key auto-generated on first start |
| `migrations.enabled` | `false` | Enable to auto-apply JSON migrations on startup |
| `migrations.directory` | `/etc/smartbotic-database/migrations` | Path to migration JSON files |

### gRPC Configuration

Under `storage.grpc` (added in v1.6.2):

| Setting | Default | Purpose |
|---------|--------:|---------|
| `max_receive_message_size_mb` | `100` | Max inbound gRPC message size. Raise for larger file uploads. |
| `max_send_message_size_mb` | `100` | Max outbound gRPC message size. |
| `resource_quota_memory_mb` | `256` | Global memory budget for gRPC inbound buffers across all RPCs combined. |
| `max_concurrent_subscribe_streams` | `50` | Max simultaneous `Subscribe` event-stream clients. Excess gets `RESOURCE_EXHAUSTED`. |
| `max_concurrent_file_streams` | `10` | Max simultaneous `UploadFile` + `DownloadFile` streams (shared pool). |

When a concurrency limit is hit, the server returns `RESOURCE_EXHAUSTED` immediately and logs `WARN` with the current / max counts. Clients should retry with backoff.

### Drop-in configuration (conf.d)

In addition to `/etc/smartbotic-database/config.json`, the server loads any `*.json` files from `/etc/smartbotic-database/conf.d/` and deep-merges them into the running config in lexicographic order.

This lets consumer projects (shadowman-cpp, callerai, your own apps) ship their own tuning without touching the upstream `config.json` that `dpkg --configure` guards as a conffile.

#### Merge semantics (RFC 7396)

- **Objects** merge recursively: sibling fields not mentioned in a drop-in are preserved from the lower-priority layer.
- **Arrays and scalars** are replaced entirely by a higher-priority layer.

Example:

```json
// /etc/smartbotic-database/config.json
{
  "log_level": "info",
  "storage": {
    "rpc_port": 9004,
    "memory": { "max_memory_mb": 512 }
  }
}

// /etc/smartbotic-database/conf.d/50-shadowman.json  (shipped by shadowman-cpp's deb)
{
  "storage": {
    "memory": { "max_memory_mb": 2048 }
  }
}

// Effective merged config — sibling rpc_port preserved, max_memory_mb overridden
{
  "log_level": "info",
  "storage": {
    "rpc_port": 9004,
    "memory": { "max_memory_mb": 2048 }
  }
}
```

#### File naming conventions

Use numeric prefixes for deterministic merge order. Last writer wins.

| File | Owner | Purpose |
|------|-------|---------|
| `00-defaults.json` | smartbotic-database (optional) | Baseline conservative tunings |
| `50-<project>.json` | consumer project's deb postinst | Workload-tuned values |
| `99-local.json` | operator | Site-specific overrides |

Files are merged in filename-sort order, so `00-` applies before `50-` applies before `99-`.

#### Failure modes

- **Missing base `config.json`** — OK, treated as `{}`
- **Missing `conf.d/` directory** — OK
- **Syntax error in any file** — server refuses to start, exit code `11`, log identifies the file + parser location
- **Unknown keys** — lenient. Individual config consumers may log `WARN` but don't block startup. This lets v1.x consumers drop a `conf.d/` file that mentions a key only v1.(x+1) knows about

#### What gets logged

At startup, the server prints each file as it merges it:

```
[INFO] Loading base config: /etc/smartbotic-database/config.json
[INFO] Merging drop-in: /etc/smartbotic-database/conf.d/50-shadowman.json
[INFO] Merging drop-in: /etc/smartbotic-database/conf.d/99-local.json
```

If a consumer wonders which tuning values are actually in effect, this is where to look.

### Eviction & Memory Pressure

smartbotic-database holds documents in memory for fast reads. When usage approaches the configured budget (`storage.memory.max_memory_mb`), eviction removes the least-useful docs to free space. Eviction is chunked, throttled, and hot-write-aware so it doesn't block foreground writes — a design requirement after the 2026-04-22 Zoe incident where one-shot eviction stalled LLM streaming writes long enough to miss their gRPC deadline.

#### Pressure levels

The server tracks four pressure levels based on current usage vs. `max_memory_mb`:

| Level | Default % | Behavior |
|-------|:-:|----------|
| `normal` | < 70% | No eviction |
| `soft` | 70–85% | Trickle eviction: one chunk per tick |
| `hard` | 85–95% | Aggressive eviction: up to `max_eviction_passes_per_trigger / 2` chunks per tick. Fires `MEMORY_PRESSURE_HIGH` event (edge-triggered) |
| `emergency` | ≥ 95% | All writes rejected with `RESOURCE_EXHAUSTED` until pressure drops below hard. Eviction runs at full rate |

Query current state via `smartbotic-db-cli status` or the `GetMemoryStats` RPC.

#### Drain cap (2.4.3+)

Eviction assumes the memory estimate falls as documents leave. When memory is
held by something eviction cannot free, that assumption breaks: the target is
never reachable, and eviction keeps trimming until the cache is empty. On
2026-08-03 a production instance did exactly this — 5996 of ~5990 documents
evicted over 30 minutes while the estimate *rose* (347 → 358 MB). Every
collection then reported 0 docs, which looks indistinguishable from data loss.
No data was actually lost; it was all still in LMDB.

Two guards now bound this:

- **Episode cap** — one pressure episode may evict at most
  `memory.eviction_max_episode_percent` (default 50%) of the resident set it
  started with, enforced per chunk. An episode re-arms only after pressure has
  been `normal` for 3 consecutive ticks.
- **No-progress detector** — after 3 consecutive ticks that evict documents
  without lowering the estimate, eviction pauses until pressure clears.

Both log at **ERROR**. Hitting either means the memory estimate is wrong, not
that the store is oversized — treat it as an alert, raise
`storage.memory.max_memory_mb`, and check what is holding memory. Set
`eviction_max_episode_percent: 0` to disable the cap (not recommended).

> **Note for v2.0+.** MemoryStore is a cache in front of LMDB; the durable copy
> lives in `<dataDir>/projects/<name>/env/`. Eviction draining the cache only
> becomes visible if the LMDB mirror is also unhealthy, which is what made the
> above incident look total. Watch for `v2.0 mirror failed` in the log.

#### Chunked eviction

Eviction never stops the world. Each tick (default every `eviction_check_interval_ms` = 5000ms, but reactive to pressure):

1. Collects candidates — docs whose `updatedAt` is older than `hot_write_floor_ms` (30 s default) ago, in non-pinned collections
2. Sorts by `(memory_priority, lastAccessedAt)` — low-priority + cold goes first
3. Applies per-collection budget: no one collection contributes more than `(share × 1.5)` of a single chunk
4. Evicts `eviction_chunk_size` docs (default 1000) in one pass
5. Sleeps `eviction_chunk_pause_ms` (default 50 ms)
6. Repeats up to `max_eviction_passes_per_trigger` times, stopping early if target reached

Docs being actively written (quiesce) are skipped via an in-flight-write counter — eviction can't land mid-upsert.

#### Per-collection priority

Use `CollectionOptions.memory_priority` to bias eviction:

| Value | Meaning |
|-------|---------|
| `low` | Evicted preferentially — cold archive collections |
| `normal` (default) | Standard LRU weight |
| `high` | Evicted last — hot operational collections |
| (pinned=true on CollectionOptions) | Never evicted |

Set via `createCollection` or migration:

```json
{
  "type": "create_collection",
  "collection": "messages",
  "options": {
    "memory_priority": "high"
  }
}
```

#### Config reference (`storage.memory`)

| Setting | Default | Meaning |
|---------|--------:|---------|
| `max_memory_mb` | 512 | Memory budget before eviction kicks in |
| `eviction_chunk_size` | 1000 | Docs evicted per chunk |
| `eviction_chunk_pause_ms` | 50 | Sleep between chunks within one tick |
| `max_eviction_passes_per_trigger` | 20 | Safety cap on chunks per tick |
| `hot_write_floor_ms` | 30000 | Docs updated within this window are unevictable |
| `memory_soft_percent` | 70 | Enter `soft` pressure |
| `memory_hard_percent` | 85 | Enter `hard` pressure — fires PRESSURE_HIGH event |
| `memory_emergency_percent` | 95 | Enter `emergency` — admission control rejects writes |
| `eviction_burst_threshold` | 10000 | Fire `MEMORY_EVICTION_BURST` event if chunk ≥ this |
| `eviction_target_percent` | 60 | Stop evicting once usage drops below this |
| `eviction_check_interval_ms` | 5000 | Wake eviction loop this often |

#### Events

Subscribe to receive memory-related notifications:

| Event type | When | Payload |
|------------|------|---------|
| `MEMORY_PRESSURE_HIGH` | Edge-triggered: pressure enters `hard` or `emergency` | `{pressure_level, pressure_percent, estimated_bytes, max_bytes}` |
| `MEMORY_EVICTION_BURST` | An eviction pass touched ≥ `eviction_burst_threshold` docs | `{evicted_docs, bytes_freed, pressure_level}` |

Both are system-level events (empty collection / id). Subscribers filter by event type.

#### Client-side retry

The client library (v1.7.0+) automatically retries write RPCs that return transient errors:

| Status | Retry? |
|--------|:-:|
| `DEADLINE_EXCEEDED` | ✅ |
| `RESOURCE_EXHAUSTED` (admission control / concurrency cap) | ✅ |
| `UNAVAILABLE` | ✅ |
| All others (INVALID_ARGUMENT, FAILED_PRECONDITION, etc.) | ❌ — final |

Config under `Client::Config`:

| Field | Default | Meaning |
|-------|--------:|---------|
| `writeRetries` | 3 | Max retry attempts |
| `writeRetryBackoffMs` | 100 | Initial backoff (exponential: 100 → 200 → 400 …) |
| `writeRetryMaxBackoffMs` | 5000 | Cap on any single backoff |
| `writeRetryJitter` | 0.25 | ± 25% jitter |

Writes wrapped: `insert`, `updateIfVersion`, `upsert`, `remove`, `patch`. Reads are NOT auto-retried — callers retry at their discretion.

#### Observability

`GetMemoryStats` RPC (and `Client::getMemoryStats()`) returns:

- Total / max memory bytes, pressure percent + level
- Per-collection: `document_count`, `estimated_bytes`, `evicted_stub_count`, `priority`
- Last eviction: timestamp, docs evicted, bytes freed

```cpp
auto s = db.getMemoryStats();
std::cout << "Memory: " << s.totalMemoryBytes << "/" << s.maxMemoryBytes
          << " (" << s.pressurePercent << "% — " << s.pressureLevel << ")" << std::endl;
for (const auto& c : s.collections) {
    std::cout << "  " << c.collection << ": " << c.documentCount << " docs, "
              << c.estimatedBytes << " bytes, priority=" << c.priority << std::endl;
}
```

Also shown in `smartbotic-db-cli status` output (if the CLI exposes it; otherwise subscribers can consume the events).

#### Recommended tunings

- **Default (512 MB)** — development or small services with ~100 K docs
- **2 GB** — production shadowman-cpp profile:

```json
// /etc/smartbotic-database/conf.d/50-shadowman.json
{
  "storage": {
    "memory": {
      "max_memory_mb": 2048,
      "memory_soft_percent": 65,
      "memory_hard_percent": 80,
      "eviction_chunk_size": 500,
      "hot_write_floor_ms": 60000
    }
  }
}
```

- **8+ GB large** — consider raising `eviction_burst_threshold` to 50000 so you don't get event spam, and consider `hot_write_floor_ms: 120000` if your write paths are slow (e.g. STT-heavy callerai)

#### Troubleshooting

**"Client::insert failed: RESOURCE_EXHAUSTED (memory pressure emergency)"** — server is in emergency pressure, client is retrying. If this persists, raise `max_memory_mb` in a conf.d drop-in or lower `memory_emergency_percent`.

**"Eviction: nothing evictable this pass"** — every candidate is either pinned, hot-written, or in-flight. Usually benign; server will re-try next tick. If persistent, consider reducing `hot_write_floor_ms` or flagging some collections as unpinned/low-priority.

**Pressure stays at `hard` despite quiet writes** — check `eviction_target_percent`; eviction stops when usage drops below this. If you want eviction to drain further during idle, raise `max_memory_mb` or lower the target.

### Configuring for Your Project

Consumer projects typically enable migrations by configuring the database via their own `postinst` script:

```bash
# Example: in your project's postinst
DB_CONFIG="/etc/smartbotic-database/config.json"
python3 -c "
import json
with open('$DB_CONFIG') as f:
    cfg = json.load(f)
s = cfg['storage']
s['migrations']['enabled'] = True
s['migrations']['directory'] = '/opt/myproject/migrations'
s['migrations']['auto_apply'] = True
with open('$DB_CONFIG', 'w') as f:
    json.dump(cfg, f, indent=2)
"
systemctl restart smartbotic-database
```

---

## Migrations

Declarative JSON-based schema migrations. Create numbered JSON files in the migrations directory.

### Migration File Format

```json
{
  "version": "001",
  "name": "create_users",
  "description": "Create users collection with encryption",
  "operations": [
    {
      "type": "create_collection",
      "collection": "users",
      "options": {
        "encrypted": true,
        "max_versions": 10
      }
    }
  ]
}
```

### Available Operations

| Operation | Fields | Description |
|-----------|--------|-------------|
| `create_collection` | `collection`, `options` | Create a new collection. Options: `encrypted`, `max_versions`, `default_ttl_seconds`, `vector_dimension` |
| `put` | `collection`, `document` | Insert/upsert a document |
| `delete` | `collection`, `id` | Delete a document |
| `delete_where_id_prefix` | `collection`, `prefix` | Bulk-delete documents whose ID starts with prefix |

### Vector-Enabled Collection Migration

```json
{
  "version": "002",
  "name": "create_memories",
  "description": "Create vector-enabled memories collection",
  "operations": [
    {
      "type": "create_collection",
      "collection": "memories",
      "options": {
        "vector_dimension": 384
      }
    }
  ]
}
```

Note: `vector_dimension` is immutable after collection creation.

---

## Docker Build Integration

When building your project in Docker for production, install the dev package in your base image instead of compiling the submodule from source.

### Dockerfile.base Example

```dockerfile
FROM debian:trixie

# Add Smartbotics APT repository
COPY smartbotics-repo.gpg /usr/share/keyrings/
RUN echo "deb [signed-by=/usr/share/keyrings/smartbotics-repo.gpg] \
    https://repository.smartbotics.ai trixie main" \
    > /etc/apt/sources.list.d/smartbotics.list

# Configure authentication
ARG REPO_USER=callerai
ARG REPO_PASS
RUN printf "machine repository.smartbotics.ai\nlogin %s\npassword %s\n" \
    "$REPO_USER" "$REPO_PASS" > /etc/apt/auth.conf.d/smartbotics.conf && \
    chmod 600 /etc/apt/auth.conf.d/smartbotics.conf

# Install client dev package (pulls shared lib as dependency)
RUN apt-get update && apt-get install -y \
    libsmartbotic-db-client-dev \
    && rm -rf /var/lib/apt/lists/*
```

Then your project's CMake uses `find_package(smartbotic-db-client REQUIRED)` and links against the pre-built shared library.

### Production Dockerfile

Your project's production `.deb` should declare:

```
Depends: smartbotic-database (>= 1.2.0), libsmartbotic-db-client (>= 1.2.0), ...
```

This ensures both the server and runtime library are installed on the target machine.

---

## CLI Administration

Install the CLI tool:

```bash
sudo apt install smartbotic-db-cli
```

The CLI connects to the database via gRPC. It does not require the server installed on the same machine — it can administer a remote instance.

```bash
# Connect to default (localhost:9004)
smartbotic-db-cli health

# Connect to a remote instance
smartbotic-db-cli --address 192.168.1.50:9004 health
```

---

## Upgrading from Legacy Packages

If your machine previously had `shadowman-database` or `callerai-storage` installed, the new `smartbotic-database` package handles the transition automatically:

```bash
sudo apt install smartbotic-database
```

What happens:
1. `dpkg` sees `Conflicts: shadowman-database` / `callerai-storage`
2. The old package is removed (its `prerm` stops the old service)
3. `smartbotic-database` is installed
4. The `postinst` script detects old data/config and migrates:
   - `/var/lib/shadowman/data/database/` → `/var/lib/smartbotic-database/`
   - `/etc/shadowman/database.json` → `/etc/smartbotic-database/config.json` (paths updated)
   - Same for callerai paths

The old data directory is preserved (copied, not moved) so rollback is safe.

### Rollback

If something goes wrong:

```bash
sudo apt remove smartbotic-database
sudo apt install shadowman-database=1.2.0-14   # or callerai-storage=0.5.18-1
```

Old packages remain in the repository.

---

## Proto File

The raw `.proto` file is available at:
- Package: `/usr/share/smartbotic-database/proto/database.proto`
- Source: `proto/database.proto`

This can be used to generate client stubs in other languages (Python, Go, etc.) if needed.