fszontagh/smartbotic-database

03412d9ad3

v1.7.3

ZIP TAR.GZ
374864e9cb

v1.7.2

ZIP TAR.GZ

v1.7.1 1e570a43b4

v1.7.1 — Replication content-loss fix

Szontágh Ferenc 3 months ago | 84 commits to main since this release

Critical replication fix

Replication was structurally working but silently dropping all user content on every replicated document. Structure survived (_id, _version, timestamps), but the actual data fields were lost. Pre-existing bug from 2026-02-05, unrelated to v1.7.0's eviction work — surfaced by the replication sanity load test added in v1.7.0's follow-up (tests/load_test/run_replication_test.sh).

If you have replication.enabled: false (the default), you're unaffected. If you've turned replication on in any deployment, upgrade to v1.7.1.

The bug

Side	Code	Behavior
Leader `database_service.cpp:515`	`entry.set_data(doc->data.dump())`	Broadcasts only the user's JSON payload
Follower `database_service.cpp:612`	`Document doc = Document::fromJson(json)`	Expects full envelope (id, collection, data, version, timestamps)

Document::fromJson reads j.value("data", {}). When the follower saw {seq: 123, content: "hello"} it looked for a "data" field inside that object, found nothing, and produced a Document with empty data. The replicated doc had the right ID and version metadata but zero user content.

The fix

One-line leader-side change: send doc->toJson() instead of doc->data.dump(). The follower's existing Document::fromJson() call then works correctly.

-     if (doc) entry.set_data(doc->data.dump());
+     if (doc) entry.set_data(doc->toJson().dump());

Follower code unchanged. The ReplicationEntry proto fields (document_id, collection, document_version) stay — they're now slightly redundant with what's in the envelope but still serve as out-of-band routing breadcrumbs.

Test evidence

Using tests/load_test/run_replication_test.sh:

Before fix (v1.7.0):

Inserted on leader:     1000
Seen on follower:       1000
Content matches:        0
Content mismatches:     1000
Verdict: FAIL

After fix (v1.7.1):

Inserted on leader:     1000/1000
Seen on follower:       1000
Content matches:        1000
Content mismatches:     0
Catch-up time:          366ms
Verdict: PASS — all 1000 docs replicated in 366ms, zero corruption

Blame

git blame puts the bug at commit a96c168 on 2026-02-05, ~2.5 months before this release. It has been silently broken since then. Nobody noticed because replication has been opt-in and most deployments run single-node.

Backward compatibility

Wire format change: ReplicationEntry.data payload format changed from "raw user JSON" to "full Document envelope JSON". A v1.7.0 follower talking to a v1.7.1 leader won't understand the new format (will try to parse envelope as user data). Upgrade leaders and followers together.
Internal data format, proto schema, disk format, client API: all unchanged.
Consumer projects: no code changes required.

Packages

Four .deb packages published to repository.smartbotics.ai (suite: trixie):

smartbotic-database_1.7.1-1_amd64.deb
libsmartbotic-db-client_1.7.1-1_amd64.deb
libsmartbotic-db-client-dev_1.7.1-1_amd64.deb
smartbotic-db-cli_1.7.1-1_amd64.deb

Downloads

v1.7.0 7fc01da994

v1.7.0 — Eviction resilience + conf.d + client retry (Zoe incident fix)

Szontágh Ferenc 3 months ago | 89 commits to main since this release

The incident this fixes

On 2026-04-22 the Zoe production instance experienced silent message loss during a shadowman-cpp LLM streaming turn. Root cause: memory eviction was triggered, one-shot evicted 137k documents (92% of the dataset) in ~3 seconds while holding collection locks, shadowman's writes to the messages collection tripped their 5s gRPC deadline, and the WebSocket chunks had already streamed to the WebUI — so on refresh the database had lost rows the UI had already displayed.

This release fixes that at four layers: smarter eviction, stricter admission control, client-side retry, and drop-in config so each consumer can tune the DB for its workload without touching the upstream config.

What's new

Drop-in configuration (`/etc/smartbotic-database/conf.d/*.json`)

Deep-merges *.json files over config.json using RFC 7396 semantics (objects merge recursively, arrays/scalars replace). Consumer projects ship their own tuning without fighting dpkg --configure's conffile prompt:

// /etc/smartbotic-database/conf.d/50-shadowman.json  (shipped by shadowman-cpp deb)
{
  "storage": {
    "memory": {
      "max_memory_mb": 2048,
      "eviction_chunk_size": 500,
      "hot_write_floor_ms": 60000
    }
  }
}

Merge order is lexicographic (00-defaults.json before 50-shadowman.json before 99-local.json). Last writer wins. Syntax errors cause exit code 11 and name the offending file.

Chunked, pressure-aware eviction (the Zoe fix)

Replaces one-shot "evict everything to target" with a steady background drip:

Chunk size (default 1000 docs/pass), chunk pause (50 ms), max passes/tick (20) — the Zoe-incident 137k eviction would now spread over ~20 passes with 50 ms pauses
Hot-write floor — docs updated in the last 30 s are unevictable (protects in-flight LLM streaming upserts)
Per-collection budget — one collection contributes at most share × 1.5 of a chunk (stops one big collection from monopolising the drop)
Quiesce — per-doc in-flight-write counter blocks eviction mid-upsert

Four-level memory pressure state machine

Level	Default %	Behavior
`normal`	< 70%	No eviction
`soft`	70–85%	Trickle: one chunk per tick
`hard`	85–95%	Aggressive + emit `MEMORY_PRESSURE_HIGH` event
`emergency`	≥ 95%	Admission control — writes return `RESOURCE_EXHAUSTED`

Edge-triggered — the PRESSURE_HIGH event doesn't spam under sustained pressure.

Per-collection memory priority

{
  "type": "create_collection",
  "collection": "messages",
  "options": { "memory_priority": "high" }
}

high / normal (default) / low. Selection sort uses (priority, lastAccessedAt) — low-priority collections evicted first; high-priority last. Pinned collections still never evict.

WAL-fallback async unlocking

find() and get() for evicted docs no longer hold the per-collection mutex during the WAL disk-read. The write lock is also released across the documentLoadCallback_ invocation (previously a global serializer across all collections). On successful page-in, the doc is hot-loaded back into the live map with updatedAt = now so the hot-write floor protects it from immediate re-eviction.

`GetMemoryStats` RPC + `Client::getMemoryStats()`

Returns:

auto s = db.getMemoryStats();
// s.totalMemoryBytes, s.maxMemoryBytes
// s.pressurePercent, s.pressureLevel ("normal"|"soft"|"hard"|"emergency")
// s.lastEvictionTimestamp / lastEvictionDocs / lastEvictionBytesFreed
// s.collections[]: collection, documentCount, estimatedBytes, evictedStubCount, priority

Not gated by read-only mode or admission control — operators need stats visibility especially under pressure.

Memory pressure events

Subscribers receive:

MEMORY_PRESSURE_HIGH when pressure transitions into hard/emergency (edge-triggered)
MEMORY_EVICTION_BURST after any chunk evicting ≥ eviction_burst_threshold docs (default 10k)

Payload JSON: {pressure_level, pressure_percent, estimated_bytes, max_bytes} / {evicted_docs, bytes_freed, pressure_level}. System-level events (empty collection/id) — filter by type.

Client-side retry on writes

The client library now automatically retries transient-failure write RPCs:

Status	Retry?
`DEADLINE_EXCEEDED`	✅
`RESOURCE_EXHAUSTED` (admission control / concurrency cap)	✅
`UNAVAILABLE`	✅
All others (`INVALID_ARGUMENT`, `FAILED_PRECONDITION`, etc.)	❌ final

Configurable via Client::Config:

config.writeRetries = 3;
config.writeRetryBackoffMs = 100;      // exponential: 100 → 200 → 400 ...
config.writeRetryMaxBackoffMs = 5000;
config.writeRetryJitter = 0.25;        // ±25%

Wraps insert, updateIfVersion, upsert, remove, patch. Reads are NOT auto-retried. Version-conflict retry inside update() is separate (application-level). Consumers using the client library (shadowman-cpp, callerai, etc.) get this for free — no code changes on their side needed to tolerate the Zoe-style eviction spike.

Config reference (`storage.memory`)

Setting	Default	Meaning
`max_memory_mb`	512	Memory budget
`eviction_chunk_size`	1000	Docs evicted per chunk
`eviction_chunk_pause_ms`	50	Sleep between chunks
`max_eviction_passes_per_trigger`	20	Safety cap on chunks/tick
`hot_write_floor_ms`	30000	Docs updated within this window unevictable
`memory_soft_percent`	70	Enter `soft` pressure
`memory_hard_percent`	85	Enter `hard` — fire PRESSURE_HIGH
`memory_emergency_percent`	95	Enter `emergency` — admission control
`eviction_burst_threshold`	10000	Fire EVICTION_BURST event at this
`eviction_target_percent`	60	Stop evicting below this
`eviction_check_interval_ms`	5000	Wake loop this often

Backward compatibility

All existing RPCs unchanged
No proto breaking changes (only new fields, new RPC, new event types)
No on-disk format changes (snapshot format unchanged from v1.6.x)
Existing deployments keep working at default settings; eviction behavior just becomes more gradual and more observable
CollectionOptions.memory_priority defaults to normal — no implicit change for existing collections
Client library: new fields on Config have safe defaults; existing code compiles unchanged

Tests

New integration tests:

tests/test_config_dropins.cpp — 8 cases: empty config, base-only, drop-in override, lexicographic order, array-replace semantics, invalid-JSON handling, non-JSON file ignoring, missing conf.d directory
tests/test_eviction.cpp — 4 cases: pressure level reporting, hot-write-floor protection, priority-based selection, chunk-size honored

All existing tests continue to pass: test_views, test_vector_storage, test_snapshot_durability, test_timestamp_precision.

Companion changes for consumers

shadowman-cpp (separate repo) should:

Raise gRPC write deadline from 5s → 15s
Stop swallowing DB-write exceptions silently
Drop a /etc/smartbotic-database/conf.d/50-shadowman.json from its deb postinst with workload-tuned values

The client-side retry logic is already in v1.7.0's client library — shadowman-cpp does NOT need to add retry code manually. It just needs to be rebuilt against the v1.7.0 client library.

callerai and other consumers: same story. Rebuild to pick up the retry behavior; optionally ship a conf.d drop-in for workload-specific tuning.

Packages

Four .deb packages published to repository.smartbotics.ai (suite: trixie):

smartbotic-database_1.7.0-1_amd64.deb
libsmartbotic-db-client_1.7.0-1_amd64.deb
libsmartbotic-db-client-dev_1.7.0-1_amd64.deb
smartbotic-db-cli_1.7.0-1_amd64.deb

Downloads

v1.6.3 df1da5479c
v1.6.3 — Filter by system fields (cursor pagination fix)

Szontágh Ferenc 3 months ago | 101 commits to main since this release
The bug

matchesFilters() always read filter fields via getJsonPath(doc.data, ...) — the JSON payload of the document. But DB-system fields (_id, _version, _created_at, _updated_at) don't live inside doc.data; they live on the Document struct itself.

Sort already knew this: the sort path at memory_store.cpp:~847 special-cases these fields and reads them from the struct. The filter path was left behind.

Symptom

Cursor-based pagination silently breaks. A typical query:
```
QueryOptions opts;
opts.filters = {{"_created_at", Op::LT, last_seen_timestamp}};
opts.sortField = "_created_at";
opts.sortDescending = true;
opts.limit = 50;
auto page = db.find("documents", opts);
```
The filter runs before the sort and filters everything out — getJsonPath returns nullopt for _created_at because the field doesn't exist in doc.data. Result: empty page every time, even when there's unambiguously matching data.

Clients using _id, _version, _created_at, or _updated_at in WHERE-style filters were affected. Clients who only sorted by these fields were unaffected (sort worked all along).

The fix

matchesFilters() now mirrors the sort path's system-field special-casing. Adds _id, _version, _created_at, _updated_at. Everything else still goes through getJsonPath(doc.data, field) as before.

The sort path doesn't handle _version (only _id, _created_at, _updated_at), but filtering by _version is cheap and unlocks useful queries like "docs modified at least N times without exposing all documents to the client." Added in this commit for symmetry.

Before / after

Before (1.6.2):
```
// Returns [] — filter excludes everything
db.find("docs", {.filters = {{"_created_at", Op::GT, 0}}});
```
After (1.6.3):
```
// Returns all documents (since _created_at > 0 is always true for real docs)
db.find("docs", {.filters = {{"_created_at", Op::GT, 0}}});
```
Backward compatibility
- No API changes
- No proto changes
- No on-disk format changes
- Queries that didn't use system fields as filters continue to behave identically
- Queries that DID use system fields as filters now return correct results instead of empty
Tests

All existing tests pass (test_views, test_vector_storage, test_timestamp_precision, test_snapshot_durability). The fix is a one-site symmetry with existing sort-path logic — no new test surface added.

Packages

Four .deb packages published to repository.smartbotics.ai (suite: trixie):
- smartbotic-database_1.6.3-1_amd64.deb
- libsmartbotic-db-client_1.6.3-1_amd64.deb
- libsmartbotic-db-client-dev_1.6.3-1_amd64.deb
- smartbotic-db-cli_1.6.3-1_amd64.deb
Downloads
- Source Code (ZIP)
- Source Code (TAR.GZ)
v1.6.2 7dc4d177be
v1.6.2 — gRPC concurrency cap + resource quota

Szontágh Ferenc 3 months ago | 102 commits to main since this release
Highlights

Bounds gRPC server memory use under load. Previously, each concurrent streaming RPC (Subscribe, UploadFile, DownloadFile) could hold up to 100 MB of buffers. With N simultaneous clients that's N × 100 MB — unbounded. This release adds three cooperating safety layers, all configurable, with defaults that preserve current behavior for healthy deployments.

What's new

Configurable gRPC settings

New storage.grpc config section:
```
"storage": {
    "grpc": {
        "max_receive_message_size_mb": 100,
        "max_send_message_size_mb": 100,
        "resource_quota_memory_mb": 256,
        "max_concurrent_subscribe_streams": 50,
        "max_concurrent_file_streams": 10
    }
}
```
Three protection layers
1. Per-message size limits — max_receive_message_size_mb / max_send_message_size_mb. Same 100 MB default as before, but now tunable without rebuilding.
2. gRPC ResourceQuota (256 MB default) — a global memory budget for all inbound buffers across all RPCs combined. gRPC natively rejects new work when the quota is saturated.
3. Per-RPC-type concurrency counters — RAII StreamGuard increments on handler entry, decrements on exit (all paths). When the limit is hit, returns RESOURCE_EXHAUSTED immediately and logs WARN:
```
[WARN] gRPC: rejecting Subscribe — concurrency limit reached (51 >= 50)
```
Separate counters for subscriber streams vs. file streams:
- max_concurrent_subscribe_streams (default 50) — bounds Subscribe
- max_concurrent_file_streams (default 10) — shared pool for UploadFile + DownloadFile
Why the split?

Subscribe streams are long-lived but use small per-message buffers (event notifications). File streams are typically shorter-lived but each can hold ~64 KB × chunks = multi-MB per stream during upload/download. Different defaults reflect those different memory profiles.

Client behavior

When the server returns RESOURCE_EXHAUSTED, clients should back off and retry. Existing client libraries (including PicoBaas SDK, and any consumer using the generated gRPC stubs) handle this status code natively — no client changes required.

Backward compatibility
- All existing behavior preserved: message size defaults unchanged, healthy deployments see no difference
- No proto changes — existing clients work unmodified
- No on-disk format changes
- Config section is additive; omitting it uses safe defaults
Operator visibility

Look for these WARN log entries if you're hitting limits under load:
```
gRPC: rejecting Subscribe — concurrency limit reached (...)
gRPC: rejecting UploadFile — concurrency limit reached (...)
gRPC: rejecting DownloadFile — concurrency limit reached (...)
```
Raise the limits if your workload genuinely needs more concurrent streams and you have the memory headroom for it. Don't just crank the numbers — each Subscribe stream is cheap, but each file stream is not.

Tests

All existing tests pass (test_views, test_vector_storage, test_timestamp_precision, test_snapshot_durability). Concurrency-cap logic itself doesn't need new unit tests — the RAII guard is trivial and the behavior is easy to verify manually (nc localhost 9004 51 times against a 50-limit server).

Packages

Four .deb packages published to repository.smartbotics.ai (suite: trixie):
- smartbotic-database_1.6.2-1_amd64.deb
- libsmartbotic-db-client_1.6.2-1_amd64.deb
- libsmartbotic-db-client-dev_1.6.2-1_amd64.deb
- smartbotic-db-cli_1.6.2-1_amd64.deb
Downloads
- Source Code (ZIP)
- Source Code (TAR.GZ)

v1.6.1 ff4062a411

v1.6.1 — Snapshot durability + tiered recovery + memory accounting

Szontágh Ferenc 3 months ago | 103 commits to main since this release

Critical bugfix release

This release ships the fix for the silent snapshot-truncation bug that took down the Zoe production instance on 2026-04-19 and emptied its database during a routine restart. All v1.x instances are affected and should upgrade. Also includes a tiered recovery system (MySQL innodb_force_recovery style), auto-readonly safety net, and memory-accounting fixes for RSS growth.

The Zoe bug (fixed)

SnapshotManager::createSnapshot() had three independent durability holes:

No error check after ofstream::write() — short writes passed silently
No flush() / fsync() before close() — no durability guarantee
Wrote directly to the final filename — partial writes survived as "valid" snapshots

Combined with a loader that had zero fallback and a recover() return value that was discarded at the call site, the database silently started empty whenever the latest snapshot was truncated — accepting writes onto a blank store. All production data on Zoe was overwritten this way.

What's fixed

Atomic snapshot writer

Writes to <path>.tmp first
Checks failbit after every write() call
flush() + fsync() on the file before considering it durable
std::filesystem::rename(tmp, final) — atomic on same filesystem
fsync() the containing directory so the rename survives crash
Calls verifySnapshot() after rename; deletes + throws if it fails
cleanupOldSnapshots() only runs after verification passes
listSnapshots() removes orphaned .tmp files from prior crashed writes

Tiered recovery modes

MySQL-style levels selectable via --recovery-mode=X flag or recovery.mode config:

Level	Name	Behavior
0	`normal`	Load latest snapshot + replay WAL. Default. Auto-escalates to older snapshot if latest is corrupt.
1	`snapshot_fallback`	Try snapshots newest→oldest, first that loads wins
2	`wal_only`	Ignore snapshots, replay WAL from 0
3	`best_effort`	snapshot_fallback → falls through to wal_only
4	`force_empty`	Start empty. Snapshots + WAL preserved on disk for forensics

Auto-readonly safety net

If recovery used anything other than "latest snapshot loaded cleanly," the DB boots in read-only mode. Prevents silent data corruption from writing onto stale state. Operator explicitly acknowledges:

smartbotic-db-cli unlock                  # live, no restart
smartbotic-database --force-readwrite     # on next restart

New RPCs + CLI

SetReadOnly / GetReadOnlyStatus gRPC RPCs
Client::lock() / unlock() / getReadOnlyStatus() methods
smartbotic-db-cli lock / unlock / status subcommands
Exit code 10 when recovery is refused
Exit code 11 for invalid --recovery-mode value

Streaming snapshot compression

Snapshot writer was allocating 2x the database size in transient buffers (uncompressed data + LZ4-bound output buffer) during snapshot creation. For a 3 GB database that meant ~6 GB RSS spike every hour. New v4 snapshot format splits the body into 16 MB LZ4 chunks with length prefixes — peak memory per chunk is ~16 MB regardless of DB size. v3 snapshots still load for backward compat.

Memory accounting

currentMemoryBytes_ (the eviction budget) was under-counting real memory by 2–3×:

estimateDocumentSize() only counted data.dump().size() + 100 — ignored JSON tree overhead, metadata strings
Version history (std::deque<DocumentVersion> per doc) was completely off-books
Vector embeddings (std::vector<float> per doc) were completely off-books

All three now properly tracked. Eviction now fires against real memory, not a fiction.

Config surface (new)

"persistence": {
    "snapshots": {
        "validate_after_write": true,
        "cleanup_only_if_verified": true
    },
    "recovery": {
        "mode": "normal",
        "auto_escalate": true,
        "allow_empty_on_fresh_install": true
    }
}

All defaults preserve safe behavior — existing deployments work unchanged.

Backward compatibility

v3 snapshot loader preserved — old snapshots still work
max_versions=0 (unlimited) still allowed — memory accounting just made honest
All existing RPCs unchanged
Consumer projects (shadowman-cpp, callerai) need no code changes

Migration for affected instances

For instances that hit the Zoe bug:

Stop the service BEFORE any writes overwhelm partial recovery options
Preserve /var/lib/smartbotic-database/ forensic state
Upgrade to v1.6.1
Start with --recovery-mode=snapshot_fallback to try older snapshots
If that fails, escalate to --recovery-mode=wal_only (replays WAL from 0)
The DB will boot read-only after non-trivial recovery — review state via smartbotic-db-cli status, extract data if needed, then smartbotic-db-cli unlock if acceptable

Tests

tests/test_snapshot_durability.cpp — 7 cases including atomic write, round-trip, post-write verification catches truncation, loadWithFallback, orphaned .tmp cleanup, all-corrupt failure, v4 10K-doc round-trip
Existing test_views, test_vector_storage, test_timestamp_precision all still pass

Documentation

New Recovery & Read-Only Mode section in docs/integration-guide.md — recovery modes, operator workflow, client/CLI API, exit codes
CLAUDE.md — Durable Snapshots + Tiered Recovery feature bullet

Packages

Four .deb packages published to repository.smartbotics.ai (suite: trixie):

smartbotic-database_1.6.1-1_amd64.deb
libsmartbotic-db-client_1.6.1-1_amd64.deb
libsmartbotic-db-client-dev_1.6.1-1_amd64.deb
smartbotic-db-cli_1.6.1-1_amd64.deb

Downloads

v1.6.0 18571feb1f
v1.6.0 — Per-collection timestamp precision

Szontágh Ferenc 3 months ago | 119 commits to main since this release
Highlights

Adds per-collection configuration for timestamp precision. Collections that receive rapid-fire writes (LLM streaming, agent tool loops) can opt into nanosecond precision to eliminate _created_at / _updated_at ties. Default precision stays ms — every existing collection is unchanged.

New
- CollectionConfig { timestamp_precision: "ms" | "ns" } — per-collection config stored in _collection_meta system collection
- CollectionConfigManager — in-memory cache with O(1) hot-path lookup, no store round-trip per write
- MemoryStore::currentTimeFor(collection) replaces currentTimeMs() at every document write site (insert, update, patch, upsert, updateIfVersion, restoreVersion, set ops)
- RPCs:
  - ConfigureCollection — flips the config atomically; future writes use the new precision
  - GetCollectionConfig — returns current config (or default)
  - MigrateCollectionTimestamps — idempotent, resumable conversion helper
Client API
```
using Cfg = smartbotic::database::Client::CollectionConfig;

db.configureCollection("llm_tokens", Cfg{"ns"});           // switch to ns
auto cfg = db.getCollectionConfig("llm_tokens");           // read back
bool isSet = db.hasCollectionConfig("llm_tokens");         // true

// One-shot migration during maintenance window
auto result = db.migrateCollectionTimestamps("llm_tokens", "ms", "ns");
// result.rowsMigrated / rowsSkipped
```
Migration semantics
- configureCollection does not auto-migrate existing data — atomic, reversible config flip only
- migrateCollectionTimestamps is idempotent/resumable via a 10^15 threshold:
  - ms values (< 10^15) are multiplied by 10^6 for ms → ns
  - ns values (≥ 10^15) are divided by 10^6 for ns → ms
  - Already-converted rows count as rows_skipped
- Metadata-only mutation — version numbers NOT incremented, history NOT updated, WAL NOT appended
Backward compatibility
- Default for new and existing collections stays ms
- No changes to on-disk format, WAL format, or snapshot format
- Consumer projects (shadowman, callerai) see zero behavior change until they explicitly call configureCollection
- Collection metadata timestamps (coll->createdAt/updatedAt) stay in ms — only document _created_at / _updated_at are affected
Recommended workflow for existing production data
1. Plan a maintenance window
2. db.configureCollection(name, {"ns"}) — atomic config flip
3. db.migrateCollectionTimestamps(name, "ms", "ns") — convert existing docs
4. Verify sample reads show ns-range timestamps
5. Resume traffic
Between steps 2 and 3, new writes are ns and old docs are ms — this is temporary but safe. The 10^15 threshold cleanly separates the two ranges, so ordering still works.

Tests
- tests/test_timestamp_precision.cpp — 6 cases including a 1000-row tight-loop uniqueness check for ns precision
Docs
- docs/integration-guide.md — new Collection Configuration section
- CLAUDE.md — Per-collection Timestamp Precision feature bullet
Packages

Four .deb packages published to repository.smartbotics.ai (suite: trixie):
- smartbotic-database_1.6.0-1_amd64.deb
- libsmartbotic-db-client_1.6.0-1_amd64.deb
- libsmartbotic-db-client-dev_1.6.0-1_amd64.deb
- smartbotic-db-cli_1.6.0-1_amd64.deb
Downloads
- Source Code (ZIP)
- Source Code (TAR.GZ)

v1.5.0 9f0512c9e8

v1.5.0 — Views

Szontágh Ferenc 3 months ago | 125 commits to main since this release

Highlights

Read-only views over collections — named projections with baked-in filters, default sort, and include/exclude field lists. Views are the database's answer to SQL views: a stable API contract plus a security enforcement boundary. Two clients can query the same underlying collection through different views and see different data.

New

ViewManager — loads views from _views system collection at startup, caches in memory for O(1) isView(name) lookups
applyProjection() — pure helper that filters document JSON by include/exclude paths (supports nested dot-notation: user.profile.name)
4 new RPCs: CreateView, DropView, ListViews, GetViewInfo
create_view migration operation — declarative view creation with JSON-level filter op naming ("EQ", "NE", "GT", ...)

Features per view

Each view carries:

include — field paths to keep (dot-notation; empty = all fields)
exclude — field paths to hide (only used when include is empty)
where — baked-in filter list, uses all operators (EQ/NE/GT/GTE/LT/LTE/IN/CONTAINS/EXISTS/REGEX/SEARCH)
default_sort — baked-in default sort; caller can override

Composition rules

Dimension	Semantic	Why
Filters	View AND caller — both applied	Security enforcement — consumers cannot escape view constraints
Sort	Caller overrides view default	Presentation preference, not security
Include/exclude	View-enforced on every response	View defines the contract
Writes	Rejected with "views are read-only"	No side-channel mutations through a view

Metadata preserved

_id, _version, _created_at, _updated_at, _created_by, _updated_by are always returned regardless of include/exclude — clients need them for identification and optimistic locking.

Client API

using Op = smartbotic::database::Client::FilterOp;

// Simple projection
db.createView("users_public", "users",
              /*include=*/{"id", "name", "avatar_url", "profile.public_bio"});

// Full-featured view with baked-in filter + sort
db.createView("active_admins", "users",
              /*include=*/{"id", "name", "role"},
              /*exclude=*/{},
              /*where=*/{
                  {"status", Op::EQ, "active"},
                  {"role",   Op::EQ, "admin"}
              },
              /*defaultSort=*/smartbotic::database::Client::Sort{"name", false});

// Query like a collection
auto results = db.find("active_admins", {
    .filters = {{"department", Op::EQ, "eng"}},   // AND-merged with view's where
    .limit = 50
});

// Listing & inspection
auto views = db.listViews();
auto info = db.getViewInfo("active_admins");
db.dropView("active_admins");

Constraints

View names cannot start with _ (reserved for system collections)
View names cannot equal an existing collection name
No view-of-view — views must target real collections
Writes through view names (insert/update/patch/upsert/remove/batch*) are rejected with a clear error

Via migrations

{
  "version": "020",
  "operations": [
    {
      "type": "create_view",
      "name": "active_admins",
      "collection": "users",
      "include": ["id", "name", "role"],
      "where": [
        {"field": "status", "op": "EQ", "value": "active"},
        {"field": "role",   "op": "EQ", "value": "admin"}
      ],
      "default_sort": {"field": "name", "descending": false}
    }
  ]
}

Migrations are idempotent — re-running a migration that creates an existing view is a no-op.

Tests

tests/test_views.cpp — 9 unit tests covering the projection helper (include/exclude precedence, nested paths, metadata preservation, missing paths)

Docs

docs/integration-guide.md — new Views section with composition tables and rationale
CLAUDE.md — Views feature bullet

Packages

Four .deb packages published to repository.smartbotics.ai (suite: trixie):

smartbotic-database_1.5.0-1_amd64.deb
libsmartbotic-db-client_1.5.0-1_amd64.deb
libsmartbotic-db-client-dev_1.5.0-1_amd64.deb
smartbotic-db-cli_1.5.0-1_amd64.deb

Downloads

v1.4.0 24b79fa014

v1.4.0 — Server-side atomic PatchDocument

Szontágh Ferenc 3 months ago | 135 commits to main since this release

Highlights

New PatchDocument RPC — atomic server-side field merge that eliminates the lost-update race for concurrent partial updates. Two clients can safely modify different fields of the same document simultaneously without any retry loop.

The race this solves

v1.3.0's update() added automatic optimistic locking with retry, which prevents writes from being silently lost — but it's still "last writer wins" for complete document replacement. If two clients read v1, one sets name and the other sets email, the last update() overwrites the other's change.

patch() solves this at the server: the read-merge-write happens inside the collection lock, using RFC 7396 merge_patch() semantics.

// Client A:                              // Client B:
db.patch("users", "u1",                   db.patch("users", "u1",
    {{"name", "Alice"}});                     {{"email", "b@x.com"}});

// Server-side, serialized by the collection lock:
// A: read doc → merge {name: "Alice"}   → write v2 → unlock
// B: read doc → merge {email: "b@x.com"} → write v3 → unlock
// Result: {name: "Alice", email: "b@x.com"} — nothing lost

New

PatchDocument RPC + PatchDocumentRequest/Response proto messages
MemoryStore::patchDocument(collection, id, patch, actor) — atomic read-merge-write inside the collection lock; returns new version or 0 if not found
Client::patch(collection, id, fields, actor) — returns new version or 0 on failure

Semantics

Dimension	Behavior
Field merge	RFC 7396 merge-patch — specified fields replace, unspecified fields preserved
Concurrency	Atomic within collection lock — no race possible
Version	Incremented once per `patch()` call
History	Pre-patch state saved to version history
Vectors	`_vector` field in patch is handled (updated or preserved)
Events	`update` event fired post-patch
WAL	Patch persisted through the standard persist callback

When to use what

Method	Use case	Concurrency safety
`patch()`	Change specific fields	Atomic server-side, no data loss
`update()`	Replace entire document	Optimistic locking + retry, last writer wins
`updateIfVersion()`	Detect conflicts manually	Fails on conflict, caller retries

Use patch() by default — it's the safest and covers most real-world updates (changing 1-3 fields at a time).

Backward compatibility

All existing RPCs unchanged
update(), updateIfVersion(), upsert() continue to work as before
No on-disk format change — patches just produce normal updated documents through the same storage path

Docs

docs/integration-guide.md — new Concurrency: update() vs patch() section

Packages

Four .deb packages published to repository.smartbotics.ai (suite: trixie):

smartbotic-database_1.4.0-1_amd64.deb
libsmartbotic-db-client_1.4.0-1_amd64.deb
libsmartbotic-db-client-dev_1.4.0-1_amd64.deb
smartbotic-db-cli_1.4.0-1_amd64.deb

Downloads

Previous Next

Releases

Critical replication fix

The bug

The fix

Test evidence

Blame

Backward compatibility

Packages

Downloads

The incident this fixes

What's new

Drop-in configuration (/etc/smartbotic-database/conf.d/*.json)

Chunked, pressure-aware eviction (the Zoe fix)

Four-level memory pressure state machine

Per-collection memory priority

WAL-fallback async unlocking

GetMemoryStats RPC + Client::getMemoryStats()

Memory pressure events

Client-side retry on writes

Config reference (storage.memory)

Backward compatibility

Tests

Companion changes for consumers

Packages

Downloads

The bug

Symptom

The fix

Before / after

Backward compatibility

Tests

Packages

Downloads

Highlights

What's new

Configurable gRPC settings

Three protection layers

Why the split?

Client behavior

Backward compatibility

Operator visibility

Tests

Packages

Downloads

Critical bugfix release

The Zoe bug (fixed)

What's fixed

Atomic snapshot writer

Tiered recovery modes

Auto-readonly safety net

New RPCs + CLI

Streaming snapshot compression

Memory accounting

Config surface (new)

Backward compatibility

Migration for affected instances

Tests

Documentation

Packages

Downloads

Highlights

New

Client API

Migration semantics

Backward compatibility

Recommended workflow for existing production data

Tests

Docs

Packages

Downloads

Highlights

New

Features per view

Composition rules

Metadata preserved

Client API

Constraints

Via migrations

Tests

Docs

Drop-in configuration (`/etc/smartbotic-database/conf.d/*.json`)

`GetMemoryStats` RPC + `Client::getMemoryStats()`

Config reference (`storage.memory`)