Smartbotic Database - Status and Roadmap

Current version: 2.11.3 (see VERSION). Last reviewed: 2026-08-10.

This is the single authoritative statement of what exists and what does not. If any other document in this repository disagrees with this one, this one is right and the other is stale - report it.

Feature-level detail lives in CLAUDE.md. This file covers only status: shipped, pending, or abandoned.

How to read the `docs/` tree

Path	What it is	Trust it?
`docs/ROADMAP.md`	This file - current status	Yes
`CLAUDE.md`	Per-release feature notes and hard-won lessons	Yes
`docs/integration-guide.md`	Consumer-facing API guide	Yes
`docs/superpowers/specs/`	Design decisions, each with a status banner	Read the banner first
`docs/superpowers/plans/`	Implementation plans for work not yet done	Yes, but re-validate against current code
`docs/superpowers/plans/archive/`	Plans for work already shipped	Historical only - do not execute
`docs/incidents/`	Post-mortems of resolved production incidents	Historical, lessons folded into `CLAUDE.md`

Storage engine history: Phases A, B, C

Older documents refer to "Phase A/B/C". That vocabulary comes from the 2026-05-15 storage-engine roadmap and is retained only for reading old comments. Do not plan new work in these terms.

Phase	Shipped	What actually happened
A	v1.10.0	yyjson became the parser on hot paths. `Document::data` stayed `nlohmann::json`. Done.
B	v1.11.0	Documents hold a binary `doc_binary::Doc` (yyjson `mut_doc`) with a lazy nlohmann view. Done.
C	v2.0.0	Superseded in flight. Phase C as designed meant writing our own engine: 16 KB slotted pages, a `BPlusTree`, an LRU buffer pool, page-level WAL redo - 2-3 months at high risk. The build-vs-buy spike chose LMDB instead (~2 weeks). Phase C's goal was met (bounded RSS, disk-backed B+ tree) but none of its named components were built.

Consequences of that substitution, which trip up readers:

There is no BufferPool, no Page, no BPlusTree in this codebase. LMDB provides all three internally.
buffer_pool_size_mb does not exist. A comment in database_service.cpp once promised max_memory_mb would be renamed to it "in v2.1". That never happened and is not planned; max_memory_mb still configures MemoryStore eviction, which no longer bounds RSS. LMDB mapsize plus OS page cache does.
"Phase C binary frame" is not a Phase C item. The relevant note is document_store_lmdb.cpp:9, "Stage 4 may optimise to a binary frame later" - part of the write-handler migration below.

Shipped

Every item below is in the installed product as of 2.11.3. CLAUDE.md has the detail and the failure modes.

JSON document store: collections, version history, field-level encryption, TTL
LMDB storage substrate, one env per project, dual-write mirror from MemoryStore
Multi-project namespaces (<project>:<collection>)
Vector storage with SIMD cosine similarity search
Views (read-only projections with baked-in filters)
Row- and column-level access policy, off by default per project
Project-scoped file storage with content-addressed, refcounted, deduplicated blobs
Per-listener TLS and bearer-token auth
Durable snapshots with tiered recovery and read-only lockout
Replication, events/subscribe, migrations, set operations
Referential integrity: relations with restrict / cascade / set_null / no_action, a DUPSORT reverse index, DescribeDelete, relations check, per-collection enforcement switches, and TTL expiry running the same policy as a manual delete
Unique constraints, enforced inside the document's own transaction
Paging fast path (no filter, no sort) and the two-pass filtered/sorted scan
Secondary indexes on declared fields: equality, IN, CONTAINS, EXISTS, ranges, intersection, result ordering, filtered totals, and distinct values / min / max. A measured selectivity guard means declaring an index cannot make a query slower. CLI: indexes, index-create, index-drop, index-values

Pending

Ordered by what a reader is most likely to need next. Nothing here has a committed date.

1. Indexing - what is left

Served from an index: equality, IN, CONTAINS, EXISTS=true, numeric and string ranges, two-list intersection, result ORDERING, filtered TOTALS, and distinct values / min / max. Remaining:

Unique constraints. Built and deliberately not shipped - see the v2.10.0 entry in CLAUDE.md. The check is correct; it cannot be enforced while applyDualWriteMirror swallows every exception and MemoryStore mutates before the mirror runs. Needs the write path restructured, which wants its own change.
NE and REGEX would need a full walk either way. A prefix-anchored REGEX (^abc) could become a range over the string tag - the one real opportunity here.
SEARCH reads whole documents by definition.
Sort + unselective filter stays a full pass, and this one is structural: an exact total_matched requires visiting every match, which is precisely what stopping early avoids. Changing it means making the total approximate - a contract decision, not an optimisation.
Covering reads are closed as not viable: the response always retains six metadata fields and only _id exists in a posting. Would need an ids-only response mode.
Compound (multi-column) indexes. Intersection covers much of the benefit; a real compound index would beat it for a pair queried constantly.

2. `encode_document` still serialises with nlohmann

storage/document_store_lmdb.cpp writes documents via doc.toJson().dump(). Measured, nlohmann::dump() is ~11x slower than yyjson_write on a large document (27.5 ms vs 2.5 ms on 2.91 MB). Swapping it to doc_binary::to_json_text is one function and no API change, but it alters the bytes every write produces, so it needs round-trip equivalence tests against an existing corpus before it can be trusted.

3. Sub-db migration remainder (was "Stage 5")

Three subsystems still persist outside LMDB:

Subsystem	Where it lives now	Target
History	`persistence/history_store.cpp`, `.hlog` files	`_history_<coll>` sub-db keyed `<doc_id>:<version>`; delete `history_store.cpp`
File metadata	on-disk JSON at `<filesDir>/records/<2-char-prefix>/<id>.meta.json`	`_files` sub-db (blobs stay on disk)
View definitions	MemoryStore `_views` system collection	`_views` sub-db keyed by qualified view name

Each is independently shippable. History is the largest.

4. Write-handler migration (was "Stage 4-finish", once labelled the v2.4 milestone)

Handlers would call doc_store_->put() directly, with id generation, timestamps and versioning moved out of MemoryStore into a WriteCoordinator. On completion MemoryStore, persistence/wal.cpp, persistence/snapshot.cpp and the eviction config knobs all delete, and the recovery-mode enum collapses.

This is the v2.x arc's architectural endpoint. It touches the write path that produced the v2.4.3, v2.4.4 and v2.8.0 LMDB handle incidents, so it wants to land in small reviewable pieces with the sub-db identity sentinel kept intact.

5. Relations - what is left after v2.11.0

Shipped in v2.11.0; see the CLAUDE.md entry for the full surface. Remaining, all deliberate and none blocking:

restrict is checked one level deep. A cascade that would destroy a restrict-protected grandchild is refused rather than recursing. Recursion needs an unbounded transaction and its own WAL-first story.
A cascade that fails after its WAL fsync still applies on the next restart. Inherent to WAL-before-LMDB; documented in relation_cascade.hpp and the error.
One narrow race remains: a write extending a TTL during a cascade's fsync-plus-commit. Only reachable for a cascade/set_null relation whose parent collection carries a TTL and is renewed after expiry. The fix is a per-document claim flag that makes the concurrent write lose visibly - not a versioned delete, which does not close it (no version-checked delete primitive exists, and the cascade never goes through remove()).
Cascade is unbounded in memory and transaction size for a very wide parent; a pre-flight child-count cap would bound both.
_relations has no direct-document-write interception, unlike _policies, so an admin editing the declaration by hand has no effect until restart. A replication follower likewise does not re-arm until restart.
No reserved _-prefix check on relation names.

5b. Open after the v2.11.1 consumer feedback round

Raised by a consumer running against 2.11.0 and NOT fixed in 2.11.1, with the reason each was left:

A child reference nested inside an array of OBJECTS cannot be expressed (nodes[].config.credentialId). Path resolution descends objects only, so there is no way to name "this field of every element". An array of scalars is supported and contributes one posting per element. Supporting the object case needs a path syntax and a reverse-index maintenance story for element-level changes; nobody has asked for it twice yet.
validate_on_write is all-or-nothing per relation. null and an absent field are already exempt (they are not references), so "no parent" IS expressible - but an empty STRING is rejected, and a consumer with legitimately empty-string references cannot opt those rows out while validating the rest. The clean answer is for the consumer to write null; a per-relation "treat empty string as absent" flag is the fallback if that is impossible.
RestoreVersion does not reject a write that touches a masked column, unlike Insert/Update/Upsert (rejectMaskedWrite). Same family as the two version-read holes 2.11.1 closed, and latent for the same reason: no deployment has enabled security. Fix when security is first armed for real.
A version's ROW visibility is decided from the current document. If a document has moved out of a principal's row predicate, its whole history becomes invisible; if it has moved in, the history it had while invisible becomes readable. Deliberate (a row predicate selects rows, and the row's identity is its current document) but worth restating before security is armed.

5c. Shutdown latency after v2.11.2

The Subscribe unwind is instant (0 ms measured). What remains in a stop, neither caused nor worsened by v2.11.2, both bounded, neither yet investigated:

~4 s inside grpc::Server::Shutdown() when a client is still connected, draining that connection's transport. With the client killed first, Shutdown returns in 1 ms. Lowering the 5 s grace would cut the tail, but it would also cancel legitimately long streaming calls sooner (DownloadFile chunks a whole file), so it is not a free change.
~4 s between "Persistence manager stopped" and "Database service stopped", i.e. joining component threads. Unmeasured per-thread; likely a poll interval somewhere that could become a condition variable.
stop() appears to run some component stops twice ("FileManager stopped" is logged again after "Database service exited cleanly"). Harmless, noisy, worth a look when either of the above is picked up.

5d. ⚠ Container stop timeout on zeus (found 2026-08-15)

Verifying the v2.11.3 unwind on zeus showed docker stop taking 10.5 s, i.e. hitting Docker's default 10 s timeout and SIGKILLing the process mid teardown - Database service stopped and exited cleanly were absent from the log. The final snapshot happened to land first, so nothing was lost, but that is luck: a slower snapshot would be killed before it renamed into place, leaving the next boot on WAL-only replay and read-only mode - the exact failure v2.11.2 was fixing, reintroduced by the container runtime rather than by the code.

A full teardown is ~10 s today (~4 s gRPC transport drain + ~4-5 s component joins; see 5c), which does not fit in 10 s. The container is now run with --stop-timeout 120, matching the unit file's TimeoutStopSec=300 in spirit.

Any future docker run for this service must carry --stop-timeout, and any docker stop in a script wants -t to match. The deb/systemd path is unaffected.

6. Smaller known gaps

Policy management has no dedicated RPCs; _policies is edited through the ordinary document API with server-side interception.
Service-wide operations (GetStats, SetReadOnly, Create/DropProject) can only be gated coarsely - they have no project to evaluate a policy against. Listener separation is the real boundary there.
event.data->dump() and filter-echo f.value.dump() in database_grpc_impl.cpp are the last nlohmann serialisers on warm paths.
_orphans_* sub-dbs from the v2.4.4 placement repair still await manual comparison.
TTL is not retro-applied: rows written before a default TTL existed keep no expiry.

Not planned

v3. v2.x is the terminal arc.
Removing nlohmann::json. It is the public client API vocabulary type - insert, get, Filter::value, QueryResult::documents. Removing it breaks every consumer call site and needs a soname bump. It is header-only, so it is not a runtime dependency of any shipped package and costs nothing at runtime. The remaining ~453 internal references are cold paths (config, migrations, policies, views) where its exceptions and value semantics are an asset.
A homegrown page engine or buffer pool. Settled: LMDB.
v1.x auto-migration as a separate package. Was slated to extract to smartbotic-database-migrate-v1 in v2.5; v2.5 never shipped and the in-binary path is harmless.

ROADMAP.md 14 KB Permalink History Raw