# Smartbotic Database - Status and Roadmap **Current version: 2.11.2** (see `VERSION`). Last reviewed: 2026-08-10. This is the single authoritative statement of what exists and what does not. If any other document in this repository disagrees with this one, this one is right and the other is stale - report it. Feature-level detail lives in `CLAUDE.md`. This file covers only *status*: shipped, pending, or abandoned. --- ## How to read the `docs/` tree | Path | What it is | Trust it? | |------|-----------|-----------| | `docs/ROADMAP.md` | This file - current status | Yes | | `CLAUDE.md` | Per-release feature notes and hard-won lessons | Yes | | `docs/integration-guide.md` | Consumer-facing API guide | Yes | | `docs/superpowers/specs/` | Design decisions, each with a status banner | Read the banner first | | `docs/superpowers/plans/` | Implementation plans for work **not yet done** | Yes, but re-validate against current code | | `docs/superpowers/plans/archive/` | Plans for work already shipped | **Historical only** - do not execute | | `docs/incidents/` | Post-mortems of resolved production incidents | Historical, lessons folded into `CLAUDE.md` | --- ## Storage engine history: Phases A, B, C Older documents refer to "Phase A/B/C". That vocabulary comes from the 2026-05-15 storage-engine roadmap and is **retained only for reading old comments**. Do not plan new work in these terms. | Phase | Shipped | What actually happened | |-------|---------|------------------------| | A | v1.10.0 | yyjson became the *parser* on hot paths. `Document::data` stayed `nlohmann::json`. Done. | | B | v1.11.0 | Documents hold a binary `doc_binary::Doc` (yyjson `mut_doc`) with a lazy nlohmann view. Done. | | C | v2.0.0 | **Superseded in flight.** Phase C as designed meant writing our own engine: 16 KB slotted pages, a `BPlusTree`, an LRU buffer pool, page-level WAL redo - 2-3 months at high risk. The build-vs-buy spike chose LMDB instead (~2 weeks). Phase C's *goal* was met (bounded RSS, disk-backed B+ tree) but **none of its named components were built.** | Consequences of that substitution, which trip up readers: - **There is no `BufferPool`, no `Page`, no `BPlusTree` in this codebase.** LMDB provides all three internally. - **`buffer_pool_size_mb` does not exist.** A comment in `database_service.cpp` once promised `max_memory_mb` would be renamed to it "in v2.1". That never happened and is not planned; `max_memory_mb` still configures MemoryStore eviction, which no longer bounds RSS. LMDB mapsize plus OS page cache does. - "Phase C binary frame" is **not** a Phase C item. The relevant note is `document_store_lmdb.cpp:9`, "Stage 4 may optimise to a binary frame later" - part of the write-handler migration below. --- ## Shipped Every item below is in the installed product as of 2.11.2. `CLAUDE.md` has the detail and the failure modes. - JSON document store: collections, version history, field-level encryption, TTL - LMDB storage substrate, one env per project, dual-write mirror from MemoryStore - Multi-project namespaces (`:`) - Vector storage with SIMD cosine similarity search - Views (read-only projections with baked-in filters) - Row- and column-level access policy, off by default per project - Project-scoped file storage with content-addressed, refcounted, deduplicated blobs - Per-listener TLS and bearer-token auth - Durable snapshots with tiered recovery and read-only lockout - Replication, events/subscribe, migrations, set operations - Referential integrity: relations with restrict / cascade / set_null / no_action, a DUPSORT reverse index, `DescribeDelete`, `relations check`, per-collection enforcement switches, and TTL expiry running the same policy as a manual delete - Unique constraints, enforced inside the document's own transaction - Paging fast path (no filter, no sort) and the two-pass filtered/sorted scan - Secondary indexes on declared fields: equality, IN, CONTAINS, EXISTS, ranges, intersection, result ordering, filtered totals, and distinct values / min / max. A measured selectivity guard means declaring an index cannot make a query slower. CLI: `indexes`, `index-create`, `index-drop`, `index-values` --- ## Pending Ordered by what a reader is most likely to need next. Nothing here has a committed date. ### 1. Indexing - what is left Served from an index: equality, `IN`, `CONTAINS`, `EXISTS=true`, numeric and string ranges, two-list intersection, result ORDERING, filtered TOTALS, and distinct values / min / max. Remaining: - **Unique constraints.** Built and deliberately not shipped - see the v2.10.0 entry in `CLAUDE.md`. The check is correct; it cannot be enforced while `applyDualWriteMirror` swallows every exception and MemoryStore mutates before the mirror runs. Needs the write path restructured, which wants its own change. - **`NE` and `REGEX`** would need a full walk either way. A prefix-anchored `REGEX` (`^abc`) could become a range over the string tag - the one real opportunity here. - **`SEARCH`** reads whole documents by definition. - **Sort + unselective filter** stays a full pass, and this one is structural: an exact `total_matched` requires visiting every match, which is precisely what stopping early avoids. Changing it means making the total approximate - a contract decision, not an optimisation. - **Covering reads** are closed as not viable: the response always retains six metadata fields and only `_id` exists in a posting. Would need an ids-only response mode. - **Compound (multi-column) indexes.** Intersection covers much of the benefit; a real compound index would beat it for a pair queried constantly. ### 2. `encode_document` still serialises with nlohmann `storage/document_store_lmdb.cpp` writes documents via `doc.toJson().dump()`. Measured, `nlohmann::dump()` is ~11x slower than `yyjson_write` on a large document (27.5 ms vs 2.5 ms on 2.91 MB). Swapping it to `doc_binary::to_json_text` is one function and no API change, but it alters the bytes every write produces, so it needs round-trip equivalence tests against an existing corpus before it can be trusted. ### 3. Sub-db migration remainder (was "Stage 5") Three subsystems still persist outside LMDB: | Subsystem | Where it lives now | Target | |-----------|-------------------|--------| | History | `persistence/history_store.cpp`, `.hlog` files | `_history_` sub-db keyed `:`; delete `history_store.cpp` | | File metadata | on-disk JSON at `/records/<2-char-prefix>/.meta.json` | `_files` sub-db (blobs stay on disk) | | View definitions | MemoryStore `_views` system collection | `_views` sub-db keyed by qualified view name | Each is independently shippable. History is the largest. ### 4. Write-handler migration (was "Stage 4-finish", once labelled the v2.4 milestone) Handlers would call `doc_store_->put()` directly, with id generation, timestamps and versioning moved out of MemoryStore into a `WriteCoordinator`. On completion MemoryStore, `persistence/wal.cpp`, `persistence/snapshot.cpp` and the eviction config knobs all delete, and the recovery-mode enum collapses. This is the v2.x arc's architectural endpoint. It touches the write path that produced the v2.4.3, v2.4.4 and v2.8.0 LMDB handle incidents, so it wants to land in small reviewable pieces with the sub-db identity sentinel kept intact. ### 5. Relations - what is left after v2.11.0 Shipped in v2.11.0; see the `CLAUDE.md` entry for the full surface. Remaining, all deliberate and none blocking: - **`restrict` is checked one level deep.** A cascade that would destroy a `restrict`-protected grandchild is refused rather than recursing. Recursion needs an unbounded transaction and its own WAL-first story. - **A cascade that fails after its WAL fsync still applies on the next restart.** Inherent to WAL-before-LMDB; documented in `relation_cascade.hpp` and the error. - **One narrow race remains:** a write extending a TTL *during* a cascade's fsync-plus-commit. Only reachable for a `cascade`/`set_null` relation whose parent collection carries a TTL and is renewed after expiry. The fix is a per-document claim flag that makes the concurrent write lose visibly - **not** a versioned delete, which does not close it (no version-checked delete primitive exists, and the cascade never goes through `remove()`). - **Cascade is unbounded in memory and transaction size** for a very wide parent; a pre-flight child-count cap would bound both. - `_relations` has no direct-document-write interception, unlike `_policies`, so an admin editing the declaration by hand has no effect until restart. A replication follower likewise does not re-arm until restart. - No reserved `_`-prefix check on relation names. ### 5b. Open after the v2.11.1 consumer feedback round Raised by a consumer running against 2.11.0 and NOT fixed in 2.11.1, with the reason each was left: - **A child reference nested inside an array of OBJECTS cannot be expressed** (`nodes[].config.credentialId`). Path resolution descends objects only, so there is no way to name "this field of every element". An array of scalars is supported and contributes one posting per element. Supporting the object case needs a path syntax and a reverse-index maintenance story for element-level changes; nobody has asked for it twice yet. - **`validate_on_write` is all-or-nothing per relation.** `null` and an absent field are already exempt (they are not references), so "no parent" IS expressible - but an empty STRING is rejected, and a consumer with legitimately empty-string references cannot opt those rows out while validating the rest. The clean answer is for the consumer to write `null`; a per-relation "treat empty string as absent" flag is the fallback if that is impossible. - **`RestoreVersion` does not reject a write that touches a masked column**, unlike `Insert`/`Update`/`Upsert` (`rejectMaskedWrite`). Same family as the two version-read holes 2.11.1 closed, and latent for the same reason: no deployment has enabled security. Fix when security is first armed for real. - **A version's ROW visibility is decided from the current document.** If a document has moved out of a principal's row predicate, its whole history becomes invisible; if it has moved in, the history it had while invisible becomes readable. Deliberate (a row predicate selects rows, and the row's identity is its current document) but worth restating before security is armed. ### 6. Smaller known gaps - Policy management has no dedicated RPCs; `_policies` is edited through the ordinary document API with server-side interception. - Service-wide operations (`GetStats`, `SetReadOnly`, `Create/DropProject`) can only be gated coarsely - they have no project to evaluate a policy against. Listener separation is the real boundary there. - `event.data->dump()` and filter-echo `f.value.dump()` in `database_grpc_impl.cpp` are the last nlohmann serialisers on warm paths. - `_orphans_*` sub-dbs from the v2.4.4 placement repair still await manual comparison. - TTL is not retro-applied: rows written before a default TTL existed keep no expiry. --- ## Not planned - **v3.** v2.x is the terminal arc. - **Removing `nlohmann::json`.** It is the public client API vocabulary type - `insert`, `get`, `Filter::value`, `QueryResult::documents`. Removing it breaks every consumer call site and needs a soname bump. It is header-only, so it is not a runtime dependency of any shipped package and costs nothing at runtime. The remaining ~453 internal references are cold paths (config, migrations, policies, views) where its exceptions and value semantics are an asset. - **A homegrown page engine or buffer pool.** Settled: LMDB. - **v1.x auto-migration as a separate package.** Was slated to extract to `smartbotic-database-migrate-v1` in v2.5; v2.5 never shipped and the in-binary path is harmless.