|
@@ -41,6 +41,7 @@ cmake --build build -j$(nproc) && ./build/tests/test_vector_storage
|
|
|
- **Auto-backup before deb upgrade** — `apt upgrade smartbotic-database` runs a `preinst` hook that snapshots `/var/lib/smartbotic-database/` to `/var/backups/smartbotic-database/pre-upgrade-<ts>-from-<version>/` via `cp -a` after the old `prerm` stops the service. Retention `3`, 1.2× free-space precheck on the backup volume (aborts the upgrade if short). Opt out with `SMARTBOTIC_DB_SKIP_BACKUP=1 apt upgrade`. Tunables: `SMARTBOTIC_DB_BACKUP_RETENTION`, `SMARTBOTIC_DB_BACKUP_ROOT`.
|
|
- **Auto-backup before deb upgrade** — `apt upgrade smartbotic-database` runs a `preinst` hook that snapshots `/var/lib/smartbotic-database/` to `/var/backups/smartbotic-database/pre-upgrade-<ts>-from-<version>/` via `cp -a` after the old `prerm` stops the service. Retention `3`, 1.2× free-space precheck on the backup volume (aborts the upgrade if short). Opt out with `SMARTBOTIC_DB_SKIP_BACKUP=1 apt upgrade`. Tunables: `SMARTBOTIC_DB_BACKUP_RETENTION`, `SMARTBOTIC_DB_BACKUP_ROOT`.
|
|
|
- **yyjson hot-path parsing (v1.10.0+)** — every JSON parse on WAL replay, snapshot deserialize, history reads, gRPC request bodies, and replication apply goes through `smartbotic::db::parse_to_nlohmann` (yyjson-backed) instead of `nlohmann::json::parse`. Measured 2.80× speedup on Zoe-shape corpora (136ms→49ms on 10k docs). In-memory representation stays `nlohmann::json` until Phase B. Wire and on-disk JSON bytes unchanged. New runtime dep: `libyyjson0`. Bench: `./build/tests/bench_json_parse <corpus.jsonl>`.
|
|
- **yyjson hot-path parsing (v1.10.0+)** — every JSON parse on WAL replay, snapshot deserialize, history reads, gRPC request bodies, and replication apply goes through `smartbotic::db::parse_to_nlohmann` (yyjson-backed) instead of `nlohmann::json::parse`. Measured 2.80× speedup on Zoe-shape corpora (136ms→49ms on 10k docs). In-memory representation stays `nlohmann::json` until Phase B. Wire and on-disk JSON bytes unchanged. New runtime dep: `libyyjson0`. Bench: `./build/tests/bench_json_parse <corpus.jsonl>`.
|
|
|
- **Binary-backed Document storage (v1.11.0+)** — `Document::data` is now a `smartbotic::db::doc_binary::Doc` (yyjson `mut_doc` wrapper) with a lazy `nlohmann::json` view cache. New accessor surface: `doc.field(name)` is the O(field-count) fast-path that reads one top-level field without materialising the full tree (the dominant read pattern), `doc.data()` is the lazy full-tree accessor, `doc.set_data(j)` re-encodes from an nlohmann tree. Wire and on-disk JSON bytes unchanged. **Honest note:** per-doc heap footprint is ~1.5% smaller than v1.10 (1618→1594 bytes on Zoe-shape) — the win is field-access CPU + Phase C scaffolding, not RSS. See `docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md` post-implementation amendment for the bench numbers. Bench: `./build/tests/bench_doc_memory --mode=<nlohmann|binary> [N]`.
|
|
- **Binary-backed Document storage (v1.11.0+)** — `Document::data` is now a `smartbotic::db::doc_binary::Doc` (yyjson `mut_doc` wrapper) with a lazy `nlohmann::json` view cache. New accessor surface: `doc.field(name)` is the O(field-count) fast-path that reads one top-level field without materialising the full tree (the dominant read pattern), `doc.data()` is the lazy full-tree accessor, `doc.set_data(j)` re-encodes from an nlohmann tree. Wire and on-disk JSON bytes unchanged. **Honest note:** per-doc heap footprint is ~1.5% smaller than v1.10 (1618→1594 bytes on Zoe-shape) — the win is field-access CPU + Phase C scaffolding, not RSS. See `docs/superpowers/specs/2026-05-15-binary-doc-format-decision.md` post-implementation amendment for the bench numbers. Bench: `./build/tests/bench_doc_memory --mode=<nlohmann|binary> [N]`.
|
|
|
|
|
+- **v2.3.1 two bug fixes found by post-2.3.0 stress + replication tests.** (1) `MemoryStore::getEvictionCandidates` compared `doc.updatedAt` against a millisecond-scaled `hotWriteFloor`; v2.2.0's default-precision flip to nanoseconds made every doc look "always within the hot-write window" so eviction never fired. Pre-existing v2.2.0 regression — load_test_mixed was never re-run after v2.2.0 shipped. Fix: pick ms- or ns-scaled floor per-doc using the 10^15 magnitude heuristic. (2) `DatabaseService::applyReplicatedEntry` called `MemoryStore::loadDocument` which bypasses the persist callback, so the v2.3 LMDB mirror never fired for replicated entries. Followers got writes into MemoryStore but the LMDB env stayed empty; v2.3 LMDB-first reads on the follower returned 0 docs. Fix: explicitly drive `applyDualWriteMirror` through the project registry after `loadDocument`. Replication catch-up time on a 100-doc test: 65ms.
|
|
|
- **v2.3.0 multi-project namespaces.** Each smartbotic-database instance now hosts multiple `project` namespaces, each with its own collections. Wire form: `<project>:<collection>` (e.g. `my_app:users`); bare names like `"users"` auto-qualify to `"default:users"` so every v2.0-v2.2 client keeps working without code changes. Storage: one LMDB env per project at `<dataDir>/projects/<name>/env/`. On first v2.3 boot the legacy `<dataDir>/env/` is atomically renamed under `projects/default/env/` (idempotent; refuses to start if both exist). Server-side: `ProjectStoreRegistry` (`service/src/storage/project_store.{hpp,cpp}`) owns the per-project envs; `DatabaseService::docStore(project)` resolves the LMDB store for a given name; mirror routing is via a resolver callback the service hands to `MemoryStore::setDocumentStoreMirror(resolver, ...)`. Client-side: `Client::Config::project` defaults to `"default"`; every outgoing collection name is auto-prepended (`qualify(coll)`) unless it already contains `:`. Per-call overrides via the qualified form always win. New gRPC RPCs: `ListProjects`, `CreateProject(name)`, `DropProject(name)` — `default` cannot be dropped. CRUD is idempotent and validated against `^[a-zA-Z_][a-zA-Z0-9_-]{0,62}$` (no `:`, no `/`, no `_-prefix). Reserved: `default` (always present); `_-prefix` (system). Tests: `test_project_addressing` (23 parser assertions), `test_project_crud` (29 placeholder assertions). 14/14 ctest green. Stress: load_test_mixed still passes against the multi-project build.
|
|
- **v2.3.0 multi-project namespaces.** Each smartbotic-database instance now hosts multiple `project` namespaces, each with its own collections. Wire form: `<project>:<collection>` (e.g. `my_app:users`); bare names like `"users"` auto-qualify to `"default:users"` so every v2.0-v2.2 client keeps working without code changes. Storage: one LMDB env per project at `<dataDir>/projects/<name>/env/`. On first v2.3 boot the legacy `<dataDir>/env/` is atomically renamed under `projects/default/env/` (idempotent; refuses to start if both exist). Server-side: `ProjectStoreRegistry` (`service/src/storage/project_store.{hpp,cpp}`) owns the per-project envs; `DatabaseService::docStore(project)` resolves the LMDB store for a given name; mirror routing is via a resolver callback the service hands to `MemoryStore::setDocumentStoreMirror(resolver, ...)`. Client-side: `Client::Config::project` defaults to `"default"`; every outgoing collection name is auto-prepended (`qualify(coll)`) unless it already contains `:`. Per-call overrides via the qualified form always win. New gRPC RPCs: `ListProjects`, `CreateProject(name)`, `DropProject(name)` — `default` cannot be dropped. CRUD is idempotent and validated against `^[a-zA-Z_][a-zA-Z0-9_-]{0,62}$` (no `:`, no `/`, no `_-prefix). Reserved: `default` (always present); `_-prefix` (system). Tests: `test_project_addressing` (23 parser assertions), `test_project_crud` (29 placeholder assertions). 14/14 ctest green. Stress: load_test_mixed still passes against the multi-project build.
|
|
|
- **v2.2.1 bug fixes from shadowman code review.** Three issues found by a code-review pass on the v2.0→v2.2.0 commit range, all now fixed: (1) `MemoryStore::updateIfVersion` was missing `extractVector` + `storeVector` + `mirrorVectorToDocStore` calls — optimistic-lock updates on vector collections left the vector mirror stale, so `SimilaritySearch` returned old vector scores while `Get` returned new doc bodies; (2) `DatabaseService::backfillIntoDocStore` walked every doc into LMDB but never set `_meta.schema_version=2`, so every boot re-backfilled the entire dataset (Zoe-scale ~1.85M docs = minutes of pre-READY blocking on every restart); (3) `SimilaritySearch` LMDB handler returned `INTERNAL` instead of `INVALID_ARGUMENT` when an unrelated throw (e.g. bad_alloc) fired after a dimension-mismatch was already observed in the scan. Exposes `smartbotic::db::storage::mark_migration_complete(env)` for the backfill marker. Tests 12/12 green.
|
|
- **v2.2.1 bug fixes from shadowman code review.** Three issues found by a code-review pass on the v2.0→v2.2.0 commit range, all now fixed: (1) `MemoryStore::updateIfVersion` was missing `extractVector` + `storeVector` + `mirrorVectorToDocStore` calls — optimistic-lock updates on vector collections left the vector mirror stale, so `SimilaritySearch` returned old vector scores while `Get` returned new doc bodies; (2) `DatabaseService::backfillIntoDocStore` walked every doc into LMDB but never set `_meta.schema_version=2`, so every boot re-backfilled the entire dataset (Zoe-scale ~1.85M docs = minutes of pre-READY blocking on every restart); (3) `SimilaritySearch` LMDB handler returned `INTERNAL` instead of `INVALID_ARGUMENT` when an unrelated throw (e.g. bad_alloc) fired after a dimension-mismatch was already observed in the scan. Exposes `smartbotic::db::storage::mark_migration_complete(env)` for the backfill marker. Tests 12/12 green.
|
|
|
- **v2.2.0 — default timestamp precision flipped to `"ns"`.** BREAKING for fresh collections. `CollectionCfg::timestampPrecision` default went from `"ms"` to `"ns"` so rapid-write collections (chat messages, metrics events, tool-call audits) get nanosecond stamps and strict ordering without needing an explicit `configureCollection()` call. Closes shadowman audit B5. Operators who need ms can still set it explicitly. Existing collections that already have a stored config record keep their value — only collections created post-upgrade WITHOUT an explicit `configureCollection()` inherit the new default. Mixed-precision rows in the same collection are read-correct via the 10^15 magnitude auto-detect; operators wanting clean rows should run `migrateCollectionTimestamps`. **`update()` docstring** updated to clarify: it OVERWRITES the doc; use `patch()` for partial mutations. Reflects shadowman audit B6.
|
|
- **v2.2.0 — default timestamp precision flipped to `"ns"`.** BREAKING for fresh collections. `CollectionCfg::timestampPrecision` default went from `"ms"` to `"ns"` so rapid-write collections (chat messages, metrics events, tool-call audits) get nanosecond stamps and strict ordering without needing an explicit `configureCollection()` call. Closes shadowman audit B5. Operators who need ms can still set it explicitly. Existing collections that already have a stored config record keep their value — only collections created post-upgrade WITHOUT an explicit `configureCollection()` inherit the new default. Mixed-precision rows in the same collection are read-correct via the 10^15 magnitude auto-detect; operators wanting clean rows should run `migrateCollectionTimestamps`. **`update()` docstring** updated to clarify: it OVERWRITES the doc; use `patch()` for partial mutations. Reflects shadowman audit B6.
|