|
@@ -0,0 +1,149 @@
|
|
|
|
|
+# v2.3 multi-project plan
|
|
|
|
|
+
|
|
|
|
|
+**Goal:** every smartbotic-database instance hosts multiple projects, each
|
|
|
|
|
+with its own namespace of collections. Backward-compatible by construction
|
|
|
|
|
+— a v2.2 client that doesn't address a project transparently uses
|
|
|
|
|
+`"default"`, which auto-migrates from the existing `<dataDir>/env/` on
|
|
|
|
|
+upgrade.
|
|
|
|
|
+
|
|
|
|
|
+## Settled design (from operator decisions today)
|
|
|
|
|
+
|
|
|
|
|
+| Axis | Decision |
|
|
|
|
|
+|------|----------|
|
|
|
|
|
+| Namespace name | `project` |
|
|
|
|
|
+| Wire shape | Collection name encodes project: `"my_app:users"` parses to `("my_app","users")`; bare `"users"` → `("default","users")` |
|
|
|
|
|
+| Separator | `:` (single, only one allowed) |
|
|
|
|
|
+| Storage layout | One LMDB env per project at `<dataDir>/projects/<name>/env/` |
|
|
|
|
|
+| Upgrade path | First v2.3 boot moves `<dataDir>/env/` → `<dataDir>/projects/default/env/` atomically |
|
|
|
|
|
+| Toggle / first-boot mode | None. Multi-project is always on. v2.2 clients without explicit project see identical behaviour via the default project. |
|
|
|
|
|
+| Project codebase rename | Deferred. v2.x arc is supposed to freeze surface area. |
|
|
|
|
|
+
|
|
|
|
|
+Parser semantics locked in by `tests/test_project_addressing.cpp` (23
|
|
|
|
|
+assertions). Strict on edge cases — leading/trailing/double `:` all
|
|
|
|
|
+throw `std::invalid_argument`.
|
|
|
|
|
+
|
|
|
|
|
+## Reserved names
|
|
|
|
|
+
|
|
|
|
|
+- `default`: always present, addressable. Idempotent on create.
|
|
|
|
|
+- `_-prefix`: reserved for system use (matches collection-name rules).
|
|
|
|
|
+- Length: 1..63 chars, `^[a-zA-Z_][a-zA-Z0-9_-]{0,62}$`.
|
|
|
|
|
+
|
|
|
|
|
+## Stage breakdown
|
|
|
|
|
+
|
|
|
|
|
+Each stage ends at a buildable, testable, committable state.
|
|
|
|
|
+
|
|
|
|
|
+### Stage A — Foundation: parser + addressing types ✅ DONE
|
|
|
|
|
+
|
|
|
|
|
+- `service/src/project_addressing.hpp` — `parseProjectCollection`,
|
|
|
|
|
+ `isValidProjectName`, constants.
|
|
|
|
|
+- `tests/test_project_addressing.cpp` — 23 assertions cover the wire-form
|
|
|
|
|
+ edge cases + the name-validation regex boundary.
|
|
|
|
|
+
|
|
|
|
|
+### Stage B — Per-project storage layout
|
|
|
|
|
+
|
|
|
|
|
+- New `ProjectStore` type wrapping `(LmdbEnv, LmdbDocumentStore)` per
|
|
|
|
|
+ project.
|
|
|
|
|
+- `DatabaseService` holds `std::map<std::string, std::unique_ptr<ProjectStore>>`
|
|
|
|
|
+ instead of single `env_`/`doc_store_`.
|
|
|
|
|
+- Lazy-open semantics: first write to project `<X>` creates
|
|
|
|
|
+ `<dataDir>/projects/<X>/env/` and registers the store.
|
|
|
|
|
+- Eager-open at boot: scan `<dataDir>/projects/*/env/`, open each.
|
|
|
|
|
+- Default project always exists — created at boot if absent.
|
|
|
|
|
+
|
|
|
|
|
+Files touched: `database_service.cpp/.hpp`, possibly a new
|
|
|
|
|
+`storage/project_store.hpp`. Test: new
|
|
|
|
|
+`tests/test_project_store.cpp` exercising open/lazy-create/list.
|
|
|
|
|
+
|
|
|
|
|
+### Stage C — v2.2 → v2.3 storage migration
|
|
|
|
|
+
|
|
|
|
|
+- On boot: if `<dataDir>/env/` exists AND `<dataDir>/projects/default/env/`
|
|
|
|
|
+ does not, atomically rename the directory.
|
|
|
|
|
+- Idempotent: if both exist (shouldn't, but defensively) refuse to start
|
|
|
|
|
+ with a clear error.
|
|
|
|
|
+- Logs the move at INFO level for operator visibility.
|
|
|
|
|
+
|
|
|
|
|
+Files touched: `database_service.cpp` (boot path). Test: extend
|
|
|
|
|
+`test_project_store.cpp` with a v2.2-layout fixture.
|
|
|
|
|
+
|
|
|
|
|
+### Stage D — Handler routing through project
|
|
|
|
|
+
|
|
|
|
|
+- Every gRPC handler that receives `collection` parses via
|
|
|
|
|
+ `parseProjectCollection` and dispatches against the resolved project's
|
|
|
|
|
+ store. Same change ~12 sites in `database_grpc_impl.cpp`.
|
|
|
|
|
+- The MemoryStore-side mirror gets the qualified name `"project:collection"`
|
|
|
|
|
+ as its in-memory key, so concurrent reads see consistent state. The
|
|
|
|
|
+ LMDB-side mirror calls the resolved project's `doc_store_->put`.
|
|
|
|
|
+
|
|
|
|
|
+Files touched: `database_grpc_impl.cpp` (every handler), `memory_store.cpp`
|
|
|
|
|
+(qualified-key handling), `database_service.cpp` (mirror routing).
|
|
|
|
|
+
|
|
|
|
|
+### Stage E — Client API
|
|
|
|
|
+
|
|
|
|
|
+- Add `Client::Config::project` field (default `"default"`). When set,
|
|
|
|
|
+ the Client prepends `project + ":"` to every bare collection name in
|
|
|
|
|
+ outgoing requests. Explicit `"other:users"` overrides the sticky
|
|
|
|
|
+ default.
|
|
|
|
|
+- Documented examples in `client.hpp`.
|
|
|
|
|
+- Per-call form `client.find("other:users", ...)` always works without
|
|
|
|
|
+ setting `Config::project`.
|
|
|
|
|
+
|
|
|
|
|
+Files touched: `client/include/smartbotic/database/client.hpp`,
|
|
|
|
|
+`client/src/client.cpp`. Tests in the existing client tests (extend or
|
|
|
|
|
+add a new file).
|
|
|
|
|
+
|
|
|
|
|
+### Stage F — Project CRUD RPCs
|
|
|
|
|
+
|
|
|
|
|
+- `ListProjects()` — returns the set of open project names. Filesystem
|
|
|
|
|
+ scan-backed; trivial.
|
|
|
|
|
+- `CreateProject(name)` — validates name, ensures
|
|
|
|
|
+ `<dataDir>/projects/<name>/env/` exists. Idempotent. Useful so admins
|
|
|
|
|
+ can pre-create empty projects before the first write.
|
|
|
|
|
+- `DropProject(name)` — refuses if `name == "default"`; closes the env
|
|
|
|
|
+ and `rm -rf`s the dir. Operator-only.
|
|
|
|
|
+
|
|
|
|
|
+Files touched: `proto/database.proto` (3 new RPCs), service
|
|
|
|
|
+implementations, client wrappers.
|
|
|
|
|
+
|
|
|
|
|
+### Stage G — Tests, docs, release
|
|
|
|
|
+
|
|
|
|
|
+- `test_project_addressing` ✅
|
|
|
|
|
+- `test_project_store` — boot, lazy-create, list, drop, default-project
|
|
|
|
|
+ always present.
|
|
|
|
|
+- `test_v22_to_v23_migration` — synthetic v2.2 dataDir, boot, verify the
|
|
|
|
|
+ env moves to `projects/default/env/`.
|
|
|
|
|
+- Integration test that hits two projects from two Client instances
|
|
|
|
|
+ concurrently and verifies isolation.
|
|
|
|
|
+- CLAUDE.md update with the v2.3 entry + back-compat statement.
|
|
|
|
|
+- VERSION bump to 2.3.0.
|
|
|
|
|
+- Local + zeus + apt repo publish.
|
|
|
|
|
+
|
|
|
|
|
+## Out of scope for v2.3
|
|
|
|
|
+
|
|
|
|
|
+- Per-project memory budgets / quotas (could come in v2.3.x).
|
|
|
|
|
+- Per-project ACLs / auth (no auth in the wire today).
|
|
|
|
|
+- Project rename (project codebase rename — deferred).
|
|
|
|
|
+- Listing projects from the CLI tool (`smartbotic-db-cli projects`) —
|
|
|
|
|
+ can wait for v2.3.x.
|
|
|
|
|
+
|
|
|
|
|
+## Open questions
|
|
|
|
|
+
|
|
|
|
|
+1. **MemoryStore: per-project or global with qualified keys?** v2.3 plan
|
|
|
|
|
+ currently says global+qualified. Cheaper, simpler, no per-project
|
|
|
|
|
+ eviction tuning. Right call for v2.3 since MemoryStore is being
|
|
|
|
|
+ decommissioned in v2.4 anyway.
|
|
|
|
|
+2. **Per-project mapsize override in config?** Add a `projects.<name>.mapsize_mb`
|
|
|
|
|
+ schema, or just take a global default? Global default for v2.3; per-project
|
|
|
|
|
+ tuning in v2.3.x if real pressure shows up.
|
|
|
|
|
+3. **Operator visibility for "you typed `:` in your collection name in
|
|
|
|
|
+ single-project mode"?** Now moot since there's no single-project
|
|
|
|
|
+ mode — but if a v2.2 client accidentally sends `"my_app:users"`, the
|
|
|
|
|
+ server will route to project `"my_app"`, NOT to a collection named
|
|
|
|
|
+ literally `"my_app:users"`. Could be surprising. Mitigation: every
|
|
|
|
|
+ v2.2 client should NOT be sending `:` in collection names today, so
|
|
|
|
|
+ the surprise window is "nobody actually does this." OK to ignore.
|
|
|
|
|
+
|
|
|
|
|
+## Estimated effort
|
|
|
|
|
+
|
|
|
|
|
+Stages A-G total: ~1-2 days of focused work. The biggest two are Stage D
|
|
|
|
|
+(handler routing — many sites, all formulaic) and Stage F (proto + RPC
|
|
|
|
|
+wiring). Stage B + C are the meatiest design pieces.
|