Parcourir la source

v2.3 Stage A: project addressing parser + plan

First step toward v2.3 multi-project support. Lays the foundation so
Stages B-G (per-project storage, handler routing, migration, client API,
project CRUD, release) can build on a tested addressing primitive
without coordination on semantics.

## What lands

- `service/src/project_addressing.hpp` — header-only.
  - `parseProjectCollection(input)` returns `{project, collection}`.
    Unqualified "users" → ("default", "users"). Qualified "my_app:users"
    → ("my_app", "users"). Strict on edge cases: leading colon, trailing
    colon, double colon, empty input all throw `std::invalid_argument`.
  - `isValidProjectName(name)` enforces `^[a-zA-Z_][a-zA-Z0-9_-]{0,62}$`
    AND rejects `_-prefix` (reserved for system).
  - Constants: `kDefaultProject = "default"`, `kProjectSeparator = ':'`,
    `kMaxProjectNameLen = 63`.

- `tests/test_project_addressing.cpp` — 23 assertions across 9 cases.
  Locks in the wire-form edge semantics; every downstream layer can
  build against them without re-deciding.

- `docs/superpowers/plans/2026-05-16-v2.3-multi-project.md` — Stage
  breakdown (A-G), settled decisions table, out-of-scope notes,
  estimated effort. Operator-readable plan for the rest of v2.3.

## Design decisions made today

- Separator is `:` (single, only one allowed). `my_app:users` parses;
  `a:b:c` errors.
- Default project name is `default`. Always present, auto-migrated from
  v2.2 layout on first boot, addressable like any other project.
- No first-boot toggle. Multi-project is always on in v2.3+. v2.2
  clients without explicit project addressing observe the same data
  they always did, via the default project.
- Storage layout: one LMDB env per project at
  `<dataDir>/projects/<name>/env/`. Real isolation; per-project backup
  becomes a directory copy.
- Project codebase rename: deferred. v2.x arc keeps the
  `smartbotic-database` name.

## Out of session

Stages B-G remain. Estimated 1-2 days of focused work — see the plan
doc for the breakdown. Tracking todos #34-#39 carry the remainder.

Build: `test_project_addressing` 23/23 passes. Full suite untouched.
fszontagh il y a 2 mois
Parent
commit
41c858a419

+ 149 - 0
docs/superpowers/plans/2026-05-16-v2.3-multi-project.md

@@ -0,0 +1,149 @@
+# v2.3 multi-project plan
+
+**Goal:** every smartbotic-database instance hosts multiple projects, each
+with its own namespace of collections. Backward-compatible by construction
+— a v2.2 client that doesn't address a project transparently uses
+`"default"`, which auto-migrates from the existing `<dataDir>/env/` on
+upgrade.
+
+## Settled design (from operator decisions today)
+
+| Axis | Decision |
+|------|----------|
+| Namespace name | `project` |
+| Wire shape | Collection name encodes project: `"my_app:users"` parses to `("my_app","users")`; bare `"users"` → `("default","users")` |
+| Separator | `:` (single, only one allowed) |
+| Storage layout | One LMDB env per project at `<dataDir>/projects/<name>/env/` |
+| Upgrade path | First v2.3 boot moves `<dataDir>/env/` → `<dataDir>/projects/default/env/` atomically |
+| Toggle / first-boot mode | None. Multi-project is always on. v2.2 clients without explicit project see identical behaviour via the default project. |
+| Project codebase rename | Deferred. v2.x arc is supposed to freeze surface area. |
+
+Parser semantics locked in by `tests/test_project_addressing.cpp` (23
+assertions). Strict on edge cases — leading/trailing/double `:` all
+throw `std::invalid_argument`.
+
+## Reserved names
+
+- `default`: always present, addressable. Idempotent on create.
+- `_-prefix`: reserved for system use (matches collection-name rules).
+- Length: 1..63 chars, `^[a-zA-Z_][a-zA-Z0-9_-]{0,62}$`.
+
+## Stage breakdown
+
+Each stage ends at a buildable, testable, committable state.
+
+### Stage A — Foundation: parser + addressing types ✅ DONE
+
+- `service/src/project_addressing.hpp` — `parseProjectCollection`,
+  `isValidProjectName`, constants.
+- `tests/test_project_addressing.cpp` — 23 assertions cover the wire-form
+  edge cases + the name-validation regex boundary.
+
+### Stage B — Per-project storage layout
+
+- New `ProjectStore` type wrapping `(LmdbEnv, LmdbDocumentStore)` per
+  project.
+- `DatabaseService` holds `std::map<std::string, std::unique_ptr<ProjectStore>>`
+  instead of single `env_`/`doc_store_`.
+- Lazy-open semantics: first write to project `<X>` creates
+  `<dataDir>/projects/<X>/env/` and registers the store.
+- Eager-open at boot: scan `<dataDir>/projects/*/env/`, open each.
+- Default project always exists — created at boot if absent.
+
+Files touched: `database_service.cpp/.hpp`, possibly a new
+`storage/project_store.hpp`. Test: new
+`tests/test_project_store.cpp` exercising open/lazy-create/list.
+
+### Stage C — v2.2 → v2.3 storage migration
+
+- On boot: if `<dataDir>/env/` exists AND `<dataDir>/projects/default/env/`
+  does not, atomically rename the directory.
+- Idempotent: if both exist (shouldn't, but defensively) refuse to start
+  with a clear error.
+- Logs the move at INFO level for operator visibility.
+
+Files touched: `database_service.cpp` (boot path). Test: extend
+`test_project_store.cpp` with a v2.2-layout fixture.
+
+### Stage D — Handler routing through project
+
+- Every gRPC handler that receives `collection` parses via
+  `parseProjectCollection` and dispatches against the resolved project's
+  store. Same change ~12 sites in `database_grpc_impl.cpp`.
+- The MemoryStore-side mirror gets the qualified name `"project:collection"`
+  as its in-memory key, so concurrent reads see consistent state. The
+  LMDB-side mirror calls the resolved project's `doc_store_->put`.
+
+Files touched: `database_grpc_impl.cpp` (every handler), `memory_store.cpp`
+(qualified-key handling), `database_service.cpp` (mirror routing).
+
+### Stage E — Client API
+
+- Add `Client::Config::project` field (default `"default"`). When set,
+  the Client prepends `project + ":"` to every bare collection name in
+  outgoing requests. Explicit `"other:users"` overrides the sticky
+  default.
+- Documented examples in `client.hpp`.
+- Per-call form `client.find("other:users", ...)` always works without
+  setting `Config::project`.
+
+Files touched: `client/include/smartbotic/database/client.hpp`,
+`client/src/client.cpp`. Tests in the existing client tests (extend or
+add a new file).
+
+### Stage F — Project CRUD RPCs
+
+- `ListProjects()` — returns the set of open project names. Filesystem
+  scan-backed; trivial.
+- `CreateProject(name)` — validates name, ensures
+  `<dataDir>/projects/<name>/env/` exists. Idempotent. Useful so admins
+  can pre-create empty projects before the first write.
+- `DropProject(name)` — refuses if `name == "default"`; closes the env
+  and `rm -rf`s the dir. Operator-only.
+
+Files touched: `proto/database.proto` (3 new RPCs), service
+implementations, client wrappers.
+
+### Stage G — Tests, docs, release
+
+- `test_project_addressing` ✅
+- `test_project_store` — boot, lazy-create, list, drop, default-project
+  always present.
+- `test_v22_to_v23_migration` — synthetic v2.2 dataDir, boot, verify the
+  env moves to `projects/default/env/`.
+- Integration test that hits two projects from two Client instances
+  concurrently and verifies isolation.
+- CLAUDE.md update with the v2.3 entry + back-compat statement.
+- VERSION bump to 2.3.0.
+- Local + zeus + apt repo publish.
+
+## Out of scope for v2.3
+
+- Per-project memory budgets / quotas (could come in v2.3.x).
+- Per-project ACLs / auth (no auth in the wire today).
+- Project rename (project codebase rename — deferred).
+- Listing projects from the CLI tool (`smartbotic-db-cli projects`) —
+  can wait for v2.3.x.
+
+## Open questions
+
+1. **MemoryStore: per-project or global with qualified keys?** v2.3 plan
+   currently says global+qualified. Cheaper, simpler, no per-project
+   eviction tuning. Right call for v2.3 since MemoryStore is being
+   decommissioned in v2.4 anyway.
+2. **Per-project mapsize override in config?** Add a `projects.<name>.mapsize_mb`
+   schema, or just take a global default? Global default for v2.3; per-project
+   tuning in v2.3.x if real pressure shows up.
+3. **Operator visibility for "you typed `:` in your collection name in
+   single-project mode"?** Now moot since there's no single-project
+   mode — but if a v2.2 client accidentally sends `"my_app:users"`, the
+   server will route to project `"my_app"`, NOT to a collection named
+   literally `"my_app:users"`. Could be surprising. Mitigation: every
+   v2.2 client should NOT be sending `:` in collection names today, so
+   the surprise window is "nobody actually does this." OK to ignore.
+
+## Estimated effort
+
+Stages A-G total: ~1-2 days of focused work. The biggest two are Stage D
+(handler routing — many sites, all formulaic) and Stage F (proto + RPC
+wiring). Stage B + C are the meatiest design pieces.

+ 102 - 0
service/src/project_addressing.hpp

@@ -0,0 +1,102 @@
+// v2.3 — project addressing.
+//
+// Collections live inside a "project" namespace. On the wire, an
+// unqualified name like "users" resolves to ("default", "users"); an
+// explicit name like "my_app:users" resolves to ("my_app", "users").
+// The default project is auto-created on first boot and is the v2.2
+// back-compat path: every existing client that doesn't set a project
+// addresses "default" and observes the same data it always did.
+//
+// Storage maps one project → one LmdbEnv at <dataDir>/projects/<name>/env/.
+// gRPC handlers parse the input here and dispatch the rest of the call
+// against the resolved project store.
+//
+// Constraints (mirrors the collection-name rules so the joined wire form
+// stays unambiguous):
+//   - project name:    ^[a-zA-Z_][a-zA-Z0-9_-]{0,62}$  (1..63 chars)
+//   - collection name: same rules as v2.2 (already enforced upstream)
+//   - separator is a single ':' (configurable below if we ever need to)
+//
+// Reserved project names:
+//   - "default" is always present and addressable; recreating it via
+//     CreateProject is a no-op (idempotent).
+//   - Names starting with '_' are reserved for system use, same as
+//     collection sub-db names.
+
+#pragma once
+
+#include <stdexcept>
+#include <string>
+#include <string_view>
+
+namespace smartbotic::database {
+
+constexpr const char* kDefaultProject = "default";
+constexpr char kProjectSeparator = ':';
+constexpr size_t kMaxProjectNameLen = 63;
+
+struct ProjectCollection {
+    std::string project;
+    std::string collection;
+};
+
+// Edge-case semantics (chosen to fail loud rather than silently coerce):
+//   ""              -> std::invalid_argument
+//   "users"         -> ("default", "users")
+//   "my_app:users"  -> ("my_app",  "users")
+//   ":users"        -> std::invalid_argument (empty project before ':' is a
+//                       user error — they typed the separator intentionally)
+//   "my_app:"       -> std::invalid_argument (empty collection)
+//   "a:b:c"         -> std::invalid_argument (multiple separators)
+//
+// The returned `project` is guaranteed non-empty and the `collection` is
+// guaranteed non-empty. Project-name validation is NOT done here — the
+// caller calls isValidProjectName() at create-time, so reads of an
+// invalid-but-existing project name still succeed via the parser (the
+// invalid name would have failed validation at create time).
+inline ProjectCollection parseProjectCollection(std::string_view input) {
+    if (input.empty()) {
+        throw std::invalid_argument("project-collection: empty input");
+    }
+    const size_t pos = input.find(kProjectSeparator);
+    if (pos == std::string_view::npos) {
+        return {kDefaultProject, std::string(input)};
+    }
+    if (pos == 0) {
+        throw std::invalid_argument(
+            "project-collection: ':' separator at position 0 (empty project) — "
+            "use an unqualified name to address the default project");
+    }
+    if (pos == input.size() - 1) {
+        throw std::invalid_argument(
+            "project-collection: empty collection after ':' separator");
+    }
+    // Reject multiple separators — ambiguous.
+    if (input.find(kProjectSeparator, pos + 1) != std::string_view::npos) {
+        throw std::invalid_argument(
+            "project-collection: only one ':' separator allowed");
+    }
+    return {std::string(input.substr(0, pos)),
+            std::string(input.substr(pos + 1))};
+}
+
+// True iff `name` matches the project-name regex AND is not a reserved
+// system name. Used by CreateProject + the lazy first-write path.
+inline bool isValidProjectName(std::string_view name) {
+    if (name.empty() || name.size() > kMaxProjectNameLen) return false;
+    if (name[0] == '_') return false;  // reserved
+    auto valid_first = [](char c) {
+        return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c == '_';
+    };
+    auto valid_rest = [](char c) {
+        return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
+            || (c >= '0' && c <= '9') || c == '_' || c == '-';
+    };
+    if (!valid_first(name[0])) return false;
+    for (size_t i = 1; i < name.size(); ++i) {
+        if (!valid_rest(name[i])) return false;
+    }
+    return true;
+}
+
+}  // namespace smartbotic::database

+ 7 - 0
tests/CMakeLists.txt

@@ -485,3 +485,10 @@ add_test(NAME lmdb_env COMMAND test_lmdb_env)
 add_test(NAME document_store COMMAND test_document_store)
 add_test(NAME migrate_v1_to_v2 COMMAND test_migrate_v1_to_v2)
 add_test(NAME dual_write_mirror COMMAND test_dual_write_mirror)
+
+# v2.3 — project addressing (header-only parser; no link deps).
+add_executable(test_project_addressing test_project_addressing.cpp)
+target_include_directories(test_project_addressing PRIVATE
+    ${CMAKE_CURRENT_SOURCE_DIR}/../service/src
+)
+add_test(NAME project_addressing COMMAND test_project_addressing)

+ 127 - 0
tests/test_project_addressing.cpp

@@ -0,0 +1,127 @@
+// v2.3 — project-addressing parser tests.
+//
+// Locks in the edge-case semantics. Anything that relies on the parser
+// (gRPC handlers, mirror routing, client) builds on these guarantees.
+
+#include <cstdlib>
+#include <iostream>
+#include <stdexcept>
+
+#include "project_addressing.hpp"
+
+using smartbotic::database::kDefaultProject;
+using smartbotic::database::ProjectCollection;
+using smartbotic::database::isValidProjectName;
+using smartbotic::database::parseProjectCollection;
+
+namespace {
+
+int g_pass = 0;
+int g_fail = 0;
+
+void check(bool cond, const char* msg) {
+    if (cond) ++g_pass;
+    else { ++g_fail; std::cerr << "FAIL: " << msg << "\n"; }
+}
+
+template <typename Fn>
+void check_throws(Fn fn, const char* msg) {
+    try {
+        fn();
+        ++g_fail;
+        std::cerr << "FAIL (did not throw): " << msg << "\n";
+    } catch (const std::invalid_argument&) {
+        ++g_pass;
+    } catch (...) {
+        ++g_fail;
+        std::cerr << "FAIL (wrong exception type): " << msg << "\n";
+    }
+}
+
+void test_unqualified_resolves_to_default() {
+    auto r = parseProjectCollection("users");
+    check(r.project == kDefaultProject, "unqualified: project == default");
+    check(r.collection == "users", "unqualified: collection preserved");
+}
+
+void test_qualified_splits_on_separator() {
+    auto r = parseProjectCollection("my_app:users");
+    check(r.project == "my_app", "qualified: project parsed");
+    check(r.collection == "users", "qualified: collection parsed");
+}
+
+void test_long_collection_name() {
+    auto r = parseProjectCollection("my_app:some_long_collection_name");
+    check(r.collection == "some_long_collection_name",
+          "qualified: long collection preserved");
+}
+
+void test_empty_input_throws() {
+    check_throws([] { parseProjectCollection(""); },
+                 "empty input must throw");
+}
+
+void test_leading_colon_throws() {
+    check_throws([] { parseProjectCollection(":users"); },
+                 "leading ':' must throw (empty project)");
+}
+
+void test_trailing_colon_throws() {
+    check_throws([] { parseProjectCollection("my_app:"); },
+                 "trailing ':' must throw (empty collection)");
+}
+
+void test_double_separator_throws() {
+    check_throws([] { parseProjectCollection("a:b:c"); },
+                 "double ':' must throw (ambiguous)");
+}
+
+void test_just_colon_throws() {
+    check_throws([] { parseProjectCollection(":"); },
+                 "':' alone must throw");
+}
+
+void test_valid_project_names() {
+    check(isValidProjectName("default"), "'default' is valid");
+    check(isValidProjectName("my_app"), "'my_app' is valid");
+    check(isValidProjectName("App1"), "'App1' is valid");
+    check(isValidProjectName("a-b-c"), "'a-b-c' is valid (dash allowed)");
+    check(isValidProjectName("_starts_underscore") == false,
+          "'_-prefix is reserved (system)");
+    check(isValidProjectName("") == false, "empty is invalid");
+    check(isValidProjectName("9starts_digit") == false,
+          "digit-start is invalid");
+    check(isValidProjectName("has:colon") == false,
+          "':' inside name is invalid");
+    check(isValidProjectName("has/slash") == false,
+          "'/' inside name is invalid (filesystem safety)");
+    check(isValidProjectName("has.dot") == false,
+          "'.' inside name is invalid");
+    check(isValidProjectName("has space") == false,
+          "whitespace inside name is invalid");
+
+    std::string too_long(64, 'a');
+    check(isValidProjectName(too_long) == false,
+          "64+ chars exceeds kMaxProjectNameLen");
+
+    std::string max_len(63, 'a');
+    check(isValidProjectName(max_len), "63 chars is the boundary");
+}
+
+}  // namespace
+
+int main() {
+    test_unqualified_resolves_to_default();
+    test_qualified_splits_on_separator();
+    test_long_collection_name();
+    test_empty_input_throws();
+    test_leading_colon_throws();
+    test_trailing_colon_throws();
+    test_double_separator_throws();
+    test_just_colon_throws();
+    test_valid_project_names();
+
+    std::cout << "project_addressing: " << g_pass << " passed, "
+              << g_fail << " failed\n";
+    return g_fail == 0 ? 0 : 1;
+}