صفحهٔ اصلی گشت‌و‌گذار راهنما
ورود
fszontagh
/
smartbotic-database
1
0
انشعاب 0
پرونده‌ها مشکلات 0 درخواست واکشی 0 ویکی
درخت: v2.7.1
شاخه‌ها تگ‌ها
smartbotic-data...
/
docs
/
superpowers
/
plans
/
2026-08-08-project-scoped-files-v2.6.0.md

2026-08-08-project-scoped-files-v2.6.0.md 16 KB
لینک دائمی تاريخچه خام

Project-Scoped Files (v2.6.0) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Give file storage a project dimension so files are namespaced like collections, which is the prerequisite for per-project file access policies.

Architecture: FileManager::FileInfo gains a project field persisted in the _files system collection. Every file RPC carries project, and the client fills it from Config::project automatically. Lookups by id verify the file's project matches the request, so an id from another namespace is NOT_FOUND rather than readable. Blob dedup stays global (disk saving) but the deduplicated flag is computed per-project so it cannot act as a cross-project existence oracle. Existing files migrate to default on first boot, idempotently.

Tech Stack: C++20, gRPC/protobuf 3.21 (proto3 field presence available), nlohmann/json, LMDB, spdlog.

Global Constraints

  • Reserved/empty project resolves to default — same rule as parseProjectCollection in service/src/project_addressing.hpp.
  • Project names validated by the existing isValidProjectName(); no new validator.
  • Wire and on-disk JSON stay JSON; no binary format change.
  • Never use em-dashes or en-dashes in any output, including code comments and commit messages. ASCII hyphens only.
  • SOVERSION stays 2. Client API changes are source-breaking, so consumers rebuild; note it in CLAUDE.md.
  • Existing _files documents have no project key. Absent means default on read, never "unknown".

Task 1: project on FileInfo, persisted and defaulted

Files:

  • Modify: service/src/files/file_manager.hpp:28-41 (add field to FileInfo)
  • Modify: service/src/files/file_manager.cpp (serialise/deserialise project, default on read)
  • Test: tests/test_file_project_scope.cpp (create)
  • Modify: tests/CMakeLists.txt (register target)

Interfaces:

  • Consumes: nothing.

  • Produces: FileManager::FileInfo::project (std::string, defaults to "default"); FileManager::listFiles(...) gains a leading const std::string& project parameter; FileManager::getFileInfo(id) unchanged, FileManager::getFileInfoIn(project, id) added returning std::nullopt on project mismatch.

  • [ ] Step 1: Write the failing test

    void test_fileinfo_persists_project() {
TmpFm fm;   // fixture boots a FileManager over a temp dir
FileManager::FileInfo in;
in.name = "a.bin"; in.fileType = "document"; in.project = "acme";
auto res = fm->storeFile({1,2,3}, in);

auto got = fm->getFileInfo(res.id);
check(got.has_value(), "file readable by id");
check(got->project == "acme", "project round-trips through _files");
}

void test_absent_project_reads_as_default() {
// A pre-2.6.0 record: metadata document with no "project" key at all.
TmpFm fm;
fm.writeLegacyRecord("legacy-1", R"({"name":"old.bin","fileType":"document"})");
auto got = fm->getFileInfo("legacy-1");
check(got.has_value(), "legacy record still readable");
check(got->project == "default", "absent project means default, not empty");
}

  • [ ] Step 2: Run test to verify it fails

Run: cmake --build build -j$(nproc) --target test_file_project_scope && ./build/tests/test_file_project_scope Expected: FAIL — FileInfo has no member project.

  • Step 3: Write minimal implementation

Add to FileInfo:

// v2.6.0 — owning project. Files are namespaced like collections so that
// per-project access policy has a referent. Records written before 2.6.0
// have no project key; absent means "default", never unknown.
std::string project = "default";

In the _files serialiser add j["project"] = info.project;, and in the deserialiser info.project = j.value("project", std::string("default"));.

  • Step 4: Run test to verify it passes

Run: ./build/tests/test_file_project_scope Expected: PASS

  • [ ] Step 5: Commit

    git add service/src/files/file_manager.hpp service/src/files/file_manager.cpp tests/test_file_project_scope.cpp tests/CMakeLists.txt
git commit -m "feat(files): persist an owning project on file metadata"

Task 2: project-scoped lookup, listing, and delete

Files:

  • Modify: service/src/files/file_manager.hpp (signatures)
  • Modify: service/src/files/file_manager.cpp (filter by project)
  • Test: tests/test_file_project_scope.cpp (extend)

Interfaces:

  • Consumes: FileInfo::project from Task 1.

  • Produces: getFileInfoIn(project, id) -> std::optional<FileInfo>; readFileIn(project, id) -> std::vector<uint8_t> (throws on mismatch); deleteFileIn(project, id) -> bool; listFiles(project, fileType, relatedId, limit, offset, checksum, name).

  • [ ] Step 1: Write the failing test

    void test_cross_project_id_is_not_readable() {
TmpFm fm;
FileInfo a; a.name="a"; a.fileType="document"; a.project="acme";
auto id = fm->storeFile({9,9,9}, a).id;

check(fm->getFileInfoIn("acme", id).has_value(), "owner project reads it");
check(!fm->getFileInfoIn("other", id).has_value(),
      "an id from another project must not resolve");
bool threw = false;
try { fm->readFileIn("other", id); } catch (const std::exception&) { threw = true; }
check(threw, "reading bytes across projects must throw, not return data");
check(!fm->deleteFileIn("other", id), "delete across projects must not delete");
check(fm->getFileInfoIn("acme", id).has_value(), "and the file survives that attempt");
}

void test_list_is_project_filtered() {
TmpFm fm;
FileInfo a; a.name="a"; a.fileType="document"; a.project="acme";  fm->storeFile({1}, a);
FileInfo b; b.name="b"; b.fileType="document"; b.project="other"; fm->storeFile({2}, b);

check(fm->listFiles("acme").totalCount == 1, "list shows only the caller's project");
check(fm->listFiles("acme").files[0].name == "a", "and the right file");
check(fm->listFiles("other").totalCount == 1, "peer project sees only its own");
}

  • [ ] Step 2: Run test to verify it fails

Run: cmake --build build -j$(nproc) --target test_file_project_scope && ./build/tests/test_file_project_scope Expected: FAIL — getFileInfoIn not declared.

  • Step 3: Write minimal implementation

Add the three *In overloads, each resolving via the existing id path then comparing info.project to the requested project and refusing on mismatch. listFiles gains project as its first parameter and filters on it before the other predicates. Keep the old getFileInfo(id) as an operator-facing, project-blind accessor and comment it as such.

  • Step 4: Run test to verify it passes

Run: ./build/tests/test_file_project_scope Expected: PASS

  • [ ] Step 5: Commit

    git add service/src/files/file_manager.hpp service/src/files/file_manager.cpp tests/test_file_project_scope.cpp
git commit -m "feat(files): scope lookup, list and delete by project"

Task 3: per-project deduplicated flag

Files:

  • Modify: service/src/files/file_manager.cpp (storeFile)
  • Test: tests/test_file_project_scope.cpp (extend)

Interfaces:

  • Consumes: listFiles(project, ...) from Task 2.

  • Produces: StoreResult::deduplicated now means "this project already referenced this checksum".

  • [ ] Step 1: Write the failing test

    void test_dedup_flag_does_not_leak_across_projects() {
TmpFm fm;
std::vector<uint8_t> bytes{7,7,7,7};
FileInfo a; a.name="a"; a.fileType="document"; a.project="acme";
auto r1 = fm->storeFile(bytes, a);
check(!r1.deduplicated, "first upload in acme is not a dedup hit");

FileInfo b; b.name="b"; b.fileType="document"; b.project="other";
auto r2 = fm->storeFile(bytes, b);
check(!r2.deduplicated,
      "same bytes from another project must NOT report deduplicated - "
      "that would be a cross-project existence oracle");

FileInfo c; c.name="c"; c.fileType="document"; c.project="acme";
auto r3 = fm->storeFile(bytes, c);
check(r3.deduplicated, "a repeat within the SAME project does report dedup");

// Blob is still shared on disk - the saving is retained.
check(fm->getRefCount(r1.checksum) == 3, "one blob, three references");
}

  • [ ] Step 2: Run test to verify it fails

Run: ./build/tests/test_file_project_scope Expected: FAIL — r2.deduplicated is true (global dedup reported verbatim).

  • Step 3: Write minimal implementation

Keep the global blob write/refcount exactly as-is. Compute the returned flag from a project-scoped metadata lookup instead of the blob's prior existence:

// v2.6.0 — dedup stays global so the disk saving is kept, but the flag we
// report is per-project. Reporting the global hit would tell project A that
// project B already holds these exact bytes.
const bool seen_in_project =
    listFiles(info.project, "", "", 1, 0, checksum).totalCount > 0;
result.deduplicated = seen_in_project;
  • Step 4: Run test to verify it passes

Run: ./build/tests/test_file_project_scope Expected: PASS

  • [ ] Step 5: Commit

    git add service/src/files/file_manager.cpp tests/test_file_project_scope.cpp
git commit -m "fix(files): compute deduplicated per project, not globally"

Task 4: project on the wire

Files:

  • Modify: proto/database.proto:545-606 (FileMetadata, DownloadFileRequest, DeleteFileRequest, GetFileInfoRequest, FileInfo, ListFilesRequest)
  • Modify: service/src/database_grpc_impl.cpp (5 file handlers)

Interfaces:

  • Consumes: the *In overloads from Task 2.

  • Produces: project field on all six messages; handlers resolve empty to default.

  • [ ] Step 1: Add the fields

    // v2.6.0 — owning project. Empty means "default", matching bare collection
// names. Files are namespaced so per-project access policy has a referent.
string project = 9;   // FileMetadata
string project = 2;   // DownloadFileRequest, DeleteFileRequest, GetFileInfoRequest
string project = 12;  // FileInfo
string project = 7;   // ListFilesRequest

  • [ ] Step 2: Route the handlers through the project-scoped calls

UploadFile copies metadata.project() (or default) into FileInfo::project. DownloadFile / DeleteFile / GetFileInfo call the *In overloads and map a nullopt/false result to NOT_FOUND. ListFiles passes the project through. FileInfo responses set project.

  • Step 3: Build

Run: cmake --build build -j$(nproc) Expected: no errors.

  • [ ] Step 4: Commit

    git add proto/database.proto service/src/database_grpc_impl.cpp
git commit -m "feat(files): carry project on every file RPC"

Task 5: client fills project automatically

Files:

  • Modify: client/src/client.cpp (uploadFile, downloadFile, deleteFile, getFileInfo, listFiles)
  • Modify: client/include/smartbotic/database/client.hpp (FileUploadMeta, file info struct gains project)
  • Test: tests/load_test/test_client_namespacing.cpp (extend), tests/load_test/test_client_namespacing.sh

Interfaces:

  • Consumes: proto project fields from Task 4.

  • Produces: file calls default to Config::project; a caller may still pass an explicit project to address another namespace.

  • [ ] Step 1: Write the failing test (append to test_client_namespacing.cpp)

    // Files must be namespaced like collections.
Client a({.address="127.0.0.1:9011", .project="proj1"});
Client b({.address="127.0.0.1:9011", .project="proj2"});
a.connect(); b.connect();

Client::FileUploadMeta m; m.name="secret.bin"; m.fileType="document";
auto up = a.uploadFile({1,2,3}, m);
ck(!up.id.empty(), "upload succeeded");
ck(a.getFileInfo(up.id).has_value(), "owner project reads its own file");
ck(!b.getFileInfo(up.id).has_value(), "peer project cannot resolve the id");
ck(b.listFiles("document").files.empty(), "peer project lists nothing");
ck(a.listFiles("document").files.size() == 1, "owner lists its own file");
ck(!b.deleteFile(up.id), "peer project cannot delete it");
ck(a.getFileInfo(up.id).has_value(), "file survives the cross-project delete");

  • [ ] Step 2: Run to verify it fails

Run: bash tests/load_test/test_client_namespacing.sh Expected: FAIL — peer project can currently read and delete the file.

  • Step 3: Set project on every outgoing file request

Mirror the qualify() convention: files use a separate project field rather than a project:name string, so set request.set_project(config_.project) in each of the five methods and read project back into the client-side struct.

  • Step 4: Run to verify it passes

Run: bash tests/load_test/test_client_namespacing.sh Expected: PASS, assertion count rises from 17 to 24.

  • [ ] Step 5: Commit

    git add client proto tests/load_test/test_client_namespacing.cpp
git commit -m "feat(files): client fills project on file RPCs"

Task 6: migrate existing files to default

Files:

  • Modify: service/src/database_service.cpp (call after recovery, beside auditSubdbPlacement())
  • Modify: service/src/database_service.hpp (declare migrateFilesToDefaultProject())
  • Test: tests/test_file_project_scope.cpp (extend)

Interfaces:

  • Consumes: FileInfo::project default from Task 1.

  • Produces: void DatabaseService::migrateFilesToDefaultProject() — idempotent, logs a count, never fails startup.

  • [ ] Step 1: Write the failing test

    void test_migration_stamps_default_and_is_idempotent() {
TmpFm fm;
fm.writeLegacyRecord("l1", R"({"name":"a","fileType":"document"})");
fm.writeLegacyRecord("l2", R"({"name":"b","fileType":"plugin"})");

check(fm.stampProjects() == 2, "first pass stamps both legacy records");
check(fm.rawRecord("l1").contains("project"), "project key now present on disk");
check(fm.stampProjects() == 0, "second pass is a no-op - idempotent");
check(fm->getFileInfoIn("default", "l1").has_value(), "readable under default");
}

  • [ ] Step 2: Run to verify it fails

Run: ./build/tests/test_file_project_scope Expected: FAIL — no stamping routine exists.

  • Step 3: Implement the migration

Walk _files, and for any document with no project key write default and save. Return the number stamped. Call it from initialize() after recovery. Log at INFO with the count, or DEBUG when zero. Advisory: catch and warn, never throw — a file-metadata stamp is not a reason to refuse service.

  • Step 4: Run to verify it passes

Run: ./build/tests/test_file_project_scope Expected: PASS

  • [ ] Step 5: Commit

    git add service/src/database_service.cpp service/src/database_service.hpp tests/test_file_project_scope.cpp
git commit -m "feat(files): stamp legacy file records with the default project"

Task 7: release v2.6.0

Files:

  • Modify: VERSION, CLAUDE.md, docs/integration-guide.md, README.md

  • [ ] Step 1: Full verification

    cmake --build build -j$(nproc)
ctest --test-dir build/tests --output-on-failure
bash tests/load_test/test_client_namespacing.sh
bash tests/load_test/test_views_multiproject.sh
bash tests/load_test/test_v24_tls_auth.sh

Expected: ctest all pass, namespacing 24/24, views ALL CHECKS PASSED, TLS 4/4.

  • Step 2: Bump and document

VERSION to 2.6.0. Add a CLAUDE.md entry covering: the breaking client change, the per-project deduplicated semantics and why, the legacy stamping migration, and that cross-project ids now return NOT_FOUND. Document the project field on files in the integration guide's file-storage section.

  • [ ] Step 3: Commit, tag, push

    git add -A
git commit -m "release(v2.6.0): project-scoped file storage"
git tag -a v2.6.0 -m "v2.6.0: project-scoped file storage"
git push origin main && git push origin v2.6.0

  • [ ] Step 4: Build and install locally

    ./packaging/build.sh --local
sudo dpkg -i dist/local/libsmartbotic-db-client_2.6.0-1_amd64.deb \
         dist/local/libsmartbotic-db-client-dev_2.6.0-1_amd64.deb \
         dist/local/smartbotic-database_2.6.0-1_amd64.deb \
         dist/local/smartbotic-db-cli_2.6.0-1_amd64.deb

Install all four together: downgrading or upgrading the server alone fails its dependency on the client lib and leaves the service dead.

  • Step 5: Verify the upgrade on the live instance

Check systemctl is-active, --version reports 2.6.0, zero [error] lines since restart, the file stamping count appears in the journal, and existing documents still read. Do NOT publish to the apt repo without asking.