# Project-Scoped Files (v2.6.0) Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Give file storage a `project` dimension so files are namespaced like collections, which is the prerequisite for per-project file access policies.

**Architecture:** `FileManager::FileInfo` gains a `project` field persisted in the `_files` system collection. Every file RPC carries `project`, and the client fills it from `Config::project` automatically. Lookups by id verify the file's project matches the request, so an id from another namespace is `NOT_FOUND` rather than readable. Blob dedup stays global (disk saving) but the `deduplicated` flag is computed per-project so it cannot act as a cross-project existence oracle. Existing files migrate to `default` on first boot, idempotently.

**Tech Stack:** C++20, gRPC/protobuf 3.21 (proto3 field presence available), nlohmann/json, LMDB, spdlog.

## Global Constraints

- Reserved/empty project resolves to `default` — same rule as `parseProjectCollection` in `service/src/project_addressing.hpp`.
- Project names validated by the existing `isValidProjectName()`; no new validator.
- Wire and on-disk JSON stay JSON; no binary format change.
- Never use em-dashes or en-dashes in any output, including code comments and commit messages. ASCII hyphens only.
- `SOVERSION` stays `2`. Client API changes are source-breaking, so consumers rebuild; note it in `CLAUDE.md`.
- Existing `_files` documents have no `project` key. Absent means `default` on read, never "unknown".

---

### Task 1: `project` on FileInfo, persisted and defaulted

**Files:**
- Modify: `service/src/files/file_manager.hpp:28-41` (add field to `FileInfo`)
- Modify: `service/src/files/file_manager.cpp` (serialise/deserialise `project`, default on read)
- Test: `tests/test_file_project_scope.cpp` (create)
- Modify: `tests/CMakeLists.txt` (register target)

**Interfaces:**
- Consumes: nothing.
- Produces: `FileManager::FileInfo::project` (`std::string`, defaults to `"default"`); `FileManager::listFiles(...)` gains a leading `const std::string& project` parameter; `FileManager::getFileInfo(id)` unchanged, `FileManager::getFileInfoIn(project, id)` added returning `std::nullopt` on project mismatch.

- [ ] **Step 1: Write the failing test**

```cpp
void test_fileinfo_persists_project() {
    TmpFm fm;   // fixture boots a FileManager over a temp dir
    FileManager::FileInfo in;
    in.name = "a.bin"; in.fileType = "document"; in.project = "acme";
    auto res = fm->storeFile({1,2,3}, in);

    auto got = fm->getFileInfo(res.id);
    check(got.has_value(), "file readable by id");
    check(got->project == "acme", "project round-trips through _files");
}

void test_absent_project_reads_as_default() {
    // A pre-2.6.0 record: metadata document with no "project" key at all.
    TmpFm fm;
    fm.writeLegacyRecord("legacy-1", R"({"name":"old.bin","fileType":"document"})");
    auto got = fm->getFileInfo("legacy-1");
    check(got.has_value(), "legacy record still readable");
    check(got->project == "default", "absent project means default, not empty");
}
```

- [ ] **Step 2: Run test to verify it fails**

Run: `cmake --build build -j$(nproc) --target test_file_project_scope && ./build/tests/test_file_project_scope`
Expected: FAIL — `FileInfo` has no member `project`.

- [ ] **Step 3: Write minimal implementation**

Add to `FileInfo`:

```cpp
// v2.6.0 — owning project. Files are namespaced like collections so that
// per-project access policy has a referent. Records written before 2.6.0
// have no project key; absent means "default", never unknown.
std::string project = "default";
```

In the `_files` serialiser add `j["project"] = info.project;`, and in the
deserialiser `info.project = j.value("project", std::string("default"));`.

- [ ] **Step 4: Run test to verify it passes**

Run: `./build/tests/test_file_project_scope`
Expected: PASS

- [ ] **Step 5: Commit**

```bash
git add service/src/files/file_manager.hpp service/src/files/file_manager.cpp tests/test_file_project_scope.cpp tests/CMakeLists.txt
git commit -m "feat(files): persist an owning project on file metadata"
```

---

### Task 2: project-scoped lookup, listing, and delete

**Files:**
- Modify: `service/src/files/file_manager.hpp` (signatures)
- Modify: `service/src/files/file_manager.cpp` (filter by project)
- Test: `tests/test_file_project_scope.cpp` (extend)

**Interfaces:**
- Consumes: `FileInfo::project` from Task 1.
- Produces: `getFileInfoIn(project, id) -> std::optional<FileInfo>`; `readFileIn(project, id) -> std::vector<uint8_t>` (throws on mismatch); `deleteFileIn(project, id) -> bool`; `listFiles(project, fileType, relatedId, limit, offset, checksum, name)`.

- [ ] **Step 1: Write the failing test**

```cpp
void test_cross_project_id_is_not_readable() {
    TmpFm fm;
    FileInfo a; a.name="a"; a.fileType="document"; a.project="acme";
    auto id = fm->storeFile({9,9,9}, a).id;

    check(fm->getFileInfoIn("acme", id).has_value(), "owner project reads it");
    check(!fm->getFileInfoIn("other", id).has_value(),
          "an id from another project must not resolve");
    bool threw = false;
    try { fm->readFileIn("other", id); } catch (const std::exception&) { threw = true; }
    check(threw, "reading bytes across projects must throw, not return data");
    check(!fm->deleteFileIn("other", id), "delete across projects must not delete");
    check(fm->getFileInfoIn("acme", id).has_value(), "and the file survives that attempt");
}

void test_list_is_project_filtered() {
    TmpFm fm;
    FileInfo a; a.name="a"; a.fileType="document"; a.project="acme";  fm->storeFile({1}, a);
    FileInfo b; b.name="b"; b.fileType="document"; b.project="other"; fm->storeFile({2}, b);

    check(fm->listFiles("acme").totalCount == 1, "list shows only the caller's project");
    check(fm->listFiles("acme").files[0].name == "a", "and the right file");
    check(fm->listFiles("other").totalCount == 1, "peer project sees only its own");
}
```

- [ ] **Step 2: Run test to verify it fails**

Run: `cmake --build build -j$(nproc) --target test_file_project_scope && ./build/tests/test_file_project_scope`
Expected: FAIL — `getFileInfoIn` not declared.

- [ ] **Step 3: Write minimal implementation**

Add the three `*In` overloads, each resolving via the existing id path then
comparing `info.project` to the requested project and refusing on mismatch.
`listFiles` gains `project` as its first parameter and filters on it before the
other predicates. Keep the old `getFileInfo(id)` as an operator-facing,
project-blind accessor and comment it as such.

- [ ] **Step 4: Run test to verify it passes**

Run: `./build/tests/test_file_project_scope`
Expected: PASS

- [ ] **Step 5: Commit**

```bash
git add service/src/files/file_manager.hpp service/src/files/file_manager.cpp tests/test_file_project_scope.cpp
git commit -m "feat(files): scope lookup, list and delete by project"
```

---

### Task 3: per-project `deduplicated` flag

**Files:**
- Modify: `service/src/files/file_manager.cpp` (`storeFile`)
- Test: `tests/test_file_project_scope.cpp` (extend)

**Interfaces:**
- Consumes: `listFiles(project, ...)` from Task 2.
- Produces: `StoreResult::deduplicated` now means "this project already referenced this checksum".

- [ ] **Step 1: Write the failing test**

```cpp
void test_dedup_flag_does_not_leak_across_projects() {
    TmpFm fm;
    std::vector<uint8_t> bytes{7,7,7,7};
    FileInfo a; a.name="a"; a.fileType="document"; a.project="acme";
    auto r1 = fm->storeFile(bytes, a);
    check(!r1.deduplicated, "first upload in acme is not a dedup hit");

    FileInfo b; b.name="b"; b.fileType="document"; b.project="other";
    auto r2 = fm->storeFile(bytes, b);
    check(!r2.deduplicated,
          "same bytes from another project must NOT report deduplicated - "
          "that would be a cross-project existence oracle");

    FileInfo c; c.name="c"; c.fileType="document"; c.project="acme";
    auto r3 = fm->storeFile(bytes, c);
    check(r3.deduplicated, "a repeat within the SAME project does report dedup");

    // Blob is still shared on disk - the saving is retained.
    check(fm->getRefCount(r1.checksum) == 3, "one blob, three references");
}
```

- [ ] **Step 2: Run test to verify it fails**

Run: `./build/tests/test_file_project_scope`
Expected: FAIL — `r2.deduplicated` is true (global dedup reported verbatim).

- [ ] **Step 3: Write minimal implementation**

Keep the global blob write/refcount exactly as-is. Compute the returned flag
from a project-scoped metadata lookup instead of the blob's prior existence:

```cpp
// v2.6.0 — dedup stays global so the disk saving is kept, but the flag we
// report is per-project. Reporting the global hit would tell project A that
// project B already holds these exact bytes.
const bool seen_in_project =
    listFiles(info.project, "", "", 1, 0, checksum).totalCount > 0;
result.deduplicated = seen_in_project;
```

- [ ] **Step 4: Run test to verify it passes**

Run: `./build/tests/test_file_project_scope`
Expected: PASS

- [ ] **Step 5: Commit**

```bash
git add service/src/files/file_manager.cpp tests/test_file_project_scope.cpp
git commit -m "fix(files): compute deduplicated per project, not globally"
```

---

### Task 4: `project` on the wire

**Files:**
- Modify: `proto/database.proto:545-606` (`FileMetadata`, `DownloadFileRequest`, `DeleteFileRequest`, `GetFileInfoRequest`, `FileInfo`, `ListFilesRequest`)
- Modify: `service/src/database_grpc_impl.cpp` (5 file handlers)

**Interfaces:**
- Consumes: the `*In` overloads from Task 2.
- Produces: `project` field on all six messages; handlers resolve empty to `default`.

- [ ] **Step 1: Add the fields**

```protobuf
// v2.6.0 — owning project. Empty means "default", matching bare collection
// names. Files are namespaced so per-project access policy has a referent.
string project = 9;   // FileMetadata
string project = 2;   // DownloadFileRequest, DeleteFileRequest, GetFileInfoRequest
string project = 12;  // FileInfo
string project = 7;   // ListFilesRequest
```

- [ ] **Step 2: Route the handlers through the project-scoped calls**

`UploadFile` copies `metadata.project()` (or `default`) into `FileInfo::project`.
`DownloadFile` / `DeleteFile` / `GetFileInfo` call the `*In` overloads and map a
`nullopt`/false result to `NOT_FOUND`. `ListFiles` passes the project through.
`FileInfo` responses set `project`.

- [ ] **Step 3: Build**

Run: `cmake --build build -j$(nproc)`
Expected: no errors.

- [ ] **Step 4: Commit**

```bash
git add proto/database.proto service/src/database_grpc_impl.cpp
git commit -m "feat(files): carry project on every file RPC"
```

---

### Task 5: client fills project automatically

**Files:**
- Modify: `client/src/client.cpp` (`uploadFile`, `downloadFile`, `deleteFile`, `getFileInfo`, `listFiles`)
- Modify: `client/include/smartbotic/database/client.hpp` (`FileUploadMeta`, file info struct gains `project`)
- Test: `tests/load_test/test_client_namespacing.cpp` (extend), `tests/load_test/test_client_namespacing.sh`

**Interfaces:**
- Consumes: proto `project` fields from Task 4.
- Produces: file calls default to `Config::project`; a caller may still pass an explicit project to address another namespace.

- [ ] **Step 1: Write the failing test** (append to `test_client_namespacing.cpp`)

```cpp
// Files must be namespaced like collections.
Client a({.address="127.0.0.1:9011", .project="proj1"});
Client b({.address="127.0.0.1:9011", .project="proj2"});
a.connect(); b.connect();

Client::FileUploadMeta m; m.name="secret.bin"; m.fileType="document";
auto up = a.uploadFile({1,2,3}, m);
ck(!up.id.empty(), "upload succeeded");
ck(a.getFileInfo(up.id).has_value(), "owner project reads its own file");
ck(!b.getFileInfo(up.id).has_value(), "peer project cannot resolve the id");
ck(b.listFiles("document").files.empty(), "peer project lists nothing");
ck(a.listFiles("document").files.size() == 1, "owner lists its own file");
ck(!b.deleteFile(up.id), "peer project cannot delete it");
ck(a.getFileInfo(up.id).has_value(), "file survives the cross-project delete");
```

- [ ] **Step 2: Run to verify it fails**

Run: `bash tests/load_test/test_client_namespacing.sh`
Expected: FAIL — peer project can currently read and delete the file.

- [ ] **Step 3: Set `project` on every outgoing file request**

Mirror the `qualify()` convention: files use a separate `project` field rather
than a `project:name` string, so set `request.set_project(config_.project)` in
each of the five methods and read `project` back into the client-side struct.

- [ ] **Step 4: Run to verify it passes**

Run: `bash tests/load_test/test_client_namespacing.sh`
Expected: PASS, assertion count rises from 17 to 24.

- [ ] **Step 5: Commit**

```bash
git add client proto tests/load_test/test_client_namespacing.cpp
git commit -m "feat(files): client fills project on file RPCs"
```

---

### Task 6: migrate existing files to `default`

**Files:**
- Modify: `service/src/database_service.cpp` (call after recovery, beside `auditSubdbPlacement()`)
- Modify: `service/src/database_service.hpp` (declare `migrateFilesToDefaultProject()`)
- Test: `tests/test_file_project_scope.cpp` (extend)

**Interfaces:**
- Consumes: `FileInfo::project` default from Task 1.
- Produces: `void DatabaseService::migrateFilesToDefaultProject()` — idempotent, logs a count, never fails startup.

- [ ] **Step 1: Write the failing test**

```cpp
void test_migration_stamps_default_and_is_idempotent() {
    TmpFm fm;
    fm.writeLegacyRecord("l1", R"({"name":"a","fileType":"document"})");
    fm.writeLegacyRecord("l2", R"({"name":"b","fileType":"plugin"})");

    check(fm.stampProjects() == 2, "first pass stamps both legacy records");
    check(fm.rawRecord("l1").contains("project"), "project key now present on disk");
    check(fm.stampProjects() == 0, "second pass is a no-op - idempotent");
    check(fm->getFileInfoIn("default", "l1").has_value(), "readable under default");
}
```

- [ ] **Step 2: Run to verify it fails**

Run: `./build/tests/test_file_project_scope`
Expected: FAIL — no stamping routine exists.

- [ ] **Step 3: Implement the migration**

Walk `_files`, and for any document with no `project` key write `default` and
save. Return the number stamped. Call it from `initialize()` after recovery.
Log at INFO with the count, or DEBUG when zero. Advisory: catch and warn, never
throw — a file-metadata stamp is not a reason to refuse service.

- [ ] **Step 4: Run to verify it passes**

Run: `./build/tests/test_file_project_scope`
Expected: PASS

- [ ] **Step 5: Commit**

```bash
git add service/src/database_service.cpp service/src/database_service.hpp tests/test_file_project_scope.cpp
git commit -m "feat(files): stamp legacy file records with the default project"
```

---

### Task 7: release v2.6.0

**Files:**
- Modify: `VERSION`, `CLAUDE.md`, `docs/integration-guide.md`, `README.md`

- [ ] **Step 1: Full verification**

```bash
cmake --build build -j$(nproc)
ctest --test-dir build/tests --output-on-failure
bash tests/load_test/test_client_namespacing.sh
bash tests/load_test/test_views_multiproject.sh
bash tests/load_test/test_v24_tls_auth.sh
```
Expected: ctest all pass, namespacing 24/24, views ALL CHECKS PASSED, TLS 4/4.

- [ ] **Step 2: Bump and document**

`VERSION` to `2.6.0`. Add a `CLAUDE.md` entry covering: the breaking client
change, the per-project `deduplicated` semantics and why, the legacy stamping
migration, and that cross-project ids now return `NOT_FOUND`. Document the
`project` field on files in the integration guide's file-storage section.

- [ ] **Step 3: Commit, tag, push**

```bash
git add -A
git commit -m "release(v2.6.0): project-scoped file storage"
git tag -a v2.6.0 -m "v2.6.0: project-scoped file storage"
git push origin main && git push origin v2.6.0
```

- [ ] **Step 4: Build and install locally**

```bash
./packaging/build.sh --local
sudo dpkg -i dist/local/libsmartbotic-db-client_2.6.0-1_amd64.deb \
             dist/local/libsmartbotic-db-client-dev_2.6.0-1_amd64.deb \
             dist/local/smartbotic-database_2.6.0-1_amd64.deb \
             dist/local/smartbotic-db-cli_2.6.0-1_amd64.deb
```

Install all four together: downgrading or upgrading the server alone fails its
dependency on the client lib and leaves the service dead.

- [ ] **Step 5: Verify the upgrade on the live instance**

Check `systemctl is-active`, `--version` reports 2.6.0, zero `[error]` lines
since restart, the file stamping count appears in the journal, and existing
documents still read. Do NOT publish to the apt repo without asking.