# Views (v1.5.0) Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add read-only views to smartbotic-database — named projections over collections that filter which fields are returned. Supports include/exclude lists with nested (dot-path) field selection. Views are stored in a `_views` system collection, created via migrations or direct RPC.

**Architecture:** Views are redirects with response-side projection. A `ViewManager` loads view definitions from the `_views` system collection. gRPC handlers for read operations (`Get`, `Find`, `Count`) check if the target name is a view — if so, they redirect to the real collection and apply projection on the response JSON. Write handlers (`Insert`, `Update`, `Patch`, `Upsert`, `Delete`) reject view names with a clear error. MemoryStore stays view-agnostic.

**Tech Stack:** C++20, nlohmann/json (already used), gRPC/Protobuf (existing service)

---

## Design Summary

### Include + exclude semantics (Option B)
- `include=[]` (empty) + `exclude=[x,y]` → return all fields except x, y
- `include=[a,b]` + `exclude=anything` → return only a, b (exclude ignored when include is non-empty)
- `include=[a,b]` + `exclude=[]` → return only a, b

### Nested paths
Dot-notation paths work through nested objects:
- `user.profile.name` → traverse `user`, then `profile`, then keep `name`
- Stop at arrays (don't descend into array elements for projection)
- Missing intermediate paths are silently skipped

### Metadata fields always returned
`_id`, `_version`, `_created_at`, `_updated_at`, `_created_by`, `_updated_by` are always returned regardless of view definition (needed for client identification and optimistic locking).

### Read-only
Writes to a view name (Insert/Update/Patch/Upsert/Delete) return `INVALID_ARGUMENT` with message "cannot write to view: views are read-only".

### No view-of-view
View's `collection` field must reference a real collection, not another view.

---

## File Structure

```
service/src/views/                                    # NEW
  view_manager.hpp                                    # ViewManager class
  view_manager.cpp
  projection.hpp                                      # applyProjection() helper
  projection.cpp

service/src/database_grpc_impl.hpp                    # Modified: add view RPCs
service/src/database_grpc_impl.cpp                    # Modified: integrate ViewManager into handlers

service/src/database_service.cpp                      # Modified: instantiate ViewManager
service/src/database_service.hpp                      # Modified: hold ViewManager reference

service/src/migrations/migration_runner.cpp           # Modified: add create_view op

client/include/smartbotic/database/client.hpp         # Modified: view API
client/src/client.cpp                                 # Modified: implement view RPCs

proto/database.proto                                  # Modified: 4 new RPCs + messages

tests/test_views.cpp                                  # NEW: integration test
tests/CMakeLists.txt                                  # Modified: add test target

VERSION                                               # Modified: 1.4.0 → 1.5.0
docs/integration-guide.md                             # Modified: Views section
CLAUDE.md                                             # Modified: note views feature
```

---

### Task 1: Bump version and add proto definitions

**Files:**
- Modify: `VERSION`
- Modify: `proto/database.proto`

- [ ] **Step 1: Bump VERSION**

```bash
echo "1.5.0" > /data/smartbotic-database/VERSION
```

- [ ] **Step 2: Add view RPCs to proto/database.proto**

Find the `DatabaseService` definition (around line 22) and add after the collection management RPCs (around line 58):

```protobuf
    // View management
    rpc CreateView(CreateViewRequest) returns (CreateViewResponse);
    rpc DropView(DropViewRequest) returns (DropViewResponse);
    rpc ListViews(ListViewsRequest) returns (ListViewsResponse);
    rpc GetViewInfo(GetViewInfoRequest) returns (GetViewInfoResponse);
```

- [ ] **Step 3: Add view message types**

Add at the end of `proto/database.proto` (before the final closing brace of any existing message):

```protobuf
// ===== View Operations =====

message ViewDefinition {
    string name = 1;                  // view name (globally unique)
    string collection = 2;            // target real collection
    repeated string include = 3;      // field paths to include (dot-notation)
    repeated string exclude = 4;      // field paths to exclude (dot-notation)
    uint64 created_at = 5;
    uint64 updated_at = 6;
}

message CreateViewRequest {
    string name = 1;
    string collection = 2;
    repeated string include = 3;
    repeated string exclude = 4;
}

message CreateViewResponse {
    bool success = 1;
    string error = 2;
}

message DropViewRequest {
    string name = 1;
}

message DropViewResponse {
    bool success = 1;
    string error = 2;
}

message ListViewsRequest {}

message ListViewsResponse {
    repeated ViewDefinition views = 1;
}

message GetViewInfoRequest {
    string name = 1;
}

message GetViewInfoResponse {
    ViewDefinition view = 1;
    bool found = 2;
}
```

- [ ] **Step 4: Build to verify proto compiles**

```bash
cd /data/smartbotic-database
cmake --build build -j$(nproc) 2>&1 | grep -E "error|Generating"
```

Expected: `Generating protobuf/gRPC code for database.proto` appears, no errors.

- [ ] **Step 5: Commit**

```bash
git add VERSION proto/database.proto
git commit -m "feat(proto): add view management RPCs and messages (v1.5.0)"
```

---

### Task 2: Create projection helper with nested path support

**Files:**
- Create: `service/src/views/projection.hpp`
- Create: `service/src/views/projection.cpp`

- [ ] **Step 1: Create projection.hpp**

```cpp
#pragma once

#include <nlohmann/json.hpp>
#include <string>
#include <vector>

namespace smartbotic::database {

/**
 * Apply projection (include/exclude) to a document.
 *
 * Semantics (Option B):
 *  - If include is non-empty: only those paths are kept (exclude is ignored)
 *  - If include is empty and exclude is non-empty: all fields kept except exclude paths
 *  - If both empty: document returned unchanged
 *
 * Paths use dot notation: "user.profile.name". Array traversal stops at arrays.
 * Missing intermediate paths are silently skipped.
 *
 * Metadata fields (_id, _version, _created_at, _updated_at, _created_by, _updated_by)
 * are always preserved regardless of include/exclude.
 */
nlohmann::json applyProjection(const nlohmann::json& document,
                                const std::vector<std::string>& include,
                                const std::vector<std::string>& exclude);

} // namespace smartbotic::database
```

- [ ] **Step 2: Create projection.cpp**

```cpp
#include "projection.hpp"

#include <algorithm>
#include <sstream>

namespace smartbotic::database {

namespace {

// Metadata fields that are always preserved
const std::vector<std::string> METADATA_FIELDS = {
    "_id", "_version", "_created_at", "_updated_at", "_created_by", "_updated_by"
};

// Split "user.profile.name" into ["user", "profile", "name"]
std::vector<std::string> splitPath(const std::string& path) {
    std::vector<std::string> parts;
    std::stringstream ss(path);
    std::string part;
    while (std::getline(ss, part, '.')) {
        if (!part.empty()) parts.push_back(part);
    }
    return parts;
}

// Recursively set src[path] into dst[path], walking nested objects
void copyPath(const nlohmann::json& src, nlohmann::json& dst,
              const std::vector<std::string>& parts, size_t idx) {
    if (idx >= parts.size()) return;
    const std::string& key = parts[idx];
    if (!src.is_object() || !src.contains(key)) return;

    if (idx == parts.size() - 1) {
        // Last segment — copy the value
        dst[key] = src[key];
        return;
    }

    // Intermediate segment — ensure dst[key] is an object, recurse
    if (!src[key].is_object()) {
        // Can't descend further into a non-object; just copy the leaf value
        dst[key] = src[key];
        return;
    }
    if (!dst.contains(key) || !dst[key].is_object()) {
        dst[key] = nlohmann::json::object();
    }
    copyPath(src[key], dst[key], parts, idx + 1);
}

// Recursively delete dst[path], walking nested objects
void deletePath(nlohmann::json& dst, const std::vector<std::string>& parts, size_t idx) {
    if (idx >= parts.size()) return;
    const std::string& key = parts[idx];
    if (!dst.is_object() || !dst.contains(key)) return;

    if (idx == parts.size() - 1) {
        dst.erase(key);
        return;
    }

    if (dst[key].is_object()) {
        deletePath(dst[key], parts, idx + 1);
    }
}

} // anonymous namespace

nlohmann::json applyProjection(const nlohmann::json& document,
                                const std::vector<std::string>& include,
                                const std::vector<std::string>& exclude) {
    // Empty include + empty exclude = no projection
    if (include.empty() && exclude.empty()) {
        return document;
    }

    if (!include.empty()) {
        // Include-only semantics: build result with only included paths
        nlohmann::json result = nlohmann::json::object();
        for (const auto& path : include) {
            auto parts = splitPath(path);
            if (parts.empty()) continue;
            copyPath(document, result, parts, 0);
        }
        // Always preserve metadata fields
        for (const auto& meta : METADATA_FIELDS) {
            if (document.contains(meta)) {
                result[meta] = document[meta];
            }
        }
        return result;
    }

    // Exclude-only semantics: copy document, delete excluded paths
    nlohmann::json result = document;
    for (const auto& path : exclude) {
        auto parts = splitPath(path);
        if (parts.empty()) continue;
        // Don't allow excluding metadata fields
        if (parts.size() == 1) {
            bool isMeta = false;
            for (const auto& meta : METADATA_FIELDS) {
                if (parts[0] == meta) { isMeta = true; break; }
            }
            if (isMeta) continue;
        }
        deletePath(result, parts, 0);
    }
    return result;
}

} // namespace smartbotic::database
```

- [ ] **Step 3: Add projection.cpp to service CMakeLists.txt**

Read `/data/smartbotic-database/service/CMakeLists.txt`. Find the `DATABASE_SERVICE_SOURCES` list. Add:

```cmake
    src/views/projection.cpp
```

between the existing sources (alphabetical order — between `replication/sync_protocol.cpp` and `migrations/migration_runner.cpp`, wherever it fits).

- [ ] **Step 4: Build to verify compilation**

```bash
cmake --build build -j$(nproc) 2>&1 | tail -5
```

Expected: Build succeeds, `projection.cpp.o` is built.

- [ ] **Step 5: Commit**

```bash
git add service/src/views/ service/CMakeLists.txt
git commit -m "feat(views): add projection helper with nested path support"
```

---

### Task 3: Create ViewManager

**Files:**
- Create: `service/src/views/view_manager.hpp`
- Create: `service/src/views/view_manager.cpp`

- [ ] **Step 1: Create view_manager.hpp**

```cpp
#pragma once

#include <memory>
#include <mutex>
#include <optional>
#include <shared_mutex>
#include <string>
#include <unordered_map>
#include <vector>

namespace smartbotic::database {

class MemoryStore;

struct ViewInfo {
    std::string name;
    std::string collection;
    std::vector<std::string> include;
    std::vector<std::string> exclude;
    uint64_t createdAt = 0;
    uint64_t updatedAt = 0;
};

/**
 * ViewManager — loads, caches, and manages view definitions.
 *
 * Views are persisted as documents in the `_views` system collection.
 * On startup, loadFromStore() populates an in-memory cache for O(1) lookup.
 * Mutations (createView/dropView) update both the store and the cache.
 */
class ViewManager {
public:
    static constexpr const char* SYSTEM_COLLECTION = "_views";

    explicit ViewManager(MemoryStore& store);

    /**
     * Load all view definitions from _views collection into the cache.
     * Called once at startup after MemoryStore is initialized.
     */
    void loadFromStore();

    /**
     * Create a view. Fails if name already exists or collection doesn't exist.
     * Returns false with errorOut populated on failure.
     */
    bool createView(const ViewInfo& view, std::string& errorOut);

    /**
     * Drop a view by name.
     */
    bool dropView(const std::string& name, std::string& errorOut);

    /**
     * O(1) check — is this name a view?
     */
    bool isView(const std::string& name) const;

    /**
     * Lookup view by name. Returns nullopt if not a view.
     */
    std::optional<ViewInfo> getView(const std::string& name) const;

    /**
     * List all views.
     */
    std::vector<ViewInfo> listViews() const;

private:
    MemoryStore& store_;
    mutable std::shared_mutex cacheMutex_;
    std::unordered_map<std::string, ViewInfo> cache_;
};

} // namespace smartbotic::database
```

- [ ] **Step 2: Create view_manager.cpp**

```cpp
#include "view_manager.hpp"

#include "../memory_store.hpp"
#include "../document.hpp"

#include <chrono>
#include <nlohmann/json.hpp>
#include <spdlog/spdlog.h>

namespace smartbotic::database {

namespace {
uint64_t nowMs() {
    return std::chrono::duration_cast<std::chrono::milliseconds>(
        std::chrono::system_clock::now().time_since_epoch()).count();
}

ViewInfo fromJson(const nlohmann::json& j) {
    ViewInfo v;
    v.name = j.value("name", "");
    v.collection = j.value("collection", "");
    if (j.contains("include") && j["include"].is_array()) {
        for (const auto& p : j["include"]) v.include.push_back(p.get<std::string>());
    }
    if (j.contains("exclude") && j["exclude"].is_array()) {
        for (const auto& p : j["exclude"]) v.exclude.push_back(p.get<std::string>());
    }
    v.createdAt = j.value("created_at", uint64_t{0});
    v.updatedAt = j.value("updated_at", uint64_t{0});
    return v;
}

nlohmann::json toJson(const ViewInfo& v) {
    return {
        {"name", v.name},
        {"collection", v.collection},
        {"include", v.include},
        {"exclude", v.exclude},
        {"created_at", v.createdAt},
        {"updated_at", v.updatedAt}
    };
}
} // anonymous namespace

ViewManager::ViewManager(MemoryStore& store) : store_(store) {}

void ViewManager::loadFromStore() {
    std::unique_lock<std::shared_mutex> lock(cacheMutex_);
    cache_.clear();

    // Ensure _views collection exists
    CollectionOptions opts;
    store_.createCollection(SYSTEM_COLLECTION, opts);

    Query q;
    auto result = store_.find(SYSTEM_COLLECTION, q);
    for (const auto& doc : result.documents) {
        ViewInfo v = fromJson(doc.data);
        if (v.name.empty()) continue;
        cache_[v.name] = v;
    }
    spdlog::info("ViewManager: loaded {} views from _views", cache_.size());
}

bool ViewManager::createView(const ViewInfo& view, std::string& errorOut) {
    if (view.name.empty()) {
        errorOut = "view name is required";
        return false;
    }
    if (view.collection.empty()) {
        errorOut = "target collection is required";
        return false;
    }
    if (view.name == view.collection) {
        errorOut = "view name cannot equal collection name";
        return false;
    }
    if (view.name.rfind("_", 0) == 0) {
        errorOut = "view name cannot start with '_' (reserved for system collections)";
        return false;
    }

    // No view-of-view
    {
        std::shared_lock<std::shared_mutex> rlock(cacheMutex_);
        if (cache_.contains(view.collection)) {
            errorOut = "target '" + view.collection + "' is a view — views can only target real collections";
            return false;
        }
        if (cache_.contains(view.name)) {
            errorOut = "view '" + view.name + "' already exists";
            return false;
        }
    }

    ViewInfo v = view;
    v.createdAt = nowMs();
    v.updatedAt = v.createdAt;

    // Persist to _views
    Document doc;
    doc.data = toJson(v);
    std::string id = store_.insert(SYSTEM_COLLECTION, v.name, doc);
    if (id.empty()) {
        errorOut = "failed to persist view definition";
        return false;
    }

    {
        std::unique_lock<std::shared_mutex> wlock(cacheMutex_);
        cache_[v.name] = v;
    }
    spdlog::info("ViewManager: created view '{}' over '{}'", v.name, v.collection);
    return true;
}

bool ViewManager::dropView(const std::string& name, std::string& errorOut) {
    {
        std::shared_lock<std::shared_mutex> rlock(cacheMutex_);
        if (!cache_.contains(name)) {
            errorOut = "view '" + name + "' does not exist";
            return false;
        }
    }

    bool removed = store_.remove(SYSTEM_COLLECTION, name);
    if (!removed) {
        errorOut = "failed to remove view from store";
        return false;
    }

    {
        std::unique_lock<std::shared_mutex> wlock(cacheMutex_);
        cache_.erase(name);
    }
    spdlog::info("ViewManager: dropped view '{}'", name);
    return true;
}

bool ViewManager::isView(const std::string& name) const {
    std::shared_lock<std::shared_mutex> lock(cacheMutex_);
    return cache_.contains(name);
}

std::optional<ViewInfo> ViewManager::getView(const std::string& name) const {
    std::shared_lock<std::shared_mutex> lock(cacheMutex_);
    auto it = cache_.find(name);
    if (it == cache_.end()) return std::nullopt;
    return it->second;
}

std::vector<ViewInfo> ViewManager::listViews() const {
    std::shared_lock<std::shared_mutex> lock(cacheMutex_);
    std::vector<ViewInfo> out;
    out.reserve(cache_.size());
    for (const auto& [_, v] : cache_) out.push_back(v);
    return out;
}

} // namespace smartbotic::database
```

- [ ] **Step 3: Add view_manager.cpp to CMakeLists.txt**

In `service/CMakeLists.txt`, `DATABASE_SERVICE_SOURCES` list, add:

```cmake
    src/views/view_manager.cpp
```

- [ ] **Step 4: Verify MemoryStore insert signature**

Read `service/src/memory_store.hpp` and confirm:
```cpp
std::string insert(const std::string& collection, const std::string& id, const Document& doc);
```

If the signature is different (e.g., different argument order or overloads), adjust the `store_.insert(...)` call in view_manager.cpp accordingly. Check `service/src/memory_store.cpp` for the exact implementation.

- [ ] **Step 5: Build**

```bash
cmake --build build -j$(nproc) 2>&1 | tail -5
```

Fix any compile errors that arise from MemoryStore API mismatches.

- [ ] **Step 6: Commit**

```bash
git add service/src/views/view_manager.hpp service/src/views/view_manager.cpp service/CMakeLists.txt
git commit -m "feat(views): add ViewManager with cached lookup and _views system collection"
```

---

### Task 4: Wire ViewManager into DatabaseService

**Files:**
- Modify: `service/src/database_service.hpp`
- Modify: `service/src/database_service.cpp`

- [ ] **Step 1: Read existing service structure**

```bash
cat /data/smartbotic-database/service/src/database_service.hpp
cat /data/smartbotic-database/service/src/database_service.cpp
```

Note how other managers (`PersistenceManager`, `EncryptionManager`, `EventManager`, `FileManager`, `ReplicationManager`, `MigrationRunner`) are instantiated and owned.

- [ ] **Step 2: Add ViewManager to DatabaseService**

In `service/src/database_service.hpp`:

1. Add include at the top:
```cpp
#include "views/view_manager.hpp"
```

2. In the class, after the other manager members (alongside `file_manager_`, `replication_manager_`, etc.), add:
```cpp
    ViewManager view_manager_;
```

3. Add a getter (near other getters):
```cpp
    ViewManager& viewManager() { return view_manager_; }
```

- [ ] **Step 3: Initialize ViewManager in constructor**

In `service/src/database_service.cpp`:

1. Find the DatabaseService constructor's member initializer list
2. Add `view_manager_(memory_store_)` alongside other manager initializations that take `memory_store_`
3. In the `start()` method (or wherever other managers load their state), after MemoryStore has loaded snapshots/WAL, add:

```cpp
    view_manager_.loadFromStore();
```

This must happen AFTER persistence has loaded existing documents (so views stored in `_views` are present in memory).

- [ ] **Step 4: Build**

```bash
cmake --build build -j$(nproc) 2>&1 | tail -5
```

- [ ] **Step 5: Run tests**

```bash
LD_LIBRARY_PATH=build/client ./build/tests/test_vector_storage
```

Expected: all tests pass (ViewManager loads empty cache when no views exist).

- [ ] **Step 6: Commit**

```bash
git add service/src/database_service.hpp service/src/database_service.cpp
git commit -m "feat(views): wire ViewManager into DatabaseService lifecycle"
```

---

### Task 5: Implement view RPC handlers

**Files:**
- Modify: `service/src/database_grpc_impl.hpp`
- Modify: `service/src/database_grpc_impl.cpp`

- [ ] **Step 1: Add ViewManager reference to DatabaseGrpcImpl**

In `service/src/database_grpc_impl.hpp`:

1. Add include:
```cpp
#include "views/view_manager.hpp"
```

2. Add constructor parameter and member. Find the existing constructor signature (it takes references to MemoryStore, PersistenceManager, etc.) and add a `ViewManager&` parameter. Add the matching `ViewManager& view_manager_;` member.

3. Declare the 4 new handler methods:
```cpp
    grpc::Status CreateView(grpc::ServerContext* context,
                            const pb::CreateViewRequest* request,
                            pb::CreateViewResponse* response) override;

    grpc::Status DropView(grpc::ServerContext* context,
                          const pb::DropViewRequest* request,
                          pb::DropViewResponse* response) override;

    grpc::Status ListViews(grpc::ServerContext* context,
                           const pb::ListViewsRequest* request,
                           pb::ListViewsResponse* response) override;

    grpc::Status GetViewInfo(grpc::ServerContext* context,
                             const pb::GetViewInfoRequest* request,
                             pb::GetViewInfoResponse* response) override;
```

- [ ] **Step 2: Update DatabaseGrpcImpl constructor in .cpp**

Find the constructor implementation in `database_grpc_impl.cpp`. Add `view_manager` to the parameter list and initializer list.

- [ ] **Step 3: Update DatabaseService to pass view_manager_**

In `service/src/database_service.cpp`, find where `DatabaseGrpcImpl` is constructed and add `view_manager_` to the argument list.

- [ ] **Step 4: Implement CreateView handler**

Add at the end of `database_grpc_impl.cpp`, before the closing namespace brace:

```cpp
grpc::Status DatabaseGrpcImpl::CreateView(
    grpc::ServerContext* /*context*/,
    const pb::CreateViewRequest* request,
    pb::CreateViewResponse* response
) {
    try {
        ViewInfo v;
        v.name = request->name();
        v.collection = request->collection();
        for (const auto& p : request->include()) v.include.push_back(p);
        for (const auto& p : request->exclude()) v.exclude.push_back(p);

        std::string err;
        bool ok = view_manager_.createView(v, err);
        response->set_success(ok);
        if (!ok) response->set_error(err);
        return grpc::Status::OK;
    } catch (const std::exception& e) {
        return grpc::Status(grpc::StatusCode::INTERNAL, e.what());
    }
}
```

- [ ] **Step 5: Implement DropView handler**

```cpp
grpc::Status DatabaseGrpcImpl::DropView(
    grpc::ServerContext* /*context*/,
    const pb::DropViewRequest* request,
    pb::DropViewResponse* response
) {
    try {
        std::string err;
        bool ok = view_manager_.dropView(request->name(), err);
        response->set_success(ok);
        if (!ok) response->set_error(err);
        return grpc::Status::OK;
    } catch (const std::exception& e) {
        return grpc::Status(grpc::StatusCode::INTERNAL, e.what());
    }
}
```

- [ ] **Step 6: Implement ListViews handler**

```cpp
grpc::Status DatabaseGrpcImpl::ListViews(
    grpc::ServerContext* /*context*/,
    const pb::ListViewsRequest* /*request*/,
    pb::ListViewsResponse* response
) {
    auto views = view_manager_.listViews();
    for (const auto& v : views) {
        auto* pb = response->add_views();
        pb->set_name(v.name);
        pb->set_collection(v.collection);
        for (const auto& p : v.include) pb->add_include(p);
        for (const auto& p : v.exclude) pb->add_exclude(p);
        pb->set_created_at(v.createdAt);
        pb->set_updated_at(v.updatedAt);
    }
    return grpc::Status::OK;
}
```

- [ ] **Step 7: Implement GetViewInfo handler**

```cpp
grpc::Status DatabaseGrpcImpl::GetViewInfo(
    grpc::ServerContext* /*context*/,
    const pb::GetViewInfoRequest* request,
    pb::GetViewInfoResponse* response
) {
    auto v = view_manager_.getView(request->name());
    if (!v) {
        response->set_found(false);
        return grpc::Status::OK;
    }
    auto* pb = response->mutable_view();
    pb->set_name(v->name);
    pb->set_collection(v->collection);
    for (const auto& p : v->include) pb->add_include(p);
    for (const auto& p : v->exclude) pb->add_exclude(p);
    pb->set_created_at(v->createdAt);
    pb->set_updated_at(v->updatedAt);
    response->set_found(true);
    return grpc::Status::OK;
}
```

- [ ] **Step 8: Build**

```bash
cmake --build build -j$(nproc) 2>&1 | tail -5
```

- [ ] **Step 9: Commit**

```bash
git add service/src/database_grpc_impl.hpp service/src/database_grpc_impl.cpp service/src/database_service.cpp
git commit -m "feat(views): implement CreateView/DropView/ListViews/GetViewInfo RPC handlers"
```

---

### Task 6: Integrate view projection into read handlers

**Files:**
- Modify: `service/src/database_grpc_impl.cpp`

Integrates the `isView` check into `Get`, `Find`, `Count` handlers. When the name is a view, redirects to the real collection and applies projection.

- [ ] **Step 1: Add projection include**

At the top of `database_grpc_impl.cpp`, add:

```cpp
#include "views/projection.hpp"
```

- [ ] **Step 2: Modify the Get handler**

Find the existing `Get` handler. At the start (after parameter validation), add:

```cpp
    std::string targetCollection = request->collection();
    std::optional<ViewInfo> view;
    if (view_manager_.isView(request->collection())) {
        view = view_manager_.getView(request->collection());
        targetCollection = view->collection;
    }
```

Then replace every use of `request->collection()` inside the Get handler with `targetCollection`.

Before writing the document data to the response, apply projection if this is a view. Find where the response document's data is set (look for `response->mutable_document()->set_data(...)`). Wrap the data like:

```cpp
    nlohmann::json docJson = nlohmann::json::parse(doc->data);
    if (view) {
        docJson = applyProjection(docJson, view->include, view->exclude);
    }
    response->mutable_document()->set_data(docJson.dump());
```

IMPORTANT: Read the existing Get handler carefully to understand how `doc->data` is used (it may be `bytes` / `std::string` already containing JSON-encoded data).

- [ ] **Step 3: Modify the Find handler**

Same pattern as Get. At the start:

```cpp
    std::string targetCollection = request->collection();
    std::optional<ViewInfo> view;
    if (view_manager_.isView(request->collection())) {
        view = view_manager_.getView(request->collection());
        targetCollection = view->collection;
    }
```

Replace `request->collection()` with `targetCollection` inside the handler.

In the loop that copies each document to the response, apply projection:

```cpp
    nlohmann::json docJson = nlohmann::json::parse(doc.data);
    if (view) {
        docJson = applyProjection(docJson, view->include, view->exclude);
    }
    pbDoc->set_data(docJson.dump());
```

- [ ] **Step 4: Modify the Count handler**

Count doesn't need projection (it returns a number), but should redirect views:

```cpp
    std::string targetCollection = request->collection();
    if (view_manager_.isView(request->collection())) {
        auto view = view_manager_.getView(request->collection());
        targetCollection = view->collection;
    }
```

Then use `targetCollection` in the store query.

- [ ] **Step 5: Build and test**

```bash
cmake --build build -j$(nproc) 2>&1 | tail -5
LD_LIBRARY_PATH=build/client ./build/tests/test_vector_storage
```

- [ ] **Step 6: Commit**

```bash
git add service/src/database_grpc_impl.cpp
git commit -m "feat(views): integrate projection into Get/Find/Count handlers"
```

---

### Task 7: Reject writes through views

**Files:**
- Modify: `service/src/database_grpc_impl.cpp`

- [ ] **Step 1: Add a helper**

Near the top of `database_grpc_impl.cpp`, add a helper function (above the class definition or as a static free function):

```cpp
namespace {
bool rejectIfView(const std::string& name, const ViewManager& vm,
                  auto* response /* has set_success + set_error */) {
    if (vm.isView(name)) {
        response->set_success(false);
        response->set_error("cannot write to view '" + name + "': views are read-only");
        return true;
    }
    return false;
}
} // anonymous namespace
```

Note: use templated lambda / auto to handle different response types. Alternatively, write four separate helpers or put the rejection inline in each handler — whichever is cleaner given existing patterns.

- [ ] **Step 2: Add rejection to Insert handler**

Find the `Insert` handler. At the start (after parsing the request), add:

```cpp
    if (view_manager_.isView(request->collection())) {
        return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
                           "cannot insert into view '" + request->collection() + "': views are read-only");
    }
```

- [ ] **Step 3: Add the same rejection to Update, Upsert, Delete, PatchDocument, BatchInsert, BatchDelete**

For each write handler, add the same check at the start. Note: for handlers whose response has `set_success()`/`set_error()` fields, you can return a "friendly" error in the response body. For handlers that return only grpc::Status, use `INVALID_ARGUMENT`.

Read each handler first to see the existing error-reporting style and match it.

- [ ] **Step 4: Build and test**

```bash
cmake --build build -j$(nproc) 2>&1 | tail -5
LD_LIBRARY_PATH=build/client ./build/tests/test_vector_storage
```

- [ ] **Step 5: Commit**

```bash
git add service/src/database_grpc_impl.cpp
git commit -m "feat(views): reject Insert/Update/Patch/Upsert/Delete on view names"
```

---

### Task 8: Add create_view migration op

**Files:**
- Modify: `service/src/migrations/migration_runner.hpp`
- Modify: `service/src/migrations/migration_runner.cpp`

- [ ] **Step 1: Wire ViewManager into MigrationRunner**

Read `service/src/migrations/migration_runner.hpp`. If it already takes a `MemoryStore&`, add a `ViewManager&` parameter. Add the member.

- [ ] **Step 2: Update construction site**

In `database_service.cpp`, find where `MigrationRunner` is constructed and pass `view_manager_`.

- [ ] **Step 3: Add create_view handler**

In `service/src/migrations/migration_runner.cpp`, find `applyOperation` (around line 209). Add a new branch (before the final `return false` / unknown operation):

```cpp
    if (type == "create_view") {
        std::string name = operation.value("name", "");
        std::string collection = operation.value("collection", "");
        if (name.empty() || collection.empty()) {
            spdlog::error("create_view: missing name or collection");
            return false;
        }

        ViewInfo v;
        v.name = name;
        v.collection = collection;
        if (operation.contains("include") && operation["include"].is_array()) {
            for (const auto& p : operation["include"]) {
                v.include.push_back(p.get<std::string>());
            }
        }
        if (operation.contains("exclude") && operation["exclude"].is_array()) {
            for (const auto& p : operation["exclude"]) {
                v.exclude.push_back(p.get<std::string>());
            }
        }

        std::string err;
        bool ok = view_manager_.createView(v, err);
        if (!ok) {
            // Treat "already exists" as success (idempotent migration)
            if (err.find("already exists") != std::string::npos) {
                spdlog::debug("create_view '{}': already exists (idempotent)", name);
                return true;
            }
            spdlog::error("create_view '{}': {}", name, err);
            return false;
        }
        spdlog::info("create_view '{}' over '{}': ok", name, collection);
        return true;
    }
```

Add include at the top of `migration_runner.cpp`:

```cpp
#include "../views/view_manager.hpp"
```

- [ ] **Step 4: Build**

```bash
cmake --build build -j$(nproc) 2>&1 | tail -5
```

- [ ] **Step 5: Commit**

```bash
git add service/src/migrations/migration_runner.hpp service/src/migrations/migration_runner.cpp service/src/database_service.cpp
git commit -m "feat(views): add create_view migration operation"
```

---

### Task 9: Add view methods to client library

**Files:**
- Modify: `client/include/smartbotic/database/client.hpp`
- Modify: `client/src/client.cpp`

- [ ] **Step 1: Add ViewDefinition struct and method declarations in client.hpp**

After the existing public methods (around the collection management section), add:

```cpp
    // ===== View Management =====

    struct ViewDefinition {
        std::string name;
        std::string collection;
        std::vector<std::string> include;
        std::vector<std::string> exclude;
        uint64_t createdAt = 0;
        uint64_t updatedAt = 0;
    };

    /**
     * Create a read-only view over a collection.
     * @param name View name (unique, cannot start with `_`)
     * @param collection Target collection (must be a real collection, not a view)
     * @param include Field paths to include (dot-notation supports nesting)
     * @param exclude Field paths to exclude (ignored if include is non-empty)
     * @return true on success, false otherwise (check server logs for details)
     */
    bool createView(const std::string& name, const std::string& collection,
                    const std::vector<std::string>& include = {},
                    const std::vector<std::string>& exclude = {});

    bool dropView(const std::string& name);
    [[nodiscard]] std::vector<ViewDefinition> listViews();
    [[nodiscard]] std::optional<ViewDefinition> getViewInfo(const std::string& name);
```

- [ ] **Step 2: Implement in client.cpp (Impl class)**

Inside the `Client::Impl` class, add:

```cpp
    bool createView(const std::string& name, const std::string& collection,
                    const std::vector<std::string>& include,
                    const std::vector<std::string>& exclude) {
        smartbotic::databasepb::CreateViewRequest request;
        request.set_name(name);
        request.set_collection(collection);
        for (const auto& p : include) request.add_include(p);
        for (const auto& p : exclude) request.add_exclude(p);

        smartbotic::databasepb::CreateViewResponse response;
        grpc::ClientContext context;
        setDeadline(context);

        auto status = stub_->CreateView(&context, request, &response);
        if (!status.ok()) {
            spdlog::error("Client::createView failed: {}", status.error_message());
            return false;
        }
        if (!response.success()) {
            spdlog::error("Client::createView rejected: {}", response.error());
            return false;
        }
        return true;
    }

    bool dropView(const std::string& name) {
        smartbotic::databasepb::DropViewRequest request;
        request.set_name(name);
        smartbotic::databasepb::DropViewResponse response;
        grpc::ClientContext context;
        setDeadline(context);

        auto status = stub_->DropView(&context, request, &response);
        if (!status.ok()) {
            spdlog::error("Client::dropView failed: {}", status.error_message());
            return false;
        }
        return response.success();
    }

    std::vector<Client::ViewDefinition> listViews() {
        smartbotic::databasepb::ListViewsRequest request;
        smartbotic::databasepb::ListViewsResponse response;
        grpc::ClientContext context;
        setDeadline(context);

        auto status = stub_->ListViews(&context, request, &response);
        std::vector<Client::ViewDefinition> out;
        if (!status.ok()) {
            spdlog::error("Client::listViews failed: {}", status.error_message());
            return out;
        }
        for (const auto& pb : response.views()) {
            Client::ViewDefinition v;
            v.name = pb.name();
            v.collection = pb.collection();
            for (const auto& p : pb.include()) v.include.push_back(p);
            for (const auto& p : pb.exclude()) v.exclude.push_back(p);
            v.createdAt = pb.created_at();
            v.updatedAt = pb.updated_at();
            out.push_back(std::move(v));
        }
        return out;
    }

    std::optional<Client::ViewDefinition> getViewInfo(const std::string& name) {
        smartbotic::databasepb::GetViewInfoRequest request;
        request.set_name(name);
        smartbotic::databasepb::GetViewInfoResponse response;
        grpc::ClientContext context;
        setDeadline(context);

        auto status = stub_->GetViewInfo(&context, request, &response);
        if (!status.ok() || !response.found()) {
            return std::nullopt;
        }
        Client::ViewDefinition v;
        v.name = response.view().name();
        v.collection = response.view().collection();
        for (const auto& p : response.view().include()) v.include.push_back(p);
        for (const auto& p : response.view().exclude()) v.exclude.push_back(p);
        v.createdAt = response.view().created_at();
        v.updatedAt = response.view().updated_at();
        return v;
    }
```

- [ ] **Step 3: Add public forwarding methods at the bottom of client.cpp**

Where the other `Client::xyz` public methods are defined (delegating to `impl_->xyz`), add:

```cpp
bool Client::createView(const std::string& name, const std::string& collection,
                        const std::vector<std::string>& include,
                        const std::vector<std::string>& exclude) {
    return impl_->createView(name, collection, include, exclude);
}

bool Client::dropView(const std::string& name) {
    return impl_->dropView(name);
}

std::vector<Client::ViewDefinition> Client::listViews() {
    return impl_->listViews();
}

std::optional<Client::ViewDefinition> Client::getViewInfo(const std::string& name) {
    return impl_->getViewInfo(name);
}
```

- [ ] **Step 4: Build**

```bash
cmake --build build -j$(nproc) 2>&1 | tail -5
```

- [ ] **Step 5: Commit**

```bash
git add client/include/smartbotic/database/client.hpp client/src/client.cpp
git commit -m "feat(views): add client API for view management"
```

---

### Task 10: Integration test

**Files:**
- Create: `tests/test_views.cpp`
- Modify: `tests/CMakeLists.txt`

- [ ] **Step 1: Read existing test structure**

```bash
cat /data/smartbotic-database/tests/test_vector_storage.cpp | head -80
cat /data/smartbotic-database/tests/CMakeLists.txt
```

Note the test harness pattern (how MemoryStore is constructed, how tests are invoked).

- [ ] **Step 2: Create tests/test_views.cpp**

Write a test that exercises `applyProjection` directly (doesn't need a running server). Follow the exact test pattern used in `test_vector_storage.cpp`:

```cpp
#include "../service/src/views/projection.hpp"

#include <cassert>
#include <iostream>
#include <nlohmann/json.hpp>

using smartbotic::database::applyProjection;

void test_empty_projection_returns_document_unchanged() {
    nlohmann::json doc = {{"name", "Alice"}, {"age", 30}};
    auto result = applyProjection(doc, {}, {});
    assert(result == doc);
    std::cout << "PASS: empty projection returns document unchanged\n";
}

void test_include_keeps_only_listed_fields() {
    nlohmann::json doc = {{"name", "Alice"}, {"email", "a@x.com"}, {"password", "secret"}};
    auto result = applyProjection(doc, {"name", "email"}, {});
    assert(result.contains("name"));
    assert(result.contains("email"));
    assert(!result.contains("password"));
    std::cout << "PASS: include keeps only listed fields\n";
}

void test_exclude_removes_fields() {
    nlohmann::json doc = {{"name", "Alice"}, {"email", "a@x.com"}, {"password", "secret"}};
    auto result = applyProjection(doc, {}, {"password"});
    assert(result.contains("name"));
    assert(result.contains("email"));
    assert(!result.contains("password"));
    std::cout << "PASS: exclude removes fields\n";
}

void test_include_takes_precedence_over_exclude() {
    nlohmann::json doc = {{"a", 1}, {"b", 2}, {"c", 3}};
    auto result = applyProjection(doc, {"a"}, {"b", "c"});
    assert(result.contains("a"));
    assert(!result.contains("b"));
    assert(!result.contains("c"));
    assert(result.size() == 1);
    std::cout << "PASS: include takes precedence over exclude\n";
}

void test_nested_include_path() {
    nlohmann::json doc = {
        {"user", {
            {"profile", {{"name", "Alice"}, {"bio", "hi"}}},
            {"private", {{"ssn", "123"}}}
        }}
    };
    auto result = applyProjection(doc, {"user.profile.name"}, {});
    assert(result["user"]["profile"]["name"] == "Alice");
    assert(!result["user"]["profile"].contains("bio"));
    assert(!result["user"].contains("private"));
    std::cout << "PASS: nested include path\n";
}

void test_nested_exclude_path() {
    nlohmann::json doc = {
        {"user", {
            {"name", "Alice"},
            {"private", {{"ssn", "123"}, {"email", "a@x.com"}}}
        }}
    };
    auto result = applyProjection(doc, {}, {"user.private.ssn"});
    assert(result["user"]["name"] == "Alice");
    assert(result["user"]["private"]["email"] == "a@x.com");
    assert(!result["user"]["private"].contains("ssn"));
    std::cout << "PASS: nested exclude path\n";
}

void test_metadata_always_preserved() {
    nlohmann::json doc = {
        {"_id", "doc-1"},
        {"_version", 5},
        {"name", "Alice"},
        {"secret", "xyz"}
    };
    auto result = applyProjection(doc, {"name"}, {});
    assert(result.contains("_id"));
    assert(result.contains("_version"));
    assert(result.contains("name"));
    assert(!result.contains("secret"));
    std::cout << "PASS: metadata always preserved\n";
}

void test_metadata_cannot_be_excluded() {
    nlohmann::json doc = {{"_id", "doc-1"}, {"_version", 5}, {"name", "Alice"}};
    auto result = applyProjection(doc, {}, {"_id", "_version"});
    assert(result.contains("_id"));
    assert(result.contains("_version"));
    std::cout << "PASS: metadata cannot be excluded\n";
}

void test_missing_path_silently_skipped() {
    nlohmann::json doc = {{"name", "Alice"}};
    auto result = applyProjection(doc, {"user.profile.name"}, {});
    // No crash, result just has metadata (none present) and nothing else
    assert(!result.contains("user"));
    std::cout << "PASS: missing path silently skipped\n";
}

int main() {
    test_empty_projection_returns_document_unchanged();
    test_include_keeps_only_listed_fields();
    test_exclude_removes_fields();
    test_include_takes_precedence_over_exclude();
    test_nested_include_path();
    test_nested_exclude_path();
    test_metadata_always_preserved();
    test_metadata_cannot_be_excluded();
    test_missing_path_silently_skipped();
    std::cout << "\nAll view projection tests PASSED!\n";
    return 0;
}
```

- [ ] **Step 3: Add to tests/CMakeLists.txt**

Read the existing CMakeLists.txt. Follow the same pattern used for `test_vector_storage` to add `test_views`:

```cmake
add_executable(test_views
    test_views.cpp
    ${CMAKE_SOURCE_DIR}/service/src/views/projection.cpp
)
target_include_directories(test_views PRIVATE
    ${CMAKE_SOURCE_DIR}/service/src
)
target_link_libraries(test_views PRIVATE
    smartbotic_db_proto
    nlohmann_json::nlohmann_json
)
```

Adjust include paths / link targets to match the existing `test_vector_storage` patterns.

- [ ] **Step 4: Build and run**

```bash
cmake --build build -j$(nproc) 2>&1 | tail -5
./build/tests/test_views
```

Expected: "All view projection tests PASSED!"

- [ ] **Step 5: Commit**

```bash
git add tests/test_views.cpp tests/CMakeLists.txt
git commit -m "test(views): add projection unit tests with nested path coverage"
```

---

### Task 11: Build and publish packages

- [ ] **Step 1: Local build**

```bash
./packaging/build.sh --local --skip-tests
```

Expected: `dist/local/*1.5.0*.deb` appear.

- [ ] **Step 2: Docker build + publish**

```bash
./packaging/build.sh --sync --suite trixie
```

Expected: `dist/debian13/*1.5.0*.deb` created, added to repo, synced to `repository.smartbotics.ai`.

- [ ] **Step 3: Verify**

```bash
ls /data/smartbotics-deb-repo/repo/pool/trixie/main/*1.5.0*
```

Expected: 4 files — smartbotic-database, libsmartbotic-db-client, libsmartbotic-db-client-dev, smartbotic-db-cli — all at 1.5.0-1.

---

### Task 12: Update documentation

**Files:**
- Modify: `docs/integration-guide.md`
- Modify: `CLAUDE.md`

- [ ] **Step 1: Add Views section to integration-guide.md**

Insert a new section after the "Document Operations" section, before "Querying":

```markdown
### Views

Views are named, read-only projections over collections. They filter which fields are returned when the view is queried. Views are useful for:

- **Schema evolution** — add fields to a collection without affecting consumers that query through a view
- **Security boundary** — hide sensitive fields (passwords, tokens) from consumers that don't need them
- **API clarity** — name and document a stable contract ("this view returns these fields")

```cpp
// Create a view (normally done via migrations, but also available at runtime)
db.createView("users_public",
              /*collection=*/"users",
              /*include=*/{"id", "name", "avatar_url", "profile.public_bio"});

// Query the view exactly like a collection
auto user = db.get("users_public", "u1");           // returns only the included fields
auto results = db.find("users_public", opts);       // filters applied on real collection, projection on response

// Inspect
auto views = db.listViews();
auto info = db.getViewInfo("users_public");

// Drop
db.dropView("users_public");
```

Semantics:

- **Include is a whitelist.** If `include` is non-empty, only those field paths are returned.
- **Exclude is a blacklist.** If `include` is empty and `exclude` is non-empty, all fields except excluded paths are returned.
- **Dot-notation** supports nested paths: `user.profile.name`.
- **Metadata fields** (`_id`, `_version`, `_created_at`, etc.) are always returned regardless of the projection.
- **Read-only.** Attempting `insert()`, `update()`, `patch()`, `upsert()`, or `remove()` on a view name returns an error.
- **No view-of-view.** A view's target must be a real collection.

#### Via migrations (recommended)

```json
{
  "version": "005",
  "name": "create_users_public_view",
  "operations": [
    {
      "type": "create_view",
      "name": "users_public",
      "collection": "users",
      "include": ["id", "name", "avatar_url", "profile.public_bio"]
    }
  ]
}
```
```

- [ ] **Step 2: Update CLAUDE.md**

Add to the "Key Features" bullet list:

```markdown
- **Views** — Read-only named projections over collections with include/exclude field lists (dot-notation for nested paths). Stored in `_views` system collection, created via migrations or `createView` RPC. Write operations on view names rejected server-side.
```

- [ ] **Step 3: Commit**

```bash
git add docs/integration-guide.md CLAUDE.md
git commit -m "docs(views): document view feature with usage examples"
```

---

### Task 13: End-to-end verification

- [ ] **Step 1: Verify all packages exist and are signed**

```bash
ls -lh dist/local/ dist/debian13/
ls /data/smartbotics-deb-repo/repo/pool/trixie/main/*1.5.0*
```

- [ ] **Step 2: Verify old packages preserved**

```bash
ls /data/smartbotics-deb-repo/repo/pool/trixie/main/smartbotic-database_1.4.0* 2>&1
ls /data/smartbotics-deb-repo/repo/pool/trixie/main/smartbotic-database_1.3.0* 2>&1
```

Expected: 1.3.0 and 1.4.0 still present in pool.

- [ ] **Step 3: Run all tests**

```bash
cmake --build build -j$(nproc)
LD_LIBRARY_PATH=build/client ./build/tests/test_vector_storage
./build/tests/test_views
```

All must pass.

- [ ] **Step 4: Push**

```bash
git push origin main
```

---

## Execution Order

```
Task 1  : Proto + VERSION bump
Task 2  : Projection helper (pure function, no deps) — testable in isolation
Task 3  : ViewManager (uses MemoryStore)
Task 4  : Wire ViewManager into DatabaseService
Task 5  : View RPC handlers (CreateView/DropView/ListViews/GetViewInfo)
Task 6  : Integrate projection into Get/Find/Count handlers
Task 7  : Reject writes on view names
Task 8  : create_view migration op
Task 9  : Client library API
Task 10 : Integration test (projection unit tests)
Task 11 : Build + publish packages
Task 12 : Documentation
Task 13 : End-to-end verification
```

Tasks 1-10 are the core implementation. Tasks 2 and 10 can be done together (tests alongside the helper). Tasks 11-13 are polish and distribution.

## Rollback Plan

- Views are opt-in — existing clients work unchanged
- `_views` collection is new — cannot break existing data
- Read handlers only invoke view logic if `isView(name)` returns true — no perf impact on non-view queries
- If needed, rollback = revert commit; old packages 1.4.0 remain in repo