Views (v1.5.0) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add read-only views to smartbotic-database — named projections over collections that filter which fields are returned. Supports include/exclude lists with nested (dot-path) field selection. Views are stored in a _views system collection, created via migrations or direct RPC.

Architecture: Views are redirects with response-side projection. A ViewManager loads view definitions from the _views system collection. gRPC handlers for read operations (Get, Find, Count) check if the target name is a view — if so, they redirect to the real collection and apply projection on the response JSON. Write handlers (Insert, Update, Patch, Upsert, Delete) reject view names with a clear error. MemoryStore stays view-agnostic.

Tech Stack: C++20, nlohmann/json (already used), gRPC/Protobuf (existing service)

Design Summary

Include + exclude semantics (Option B)

include=[] (empty) + exclude=[x,y] → return all fields except x, y
include=[a,b] + exclude=anything → return only a, b (exclude ignored when include is non-empty)
include=[a,b] + exclude=[] → return only a, b

Nested paths

Dot-notation paths work through nested objects:

user.profile.name → traverse user, then profile, then keep name
Stop at arrays (don't descend into array elements for projection)
Missing intermediate paths are silently skipped

Metadata fields always returned

_id, _version, _created_at, _updated_at, _created_by, _updated_by are always returned regardless of view definition (needed for client identification and optimistic locking).

Read-only

Writes to a view name (Insert/Update/Patch/Upsert/Delete) return INVALID_ARGUMENT with message "cannot write to view: views are read-only".

No view-of-view

View's collection field must reference a real collection, not another view.

File Structure

service/src/views/                                    # NEW
  view_manager.hpp                                    # ViewManager class
  view_manager.cpp
  projection.hpp                                      # applyProjection() helper
  projection.cpp

service/src/database_grpc_impl.hpp                    # Modified: add view RPCs
service/src/database_grpc_impl.cpp                    # Modified: integrate ViewManager into handlers

service/src/database_service.cpp                      # Modified: instantiate ViewManager
service/src/database_service.hpp                      # Modified: hold ViewManager reference

service/src/migrations/migration_runner.cpp           # Modified: add create_view op

client/include/smartbotic/database/client.hpp         # Modified: view API
client/src/client.cpp                                 # Modified: implement view RPCs

proto/database.proto                                  # Modified: 4 new RPCs + messages

tests/test_views.cpp                                  # NEW: integration test
tests/CMakeLists.txt                                  # Modified: add test target

VERSION                                               # Modified: 1.4.0 → 1.5.0
docs/integration-guide.md                             # Modified: Views section
CLAUDE.md                                             # Modified: note views feature

Task 1: Bump version and add proto definitions

Files:

Modify: VERSION
Modify: proto/database.proto

[ ] Step 1: Bump VERSION

echo "1.5.0" > /data/smartbotic-database/VERSION

[ ] Step 2: Add view RPCs to proto/database.proto

Find the DatabaseService definition (around line 22) and add after the collection management RPCs (around line 58):

    // View management
    rpc CreateView(CreateViewRequest) returns (CreateViewResponse);
    rpc DropView(DropViewRequest) returns (DropViewResponse);
    rpc ListViews(ListViewsRequest) returns (ListViewsResponse);
    rpc GetViewInfo(GetViewInfoRequest) returns (GetViewInfoResponse);

Step 3: Add view message types

Add at the end of proto/database.proto (before the final closing brace of any existing message):

// ===== View Operations =====

message ViewDefinition {
    string name = 1;                  // view name (globally unique)
    string collection = 2;            // target real collection
    repeated string include = 3;      // field paths to include (dot-notation)
    repeated string exclude = 4;      // field paths to exclude (dot-notation)
    uint64 created_at = 5;
    uint64 updated_at = 6;
}

message CreateViewRequest {
    string name = 1;
    string collection = 2;
    repeated string include = 3;
    repeated string exclude = 4;
}

message CreateViewResponse {
    bool success = 1;
    string error = 2;
}

message DropViewRequest {
    string name = 1;
}

message DropViewResponse {
    bool success = 1;
    string error = 2;
}

message ListViewsRequest {}

message ListViewsResponse {
    repeated ViewDefinition views = 1;
}

message GetViewInfoRequest {
    string name = 1;
}

message GetViewInfoResponse {
    ViewDefinition view = 1;
    bool found = 2;
}

[ ] Step 4: Build to verify proto compiles

cd /data/smartbotic-database
cmake --build build -j$(nproc) 2>&1 | grep -E "error|Generating"

Expected: Generating protobuf/gRPC code for database.proto appears, no errors.

[ ] Step 5: Commit

git add VERSION proto/database.proto
git commit -m "feat(proto): add view management RPCs and messages (v1.5.0)"

Task 2: Create projection helper with nested path support

Files:

Create: service/src/views/projection.hpp
Create: service/src/views/projection.cpp

[ ] Step 1: Create projection.hpp

#pragma once

#include <nlohmann/json.hpp>
#include <string>
#include <vector>

namespace smartbotic::database {

/**
* Apply projection (include/exclude) to a document.
*
* Semantics (Option B):
*  - If include is non-empty: only those paths are kept (exclude is ignored)
*  - If include is empty and exclude is non-empty: all fields kept except exclude paths
*  - If both empty: document returned unchanged
*
* Paths use dot notation: "user.profile.name". Array traversal stops at arrays.
* Missing intermediate paths are silently skipped.
*
* Metadata fields (_id, _version, _created_at, _updated_at, _created_by, _updated_by)
* are always preserved regardless of include/exclude.
*/
nlohmann::json applyProjection(const nlohmann::json& document,
                            const std::vector<std::string>& include,
                            const std::vector<std::string>& exclude);

} // namespace smartbotic::database

[ ] Step 2: Create projection.cpp

#include "projection.hpp"

#include <algorithm>
#include <sstream>

namespace smartbotic::database {

namespace {

// Metadata fields that are always preserved
const std::vector<std::string> METADATA_FIELDS = {
"_id", "_version", "_created_at", "_updated_at", "_created_by", "_updated_by"
};

// Split "user.profile.name" into ["user", "profile", "name"]
std::vector<std::string> splitPath(const std::string& path) {
std::vector<std::string> parts;
std::stringstream ss(path);
std::string part;
while (std::getline(ss, part, '.')) {
    if (!part.empty()) parts.push_back(part);
}
return parts;
}

// Recursively set src[path] into dst[path], walking nested objects
void copyPath(const nlohmann::json& src, nlohmann::json& dst,
          const std::vector<std::string>& parts, size_t idx) {
if (idx >= parts.size()) return;
const std::string& key = parts[idx];
if (!src.is_object() || !src.contains(key)) return;

if (idx == parts.size() - 1) {
    // Last segment — copy the value
    dst[key] = src[key];
    return;
}

// Intermediate segment — ensure dst[key] is an object, recurse
if (!src[key].is_object()) {
    // Can't descend further into a non-object; just copy the leaf value
    dst[key] = src[key];
    return;
}
if (!dst.contains(key) || !dst[key].is_object()) {
    dst[key] = nlohmann::json::object();
}
copyPath(src[key], dst[key], parts, idx + 1);
}

// Recursively delete dst[path], walking nested objects
void deletePath(nlohmann::json& dst, const std::vector<std::string>& parts, size_t idx) {
if (idx >= parts.size()) return;
const std::string& key = parts[idx];
if (!dst.is_object() || !dst.contains(key)) return;

if (idx == parts.size() - 1) {
    dst.erase(key);
    return;
}

if (dst[key].is_object()) {
    deletePath(dst[key], parts, idx + 1);
}
}

} // anonymous namespace

nlohmann::json applyProjection(const nlohmann::json& document,
                            const std::vector<std::string>& include,
                            const std::vector<std::string>& exclude) {
// Empty include + empty exclude = no projection
if (include.empty() && exclude.empty()) {
    return document;
}

if (!include.empty()) {
    // Include-only semantics: build result with only included paths
    nlohmann::json result = nlohmann::json::object();
    for (const auto& path : include) {
        auto parts = splitPath(path);
        if (parts.empty()) continue;
        copyPath(document, result, parts, 0);
    }
    // Always preserve metadata fields
    for (const auto& meta : METADATA_FIELDS) {
        if (document.contains(meta)) {
            result[meta] = document[meta];
        }
    }
    return result;
}

// Exclude-only semantics: copy document, delete excluded paths
nlohmann::json result = document;
for (const auto& path : exclude) {
    auto parts = splitPath(path);
    if (parts.empty()) continue;
    // Don't allow excluding metadata fields
    if (parts.size() == 1) {
        bool isMeta = false;
        for (const auto& meta : METADATA_FIELDS) {
            if (parts[0] == meta) { isMeta = true; break; }
        }
        if (isMeta) continue;
    }
    deletePath(result, parts, 0);
}
return result;
}

} // namespace smartbotic::database

[ ] Step 3: Add projection.cpp to service CMakeLists.txt

Read /data/smartbotic-database/service/CMakeLists.txt. Find the DATABASE_SERVICE_SOURCES list. Add:

    src/views/projection.cpp

between the existing sources (alphabetical order — between replication/sync_protocol.cpp and migrations/migration_runner.cpp, wherever it fits).

[ ] Step 4: Build to verify compilation

cmake --build build -j$(nproc) 2>&1 | tail -5

Expected: Build succeeds, projection.cpp.o is built.

[ ] Step 5: Commit

git add service/src/views/ service/CMakeLists.txt
git commit -m "feat(views): add projection helper with nested path support"

Task 3: Create ViewManager

Files:

Create: service/src/views/view_manager.hpp
Create: service/src/views/view_manager.cpp

[ ] Step 1: Create view_manager.hpp

#pragma once

#include <memory>
#include <mutex>
#include <optional>
#include <shared_mutex>
#include <string>
#include <unordered_map>
#include <vector>

namespace smartbotic::database {

class MemoryStore;

struct ViewInfo {
std::string name;
std::string collection;
std::vector<std::string> include;
std::vector<std::string> exclude;
uint64_t createdAt = 0;
uint64_t updatedAt = 0;
};

/**
* ViewManager — loads, caches, and manages view definitions.
*
* Views are persisted as documents in the `_views` system collection.
* On startup, loadFromStore() populates an in-memory cache for O(1) lookup.
* Mutations (createView/dropView) update both the store and the cache.
*/
class ViewManager {
public:
static constexpr const char* SYSTEM_COLLECTION = "_views";

explicit ViewManager(MemoryStore& store);

/**
 * Load all view definitions from _views collection into the cache.
 * Called once at startup after MemoryStore is initialized.
 */
void loadFromStore();

/**
 * Create a view. Fails if name already exists or collection doesn't exist.
 * Returns false with errorOut populated on failure.
 */
bool createView(const ViewInfo& view, std::string& errorOut);

/**
 * Drop a view by name.
 */
bool dropView(const std::string& name, std::string& errorOut);

/**
 * O(1) check — is this name a view?
 */
bool isView(const std::string& name) const;

/**
 * Lookup view by name. Returns nullopt if not a view.
 */
std::optional<ViewInfo> getView(const std::string& name) const;

/**
 * List all views.
 */
std::vector<ViewInfo> listViews() const;

private:
MemoryStore& store_;
mutable std::shared_mutex cacheMutex_;
std::unordered_map<std::string, ViewInfo> cache_;
};

} // namespace smartbotic::database

[ ] Step 2: Create view_manager.cpp

#include "view_manager.hpp"

#include "../memory_store.hpp"
#include "../document.hpp"

#include <chrono>
#include <nlohmann/json.hpp>
#include <spdlog/spdlog.h>

namespace smartbotic::database {

namespace {
uint64_t nowMs() {
return std::chrono::duration_cast<std::chrono::milliseconds>(
    std::chrono::system_clock::now().time_since_epoch()).count();
}

ViewInfo fromJson(const nlohmann::json& j) {
ViewInfo v;
v.name = j.value("name", "");
v.collection = j.value("collection", "");
if (j.contains("include") && j["include"].is_array()) {
    for (const auto& p : j["include"]) v.include.push_back(p.get<std::string>());
}
if (j.contains("exclude") && j["exclude"].is_array()) {
    for (const auto& p : j["exclude"]) v.exclude.push_back(p.get<std::string>());
}
v.createdAt = j.value("created_at", uint64_t{0});
v.updatedAt = j.value("updated_at", uint64_t{0});
return v;
}

nlohmann::json toJson(const ViewInfo& v) {
return {
    {"name", v.name},
    {"collection", v.collection},
    {"include", v.include},
    {"exclude", v.exclude},
    {"created_at", v.createdAt},
    {"updated_at", v.updatedAt}
};
}
} // anonymous namespace

ViewManager::ViewManager(MemoryStore& store) : store_(store) {}

void ViewManager::loadFromStore() {
std::unique_lock<std::shared_mutex> lock(cacheMutex_);
cache_.clear();

// Ensure _views collection exists
CollectionOptions opts;
store_.createCollection(SYSTEM_COLLECTION, opts);

Query q;
auto result = store_.find(SYSTEM_COLLECTION, q);
for (const auto& doc : result.documents) {
    ViewInfo v = fromJson(doc.data);
    if (v.name.empty()) continue;
    cache_[v.name] = v;
}
spdlog::info("ViewManager: loaded {} views from _views", cache_.size());
}

bool ViewManager::createView(const ViewInfo& view, std::string& errorOut) {
if (view.name.empty()) {
    errorOut = "view name is required";
    return false;
}
if (view.collection.empty()) {
    errorOut = "target collection is required";
    return false;
}
if (view.name == view.collection) {
    errorOut = "view name cannot equal collection name";
    return false;
}
if (view.name.rfind("_", 0) == 0) {
    errorOut = "view name cannot start with '_' (reserved for system collections)";
    return false;
}

// No view-of-view
{
    std::shared_lock<std::shared_mutex> rlock(cacheMutex_);
    if (cache_.contains(view.collection)) {
        errorOut = "target '" + view.collection + "' is a view — views can only target real collections";
        return false;
    }
    if (cache_.contains(view.name)) {
        errorOut = "view '" + view.name + "' already exists";
        return false;
    }
}

ViewInfo v = view;
v.createdAt = nowMs();
v.updatedAt = v.createdAt;

// Persist to _views
Document doc;
doc.data = toJson(v);
std::string id = store_.insert(SYSTEM_COLLECTION, v.name, doc);
if (id.empty()) {
    errorOut = "failed to persist view definition";
    return false;
}

{
    std::unique_lock<std::shared_mutex> wlock(cacheMutex_);
    cache_[v.name] = v;
}
spdlog::info("ViewManager: created view '{}' over '{}'", v.name, v.collection);
return true;
}

bool ViewManager::dropView(const std::string& name, std::string& errorOut) {
{
    std::shared_lock<std::shared_mutex> rlock(cacheMutex_);
    if (!cache_.contains(name)) {
        errorOut = "view '" + name + "' does not exist";
        return false;
    }
}

bool removed = store_.remove(SYSTEM_COLLECTION, name);
if (!removed) {
    errorOut = "failed to remove view from store";
    return false;
}

{
    std::unique_lock<std::shared_mutex> wlock(cacheMutex_);
    cache_.erase(name);
}
spdlog::info("ViewManager: dropped view '{}'", name);
return true;
}

bool ViewManager::isView(const std::string& name) const {
std::shared_lock<std::shared_mutex> lock(cacheMutex_);
return cache_.contains(name);
}

std::optional<ViewInfo> ViewManager::getView(const std::string& name) const {
std::shared_lock<std::shared_mutex> lock(cacheMutex_);
auto it = cache_.find(name);
if (it == cache_.end()) return std::nullopt;
return it->second;
}

std::vector<ViewInfo> ViewManager::listViews() const {
std::shared_lock<std::shared_mutex> lock(cacheMutex_);
std::vector<ViewInfo> out;
out.reserve(cache_.size());
for (const auto& [_, v] : cache_) out.push_back(v);
return out;
}

} // namespace smartbotic::database

[ ] Step 3: Add view_manager.cpp to CMakeLists.txt

In service/CMakeLists.txt, DATABASE_SERVICE_SOURCES list, add:

    src/views/view_manager.cpp

Step 4: Verify MemoryStore insert signature

Read service/src/memory_store.hpp and confirm:

std::string insert(const std::string& collection, const std::string& id, const Document& doc);

If the signature is different (e.g., different argument order or overloads), adjust the store_.insert(...) call in view_manager.cpp accordingly. Check service/src/memory_store.cpp for the exact implementation.

[ ] Step 5: Build

cmake --build build -j$(nproc) 2>&1 | tail -5

Fix any compile errors that arise from MemoryStore API mismatches.

[ ] Step 6: Commit

git add service/src/views/view_manager.hpp service/src/views/view_manager.cpp service/CMakeLists.txt
git commit -m "feat(views): add ViewManager with cached lookup and _views system collection"

Task 4: Wire ViewManager into DatabaseService

Files:

Modify: service/src/database_service.hpp
Modify: service/src/database_service.cpp

[ ] Step 1: Read existing service structure

cat /data/smartbotic-database/service/src/database_service.hpp
cat /data/smartbotic-database/service/src/database_service.cpp

Note how other managers (PersistenceManager, EncryptionManager, EventManager, FileManager, ReplicationManager, MigrationRunner) are instantiated and owned.

Step 2: Add ViewManager to DatabaseService

In service/src/database_service.hpp:

Add include at the top:
```
#include "views/view_manager.hpp"
```
In the class, after the other manager members (alongside file_manager_, replication_manager_, etc.), add:
```
ViewManager view_manager_;
```

Add a getter (near other getters):

ViewManager& viewManager() { return view_manager_; }

Step 3: Initialize ViewManager in constructor

In service/src/database_service.cpp:

Find the DatabaseService constructor's member initializer list
Add view_manager_(memory_store_) alongside other manager initializations that take memory_store_
In the start() method (or wherever other managers load their state), after MemoryStore has loaded snapshots/WAL, add:
```
view_manager_.loadFromStore();
```

This must happen AFTER persistence has loaded existing documents (so views stored in _views are present in memory).

[ ] Step 4: Build

cmake --build build -j$(nproc) 2>&1 | tail -5

[ ] Step 5: Run tests

LD_LIBRARY_PATH=build/client ./build/tests/test_vector_storage

Expected: all tests pass (ViewManager loads empty cache when no views exist).

[ ] Step 6: Commit

git add service/src/database_service.hpp service/src/database_service.cpp
git commit -m "feat(views): wire ViewManager into DatabaseService lifecycle"

Task 5: Implement view RPC handlers

Files:

Modify: service/src/database_grpc_impl.hpp
Modify: service/src/database_grpc_impl.cpp
[ ] Step 1: Add ViewManager reference to DatabaseGrpcImpl

In service/src/database_grpc_impl.hpp:

Add include:
```
#include "views/view_manager.hpp"
```
Add constructor parameter and member. Find the existing constructor signature (it takes references to MemoryStore, PersistenceManager, etc.) and add a ViewManager& parameter. Add the matching ViewManager& view_manager_; member.

Declare the 4 new handler methods:

grpc::Status CreateView(grpc::ServerContext* context,
                        const pb::CreateViewRequest* request,
                        pb::CreateViewResponse* response) override;

grpc::Status DropView(grpc::ServerContext* context,
                      const pb::DropViewRequest* request,
                      pb::DropViewResponse* response) override;

grpc::Status ListViews(grpc::ServerContext* context,
                       const pb::ListViewsRequest* request,
                       pb::ListViewsResponse* response) override;

grpc::Status GetViewInfo(grpc::ServerContext* context,
                         const pb::GetViewInfoRequest* request,
                         pb::GetViewInfoResponse* response) override;

Step 2: Update DatabaseGrpcImpl constructor in .cpp

Find the constructor implementation in database_grpc_impl.cpp. Add view_manager to the parameter list and initializer list.

Step 3: Update DatabaseService to pass viewmanager

In service/src/database_service.cpp, find where DatabaseGrpcImpl is constructed and add view_manager_ to the argument list.

Step 4: Implement CreateView handler

Add at the end of database_grpc_impl.cpp, before the closing namespace brace:

grpc::Status DatabaseGrpcImpl::CreateView(
    grpc::ServerContext* /*context*/,
    const pb::CreateViewRequest* request,
    pb::CreateViewResponse* response
) {
    try {
        ViewInfo v;
        v.name = request->name();
        v.collection = request->collection();
        for (const auto& p : request->include()) v.include.push_back(p);
        for (const auto& p : request->exclude()) v.exclude.push_back(p);

        std::string err;
        bool ok = view_manager_.createView(v, err);
        response->set_success(ok);
        if (!ok) response->set_error(err);
        return grpc::Status::OK;
    } catch (const std::exception& e) {
        return grpc::Status(grpc::StatusCode::INTERNAL, e.what());
    }
}

[ ] Step 5: Implement DropView handler

grpc::Status DatabaseGrpcImpl::DropView(
grpc::ServerContext* /*context*/,
const pb::DropViewRequest* request,
pb::DropViewResponse* response
) {
try {
    std::string err;
    bool ok = view_manager_.dropView(request->name(), err);
    response->set_success(ok);
    if (!ok) response->set_error(err);
    return grpc::Status::OK;
} catch (const std::exception& e) {
    return grpc::Status(grpc::StatusCode::INTERNAL, e.what());
}
}

[ ] Step 6: Implement ListViews handler

grpc::Status DatabaseGrpcImpl::ListViews(
grpc::ServerContext* /*context*/,
const pb::ListViewsRequest* /*request*/,
pb::ListViewsResponse* response
) {
auto views = view_manager_.listViews();
for (const auto& v : views) {
    auto* pb = response->add_views();
    pb->set_name(v.name);
    pb->set_collection(v.collection);
    for (const auto& p : v.include) pb->add_include(p);
    for (const auto& p : v.exclude) pb->add_exclude(p);
    pb->set_created_at(v.createdAt);
    pb->set_updated_at(v.updatedAt);
}
return grpc::Status::OK;
}

[ ] Step 7: Implement GetViewInfo handler

grpc::Status DatabaseGrpcImpl::GetViewInfo(
grpc::ServerContext* /*context*/,
const pb::GetViewInfoRequest* request,
pb::GetViewInfoResponse* response
) {
auto v = view_manager_.getView(request->name());
if (!v) {
    response->set_found(false);
    return grpc::Status::OK;
}
auto* pb = response->mutable_view();
pb->set_name(v->name);
pb->set_collection(v->collection);
for (const auto& p : v->include) pb->add_include(p);
for (const auto& p : v->exclude) pb->add_exclude(p);
pb->set_created_at(v->createdAt);
pb->set_updated_at(v->updatedAt);
response->set_found(true);
return grpc::Status::OK;
}

[ ] Step 8: Build

cmake --build build -j$(nproc) 2>&1 | tail -5

[ ] Step 9: Commit

git add service/src/database_grpc_impl.hpp service/src/database_grpc_impl.cpp service/src/database_service.cpp
git commit -m "feat(views): implement CreateView/DropView/ListViews/GetViewInfo RPC handlers"

Task 6: Integrate view projection into read handlers

Files:

Modify: service/src/database_grpc_impl.cpp

Integrates the isView check into Get, Find, Count handlers. When the name is a view, redirects to the real collection and applies projection.

Step 1: Add projection include

At the top of database_grpc_impl.cpp, add:

#include "views/projection.hpp"

Step 2: Modify the Get handler

Find the existing Get handler. At the start (after parameter validation), add:

    std::string targetCollection = request->collection();
    std::optional<ViewInfo> view;
    if (view_manager_.isView(request->collection())) {
        view = view_manager_.getView(request->collection());
        targetCollection = view->collection;
    }

Then replace every use of request->collection() inside the Get handler with targetCollection.

Before writing the document data to the response, apply projection if this is a view. Find where the response document's data is set (look for response->mutable_document()->set_data(...)). Wrap the data like:

    nlohmann::json docJson = nlohmann::json::parse(doc->data);
    if (view) {
        docJson = applyProjection(docJson, view->include, view->exclude);
    }
    response->mutable_document()->set_data(docJson.dump());

IMPORTANT: Read the existing Get handler carefully to understand how doc->data is used (it may be bytes / std::string already containing JSON-encoded data).

Step 3: Modify the Find handler

Same pattern as Get. At the start:

    std::string targetCollection = request->collection();
    std::optional<ViewInfo> view;
    if (view_manager_.isView(request->collection())) {
        view = view_manager_.getView(request->collection());
        targetCollection = view->collection;
    }

Replace request->collection() with targetCollection inside the handler.

In the loop that copies each document to the response, apply projection:

    nlohmann::json docJson = nlohmann::json::parse(doc.data);
    if (view) {
        docJson = applyProjection(docJson, view->include, view->exclude);
    }
    pbDoc->set_data(docJson.dump());

Step 4: Modify the Count handler

Count doesn't need projection (it returns a number), but should redirect views:

    std::string targetCollection = request->collection();
    if (view_manager_.isView(request->collection())) {
        auto view = view_manager_.getView(request->collection());
        targetCollection = view->collection;
    }

Then use targetCollection in the store query.

[ ] Step 5: Build and test

cmake --build build -j$(nproc) 2>&1 | tail -5
LD_LIBRARY_PATH=build/client ./build/tests/test_vector_storage

[ ] Step 6: Commit

git add service/src/database_grpc_impl.cpp
git commit -m "feat(views): integrate projection into Get/Find/Count handlers"

Task 7: Reject writes through views

Files:

Modify: service/src/database_grpc_impl.cpp
[ ] Step 1: Add a helper

Near the top of database_grpc_impl.cpp, add a helper function (above the class definition or as a static free function):

namespace {
bool rejectIfView(const std::string& name, const ViewManager& vm,
                  auto* response /* has set_success + set_error */) {
    if (vm.isView(name)) {
        response->set_success(false);
        response->set_error("cannot write to view '" + name + "': views are read-only");
        return true;
    }
    return false;
}
} // anonymous namespace

Note: use templated lambda / auto to handle different response types. Alternatively, write four separate helpers or put the rejection inline in each handler — whichever is cleaner given existing patterns.

Step 2: Add rejection to Insert handler

Find the Insert handler. At the start (after parsing the request), add:

    if (view_manager_.isView(request->collection())) {
        return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
                           "cannot insert into view '" + request->collection() + "': views are read-only");
    }

Step 3: Add the same rejection to Update, Upsert, Delete, PatchDocument, BatchInsert, BatchDelete

For each write handler, add the same check at the start. Note: for handlers whose response has set_success()/set_error() fields, you can return a "friendly" error in the response body. For handlers that return only grpc::Status, use INVALID_ARGUMENT.

Read each handler first to see the existing error-reporting style and match it.

[ ] Step 4: Build and test

cmake --build build -j$(nproc) 2>&1 | tail -5
LD_LIBRARY_PATH=build/client ./build/tests/test_vector_storage

[ ] Step 5: Commit

git add service/src/database_grpc_impl.cpp
git commit -m "feat(views): reject Insert/Update/Patch/Upsert/Delete on view names"

Task 8: Add create_view migration op

Files:

Modify: service/src/migrations/migration_runner.hpp
Modify: service/src/migrations/migration_runner.cpp
[ ] Step 1: Wire ViewManager into MigrationRunner

Read service/src/migrations/migration_runner.hpp. If it already takes a MemoryStore&, add a ViewManager& parameter. Add the member.

Step 2: Update construction site

In database_service.cpp, find where MigrationRunner is constructed and pass view_manager_.

Step 3: Add create_view handler

In service/src/migrations/migration_runner.cpp, find applyOperation (around line 209). Add a new branch (before the final return false / unknown operation):

    if (type == "create_view") {
        std::string name = operation.value("name", "");
        std::string collection = operation.value("collection", "");
        if (name.empty() || collection.empty()) {
            spdlog::error("create_view: missing name or collection");
            return false;
        }

        ViewInfo v;
        v.name = name;
        v.collection = collection;
        if (operation.contains("include") && operation["include"].is_array()) {
            for (const auto& p : operation["include"]) {
                v.include.push_back(p.get<std::string>());
            }
        }
        if (operation.contains("exclude") && operation["exclude"].is_array()) {
            for (const auto& p : operation["exclude"]) {
                v.exclude.push_back(p.get<std::string>());
            }
        }

        std::string err;
        bool ok = view_manager_.createView(v, err);
        if (!ok) {
            // Treat "already exists" as success (idempotent migration)
            if (err.find("already exists") != std::string::npos) {
                spdlog::debug("create_view '{}': already exists (idempotent)", name);
                return true;
            }
            spdlog::error("create_view '{}': {}", name, err);
            return false;
        }
        spdlog::info("create_view '{}' over '{}': ok", name, collection);
        return true;
    }

Add include at the top of migration_runner.cpp:

#include "../views/view_manager.hpp"

[ ] Step 4: Build

cmake --build build -j$(nproc) 2>&1 | tail -5

[ ] Step 5: Commit

git add service/src/migrations/migration_runner.hpp service/src/migrations/migration_runner.cpp service/src/database_service.cpp
git commit -m "feat(views): add create_view migration operation"

Task 9: Add view methods to client library

Files:

Modify: client/include/smartbotic/database/client.hpp
Modify: client/src/client.cpp
[ ] Step 1: Add ViewDefinition struct and method declarations in client.hpp

After the existing public methods (around the collection management section), add:

    // ===== View Management =====

    struct ViewDefinition {
        std::string name;
        std::string collection;
        std::vector<std::string> include;
        std::vector<std::string> exclude;
        uint64_t createdAt = 0;
        uint64_t updatedAt = 0;
    };

    /**
     * Create a read-only view over a collection.
     * @param name View name (unique, cannot start with `_`)
     * @param collection Target collection (must be a real collection, not a view)
     * @param include Field paths to include (dot-notation supports nesting)
     * @param exclude Field paths to exclude (ignored if include is non-empty)
     * @return true on success, false otherwise (check server logs for details)
     */
    bool createView(const std::string& name, const std::string& collection,
                    const std::vector<std::string>& include = {},
                    const std::vector<std::string>& exclude = {});

    bool dropView(const std::string& name);
    [[nodiscard]] std::vector<ViewDefinition> listViews();
    [[nodiscard]] std::optional<ViewDefinition> getViewInfo(const std::string& name);

Step 2: Implement in client.cpp (Impl class)

Inside the Client::Impl class, add:

    bool createView(const std::string& name, const std::string& collection,
                    const std::vector<std::string>& include,
                    const std::vector<std::string>& exclude) {
        smartbotic::databasepb::CreateViewRequest request;
        request.set_name(name);
        request.set_collection(collection);
        for (const auto& p : include) request.add_include(p);
        for (const auto& p : exclude) request.add_exclude(p);

        smartbotic::databasepb::CreateViewResponse response;
        grpc::ClientContext context;
        setDeadline(context);

        auto status = stub_->CreateView(&context, request, &response);
        if (!status.ok()) {
            spdlog::error("Client::createView failed: {}", status.error_message());
            return false;
        }
        if (!response.success()) {
            spdlog::error("Client::createView rejected: {}", response.error());
            return false;
        }
        return true;
    }

    bool dropView(const std::string& name) {
        smartbotic::databasepb::DropViewRequest request;
        request.set_name(name);
        smartbotic::databasepb::DropViewResponse response;
        grpc::ClientContext context;
        setDeadline(context);

        auto status = stub_->DropView(&context, request, &response);
        if (!status.ok()) {
            spdlog::error("Client::dropView failed: {}", status.error_message());
            return false;
        }
        return response.success();
    }

    std::vector<Client::ViewDefinition> listViews() {
        smartbotic::databasepb::ListViewsRequest request;
        smartbotic::databasepb::ListViewsResponse response;
        grpc::ClientContext context;
        setDeadline(context);

        auto status = stub_->ListViews(&context, request, &response);
        std::vector<Client::ViewDefinition> out;
        if (!status.ok()) {
            spdlog::error("Client::listViews failed: {}", status.error_message());
            return out;
        }
        for (const auto& pb : response.views()) {
            Client::ViewDefinition v;
            v.name = pb.name();
            v.collection = pb.collection();
            for (const auto& p : pb.include()) v.include.push_back(p);
            for (const auto& p : pb.exclude()) v.exclude.push_back(p);
            v.createdAt = pb.created_at();
            v.updatedAt = pb.updated_at();
            out.push_back(std::move(v));
        }
        return out;
    }

    std::optional<Client::ViewDefinition> getViewInfo(const std::string& name) {
        smartbotic::databasepb::GetViewInfoRequest request;
        request.set_name(name);
        smartbotic::databasepb::GetViewInfoResponse response;
        grpc::ClientContext context;
        setDeadline(context);

        auto status = stub_->GetViewInfo(&context, request, &response);
        if (!status.ok() || !response.found()) {
            return std::nullopt;
        }
        Client::ViewDefinition v;
        v.name = response.view().name();
        v.collection = response.view().collection();
        for (const auto& p : response.view().include()) v.include.push_back(p);
        for (const auto& p : response.view().exclude()) v.exclude.push_back(p);
        v.createdAt = response.view().created_at();
        v.updatedAt = response.view().updated_at();
        return v;
    }

Step 3: Add public forwarding methods at the bottom of client.cpp

Where the other Client::xyz public methods are defined (delegating to impl_->xyz), add:

bool Client::createView(const std::string& name, const std::string& collection,
                        const std::vector<std::string>& include,
                        const std::vector<std::string>& exclude) {
    return impl_->createView(name, collection, include, exclude);
}

bool Client::dropView(const std::string& name) {
    return impl_->dropView(name);
}

std::vector<Client::ViewDefinition> Client::listViews() {
    return impl_->listViews();
}

std::optional<Client::ViewDefinition> Client::getViewInfo(const std::string& name) {
    return impl_->getViewInfo(name);
}

[ ] Step 4: Build

cmake --build build -j$(nproc) 2>&1 | tail -5

[ ] Step 5: Commit

git add client/include/smartbotic/database/client.hpp client/src/client.cpp
git commit -m "feat(views): add client API for view management"

Task 10: Integration test

Files:

Create: tests/test_views.cpp
Modify: tests/CMakeLists.txt

[ ] Step 1: Read existing test structure

cat /data/smartbotic-database/tests/test_vector_storage.cpp | head -80
cat /data/smartbotic-database/tests/CMakeLists.txt

Note the test harness pattern (how MemoryStore is constructed, how tests are invoked).

Step 2: Create tests/test_views.cpp

Write a test that exercises applyProjection directly (doesn't need a running server). Follow the exact test pattern used in test_vector_storage.cpp:

#include "../service/src/views/projection.hpp"

#include <cassert>
#include <iostream>
#include <nlohmann/json.hpp>

using smartbotic::database::applyProjection;

void test_empty_projection_returns_document_unchanged() {
    nlohmann::json doc = {{"name", "Alice"}, {"age", 30}};
    auto result = applyProjection(doc, {}, {});
    assert(result == doc);
    std::cout << "PASS: empty projection returns document unchanged\n";
}

void test_include_keeps_only_listed_fields() {
    nlohmann::json doc = {{"name", "Alice"}, {"email", "a@x.com"}, {"password", "secret"}};
    auto result = applyProjection(doc, {"name", "email"}, {});
    assert(result.contains("name"));
    assert(result.contains("email"));
    assert(!result.contains("password"));
    std::cout << "PASS: include keeps only listed fields\n";
}

void test_exclude_removes_fields() {
    nlohmann::json doc = {{"name", "Alice"}, {"email", "a@x.com"}, {"password", "secret"}};
    auto result = applyProjection(doc, {}, {"password"});
    assert(result.contains("name"));
    assert(result.contains("email"));
    assert(!result.contains("password"));
    std::cout << "PASS: exclude removes fields\n";
}

void test_include_takes_precedence_over_exclude() {
    nlohmann::json doc = {{"a", 1}, {"b", 2}, {"c", 3}};
    auto result = applyProjection(doc, {"a"}, {"b", "c"});
    assert(result.contains("a"));
    assert(!result.contains("b"));
    assert(!result.contains("c"));
    assert(result.size() == 1);
    std::cout << "PASS: include takes precedence over exclude\n";
}

void test_nested_include_path() {
    nlohmann::json doc = {
        {"user", {
            {"profile", {{"name", "Alice"}, {"bio", "hi"}}},
            {"private", {{"ssn", "123"}}}
        }}
    };
    auto result = applyProjection(doc, {"user.profile.name"}, {});
    assert(result["user"]["profile"]["name"] == "Alice");
    assert(!result["user"]["profile"].contains("bio"));
    assert(!result["user"].contains("private"));
    std::cout << "PASS: nested include path\n";
}

void test_nested_exclude_path() {
    nlohmann::json doc = {
        {"user", {
            {"name", "Alice"},
            {"private", {{"ssn", "123"}, {"email", "a@x.com"}}}
        }}
    };
    auto result = applyProjection(doc, {}, {"user.private.ssn"});
    assert(result["user"]["name"] == "Alice");
    assert(result["user"]["private"]["email"] == "a@x.com");
    assert(!result["user"]["private"].contains("ssn"));
    std::cout << "PASS: nested exclude path\n";
}

void test_metadata_always_preserved() {
    nlohmann::json doc = {
        {"_id", "doc-1"},
        {"_version", 5},
        {"name", "Alice"},
        {"secret", "xyz"}
    };
    auto result = applyProjection(doc, {"name"}, {});
    assert(result.contains("_id"));
    assert(result.contains("_version"));
    assert(result.contains("name"));
    assert(!result.contains("secret"));
    std::cout << "PASS: metadata always preserved\n";
}

void test_metadata_cannot_be_excluded() {
    nlohmann::json doc = {{"_id", "doc-1"}, {"_version", 5}, {"name", "Alice"}};
    auto result = applyProjection(doc, {}, {"_id", "_version"});
    assert(result.contains("_id"));
    assert(result.contains("_version"));
    std::cout << "PASS: metadata cannot be excluded\n";
}

void test_missing_path_silently_skipped() {
    nlohmann::json doc = {{"name", "Alice"}};
    auto result = applyProjection(doc, {"user.profile.name"}, {});
    // No crash, result just has metadata (none present) and nothing else
    assert(!result.contains("user"));
    std::cout << "PASS: missing path silently skipped\n";
}

int main() {
    test_empty_projection_returns_document_unchanged();
    test_include_keeps_only_listed_fields();
    test_exclude_removes_fields();
    test_include_takes_precedence_over_exclude();
    test_nested_include_path();
    test_nested_exclude_path();
    test_metadata_always_preserved();
    test_metadata_cannot_be_excluded();
    test_missing_path_silently_skipped();
    std::cout << "\nAll view projection tests PASSED!\n";
    return 0;
}

Step 3: Add to tests/CMakeLists.txt

Read the existing CMakeLists.txt. Follow the same pattern used for test_vector_storage to add test_views:

add_executable(test_views
    test_views.cpp
    ${CMAKE_SOURCE_DIR}/service/src/views/projection.cpp
)
target_include_directories(test_views PRIVATE
    ${CMAKE_SOURCE_DIR}/service/src
)
target_link_libraries(test_views PRIVATE
    smartbotic_db_proto
    nlohmann_json::nlohmann_json
)

Adjust include paths / link targets to match the existing test_vector_storage patterns.

[ ] Step 4: Build and run

cmake --build build -j$(nproc) 2>&1 | tail -5
./build/tests/test_views

Expected: "All view projection tests PASSED!"

[ ] Step 5: Commit

git add tests/test_views.cpp tests/CMakeLists.txt
git commit -m "test(views): add projection unit tests with nested path coverage"

Task 11: Build and publish packages

[ ] Step 1: Local build

./packaging/build.sh --local --skip-tests

Expected: dist/local/*1.5.0*.deb appear.

[ ] Step 2: Docker build + publish

./packaging/build.sh --sync --suite trixie

Expected: dist/debian13/*1.5.0*.deb created, added to repo, synced to repository.smartbotics.ai.

[ ] Step 3: Verify

ls /data/smartbotics-deb-repo/repo/pool/trixie/main/*1.5.0*

Expected: 4 files — smartbotic-database, libsmartbotic-db-client, libsmartbotic-db-client-dev, smartbotic-db-cli — all at 1.5.0-1.

Task 12: Update documentation

Files:

Modify: docs/integration-guide.md
Modify: CLAUDE.md
[ ] Step 1: Add Views section to integration-guide.md

Insert a new section after the "Document Operations" section, before "Querying":

### Views

Views are named, read-only projections over collections. They filter which fields are returned when the view is queried. Views are useful for:

- **Schema evolution** — add fields to a collection without affecting consumers that query through a view
- **Security boundary** — hide sensitive fields (passwords, tokens) from consumers that don't need them
- **API clarity** — name and document a stable contract ("this view returns these fields")

cpp // Create a view (normally done via migrations, but also available at runtime) db.createView("users_public",

          /*collection=*/"users",
          /*include=*/{"id", "name", "avatar_url", "profile.public_bio"});

// Query the view exactly like a collection auto user = db.get("users_public", "u1"); // returns only the included fields auto results = db.find("users_public", opts); // filters applied on real collection, projection on response

// Inspect auto views = db.listViews(); auto info = db.getViewInfo("users_public");

// Drop db.dropView("users_public");


Semantics:

- **Include is a whitelist.** If `include` is non-empty, only those field paths are returned.
- **Exclude is a blacklist.** If `include` is empty and `exclude` is non-empty, all fields except excluded paths are returned.
- **Dot-notation** supports nested paths: `user.profile.name`.
- **Metadata fields** (`_id`, `_version`, `_created_at`, etc.) are always returned regardless of the projection.
- **Read-only.** Attempting `insert()`, `update()`, `patch()`, `upsert()`, or `remove()` on a view name returns an error.
- **No view-of-view.** A view's target must be a real collection.

#### Via migrations (recommended)

json { "version": "005", "name": "create_users_public_view", "operations": [

{
  "type": "create_view",
  "name": "users_public",
  "collection": "users",
  "include": ["id", "name", "avatar_url", "profile.public_bio"]
}

] }

Step 2: Update CLAUDE.md

Add to the "Key Features" bullet list:

- **Views** — Read-only named projections over collections with include/exclude field lists (dot-notation for nested paths). Stored in `_views` system collection, created via migrations or `createView` RPC. Write operations on view names rejected server-side.

[ ] Step 3: Commit

git add docs/integration-guide.md CLAUDE.md
git commit -m "docs(views): document view feature with usage examples"

Task 13: End-to-end verification

[ ] Step 1: Verify all packages exist and are signed

ls -lh dist/local/ dist/debian13/
ls /data/smartbotics-deb-repo/repo/pool/trixie/main/*1.5.0*

[ ] Step 2: Verify old packages preserved

ls /data/smartbotics-deb-repo/repo/pool/trixie/main/smartbotic-database_1.4.0* 2>&1
ls /data/smartbotics-deb-repo/repo/pool/trixie/main/smartbotic-database_1.3.0* 2>&1

Expected: 1.3.0 and 1.4.0 still present in pool.

[ ] Step 3: Run all tests

cmake --build build -j$(nproc)
LD_LIBRARY_PATH=build/client ./build/tests/test_vector_storage
./build/tests/test_views

All must pass.

[ ] Step 4: Push
```
git push origin main
```

Execution Order

Task 1  : Proto + VERSION bump
Task 2  : Projection helper (pure function, no deps) — testable in isolation
Task 3  : ViewManager (uses MemoryStore)
Task 4  : Wire ViewManager into DatabaseService
Task 5  : View RPC handlers (CreateView/DropView/ListViews/GetViewInfo)
Task 6  : Integrate projection into Get/Find/Count handlers
Task 7  : Reject writes on view names
Task 8  : create_view migration op
Task 9  : Client library API
Task 10 : Integration test (projection unit tests)
Task 11 : Build + publish packages
Task 12 : Documentation
Task 13 : End-to-end verification

Tasks 1-10 are the core implementation. Tasks 2 and 10 can be done together (tests alongside the helper). Tasks 11-13 are polish and distribution.

Rollback Plan

Views are opt-in — existing clients work unchanged
_views collection is new — cannot break existing data
Read handlers only invoke view logic if isView(name) returns true — no perf impact on non-view queries
If needed, rollback = revert commit; old packages 1.4.0 remain in repo

2026-04-13-views-v1.5.0.md 49 KB Permalink History Raw

Views (v1.5.0) Implementation Plan

Design Summary

Include + exclude semantics (Option B)

Nested paths

Metadata fields always returned

Read-only

No view-of-view

File Structure

Task 1: Bump version and add proto definitions

Task 2: Create projection helper with nested path support

Task 3: Create ViewManager

Task 4: Wire ViewManager into DatabaseService

Task 5: Implement view RPC handlers

Task 6: Integrate view projection into read handlers

Task 7: Reject writes through views

Task 8: Add create_view migration op

Task 9: Add view methods to client library

Task 10: Integration test

Task 11: Build and publish packages

Task 12: Update documentation

Task 13: End-to-end verification

Execution Order

Rollback Plan

2026-04-13-views-v1.5.0.md 49 KB

Permalink History Raw