Initial extraction from CallerAI storage service

Features:
- Standalone database service with health check and replication
- Migration system for declarative JSON-based initialization
- Client library for injection into projects via git submodule
- Self-contained packaging for independent deployment

Architecture:
- service/: Full database service with gRPC API
- client/: Lightweight client library
- proto/: gRPC protocol definitions (DatabaseService, DatabaseReplication)

All namespaces changed from callerai::storage to smartbotic::database.
Proto package changed from callerai.storagepb to smartbotic.databasepb.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

CMakeLists.txt

@@ -0,0 +1,76 @@
+# Smartbotic Database
+# Standalone document database with gRPC API
+
+cmake_minimum_required(VERSION 3.20)
+
+# Read version from VERSION file
+file(READ "${CMAKE_CURRENT_SOURCE_DIR}/VERSION" SMARTBOTIC_DB_VERSION)
+string(STRIP "${SMARTBOTIC_DB_VERSION}" SMARTBOTIC_DB_VERSION)
+
+# Detect standalone vs embedded mode
+if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
+    # Building as standalone project
+    project(smartbotic-database VERSION ${SMARTBOTIC_DB_VERSION} LANGUAGES CXX C)
+    set(SMARTBOTIC_DB_STANDALONE ON)
+else()
+    # Building as embedded submodule
+    set(SMARTBOTIC_DB_STANDALONE OFF)
+endif()
+
+# C++ settings
+set(CMAKE_CXX_STANDARD 20)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+set(CMAKE_CXX_EXTENSIONS OFF)
+
+# Build options
+option(SMARTBOTIC_DB_BUILD_CLIENT "Build the database client library" ON)
+option(SMARTBOTIC_DB_BUILD_SERVICE "Build the database service executable" ${SMARTBOTIC_DB_STANDALONE})
+
+# Find required packages
+find_package(Protobuf REQUIRED)
+find_package(gRPC CONFIG QUIET)
+if(NOT gRPC_FOUND)
+    find_package(PkgConfig REQUIRED)
+    pkg_check_modules(GRPC REQUIRED grpc++)
+endif()
+find_package(nlohmann_json 3.2.0 QUIET)
+if(NOT nlohmann_json_FOUND)
+    find_package(PkgConfig REQUIRED)
+    pkg_check_modules(NLOHMANN_JSON REQUIRED nlohmann_json)
+endif()
+
+# Find spdlog
+find_package(spdlog QUIET)
+if(NOT spdlog_FOUND)
+    find_package(PkgConfig REQUIRED)
+    pkg_check_modules(SPDLOG REQUIRED spdlog)
+endif()
+
+# Optional: systemd support
+find_package(PkgConfig)
+if(PKG_CONFIG_FOUND)
+    pkg_check_modules(SYSTEMD QUIET libsystemd)
+endif()
+
+# Proto compilation
+add_subdirectory(proto)
+
+# Client library
+if(SMARTBOTIC_DB_BUILD_CLIENT)
+    add_subdirectory(client)
+endif()
+
+# Service executable
+if(SMARTBOTIC_DB_BUILD_SERVICE)
+    add_subdirectory(service)
+endif()
+
+# Print configuration summary
+if(SMARTBOTIC_DB_STANDALONE)
+    message(STATUS "")
+    message(STATUS "Smartbotic Database ${SMARTBOTIC_DB_VERSION}")
+    message(STATUS "  Build client library: ${SMARTBOTIC_DB_BUILD_CLIENT}")
+    message(STATUS "  Build service:        ${SMARTBOTIC_DB_BUILD_SERVICE}")
+    message(STATUS "  Systemd support:      ${SYSTEMD_FOUND}")
+    message(STATUS "")
+endif()

README.md

@@ -0,0 +1,112 @@
+# Smartbotic Database
+
+A standalone key-value document database with gRPC API, designed for microservice architectures.
+
+## Features
+
+- **Document Storage** - JSON document store with collections
+- **Health Check** - gRPC `HealthCheck()` RPC for service status verification
+- **Replication** - Multi-master replication via `StorageReplication` gRPC service
+- **Persistence** - WAL (Write-Ahead Log) + snapshots with LZ4 compression
+- **Encryption** - Field-level AES-256-GCM encryption for sensitive data
+- **Events** - Pub/sub event subscription for real-time updates
+- **File Storage** - Binary file management with streaming support
+- **Migrations** - Declarative JSON-based schema migrations
+
+## Architecture
+
+```
+smartbotic-database/
+├── client/           # Client library (injectable into projects)
+├── service/          # Database service (standalone deployable)
+├── proto/            # gRPC protocol buffer definitions
+└── packaging/        # Docker build and systemd files
+```
+
+## Building
+
+### Standalone Build
+
+```bash
+cmake -B build -G Ninja
+ninja -C build
+```
+
+### As Embedded Submodule
+
+When used as a submodule, configure build options:
+
+```cmake
+set(SMARTBOTIC_DB_BUILD_CLIENT ON)   # Always need client
+set(SMARTBOTIC_DB_BUILD_SERVICE ON)  # Optional: include service binary
+add_subdirectory(external/smartbotic-database)
+```
+
+## Configuration
+
+```json
+{
+  "database": {
+    "bind_address": "0.0.0.0",
+    "rpc_port": 9004,
+    "data_directory": "/var/lib/smartbotic/database",
+    "migrations": {
+      "enabled": true,
+      "directory": "/etc/smartbotic/migrations",
+      "auto_apply": true
+    },
+    "persistence": {
+      "wal_sync_interval_ms": 100,
+      "snapshot_interval_sec": 3600
+    },
+    "encryption": {
+      "enabled": true,
+      "auto_generate_key": true
+    },
+    "replication": {
+      "enabled": false,
+      "peers": []
+    }
+  }
+}
+```
+
+## Migrations
+
+Create JSON migration files in the migrations directory:
+
+```json
+{
+  "version": "001",
+  "name": "create_users",
+  "description": "Create users collection",
+  "operations": [
+    {
+      "type": "create_collection",
+      "collection": "users",
+      "options": {"encrypted": true}
+    }
+  ]
+}
+```
+
+Supported operations:
+- `create_collection` - Create collection if not exists
+- `drop_collection` - Drop collection (requires `confirm: true`)
+- `insert` - Insert document (fails if exists)
+- `insert_if_not_exists` - Insert only if ID doesn't exist
+- `upsert` - Insert or update document
+- `update` - Update existing document
+- `delete` - Delete document
+
+## CMake Targets
+
+| Target | Description |
+|--------|-------------|
+| `smartbotic_db_proto` | Generated protobuf/gRPC code |
+| `smartbotic-db-client` | Client library |
+| `smartbotic-database` | Service executable |
+
+## License
+
+Proprietary - Smartbotics AI

VERSION

@@ -0,0 +1 @@
+1.0.0

client/CMakeLists.txt

@@ -0,0 +1,31 @@
+# Smartbotic Database Client Library
+
+add_library(smartbotic-db-client STATIC
+    src/client.cpp
+)
+
+target_include_directories(smartbotic-db-client PUBLIC
+    $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
+    $<INSTALL_INTERFACE:include>
+)
+
+# Link dependencies
+target_link_libraries(smartbotic-db-client PUBLIC
+    smartbotic_db_proto
+)
+
+if(TARGET nlohmann_json::nlohmann_json)
+    target_link_libraries(smartbotic-db-client PUBLIC nlohmann_json::nlohmann_json)
+else()
+    target_include_directories(smartbotic-db-client PUBLIC ${NLOHMANN_JSON_INCLUDE_DIRS})
+endif()
+
+if(TARGET spdlog::spdlog)
+    target_link_libraries(smartbotic-db-client PUBLIC spdlog::spdlog)
+else()
+    target_link_libraries(smartbotic-db-client PUBLIC ${SPDLOG_LIBRARIES})
+    target_include_directories(smartbotic-db-client PUBLIC ${SPDLOG_INCLUDE_DIRS})
+endif()
+
+# Export alias
+add_library(smartbotic::db-client ALIAS smartbotic-db-client)

client/include/smartbotic/database/client.hpp

@@ -0,0 +1,186 @@
+#pragma once
+
+#include <nlohmann/json.hpp>
+
+#include <functional>
+#include <memory>
+#include <optional>
+#include <string>
+#include <vector>
+
+namespace smartbotic::database {
+
+/**
+ * Client for the Database service.
+ * Provides a simple interface for other services to interact with the database.
+ * Uses PIMPL to hide gRPC implementation details.
+ */
+class Client {
+public:
+    struct Config {
+        std::string address = "localhost:9004";
+        uint32_t timeoutMs = 5000;
+        uint32_t maxRetries = 3;
+    };
+
+    explicit Client(Config config);
+    ~Client();
+
+    // Move-only
+    Client(Client&&) noexcept;
+    Client& operator=(Client&&) noexcept;
+    Client(const Client&) = delete;
+    Client& operator=(const Client&) = delete;
+
+    /**
+     * Connect to the database service.
+     */
+    bool connect();
+
+    /**
+     * Check if connected.
+     */
+    [[nodiscard]] bool isConnected() const;
+
+    // ===== Document Operations =====
+
+    /**
+     * Insert a document.
+     * @param actor User ID performing the operation (for audit trail)
+     * @return Document ID
+     */
+    std::string insert(const std::string& collection, const nlohmann::json& data,
+                      const std::string& id = "", uint32_t ttlSeconds = 0,
+                      const std::string& actor = "");
+
+    /**
+     * Get a document by ID.
+     */
+    [[nodiscard]] std::optional<nlohmann::json> get(const std::string& collection, const std::string& id);
+
+    /**
+     * Update a document.
+     * @param actor User ID performing the operation (for audit trail)
+     */
+    bool update(const std::string& collection, const std::string& id, const nlohmann::json& data,
+                const std::string& actor = "");
+
+    /**
+     * Update with optimistic locking.
+     * @param actor User ID performing the operation (for audit trail)
+     */
+    bool updateIfVersion(const std::string& collection, const std::string& id,
+                        const nlohmann::json& data, uint64_t expectedVersion,
+                        const std::string& actor = "");
+
+    /**
+     * Upsert a document.
+     * @param actor User ID performing the operation (for audit trail)
+     * @return {id, isNew}
+     */
+    std::pair<std::string, bool> upsert(const std::string& collection, const nlohmann::json& data,
+                                        const std::string& id = "", uint32_t ttlSeconds = 0,
+                                        const std::string& actor = "");
+
+    /**
+     * Delete a document.
+     */
+    bool remove(const std::string& collection, const std::string& id);
+
+    /**
+     * Check if a document exists.
+     */
+    [[nodiscard]] bool exists(const std::string& collection, const std::string& id);
+
+    // ===== Query Operations =====
+
+    struct QueryOptions {
+        std::vector<std::pair<std::string, nlohmann::json>> filters;  // field, value pairs (EQ only)
+        std::string sortField;
+        bool sortDescending = false;
+        uint32_t limit = 100;
+        uint32_t offset = 0;
+    };
+
+    /**
+     * Find documents matching a query.
+     */
+    [[nodiscard]] std::vector<nlohmann::json> find(const std::string& collection,
+                                                   const QueryOptions& options);
+
+    /**
+     * Find all documents in a collection (no filters).
+     */
+    [[nodiscard]] std::vector<nlohmann::json> find(const std::string& collection);
+
+    /**
+     * Count documents matching filters.
+     */
+    [[nodiscard]] uint64_t count(const std::string& collection,
+                                 const std::vector<std::pair<std::string, nlohmann::json>>& filters);
+
+    /**
+     * Count all documents in a collection.
+     */
+    [[nodiscard]] uint64_t count(const std::string& collection);
+
+    // ===== Set Operations (Redis Compatibility) =====
+
+    bool setAdd(const std::string& collection, const std::string& setId, const std::string& member);
+    bool setRemove(const std::string& collection, const std::string& setId, const std::string& member);
+    [[nodiscard]] std::vector<std::string> setMembers(const std::string& collection, const std::string& setId);
+    [[nodiscard]] bool setIsMember(const std::string& collection, const std::string& setId, const std::string& member);
+
+    // ===== Collection Management =====
+
+    bool createCollection(const std::string& name, uint32_t defaultTtlSeconds = 0, bool encrypted = false);
+    bool dropCollection(const std::string& name);
+    [[nodiscard]] std::vector<std::string> listCollections();
+
+    struct CollectionInfo {
+        std::string name;
+        uint64_t documentCount = 0;
+        uint64_t sizeBytes = 0;
+        uint32_t defaultTtlSeconds = 0;
+        bool encrypted = false;
+        uint64_t createdAt = 0;
+        uint64_t updatedAt = 0;
+    };
+
+    /**
+     * Get collection info including size and document count.
+     */
+    [[nodiscard]] std::optional<CollectionInfo> getCollectionInfo(const std::string& name);
+
+    // ===== Event Subscription =====
+
+    using EventCallback = std::function<void(const std::string& collection,
+                                             const std::string& id,
+                                             const std::string& eventType,
+                                             const std::optional<nlohmann::json>& data)>;
+
+    /**
+     * Subscribe to database events.
+     * @return Subscription handle (call reset() to unsubscribe)
+     */
+    std::shared_ptr<void> subscribe(const std::vector<std::string>& collections, EventCallback callback);
+
+    // ===== Health =====
+
+    struct HealthInfo {
+        bool healthy = false;
+        uint64_t uptimeMs = 0;
+        uint64_t documentCount = 0;
+        uint64_t memoryUsedBytes = 0;
+        uint64_t walSizeBytes = 0;
+    };
+
+    [[nodiscard]] bool healthCheck();
+    [[nodiscard]] std::optional<HealthInfo> getHealthInfo();
+
+private:
+    class Impl;
+    std::unique_ptr<Impl> impl_;
+};
+
+} // namespace smartbotic::database

client/include/smartbotic/database/document.hpp

@@ -0,0 +1,49 @@
+#pragma once
+
+// This header provides document types for client usage.
+// It's a simplified version focused on client-side operations.
+
+#include <nlohmann/json.hpp>
+
+#include <optional>
+#include <string>
+#include <vector>
+
+namespace smartbotic::database {
+
+/**
+ * Query options for find operations.
+ */
+struct QueryOptions {
+    std::vector<std::pair<std::string, nlohmann::json>> filters;  // field, value pairs
+    std::string sortField;
+    bool sortDescending = false;
+    uint32_t limit = 100;
+    uint32_t offset = 0;
+};
+
+/**
+ * Collection information.
+ */
+struct CollectionInfo {
+    std::string name;
+    uint64_t documentCount = 0;
+    uint64_t sizeBytes = 0;
+    uint32_t defaultTtlSeconds = 0;
+    bool encrypted = false;
+    uint64_t createdAt = 0;
+    uint64_t updatedAt = 0;
+};
+
+/**
+ * Health information.
+ */
+struct HealthInfo {
+    bool healthy = false;
+    uint64_t uptimeMs = 0;
+    uint64_t documentCount = 0;
+    uint64_t memoryUsedBytes = 0;
+    uint64_t walSizeBytes = 0;
+};
+
+} // namespace smartbotic::database

client/src/client.cpp

@@ -0,0 +1,733 @@
+#include "smartbotic/database/client.hpp"
+
+#include <database.grpc.pb.h>
+
+#include <grpcpp/grpcpp.h>
+#include <spdlog/spdlog.h>
+
+#include <atomic>
+#include <mutex>
+#include <thread>
+
+namespace smartbotic::database {
+
+// ===== PIMPL Implementation =====
+
+class Client::Impl {
+public:
+    explicit Impl(Config config) : config_(std::move(config)) {}
+
+    ~Impl() {
+        disconnect();
+    }
+
+    bool connect() {
+        try {
+            auto channelArgs = grpc::ChannelArguments();
+            // Use longer keepalive intervals to avoid "too_many_pings" errors from server
+            channelArgs.SetInt(GRPC_ARG_KEEPALIVE_TIME_MS, 60000);  // 60 seconds
+            channelArgs.SetInt(GRPC_ARG_KEEPALIVE_TIMEOUT_MS, 20000);  // 20 seconds
+            channelArgs.SetInt(GRPC_ARG_KEEPALIVE_PERMIT_WITHOUT_CALLS, 0);  // Only ping when there are active calls
+            // Set max message size to match server (100MB for file uploads)
+            channelArgs.SetMaxReceiveMessageSize(100 * 1024 * 1024);
+            channelArgs.SetMaxSendMessageSize(100 * 1024 * 1024);
+
+            channel_ = grpc::CreateCustomChannel(
+                config_.address,
+                grpc::InsecureChannelCredentials(),
+                channelArgs
+            );
+
+            stub_ = smartbotic::databasepb::DatabaseService::NewStub(channel_);
+            connected_ = true;
+
+            spdlog::info("Database client connected to {}", config_.address);
+            return true;
+        } catch (const std::exception& e) {
+            spdlog::error("Database client connection failed: {}", e.what());
+            return false;
+        }
+    }
+
+    void disconnect() {
+        connected_ = false;
+        stub_.reset();
+        channel_.reset();
+    }
+
+    bool isConnected() const {
+        if (!connected_ || !channel_) {
+            return false;
+        }
+        // Check if channel is in a usable state (not failed or shutdown)
+        auto state = channel_->GetState(false);
+        return state == GRPC_CHANNEL_READY ||
+               state == GRPC_CHANNEL_IDLE ||
+               state == GRPC_CHANNEL_CONNECTING;
+    }
+
+    // ===== Document Operations =====
+
+    std::string insert(const std::string& collection, const nlohmann::json& data,
+                      const std::string& id, uint32_t ttlSeconds,
+                      const std::string& actor) {
+        smartbotic::databasepb::InsertRequest request;
+        request.set_collection(collection);
+        request.set_data(data.dump());
+        if (!id.empty()) {
+            request.set_id(id);
+        }
+        if (ttlSeconds > 0) {
+            request.set_ttl_seconds(ttlSeconds);
+        }
+        if (!actor.empty()) {
+            request.set_actor(actor);
+        }
+
+        smartbotic::databasepb::InsertResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->Insert(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::insert failed: {}", status.error_message());
+            throw std::runtime_error(status.error_message());
+        }
+
+        return response.id();
+    }
+
+    std::optional<nlohmann::json> get(const std::string& collection, const std::string& id) {
+        smartbotic::databasepb::GetRequest request;
+        request.set_collection(collection);
+        request.set_id(id);
+
+        smartbotic::databasepb::GetResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->Get(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::get failed: {}", status.error_message());
+            return std::nullopt;
+        }
+
+        if (!response.found()) {
+            return std::nullopt;
+        }
+
+        auto json = nlohmann::json::parse(response.document().data());
+        json["_id"] = response.document().id();  // Include document ID in returned data
+        return json;
+    }
+
+    bool update(const std::string& collection, const std::string& id, const nlohmann::json& data,
+                const std::string& actor) {
+        smartbotic::databasepb::UpdateRequest request;
+        request.set_collection(collection);
+        request.set_id(id);
+        request.set_data(data.dump());
+        if (!actor.empty()) {
+            request.set_actor(actor);
+        }
+
+        smartbotic::databasepb::UpdateResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->Update(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::update failed: {}", status.error_message());
+            return false;
+        }
+
+        return response.success();
+    }
+
+    bool updateIfVersion(const std::string& collection, const std::string& id,
+                        const nlohmann::json& data, uint64_t expectedVersion,
+                        const std::string& actor) {
+        smartbotic::databasepb::UpdateRequest request;
+        request.set_collection(collection);
+        request.set_id(id);
+        request.set_data(data.dump());
+        request.set_expected_version(expectedVersion);
+        if (!actor.empty()) {
+            request.set_actor(actor);
+        }
+
+        smartbotic::databasepb::UpdateResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->Update(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::updateIfVersion failed: {}", status.error_message());
+            return false;
+        }
+
+        return response.success();
+    }
+
+    std::pair<std::string, bool> upsert(const std::string& collection, const nlohmann::json& data,
+                                        const std::string& id, uint32_t ttlSeconds,
+                                        const std::string& actor) {
+        smartbotic::databasepb::UpsertRequest request;
+        request.set_collection(collection);
+        request.set_data(data.dump());
+        if (!id.empty()) {
+            request.set_id(id);
+        }
+        if (ttlSeconds > 0) {
+            request.set_ttl_seconds(ttlSeconds);
+        }
+        if (!actor.empty()) {
+            request.set_actor(actor);
+        }
+
+        smartbotic::databasepb::UpsertResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->Upsert(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::upsert failed: {}", status.error_message());
+            throw std::runtime_error(status.error_message());
+        }
+
+        return {response.id(), response.inserted()};
+    }
+
+    bool remove(const std::string& collection, const std::string& id) {
+        smartbotic::databasepb::DeleteRequest request;
+        request.set_collection(collection);
+        request.set_id(id);
+
+        smartbotic::databasepb::DeleteResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->Delete(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::remove failed: {}", status.error_message());
+            return false;
+        }
+
+        return response.deleted();
+    }
+
+    bool exists(const std::string& collection, const std::string& id) {
+        smartbotic::databasepb::ExistsRequest request;
+        request.set_collection(collection);
+        request.set_id(id);
+
+        smartbotic::databasepb::ExistsResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->Exists(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::exists failed: {}", status.error_message());
+            return false;
+        }
+
+        return response.exists();
+    }
+
+    // ===== Query Operations =====
+
+    std::vector<nlohmann::json> find(const std::string& collection,
+                                     const Client::QueryOptions& options) {
+        smartbotic::databasepb::FindRequest request;
+        request.set_collection(collection);
+
+        // Set filters
+        for (const auto& [field, value] : options.filters) {
+            auto* filter = request.add_filters();
+            filter->set_field(field);
+            filter->set_value(value.dump());
+            // Use SEARCH op for _search field, EQ for others
+            if (field == "_search") {
+                filter->set_op(smartbotic::databasepb::FILTER_OP_SEARCH);
+            } else {
+                filter->set_op(smartbotic::databasepb::FILTER_OP_EQ);
+            }
+        }
+
+        // Set sorting
+        if (!options.sortField.empty()) {
+            auto* sort = request.mutable_sort();
+            sort->set_field(options.sortField);
+            sort->set_descending(options.sortDescending);
+        }
+
+        // Set pagination
+        request.set_limit(options.limit);
+        request.set_offset(options.offset);
+
+        smartbotic::databasepb::FindResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->Find(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::find failed: {}", status.error_message());
+            return {};
+        }
+
+        std::vector<nlohmann::json> results;
+        results.reserve(response.documents_size());
+        for (const auto& doc : response.documents()) {
+            auto json = nlohmann::json::parse(doc.data());
+            json["_id"] = doc.id();  // Include document ID in returned data
+            json["_created_at"] = doc.created_at();  // Creation timestamp (ms)
+            json["_updated_at"] = doc.updated_at();  // Last update timestamp (ms)
+            json["_created_by"] = doc.created_by();  // User ID who created
+            json["_updated_by"] = doc.updated_by();  // User ID who last updated
+            results.push_back(json);
+        }
+
+        return results;
+    }
+
+    uint64_t count(const std::string& collection,
+                   const std::vector<std::pair<std::string, nlohmann::json>>& filters) {
+        smartbotic::databasepb::CountRequest request;
+        request.set_collection(collection);
+
+        for (const auto& [field, value] : filters) {
+            auto* filter = request.add_filters();
+            filter->set_field(field);
+            filter->set_value(value.dump());
+            // Use SEARCH op for _search field, EQ for others
+            if (field == "_search") {
+                filter->set_op(smartbotic::databasepb::FILTER_OP_SEARCH);
+            } else {
+                filter->set_op(smartbotic::databasepb::FILTER_OP_EQ);
+            }
+        }
+
+        smartbotic::databasepb::CountResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->Count(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::count failed: {}", status.error_message());
+            return 0;
+        }
+
+        return response.count();
+    }
+
+    // ===== Set Operations =====
+
+    bool setAdd(const std::string& collection, const std::string& setId, const std::string& member) {
+        smartbotic::databasepb::SetAddRequest request;
+        request.set_collection(collection);
+        request.set_set_id(setId);
+        request.set_member(member);
+
+        smartbotic::databasepb::SetAddResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->SetAdd(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::setAdd failed: {}", status.error_message());
+            return false;
+        }
+
+        return response.added();
+    }
+
+    bool setRemove(const std::string& collection, const std::string& setId, const std::string& member) {
+        smartbotic::databasepb::SetRemoveRequest request;
+        request.set_collection(collection);
+        request.set_set_id(setId);
+        request.set_member(member);
+
+        smartbotic::databasepb::SetRemoveResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->SetRemove(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::setRemove failed: {}", status.error_message());
+            return false;
+        }
+
+        return response.removed();
+    }
+
+    std::vector<std::string> setMembers(const std::string& collection, const std::string& setId) {
+        smartbotic::databasepb::SetMembersRequest request;
+        request.set_collection(collection);
+        request.set_set_id(setId);
+
+        smartbotic::databasepb::SetMembersResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->SetMembers(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::setMembers failed: {}", status.error_message());
+            return {};
+        }
+
+        return {response.members().begin(), response.members().end()};
+    }
+
+    bool setIsMember(const std::string& collection, const std::string& setId, const std::string& member) {
+        smartbotic::databasepb::SetIsMemberRequest request;
+        request.set_collection(collection);
+        request.set_set_id(setId);
+        request.set_member(member);
+
+        smartbotic::databasepb::SetIsMemberResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->SetIsMember(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::setIsMember failed: {}", status.error_message());
+            return false;
+        }
+
+        return response.is_member();
+    }
+
+    // ===== Collection Management =====
+
+    bool createCollection(const std::string& name, uint32_t defaultTtlSeconds, bool encrypted) {
+        smartbotic::databasepb::CreateCollectionRequest request;
+        request.set_name(name);
+        auto* options = request.mutable_options();
+        if (defaultTtlSeconds > 0) {
+            options->set_default_ttl_seconds(defaultTtlSeconds);
+        }
+        options->set_encrypted(encrypted);
+
+        smartbotic::databasepb::CreateCollectionResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->CreateCollection(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::createCollection failed: {}", status.error_message());
+            return false;
+        }
+
+        return response.created();
+    }
+
+    bool dropCollection(const std::string& name) {
+        smartbotic::databasepb::DropCollectionRequest request;
+        request.set_name(name);
+
+        smartbotic::databasepb::DropCollectionResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->DropCollection(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::dropCollection failed: {}", status.error_message());
+            return false;
+        }
+
+        return response.dropped();
+    }
+
+    std::vector<std::string> listCollections() {
+        smartbotic::databasepb::ListCollectionsRequest request;
+
+        smartbotic::databasepb::ListCollectionsResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->ListCollections(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::listCollections failed: {}", status.error_message());
+            return {};
+        }
+
+        return {response.names().begin(), response.names().end()};
+    }
+
+    std::optional<Client::CollectionInfo> getCollectionInfo(const std::string& name) {
+        smartbotic::databasepb::GetCollectionInfoRequest request;
+        request.set_name(name);
+
+        smartbotic::databasepb::GetCollectionInfoResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->GetCollectionInfo(&context, request, &response);
+        if (!status.ok()) {
+            spdlog::error("Client::getCollectionInfo failed: {}", status.error_message());
+            return std::nullopt;
+        }
+
+        if (!response.found()) {
+            return std::nullopt;
+        }
+
+        const auto& info = response.info();
+        Client::CollectionInfo result;
+        result.name = info.name();
+        result.documentCount = info.document_count();
+        result.sizeBytes = info.size_bytes();
+        result.defaultTtlSeconds = info.options().default_ttl_seconds();
+        result.encrypted = info.options().encrypted();
+        result.createdAt = info.created_at();
+        result.updatedAt = info.updated_at();
+        return result;
+    }
+
+    // ===== Event Subscription =====
+
+    class SubscriptionHandle {
+    public:
+        SubscriptionHandle(std::shared_ptr<grpc::ClientContext> ctx,
+                          std::unique_ptr<grpc::ClientReader<smartbotic::databasepb::DatabaseEvent>> reader,
+                          std::thread readerThread)
+            : context_(std::move(ctx))
+            , reader_(std::move(reader))
+            , readerThread_(std::move(readerThread))
+            , active_(true) {}
+
+        ~SubscriptionHandle() {
+            cancel();
+        }
+
+        void cancel() {
+            if (active_.exchange(false)) {
+                context_->TryCancel();
+                if (readerThread_.joinable()) {
+                    readerThread_.join();
+                }
+            }
+        }
+
+    private:
+        std::shared_ptr<grpc::ClientContext> context_;
+        std::unique_ptr<grpc::ClientReader<smartbotic::databasepb::DatabaseEvent>> reader_;
+        std::thread readerThread_;
+        std::atomic<bool> active_;
+    };
+
+    std::shared_ptr<void> subscribe(const std::vector<std::string>& collections,
+                                    Client::EventCallback callback) {
+        auto context = std::make_shared<grpc::ClientContext>();
+
+        smartbotic::databasepb::SubscribeRequest request;
+        for (const auto& coll : collections) {
+            request.add_collections(coll);
+        }
+        request.set_include_data(true);
+
+        auto reader = stub_->Subscribe(context.get(), request);
+
+        // Create reader thread
+        auto readerThread = std::thread([reader = reader.get(), callback = std::move(callback)]() {
+            smartbotic::databasepb::DatabaseEvent event;
+            while (reader->Read(&event)) {
+                std::optional<nlohmann::json> data;
+                if (!event.data().empty()) {
+                    data = nlohmann::json::parse(event.data());
+                }
+
+                // Convert proto event type to string
+                std::string eventType;
+                switch (event.type()) {
+                    case smartbotic::databasepb::EVENT_INSERT: eventType = "insert"; break;
+                    case smartbotic::databasepb::EVENT_UPDATE: eventType = "update"; break;
+                    case smartbotic::databasepb::EVENT_DELETE: eventType = "delete"; break;
+                    case smartbotic::databasepb::EVENT_EXPIRE: eventType = "expire"; break;
+                    case smartbotic::databasepb::EVENT_INVALIDATE: eventType = "invalidate"; break;
+                    default: eventType = "unknown"; break;
+                }
+
+                callback(
+                    event.collection(),
+                    event.document_id(),
+                    eventType,
+                    data
+                );
+            }
+        });
+
+        return std::make_shared<SubscriptionHandle>(
+            std::move(context),
+            std::move(reader),
+            std::move(readerThread)
+        );
+    }
+
+    // ===== Health =====
+
+    bool healthCheck() {
+        smartbotic::databasepb::HealthCheckRequest request;
+
+        smartbotic::databasepb::HealthCheckResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->HealthCheck(&context, request, &response);
+        if (!status.ok()) {
+            return false;
+        }
+
+        return response.healthy();
+    }
+
+    std::optional<Client::HealthInfo> getHealthInfo() {
+        smartbotic::databasepb::HealthCheckRequest request;
+
+        smartbotic::databasepb::HealthCheckResponse response;
+        grpc::ClientContext context;
+        setDeadline(context);
+
+        auto status = stub_->HealthCheck(&context, request, &response);
+        if (!status.ok()) {
+            return std::nullopt;
+        }
+
+        Client::HealthInfo info;
+        info.healthy = response.healthy();
+        info.uptimeMs = response.uptime_seconds() * 1000;
+        info.documentCount = response.document_count();
+        info.memoryUsedBytes = response.memory_used_bytes();
+        info.walSizeBytes = response.wal_size_bytes();
+        return info;
+    }
+
+private:
+    void setDeadline(grpc::ClientContext& context) {
+        context.set_deadline(
+            std::chrono::system_clock::now() + std::chrono::milliseconds(config_.timeoutMs)
+        );
+    }
+
+    Config config_;
+    std::shared_ptr<grpc::Channel> channel_;
+    std::unique_ptr<smartbotic::databasepb::DatabaseService::Stub> stub_;
+    std::atomic<bool> connected_{false};
+};
+
+// ===== Client Public Interface Implementation =====
+
+Client::Client(Config config)
+    : impl_(std::make_unique<Impl>(std::move(config))) {}
+
+Client::~Client() = default;
+
+Client::Client(Client&&) noexcept = default;
+Client& Client::operator=(Client&&) noexcept = default;
+
+bool Client::connect() {
+    return impl_->connect();
+}
+
+bool Client::isConnected() const {
+    return impl_->isConnected();
+}
+
+std::string Client::insert(const std::string& collection, const nlohmann::json& data,
+                                  const std::string& id, uint32_t ttlSeconds,
+                                  const std::string& actor) {
+    return impl_->insert(collection, data, id, ttlSeconds, actor);
+}
+
+std::optional<nlohmann::json> Client::get(const std::string& collection, const std::string& id) {
+    return impl_->get(collection, id);
+}
+
+bool Client::update(const std::string& collection, const std::string& id, const nlohmann::json& data,
+                           const std::string& actor) {
+    return impl_->update(collection, id, data, actor);
+}
+
+bool Client::updateIfVersion(const std::string& collection, const std::string& id,
+                                    const nlohmann::json& data, uint64_t expectedVersion,
+                                    const std::string& actor) {
+    return impl_->updateIfVersion(collection, id, data, expectedVersion, actor);
+}
+
+std::pair<std::string, bool> Client::upsert(const std::string& collection, const nlohmann::json& data,
+                                                    const std::string& id, uint32_t ttlSeconds,
+                                                    const std::string& actor) {
+    return impl_->upsert(collection, data, id, ttlSeconds, actor);
+}
+
+bool Client::remove(const std::string& collection, const std::string& id) {
+    return impl_->remove(collection, id);
+}
+
+bool Client::exists(const std::string& collection, const std::string& id) {
+    return impl_->exists(collection, id);
+}
+
+std::vector<nlohmann::json> Client::find(const std::string& collection,
+                                                 const QueryOptions& options) {
+    return impl_->find(collection, options);
+}
+
+std::vector<nlohmann::json> Client::find(const std::string& collection) {
+    return impl_->find(collection, QueryOptions{});
+}
+
+uint64_t Client::count(const std::string& collection,
+                               const std::vector<std::pair<std::string, nlohmann::json>>& filters) {
+    return impl_->count(collection, filters);
+}
+
+uint64_t Client::count(const std::string& collection) {
+    return impl_->count(collection, {});
+}
+
+bool Client::setAdd(const std::string& collection, const std::string& setId, const std::string& member) {
+    return impl_->setAdd(collection, setId, member);
+}
+
+bool Client::setRemove(const std::string& collection, const std::string& setId, const std::string& member) {
+    return impl_->setRemove(collection, setId, member);
+}
+
+std::vector<std::string> Client::setMembers(const std::string& collection, const std::string& setId) {
+    return impl_->setMembers(collection, setId);
+}
+
+bool Client::setIsMember(const std::string& collection, const std::string& setId, const std::string& member) {
+    return impl_->setIsMember(collection, setId, member);
+}
+
+bool Client::createCollection(const std::string& name, uint32_t defaultTtlSeconds, bool encrypted) {
+    return impl_->createCollection(name, defaultTtlSeconds, encrypted);
+}
+
+bool Client::dropCollection(const std::string& name) {
+    return impl_->dropCollection(name);
+}
+
+std::vector<std::string> Client::listCollections() {
+    return impl_->listCollections();
+}
+
+std::optional<Client::CollectionInfo> Client::getCollectionInfo(const std::string& name) {
+    return impl_->getCollectionInfo(name);
+}
+
+std::shared_ptr<void> Client::subscribe(const std::vector<std::string>& collections, EventCallback callback) {
+    return impl_->subscribe(collections, std::move(callback));
+}
+
+bool Client::healthCheck() {
+    return impl_->healthCheck();
+}
+
+std::optional<Client::HealthInfo> Client::getHealthInfo() {
+    return impl_->getHealthInfo();
+}
+
+} // namespace smartbotic::database

packaging/build.sh

@@ -0,0 +1,106 @@
+#!/bin/bash
+# Build smartbotic-database package
+
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT_DIR="$(dirname "$SCRIPT_DIR")"
+
+# Read version from VERSION file
+VERSION=$(cat "$ROOT_DIR/VERSION" | tr -d '[:space:]')
+PLATFORM="debian13"
+
+# Parse arguments
+REBUILD_BASE=false
+CLEAR_CACHE=false
+NO_CACHE=false
+BUILD_VERSION=""
+
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        --rebuild-base)
+            REBUILD_BASE=true
+            shift
+            ;;
+        --clear-cache)
+            CLEAR_CACHE=true
+            shift
+            ;;
+        --no-cache)
+            NO_CACHE=true
+            shift
+            ;;
+        *)
+            BUILD_VERSION="$1"
+            shift
+            ;;
+    esac
+done
+
+# Use provided version or default from VERSION file
+VERSION="${BUILD_VERSION:-$VERSION}"
+
+echo "Building smartbotic-database version $VERSION"
+
+# Create dist directory
+mkdir -p "$SCRIPT_DIR/dist"
+
+# Build using Docker if Dockerfile exists
+if [ -f "$SCRIPT_DIR/Dockerfile.build" ]; then
+    DOCKER_BUILD_ARGS=""
+
+    if [ "$NO_CACHE" = true ]; then
+        DOCKER_BUILD_ARGS="--no-cache"
+    fi
+
+    docker buildx build \
+        $DOCKER_BUILD_ARGS \
+        -f "$SCRIPT_DIR/Dockerfile.build" \
+        --build-arg BUILD_VERSION="$VERSION" \
+        --build-arg BUILD_GIT_COMMIT="$(git -C "$ROOT_DIR" rev-parse HEAD 2>/dev/null || echo 'unknown')" \
+        --build-arg BUILD_GIT_BRANCH="$(git -C "$ROOT_DIR" rev-parse --abbrev-ref HEAD 2>/dev/null || echo 'unknown')" \
+        --target packages \
+        --output "type=local,dest=$SCRIPT_DIR/dist/" \
+        "$ROOT_DIR"
+else
+    # Fall back to local build
+    echo "No Dockerfile.build found, building locally..."
+
+    BUILD_DIR="$ROOT_DIR/build"
+    PACKAGE_DIR="$SCRIPT_DIR/dist/smartbotic-database-$VERSION-$PLATFORM"
+
+    # Configure and build
+    cmake -B "$BUILD_DIR" -G Ninja \
+        -DCMAKE_BUILD_TYPE=Release \
+        -DSMARTBOTIC_DB_BUILD_SERVICE=ON \
+        -DSMARTBOTIC_DB_BUILD_CLIENT=ON \
+        "$ROOT_DIR"
+
+    ninja -C "$BUILD_DIR"
+
+    # Create package structure
+    rm -rf "$PACKAGE_DIR"
+    mkdir -p "$PACKAGE_DIR/bin"
+    mkdir -p "$PACKAGE_DIR/etc/systemd/system"
+
+    # Copy binary
+    cp "$BUILD_DIR/service/smartbotic-database" "$PACKAGE_DIR/bin/"
+
+    # Copy config
+    cp "$ROOT_DIR/service/config/database.json.example" "$PACKAGE_DIR/etc/smartbotic-database.json"
+
+    # Copy systemd unit
+    if [ -f "$SCRIPT_DIR/systemd/smartbotic-database.service" ]; then
+        cp "$SCRIPT_DIR/systemd/smartbotic-database.service" "$PACKAGE_DIR/etc/systemd/system/"
+    fi
+
+    # Create VERSION file
+    echo "$VERSION" > "$PACKAGE_DIR/VERSION"
+
+    # Create tarball
+    cd "$SCRIPT_DIR/dist"
+    tar -czf "smartbotic-database-$VERSION-$PLATFORM.tar.gz" "smartbotic-database-$VERSION-$PLATFORM"
+    rm -rf "smartbotic-database-$VERSION-$PLATFORM"
+
+    echo "Package created: $SCRIPT_DIR/dist/smartbotic-database-$VERSION-$PLATFORM.tar.gz"
+fi

packaging/systemd/smartbotic-database.service

@@ -0,0 +1,29 @@
+[Unit]
+Description=Smartbotic Database Service
+Documentation=https://git.smartbotics.ai/fszontagh/smartbotic-database
+After=network.target
+
+[Service]
+Type=notify
+ExecStart=/opt/smartbotic/bin/smartbotic-database --config /etc/smartbotic/database.json
+Restart=on-failure
+RestartSec=5
+TimeoutStartSec=30
+WatchdogSec=60
+
+# Security hardening
+NoNewPrivileges=yes
+ProtectSystem=strict
+ProtectHome=read-only
+PrivateTmp=yes
+ReadWritePaths=/var/lib/smartbotic/database
+
+# Resource limits
+LimitNOFILE=65536
+LimitNPROC=4096
+
+# Environment
+Environment=LOG_LEVEL=info
+
+[Install]
+WantedBy=multi-user.target

proto/CMakeLists.txt

@@ -0,0 +1,74 @@
+# Proto compilation for smartbotic-database
+
+# Get the protobuf compiler and gRPC plugin
+find_program(PROTOC protoc REQUIRED)
+find_program(GRPC_CPP_PLUGIN grpc_cpp_plugin REQUIRED)
+
+# Get the include directory for well-known types
+get_target_property(PROTOBUF_INCLUDE_DIR protobuf::libprotobuf INTERFACE_INCLUDE_DIRECTORIES)
+if(NOT PROTOBUF_INCLUDE_DIR)
+    # Fallback for system protobuf
+    set(PROTOBUF_INCLUDE_DIR "/usr/include")
+endif()
+
+# Create output directory
+set(PROTO_OUTPUT_DIR "${CMAKE_CURRENT_BINARY_DIR}/generated")
+file(MAKE_DIRECTORY ${PROTO_OUTPUT_DIR})
+
+# Proto file
+set(PROTO_FILE "${CMAKE_CURRENT_SOURCE_DIR}/database.proto")
+
+# Generated files
+set(PROTO_SRCS "${PROTO_OUTPUT_DIR}/database.pb.cc")
+set(PROTO_HDRS "${PROTO_OUTPUT_DIR}/database.pb.h")
+set(GRPC_SRCS "${PROTO_OUTPUT_DIR}/database.grpc.pb.cc")
+set(GRPC_HDRS "${PROTO_OUTPUT_DIR}/database.grpc.pb.h")
+
+# Generate proto and gRPC files
+add_custom_command(
+    OUTPUT ${PROTO_SRCS} ${PROTO_HDRS} ${GRPC_SRCS} ${GRPC_HDRS}
+    COMMAND ${PROTOC}
+        --proto_path=${CMAKE_CURRENT_SOURCE_DIR}
+        --proto_path=${PROTOBUF_INCLUDE_DIR}
+        --cpp_out=${PROTO_OUTPUT_DIR}
+        --grpc_out=${PROTO_OUTPUT_DIR}
+        --plugin=protoc-gen-grpc=${GRPC_CPP_PLUGIN}
+        ${PROTO_FILE}
+    DEPENDS ${PROTO_FILE}
+    COMMENT "Generating protobuf/gRPC code for database.proto"
+    VERBATIM
+)
+
+# Create proto library
+add_library(smartbotic_db_proto STATIC
+    ${PROTO_SRCS}
+    ${GRPC_SRCS}
+)
+
+target_include_directories(smartbotic_db_proto PUBLIC
+    $<BUILD_INTERFACE:${PROTO_OUTPUT_DIR}>
+    $<INSTALL_INTERFACE:include>
+)
+
+# Suppress warnings in generated code
+target_compile_options(smartbotic_db_proto PRIVATE
+    -Wno-unused-parameter
+    -Wno-array-bounds
+)
+
+# Link dependencies
+if(TARGET gRPC::grpc++)
+    target_link_libraries(smartbotic_db_proto PUBLIC
+        protobuf::libprotobuf
+        gRPC::grpc++
+    )
+else()
+    target_link_libraries(smartbotic_db_proto PUBLIC
+        protobuf
+        ${GRPC_LIBRARIES}
+    )
+    target_include_directories(smartbotic_db_proto PUBLIC ${GRPC_INCLUDE_DIRS})
+endif()
+
+# Export for other targets
+add_library(smartbotic::db_proto ALIAS smartbotic_db_proto)

proto/database.proto

@@ -0,0 +1,544 @@
+// Smartbotic Database Service
+// Document-oriented storage with persistence and replication
+
+syntax = "proto3";
+
+package smartbotic.databasepb;
+
+// Error code range: 6000-6999 for Database errors
+// 6000: Generic database error
+// 6001: Collection not found
+// 6002: Document not found
+// 6003: Document already exists
+// 6004: Version conflict
+// 6005: Invalid query
+// 6006: File not found
+// 6007: File too large
+// 6008: Encryption error
+// 6009: Replication error
+
+// ===== Main Database Service =====
+
+service DatabaseService {
+    // Document operations
+    rpc Insert(InsertRequest) returns (InsertResponse);
+    rpc Get(GetRequest) returns (GetResponse);
+    rpc Update(UpdateRequest) returns (UpdateResponse);
+    rpc Upsert(UpsertRequest) returns (UpsertResponse);
+    rpc Delete(DeleteRequest) returns (DeleteResponse);
+    rpc Exists(ExistsRequest) returns (ExistsResponse);
+
+    // Batch operations
+    rpc BatchInsert(BatchInsertRequest) returns (BatchInsertResponse);
+    rpc BatchGet(BatchGetRequest) returns (BatchGetResponse);
+    rpc BatchDelete(BatchDeleteRequest) returns (BatchDeleteResponse);
+
+    // Query operations
+    rpc Find(FindRequest) returns (FindResponse);
+    rpc Count(CountRequest) returns (CountResponse);
+
+    // Set operations (Redis compatibility)
+    rpc SetAdd(SetAddRequest) returns (SetAddResponse);
+    rpc SetRemove(SetRemoveRequest) returns (SetRemoveResponse);
+    rpc SetMembers(SetMembersRequest) returns (SetMembersResponse);
+    rpc SetIsMember(SetIsMemberRequest) returns (SetIsMemberResponse);
+
+    // Collection management
+    rpc CreateCollection(CreateCollectionRequest) returns (CreateCollectionResponse);
+    rpc DropCollection(DropCollectionRequest) returns (DropCollectionResponse);
+    rpc ListCollections(ListCollectionsRequest) returns (ListCollectionsResponse);
+    rpc GetCollectionInfo(GetCollectionInfoRequest) returns (GetCollectionInfoResponse);
+
+    // File operations
+    rpc UploadFile(stream FileChunk) returns (UploadFileResponse);
+    rpc DownloadFile(DownloadFileRequest) returns (stream FileChunk);
+    rpc DeleteFile(DeleteFileRequest) returns (DeleteFileResponse);
+    rpc GetFileInfo(GetFileInfoRequest) returns (FileInfo);
+    rpc ListFiles(ListFilesRequest) returns (ListFilesResponse);
+
+    // Event subscription
+    rpc Subscribe(SubscribeRequest) returns (stream DatabaseEvent);
+
+    // Health and stats
+    rpc HealthCheck(HealthCheckRequest) returns (HealthCheckResponse);
+    rpc GetStats(GetStatsRequest) returns (GetStatsResponse);
+}
+
+// ===== Replication Service =====
+
+service DatabaseReplication {
+    // Bidirectional streaming for real-time sync
+    rpc SyncStream(stream ReplicationMessage) returns (stream ReplicationMessage);
+
+    // Pull missed entries (for recovery)
+    rpc GetEntriesSince(GetEntriesRequest) returns (stream ReplicationEntry);
+
+    // Get current node state
+    rpc GetNodeState(GetNodeStateRequest) returns (NodeState);
+}
+
+// ===== Document Types =====
+
+message Document {
+    string id = 1;
+    string collection = 2;
+    bytes data = 3;               // JSON-encoded document data
+    uint64 version = 4;
+    uint64 created_at = 5;        // Timestamp in milliseconds
+    uint64 updated_at = 6;
+    uint64 expires_at = 7;        // 0 = no expiration
+    string node_id = 8;           // Origin node for replication
+    bool encrypted = 9;
+    repeated string encrypted_fields = 10;
+    string created_by = 11;       // User ID who created the document
+    string updated_by = 12;       // User ID who last updated the document
+}
+
+// ===== Document Operations =====
+
+message InsertRequest {
+    string collection = 1;
+    bytes data = 2;               // JSON document
+    string id = 3;                // Optional, auto-generated if empty
+    uint32 ttl_seconds = 4;       // 0 = use collection default
+    string actor = 5;             // User ID performing the operation (for audit)
+}
+
+message InsertResponse {
+    string id = 1;
+    uint64 version = 2;
+}
+
+message GetRequest {
+    string collection = 1;
+    string id = 2;
+}
+
+message GetResponse {
+    Document document = 1;
+    bool found = 2;
+}
+
+message UpdateRequest {
+    string collection = 1;
+    string id = 2;
+    bytes data = 3;
+    uint64 expected_version = 4;  // 0 = no version check (force update)
+    string actor = 5;             // User ID performing the operation (for audit)
+}
+
+message UpdateResponse {
+    uint64 new_version = 1;
+    bool success = 2;
+    string error = 3;
+}
+
+message UpsertRequest {
+    string collection = 1;
+    bytes data = 2;
+    string id = 3;
+    uint32 ttl_seconds = 4;
+    string actor = 5;             // User ID performing the operation (for audit)
+}
+
+message UpsertResponse {
+    string id = 1;
+    uint64 version = 2;
+    bool inserted = 3;            // true if inserted, false if updated
+}
+
+message DeleteRequest {
+    string collection = 1;
+    string id = 2;
+}
+
+message DeleteResponse {
+    bool deleted = 1;
+}
+
+message ExistsRequest {
+    string collection = 1;
+    string id = 2;
+}
+
+message ExistsResponse {
+    bool exists = 1;
+}
+
+// ===== Batch Operations =====
+
+message BatchInsertRequest {
+    string collection = 1;
+    repeated BatchInsertItem items = 2;
+    string actor = 3;             // User ID performing the operation (for audit)
+}
+
+message BatchInsertItem {
+    bytes data = 1;
+    string id = 2;
+    uint32 ttl_seconds = 3;
+}
+
+message BatchInsertResponse {
+    repeated string ids = 1;
+    uint32 success_count = 2;
+    uint32 error_count = 3;
+}
+
+message BatchGetRequest {
+    string collection = 1;
+    repeated string ids = 2;
+}
+
+message BatchGetResponse {
+    repeated Document documents = 1;
+}
+
+message BatchDeleteRequest {
+    string collection = 1;
+    repeated string ids = 2;
+}
+
+message BatchDeleteResponse {
+    uint64 deleted_count = 1;
+}
+
+// ===== Query Operations =====
+
+message FindRequest {
+    string collection = 1;
+    repeated Filter filters = 2;
+    Sort sort = 3;
+    uint32 limit = 4;             // Default 100
+    uint32 offset = 5;
+    repeated string projection = 6;  // Fields to include (empty = all)
+}
+
+message Filter {
+    string field = 1;
+    FilterOp op = 2;
+    bytes value = 3;              // JSON-encoded value
+}
+
+enum FilterOp {
+    FILTER_OP_UNSPECIFIED = 0;
+    FILTER_OP_EQ = 1;
+    FILTER_OP_NE = 2;
+    FILTER_OP_GT = 3;
+    FILTER_OP_GTE = 4;
+    FILTER_OP_LT = 5;
+    FILTER_OP_LTE = 6;
+    FILTER_OP_IN = 7;
+    FILTER_OP_CONTAINS = 8;
+    FILTER_OP_EXISTS = 9;
+    FILTER_OP_REGEX = 10;
+    FILTER_OP_SEARCH = 11;  // Full-text search across ID and string fields
+}
+
+message Sort {
+    string field = 1;
+    bool descending = 2;
+}
+
+message FindResponse {
+    repeated Document documents = 1;
+    uint64 total_count = 2;
+    bool has_more = 3;
+}
+
+message CountRequest {
+    string collection = 1;
+    repeated Filter filters = 2;
+}
+
+message CountResponse {
+    uint64 count = 1;
+}
+
+// ===== Set Operations =====
+
+message SetAddRequest {
+    string collection = 1;
+    string set_id = 2;
+    string member = 3;
+}
+
+message SetAddResponse {
+    bool added = 1;               // false if already exists
+}
+
+message SetRemoveRequest {
+    string collection = 1;
+    string set_id = 2;
+    string member = 3;
+}
+
+message SetRemoveResponse {
+    bool removed = 1;
+}
+
+message SetMembersRequest {
+    string collection = 1;
+    string set_id = 2;
+}
+
+message SetMembersResponse {
+    repeated string members = 1;
+}
+
+message SetIsMemberRequest {
+    string collection = 1;
+    string set_id = 2;
+    string member = 3;
+}
+
+message SetIsMemberResponse {
+    bool is_member = 1;
+}
+
+// ===== Collection Management =====
+
+message CreateCollectionRequest {
+    string name = 1;
+    CollectionOptions options = 2;
+}
+
+message CollectionOptions {
+    bool auto_create_id = 1;
+    uint32 default_ttl_seconds = 2;
+    bool encrypted = 3;
+    repeated string sensitive_fields = 4;
+}
+
+message CreateCollectionResponse {
+    bool created = 1;
+}
+
+message DropCollectionRequest {
+    string name = 1;
+}
+
+message DropCollectionResponse {
+    bool dropped = 1;
+}
+
+message ListCollectionsRequest {
+}
+
+message ListCollectionsResponse {
+    repeated string names = 1;
+}
+
+message GetCollectionInfoRequest {
+    string name = 1;
+}
+
+message GetCollectionInfoResponse {
+    CollectionInfo info = 1;
+    bool found = 2;
+}
+
+message CollectionInfo {
+    string name = 1;
+    uint64 document_count = 2;
+    uint64 size_bytes = 3;
+    CollectionOptions options = 4;
+    uint64 created_at = 5;
+    uint64 updated_at = 6;
+}
+
+// ===== File Operations =====
+
+message FileChunk {
+    oneof content {
+        FileMetadata metadata = 1;    // First chunk includes metadata
+        bytes data = 2;               // Subsequent chunks are data
+    }
+}
+
+message FileMetadata {
+    string id = 1;                    // Set by server on upload response
+    string name = 2;
+    string mime_type = 3;
+    uint64 size = 4;
+    string file_type = 5;             // audio/recording, pcap, upload
+    string related_id = 6;            // e.g., call_id for recordings
+    map<string, string> metadata = 7; // Custom metadata
+}
+
+message UploadFileResponse {
+    string id = 1;
+    uint64 size = 2;
+    string checksum = 3;              // SHA-256
+}
+
+message DownloadFileRequest {
+    string id = 1;
+}
+
+message DeleteFileRequest {
+    string id = 1;
+}
+
+message DeleteFileResponse {
+    bool deleted = 1;
+}
+
+message GetFileInfoRequest {
+    string id = 1;
+}
+
+message FileInfo {
+    string id = 1;
+    string name = 2;
+    string mime_type = 3;
+    uint64 size = 4;
+    string file_type = 5;
+    string related_id = 6;
+    string checksum = 7;
+    uint64 created_at = 8;
+    map<string, string> metadata = 9;
+}
+
+message ListFilesRequest {
+    string file_type = 1;             // Filter by type (optional)
+    string related_id = 2;            // Filter by related ID (optional)
+    uint32 limit = 3;
+    uint32 offset = 4;
+}
+
+message ListFilesResponse {
+    repeated FileInfo files = 1;
+    uint64 total_count = 2;
+    bool has_more = 3;
+}
+
+// ===== Event Subscription =====
+
+message SubscribeRequest {
+    repeated string collections = 1;  // Empty = all collections
+    repeated string patterns = 2;     // Glob patterns (e.g., "assistants:*")
+    bool include_data = 3;            // Include document data in events
+}
+
+message DatabaseEvent {
+    EventType type = 1;
+    string collection = 2;
+    string document_id = 3;
+    uint64 timestamp = 4;
+    string node_id = 5;
+    bytes data = 6;                   // JSON document (if include_data)
+}
+
+enum EventType {
+    EVENT_UNKNOWN = 0;
+    EVENT_INSERT = 1;
+    EVENT_UPDATE = 2;
+    EVENT_DELETE = 3;
+    EVENT_EXPIRE = 4;
+    EVENT_INVALIDATE = 5;
+}
+
+// ===== Replication =====
+
+message ReplicationEntry {
+    uint64 sequence = 1;
+    uint64 global_timestamp = 2;
+    string node_id = 3;
+    OperationType op = 4;
+    string collection = 5;
+    string document_id = 6;
+    uint64 document_version = 7;
+    bytes data = 8;
+}
+
+enum OperationType {
+    OP_UNKNOWN = 0;
+    OP_INSERT = 1;
+    OP_UPDATE = 2;
+    OP_DELETE = 3;
+    OP_UPSERT = 4;
+    OP_CREATE_COLLECTION = 5;
+    OP_DROP_COLLECTION = 6;
+}
+
+message ReplicationMessage {
+    oneof content {
+        ReplicationEntry entry = 1;
+        Heartbeat heartbeat = 2;
+        WatermarkUpdate watermark = 3;
+        SyncRequest sync_request = 4;
+    }
+}
+
+message Heartbeat {
+    string node_id = 1;
+    uint64 timestamp = 2;
+    uint64 sequence = 3;
+}
+
+message WatermarkUpdate {
+    string node_id = 1;
+    uint64 sequence = 2;
+}
+
+message SyncRequest {
+    uint64 from_sequence = 1;
+}
+
+message GetEntriesRequest {
+    uint64 from_sequence = 1;
+    uint32 limit = 2;
+}
+
+message GetNodeStateRequest {
+}
+
+message NodeState {
+    string node_id = 1;
+    uint64 current_sequence = 2;
+    map<string, uint64> peer_watermarks = 3;
+    uint64 document_count = 4;
+    repeated string collections = 5;
+    bool healthy = 6;
+    // Per-collection sequence tracking for collection discovery
+    map<string, uint64> collection_sequences = 7;
+}
+
+// ===== Health and Stats =====
+
+message HealthCheckRequest {
+}
+
+message HealthCheckResponse {
+    bool healthy = 1;
+    uint64 uptime_seconds = 2;
+    uint64 document_count = 3;
+    uint64 memory_used_bytes = 4;
+    uint64 wal_size_bytes = 5;
+    repeated PeerHealth peers = 6;
+}
+
+message PeerHealth {
+    string node_id = 1;
+    bool connected = 2;
+    uint64 latency_ms = 3;
+    uint64 lag_entries = 4;
+}
+
+message GetStatsRequest {
+}
+
+message GetStatsResponse {
+    uint64 total_documents = 1;
+    uint64 total_collections = 2;
+    uint64 memory_used_bytes = 3;
+    uint64 wal_sequence = 4;
+    uint64 wal_size_bytes = 5;
+    uint64 snapshot_count = 6;
+    uint64 last_snapshot_sequence = 7;
+    uint64 insert_count = 8;
+    uint64 update_count = 9;
+    uint64 delete_count = 10;
+    uint64 query_count = 11;
+}

service/CMakeLists.txt

@@ -0,0 +1,75 @@
+# Smartbotic Database Service
+
+# Find OpenSSL for encryption
+find_package(OpenSSL REQUIRED)
+
+# Find optional LZ4 for compression
+find_package(PkgConfig)
+if(PKG_CONFIG_FOUND)
+    pkg_check_modules(LZ4 QUIET liblz4)
+endif()
+
+# Source files
+set(DATABASE_SERVICE_SOURCES
+    src/main.cpp
+    src/database_service.cpp
+    src/database_grpc_impl.cpp
+    src/memory_store.cpp
+    src/persistence/persistence_manager.cpp
+    src/persistence/wal.cpp
+    src/persistence/snapshot.cpp
+    src/encryption/encryption_manager.cpp
+    src/events/event_manager.cpp
+    src/files/file_manager.cpp
+    src/files/file_store.cpp
+    src/replication/replication_manager.cpp
+    src/replication/sync_protocol.cpp
+    src/replication/conflict_resolver.cpp
+    src/migrations/migration_runner.cpp
+)
+
+# Create executable
+add_executable(smartbotic-database ${DATABASE_SERVICE_SOURCES})
+
+target_include_directories(smartbotic-database PRIVATE
+    ${CMAKE_CURRENT_SOURCE_DIR}/src
+)
+
+# Link dependencies
+target_link_libraries(smartbotic-database PRIVATE
+    smartbotic_db_proto
+    OpenSSL::SSL
+    OpenSSL::Crypto
+)
+
+if(TARGET nlohmann_json::nlohmann_json)
+    target_link_libraries(smartbotic-database PRIVATE nlohmann_json::nlohmann_json)
+else()
+    target_include_directories(smartbotic-database PRIVATE ${NLOHMANN_JSON_INCLUDE_DIRS})
+endif()
+
+if(TARGET spdlog::spdlog)
+    target_link_libraries(smartbotic-database PRIVATE spdlog::spdlog)
+else()
+    target_link_libraries(smartbotic-database PRIVATE ${SPDLOG_LIBRARIES})
+    target_include_directories(smartbotic-database PRIVATE ${SPDLOG_INCLUDE_DIRS})
+endif()
+
+# LZ4 compression support
+if(LZ4_FOUND)
+    target_compile_definitions(smartbotic-database PRIVATE HAVE_LZ4)
+    target_link_libraries(smartbotic-database PRIVATE ${LZ4_LIBRARIES})
+    target_include_directories(smartbotic-database PRIVATE ${LZ4_INCLUDE_DIRS})
+endif()
+
+# Systemd support
+if(SYSTEMD_FOUND)
+    target_compile_definitions(smartbotic-database PRIVATE HAVE_SYSTEMD)
+    target_link_libraries(smartbotic-database PRIVATE ${SYSTEMD_LIBRARIES})
+    target_include_directories(smartbotic-database PRIVATE ${SYSTEMD_INCLUDE_DIRS})
+endif()
+
+# Installation
+install(TARGETS smartbotic-database
+    RUNTIME DESTINATION bin
+)

service/config/database.json.example

@@ -0,0 +1,33 @@
+{
+    "log_level": "info",
+
+    "webapi_url": "http://localhost:8080",
+    "service_key": "${INTERNAL_SERVICE_KEY}",
+
+    "storage": {
+        "bind_address": "0.0.0.0",
+        "rpc_port": 9004,
+        "node_id": "${STORAGE_NODE_ID:storage-1}",
+        "data_directory": "/var/lib/callerai/storage",
+        "persistence": {
+            "wal_sync_interval_ms": 100,
+            "snapshot_interval_sec": 3600,
+            "compression": "lz4"
+        },
+        "files": {
+            "max_file_size_mb": 500,
+            "allowed_types": ["audio/*", "application/vnd.tcpdump.pcap", "application/octet-stream"],
+            "cleanup_orphans_interval_sec": 3600
+        },
+        "encryption": {
+            "enabled": true,
+            "key_file": "/var/lib/callerai/storage/storage.key",
+            "auto_generate_key": true
+        },
+        "replication": {
+            "enabled": false,
+            "peers": [],
+            "conflict_resolution": "last_writer_wins"
+        }
+    }
+}

service/src/database_grpc_impl.cpp

@@ -0,0 +1,871 @@
+#include "database_grpc_impl.hpp"
+#include "persistence/wal.hpp"
+
+#include <spdlog/spdlog.h>
+#include <nlohmann/json.hpp>
+
+namespace smartbotic::database {
+
+// Namespace alias for proto types
+namespace pb = smartbotic::databasepb;
+
+// ===== DatabaseGrpcImpl =====
+
+DatabaseGrpcImpl::DatabaseGrpcImpl(
+    MemoryStore& store,
+    PersistenceManager& persistence,
+    EventManager& events,
+    FileManager& files,
+    EncryptionManager& encryption
+) : store_(store)
+  , persistence_(persistence)
+  , events_(events)
+  , files_(files)
+  , encryption_(encryption)
+{
+}
+
+// ===== Document Operations =====
+
+grpc::Status DatabaseGrpcImpl::Insert(
+    grpc::ServerContext* /*context*/,
+    const pb::InsertRequest* request,
+    pb::InsertResponse* response
+) {
+    try {
+        nlohmann::json data = nlohmann::json::parse(
+            request->data().begin(), request->data().end()
+        );
+
+        Document doc;
+        doc.id = request->id();
+        doc.data = data;
+        if (request->ttl_seconds() > 0) {
+            doc.setTtlSeconds(request->ttl_seconds());
+        }
+
+        // Set audit fields
+        if (!request->actor().empty()) {
+            doc.createdBy = request->actor();
+            doc.updatedBy = request->actor();
+        }
+
+        // Encrypt sensitive fields if needed
+        encryption_.encryptSensitiveFields(doc);
+
+        std::string id = store_.insert(request->collection(), doc);
+
+        response->set_id(id);
+        response->set_version(1);
+
+        return grpc::Status::OK;
+
+    } catch (const nlohmann::json::exception& e) {
+        return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
+                           "Invalid JSON: " + std::string(e.what()));
+    } catch (const std::exception& e) {
+        return grpc::Status(grpc::StatusCode::INTERNAL, e.what());
+    }
+}
+
+grpc::Status DatabaseGrpcImpl::Get(
+    grpc::ServerContext* /*context*/,
+    const pb::GetRequest* request,
+    pb::GetResponse* response
+) {
+    auto doc = store_.get(request->collection(), request->id());
+    if (!doc) {
+        response->set_found(false);
+        return grpc::Status::OK;
+    }
+
+    // Decrypt sensitive fields
+    encryption_.decryptSensitiveFields(*doc);
+
+    *response->mutable_document() = toProto(*doc);
+    response->set_found(true);
+
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::Update(
+    grpc::ServerContext* /*context*/,
+    const pb::UpdateRequest* request,
+    pb::UpdateResponse* response
+) {
+    try {
+        nlohmann::json data = nlohmann::json::parse(
+            request->data().begin(), request->data().end()
+        );
+
+        Document doc;
+        doc.data = data;
+
+        // Set audit field for update
+        if (!request->actor().empty()) {
+            doc.updatedBy = request->actor();
+        }
+
+        encryption_.encryptSensitiveFields(doc);
+
+        bool success;
+        if (request->expected_version() > 0) {
+            success = store_.updateIfVersion(
+                request->collection(),
+                request->id(),
+                doc,
+                request->expected_version()
+            );
+            if (!success) {
+                response->set_success(false);
+                response->set_error("Version conflict");
+                return grpc::Status::OK;
+            }
+        } else {
+            success = store_.update(request->collection(), request->id(), doc);
+            if (!success) {
+                response->set_success(false);
+                response->set_error("Document not found");
+                return grpc::Status::OK;
+            }
+        }
+
+        // Get updated version
+        auto updated = store_.get(request->collection(), request->id());
+        response->set_new_version(updated ? updated->version : 0);
+        response->set_success(true);
+
+        return grpc::Status::OK;
+
+    } catch (const nlohmann::json::exception& e) {
+        return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
+                           "Invalid JSON: " + std::string(e.what()));
+    } catch (const std::exception& e) {
+        return grpc::Status(grpc::StatusCode::INTERNAL, e.what());
+    }
+}
+
+grpc::Status DatabaseGrpcImpl::Upsert(
+    grpc::ServerContext* /*context*/,
+    const pb::UpsertRequest* request,
+    pb::UpsertResponse* response
+) {
+    try {
+        nlohmann::json data = nlohmann::json::parse(
+            request->data().begin(), request->data().end()
+        );
+
+        bool existed = store_.exists(request->collection(), request->id());
+
+        Document doc;
+        doc.id = request->id();
+        doc.data = data;
+        if (request->ttl_seconds() > 0) {
+            doc.setTtlSeconds(request->ttl_seconds());
+        }
+
+        // Set audit fields
+        if (!request->actor().empty()) {
+            if (!existed) {
+                doc.createdBy = request->actor();
+            }
+            doc.updatedBy = request->actor();
+        }
+
+        encryption_.encryptSensitiveFields(doc);
+
+        std::string id = store_.upsert(request->collection(), doc);
+
+        auto updated = store_.get(request->collection(), id);
+        response->set_id(id);
+        response->set_version(updated ? updated->version : 1);
+        response->set_inserted(!existed);
+
+        return grpc::Status::OK;
+
+    } catch (const nlohmann::json::exception& e) {
+        return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
+                           "Invalid JSON: " + std::string(e.what()));
+    } catch (const std::exception& e) {
+        return grpc::Status(grpc::StatusCode::INTERNAL, e.what());
+    }
+}
+
+grpc::Status DatabaseGrpcImpl::Delete(
+    grpc::ServerContext* /*context*/,
+    const pb::DeleteRequest* request,
+    pb::DeleteResponse* response
+) {
+    bool deleted = store_.remove(request->collection(), request->id());
+    response->set_deleted(deleted);
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::Exists(
+    grpc::ServerContext* /*context*/,
+    const pb::ExistsRequest* request,
+    pb::ExistsResponse* response
+) {
+    bool exists = store_.exists(request->collection(), request->id());
+    response->set_exists(exists);
+    return grpc::Status::OK;
+}
+
+// ===== Batch Operations =====
+
+grpc::Status DatabaseGrpcImpl::BatchInsert(
+    grpc::ServerContext* /*context*/,
+    const pb::BatchInsertRequest* request,
+    pb::BatchInsertResponse* response
+) {
+    uint32_t successCount = 0;
+    uint32_t errorCount = 0;
+    std::string actor = request->actor();
+
+    for (const auto& item : request->items()) {
+        try {
+            nlohmann::json data = nlohmann::json::parse(
+                item.data().begin(), item.data().end()
+            );
+
+            Document doc;
+            doc.id = item.id();
+            doc.data = data;
+            if (item.ttl_seconds() > 0) {
+                doc.setTtlSeconds(item.ttl_seconds());
+            }
+
+            // Set audit fields
+            if (!actor.empty()) {
+                doc.createdBy = actor;
+                doc.updatedBy = actor;
+            }
+
+            encryption_.encryptSensitiveFields(doc);
+
+            std::string id = store_.insert(request->collection(), doc);
+            response->add_ids(id);
+            successCount++;
+
+        } catch (const std::exception&) {
+            errorCount++;
+        }
+    }
+
+    response->set_success_count(successCount);
+    response->set_error_count(errorCount);
+
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::BatchGet(
+    grpc::ServerContext* /*context*/,
+    const pb::BatchGetRequest* request,
+    pb::BatchGetResponse* response
+) {
+    for (const auto& id : request->ids()) {
+        auto doc = store_.get(request->collection(), id);
+        if (doc) {
+            encryption_.decryptSensitiveFields(*doc);
+            *response->add_documents() = toProto(*doc);
+        }
+    }
+
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::BatchDelete(
+    grpc::ServerContext* /*context*/,
+    const pb::BatchDeleteRequest* request,
+    pb::BatchDeleteResponse* response
+) {
+    std::vector<std::string> ids(request->ids().begin(), request->ids().end());
+    uint64_t deleted = store_.bulkDelete(request->collection(), ids);
+    response->set_deleted_count(deleted);
+    return grpc::Status::OK;
+}
+
+// ===== Query Operations =====
+
+grpc::Status DatabaseGrpcImpl::Find(
+    grpc::ServerContext* /*context*/,
+    const pb::FindRequest* request,
+    pb::FindResponse* response
+) {
+    Query query = fromProtoQuery(*request);
+    QueryResult result = store_.find(request->collection(), query);
+
+    response->set_total_count(result.totalCount);
+    response->set_has_more(result.hasMore);
+
+    for (auto& doc : result.documents) {
+        encryption_.decryptSensitiveFields(doc);
+        *response->add_documents() = toProto(doc);
+    }
+
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::Count(
+    grpc::ServerContext* /*context*/,
+    const pb::CountRequest* request,
+    pb::CountResponse* response
+) {
+    auto filters = fromProtoFilters(request->filters());
+    uint64_t count = store_.count(request->collection(), filters);
+    response->set_count(count);
+    return grpc::Status::OK;
+}
+
+// ===== Set Operations =====
+
+grpc::Status DatabaseGrpcImpl::SetAdd(
+    grpc::ServerContext* /*context*/,
+    const pb::SetAddRequest* request,
+    pb::SetAddResponse* response
+) {
+    bool added = store_.setAdd(request->collection(), request->set_id(), request->member());
+    response->set_added(added);
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::SetRemove(
+    grpc::ServerContext* /*context*/,
+    const pb::SetRemoveRequest* request,
+    pb::SetRemoveResponse* response
+) {
+    bool removed = store_.setRemove(request->collection(), request->set_id(), request->member());
+    response->set_removed(removed);
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::SetMembers(
+    grpc::ServerContext* /*context*/,
+    const pb::SetMembersRequest* request,
+    pb::SetMembersResponse* response
+) {
+    auto members = store_.setMembers(request->collection(), request->set_id());
+    for (const auto& member : members) {
+        response->add_members(member);
+    }
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::SetIsMember(
+    grpc::ServerContext* /*context*/,
+    const pb::SetIsMemberRequest* request,
+    pb::SetIsMemberResponse* response
+) {
+    bool isMember = store_.setIsMember(request->collection(), request->set_id(), request->member());
+    response->set_is_member(isMember);
+    return grpc::Status::OK;
+}
+
+// ===== Collection Management =====
+
+grpc::Status DatabaseGrpcImpl::CreateCollection(
+    grpc::ServerContext* /*context*/,
+    const pb::CreateCollectionRequest* request,
+    pb::CreateCollectionResponse* response
+) {
+    CollectionOptions opts;
+    // Proto3 defaults bools to false, but we want autoCreateId to default to true
+    // Since proto3 can't distinguish "not set" from "false", we always default to true
+    // Users who want autoCreateId=false must use a different field (e.g., disable_auto_id)
+    opts.autoCreateId = true;
+    opts.defaultTtlSeconds = request->options().default_ttl_seconds();
+    opts.encrypted = request->options().encrypted();
+    for (const auto& field : request->options().sensitive_fields()) {
+        opts.sensitiveFields.push_back(field);
+    }
+
+    bool created = store_.createCollection(request->name(), opts);
+    response->set_created(created);
+
+    if (created) {
+        persistence_.logCreateCollection(request->name(), opts);
+    }
+
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::DropCollection(
+    grpc::ServerContext* /*context*/,
+    const pb::DropCollectionRequest* request,
+    pb::DropCollectionResponse* response
+) {
+    bool dropped = store_.dropCollection(request->name());
+    response->set_dropped(dropped);
+
+    if (dropped) {
+        persistence_.logDropCollection(request->name());
+    }
+
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::ListCollections(
+    grpc::ServerContext* /*context*/,
+    const pb::ListCollectionsRequest* /*request*/,
+    pb::ListCollectionsResponse* response
+) {
+    auto names = store_.listCollections();
+    for (const auto& name : names) {
+        response->add_names(name);
+    }
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::GetCollectionInfo(
+    grpc::ServerContext* /*context*/,
+    const pb::GetCollectionInfoRequest* request,
+    pb::GetCollectionInfoResponse* response
+) {
+    auto info = store_.getCollectionInfo(request->name());
+    if (!info) {
+        response->set_found(false);
+        return grpc::Status::OK;
+    }
+
+    *response->mutable_info() = toProtoCollInfo(*info);
+    response->set_found(true);
+
+    return grpc::Status::OK;
+}
+
+// ===== File Operations =====
+
+grpc::Status DatabaseGrpcImpl::UploadFile(
+    grpc::ServerContext* /*context*/,
+    grpc::ServerReader<pb::FileChunk>* reader,
+    pb::UploadFileResponse* response
+) {
+    pb::FileChunk chunk;
+    std::string fileId;
+    std::vector<uint8_t> fileData;
+    pb::FileMetadata metadata;
+
+    while (reader->Read(&chunk)) {
+        if (chunk.has_metadata()) {
+            metadata = chunk.metadata();
+        } else if (chunk.has_data()) {
+            const auto& data = chunk.data();
+            fileData.insert(fileData.end(), data.begin(), data.end());
+        }
+    }
+
+    try {
+        FileManager::FileInfo info;
+        info.name = metadata.name();
+        info.mimeType = metadata.mime_type();
+        info.fileType = metadata.file_type();
+        info.relatedId = metadata.related_id();
+        for (const auto& [key, value] : metadata.metadata()) {
+            info.metadata[key] = value;
+        }
+
+        fileId = files_.storeFile(fileData, info);
+
+        response->set_id(fileId);
+        response->set_size(fileData.size());
+        response->set_checksum(files_.getChecksum(fileId));
+
+        return grpc::Status::OK;
+
+    } catch (const std::exception& e) {
+        return grpc::Status(grpc::StatusCode::INTERNAL, e.what());
+    }
+}
+
+grpc::Status DatabaseGrpcImpl::DownloadFile(
+    grpc::ServerContext* context,
+    const pb::DownloadFileRequest* request,
+    grpc::ServerWriter<pb::FileChunk>* writer
+) {
+    try {
+        auto info = files_.getFileInfo(request->id());
+        if (!info) {
+            return grpc::Status(grpc::StatusCode::NOT_FOUND, "File not found");
+        }
+
+        // Send metadata first
+        pb::FileChunk metadataChunk;
+        auto* meta = metadataChunk.mutable_metadata();
+        meta->set_id(info->id);
+        meta->set_name(info->name);
+        meta->set_mime_type(info->mimeType);
+        meta->set_size(info->size);
+        meta->set_file_type(info->fileType);
+        writer->Write(metadataChunk);
+
+        // Stream file data in chunks
+        constexpr size_t CHUNK_SIZE = 64 * 1024;  // 64KB chunks
+        auto data = files_.readFile(request->id());
+
+        for (size_t offset = 0; offset < data.size() && !context->IsCancelled(); offset += CHUNK_SIZE) {
+            pb::FileChunk dataChunk;
+            size_t chunkSize = std::min(CHUNK_SIZE, data.size() - offset);
+            dataChunk.set_data(data.data() + offset, chunkSize);
+            writer->Write(dataChunk);
+        }
+
+        return grpc::Status::OK;
+
+    } catch (const std::exception& e) {
+        return grpc::Status(grpc::StatusCode::INTERNAL, e.what());
+    }
+}
+
+grpc::Status DatabaseGrpcImpl::DeleteFile(
+    grpc::ServerContext* /*context*/,
+    const pb::DeleteFileRequest* request,
+    pb::DeleteFileResponse* response
+) {
+    bool deleted = files_.deleteFile(request->id());
+    response->set_deleted(deleted);
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::GetFileInfo(
+    grpc::ServerContext* /*context*/,
+    const pb::GetFileInfoRequest* request,
+    pb::FileInfo* response
+) {
+    auto info = files_.getFileInfo(request->id());
+    if (!info) {
+        return grpc::Status(grpc::StatusCode::NOT_FOUND, "File not found");
+    }
+
+    response->set_id(info->id);
+    response->set_name(info->name);
+    response->set_mime_type(info->mimeType);
+    response->set_size(info->size);
+    response->set_file_type(info->fileType);
+    response->set_related_id(info->relatedId);
+    response->set_checksum(info->checksum);
+    response->set_created_at(info->createdAt);
+    for (const auto& [key, value] : info->metadata) {
+        (*response->mutable_metadata())[key] = value;
+    }
+
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::ListFiles(
+    grpc::ServerContext* /*context*/,
+    const pb::ListFilesRequest* request,
+    pb::ListFilesResponse* response
+) {
+    auto result = files_.listFiles(request->file_type(), request->related_id(),
+                                   request->limit(), request->offset());
+
+    response->set_total_count(result.totalCount);
+    response->set_has_more(result.hasMore);
+
+    for (const auto& info : result.files) {
+        auto* file = response->add_files();
+        file->set_id(info.id);
+        file->set_name(info.name);
+        file->set_mime_type(info.mimeType);
+        file->set_size(info.size);
+        file->set_file_type(info.fileType);
+        file->set_related_id(info.relatedId);
+        file->set_checksum(info.checksum);
+        file->set_created_at(info.createdAt);
+    }
+
+    return grpc::Status::OK;
+}
+
+// ===== Event Subscription =====
+
+grpc::Status DatabaseGrpcImpl::Subscribe(
+    grpc::ServerContext* context,
+    const pb::SubscribeRequest* request,
+    grpc::ServerWriter<pb::DatabaseEvent>* writer
+) {
+    std::vector<std::string> collections(request->collections().begin(), request->collections().end());
+    std::vector<std::string> patterns(request->patterns().begin(), request->patterns().end());
+    bool includeData = request->include_data();
+
+    uint64_t subId = events_.subscribe(collections, patterns, [writer, includeData, context](const DatabaseEvent& event) {
+        if (context->IsCancelled()) return;
+
+        pb::DatabaseEvent protoEvent;
+        protoEvent.set_type(static_cast<pb::EventType>(static_cast<int>(event.type) + 1));
+        protoEvent.set_collection(event.collection);
+        protoEvent.set_document_id(event.documentId);
+        protoEvent.set_timestamp(event.timestamp);
+        protoEvent.set_node_id(event.nodeId);
+
+        if (includeData && event.data) {
+            std::string dataStr = event.data->dump();
+            protoEvent.set_data(dataStr);
+        }
+
+        writer->Write(protoEvent);
+    });
+
+    // Block until cancelled
+    while (!context->IsCancelled()) {
+        std::this_thread::sleep_for(std::chrono::milliseconds(100));
+    }
+
+    events_.unsubscribe(subId);
+
+    return grpc::Status::OK;
+}
+
+// ===== Health and Stats =====
+
+grpc::Status DatabaseGrpcImpl::HealthCheck(
+    grpc::ServerContext* /*context*/,
+    const pb::HealthCheckRequest* /*request*/,
+    pb::HealthCheckResponse* response
+) {
+    auto stats = store_.getStats();
+    auto persistStats = persistence_.getStats();
+
+    response->set_healthy(true);
+    response->set_document_count(stats.totalDocuments);
+    response->set_memory_used_bytes(stats.estimatedMemoryBytes);
+    response->set_wal_size_bytes(persistStats.walSizeBytes);
+
+    // Calculate uptime
+    auto now = std::chrono::steady_clock::now();
+    auto uptimeSeconds = std::chrono::duration_cast<std::chrono::seconds>(now - startTime_).count();
+    response->set_uptime_seconds(static_cast<uint64_t>(uptimeSeconds));
+
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseGrpcImpl::GetStats(
+    grpc::ServerContext* /*context*/,
+    const pb::GetStatsRequest* /*request*/,
+    pb::GetStatsResponse* response
+) {
+    auto stats = store_.getStats();
+    auto persistStats = persistence_.getStats();
+
+    response->set_total_documents(stats.totalDocuments);
+    response->set_total_collections(stats.totalCollections);
+    response->set_memory_used_bytes(stats.estimatedMemoryBytes);
+    response->set_wal_sequence(persistStats.walSequence);
+    response->set_wal_size_bytes(persistStats.walSizeBytes);
+    response->set_snapshot_count(persistStats.snapshotCount);
+    response->set_last_snapshot_sequence(persistStats.lastSnapshotSequence);
+    response->set_insert_count(stats.insertCount);
+    response->set_update_count(stats.updateCount);
+    response->set_delete_count(stats.deleteCount);
+    response->set_query_count(stats.queryCount);
+
+    return grpc::Status::OK;
+}
+
+// ===== Helper Methods =====
+
+pb::Document DatabaseGrpcImpl::toProto(const Document& doc) {
+    pb::Document proto;
+    proto.set_id(doc.id);
+    proto.set_collection(doc.collection);
+    proto.set_data(doc.data.dump());
+    proto.set_version(doc.version);
+    proto.set_created_at(doc.createdAt);
+    proto.set_updated_at(doc.updatedAt);
+    proto.set_expires_at(doc.expiresAt);
+    proto.set_node_id(doc.nodeId);
+    proto.set_encrypted(doc.encrypted);
+    for (const auto& field : doc.encryptedFields) {
+        proto.add_encrypted_fields(field);
+    }
+    proto.set_created_by(doc.createdBy);
+    proto.set_updated_by(doc.updatedBy);
+    return proto;
+}
+
+Document DatabaseGrpcImpl::fromProto(const pb::Document& proto) {
+    Document doc;
+    doc.id = proto.id();
+    doc.collection = proto.collection();
+    doc.data = nlohmann::json::parse(proto.data());
+    doc.version = proto.version();
+    doc.createdAt = proto.created_at();
+    doc.updatedAt = proto.updated_at();
+    doc.expiresAt = proto.expires_at();
+    doc.nodeId = proto.node_id();
+    doc.encrypted = proto.encrypted();
+    for (const auto& field : proto.encrypted_fields()) {
+        doc.encryptedFields.push_back(field);
+    }
+    doc.createdBy = proto.created_by();
+    doc.updatedBy = proto.updated_by();
+    return doc;
+}
+
+pb::CollectionInfo DatabaseGrpcImpl::toProtoCollInfo(const struct CollectionInfo& info) {
+    pb::CollectionInfo proto;
+    proto.set_name(info.name);
+    proto.set_document_count(info.documentCount);
+    proto.set_size_bytes(info.sizeBytes);
+    proto.mutable_options()->set_auto_create_id(info.options.autoCreateId);
+    proto.mutable_options()->set_default_ttl_seconds(info.options.defaultTtlSeconds);
+    proto.mutable_options()->set_encrypted(info.options.encrypted);
+    for (const auto& field : info.options.sensitiveFields) {
+        proto.mutable_options()->add_sensitive_fields(field);
+    }
+    proto.set_created_at(info.createdAt);
+    proto.set_updated_at(info.updatedAt);
+    return proto;
+}
+
+std::vector<Filter> DatabaseGrpcImpl::fromProtoFilters(
+    const google::protobuf::RepeatedPtrField<pb::Filter>& filters
+) {
+    std::vector<Filter> result;
+    for (const auto& f : filters) {
+        Filter filter;
+        filter.field = f.field();
+        filter.op = static_cast<FilterOp>(f.op() - 1);  // Adjust for UNSPECIFIED
+        if (!f.value().empty()) {
+            filter.value = nlohmann::json::parse(f.value());
+        }
+        result.push_back(filter);
+    }
+    return result;
+}
+
+Query DatabaseGrpcImpl::fromProtoQuery(const pb::FindRequest& request) {
+    Query query;
+    query.filters = fromProtoFilters(request.filters());
+    if (request.has_sort() && !request.sort().field().empty()) {
+        query.sort = Sort{request.sort().field(), request.sort().descending()};
+    }
+    query.limit = request.limit() > 0 ? request.limit() : 100;
+    query.offset = request.offset();
+    for (const auto& field : request.projection()) {
+        query.projection.push_back(field);
+    }
+    return query;
+}
+
+// ===== DatabaseReplicationGrpcImpl =====
+
+DatabaseReplicationGrpcImpl::DatabaseReplicationGrpcImpl(
+    MemoryStore& store,
+    ReplicationManager& replication,
+    PersistenceManager& persistence
+) : store_(store)
+  , replication_(replication)
+  , persistence_(persistence)
+{
+}
+
+grpc::Status DatabaseReplicationGrpcImpl::SyncStream(
+    grpc::ServerContext* context,
+    grpc::ServerReaderWriter<pb::ReplicationMessage, pb::ReplicationMessage>* stream
+) {
+    // Handle bidirectional streaming for replication
+    pb::ReplicationMessage msg;
+    while (!context->IsCancelled() && stream->Read(&msg)) {
+        if (msg.has_entry()) {
+            replication_.handleIncomingEntry(msg.entry());
+        } else if (msg.has_heartbeat()) {
+            replication_.handleHeartbeat(msg.heartbeat());
+        } else if (msg.has_watermark()) {
+            replication_.handleWatermarkUpdate(msg.watermark());
+        } else if (msg.has_sync_request()) {
+            replication_.handleSyncRequest(msg.sync_request(), stream);
+        }
+    }
+
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseReplicationGrpcImpl::GetEntriesSince(
+    grpc::ServerContext* context,
+    const pb::GetEntriesRequest* request,
+    grpc::ServerWriter<pb::ReplicationEntry>* writer
+) {
+    auto nodeState = replication_.getNodeState();
+    uint32_t limit = request->limit() > 0 ? request->limit() : 1000;
+
+    persistence_.replayWal(request->from_sequence(), limit,
+        [writer, context, &nodeState](const WalEntry& walEntry) {
+            if (context->IsCancelled()) {
+                return;
+            }
+
+            pb::ReplicationEntry entry;
+            entry.set_sequence(walEntry.sequence);
+            entry.set_global_timestamp(walEntry.timestamp);
+            entry.set_node_id(nodeState.nodeId);
+            entry.set_collection(walEntry.collection);
+            entry.set_document_id(walEntry.documentId);
+
+            // Convert operation type
+            switch (walEntry.opType) {
+                case WalOpType::INSERT:
+                    entry.set_op(pb::OP_INSERT);
+                    break;
+                case WalOpType::UPDATE:
+                    entry.set_op(pb::OP_UPDATE);
+                    break;
+                case WalOpType::DELETE:
+                    entry.set_op(pb::OP_DELETE);
+                    break;
+                case WalOpType::UPSERT:
+                    entry.set_op(pb::OP_UPSERT);
+                    break;
+                case WalOpType::CREATE_COLLECTION:
+                    entry.set_op(pb::OP_CREATE_COLLECTION);
+                    break;
+                case WalOpType::DROP_COLLECTION:
+                    entry.set_op(pb::OP_DROP_COLLECTION);
+                    break;
+                default:
+                    entry.set_op(pb::OP_UNKNOWN);
+                    break;
+            }
+
+            // Add data if present
+            if (walEntry.data) {
+                entry.set_data(walEntry.data->dump());
+            }
+
+            writer->Write(entry);
+        });
+
+    return grpc::Status::OK;
+}
+
+grpc::Status DatabaseReplicationGrpcImpl::GetNodeState(
+    grpc::ServerContext* /*context*/,
+    const pb::GetNodeStateRequest* /*request*/,
+    pb::NodeState* response
+) {
+    auto state = replication_.getNodeState();
+
+    response->set_node_id(state.nodeId);
+    // Use actual WAL sequence which includes all writes
+    response->set_current_sequence(persistence_.currentWalSequence());
+    response->set_document_count(store_.getStats().totalDocuments);
+    response->set_healthy(true);
+
+    for (const auto& name : store_.listCollections()) {
+        response->add_collections(name);
+    }
+
+    for (const auto& [peerId, watermark] : state.peerWatermarks) {
+        (*response->mutable_peer_watermarks())[peerId] = watermark;
+    }
+
+    // Add per-collection sequences for collection discovery
+    auto collectionSeqs = persistence_.getCollectionSequences();
+    for (const auto& [collName, seq] : collectionSeqs) {
+        (*response->mutable_collection_sequences())[collName] = seq;
+    }
+
+    return grpc::Status::OK;
+}
+
+} // namespace smartbotic::database

service/src/database_grpc_impl.hpp

@@ -0,0 +1,267 @@
+#pragma once
+
+#include "memory_store.hpp"
+#include "persistence/persistence_manager.hpp"
+#include "events/event_manager.hpp"
+#include "files/file_manager.hpp"
+#include "encryption/encryption_manager.hpp"
+#include "replication/replication_manager.hpp"
+
+#include <database.grpc.pb.h>
+
+#include <grpcpp/grpcpp.h>
+#include <chrono>
+#include <memory>
+#include <string>
+
+namespace smartbotic::database {
+
+// Namespace alias for proto types
+namespace pb = smartbotic::databasepb;
+
+/**
+ * gRPC service implementation for the Storage service.
+ */
+class DatabaseGrpcImpl final : public pb::DatabaseService::Service {
+public:
+    DatabaseGrpcImpl(
+        MemoryStore& store,
+        PersistenceManager& persistence,
+        EventManager& events,
+        FileManager& files,
+        EncryptionManager& encryption
+    );
+
+    ~DatabaseGrpcImpl() override = default;
+
+    // ===== Document Operations =====
+
+    grpc::Status Insert(
+        grpc::ServerContext* context,
+        const pb::InsertRequest* request,
+        pb::InsertResponse* response
+    ) override;
+
+    grpc::Status Get(
+        grpc::ServerContext* context,
+        const pb::GetRequest* request,
+        pb::GetResponse* response
+    ) override;
+
+    grpc::Status Update(
+        grpc::ServerContext* context,
+        const pb::UpdateRequest* request,
+        pb::UpdateResponse* response
+    ) override;
+
+    grpc::Status Upsert(
+        grpc::ServerContext* context,
+        const pb::UpsertRequest* request,
+        pb::UpsertResponse* response
+    ) override;
+
+    grpc::Status Delete(
+        grpc::ServerContext* context,
+        const pb::DeleteRequest* request,
+        pb::DeleteResponse* response
+    ) override;
+
+    grpc::Status Exists(
+        grpc::ServerContext* context,
+        const pb::ExistsRequest* request,
+        pb::ExistsResponse* response
+    ) override;
+
+    // ===== Batch Operations =====
+
+    grpc::Status BatchInsert(
+        grpc::ServerContext* context,
+        const pb::BatchInsertRequest* request,
+        pb::BatchInsertResponse* response
+    ) override;
+
+    grpc::Status BatchGet(
+        grpc::ServerContext* context,
+        const pb::BatchGetRequest* request,
+        pb::BatchGetResponse* response
+    ) override;
+
+    grpc::Status BatchDelete(
+        grpc::ServerContext* context,
+        const pb::BatchDeleteRequest* request,
+        pb::BatchDeleteResponse* response
+    ) override;
+
+    // ===== Query Operations =====
+
+    grpc::Status Find(
+        grpc::ServerContext* context,
+        const pb::FindRequest* request,
+        pb::FindResponse* response
+    ) override;
+
+    grpc::Status Count(
+        grpc::ServerContext* context,
+        const pb::CountRequest* request,
+        pb::CountResponse* response
+    ) override;
+
+    // ===== Set Operations =====
+
+    grpc::Status SetAdd(
+        grpc::ServerContext* context,
+        const pb::SetAddRequest* request,
+        pb::SetAddResponse* response
+    ) override;
+
+    grpc::Status SetRemove(
+        grpc::ServerContext* context,
+        const pb::SetRemoveRequest* request,
+        pb::SetRemoveResponse* response
+    ) override;
+
+    grpc::Status SetMembers(
+        grpc::ServerContext* context,
+        const pb::SetMembersRequest* request,
+        pb::SetMembersResponse* response
+    ) override;
+
+    grpc::Status SetIsMember(
+        grpc::ServerContext* context,
+        const pb::SetIsMemberRequest* request,
+        pb::SetIsMemberResponse* response
+    ) override;
+
+    // ===== Collection Management =====
+
+    grpc::Status CreateCollection(
+        grpc::ServerContext* context,
+        const pb::CreateCollectionRequest* request,
+        pb::CreateCollectionResponse* response
+    ) override;
+
+    grpc::Status DropCollection(
+        grpc::ServerContext* context,
+        const pb::DropCollectionRequest* request,
+        pb::DropCollectionResponse* response
+    ) override;
+
+    grpc::Status ListCollections(
+        grpc::ServerContext* context,
+        const pb::ListCollectionsRequest* request,
+        pb::ListCollectionsResponse* response
+    ) override;
+
+    grpc::Status GetCollectionInfo(
+        grpc::ServerContext* context,
+        const pb::GetCollectionInfoRequest* request,
+        pb::GetCollectionInfoResponse* response
+    ) override;
+
+    // ===== File Operations =====
+
+    grpc::Status UploadFile(
+        grpc::ServerContext* context,
+        grpc::ServerReader<pb::FileChunk>* reader,
+        pb::UploadFileResponse* response
+    ) override;
+
+    grpc::Status DownloadFile(
+        grpc::ServerContext* context,
+        const pb::DownloadFileRequest* request,
+        grpc::ServerWriter<pb::FileChunk>* writer
+    ) override;
+
+    grpc::Status DeleteFile(
+        grpc::ServerContext* context,
+        const pb::DeleteFileRequest* request,
+        pb::DeleteFileResponse* response
+    ) override;
+
+    grpc::Status GetFileInfo(
+        grpc::ServerContext* context,
+        const pb::GetFileInfoRequest* request,
+        pb::FileInfo* response
+    ) override;
+
+    grpc::Status ListFiles(
+        grpc::ServerContext* context,
+        const pb::ListFilesRequest* request,
+        pb::ListFilesResponse* response
+    ) override;
+
+    // ===== Event Subscription =====
+
+    grpc::Status Subscribe(
+        grpc::ServerContext* context,
+        const pb::SubscribeRequest* request,
+        grpc::ServerWriter<pb::DatabaseEvent>* writer
+    ) override;
+
+    // ===== Health and Stats =====
+
+    grpc::Status HealthCheck(
+        grpc::ServerContext* context,
+        const pb::HealthCheckRequest* request,
+        pb::HealthCheckResponse* response
+    ) override;
+
+    grpc::Status GetStats(
+        grpc::ServerContext* context,
+        const pb::GetStatsRequest* request,
+        pb::GetStatsResponse* response
+    ) override;
+
+private:
+    // Convert internal types to proto types
+    static pb::Document toProto(const Document& doc);
+    static Document fromProto(const pb::Document& proto);
+    static pb::CollectionInfo toProtoCollInfo(const struct CollectionInfo& info);
+    static std::vector<Filter> fromProtoFilters(const google::protobuf::RepeatedPtrField<pb::Filter>& filters);
+    static Query fromProtoQuery(const pb::FindRequest& request);
+
+    MemoryStore& store_;
+    PersistenceManager& persistence_;
+    EventManager& events_;
+    FileManager& files_;
+    EncryptionManager& encryption_;
+    std::chrono::steady_clock::time_point startTime_ = std::chrono::steady_clock::now();
+};
+
+/**
+ * gRPC service implementation for Storage Replication.
+ */
+class DatabaseReplicationGrpcImpl final : public pb::StorageReplication::Service {
+public:
+    DatabaseReplicationGrpcImpl(
+        MemoryStore& store,
+        ReplicationManager& replication,
+        PersistenceManager& persistence
+    );
+
+    ~DatabaseReplicationGrpcImpl() override = default;
+
+    grpc::Status SyncStream(
+        grpc::ServerContext* context,
+        grpc::ServerReaderWriter<pb::ReplicationMessage, pb::ReplicationMessage>* stream
+    ) override;
+
+    grpc::Status GetEntriesSince(
+        grpc::ServerContext* context,
+        const pb::GetEntriesRequest* request,
+        grpc::ServerWriter<pb::ReplicationEntry>* writer
+    ) override;
+
+    grpc::Status GetNodeState(
+        grpc::ServerContext* context,
+        const pb::GetNodeStateRequest* request,
+        pb::NodeState* response
+    ) override;
+
+private:
+    MemoryStore& store_;
+    ReplicationManager& replication_;
+    PersistenceManager& persistence_;
+};
+
+} // namespace smartbotic::database

service/src/database_service.cpp

@@ -0,0 +1,487 @@
+#include "database_service.hpp"
+
+#include <grpcpp/grpcpp.h>
+#include <spdlog/spdlog.h>
+
+#include <fstream>
+
+namespace smartbotic::database {
+
+DatabaseService::DatabaseService(Config config)
+    : config_(std::move(config))
+{
+}
+
+DatabaseService::~DatabaseService() {
+    stop();
+}
+
+bool DatabaseService::initialize() {
+    spdlog::info("Initializing database service (node: {})", config_.nodeId);
+
+    try {
+        // Create data directory if needed
+        std::error_code ec;
+        std::filesystem::create_directories(config_.dataDirectory, ec);
+        if (ec) {
+            spdlog::error("Failed to create data directory: {}", ec.message());
+            return false;
+        }
+
+        setupComponents();
+
+        // Initialize encryption
+        if (config_.encryptionEnabled) {
+            if (!encryption_->initialize()) {
+                spdlog::error("Failed to initialize encryption");
+                return false;
+            }
+        }
+
+        // Recover from persistence
+        persistence_->recover(*store_);
+
+        // Set replication sequence after recovery (WAL sequence is now known)
+        replication_->setSequence(persistence_->currentWalSequence());
+
+        // Run migrations if enabled
+        if (config_.migrations.enabled && !config_.migrations.directory.empty()) {
+            if (!runMigrations()) {
+                if (config_.migrations.failOnError) {
+                    spdlog::error("Failed to run migrations");
+                    return false;
+                }
+                spdlog::warn("Some migrations failed, continuing anyway");
+            }
+        }
+
+        spdlog::info("Database service initialized successfully");
+        return true;
+
+    } catch (const std::exception& e) {
+        spdlog::error("Failed to initialize database service: {}", e.what());
+        return false;
+    }
+}
+
+bool DatabaseService::runMigrations() {
+    if (!config_.migrations.enabled) {
+        return true;
+    }
+
+    MigrationRunner::Config migrationConfig;
+    migrationConfig.directory = config_.migrations.directory;
+    migrationConfig.autoApply = config_.migrations.autoApply;
+    migrationConfig.failOnError = config_.migrations.failOnError;
+
+    migrationRunner_ = std::make_unique<MigrationRunner>(*store_, migrationConfig);
+    return migrationRunner_->runMigrations();
+}
+
+void DatabaseService::start() {
+    if (running_.exchange(true)) {
+        return;
+    }
+
+    spdlog::info("Starting database service on {}:{}", config_.bindAddress, config_.rpcPort);
+
+    // Start components
+    store_->start();
+    persistence_->start();
+    events_->start();
+    files_->start();
+
+    if (config_.replicationEnabled) {
+        replication_->start();
+        // Set initial local collections for discovery
+        replication_->setLocalCollections(store_->listCollections());
+    }
+
+    // Start gRPC server
+    startGrpcServer();
+
+    spdlog::info("Database service started");
+}
+
+void DatabaseService::stop() {
+    if (!running_.exchange(false)) {
+        return;
+    }
+
+    spdlog::info("Stopping database service...");
+
+    // Stop gRPC server first
+    stopGrpcServer();
+
+    // Stop components in reverse order
+    if (replication_) {
+        replication_->stop();
+    }
+    if (files_) {
+        files_->stop();
+    }
+    if (events_) {
+        events_->stop();
+    }
+    if (persistence_) {
+        persistence_->stop();
+    }
+    if (store_) {
+        store_->stop();
+    }
+
+    spdlog::info("Database service stopped");
+}
+
+void DatabaseService::wait() {
+    if (serverThread_.joinable()) {
+        serverThread_.join();
+    }
+}
+
+void DatabaseService::signalStop() {
+    stopRequested_ = true;
+    stop();
+}
+
+nlohmann::json DatabaseService::getStats() const {
+    nlohmann::json stats;
+
+    if (store_) {
+        auto storeStats = store_->getStats();
+        stats["documents"] = storeStats.totalDocuments;
+        stats["collections"] = storeStats.totalCollections;
+        stats["memory_bytes"] = storeStats.estimatedMemoryBytes;
+        stats["inserts"] = storeStats.insertCount;
+        stats["updates"] = storeStats.updateCount;
+        stats["deletes"] = storeStats.deleteCount;
+        stats["queries"] = storeStats.queryCount;
+    }
+
+    if (persistence_) {
+        auto persistStats = persistence_->getStats();
+        stats["wal_sequence"] = persistStats.walSequence;
+        stats["wal_size_bytes"] = persistStats.walSizeBytes;
+        stats["snapshot_count"] = persistStats.snapshotCount;
+        stats["last_snapshot_sequence"] = persistStats.lastSnapshotSequence;
+    }
+
+    if (events_) {
+        stats["subscriptions"] = events_->subscriptionCount();
+    }
+
+    return stats;
+}
+
+DatabaseService::Config DatabaseService::loadConfig(const std::filesystem::path& configPath) {
+    std::ifstream file(configPath);
+    if (!file) {
+        throw std::runtime_error("Failed to open config file: " + configPath.string());
+    }
+
+    nlohmann::json json;
+    file >> json;
+
+    Config config;
+
+    // Check for both "storage" and "database" keys for backward compatibility
+    nlohmann::json* dbConfig = nullptr;
+    if (json.contains("database")) {
+        dbConfig = &json["database"];
+    } else if (json.contains("storage")) {
+        dbConfig = &json["storage"];
+    }
+
+    if (dbConfig) {
+        auto& db = *dbConfig;
+
+        config.nodeId = db.value("node_id", config.nodeId);
+        config.bindAddress = db.value("bind_address", config.bindAddress);
+        config.rpcPort = db.value("rpc_port", config.rpcPort);
+
+        // Expand environment variables in data directory
+        std::string dataDir = db.value("data_directory", "");
+        if (dataDir.find("${HOME}") != std::string::npos) {
+            const char* home = std::getenv("HOME");
+            if (home) {
+                size_t pos = dataDir.find("${HOME}");
+                dataDir.replace(pos, 7, home);
+            }
+        }
+        config.dataDirectory = dataDir;
+
+        // Migrations settings
+        if (db.contains("migrations")) {
+            auto& migrations = db["migrations"];
+            config.migrations.enabled = migrations.value("enabled", config.migrations.enabled);
+            config.migrations.autoApply = migrations.value("auto_apply", config.migrations.autoApply);
+            config.migrations.failOnError = migrations.value("fail_on_error", config.migrations.failOnError);
+
+            std::string migDir = migrations.value("directory", "");
+            if (migDir.find("${HOME}") != std::string::npos) {
+                const char* home = std::getenv("HOME");
+                if (home) {
+                    size_t pos = migDir.find("${HOME}");
+                    migDir.replace(pos, 7, home);
+                }
+            }
+            config.migrations.directory = migDir;
+        }
+
+        // Persistence settings
+        if (db.contains("persistence")) {
+            auto& persistence = db["persistence"];
+            config.walSyncIntervalMs = persistence.value("wal_sync_interval_ms", config.walSyncIntervalMs);
+            config.snapshotIntervalSec = persistence.value("snapshot_interval_sec", config.snapshotIntervalSec);
+            config.compressionEnabled = persistence.value("compression", "lz4") != "none";
+        }
+
+        // File settings
+        if (db.contains("files")) {
+            auto& files = db["files"];
+            config.maxFileSizeMb = files.value("max_file_size_mb", config.maxFileSizeMb);
+            config.fileCleanupIntervalSec = files.value("cleanup_orphans_interval_sec", config.fileCleanupIntervalSec);
+            if (files.contains("allowed_types")) {
+                for (const auto& type : files["allowed_types"]) {
+                    config.allowedFileTypes.push_back(type.get<std::string>());
+                }
+            }
+        }
+
+        // Encryption settings
+        if (db.contains("encryption")) {
+            auto& encryption = db["encryption"];
+            config.encryptionEnabled = encryption.value("enabled", config.encryptionEnabled);
+            config.autoGenerateKey = encryption.value("auto_generate_key", config.autoGenerateKey);
+
+            std::string keyFile = encryption.value("key_file", "");
+            if (keyFile.find("${HOME}") != std::string::npos) {
+                const char* home = std::getenv("HOME");
+                if (home) {
+                    size_t pos = keyFile.find("${HOME}");
+                    keyFile.replace(pos, 7, home);
+                }
+            }
+            config.keyFilePath = keyFile;
+        }
+
+        // Replication settings
+        if (db.contains("replication")) {
+            auto& replication = db["replication"];
+            config.replicationEnabled = replication.value("enabled", config.replicationEnabled);
+            config.conflictResolution = replication.value("conflict_resolution", config.conflictResolution);
+            if (replication.contains("peers")) {
+                for (const auto& peer : replication["peers"]) {
+                    config.peerAddresses.push_back(peer.get<std::string>());
+                }
+            }
+        }
+    }
+
+    return config;
+}
+
+void DatabaseService::setupComponents() {
+    // Create memory store
+    MemoryStore::Config storeConfig;
+    storeConfig.nodeId = config_.nodeId;
+    store_ = std::make_unique<MemoryStore>(storeConfig);
+
+    // Create persistence manager
+    PersistenceManager::Config persistConfig;
+    persistConfig.dataDir = config_.dataDirectory;
+    persistConfig.walSyncIntervalMs = config_.walSyncIntervalMs;
+    persistConfig.snapshotIntervalSec = config_.snapshotIntervalSec;
+    persistConfig.compressionEnabled = config_.compressionEnabled;
+    persistence_ = std::make_unique<PersistenceManager>(persistConfig);
+
+    // Connect store callbacks to persistence - uses setPersistCallback for WAL logging
+    store_->setPersistCallback([this](const std::string& collection, const std::string& id,
+                                       const std::optional<Document>& doc, EventType eventType) {
+        // Log to WAL based on event type
+        switch (eventType) {
+            case EventType::INSERT:
+                if (doc) persistence_->logInsert(collection, *doc);
+                break;
+            case EventType::UPDATE:
+                if (doc) persistence_->logUpdate(collection, *doc);
+                break;
+            case EventType::DELETE:
+                persistence_->logDelete(collection, id);
+                break;
+            default:
+                break;
+        }
+
+        // Queue for replication broadcast
+        if (replication_ && config_.replicationEnabled) {
+            databasepb::ReplicationEntry entry;
+            entry.set_collection(collection);
+            entry.set_document_id(id);
+            entry.set_global_timestamp(
+                std::chrono::duration_cast<std::chrono::milliseconds>(
+                    std::chrono::system_clock::now().time_since_epoch()
+                ).count());
+
+            switch (eventType) {
+                case EventType::INSERT:
+                    entry.set_op(databasepb::OP_INSERT);
+                    if (doc) entry.set_data(doc->data.dump());
+                    break;
+                case EventType::UPDATE:
+                    entry.set_op(databasepb::OP_UPDATE);
+                    if (doc) entry.set_data(doc->data.dump());
+                    break;
+                case EventType::DELETE:
+                    entry.set_op(databasepb::OP_DELETE);
+                    break;
+                default:
+                    break;
+            }
+
+            replication_->queueForReplication(entry);
+        }
+
+        // Publish event
+        if (events_) {
+            DatabaseEvent event;
+            event.type = eventType;
+            event.collection = collection;
+            event.documentId = id;
+            event.timestamp = std::chrono::duration_cast<std::chrono::milliseconds>(
+                std::chrono::system_clock::now().time_since_epoch()
+            ).count();
+            event.nodeId = config_.nodeId;
+            if (doc) {
+                event.data = doc->data;
+            }
+            events_->publish(event);
+        }
+    });
+
+    // Create encryption manager
+    EncryptionManager::Config encryptConfig;
+    encryptConfig.enabled = config_.encryptionEnabled;
+    encryptConfig.keyFilePath = config_.keyFilePath;
+    encryptConfig.autoGenerateKey = config_.autoGenerateKey;
+    encryption_ = std::make_unique<EncryptionManager>(encryptConfig);
+
+    // Create event manager
+    EventManager::Config eventConfig;
+    eventConfig.nodeId = config_.nodeId;
+    events_ = std::make_unique<EventManager>(eventConfig);
+
+    // Create file manager
+    FileManager::Config fileConfig;
+    fileConfig.filesDir = config_.dataDirectory / "files";
+    fileConfig.maxFileSizeMb = config_.maxFileSizeMb;
+    fileConfig.allowedMimeTypes = config_.allowedFileTypes;
+    fileConfig.cleanupIntervalSec = config_.fileCleanupIntervalSec;
+    files_ = std::make_unique<FileManager>(fileConfig);
+
+    // Create replication manager
+    ReplicationManager::Config replConfig;
+    replConfig.nodeId = config_.nodeId;
+    replConfig.peerAddresses = config_.peerAddresses;
+    replConfig.conflictResolution = config_.conflictResolution;
+    replication_ = std::make_unique<ReplicationManager>(replConfig);
+
+    // Wire replication entry apply callback
+    replication_->setEntryApplyCallback([this](const databasepb::ReplicationEntry& entry) {
+        applyReplicatedEntry(entry);
+    });
+
+    // Create gRPC implementations
+    storageImpl_ = std::make_unique<DatabaseGrpcImpl>(
+        *store_, *persistence_, *events_, *files_, *encryption_
+    );
+    replicationImpl_ = std::make_unique<DatabaseReplicationGrpcImpl>(
+        *store_, *replication_, *persistence_
+    );
+}
+
+void DatabaseService::applyReplicatedEntry(const databasepb::ReplicationEntry& entry) {
+    try {
+        switch (entry.op()) {
+            case databasepb::OP_INSERT:
+            case databasepb::OP_UPDATE:
+            case databasepb::OP_UPSERT: {
+                if (entry.data().empty()) {
+                    spdlog::warn("Replication entry has no data for op {}", static_cast<int>(entry.op()));
+                    return;
+                }
+
+                auto json = nlohmann::json::parse(entry.data());
+                Document doc = Document::fromJson(json);
+                doc.id = entry.document_id();
+                doc.nodeId = entry.node_id();
+
+                // Use loadDocument to bypass normal callbacks (avoid re-replication)
+                store_->loadDocument(entry.collection(), doc);
+                spdlog::trace("Applied replicated {} to {}/{} from {}",
+                             entry.op() == databasepb::OP_INSERT ? "insert" :
+                             entry.op() == databasepb::OP_UPDATE ? "update" : "upsert",
+                             entry.collection(), entry.document_id(), entry.node_id());
+                break;
+            }
+
+            case databasepb::OP_DELETE: {
+                store_->remove(entry.collection(), entry.document_id());
+                spdlog::trace("Applied replicated delete to {}/{} from {}",
+                             entry.collection(), entry.document_id(), entry.node_id());
+                break;
+            }
+
+            case databasepb::OP_CREATE_COLLECTION: {
+                CollectionOptions options;
+                store_->createCollection(entry.collection(), options);
+                spdlog::trace("Applied replicated create collection {} from {}",
+                             entry.collection(), entry.node_id());
+                break;
+            }
+
+            case databasepb::OP_DROP_COLLECTION: {
+                store_->dropCollection(entry.collection());
+                spdlog::trace("Applied replicated drop collection {} from {}",
+                             entry.collection(), entry.node_id());
+                break;
+            }
+
+            default:
+                spdlog::warn("Unknown replication operation type: {}", static_cast<int>(entry.op()));
+                break;
+        }
+    } catch (const std::exception& e) {
+        spdlog::error("Failed to apply replicated entry: {}", e.what());
+    }
+}
+
+void DatabaseService::startGrpcServer() {
+    std::string serverAddress = config_.bindAddress + ":" + std::to_string(config_.rpcPort);
+
+    grpc::ServerBuilder builder;
+    builder.AddListeningPort(serverAddress, grpc::InsecureServerCredentials());
+    builder.RegisterService(storageImpl_.get());
+    builder.RegisterService(replicationImpl_.get());
+
+    // Configure server
+    builder.SetMaxReceiveMessageSize(100 * 1024 * 1024);  // 100MB for file uploads
+    builder.SetMaxSendMessageSize(100 * 1024 * 1024);
+
+    grpcServer_ = builder.BuildAndStart();
+
+    if (!grpcServer_) {
+        throw std::runtime_error("Failed to start gRPC server on " + serverAddress);
+    }
+
+    spdlog::info("gRPC server listening on {}", serverAddress);
+}
+
+void DatabaseService::stopGrpcServer() {
+    if (grpcServer_) {
+        grpcServer_->Shutdown();
+        grpcServer_.reset();
+    }
+}
+
+} // namespace smartbotic::database

service/src/database_service.hpp

@@ -0,0 +1,149 @@
+#pragma once
+
+#include "document.hpp"
+#include "memory_store.hpp"
+#include "database_grpc_impl.hpp"
+#include "persistence/persistence_manager.hpp"
+#include "encryption/encryption_manager.hpp"
+#include "events/event_manager.hpp"
+#include "files/file_manager.hpp"
+#include "replication/replication_manager.hpp"
+#include "migrations/migration_runner.hpp"
+
+#include <nlohmann/json.hpp>
+
+#include <atomic>
+#include <filesystem>
+#include <memory>
+#include <string>
+#include <thread>
+
+namespace grpc {
+class Server;
+}
+
+namespace smartbotic::database {
+
+/**
+ * Main storage service application.
+ * Coordinates all components and manages the gRPC server.
+ */
+class DatabaseService {
+public:
+    struct Config {
+        std::string nodeId = "storage-1";
+        std::string bindAddress = "0.0.0.0";
+        uint16_t rpcPort = 9004;
+
+        // Bootstrap/service registration
+        std::string webapiUrl;
+        std::string serviceKey;
+        uint32_t heartbeatIntervalSec = 30;
+
+        std::filesystem::path dataDirectory;
+        std::filesystem::path keyFilePath;
+
+        // Persistence settings
+        uint32_t walSyncIntervalMs = 100;
+        uint32_t snapshotIntervalSec = 3600;
+        bool compressionEnabled = true;
+
+        // File storage settings
+        uint64_t maxFileSizeMb = 500;
+        std::vector<std::string> allowedFileTypes;
+        uint32_t fileCleanupIntervalSec = 3600;
+
+        // Encryption settings
+        bool encryptionEnabled = true;
+        bool autoGenerateKey = true;
+
+        // Replication settings
+        bool replicationEnabled = true;
+        std::vector<std::string> peerAddresses;
+        std::string conflictResolution = "last_writer_wins";
+
+        // Migration settings
+        struct MigrationsConfig {
+            bool enabled = true;
+            std::filesystem::path directory;
+            bool autoApply = true;
+            bool failOnError = true;
+        } migrations;
+    };
+
+    explicit DatabaseService(Config config);
+    ~DatabaseService();
+
+    /**
+     * Initialize the service.
+     * Loads configuration, initializes components, and prepares for startup.
+     */
+    bool initialize();
+
+    /**
+     * Start the service.
+     * Starts the gRPC server and all background threads.
+     */
+    void start();
+
+    /**
+     * Stop the service.
+     * Gracefully shuts down all components.
+     */
+    void stop();
+
+    /**
+     * Wait for the service to stop.
+     */
+    void wait();
+
+    /**
+     * Check if the service is running.
+     */
+    [[nodiscard]] bool isRunning() const { return running_; }
+
+    /**
+     * Signal the service to stop (for signal handlers).
+     */
+    void signalStop();
+
+    /**
+     * Get service statistics.
+     */
+    [[nodiscard]] nlohmann::json getStats() const;
+
+    /**
+     * Load configuration from a JSON file.
+     */
+    static Config loadConfig(const std::filesystem::path& configPath);
+
+private:
+    void setupComponents();
+    void startGrpcServer();
+    void stopGrpcServer();
+    void applyReplicatedEntry(const databasepb::ReplicationEntry& entry);
+    bool runMigrations();
+
+    Config config_;
+    std::atomic<bool> running_{false};
+    std::atomic<bool> stopRequested_{false};
+
+    // Components
+    std::unique_ptr<MemoryStore> store_;
+    std::unique_ptr<PersistenceManager> persistence_;
+    std::unique_ptr<EncryptionManager> encryption_;
+    std::unique_ptr<EventManager> events_;
+    std::unique_ptr<FileManager> files_;
+    std::unique_ptr<ReplicationManager> replication_;
+
+    // gRPC
+    std::unique_ptr<DatabaseGrpcImpl> storageImpl_;
+    std::unique_ptr<DatabaseReplicationGrpcImpl> replicationImpl_;
+    std::unique_ptr<grpc::Server> grpcServer_;
+    std::thread serverThread_;
+
+    // Migration runner
+    std::unique_ptr<MigrationRunner> migrationRunner_;
+};
+
+} // namespace smartbotic::database

service/src/document.hpp

@@ -0,0 +1,301 @@
+#pragma once
+
+#include <string>
+#include <vector>
+#include <chrono>
+#include <optional>
+#include <nlohmann/json.hpp>
+
+namespace smartbotic::database {
+
+/**
+ * Represents a document stored in a collection.
+ * Documents are JSON objects with metadata for versioning, TTL, and encryption.
+ * Named Document to avoid conflict with proto-generated Document message.
+ */
+struct Document {
+    std::string id;                              // Unique document ID within collection
+    std::string collection;                      // Collection name
+    nlohmann::json data;                         // Document content (JSON)
+    uint64_t version = 1;                        // Monotonic version for optimistic locking
+    uint64_t createdAt = 0;                      // Creation timestamp (ms since epoch)
+    uint64_t updatedAt = 0;                      // Last modification timestamp (ms since epoch)
+    uint64_t expiresAt = 0;                      // TTL expiration (0 = no expiration)
+    std::string nodeId;                          // Origin node for replication
+    bool encrypted = false;                      // Whether sensitive fields are encrypted
+    std::vector<std::string> encryptedFields;   // List of encrypted field paths
+    std::string createdBy;                       // User ID who created the document
+    std::string updatedBy;                       // User ID who last updated the document
+
+    // Check if document has expired
+    [[nodiscard]] bool isExpired() const {
+        if (expiresAt == 0) return false;
+        auto now = std::chrono::duration_cast<std::chrono::milliseconds>(
+            std::chrono::system_clock::now().time_since_epoch()
+        ).count();
+        return static_cast<uint64_t>(now) >= expiresAt;
+    }
+
+    // Set TTL in seconds from now
+    void setTtlSeconds(uint32_t seconds) {
+        if (seconds == 0) {
+            expiresAt = 0;
+        } else {
+            auto now = std::chrono::duration_cast<std::chrono::milliseconds>(
+                std::chrono::system_clock::now().time_since_epoch()
+            ).count();
+            expiresAt = static_cast<uint64_t>(now) + (seconds * 1000ULL);
+        }
+    }
+
+    // Get remaining TTL in seconds (0 if expired or no TTL)
+    [[nodiscard]] uint32_t getRemainingTtlSeconds() const {
+        if (expiresAt == 0) return 0;
+        auto now = std::chrono::duration_cast<std::chrono::milliseconds>(
+            std::chrono::system_clock::now().time_since_epoch()
+        ).count();
+        if (static_cast<uint64_t>(now) >= expiresAt) return 0;
+        return static_cast<uint32_t>((expiresAt - static_cast<uint64_t>(now)) / 1000);
+    }
+
+    // JSON serialization
+    [[nodiscard]] nlohmann::json toJson() const {
+        nlohmann::json j;
+        j["id"] = id;
+        j["collection"] = collection;
+        j["data"] = data;
+        j["version"] = version;
+        j["createdAt"] = createdAt;
+        j["updatedAt"] = updatedAt;
+        j["expiresAt"] = expiresAt;
+        j["nodeId"] = nodeId;
+        j["encrypted"] = encrypted;
+        j["encryptedFields"] = encryptedFields;
+        j["createdBy"] = createdBy;
+        j["updatedBy"] = updatedBy;
+        return j;
+    }
+
+    // JSON deserialization
+    static Document fromJson(const nlohmann::json& j) {
+        Document doc;
+        doc.id = j.value("id", "");
+        doc.collection = j.value("collection", "");
+        doc.data = j.value("data", nlohmann::json::object());
+        doc.version = j.value("version", 1ULL);
+        doc.createdAt = j.value("createdAt", 0ULL);
+        doc.updatedAt = j.value("updatedAt", 0ULL);
+        doc.expiresAt = j.value("expiresAt", 0ULL);
+        doc.nodeId = j.value("nodeId", "");
+        doc.encrypted = j.value("encrypted", false);
+        doc.encryptedFields = j.value("encryptedFields", std::vector<std::string>{});
+        doc.createdBy = j.value("createdBy", "");
+        doc.updatedBy = j.value("updatedBy", "");
+        return doc;
+    }
+};
+
+/**
+ * Options for creating a collection.
+ * Named CollectionOptions to avoid conflict with proto-generated type.
+ */
+struct CollectionOptions {
+    bool autoCreateId = true;                    // Auto-generate IDs if not provided
+    uint32_t defaultTtlSeconds = 0;              // Default TTL for documents (0 = none)
+    bool encrypted = false;                      // Encrypt all documents by default
+    std::vector<std::string> sensitiveFields;   // Fields to always encrypt
+
+    [[nodiscard]] nlohmann::json toJson() const {
+        nlohmann::json j;
+        j["autoCreateId"] = autoCreateId;
+        j["defaultTtlSeconds"] = defaultTtlSeconds;
+        j["encrypted"] = encrypted;
+        j["sensitiveFields"] = sensitiveFields;
+        return j;
+    }
+
+    static CollectionOptions fromJson(const nlohmann::json& j) {
+        CollectionOptions opts;
+        opts.autoCreateId = j.value("autoCreateId", true);
+        opts.defaultTtlSeconds = j.value("defaultTtlSeconds", 0U);
+        opts.encrypted = j.value("encrypted", false);
+        opts.sensitiveFields = j.value("sensitiveFields", std::vector<std::string>{});
+        return opts;
+    }
+};
+
+/**
+ * Collection metadata.
+ */
+struct CollectionInfo {
+    std::string name;
+    uint64_t documentCount = 0;
+    uint64_t sizeBytes = 0;
+    CollectionOptions options;
+    uint64_t createdAt = 0;
+    uint64_t updatedAt = 0;
+
+    [[nodiscard]] nlohmann::json toJson() const {
+        nlohmann::json j;
+        j["name"] = name;
+        j["documentCount"] = documentCount;
+        j["sizeBytes"] = sizeBytes;
+        j["options"] = options.toJson();
+        j["createdAt"] = createdAt;
+        j["updatedAt"] = updatedAt;
+        return j;
+    }
+};
+
+/**
+ * Filter operator for queries.
+ */
+enum class FilterOp {
+    EQ,        // Equal
+    NE,        // Not equal
+    GT,        // Greater than
+    GTE,       // Greater than or equal
+    LT,        // Less than
+    LTE,       // Less than or equal
+    IN,        // Value in array
+    CONTAINS,  // Array contains value
+    EXISTS,    // Field exists
+    REGEX,     // Regex match
+    SEARCH     // Full-text search across ID and string fields
+};
+
+/**
+ * Query filter.
+ */
+struct Filter {
+    std::string field;       // JSON path (e.g., "status" or "user.email")
+    FilterOp op = FilterOp::EQ;
+    nlohmann::json value;
+
+    [[nodiscard]] nlohmann::json toJson() const {
+        nlohmann::json j;
+        j["field"] = field;
+        j["op"] = static_cast<int>(op);
+        j["value"] = value;
+        return j;
+    }
+
+    static Filter fromJson(const nlohmann::json& j) {
+        Filter f;
+        f.field = j.value("field", "");
+        f.op = static_cast<FilterOp>(j.value("op", 0));
+        f.value = j.value("value", nlohmann::json());
+        return f;
+    }
+};
+
+/**
+ * Sort specification.
+ */
+struct Sort {
+    std::string field;
+    bool descending = false;
+
+    [[nodiscard]] nlohmann::json toJson() const {
+        nlohmann::json j;
+        j["field"] = field;
+        j["descending"] = descending;
+        return j;
+    }
+
+    static Sort fromJson(const nlohmann::json& j) {
+        Sort s;
+        s.field = j.value("field", "");
+        s.descending = j.value("descending", false);
+        return s;
+    }
+};
+
+/**
+ * Query specification.
+ */
+struct Query {
+    std::vector<Filter> filters;
+    std::optional<Sort> sort;
+    uint32_t limit = 100;
+    uint32_t offset = 0;
+    std::vector<std::string> projection;  // Fields to include (empty = all)
+
+    [[nodiscard]] nlohmann::json toJson() const {
+        nlohmann::json j;
+        j["filters"] = nlohmann::json::array();
+        for (const auto& f : filters) {
+            j["filters"].push_back(f.toJson());
+        }
+        if (sort) {
+            j["sort"] = sort->toJson();
+        }
+        j["limit"] = limit;
+        j["offset"] = offset;
+        j["projection"] = projection;
+        return j;
+    }
+
+    static Query fromJson(const nlohmann::json& j) {
+        Query q;
+        if (j.contains("filters") && j["filters"].is_array()) {
+            for (const auto& f : j["filters"]) {
+                q.filters.push_back(Filter::fromJson(f));
+            }
+        }
+        if (j.contains("sort") && !j["sort"].is_null()) {
+            q.sort = Sort::fromJson(j["sort"]);
+        }
+        q.limit = j.value("limit", 100U);
+        q.offset = j.value("offset", 0U);
+        q.projection = j.value("projection", std::vector<std::string>{});
+        return q;
+    }
+};
+
+/**
+ * Result of a find query.
+ */
+struct QueryResult {
+    std::vector<Document> documents;
+    uint64_t totalCount = 0;
+    bool hasMore = false;
+};
+
+/**
+ * Storage event types.
+ */
+enum class EventType {
+    INSERT,
+    UPDATE,
+    DELETE,
+    EXPIRE,
+    INVALIDATE
+};
+
+/**
+ * Storage event for pub/sub.
+ */
+struct DatabaseEvent {
+    EventType type;
+    std::string collection;
+    std::string documentId;
+    uint64_t timestamp = 0;
+    std::string nodeId;
+    std::optional<nlohmann::json> data;  // Document data (if included)
+
+    [[nodiscard]] nlohmann::json toJson() const {
+        nlohmann::json j;
+        j["type"] = static_cast<int>(type);
+        j["collection"] = collection;
+        j["documentId"] = documentId;
+        j["timestamp"] = timestamp;
+        j["nodeId"] = nodeId;
+        if (data) {
+            j["data"] = *data;
+        }
+        return j;
+    }
+};
+
+} // namespace smartbotic::database

service/src/encryption/encryption_manager.cpp

@@ -0,0 +1,642 @@
+#include "encryption_manager.hpp"
+
+#include <spdlog/spdlog.h>
+
+#include <openssl/evp.h>
+#include <openssl/rand.h>
+#include <openssl/err.h>
+
+#include <algorithm>
+#include <cstring>
+#include <fstream>
+#include <sstream>
+#include <stdexcept>
+
+namespace smartbotic::database {
+
+namespace {
+
+constexpr size_t KEY_SIZE = 32;     // AES-256
+constexpr size_t IV_SIZE = 12;      // GCM IV
+constexpr size_t TAG_SIZE = 16;     // GCM Auth tag
+constexpr char ENCRYPTED_PREFIX[] = "$ENC$";
+
+// Base64 encoding/decoding helpers
+std::string base64Encode(const std::vector<uint8_t>& data) {
+    static const char* alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
+
+    std::string result;
+    result.reserve((data.size() + 2) / 3 * 4);
+
+    for (size_t i = 0; i < data.size(); i += 3) {
+        uint32_t octet_a = data[i];
+        uint32_t octet_b = (i + 1 < data.size()) ? data[i + 1] : 0;
+        uint32_t octet_c = (i + 2 < data.size()) ? data[i + 2] : 0;
+
+        uint32_t triple = (octet_a << 16) | (octet_b << 8) | octet_c;
+
+        result += alphabet[(triple >> 18) & 0x3F];
+        result += alphabet[(triple >> 12) & 0x3F];
+        result += (i + 1 < data.size()) ? alphabet[(triple >> 6) & 0x3F] : '=';
+        result += (i + 2 < data.size()) ? alphabet[triple & 0x3F] : '=';
+    }
+
+    return result;
+}
+
+std::vector<uint8_t> base64Decode(const std::string& encoded) {
+    static const int decode_table[] = {
+        -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+        -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+        -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,62,-1,-1,-1,63,
+        52,53,54,55,56,57,58,59,60,61,-1,-1,-1,-1,-1,-1,
+        -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9,10,11,12,13,14,
+        15,16,17,18,19,20,21,22,23,24,25,-1,-1,-1,-1,-1,
+        -1,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,
+        41,42,43,44,45,46,47,48,49,50,51,-1,-1,-1,-1,-1
+    };
+
+    std::vector<uint8_t> result;
+    result.reserve(encoded.size() * 3 / 4);
+
+    uint32_t buf = 0;
+    int bits = 0;
+
+    for (char c : encoded) {
+        if (c == '=') break;
+        if (c < 0 || c >= 128) continue;
+
+        int val = decode_table[static_cast<unsigned char>(c)];
+        if (val < 0) continue;
+
+        buf = (buf << 6) | val;
+        bits += 6;
+
+        if (bits >= 8) {
+            bits -= 8;
+            result.push_back(static_cast<uint8_t>((buf >> bits) & 0xFF));
+        }
+    }
+
+    return result;
+}
+
+// Convert glob pattern to regex
+std::string globToRegex(const std::string& glob) {
+    std::string regex;
+    regex.reserve(glob.size() * 2);
+
+    for (char c : glob) {
+        switch (c) {
+            case '*':
+                regex += ".*";
+                break;
+            case '?':
+                regex += ".";
+                break;
+            case '.':
+            case '+':
+            case '^':
+            case '$':
+            case '(':
+            case ')':
+            case '[':
+            case ']':
+            case '{':
+            case '}':
+            case '|':
+            case '\\':
+                regex += '\\';
+                regex += c;
+                break;
+            default:
+                regex += c;
+                break;
+        }
+    }
+
+    return regex;
+}
+
+} // anonymous namespace
+
+EncryptionManager::EncryptionManager(Config config)
+    : config_(std::move(config))
+{
+    // Compile sensitive field patterns
+    for (const auto& pattern : config_.sensitiveFieldPatterns) {
+        try {
+            sensitivePatterns_.emplace_back(
+                globToRegex(pattern),
+                std::regex::icase
+            );
+        } catch (const std::regex_error& e) {
+            spdlog::warn("Invalid sensitive field pattern '{}': {}", pattern, e.what());
+        }
+    }
+}
+
+EncryptionManager::~EncryptionManager() {
+    // Securely clear the master key
+    if (!masterKey_.empty()) {
+        std::fill(masterKey_.begin(), masterKey_.end(), 0);
+        masterKey_.clear();
+    }
+}
+
+bool EncryptionManager::initialize() {
+    if (!config_.enabled) {
+        spdlog::info("Encryption disabled");
+        initialized_ = true;
+        return true;
+    }
+
+    std::lock_guard lock(mutex_);
+
+    // Try to load existing key
+    bool keyLoaded = false;
+
+    if (config_.keyProvider == "file") {
+        keyLoaded = loadKeyFromFile();
+    } else if (config_.keyProvider == "env") {
+        keyLoaded = loadKeyFromEnv();
+    }
+
+    // Generate new key if needed and allowed
+    if (!keyLoaded && config_.autoGenerateKey) {
+        if (!generateKey()) {
+            spdlog::error("Failed to generate encryption key");
+            return false;
+        }
+
+        // Save the generated key if using file provider
+        if (config_.keyProvider == "file") {
+            if (!saveKeyToFile()) {
+                spdlog::error("CRITICAL: Failed to save encryption key to file - data will be lost on restart!");
+                return false;
+            }
+        }
+    } else if (!keyLoaded) {
+        spdlog::error("No encryption key available and auto-generation disabled");
+        return false;
+    }
+
+    initialized_ = true;
+    spdlog::info("Encryption initialized (key provider: {})", config_.keyProvider);
+
+    return true;
+}
+
+void EncryptionManager::encryptSensitiveFields(Document& doc) {
+    if (!isEnabled()) {
+        return;
+    }
+
+    std::lock_guard lock(mutex_);
+
+    // Traverse JSON and encrypt sensitive fields
+    encryptJsonField(doc.data, "");
+
+    doc.encrypted = true;
+}
+
+void EncryptionManager::decryptSensitiveFields(Document& doc) {
+    if (!isEnabled() || !doc.encrypted) {
+        return;
+    }
+
+    std::lock_guard lock(mutex_);
+
+    // Traverse JSON and decrypt sensitive fields
+    decryptJsonField(doc.data, "");
+}
+
+std::string EncryptionManager::encryptValue(const std::string& plaintext) {
+    if (!isEnabled()) {
+        return plaintext;
+    }
+
+    SPDLOG_INFO("Encrypting value of length {} bytes, first byte={:02x}, last byte={:02x}",
+                plaintext.length(),
+                plaintext.empty() ? 0 : static_cast<uint8_t>(plaintext[0]),
+                plaintext.empty() ? 0 : static_cast<uint8_t>(plaintext[plaintext.length()-1]));
+    std::vector<uint8_t> plaintextBytes(plaintext.begin(), plaintext.end());
+    auto encrypted = encryptData(plaintextBytes);
+    SPDLOG_INFO("Encrypted data size: {} bytes (IV=12 + CT={} + Tag=16)", encrypted.size(), encrypted.size() - 28);
+
+    return std::string(ENCRYPTED_PREFIX) + base64Encode(encrypted);
+}
+
+std::string EncryptionManager::decryptValue(const std::string& ciphertext) {
+    if (!isEnabled()) {
+        return ciphertext;
+    }
+
+    // Check for encryption prefix
+    if (ciphertext.find(ENCRYPTED_PREFIX) != 0) {
+        return ciphertext;  // Not encrypted
+    }
+
+    std::string encoded = ciphertext.substr(strlen(ENCRYPTED_PREFIX));
+    auto encrypted = base64Decode(encoded);
+
+    auto decrypted = decryptData(encrypted);
+
+    return std::string(decrypted.begin(), decrypted.end());
+}
+
+std::vector<uint8_t> EncryptionManager::encryptData(const std::vector<uint8_t>& plaintext) {
+    if (!isEnabled() || masterKey_.empty()) {
+        return plaintext;
+    }
+
+    // Generate random IV
+    std::vector<uint8_t> iv(IV_SIZE);
+    if (RAND_bytes(iv.data(), static_cast<int>(IV_SIZE)) != 1) {
+        throw std::runtime_error("Failed to generate random IV");
+    }
+
+    // Create cipher context
+    EVP_CIPHER_CTX* ctx = EVP_CIPHER_CTX_new();
+    if (!ctx) {
+        throw std::runtime_error("Failed to create cipher context");
+    }
+
+    std::vector<uint8_t> ciphertext(plaintext.size() + EVP_MAX_BLOCK_LENGTH);
+    std::vector<uint8_t> tag(TAG_SIZE);
+    int len = 0;
+    int ciphertextLen = 0;
+
+    try {
+        // Initialize encryption
+        if (EVP_EncryptInit_ex(ctx, EVP_aes_256_gcm(), nullptr, nullptr, nullptr) != 1) {
+            throw std::runtime_error("Failed to initialize AES-256-GCM");
+        }
+
+        // Set IV length
+        if (EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, static_cast<int>(IV_SIZE), nullptr) != 1) {
+            throw std::runtime_error("Failed to set IV length");
+        }
+
+        // Set key and IV
+        if (EVP_EncryptInit_ex(ctx, nullptr, nullptr, masterKey_.data(), iv.data()) != 1) {
+            throw std::runtime_error("Failed to set key and IV");
+        }
+
+        // Encrypt data
+        if (EVP_EncryptUpdate(ctx, ciphertext.data(), &len, plaintext.data(),
+                             static_cast<int>(plaintext.size())) != 1) {
+            throw std::runtime_error("Failed to encrypt data");
+        }
+        ciphertextLen = len;
+
+        // Finalize encryption
+        if (EVP_EncryptFinal_ex(ctx, ciphertext.data() + len, &len) != 1) {
+            throw std::runtime_error("Failed to finalize encryption");
+        }
+        ciphertextLen += len;
+
+        // Get auth tag
+        if (EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_GET_TAG, static_cast<int>(TAG_SIZE), tag.data()) != 1) {
+            throw std::runtime_error("Failed to get auth tag");
+        }
+
+        EVP_CIPHER_CTX_free(ctx);
+
+    } catch (...) {
+        EVP_CIPHER_CTX_free(ctx);
+        throw;
+    }
+
+    // Combine: IV + ciphertext + tag
+    std::vector<uint8_t> result;
+    result.reserve(IV_SIZE + ciphertextLen + TAG_SIZE);
+    result.insert(result.end(), iv.begin(), iv.end());
+    result.insert(result.end(), ciphertext.begin(), ciphertext.begin() + ciphertextLen);
+    result.insert(result.end(), tag.begin(), tag.end());
+
+    return result;
+}
+
+std::vector<uint8_t> EncryptionManager::decryptData(const std::vector<uint8_t>& ciphertext) {
+    if (!isEnabled() || masterKey_.empty()) {
+        return ciphertext;
+    }
+
+    if (ciphertext.size() < IV_SIZE + TAG_SIZE) {
+        throw std::runtime_error("Ciphertext too short");
+    }
+
+    // Extract IV, encrypted data, and tag
+    std::vector<uint8_t> iv(ciphertext.begin(), ciphertext.begin() + IV_SIZE);
+    std::vector<uint8_t> tag(ciphertext.end() - TAG_SIZE, ciphertext.end());
+    std::vector<uint8_t> encrypted(ciphertext.begin() + IV_SIZE, ciphertext.end() - TAG_SIZE);
+
+    // Create cipher context
+    EVP_CIPHER_CTX* ctx = EVP_CIPHER_CTX_new();
+    if (!ctx) {
+        throw std::runtime_error("Failed to create cipher context");
+    }
+
+    std::vector<uint8_t> plaintext(encrypted.size() + EVP_MAX_BLOCK_LENGTH);
+    int len = 0;
+    int plaintextLen = 0;
+
+    try {
+        // Initialize decryption
+        if (EVP_DecryptInit_ex(ctx, EVP_aes_256_gcm(), nullptr, nullptr, nullptr) != 1) {
+            throw std::runtime_error("Failed to initialize AES-256-GCM decryption");
+        }
+
+        // Set IV length
+        if (EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, static_cast<int>(IV_SIZE), nullptr) != 1) {
+            throw std::runtime_error("Failed to set IV length");
+        }
+
+        // Set key and IV
+        if (EVP_DecryptInit_ex(ctx, nullptr, nullptr, masterKey_.data(), iv.data()) != 1) {
+            throw std::runtime_error("Failed to set key and IV");
+        }
+
+        // Decrypt data
+        if (EVP_DecryptUpdate(ctx, plaintext.data(), &len, encrypted.data(),
+                             static_cast<int>(encrypted.size())) != 1) {
+            throw std::runtime_error("Failed to decrypt data");
+        }
+        plaintextLen = len;
+
+        // Set expected auth tag
+        if (EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_TAG, static_cast<int>(TAG_SIZE),
+                               const_cast<uint8_t*>(tag.data())) != 1) {
+            throw std::runtime_error("Failed to set auth tag");
+        }
+
+        // Finalize decryption and verify tag
+        if (EVP_DecryptFinal_ex(ctx, plaintext.data() + len, &len) != 1) {
+            throw std::runtime_error("Decryption failed: authentication failed");
+        }
+        plaintextLen += len;
+
+        EVP_CIPHER_CTX_free(ctx);
+
+    } catch (...) {
+        EVP_CIPHER_CTX_free(ctx);
+        throw;
+    }
+
+    plaintext.resize(plaintextLen);
+    return plaintext;
+}
+
+bool EncryptionManager::isSensitiveField(const std::string& fieldPath) const {
+    for (const auto& pattern : sensitivePatterns_) {
+        if (std::regex_match(fieldPath, pattern)) {
+            return true;
+        }
+    }
+    return false;
+}
+
+bool EncryptionManager::loadKeyFromFile() {
+    if (config_.keyFilePath.empty()) {
+        return false;
+    }
+
+    if (!std::filesystem::exists(config_.keyFilePath)) {
+        spdlog::debug("Key file not found: {}", config_.keyFilePath.string());
+        return false;
+    }
+
+    std::ifstream file(config_.keyFilePath, std::ios::binary);
+    if (!file) {
+        spdlog::error("Failed to open key file: {}", config_.keyFilePath.string());
+        return false;
+    }
+
+    masterKey_.resize(KEY_SIZE);
+    file.read(reinterpret_cast<char*>(masterKey_.data()), KEY_SIZE);
+
+    if (file.gcount() != static_cast<std::streamsize>(KEY_SIZE)) {
+        spdlog::error("Invalid key file size: expected {}, got {}", KEY_SIZE, file.gcount());
+        masterKey_.clear();
+        return false;
+    }
+
+    // Log key fingerprint for debugging (first 8 bytes as hex)
+    std::string fingerprint;
+    for (size_t i = 0; i < std::min(size_t(8), masterKey_.size()); ++i) {
+        char buf[3];
+        snprintf(buf, sizeof(buf), "%02x", masterKey_[i]);
+        fingerprint += buf;
+    }
+
+    // Verify the key works by doing a test encrypt/decrypt round-trip
+    const std::string testData = "CallerAI-KeyVerification-Test";
+    try {
+        std::vector<uint8_t> testPlaintext(testData.begin(), testData.end());
+
+        // Temporarily mark as initialized for the test
+        bool wasInitialized = initialized_;
+        initialized_ = true;
+
+        auto encrypted = encryptData(testPlaintext);
+        auto decrypted = decryptData(encrypted);
+
+        initialized_ = wasInitialized;
+
+        if (decrypted != testPlaintext) {
+            spdlog::error("Key verification failed: encrypt/decrypt round-trip mismatch (fingerprint: {}...)", fingerprint);
+            masterKey_.clear();
+            return false;
+        }
+    } catch (const std::exception& e) {
+        spdlog::error("Key verification failed: {} (fingerprint: {}...)", e.what(), fingerprint);
+        masterKey_.clear();
+        return false;
+    }
+
+    spdlog::info("Loaded and verified encryption key from {} (fingerprint: {}...)",
+                 config_.keyFilePath.string(), fingerprint);
+    return true;
+}
+
+bool EncryptionManager::loadKeyFromEnv() {
+    const char* keyStr = std::getenv(config_.keyEnvVar.c_str());
+    if (!keyStr) {
+        spdlog::debug("Environment variable {} not set", config_.keyEnvVar);
+        return false;
+    }
+
+    // Expect hex-encoded key
+    std::string keyHex = keyStr;
+    if (keyHex.size() != KEY_SIZE * 2) {
+        spdlog::error("Invalid key in environment variable (expected {} hex chars)", KEY_SIZE * 2);
+        return false;
+    }
+
+    masterKey_.resize(KEY_SIZE);
+    for (size_t i = 0; i < KEY_SIZE; ++i) {
+        std::string byteStr = keyHex.substr(i * 2, 2);
+        masterKey_[i] = static_cast<uint8_t>(std::stoul(byteStr, nullptr, 16));
+    }
+
+    spdlog::info("Loaded encryption key from environment variable {}", config_.keyEnvVar);
+    return true;
+}
+
+bool EncryptionManager::generateKey() {
+    masterKey_.resize(KEY_SIZE);
+
+    if (RAND_bytes(masterKey_.data(), static_cast<int>(KEY_SIZE)) != 1) {
+        spdlog::error("Failed to generate random key");
+        masterKey_.clear();
+        return false;
+    }
+
+    spdlog::info("Generated new encryption key");
+    return true;
+}
+
+bool EncryptionManager::saveKeyToFile() {
+    if (config_.keyFilePath.empty() || masterKey_.empty()) {
+        return false;
+    }
+
+    // Create parent directories if needed
+    auto parentDir = config_.keyFilePath.parent_path();
+    if (!parentDir.empty() && !std::filesystem::exists(parentDir)) {
+        std::error_code ec;
+        std::filesystem::create_directories(parentDir, ec);
+        if (ec) {
+            spdlog::error("Failed to create key directory: {}", ec.message());
+            return false;
+        }
+    }
+
+    // Write key file with restrictive permissions
+    {
+        std::ofstream file(config_.keyFilePath, std::ios::binary | std::ios::trunc);
+        if (!file) {
+            spdlog::error("Failed to create key file: {}", config_.keyFilePath.string());
+            return false;
+        }
+
+        file.write(reinterpret_cast<const char*>(masterKey_.data()), masterKey_.size());
+
+        // Check for write errors
+        if (!file.good()) {
+            spdlog::error("Failed to write key file: stream error");
+            return false;
+        }
+
+        file.flush();
+        if (!file.good()) {
+            spdlog::error("Failed to flush key file: stream error");
+            return false;
+        }
+
+        file.close();
+    }
+
+    // CRITICAL: Verify the written key by reading it back
+    {
+        std::ifstream verifyFile(config_.keyFilePath, std::ios::binary);
+        if (!verifyFile) {
+            spdlog::error("Failed to verify key file: cannot reopen for reading");
+            return false;
+        }
+
+        std::vector<uint8_t> savedKey(KEY_SIZE);
+        verifyFile.read(reinterpret_cast<char*>(savedKey.data()), KEY_SIZE);
+
+        if (verifyFile.gcount() != static_cast<std::streamsize>(KEY_SIZE)) {
+            spdlog::error("Failed to verify key file: wrong size (expected {}, got {})",
+                         KEY_SIZE, verifyFile.gcount());
+            return false;
+        }
+
+        // Compare the saved key with the in-memory key
+        if (savedKey != masterKey_) {
+            spdlog::error("CRITICAL: Saved key does not match in-memory key! Key file may be corrupted.");
+            return false;
+        }
+    }
+
+    // Set restrictive permissions (owner read/write only)
+    std::error_code ec;
+    std::filesystem::permissions(
+        config_.keyFilePath,
+        std::filesystem::perms::owner_read | std::filesystem::perms::owner_write,
+        std::filesystem::perm_options::replace,
+        ec
+    );
+
+    if (ec) {
+        spdlog::warn("Failed to set key file permissions: {}", ec.message());
+    }
+
+    // Log key fingerprint for debugging (first 8 bytes as hex)
+    std::string fingerprint;
+    for (size_t i = 0; i < std::min(size_t(8), masterKey_.size()); ++i) {
+        char buf[3];
+        snprintf(buf, sizeof(buf), "%02x", masterKey_[i]);
+        fingerprint += buf;
+    }
+    spdlog::info("Saved and verified encryption key to {} (fingerprint: {}...)",
+                 config_.keyFilePath.string(), fingerprint);
+    return true;
+}
+
+void EncryptionManager::encryptJsonField(nlohmann::json& obj, const std::string& path) {
+    if (obj.is_object()) {
+        for (auto& [key, value] : obj.items()) {
+            std::string fieldPath = path.empty() ? key : path + "." + key;
+
+            if (value.is_string() && isSensitiveField(fieldPath)) {
+                std::string plaintext = value.get<std::string>();
+                // Skip if already encrypted (prevent double encryption)
+                if (plaintext.find(ENCRYPTED_PREFIX) != 0) {
+                    value = encryptValue(plaintext);
+                }
+            } else if (value.is_object() || value.is_array()) {
+                // Recurse into nested objects/arrays
+                encryptJsonField(value, fieldPath);
+            }
+        }
+    } else if (obj.is_array()) {
+        for (size_t i = 0; i < obj.size(); ++i) {
+            std::string fieldPath = path + "[" + std::to_string(i) + "]";
+            encryptJsonField(obj[i], fieldPath);
+        }
+    }
+}
+
+void EncryptionManager::decryptJsonField(nlohmann::json& obj, const std::string& path) {
+    if (obj.is_object()) {
+        for (auto& [key, value] : obj.items()) {
+            std::string fieldPath = path.empty() ? key : path + "." + key;
+
+            if (value.is_string()) {
+                std::string str = value.get<std::string>();
+                if (str.find(ENCRYPTED_PREFIX) == 0) {
+                    // Decrypt this field
+                    try {
+                        value = decryptValue(str);
+                    } catch (const std::exception& e) {
+                        spdlog::error("Failed to decrypt field {}: {}", fieldPath, e.what());
+                    }
+                }
+            } else if (value.is_object() || value.is_array()) {
+                // Recurse into nested objects/arrays
+                decryptJsonField(value, fieldPath);
+            }
+        }
+    } else if (obj.is_array()) {
+        for (size_t i = 0; i < obj.size(); ++i) {
+            std::string fieldPath = path + "[" + std::to_string(i) + "]";
+            decryptJsonField(obj[i], fieldPath);
+        }
+    }
+}
+
+} // namespace smartbotic::database

service/src/encryption/encryption_manager.hpp

@@ -0,0 +1,101 @@
+#pragma once
+
+#include "../document.hpp"
+
+#include <filesystem>
+#include <mutex>
+#include <regex>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+namespace smartbotic::database {
+
+/**
+ * Encryption manager for field-level encryption of sensitive data.
+ * Uses AES-256-GCM for authenticated encryption.
+ */
+class EncryptionManager {
+public:
+    struct Config {
+        bool enabled = true;
+        std::string keyProvider = "file";     // "file", "env"
+        std::filesystem::path keyFilePath;
+        std::string keyEnvVar = "STORAGE_MASTER_KEY";
+        bool autoGenerateKey = true;
+        std::vector<std::string> sensitiveFieldPatterns = {
+            "*.api_key",
+            "*.apiKey",
+            "*.password",
+            "*.secret",
+            "*.token",
+            "*.credentials"
+        };
+    };
+
+    explicit EncryptionManager(Config config);
+    ~EncryptionManager();
+
+    /**
+     * Initialize the encryption manager.
+     * Loads or generates the master key.
+     */
+    bool initialize();
+
+    /**
+     * Check if encryption is enabled.
+     */
+    [[nodiscard]] bool isEnabled() const { return config_.enabled && initialized_; }
+
+    /**
+     * Encrypt sensitive fields in a document.
+     */
+    void encryptSensitiveFields(Document& doc);
+
+    /**
+     * Decrypt sensitive fields in a document.
+     */
+    void decryptSensitiveFields(Document& doc);
+
+    /**
+     * Encrypt a single string value.
+     */
+    [[nodiscard]] std::string encryptValue(const std::string& plaintext);
+
+    /**
+     * Decrypt a single string value.
+     */
+    [[nodiscard]] std::string decryptValue(const std::string& ciphertext);
+
+    /**
+     * Encrypt binary data.
+     */
+    [[nodiscard]] std::vector<uint8_t> encryptData(const std::vector<uint8_t>& plaintext);
+
+    /**
+     * Decrypt binary data.
+     */
+    [[nodiscard]] std::vector<uint8_t> decryptData(const std::vector<uint8_t>& ciphertext);
+
+    /**
+     * Check if a field path matches sensitive patterns.
+     */
+    [[nodiscard]] bool isSensitiveField(const std::string& fieldPath) const;
+
+private:
+    bool loadKeyFromFile();
+    bool loadKeyFromEnv();
+    bool generateKey();
+    bool saveKeyToFile();
+
+    void encryptJsonField(nlohmann::json& obj, const std::string& path);
+    void decryptJsonField(nlohmann::json& obj, const std::string& path);
+
+    Config config_;
+    bool initialized_ = false;
+    std::vector<uint8_t> masterKey_;
+    std::vector<std::regex> sensitivePatterns_;
+    mutable std::mutex mutex_;
+};
+
+} // namespace smartbotic::database

service/src/events/event_manager.cpp

@@ -0,0 +1,189 @@
+#include "event_manager.hpp"
+
+#include <spdlog/spdlog.h>
+
+#include <algorithm>
+#include <chrono>
+#include <regex>
+
+namespace smartbotic::database {
+
+EventManager::EventManager(Config config)
+    : config_(std::move(config))
+{
+}
+
+EventManager::~EventManager() {
+    stop();
+}
+
+void EventManager::start() {
+    if (running_.exchange(true)) {
+        return;  // Already running
+    }
+
+    spdlog::info("EventManager started (node: {})", config_.nodeId);
+}
+
+void EventManager::stop() {
+    if (!running_.exchange(false)) {
+        return;  // Not running
+    }
+
+    // Clear all subscriptions
+    {
+        std::unique_lock lock(mutex_);
+        subscriptions_.clear();
+    }
+
+    spdlog::info("EventManager stopped");
+}
+
+uint64_t EventManager::subscribe(
+    const std::vector<std::string>& collections,
+    const std::vector<std::string>& patterns,
+    EventCallback callback
+) {
+    uint64_t id = nextSubscriptionId_.fetch_add(1);
+
+    std::unique_lock lock(mutex_);
+    subscriptions_.push_back(Subscription{
+        .id = id,
+        .collections = collections,
+        .patterns = patterns,
+        .callback = std::move(callback)
+    });
+
+    spdlog::debug("EventManager: new subscription {} for {} collections, {} patterns",
+                  id, collections.size(), patterns.size());
+
+    return id;
+}
+
+void EventManager::unsubscribe(uint64_t subscriptionId) {
+    std::unique_lock lock(mutex_);
+
+    auto it = std::find_if(subscriptions_.begin(), subscriptions_.end(),
+        [subscriptionId](const Subscription& sub) {
+            return sub.id == subscriptionId;
+        });
+
+    if (it != subscriptions_.end()) {
+        subscriptions_.erase(it);
+        spdlog::debug("EventManager: removed subscription {}", subscriptionId);
+    }
+}
+
+void EventManager::publish(const DatabaseEvent& event) {
+    if (!running_) {
+        return;
+    }
+
+    // Get matching subscriptions
+    std::vector<EventCallback> matchingCallbacks;
+
+    {
+        std::shared_lock lock(mutex_);
+        for (const auto& sub : subscriptions_) {
+            if (matchesSubscription(sub, event)) {
+                matchingCallbacks.push_back(sub.callback);
+            }
+        }
+    }
+
+    // Call callbacks outside the lock
+    for (const auto& callback : matchingCallbacks) {
+        try {
+            callback(event);
+        } catch (const std::exception& e) {
+            spdlog::error("EventManager: callback error: {}", e.what());
+        }
+    }
+
+    spdlog::trace("EventManager: published {} event for {}/{} to {} subscribers",
+                  static_cast<int>(event.type), event.collection, event.documentId,
+                  matchingCallbacks.size());
+}
+
+uint64_t EventManager::subscriptionCount() const {
+    std::shared_lock lock(mutex_);
+    return subscriptions_.size();
+}
+
+bool EventManager::matchesSubscription(const Subscription& sub, const DatabaseEvent& event) const {
+    // If no collections specified, match all
+    if (sub.collections.empty() && sub.patterns.empty()) {
+        return true;
+    }
+
+    // Check exact collection match
+    for (const auto& coll : sub.collections) {
+        if (coll == event.collection) {
+            return true;
+        }
+    }
+
+    // Check pattern match
+    for (const auto& pattern : sub.patterns) {
+        // Check against "collection:documentId" format
+        std::string fullKey = event.collection + ":" + event.documentId;
+        if (matchesPattern(pattern, fullKey)) {
+            return true;
+        }
+
+        // Also check against just the collection
+        if (matchesPattern(pattern, event.collection)) {
+            return true;
+        }
+    }
+
+    return false;
+}
+
+bool EventManager::matchesPattern(const std::string& pattern, const std::string& value) const {
+    // Convert glob pattern to regex
+    // * matches any sequence of characters
+    // ? matches any single character
+
+    std::string regexPattern;
+    regexPattern.reserve(pattern.size() * 2);
+
+    for (char c : pattern) {
+        switch (c) {
+            case '*':
+                regexPattern += ".*";
+                break;
+            case '?':
+                regexPattern += ".";
+                break;
+            case '.':
+            case '+':
+            case '^':
+            case '$':
+            case '(':
+            case ')':
+            case '[':
+            case ']':
+            case '{':
+            case '}':
+            case '|':
+            case '\\':
+                regexPattern += '\\';
+                regexPattern += c;
+                break;
+            default:
+                regexPattern += c;
+                break;
+        }
+    }
+
+    try {
+        std::regex re(regexPattern, std::regex::icase);
+        return std::regex_match(value, re);
+    } catch (const std::regex_error&) {
+        // If regex fails, fall back to exact match
+        return pattern == value;
+    }
+}
+
+} // namespace smartbotic::database

service/src/events/event_manager.hpp

@@ -0,0 +1,84 @@
+#pragma once
+
+#include "../document.hpp"
+
+#include <atomic>
+#include <functional>
+#include <mutex>
+#include <shared_mutex>
+#include <string>
+#include <thread>
+#include <unordered_map>
+#include <vector>
+
+namespace smartbotic::database {
+
+/**
+ * Event manager for pub/sub functionality.
+ * Replaces Redis PUBLISH/SUBSCRIBE with gRPC streaming.
+ */
+class EventManager {
+public:
+    using EventCallback = std::function<void(const DatabaseEvent&)>;
+
+    struct Config {
+        std::string nodeId;
+
+        Config() : nodeId("storage-1") {}
+    };
+
+    explicit EventManager(Config config = Config{});
+    ~EventManager();
+
+    // Lifecycle
+    void start();
+    void stop();
+
+    /**
+     * Subscribe to storage events.
+     * @param collections List of collections to subscribe to (empty = all)
+     * @param patterns Glob patterns (e.g., "assistants:*")
+     * @param callback Function called for each matching event
+     * @return Subscription ID
+     */
+    uint64_t subscribe(
+        const std::vector<std::string>& collections,
+        const std::vector<std::string>& patterns,
+        EventCallback callback
+    );
+
+    /**
+     * Unsubscribe from events.
+     */
+    void unsubscribe(uint64_t subscriptionId);
+
+    /**
+     * Publish an event.
+     */
+    void publish(const DatabaseEvent& event);
+
+    /**
+     * Get subscription count.
+     */
+    [[nodiscard]] uint64_t subscriptionCount() const;
+
+private:
+    struct Subscription {
+        uint64_t id;
+        std::vector<std::string> collections;
+        std::vector<std::string> patterns;
+        EventCallback callback;
+    };
+
+    bool matchesSubscription(const Subscription& sub, const DatabaseEvent& event) const;
+    bool matchesPattern(const std::string& pattern, const std::string& value) const;
+
+    Config config_;
+    std::atomic<bool> running_{false};
+    std::atomic<uint64_t> nextSubscriptionId_{1};
+
+    std::vector<Subscription> subscriptions_;
+    mutable std::shared_mutex mutex_;
+};
+
+} // namespace smartbotic::database

service/src/files/file_manager.cpp

@@ -0,0 +1,337 @@
+#include "file_manager.hpp"
+
+#include <spdlog/spdlog.h>
+#include <nlohmann/json.hpp>
+
+#include <openssl/sha.h>
+
+#include <algorithm>
+#include <chrono>
+#include <fstream>
+#include <iomanip>
+#include <random>
+#include <sstream>
+
+namespace smartbotic::database {
+
+FileManager::FileManager(Config config)
+    : config_(std::move(config))
+{
+}
+
+FileManager::~FileManager() {
+    stop();
+}
+
+void FileManager::start() {
+    // Create files directory if it doesn't exist
+    std::error_code ec;
+    std::filesystem::create_directories(config_.filesDir, ec);
+    if (ec) {
+        spdlog::error("Failed to create files directory: {}", ec.message());
+        throw std::runtime_error("Failed to create files directory");
+    }
+
+    // Create subdirectories for different file types
+    std::filesystem::create_directories(config_.filesDir / "audio", ec);
+    std::filesystem::create_directories(config_.filesDir / "pcap", ec);
+    std::filesystem::create_directories(config_.filesDir / "uploads", ec);
+
+    spdlog::info("FileManager started (directory: {})", config_.filesDir.string());
+}
+
+void FileManager::stop() {
+    spdlog::info("FileManager stopped");
+}
+
+std::string FileManager::storeFile(const std::vector<uint8_t>& data, FileInfo info) {
+    // Validate file size
+    uint64_t maxBytes = config_.maxFileSizeMb * 1024 * 1024;
+    if (data.size() > maxBytes) {
+        throw std::runtime_error("File too large (max: " + std::to_string(config_.maxFileSizeMb) + " MB)");
+    }
+
+    // Validate MIME type if restrictions are set
+    if (!config_.allowedMimeTypes.empty()) {
+        bool allowed = false;
+        for (const auto& pattern : config_.allowedMimeTypes) {
+            if (matchMimeType(pattern, info.mimeType)) {
+                allowed = true;
+                break;
+            }
+        }
+        if (!allowed) {
+            throw std::runtime_error("MIME type not allowed: " + info.mimeType);
+        }
+    }
+
+    // Generate ID if not provided
+    if (info.id.empty()) {
+        info.id = generateId();
+    }
+
+    // Calculate checksum
+    info.checksum = calculateChecksum(data);
+
+    // Set size and timestamp
+    info.size = data.size();
+    info.createdAt = std::chrono::duration_cast<std::chrono::milliseconds>(
+        std::chrono::system_clock::now().time_since_epoch()
+    ).count();
+
+    // Determine file path
+    auto filePath = getFilePath(info.id);
+
+    // Create parent directories if needed
+    std::error_code ec;
+    std::filesystem::create_directories(filePath.parent_path(), ec);
+    if (ec) {
+        throw std::runtime_error("Failed to create directory: " + ec.message());
+    }
+
+    // Write data file
+    {
+        std::ofstream file(filePath, std::ios::binary);
+        if (!file) {
+            throw std::runtime_error("Failed to create file: " + filePath.string());
+        }
+        file.write(reinterpret_cast<const char*>(data.data()), data.size());
+        if (!file) {
+            throw std::runtime_error("Failed to write file: " + filePath.string());
+        }
+    }
+
+    // Write metadata file
+    {
+        nlohmann::json meta;
+        meta["id"] = info.id;
+        meta["name"] = info.name;
+        meta["mimeType"] = info.mimeType;
+        meta["size"] = info.size;
+        meta["fileType"] = info.fileType;
+        meta["relatedId"] = info.relatedId;
+        meta["checksum"] = info.checksum;
+        meta["createdAt"] = info.createdAt;
+        meta["metadata"] = info.metadata;
+
+        auto metaPath = filePath.string() + ".meta.json";
+        std::ofstream metaFile(metaPath);
+        if (!metaFile) {
+            // Clean up data file
+            std::filesystem::remove(filePath);
+            throw std::runtime_error("Failed to create metadata file");
+        }
+        metaFile << meta.dump(2);
+    }
+
+    spdlog::debug("Stored file {} ({} bytes)", info.id, data.size());
+
+    return info.id;
+}
+
+std::vector<uint8_t> FileManager::readFile(const std::string& id) const {
+    auto filePath = getFilePath(id);
+
+    if (!std::filesystem::exists(filePath)) {
+        throw std::runtime_error("File not found: " + id);
+    }
+
+    std::ifstream file(filePath, std::ios::binary | std::ios::ate);
+    if (!file) {
+        throw std::runtime_error("Failed to open file: " + id);
+    }
+
+    auto size = file.tellg();
+    file.seekg(0);
+
+    std::vector<uint8_t> data(size);
+    file.read(reinterpret_cast<char*>(data.data()), size);
+
+    return data;
+}
+
+bool FileManager::deleteFile(const std::string& id) {
+    auto filePath = getFilePath(id);
+    auto metaPath = std::filesystem::path(filePath.string() + ".meta.json");
+
+    std::error_code ec;
+    bool deletedData = std::filesystem::remove(filePath, ec);
+    bool deletedMeta = std::filesystem::remove(metaPath, ec);
+
+    if (deletedData || deletedMeta) {
+        spdlog::debug("Deleted file {}", id);
+    }
+
+    return deletedData || deletedMeta;
+}
+
+std::optional<FileManager::FileInfo> FileManager::getFileInfo(const std::string& id) const {
+    auto filePath = getFilePath(id);
+    auto metaPath = std::filesystem::path(filePath.string() + ".meta.json");
+
+    if (!std::filesystem::exists(metaPath)) {
+        return std::nullopt;
+    }
+
+    std::ifstream metaFile(metaPath);
+    if (!metaFile) {
+        return std::nullopt;
+    }
+
+    try {
+        nlohmann::json meta;
+        metaFile >> meta;
+
+        FileInfo info;
+        info.id = meta.value("id", "");
+        info.name = meta.value("name", "");
+        info.mimeType = meta.value("mimeType", "");
+        info.size = meta.value("size", 0ULL);
+        info.fileType = meta.value("fileType", "");
+        info.relatedId = meta.value("relatedId", "");
+        info.checksum = meta.value("checksum", "");
+        info.createdAt = meta.value("createdAt", 0ULL);
+
+        if (meta.contains("metadata") && meta["metadata"].is_object()) {
+            for (const auto& [key, value] : meta["metadata"].items()) {
+                if (value.is_string()) {
+                    info.metadata[key] = value.get<std::string>();
+                }
+            }
+        }
+
+        return info;
+
+    } catch (const nlohmann::json::exception&) {
+        return std::nullopt;
+    }
+}
+
+std::string FileManager::getChecksum(const std::string& id) const {
+    auto info = getFileInfo(id);
+    return info ? info->checksum : "";
+}
+
+FileManager::ListResult FileManager::listFiles(
+    const std::string& fileType,
+    const std::string& relatedId,
+    uint32_t limit,
+    uint32_t offset
+) const {
+    ListResult result;
+    std::vector<FileInfo> allFiles;
+
+    // Scan all subdirectories
+    std::vector<std::filesystem::path> subDirs = {
+        config_.filesDir / "audio",
+        config_.filesDir / "pcap",
+        config_.filesDir / "uploads"
+    };
+
+    for (const auto& subDir : subDirs) {
+        if (!std::filesystem::exists(subDir)) {
+            continue;
+        }
+
+        for (const auto& entry : std::filesystem::recursive_directory_iterator(subDir)) {
+            if (entry.is_regular_file() && entry.path().string().ends_with(".meta.json")) {
+                // Extract ID from path
+                std::string metaPath = entry.path().string();
+                std::string filePath = metaPath.substr(0, metaPath.length() - 10);  // Remove ".meta.json"
+                std::string id = std::filesystem::path(filePath).filename().string();
+
+                auto info = getFileInfo(id);
+                if (!info) {
+                    continue;
+                }
+
+                // Apply filters
+                if (!fileType.empty() && info->fileType != fileType) {
+                    continue;
+                }
+                if (!relatedId.empty() && info->relatedId != relatedId) {
+                    continue;
+                }
+
+                allFiles.push_back(*info);
+            }
+        }
+    }
+
+    // Sort by creation time (newest first)
+    std::sort(allFiles.begin(), allFiles.end(),
+              [](const FileInfo& a, const FileInfo& b) {
+                  return a.createdAt > b.createdAt;
+              });
+
+    // Apply pagination
+    result.totalCount = allFiles.size();
+
+    if (offset >= allFiles.size()) {
+        return result;
+    }
+
+    auto start = allFiles.begin() + offset;
+    auto end = std::min(start + limit, allFiles.end());
+
+    result.files.assign(start, end);
+    result.hasMore = (offset + limit < allFiles.size());
+
+    return result;
+}
+
+std::string FileManager::generateId() const {
+    static const char* charset = "0123456789abcdef";
+
+    std::random_device rd;
+    std::mt19937 gen(rd());
+    std::uniform_int_distribution<> dis(0, 15);
+
+    std::string id = "file_";
+    for (int i = 0; i < 16; ++i) {
+        id += charset[dis(gen)];
+    }
+
+    return id;
+}
+
+std::filesystem::path FileManager::getFilePath(const std::string& id) const {
+    // Use first 2 characters of ID for sharding into subdirectories
+    // This helps with filesystem performance for large numbers of files
+    std::string subDir;
+    if (id.starts_with("file_") && id.size() >= 7) {
+        subDir = id.substr(5, 2);
+    } else if (id.size() >= 2) {
+        subDir = id.substr(0, 2);
+    } else {
+        subDir = "00";
+    }
+
+    // Default to uploads directory
+    return config_.filesDir / "uploads" / subDir / id;
+}
+
+std::string FileManager::calculateChecksum(const std::vector<uint8_t>& data) {
+    unsigned char hash[SHA256_DIGEST_LENGTH];
+    SHA256(data.data(), data.size(), hash);
+
+    std::ostringstream oss;
+    oss << "sha256:";
+    for (int i = 0; i < SHA256_DIGEST_LENGTH; ++i) {
+        oss << std::hex << std::setw(2) << std::setfill('0') << static_cast<int>(hash[i]);
+    }
+
+    return oss.str();
+}
+
+bool FileManager::matchMimeType(const std::string& pattern, const std::string& mimeType) const {
+    // Handle wildcard patterns like "audio/*"
+    if (pattern.ends_with("/*")) {
+        std::string prefix = pattern.substr(0, pattern.size() - 1);
+        return mimeType.starts_with(prefix);
+    }
+
+    return pattern == mimeType;
+}
+
+} // namespace smartbotic::database

service/src/files/file_manager.hpp

@@ -0,0 +1,96 @@
+#pragma once
+
+#include <cstdint>
+#include <filesystem>
+#include <optional>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+namespace smartbotic::database {
+
+/**
+ * File manager for storing binary files (audio recordings, PCAP files, etc.).
+ */
+class FileManager {
+public:
+    struct Config {
+        std::filesystem::path filesDir;
+        uint64_t maxFileSizeMb = 500;
+        std::vector<std::string> allowedMimeTypes;
+        uint32_t cleanupIntervalSec = 3600;
+    };
+
+    struct FileInfo {
+        std::string id;
+        std::string name;
+        std::string mimeType;
+        uint64_t size = 0;
+        std::string fileType;         // audio/recording, pcap, upload
+        std::string relatedId;        // e.g., call_id
+        std::string checksum;         // SHA-256
+        uint64_t createdAt = 0;
+        std::unordered_map<std::string, std::string> metadata;
+    };
+
+    struct ListResult {
+        std::vector<FileInfo> files;
+        uint64_t totalCount = 0;
+        bool hasMore = false;
+    };
+
+    explicit FileManager(Config config);
+    ~FileManager();
+
+    // Lifecycle
+    void start();
+    void stop();
+
+    /**
+     * Store a file.
+     * @param data File content
+     * @param info File metadata
+     * @return File ID
+     */
+    std::string storeFile(const std::vector<uint8_t>& data, FileInfo info);
+
+    /**
+     * Read a file.
+     */
+    [[nodiscard]] std::vector<uint8_t> readFile(const std::string& id) const;
+
+    /**
+     * Delete a file.
+     */
+    bool deleteFile(const std::string& id);
+
+    /**
+     * Get file info.
+     */
+    [[nodiscard]] std::optional<FileInfo> getFileInfo(const std::string& id) const;
+
+    /**
+     * Get file checksum.
+     */
+    [[nodiscard]] std::string getChecksum(const std::string& id) const;
+
+    /**
+     * List files with optional filters.
+     */
+    [[nodiscard]] ListResult listFiles(
+        const std::string& fileType = "",
+        const std::string& relatedId = "",
+        uint32_t limit = 100,
+        uint32_t offset = 0
+    ) const;
+
+private:
+    std::string generateId() const;
+    std::filesystem::path getFilePath(const std::string& id) const;
+    static std::string calculateChecksum(const std::vector<uint8_t>& data);
+    bool matchMimeType(const std::string& pattern, const std::string& mimeType) const;
+
+    Config config_;
+};
+
+} // namespace smartbotic::database

service/src/files/file_store.cpp

@@ -0,0 +1,117 @@
+#include "file_store.hpp"
+
+#include <spdlog/spdlog.h>
+
+#include <fstream>
+#include <stdexcept>
+
+namespace smartbotic::database {
+
+FileStore::FileStore(Config config)
+    : config_(std::move(config))
+{
+    // Create base directory if needed
+    std::error_code ec;
+    std::filesystem::create_directories(config_.baseDir, ec);
+    if (ec) {
+        spdlog::error("Failed to create file store directory: {}", ec.message());
+    }
+}
+
+FileStore::~FileStore() = default;
+
+void FileStore::write(const std::filesystem::path& relativePath, const std::vector<uint8_t>& data) {
+    auto absPath = getAbsolutePath(relativePath);
+
+    // Create parent directories
+    std::error_code ec;
+    std::filesystem::create_directories(absPath.parent_path(), ec);
+    if (ec) {
+        throw std::runtime_error("Failed to create directory: " + ec.message());
+    }
+
+    std::ofstream file(absPath, std::ios::binary);
+    if (!file) {
+        throw std::runtime_error("Failed to create file: " + absPath.string());
+    }
+
+    file.write(reinterpret_cast<const char*>(data.data()), data.size());
+
+    if (!file) {
+        throw std::runtime_error("Failed to write file: " + absPath.string());
+    }
+}
+
+std::vector<uint8_t> FileStore::read(const std::filesystem::path& relativePath) const {
+    auto absPath = getAbsolutePath(relativePath);
+
+    std::ifstream file(absPath, std::ios::binary | std::ios::ate);
+    if (!file) {
+        throw std::runtime_error("Failed to open file: " + absPath.string());
+    }
+
+    auto size = file.tellg();
+    file.seekg(0);
+
+    std::vector<uint8_t> data(size);
+    file.read(reinterpret_cast<char*>(data.data()), size);
+
+    return data;
+}
+
+std::vector<uint8_t> FileStore::readChunk(
+    const std::filesystem::path& relativePath,
+    size_t offset,
+    size_t length
+) const {
+    auto absPath = getAbsolutePath(relativePath);
+
+    std::ifstream file(absPath, std::ios::binary);
+    if (!file) {
+        throw std::runtime_error("Failed to open file: " + absPath.string());
+    }
+
+    // Get file size
+    file.seekg(0, std::ios::end);
+    auto fileSize = static_cast<size_t>(file.tellg());
+
+    if (offset >= fileSize) {
+        return {};  // Offset beyond file end
+    }
+
+    // Adjust length if needed
+    size_t actualLength = std::min(length, fileSize - offset);
+
+    file.seekg(offset);
+    std::vector<uint8_t> data(actualLength);
+    file.read(reinterpret_cast<char*>(data.data()), actualLength);
+
+    return data;
+}
+
+bool FileStore::exists(const std::filesystem::path& relativePath) const {
+    return std::filesystem::exists(getAbsolutePath(relativePath));
+}
+
+size_t FileStore::getSize(const std::filesystem::path& relativePath) const {
+    auto absPath = getAbsolutePath(relativePath);
+
+    if (!std::filesystem::exists(absPath)) {
+        return 0;
+    }
+
+    return static_cast<size_t>(std::filesystem::file_size(absPath));
+}
+
+bool FileStore::remove(const std::filesystem::path& relativePath) {
+    auto absPath = getAbsolutePath(relativePath);
+
+    std::error_code ec;
+    return std::filesystem::remove(absPath, ec);
+}
+
+std::filesystem::path FileStore::getAbsolutePath(const std::filesystem::path& relativePath) const {
+    return config_.baseDir / relativePath;
+}
+
+} // namespace smartbotic::database

service/src/files/file_store.hpp

@@ -0,0 +1,75 @@
+#pragma once
+
+#include <cstdint>
+#include <filesystem>
+#include <string>
+#include <vector>
+
+namespace smartbotic::database {
+
+/**
+ * Low-level file store for binary file operations.
+ * Provides streaming read/write for large files.
+ */
+class FileStore {
+public:
+    struct Config {
+        std::filesystem::path baseDir;
+        size_t chunkSize = 64 * 1024;  // 64KB chunks for streaming
+    };
+
+    explicit FileStore(Config config);
+    ~FileStore();
+
+    /**
+     * Write data to a file.
+     * @param relativePath Path relative to base directory
+     * @param data Data to write
+     */
+    void write(const std::filesystem::path& relativePath, const std::vector<uint8_t>& data);
+
+    /**
+     * Read entire file.
+     * @param relativePath Path relative to base directory
+     * @return File contents
+     */
+    [[nodiscard]] std::vector<uint8_t> read(const std::filesystem::path& relativePath) const;
+
+    /**
+     * Read a chunk of a file.
+     * @param relativePath Path relative to base directory
+     * @param offset Starting offset
+     * @param length Number of bytes to read
+     * @return File chunk
+     */
+    [[nodiscard]] std::vector<uint8_t> readChunk(
+        const std::filesystem::path& relativePath,
+        size_t offset,
+        size_t length
+    ) const;
+
+    /**
+     * Check if a file exists.
+     */
+    [[nodiscard]] bool exists(const std::filesystem::path& relativePath) const;
+
+    /**
+     * Get file size.
+     */
+    [[nodiscard]] size_t getSize(const std::filesystem::path& relativePath) const;
+
+    /**
+     * Delete a file.
+     */
+    bool remove(const std::filesystem::path& relativePath);
+
+    /**
+     * Get absolute path.
+     */
+    [[nodiscard]] std::filesystem::path getAbsolutePath(const std::filesystem::path& relativePath) const;
+
+private:
+    Config config_;
+};
+
+} // namespace smartbotic::database

service/src/main.cpp

@@ -0,0 +1,203 @@
+#include "database_service.hpp"
+
+#include <spdlog/spdlog.h>
+#include <spdlog/sinks/stdout_color_sinks.h>
+
+#include <csignal>
+#include <cstdlib>
+#include <filesystem>
+#include <iostream>
+
+#ifdef HAVE_SYSTEMD
+#include <systemd/sd-daemon.h>
+#endif
+
+namespace {
+
+smartbotic::database::DatabaseService* g_service = nullptr;
+
+void signalHandler(int signal) {
+    spdlog::info("Received signal {}", signal);
+    if (g_service) {
+        g_service->signalStop();
+    }
+}
+
+void printUsage(const char* programName) {
+    std::cout << "Usage: " << programName << " [OPTIONS]\n"
+              << "\n"
+              << "Options:\n"
+              << "  --config, -c PATH    Configuration file path\n"
+              << "  --help, -h           Show this help message\n"
+              << "  --version, -v        Show version information\n"
+              << "\n"
+              << "Environment variables:\n"
+              << "  DATABASE_NODE_ID     Override node ID from config\n"
+              << "  DATABASE_MASTER_KEY  Encryption master key (hex-encoded)\n"
+              << std::endl;
+}
+
+void printVersion() {
+    std::cout << "smartbotic-database version 1.0.0\n"
+              << "Smartbotic Database Service\n"
+              << std::endl;
+}
+
+void initLogging(const std::string& serviceName) {
+    try {
+        // Create console sink
+        auto console_sink = std::make_shared<spdlog::sinks::stdout_color_sink_mt>();
+        console_sink->set_level(spdlog::level::debug);
+
+        // Create default logger with console sink
+        auto logger = std::make_shared<spdlog::logger>(serviceName, console_sink);
+        logger->set_level(spdlog::level::info);
+
+        // Check environment for log level
+        if (const char* logLevel = std::getenv("LOG_LEVEL")) {
+            std::string level(logLevel);
+            if (level == "trace") logger->set_level(spdlog::level::trace);
+            else if (level == "debug") logger->set_level(spdlog::level::debug);
+            else if (level == "info") logger->set_level(spdlog::level::info);
+            else if (level == "warn") logger->set_level(spdlog::level::warn);
+            else if (level == "error") logger->set_level(spdlog::level::err);
+        }
+
+        spdlog::set_default_logger(logger);
+        spdlog::set_pattern("[%Y-%m-%d %H:%M:%S.%e] [%^%l%$] [%n] %v");
+
+    } catch (const spdlog::spdlog_ex& ex) {
+        std::cerr << "Log initialization failed: " << ex.what() << std::endl;
+    }
+}
+
+std::filesystem::path findConfigFile(int argc, char* argv[]) {
+    // Check command line arguments
+    for (int i = 1; i < argc; ++i) {
+        std::string arg = argv[i];
+        if ((arg == "--config" || arg == "-c") && i + 1 < argc) {
+            return argv[i + 1];
+        }
+    }
+
+    // Check environment variable
+    if (const char* configPath = std::getenv("DATABASE_CONFIG")) {
+        return configPath;
+    }
+
+    // Default paths
+    std::vector<std::filesystem::path> defaultPaths = {
+        "./config/database.json",
+        "./config/storage.json",  // Backward compatibility
+        "/etc/smartbotic/database.json",
+        "/etc/callerai/storage.json",  // Backward compatibility
+        std::filesystem::path(std::getenv("HOME") ? std::getenv("HOME") : "") / ".config/smartbotic/database.json"
+    };
+
+    for (const auto& path : defaultPaths) {
+        if (std::filesystem::exists(path)) {
+            return path;
+        }
+    }
+
+    return "./config/database.json";
+}
+
+} // anonymous namespace
+
+int main(int argc, char* argv[]) {
+    // Parse command line arguments
+    for (int i = 1; i < argc; ++i) {
+        std::string arg = argv[i];
+        if (arg == "--help" || arg == "-h") {
+            printUsage(argv[0]);
+            return 0;
+        }
+        if (arg == "--version" || arg == "-v") {
+            printVersion();
+            return 0;
+        }
+    }
+
+    try {
+        // Initialize logging
+        initLogging("database");
+
+        // Find configuration file
+        auto configPath = findConfigFile(argc, argv);
+        spdlog::info("Loading configuration from {}", configPath.string());
+
+        // Load configuration
+        smartbotic::database::DatabaseService::Config config;
+        if (std::filesystem::exists(configPath)) {
+            config = smartbotic::database::DatabaseService::loadConfig(configPath);
+        } else {
+            spdlog::warn("Configuration file not found, using defaults");
+            // Set default data directory
+            const char* home = std::getenv("HOME");
+            if (home) {
+                config.dataDirectory = std::filesystem::path(home) / ".local/share/smartbotic/database";
+                config.keyFilePath = std::filesystem::path(home) / ".config/smartbotic/database.key";
+            } else {
+                config.dataDirectory = "/var/lib/smartbotic/database";
+                config.keyFilePath = "/etc/smartbotic/database.key";
+            }
+        }
+
+        // Override node ID from environment if set
+        if (const char* nodeId = std::getenv("DATABASE_NODE_ID")) {
+            config.nodeId = nodeId;
+        }
+
+        // Create service
+        smartbotic::database::DatabaseService service(config);
+        g_service = &service;
+
+        // Set up signal handlers
+        std::signal(SIGINT, signalHandler);
+        std::signal(SIGTERM, signalHandler);
+
+        // Initialize service
+        if (!service.initialize()) {
+            spdlog::error("Failed to initialize database service");
+            return 1;
+        }
+
+        // Start service
+        service.start();
+
+#ifdef HAVE_SYSTEMD
+        // Notify systemd we're ready
+        sd_notify(0, "READY=1");
+        spdlog::info("Notified systemd: READY");
+#endif
+
+        // Wait for shutdown
+        int watchdogCounter = 0;
+        while (service.isRunning()) {
+            std::this_thread::sleep_for(std::chrono::seconds(1));
+#ifdef HAVE_SYSTEMD
+            // Send watchdog notification every 30 seconds (half of WatchdogSec=60)
+            if (++watchdogCounter >= 30) {
+                sd_notify(0, "WATCHDOG=1");
+                watchdogCounter = 0;
+            }
+#endif
+        }
+
+#ifdef HAVE_SYSTEMD
+        sd_notify(0, "STOPPING=1");
+#endif
+
+        // Stop service
+        service.stop();
+        g_service = nullptr;
+
+        spdlog::info("Database service exited cleanly");
+        return 0;
+
+    } catch (const std::exception& e) {
+        spdlog::error("Fatal error: {}", e.what());
+        return 1;
+    }
+}

service/src/memory_store.cpp

@@ -0,0 +1,1225 @@
+#include "memory_store.hpp"
+
+#include <spdlog/spdlog.h>
+
+#include <algorithm>
+#include <random>
+#include <regex>
+#include <sstream>
+#include <iomanip>
+
+namespace smartbotic::database {
+
+MemoryStore::MemoryStore(Config config)
+    : config_(std::move(config)) {
+}
+
+MemoryStore::~MemoryStore() {
+    stop();
+}
+
+void MemoryStore::start() {
+    if (running_.exchange(true)) {
+        return;  // Already running
+    }
+
+    // Start expiration thread
+    expirationThread_ = std::thread(&MemoryStore::expirationLoop, this);
+}
+
+void MemoryStore::stop() {
+    if (!running_.exchange(false)) {
+        return;  // Already stopped
+    }
+
+    if (expirationThread_.joinable()) {
+        expirationThread_.join();
+    }
+}
+
+void MemoryStore::setEventCallback(EventCallback callback) {
+    std::lock_guard<std::mutex> lock(callbackMutex_);
+    eventCallback_ = std::move(callback);
+}
+
+void MemoryStore::setPersistCallback(PersistCallback callback) {
+    std::lock_guard<std::mutex> lock(callbackMutex_);
+    persistCallback_ = std::move(callback);
+}
+
+// ===== Collection Management =====
+
+bool MemoryStore::createCollection(const std::string& name, const CollectionOptions& options) {
+    std::unique_lock<std::shared_mutex> lock(globalMutex_);
+
+    if (collections_.find(name) != collections_.end()) {
+        return false;  // Already exists
+    }
+
+    auto coll = std::make_unique<CollectionData>();
+    coll->options = options;
+    coll->createdAt = currentTimeMs();
+    coll->updatedAt = coll->createdAt;
+
+    collections_[name] = std::move(coll);
+
+    {
+        std::lock_guard<std::mutex> statsLock(statsMutex_);
+        stats_.totalCollections++;
+    }
+
+    return true;
+}
+
+bool MemoryStore::dropCollection(const std::string& name) {
+    std::unique_lock<std::shared_mutex> lock(globalMutex_);
+
+    auto it = collections_.find(name);
+    if (it == collections_.end()) {
+        return false;
+    }
+
+    uint64_t docCount = 0;
+    {
+        std::shared_lock<std::shared_mutex> collLock(it->second->mutex);
+        docCount = it->second->documents.size();
+    }
+
+    collections_.erase(it);
+
+    {
+        std::lock_guard<std::mutex> statsLock(statsMutex_);
+        stats_.totalCollections--;
+        stats_.totalDocuments -= docCount;
+    }
+
+    return true;
+}
+
+std::vector<std::string> MemoryStore::listCollections() const {
+    std::shared_lock<std::shared_mutex> lock(globalMutex_);
+
+    std::vector<std::string> names;
+    names.reserve(collections_.size());
+    for (const auto& [name, _] : collections_) {
+        names.push_back(name);
+    }
+    return names;
+}
+
+std::optional<CollectionInfo> MemoryStore::getCollectionInfo(const std::string& name) const {
+    std::shared_lock<std::shared_mutex> globalLock(globalMutex_);
+
+    auto it = collections_.find(name);
+    if (it == collections_.end()) {
+        return std::nullopt;
+    }
+
+    std::shared_lock<std::shared_mutex> collLock(it->second->mutex);
+
+    CollectionInfo info;
+    info.name = name;
+    info.documentCount = it->second->documents.size();
+    info.options = it->second->options;
+    info.createdAt = it->second->createdAt;
+    info.updatedAt = it->second->updatedAt;
+
+    // Estimate size
+    for (const auto& [id, doc] : it->second->documents) {
+        info.sizeBytes += estimateDocumentSize(doc);
+    }
+
+    return info;
+}
+
+bool MemoryStore::collectionExists(const std::string& name) const {
+    std::shared_lock<std::shared_mutex> lock(globalMutex_);
+    return collections_.find(name) != collections_.end();
+}
+
+// ===== Document CRUD =====
+
+std::string MemoryStore::insert(const std::string& collection, Document doc) {
+    CollectionData* coll = getOrCreateCollection(collection);
+
+    std::unique_lock<std::shared_mutex> lock(coll->mutex);
+
+    // Generate ID if not provided
+    if (doc.id.empty()) {
+        if (coll->options.autoCreateId) {
+            doc.id = generateId();
+        } else {
+            throw std::runtime_error("Document ID is required");
+        }
+    }
+
+    // Check if ID already exists
+    if (coll->documents.find(doc.id) != coll->documents.end()) {
+        throw std::runtime_error("Document with ID '" + doc.id + "' already exists");
+    }
+
+    // Set metadata
+    doc.collection = collection;
+    doc.version = 1;
+    doc.createdAt = currentTimeMs();
+    doc.updatedAt = doc.createdAt;
+    doc.nodeId = config_.nodeId;
+
+    // Apply default TTL if not set
+    if (doc.expiresAt == 0 && coll->options.defaultTtlSeconds > 0) {
+        doc.setTtlSeconds(coll->options.defaultTtlSeconds);
+    }
+
+    // Add to expiration index
+    if (doc.expiresAt > 0) {
+        addToExpirationIndex(*coll, doc.id, doc.expiresAt);
+    }
+
+    std::string docId = doc.id;
+    coll->documents[docId] = doc;
+    coll->updatedAt = doc.updatedAt;
+
+    lock.unlock();
+
+    // Update stats
+    {
+        std::lock_guard<std::mutex> statsLock(statsMutex_);
+        stats_.totalDocuments++;
+        stats_.insertCount++;
+    }
+
+    // Emit callbacks
+    emitPersist(collection, docId, doc, EventType::INSERT);
+    emitEvent(EventType::INSERT, collection, docId, doc.data);
+
+    return docId;
+}
+
+std::optional<Document> MemoryStore::get(const std::string& collection, const std::string& id) const {
+    const CollectionData* coll = getCollection(collection);
+    if (!coll) {
+        return std::nullopt;
+    }
+
+    std::shared_lock<std::shared_mutex> lock(coll->mutex);
+
+    auto it = coll->documents.find(id);
+    if (it == coll->documents.end()) {
+        return std::nullopt;
+    }
+
+    // Check expiration
+    if (it->second.isExpired()) {
+        return std::nullopt;
+    }
+
+    return it->second;
+}
+
+bool MemoryStore::update(const std::string& collection, const std::string& id, const Document& doc) {
+    CollectionData* coll = getOrCreateCollection(collection);
+
+    std::unique_lock<std::shared_mutex> lock(coll->mutex);
+
+    auto it = coll->documents.find(id);
+    if (it == coll->documents.end()) {
+        return false;
+    }
+
+    // Remove from old expiration index
+    if (it->second.expiresAt > 0) {
+        removeFromExpirationIndex(*coll, id, it->second.expiresAt);
+    }
+
+    // Update document
+    Document updated = doc;
+    updated.id = id;
+    updated.collection = collection;
+    updated.version = it->second.version + 1;
+    updated.createdAt = it->second.createdAt;
+    updated.createdBy = it->second.createdBy;  // Preserve original creator
+    updated.updatedAt = currentTimeMs();
+    updated.nodeId = config_.nodeId;
+
+    // Add to new expiration index
+    if (updated.expiresAt > 0) {
+        addToExpirationIndex(*coll, id, updated.expiresAt);
+    }
+
+    it->second = updated;
+    coll->updatedAt = updated.updatedAt;
+
+    lock.unlock();
+
+    // Update stats
+    {
+        std::lock_guard<std::mutex> statsLock(statsMutex_);
+        stats_.updateCount++;
+    }
+
+    // Emit callbacks
+    emitPersist(collection, id, updated, EventType::UPDATE);
+    emitEvent(EventType::UPDATE, collection, id, updated.data);
+
+    return true;
+}
+
+std::string MemoryStore::upsert(const std::string& collection, Document doc) {
+    CollectionData* coll = getOrCreateCollection(collection);
+
+    std::unique_lock<std::shared_mutex> lock(coll->mutex);
+
+    // Generate ID if not provided
+    if (doc.id.empty()) {
+        if (coll->options.autoCreateId) {
+            doc.id = generateId();
+        } else {
+            throw std::runtime_error("Document ID is required");
+        }
+    }
+
+    std::string docId = doc.id;
+    bool isInsert = false;
+
+    auto it = coll->documents.find(docId);
+    if (it == coll->documents.end()) {
+        // Insert
+        isInsert = true;
+        doc.collection = collection;
+        doc.version = 1;
+        doc.createdAt = currentTimeMs();
+        doc.updatedAt = doc.createdAt;
+        doc.nodeId = config_.nodeId;
+
+        if (doc.expiresAt == 0 && coll->options.defaultTtlSeconds > 0) {
+            doc.setTtlSeconds(coll->options.defaultTtlSeconds);
+        }
+
+        if (doc.expiresAt > 0) {
+            addToExpirationIndex(*coll, docId, doc.expiresAt);
+        }
+
+        coll->documents[docId] = doc;
+    } else {
+        // Update
+        if (it->second.expiresAt > 0) {
+            removeFromExpirationIndex(*coll, docId, it->second.expiresAt);
+        }
+
+        doc.id = docId;
+        doc.collection = collection;
+        doc.version = it->second.version + 1;
+        doc.createdAt = it->second.createdAt;
+        doc.createdBy = it->second.createdBy;  // Preserve original creator
+        doc.updatedAt = currentTimeMs();
+        doc.nodeId = config_.nodeId;
+
+        if (doc.expiresAt > 0) {
+            addToExpirationIndex(*coll, docId, doc.expiresAt);
+        }
+
+        it->second = doc;
+    }
+
+    coll->updatedAt = doc.updatedAt;
+
+    lock.unlock();
+
+    // Update stats
+    {
+        std::lock_guard<std::mutex> statsLock(statsMutex_);
+        if (isInsert) {
+            stats_.totalDocuments++;
+            stats_.insertCount++;
+        } else {
+            stats_.updateCount++;
+        }
+    }
+
+    // Emit callbacks
+    emitPersist(collection, docId, doc, isInsert ? EventType::INSERT : EventType::UPDATE);
+    emitEvent(isInsert ? EventType::INSERT : EventType::UPDATE, collection, docId, doc.data);
+
+    return docId;
+}
+
+bool MemoryStore::remove(const std::string& collection, const std::string& id) {
+    CollectionData* coll = getOrCreateCollection(collection);
+
+    std::unique_lock<std::shared_mutex> lock(coll->mutex);
+
+    auto it = coll->documents.find(id);
+    if (it == coll->documents.end()) {
+        return false;
+    }
+
+    // Remove from expiration index
+    if (it->second.expiresAt > 0) {
+        removeFromExpirationIndex(*coll, id, it->second.expiresAt);
+    }
+
+    coll->documents.erase(it);
+    coll->updatedAt = currentTimeMs();
+
+    lock.unlock();
+
+    // Update stats
+    {
+        std::lock_guard<std::mutex> statsLock(statsMutex_);
+        stats_.totalDocuments--;
+        stats_.deleteCount++;
+    }
+
+    // Emit callbacks
+    emitPersist(collection, id, std::nullopt, EventType::DELETE);
+    emitEvent(EventType::DELETE, collection, id);
+
+    return true;
+}
+
+bool MemoryStore::exists(const std::string& collection, const std::string& id) const {
+    const CollectionData* coll = getCollection(collection);
+    if (!coll) {
+        return false;
+    }
+
+    std::shared_lock<std::shared_mutex> lock(coll->mutex);
+
+    auto it = coll->documents.find(id);
+    if (it == coll->documents.end()) {
+        return false;
+    }
+
+    return !it->second.isExpired();
+}
+
+bool MemoryStore::updateIfVersion(const std::string& collection, const std::string& id,
+                                   const Document& doc, uint64_t expectedVersion) {
+    CollectionData* coll = getOrCreateCollection(collection);
+
+    std::unique_lock<std::shared_mutex> lock(coll->mutex);
+
+    auto it = coll->documents.find(id);
+    if (it == coll->documents.end()) {
+        return false;
+    }
+
+    if (it->second.version != expectedVersion) {
+        return false;
+    }
+
+    // Remove from old expiration index
+    if (it->second.expiresAt > 0) {
+        removeFromExpirationIndex(*coll, id, it->second.expiresAt);
+    }
+
+    // Update document
+    Document updated = doc;
+    updated.id = id;
+    updated.collection = collection;
+    updated.version = expectedVersion + 1;
+    updated.createdAt = it->second.createdAt;
+    updated.createdBy = it->second.createdBy;  // Preserve original creator
+    updated.updatedAt = currentTimeMs();
+    updated.nodeId = config_.nodeId;
+
+    // Add to new expiration index
+    if (updated.expiresAt > 0) {
+        addToExpirationIndex(*coll, id, updated.expiresAt);
+    }
+
+    it->second = updated;
+    coll->updatedAt = updated.updatedAt;
+
+    lock.unlock();
+
+    // Update stats
+    {
+        std::lock_guard<std::mutex> statsLock(statsMutex_);
+        stats_.updateCount++;
+    }
+
+    // Emit callbacks
+    emitPersist(collection, id, updated, EventType::UPDATE);
+    emitEvent(EventType::UPDATE, collection, id, updated.data);
+
+    return true;
+}
+
+// ===== Query Operations =====
+
+QueryResult MemoryStore::find(const std::string& collection, const Query& query) const {
+    const CollectionData* coll = getCollection(collection);
+    if (!coll) {
+        return {};
+    }
+
+    std::shared_lock<std::shared_mutex> lock(coll->mutex);
+
+    // Update query count
+    {
+        std::lock_guard<std::mutex> statsLock(statsMutex_);
+        const_cast<MemoryStore*>(this)->stats_.queryCount++;
+    }
+
+    // Collect matching documents
+    std::vector<Document> matches;
+    for (const auto& [id, doc] : coll->documents) {
+        if (doc.isExpired()) {
+            continue;
+        }
+        if (matchesFilters(doc, query.filters)) {
+            matches.push_back(doc);
+        }
+    }
+
+    // Sort if requested
+    if (query.sort && !query.sort->field.empty()) {
+        SPDLOG_DEBUG("Sorting {} documents by field '{}' descending={}",
+                     matches.size(), query.sort->field, query.sort->descending);
+        std::sort(matches.begin(), matches.end(),
+            [&query](const Document& a, const Document& b) {
+                std::optional<nlohmann::json> valA, valB;
+
+                // Handle special fields that are stored as struct members, not in data
+                if (query.sort->field == "_id") {
+                    valA = nlohmann::json(a.id);
+                    valB = nlohmann::json(b.id);
+                } else if (query.sort->field == "_created_at") {
+                    valA = nlohmann::json(a.createdAt);
+                    valB = nlohmann::json(b.createdAt);
+                } else if (query.sort->field == "_updated_at") {
+                    valA = nlohmann::json(a.updatedAt);
+                    valB = nlohmann::json(b.updatedAt);
+                } else {
+                    valA = getJsonPath(a.data, query.sort->field);
+                    valB = getJsonPath(b.data, query.sort->field);
+                }
+
+                if (!valA && !valB) {
+                    // Both null - use ID as tiebreaker for stable ordering
+                    return query.sort->descending ? a.id > b.id : a.id < b.id;
+                }
+                if (!valA) return query.sort->descending;
+                if (!valB) return !query.sort->descending;
+
+                // Compare primary sort values
+                if (*valA == *valB) {
+                    // Equal values - use ID as tiebreaker for stable ordering
+                    return query.sort->descending ? a.id > b.id : a.id < b.id;
+                }
+
+                bool less = *valA < *valB;
+                return query.sort->descending ? !less : less;
+            });
+
+        // Log first few document timestamps after sorting
+        if (matches.size() > 0) {
+            std::string debugInfo = "First documents after sort: ";
+            size_t count = std::min<size_t>(5, matches.size());
+            for (size_t i = 0; i < count; ++i) {
+                if (query.sort->field == "_created_at") {
+                    debugInfo += "[" + matches[i].id + ": " + std::to_string(matches[i].createdAt) + "] ";
+                } else if (query.sort->field == "_updated_at") {
+                    debugInfo += "[" + matches[i].id + ": " + std::to_string(matches[i].updatedAt) + "] ";
+                } else {
+                    auto val = getJsonPath(matches[i].data, query.sort->field);
+                    debugInfo += "[" + matches[i].id + ": " + (val ? val->dump() : "null") + "] ";
+                }
+            }
+            SPDLOG_DEBUG("{}", debugInfo);
+        }
+    }
+
+    // Build result with pagination
+    QueryResult result;
+    result.totalCount = matches.size();
+
+    uint32_t start = std::min(query.offset, static_cast<uint32_t>(matches.size()));
+    uint32_t end = std::min(query.offset + query.limit, static_cast<uint32_t>(matches.size()));
+
+    for (uint32_t i = start; i < end; ++i) {
+        result.documents.push_back(matches[i]);
+    }
+
+    result.hasMore = end < matches.size();
+
+    return result;
+}
+
+uint64_t MemoryStore::count(const std::string& collection, const std::vector<Filter>& filters) const {
+    const CollectionData* coll = getCollection(collection);
+    if (!coll) {
+        return 0;
+    }
+
+    std::shared_lock<std::shared_mutex> lock(coll->mutex);
+
+    if (filters.empty()) {
+        // Count non-expired documents
+        uint64_t count = 0;
+        for (const auto& [id, doc] : coll->documents) {
+            if (!doc.isExpired()) {
+                count++;
+            }
+        }
+        return count;
+    }
+
+    uint64_t count = 0;
+    for (const auto& [id, doc] : coll->documents) {
+        if (!doc.isExpired() && matchesFilters(doc, filters)) {
+            count++;
+        }
+    }
+    return count;
+}
+
+// ===== Set Operations =====
+
+bool MemoryStore::setAdd(const std::string& collection, const std::string& setId, const std::string& member) {
+    CollectionData* coll = getOrCreateCollection(collection);
+
+    std::unique_lock<std::shared_mutex> lock(coll->mutex);
+
+    auto it = coll->documents.find(setId);
+    if (it == coll->documents.end()) {
+        // Create new set document
+        Document doc;
+        doc.id = setId;
+        doc.collection = collection;
+        doc.data = nlohmann::json::object();
+        doc.data["members"] = nlohmann::json::array({member});
+        doc.version = 1;
+        doc.createdAt = currentTimeMs();
+        doc.updatedAt = doc.createdAt;
+        doc.nodeId = config_.nodeId;
+
+        if (coll->options.defaultTtlSeconds > 0) {
+            doc.setTtlSeconds(coll->options.defaultTtlSeconds);
+            addToExpirationIndex(*coll, setId, doc.expiresAt);
+        }
+
+        coll->documents[setId] = doc;
+        coll->updatedAt = doc.updatedAt;
+
+        lock.unlock();
+
+        {
+            std::lock_guard<std::mutex> statsLock(statsMutex_);
+            stats_.totalDocuments++;
+            stats_.insertCount++;
+        }
+
+        emitPersist(collection, setId, doc, EventType::INSERT);
+        emitEvent(EventType::INSERT, collection, setId, doc.data);
+
+        return true;
+    }
+
+    // Check if member already exists
+    auto& members = it->second.data["members"];
+    if (!members.is_array()) {
+        members = nlohmann::json::array();
+    }
+
+    for (const auto& m : members) {
+        if (m.is_string() && m.get<std::string>() == member) {
+            return false;  // Already exists
+        }
+    }
+
+    members.push_back(member);
+    it->second.version++;
+    it->second.updatedAt = currentTimeMs();
+    it->second.nodeId = config_.nodeId;
+    coll->updatedAt = it->second.updatedAt;
+
+    Document updated = it->second;
+
+    lock.unlock();
+
+    {
+        std::lock_guard<std::mutex> statsLock(statsMutex_);
+        stats_.updateCount++;
+    }
+
+    emitPersist(collection, setId, updated, EventType::UPDATE);
+    emitEvent(EventType::UPDATE, collection, setId, updated.data);
+
+    return true;
+}
+
+bool MemoryStore::setRemove(const std::string& collection, const std::string& setId, const std::string& member) {
+    CollectionData* coll = getOrCreateCollection(collection);
+
+    std::unique_lock<std::shared_mutex> lock(coll->mutex);
+
+    auto it = coll->documents.find(setId);
+    if (it == coll->documents.end()) {
+        return false;
+    }
+
+    auto& members = it->second.data["members"];
+    if (!members.is_array()) {
+        return false;
+    }
+
+    bool found = false;
+    auto newMembers = nlohmann::json::array();
+    for (const auto& m : members) {
+        if (m.is_string() && m.get<std::string>() == member) {
+            found = true;
+        } else {
+            newMembers.push_back(m);
+        }
+    }
+
+    if (!found) {
+        return false;
+    }
+
+    it->second.data["members"] = newMembers;
+    it->second.version++;
+    it->second.updatedAt = currentTimeMs();
+    it->second.nodeId = config_.nodeId;
+    coll->updatedAt = it->second.updatedAt;
+
+    Document updated = it->second;
+
+    lock.unlock();
+
+    {
+        std::lock_guard<std::mutex> statsLock(statsMutex_);
+        stats_.updateCount++;
+    }
+
+    emitPersist(collection, setId, updated, EventType::UPDATE);
+    emitEvent(EventType::UPDATE, collection, setId, updated.data);
+
+    return true;
+}
+
+std::vector<std::string> MemoryStore::setMembers(const std::string& collection, const std::string& setId) const {
+    const CollectionData* coll = getCollection(collection);
+    if (!coll) {
+        return {};
+    }
+
+    std::shared_lock<std::shared_mutex> lock(coll->mutex);
+
+    auto it = coll->documents.find(setId);
+    if (it == coll->documents.end() || it->second.isExpired()) {
+        return {};
+    }
+
+    const auto& members = it->second.data["members"];
+    if (!members.is_array()) {
+        return {};
+    }
+
+    std::vector<std::string> result;
+    for (const auto& m : members) {
+        if (m.is_string()) {
+            result.push_back(m.get<std::string>());
+        }
+    }
+    return result;
+}
+
+bool MemoryStore::setIsMember(const std::string& collection, const std::string& setId, const std::string& member) const {
+    const CollectionData* coll = getCollection(collection);
+    if (!coll) {
+        return false;
+    }
+
+    std::shared_lock<std::shared_mutex> lock(coll->mutex);
+
+    auto it = coll->documents.find(setId);
+    if (it == coll->documents.end() || it->second.isExpired()) {
+        return false;
+    }
+
+    const auto& members = it->second.data["members"];
+    if (!members.is_array()) {
+        return false;
+    }
+
+    for (const auto& m : members) {
+        if (m.is_string() && m.get<std::string>() == member) {
+            return true;
+        }
+    }
+    return false;
+}
+
+// ===== Bulk Operations =====
+
+std::vector<std::string> MemoryStore::bulkInsert(const std::string& collection, std::vector<Document> docs) {
+    std::vector<std::string> ids;
+    ids.reserve(docs.size());
+
+    for (auto& doc : docs) {
+        try {
+            ids.push_back(insert(collection, std::move(doc)));
+        } catch (const std::exception&) {
+            // Continue with other documents
+        }
+    }
+
+    return ids;
+}
+
+uint64_t MemoryStore::bulkDelete(const std::string& collection, const std::vector<std::string>& ids) {
+    uint64_t deleted = 0;
+    for (const auto& id : ids) {
+        if (remove(collection, id)) {
+            deleted++;
+        }
+    }
+    return deleted;
+}
+
+// ===== TTL Management =====
+
+uint64_t MemoryStore::expireDocuments() {
+    uint64_t expired = 0;
+    auto now = currentTimeMs();
+
+    std::shared_lock<std::shared_mutex> globalLock(globalMutex_);
+
+    for (auto& [collName, coll] : collections_) {
+        std::unique_lock<std::shared_mutex> collLock(coll->mutex);
+
+        // Find expired documents
+        std::vector<std::string> toExpire;
+        auto it = coll->expirationIndex.begin();
+        while (it != coll->expirationIndex.end() && it->first <= now) {
+            for (const auto& id : it->second) {
+                toExpire.push_back(id);
+            }
+            it = coll->expirationIndex.erase(it);
+        }
+
+        // Remove expired documents
+        for (const auto& id : toExpire) {
+            auto docIt = coll->documents.find(id);
+            if (docIt != coll->documents.end()) {
+                coll->documents.erase(docIt);
+                expired++;
+
+                // Emit callbacks (unlock first to avoid deadlock)
+                collLock.unlock();
+
+                {
+                    std::lock_guard<std::mutex> statsLock(statsMutex_);
+                    stats_.totalDocuments--;
+                    stats_.expiredCount++;
+                }
+
+                emitPersist(collName, id, std::nullopt, EventType::EXPIRE);
+                emitEvent(EventType::EXPIRE, collName, id);
+
+                collLock.lock();
+            }
+        }
+    }
+
+    return expired;
+}
+
+// ===== Statistics =====
+
+MemoryStore::Stats MemoryStore::getStats() const {
+    std::lock_guard<std::mutex> lock(statsMutex_);
+
+    Stats stats = stats_;
+
+    // Calculate estimated memory
+    std::shared_lock<std::shared_mutex> globalLock(globalMutex_);
+    stats.estimatedMemoryBytes = 0;
+    for (const auto& [name, coll] : collections_) {
+        std::shared_lock<std::shared_mutex> collLock(coll->mutex);
+        for (const auto& [id, doc] : coll->documents) {
+            stats.estimatedMemoryBytes += estimateDocumentSize(doc);
+        }
+    }
+
+    return stats;
+}
+
+// ===== Snapshot/Recovery Support =====
+
+std::vector<Document> MemoryStore::getAllDocuments(const std::string& collection) const {
+    const CollectionData* coll = getCollection(collection);
+    if (!coll) {
+        return {};
+    }
+
+    std::shared_lock<std::shared_mutex> lock(coll->mutex);
+
+    std::vector<Document> docs;
+    docs.reserve(coll->documents.size());
+    for (const auto& [id, doc] : coll->documents) {
+        docs.push_back(doc);
+    }
+    return docs;
+}
+
+std::vector<std::pair<std::string, CollectionOptions>> MemoryStore::getAllCollectionsWithOptions() const {
+    std::shared_lock<std::shared_mutex> lock(globalMutex_);
+
+    std::vector<std::pair<std::string, CollectionOptions>> result;
+    result.reserve(collections_.size());
+    for (const auto& [name, coll] : collections_) {
+        std::shared_lock<std::shared_mutex> collLock(coll->mutex);
+        result.emplace_back(name, coll->options);
+    }
+    return result;
+}
+
+void MemoryStore::loadDocument(const std::string& collection, Document doc) {
+    CollectionData* coll = getOrCreateCollection(collection);
+
+    std::unique_lock<std::shared_mutex> lock(coll->mutex);
+
+    doc.collection = collection;
+
+    // Add to expiration index
+    if (doc.expiresAt > 0) {
+        addToExpirationIndex(*coll, doc.id, doc.expiresAt);
+    }
+
+    coll->documents[doc.id] = std::move(doc);
+
+    {
+        std::lock_guard<std::mutex> statsLock(statsMutex_);
+        stats_.totalDocuments++;
+    }
+}
+
+void MemoryStore::clear() {
+    std::unique_lock<std::shared_mutex> lock(globalMutex_);
+    collections_.clear();
+
+    std::lock_guard<std::mutex> statsLock(statsMutex_);
+    stats_ = Stats{};
+}
+
+// ===== Private Methods =====
+
+std::string MemoryStore::generateId() const {
+    static std::random_device rd;
+    static std::mt19937_64 gen(rd());
+    static std::uniform_int_distribution<uint64_t> dis;
+
+    auto now = currentTimeMs();
+    uint64_t random = dis(gen);
+
+    std::ostringstream oss;
+    oss << std::hex << std::setfill('0')
+        << std::setw(12) << now
+        << std::setw(16) << random;
+
+    return oss.str();
+}
+
+uint64_t MemoryStore::currentTimeMs() {
+    return static_cast<uint64_t>(
+        std::chrono::duration_cast<std::chrono::milliseconds>(
+            std::chrono::system_clock::now().time_since_epoch()
+        ).count()
+    );
+}
+
+bool MemoryStore::matchesFilters(const Document& doc, const std::vector<Filter>& filters) const {
+    for (const auto& filter : filters) {
+        auto value = getJsonPath(doc.data, filter.field);
+
+        switch (filter.op) {
+            case FilterOp::EXISTS:
+                if (value.has_value() != filter.value.get<bool>()) {
+                    return false;
+                }
+                break;
+
+            case FilterOp::EQ:
+            case FilterOp::NE:
+            case FilterOp::GT:
+            case FilterOp::GTE:
+            case FilterOp::LT:
+            case FilterOp::LTE:
+                if (!value) {
+                    return false;
+                }
+                if (!compareJson(*value, filter.op, filter.value)) {
+                    return false;
+                }
+                break;
+
+            case FilterOp::IN:
+                if (!value || !filter.value.is_array()) {
+                    return false;
+                }
+                {
+                    bool found = false;
+                    for (const auto& v : filter.value) {
+                        if (*value == v) {
+                            found = true;
+                            break;
+                        }
+                    }
+                    if (!found) return false;
+                }
+                break;
+
+            case FilterOp::CONTAINS:
+                if (!value || !value->is_array()) {
+                    return false;
+                }
+                {
+                    bool found = false;
+                    for (const auto& v : *value) {
+                        if (v == filter.value) {
+                            found = true;
+                            break;
+                        }
+                    }
+                    if (!found) return false;
+                }
+                break;
+
+            case FilterOp::REGEX:
+                if (!value || !value->is_string() || !filter.value.is_string()) {
+                    return false;
+                }
+                try {
+                    std::regex re(filter.value.get<std::string>());
+                    if (!std::regex_search(value->get<std::string>(), re)) {
+                        return false;
+                    }
+                } catch (const std::regex_error&) {
+                    return false;
+                }
+                break;
+
+            case FilterOp::SEARCH:
+                // Full-text search across document ID and string fields
+                if (!filter.value.is_string()) {
+                    return false;
+                }
+                if (!matchesSearch(doc, filter.value.get<std::string>())) {
+                    return false;
+                }
+                break;
+        }
+    }
+
+    return true;
+}
+
+std::optional<nlohmann::json> MemoryStore::getJsonPath(const nlohmann::json& obj, const std::string& path) {
+    if (path.empty() || !obj.is_object()) {
+        return std::nullopt;
+    }
+
+    // Split path by '.'
+    std::vector<std::string> parts;
+    std::istringstream iss(path);
+    std::string part;
+    while (std::getline(iss, part, '.')) {
+        parts.push_back(part);
+    }
+
+    const nlohmann::json* current = &obj;
+    for (const auto& p : parts) {
+        if (!current->is_object() || !current->contains(p)) {
+            return std::nullopt;
+        }
+        current = &(*current)[p];
+    }
+
+    return std::make_optional(*current);
+}
+
+bool MemoryStore::compareJson(const nlohmann::json& a, FilterOp op, const nlohmann::json& b) {
+    switch (op) {
+        case FilterOp::EQ:
+            return a == b;
+        case FilterOp::NE:
+            return a != b;
+        case FilterOp::GT:
+            return a > b;
+        case FilterOp::GTE:
+            return a >= b;
+        case FilterOp::LT:
+            return a < b;
+        case FilterOp::LTE:
+            return a <= b;
+        default:
+            return false;
+    }
+}
+
+bool MemoryStore::matchesSearch(const Document& doc, const std::string& searchTerm) const {
+    if (searchTerm.empty()) {
+        return true;  // Empty search matches everything
+    }
+
+    // Convert search term to lowercase for case-insensitive matching
+    std::string lowerSearchTerm = searchTerm;
+    std::transform(lowerSearchTerm.begin(), lowerSearchTerm.end(), lowerSearchTerm.begin(),
+                   [](unsigned char c) { return std::tolower(c); });
+
+    // Check document ID
+    std::string lowerId = doc.id;
+    std::transform(lowerId.begin(), lowerId.end(), lowerId.begin(),
+                   [](unsigned char c) { return std::tolower(c); });
+    if (lowerId.find(lowerSearchTerm) != std::string::npos) {
+        return true;
+    }
+
+    // Search in document data
+    return searchInJson(doc.data, lowerSearchTerm);
+}
+
+bool MemoryStore::searchInJson(const nlohmann::json& obj, const std::string& lowerSearchTerm) {
+    if (obj.is_string()) {
+        std::string lowerValue = obj.get<std::string>();
+        std::transform(lowerValue.begin(), lowerValue.end(), lowerValue.begin(),
+                       [](unsigned char c) { return std::tolower(c); });
+        return lowerValue.find(lowerSearchTerm) != std::string::npos;
+    }
+
+    if (obj.is_object()) {
+        for (const auto& [key, value] : obj.items()) {
+            // Search in keys too
+            std::string lowerKey = key;
+            std::transform(lowerKey.begin(), lowerKey.end(), lowerKey.begin(),
+                           [](unsigned char c) { return std::tolower(c); });
+            if (lowerKey.find(lowerSearchTerm) != std::string::npos) {
+                return true;
+            }
+            // Recursively search in values
+            if (searchInJson(value, lowerSearchTerm)) {
+                return true;
+            }
+        }
+    }
+
+    if (obj.is_array()) {
+        for (const auto& item : obj) {
+            if (searchInJson(item, lowerSearchTerm)) {
+                return true;
+            }
+        }
+    }
+
+    // For numbers, convert to string and search
+    if (obj.is_number()) {
+        std::string numStr = obj.dump();
+        if (numStr.find(lowerSearchTerm) != std::string::npos) {
+            return true;
+        }
+    }
+
+    return false;
+}
+
+void MemoryStore::emitEvent(EventType type, const std::string& collection, const std::string& id,
+                            const std::optional<nlohmann::json>& data) {
+    std::lock_guard<std::mutex> lock(callbackMutex_);
+    if (eventCallback_) {
+        DatabaseEvent event;
+        event.type = type;
+        event.collection = collection;
+        event.documentId = id;
+        event.timestamp = currentTimeMs();
+        event.nodeId = config_.nodeId;
+        event.data = data;
+        eventCallback_(event);
+    }
+}
+
+void MemoryStore::emitPersist(const std::string& collection, const std::string& id,
+                              const std::optional<Document>& doc, EventType eventType) {
+    std::lock_guard<std::mutex> lock(callbackMutex_);
+    if (persistCallback_) {
+        persistCallback_(collection, id, doc, eventType);
+    }
+}
+
+void MemoryStore::addToExpirationIndex(CollectionData& coll, const std::string& id, uint64_t expiresAt) {
+    coll.expirationIndex[expiresAt].insert(id);
+}
+
+void MemoryStore::removeFromExpirationIndex(CollectionData& coll, const std::string& id, uint64_t expiresAt) {
+    auto it = coll.expirationIndex.find(expiresAt);
+    if (it != coll.expirationIndex.end()) {
+        it->second.erase(id);
+        if (it->second.empty()) {
+            coll.expirationIndex.erase(it);
+        }
+    }
+}
+
+uint64_t MemoryStore::estimateDocumentSize(const Document& doc) {
+    // Rough estimate: ID + collection + JSON dump size + metadata
+    return doc.id.size() + doc.collection.size() + doc.data.dump().size() + 100;
+}
+
+void MemoryStore::expirationLoop() {
+    while (running_.load()) {
+        std::this_thread::sleep_for(
+            std::chrono::milliseconds(config_.expirationCheckIntervalMs)
+        );
+
+        if (running_.load()) {
+            expireDocuments();
+        }
+    }
+}
+
+MemoryStore::CollectionData* MemoryStore::getOrCreateCollection(const std::string& name) {
+    {
+        std::shared_lock<std::shared_mutex> lock(globalMutex_);
+        auto it = collections_.find(name);
+        if (it != collections_.end()) {
+            return it->second.get();
+        }
+    }
+
+    // Create collection
+    std::unique_lock<std::shared_mutex> lock(globalMutex_);
+
+    // Double-check after acquiring exclusive lock
+    auto it = collections_.find(name);
+    if (it != collections_.end()) {
+        return it->second.get();
+    }
+
+    auto coll = std::make_unique<CollectionData>();
+    coll->createdAt = currentTimeMs();
+    coll->updatedAt = coll->createdAt;
+
+    auto* ptr = coll.get();
+    collections_[name] = std::move(coll);
+
+    {
+        std::lock_guard<std::mutex> statsLock(statsMutex_);
+        stats_.totalCollections++;
+    }
+
+    return ptr;
+}
+
+const MemoryStore::CollectionData* MemoryStore::getCollection(const std::string& name) const {
+    std::shared_lock<std::shared_mutex> lock(globalMutex_);
+    auto it = collections_.find(name);
+    if (it != collections_.end()) {
+        return it->second.get();
+    }
+    return nullptr;
+}
+
+} // namespace smartbotic::database

service/src/memory_store.hpp

@@ -0,0 +1,322 @@
+#pragma once
+
+#include "document.hpp"
+
+#include <atomic>
+#include <functional>
+#include <map>
+#include <memory>
+#include <mutex>
+#include <optional>
+#include <shared_mutex>
+#include <string>
+#include <thread>
+#include <unordered_map>
+#include <unordered_set>
+#include <vector>
+
+namespace smartbotic::database {
+
+/**
+ * Callback for storage events.
+ */
+using EventCallback = std::function<void(const DatabaseEvent&)>;
+
+/**
+ * Callback for WAL/persistence operations.
+ */
+using PersistCallback = std::function<void(const std::string& collection,
+                                           const std::string& id,
+                                           const std::optional<Document>& doc,
+                                           EventType eventType)>;
+
+/**
+ * In-memory storage engine with collection-based document storage.
+ *
+ * Features:
+ * - Thread-safe with per-collection read-write locks
+ * - TTL support with background expiration
+ * - Optimistic locking via document versions
+ * - Set operations for compatibility with Redis patterns
+ * - Event callbacks for persistence and pub/sub
+ */
+class MemoryStore {
+public:
+    struct Config {
+        uint64_t maxMemoryBytes;
+        uint32_t expirationCheckIntervalMs;
+        std::string nodeId;
+
+        Config()
+            : maxMemoryBytes(1024ULL * 1024 * 1024)  // 1GB default
+            , expirationCheckIntervalMs(1000)         // Check every second
+            , nodeId("storage-1")
+        {}
+    };
+
+    explicit MemoryStore(Config config = Config{});
+    ~MemoryStore();
+
+    // Non-copyable, non-movable
+    MemoryStore(const MemoryStore&) = delete;
+    MemoryStore& operator=(const MemoryStore&) = delete;
+    MemoryStore(MemoryStore&&) = delete;
+    MemoryStore& operator=(MemoryStore&&) = delete;
+
+    // Lifecycle
+    void start();
+    void stop();
+    [[nodiscard]] bool isRunning() const { return running_.load(); }
+
+    // Event callbacks
+    void setEventCallback(EventCallback callback);
+    void setPersistCallback(PersistCallback callback);
+
+    // ===== Collection Management =====
+
+    /**
+     * Create a new collection.
+     * @return true if created, false if already exists
+     */
+    bool createCollection(const std::string& name, const CollectionOptions& options = {});
+
+    /**
+     * Drop a collection and all its documents.
+     * @return true if dropped, false if not found
+     */
+    bool dropCollection(const std::string& name);
+
+    /**
+     * List all collection names.
+     */
+    [[nodiscard]] std::vector<std::string> listCollections() const;
+
+    /**
+     * Get collection info.
+     */
+    [[nodiscard]] std::optional<CollectionInfo> getCollectionInfo(const std::string& name) const;
+
+    /**
+     * Check if collection exists.
+     */
+    [[nodiscard]] bool collectionExists(const std::string& name) const;
+
+    // ===== Document CRUD =====
+
+    /**
+     * Insert a new document.
+     * @param collection Collection name
+     * @param doc Document to insert (id will be auto-generated if empty)
+     * @return Document ID
+     * @throws std::runtime_error if collection doesn't exist or ID already exists
+     */
+    std::string insert(const std::string& collection, Document doc);
+
+    /**
+     * Get a document by ID.
+     * @return Document if found and not expired, nullopt otherwise
+     */
+    [[nodiscard]] std::optional<Document> get(const std::string& collection, const std::string& id) const;
+
+    /**
+     * Update an existing document.
+     * @return true if updated, false if not found
+     */
+    bool update(const std::string& collection, const std::string& id, const Document& doc);
+
+    /**
+     * Insert or update a document.
+     * @return Document ID
+     */
+    std::string upsert(const std::string& collection, Document doc);
+
+    /**
+     * Delete a document.
+     * @return true if deleted, false if not found
+     */
+    bool remove(const std::string& collection, const std::string& id);
+
+    /**
+     * Check if a document exists.
+     */
+    [[nodiscard]] bool exists(const std::string& collection, const std::string& id) const;
+
+    /**
+     * Update with optimistic locking.
+     * @param expectedVersion Expected version of the document
+     * @return true if updated, false if version mismatch or not found
+     */
+    bool updateIfVersion(const std::string& collection, const std::string& id,
+                         const Document& doc, uint64_t expectedVersion);
+
+    // ===== Query Operations =====
+
+    /**
+     * Find documents matching a query.
+     */
+    [[nodiscard]] QueryResult find(const std::string& collection, const Query& query) const;
+
+    /**
+     * Count documents matching filters.
+     */
+    [[nodiscard]] uint64_t count(const std::string& collection, const std::vector<Filter>& filters = {}) const;
+
+    // ===== Set Operations (Redis Compatibility) =====
+
+    /**
+     * Add a member to a set.
+     * Sets are stored as documents with a "members" array.
+     * @return true if member was added, false if already exists
+     */
+    bool setAdd(const std::string& collection, const std::string& setId, const std::string& member);
+
+    /**
+     * Remove a member from a set.
+     * @return true if member was removed, false if not found
+     */
+    bool setRemove(const std::string& collection, const std::string& setId, const std::string& member);
+
+    /**
+     * Get all members of a set.
+     */
+    [[nodiscard]] std::vector<std::string> setMembers(const std::string& collection, const std::string& setId) const;
+
+    /**
+     * Check if a value is a member of a set.
+     */
+    [[nodiscard]] bool setIsMember(const std::string& collection, const std::string& setId, const std::string& member) const;
+
+    // ===== Bulk Operations =====
+
+    /**
+     * Insert multiple documents.
+     * @return Vector of inserted document IDs
+     */
+    std::vector<std::string> bulkInsert(const std::string& collection, std::vector<Document> docs);
+
+    /**
+     * Delete multiple documents by ID.
+     * @return Number of documents deleted
+     */
+    uint64_t bulkDelete(const std::string& collection, const std::vector<std::string>& ids);
+
+    // ===== TTL Management =====
+
+    /**
+     * Expire documents that have passed their TTL.
+     * Called automatically by background thread.
+     * @return Number of documents expired
+     */
+    uint64_t expireDocuments();
+
+    // ===== Statistics =====
+
+    struct Stats {
+        uint64_t totalDocuments = 0;
+        uint64_t totalCollections = 0;
+        uint64_t estimatedMemoryBytes = 0;
+        uint64_t expiredCount = 0;
+        uint64_t insertCount = 0;
+        uint64_t updateCount = 0;
+        uint64_t deleteCount = 0;
+        uint64_t queryCount = 0;
+    };
+
+    [[nodiscard]] Stats getStats() const;
+
+    // ===== Snapshot/Recovery Support =====
+
+    /**
+     * Get all documents in a collection (for snapshotting).
+     */
+    [[nodiscard]] std::vector<Document> getAllDocuments(const std::string& collection) const;
+
+    /**
+     * Get all collections with their options (for snapshotting).
+     */
+    [[nodiscard]] std::vector<std::pair<std::string, CollectionOptions>> getAllCollectionsWithOptions() const;
+
+    /**
+     * Load a document directly (for recovery, bypasses event callbacks).
+     */
+    void loadDocument(const std::string& collection, Document doc);
+
+    /**
+     * Clear all data (for testing/recovery).
+     */
+    void clear();
+
+private:
+    struct CollectionData {
+        CollectionOptions options;
+        std::unordered_map<std::string, Document> documents;
+        std::map<uint64_t, std::unordered_set<std::string>> expirationIndex;  // expiresAt -> ids
+        uint64_t createdAt = 0;
+        uint64_t updatedAt = 0;
+        mutable std::shared_mutex mutex;
+    };
+
+    // Generate a unique document ID
+    [[nodiscard]] std::string generateId() const;
+
+    // Get current timestamp in milliseconds
+    [[nodiscard]] static uint64_t currentTimeMs();
+
+    // Apply filters to a document
+    [[nodiscard]] bool matchesFilters(const Document& doc, const std::vector<Filter>& filters) const;
+
+    // Get JSON value at a path (e.g., "user.email")
+    [[nodiscard]] static std::optional<nlohmann::json> getJsonPath(const nlohmann::json& obj, const std::string& path);
+
+    // Compare JSON values
+    [[nodiscard]] static bool compareJson(const nlohmann::json& a, FilterOp op, const nlohmann::json& b);
+
+    // Search matching - case-insensitive search across document ID and string fields
+    [[nodiscard]] bool matchesSearch(const Document& doc, const std::string& searchTerm) const;
+
+    // Recursively search JSON for string matches (case-insensitive)
+    [[nodiscard]] static bool searchInJson(const nlohmann::json& obj, const std::string& lowerSearchTerm);
+
+    // Emit an event
+    void emitEvent(EventType type, const std::string& collection, const std::string& id,
+                   const std::optional<nlohmann::json>& data = std::nullopt);
+
+    // Emit a persist callback
+    void emitPersist(const std::string& collection, const std::string& id,
+                     const std::optional<Document>& doc, EventType eventType);
+
+    // Update expiration index
+    void addToExpirationIndex(CollectionData& coll, const std::string& id, uint64_t expiresAt);
+    void removeFromExpirationIndex(CollectionData& coll, const std::string& id, uint64_t expiresAt);
+
+    // Estimate document memory size
+    [[nodiscard]] static uint64_t estimateDocumentSize(const Document& doc);
+
+    // Background expiration thread
+    void expirationLoop();
+
+    // Ensure collection exists, returns pointer to collection data
+    CollectionData* getOrCreateCollection(const std::string& name);
+    const CollectionData* getCollection(const std::string& name) const;
+
+    Config config_;
+    std::atomic<bool> running_{false};
+
+    // Collections map with global mutex for collection creation/deletion
+    std::unordered_map<std::string, std::unique_ptr<CollectionData>> collections_;
+    mutable std::shared_mutex globalMutex_;
+
+    // Event callbacks
+    EventCallback eventCallback_;
+    PersistCallback persistCallback_;
+    std::mutex callbackMutex_;
+
+    // Background expiration thread
+    std::thread expirationThread_;
+
+    // Statistics
+    mutable std::mutex statsMutex_;
+    Stats stats_;
+};
+
+} // namespace smartbotic::database

service/src/migrations/migration_runner.cpp

@@ -0,0 +1,395 @@
+#include "migration_runner.hpp"
+
+#include <spdlog/spdlog.h>
+#include <nlohmann/json.hpp>
+
+#include <algorithm>
+#include <fstream>
+#include <iomanip>
+#include <sstream>
+
+namespace smartbotic::database {
+
+MigrationRunner::MigrationRunner(MemoryStore& store, Config config)
+    : store_(store)
+    , config_(std::move(config))
+{
+    // Ensure migrations collection exists
+    store_.createCollection(MIGRATIONS_COLLECTION, CollectionOptions{});
+}
+
+bool MigrationRunner::runMigrations() {
+    if (!config_.autoApply) {
+        spdlog::info("Migrations auto-apply disabled, skipping");
+        return true;
+    }
+
+    if (!std::filesystem::exists(config_.directory)) {
+        spdlog::info("Migrations directory does not exist: {}", config_.directory.string());
+        return true;
+    }
+
+    auto files = getMigrationFiles();
+    if (files.empty()) {
+        spdlog::debug("No migration files found");
+        return true;
+    }
+
+    spdlog::info("Found {} migration files, checking for pending...", files.size());
+
+    int applied = 0;
+    int skipped = 0;
+    bool allSuccess = true;
+
+    for (const auto& file : files) {
+        // Read migration file
+        std::ifstream ifs(file);
+        if (!ifs) {
+            spdlog::error("Failed to read migration file: {}", file.string());
+            if (config_.failOnError) {
+                return false;
+            }
+            allSuccess = false;
+            continue;
+        }
+
+        nlohmann::json migration;
+        try {
+            ifs >> migration;
+        } catch (const std::exception& e) {
+            spdlog::error("Failed to parse migration file {}: {}", file.string(), e.what());
+            if (config_.failOnError) {
+                return false;
+            }
+            allSuccess = false;
+            continue;
+        }
+
+        std::string version = migration.value("version", "");
+        std::string name = migration.value("name", "");
+
+        if (version.empty() || name.empty()) {
+            spdlog::error("Migration file {} missing version or name", file.string());
+            if (config_.failOnError) {
+                return false;
+            }
+            allSuccess = false;
+            continue;
+        }
+
+        // Check if already applied
+        if (isMigrationApplied(version, name)) {
+            spdlog::debug("Migration {}/{} already applied, skipping", version, name);
+            ++skipped;
+            continue;
+        }
+
+        // Apply migration
+        spdlog::info("Applying migration {}/{}: {}",
+                     version, name,
+                     migration.value("description", ""));
+
+        if (!applyMigration(file)) {
+            spdlog::error("Failed to apply migration {}/{}", version, name);
+            if (config_.failOnError) {
+                return false;
+            }
+            allSuccess = false;
+            continue;
+        }
+
+        ++applied;
+    }
+
+    spdlog::info("Migrations complete: {} applied, {} skipped", applied, skipped);
+    return allSuccess;
+}
+
+std::vector<std::string> MigrationRunner::pendingMigrations() const {
+    std::vector<std::string> pending;
+
+    auto files = getMigrationFiles();
+    for (const auto& file : files) {
+        std::ifstream ifs(file);
+        if (!ifs) continue;
+
+        nlohmann::json migration;
+        try {
+            ifs >> migration;
+        } catch (...) {
+            continue;
+        }
+
+        std::string version = migration.value("version", "");
+        std::string name = migration.value("name", "");
+
+        if (!version.empty() && !name.empty() && !isMigrationApplied(version, name)) {
+            pending.push_back(version + "_" + name);
+        }
+    }
+
+    return pending;
+}
+
+std::vector<std::string> MigrationRunner::appliedMigrations() const {
+    std::vector<std::string> applied;
+
+    Query query;
+    query.limit = 1000;
+    auto result = store_.find(MIGRATIONS_COLLECTION, query);
+
+    for (const auto& doc : result.documents) {
+        applied.push_back(doc.id);
+    }
+
+    return applied;
+}
+
+bool MigrationRunner::applyMigration(const std::filesystem::path& file) {
+    std::ifstream ifs(file);
+    if (!ifs) {
+        return false;
+    }
+
+    std::stringstream buffer;
+    buffer << ifs.rdbuf();
+    std::string content = buffer.str();
+
+    nlohmann::json migration;
+    try {
+        migration = nlohmann::json::parse(content);
+    } catch (const std::exception& e) {
+        spdlog::error("Failed to parse migration: {}", e.what());
+        return false;
+    }
+
+    std::string version = migration.value("version", "");
+    std::string name = migration.value("name", "");
+
+    if (!migration.contains("operations") || !migration["operations"].is_array()) {
+        spdlog::error("Migration {} has no operations array", file.string());
+        return false;
+    }
+
+    // Apply each operation
+    for (const auto& op : migration["operations"]) {
+        if (!applyOperation(op)) {
+            spdlog::error("Migration {}/{} failed at operation: {}",
+                         version, name, op.dump());
+            return false;
+        }
+    }
+
+    // Record migration as applied
+    recordMigration(version, name, calculateChecksum(content));
+
+    return true;
+}
+
+bool MigrationRunner::applyOperation(const nlohmann::json& operation) {
+    std::string type = operation.value("type", "");
+
+    if (type == "create_collection") {
+        std::string collection = operation.value("collection", "");
+        if (collection.empty()) {
+            spdlog::error("create_collection: missing collection name");
+            return false;
+        }
+
+        CollectionOptions options;
+        if (operation.contains("options")) {
+            auto& opts = operation["options"];
+            options.encrypted = opts.value("encrypted", false);
+            if (opts.contains("sensitive_fields")) {
+                for (const auto& field : opts["sensitive_fields"]) {
+                    options.sensitiveFields.push_back(field.get<std::string>());
+                }
+            }
+            options.defaultTtlSeconds = opts.value("default_ttl_seconds", 0U);
+        }
+
+        bool created = store_.createCollection(collection, options);
+        spdlog::debug("create_collection '{}': {}", collection, created ? "created" : "already exists");
+        return true;  // Not an error if already exists
+    }
+
+    if (type == "drop_collection") {
+        std::string collection = operation.value("collection", "");
+        bool confirm = operation.value("confirm", false);
+
+        if (collection.empty()) {
+            spdlog::error("drop_collection: missing collection name");
+            return false;
+        }
+        if (!confirm) {
+            spdlog::error("drop_collection: requires confirm: true for safety");
+            return false;
+        }
+
+        store_.dropCollection(collection);
+        spdlog::debug("drop_collection '{}': dropped", collection);
+        return true;
+    }
+
+    if (type == "insert") {
+        std::string collection = operation.value("collection", "");
+        std::string id = operation.value("id", "");
+
+        if (collection.empty() || !operation.contains("data")) {
+            spdlog::error("insert: missing collection or data");
+            return false;
+        }
+
+        Document doc;
+        doc.id = id;
+        doc.data = operation["data"];
+
+        try {
+            store_.insert(collection, doc);
+            spdlog::debug("insert into '{}': {}", collection, doc.id);
+        } catch (const std::exception& e) {
+            spdlog::error("insert failed: {}", e.what());
+            return false;
+        }
+        return true;
+    }
+
+    if (type == "insert_if_not_exists") {
+        std::string collection = operation.value("collection", "");
+        std::string id = operation.value("id", "");
+
+        if (collection.empty() || id.empty() || !operation.contains("data")) {
+            spdlog::error("insert_if_not_exists: missing collection, id, or data");
+            return false;
+        }
+
+        // Check if document exists
+        if (store_.exists(collection, id)) {
+            spdlog::debug("insert_if_not_exists '{}': already exists, skipping", id);
+            return true;
+        }
+
+        Document doc;
+        doc.id = id;
+        doc.data = operation["data"];
+
+        try {
+            store_.insert(collection, doc);
+            spdlog::debug("insert_if_not_exists into '{}': {}", collection, id);
+        } catch (const std::exception& e) {
+            spdlog::error("insert_if_not_exists failed: {}", e.what());
+            return false;
+        }
+        return true;
+    }
+
+    if (type == "upsert") {
+        std::string collection = operation.value("collection", "");
+        std::string id = operation.value("id", "");
+
+        if (collection.empty() || !operation.contains("data")) {
+            spdlog::error("upsert: missing collection or data");
+            return false;
+        }
+
+        Document doc;
+        doc.id = id;
+        doc.data = operation["data"];
+
+        store_.upsert(collection, doc);
+        spdlog::debug("upsert into '{}': {}", collection, doc.id);
+        return true;
+    }
+
+    if (type == "update") {
+        std::string collection = operation.value("collection", "");
+        std::string id = operation.value("id", "");
+
+        if (collection.empty() || id.empty() || !operation.contains("data")) {
+            spdlog::error("update: missing collection, id, or data");
+            return false;
+        }
+
+        Document doc;
+        doc.id = id;
+        doc.data = operation["data"];
+
+        bool updated = store_.update(collection, id, doc);
+        spdlog::debug("update '{}': {} ({})", id, updated ? "updated" : "not found",
+                     updated ? "success" : "failed");
+        return updated;
+    }
+
+    if (type == "delete") {
+        std::string collection = operation.value("collection", "");
+        std::string id = operation.value("id", "");
+
+        if (collection.empty() || id.empty()) {
+            spdlog::error("delete: missing collection or id");
+            return false;
+        }
+
+        store_.remove(collection, id);
+        spdlog::debug("delete from '{}': {}", collection, id);
+        return true;
+    }
+
+    spdlog::error("Unknown operation type: {}", type);
+    return false;
+}
+
+bool MigrationRunner::isMigrationApplied(const std::string& version, const std::string& name) const {
+    std::string id = version + "_" + name;
+    return store_.exists(MIGRATIONS_COLLECTION, id);
+}
+
+void MigrationRunner::recordMigration(const std::string& version, const std::string& name, const std::string& checksum) {
+    std::string id = version + "_" + name;
+
+    Document doc;
+    doc.id = id;
+    doc.data = {
+        {"version", version},
+        {"name", name},
+        {"applied_at", std::chrono::duration_cast<std::chrono::milliseconds>(
+            std::chrono::system_clock::now().time_since_epoch()).count()},
+        {"status", "applied"},
+        {"checksum", checksum}
+    };
+
+    store_.insert(MIGRATIONS_COLLECTION, doc);
+}
+
+std::string MigrationRunner::calculateChecksum(const std::string& content) {
+    // Simple hash for now - could use SHA256 if crypto is available
+    std::hash<std::string> hasher;
+    auto hash = hasher(content);
+
+    std::stringstream ss;
+    ss << "hash:" << std::hex << hash;
+    return ss.str();
+}
+
+std::vector<std::filesystem::path> MigrationRunner::getMigrationFiles() const {
+    std::vector<std::filesystem::path> files;
+
+    if (!std::filesystem::exists(config_.directory)) {
+        return files;
+    }
+
+    for (const auto& entry : std::filesystem::directory_iterator(config_.directory)) {
+        if (entry.is_regular_file()) {
+            auto ext = entry.path().extension();
+            if (ext == ".json") {
+                files.push_back(entry.path());
+            }
+        }
+    }
+
+    // Sort by filename (assumes 001_name.json format)
+    std::sort(files.begin(), files.end());
+
+    return files;
+}
+
+} // namespace smartbotic::database

service/src/migrations/migration_runner.hpp

@@ -0,0 +1,97 @@
+#pragma once
+
+#include "../memory_store.hpp"
+
+#include <filesystem>
+#include <string>
+#include <vector>
+
+namespace smartbotic::database {
+
+/**
+ * Migration runner that applies JSON-based migrations to the database.
+ *
+ * Migrations are JSON files in the migrations directory with this format:
+ * {
+ *   "version": "001",
+ *   "name": "create_users",
+ *   "description": "Create users collection",
+ *   "operations": [
+ *     {"type": "create_collection", "collection": "users", "options": {...}},
+ *     {"type": "insert_if_not_exists", "collection": "settings", "id": "key", "data": {...}}
+ *   ]
+ * }
+ *
+ * Supported operations:
+ * - create_collection: Create collection if not exists
+ * - drop_collection: Drop collection (requires confirm: true)
+ * - insert: Insert document (fails if exists)
+ * - insert_if_not_exists: Insert only if ID doesn't exist
+ * - upsert: Insert or update document
+ * - update: Update existing document
+ * - delete: Delete document
+ */
+class MigrationRunner {
+public:
+    struct Config {
+        std::filesystem::path directory;
+        bool autoApply = true;
+        bool failOnError = true;
+    };
+
+    explicit MigrationRunner(MemoryStore& store, Config config);
+
+    /**
+     * Run all pending migrations.
+     * @return true if all migrations succeeded
+     */
+    bool runMigrations();
+
+    /**
+     * Get list of pending migrations (not yet applied).
+     */
+    std::vector<std::string> pendingMigrations() const;
+
+    /**
+     * Get list of applied migrations.
+     */
+    std::vector<std::string> appliedMigrations() const;
+
+private:
+    /**
+     * Apply a single migration file.
+     */
+    bool applyMigration(const std::filesystem::path& file);
+
+    /**
+     * Check if a migration has already been applied.
+     */
+    bool isMigrationApplied(const std::string& version, const std::string& name) const;
+
+    /**
+     * Record a migration as applied.
+     */
+    void recordMigration(const std::string& version, const std::string& name, const std::string& checksum);
+
+    /**
+     * Apply a single operation within a migration.
+     */
+    bool applyOperation(const nlohmann::json& operation);
+
+    /**
+     * Calculate checksum for migration content.
+     */
+    static std::string calculateChecksum(const std::string& content);
+
+    /**
+     * Get sorted list of migration files in directory.
+     */
+    std::vector<std::filesystem::path> getMigrationFiles() const;
+
+    MemoryStore& store_;
+    Config config_;
+
+    static constexpr const char* MIGRATIONS_COLLECTION = "_migrations";
+};
+
+} // namespace smartbotic::database

service/src/persistence/persistence_manager.cpp

@@ -0,0 +1,365 @@
+#include "persistence_manager.hpp"
+
+#include <spdlog/spdlog.h>
+
+namespace smartbotic::database {
+
+PersistenceManager::PersistenceManager(Config config)
+    : config_(std::move(config)) {
+
+    // Create data directory if it doesn't exist
+    std::error_code ec;
+    std::filesystem::create_directories(config_.dataDir, ec);
+
+    // Initialize WAL
+    WriteAheadLog::Config walConfig;
+    walConfig.walDir = config_.dataDir / "wal";
+    walConfig.syncIntervalMs = config_.walSyncIntervalMs;
+    walConfig.maxFileSizeBytes = config_.maxWalSizeMb * 1024 * 1024;
+    wal_ = std::make_unique<WriteAheadLog>(walConfig);
+
+    // Initialize snapshot manager
+    SnapshotManager::Config snapConfig;
+    snapConfig.snapshotDir = config_.dataDir / "snapshots";
+    snapConfig.compressionEnabled = config_.compressionEnabled;
+    snapConfig.maxSnapshots = config_.maxSnapshots;
+    snapshotMgr_ = std::make_unique<SnapshotManager>(snapConfig);
+}
+
+PersistenceManager::~PersistenceManager() {
+    stop();
+}
+
+bool PersistenceManager::start() {
+    if (running_.exchange(true)) {
+        return true;  // Already running
+    }
+
+    // Open WAL
+    if (!wal_->open()) {
+        spdlog::error("Failed to open WAL");
+        running_.store(false);
+        return false;
+    }
+
+    lastSnapshotTime_ = std::chrono::steady_clock::now();
+
+    // Start background threads
+    walSyncThread_ = std::thread(&PersistenceManager::walSyncLoop, this);
+    snapshotThread_ = std::thread(&PersistenceManager::snapshotLoop, this);
+
+    spdlog::info("Persistence manager started, WAL sequence: {}", wal_->currentSequence());
+    return true;
+}
+
+void PersistenceManager::stop() {
+    if (!running_.exchange(false)) {
+        return;  // Already stopped
+    }
+
+    // Signal snapshot thread to stop
+    {
+        std::lock_guard<std::mutex> lock(snapshotMutex_);
+        snapshotCv_.notify_all();
+    }
+
+    // Wait for threads to finish
+    if (walSyncThread_.joinable()) {
+        walSyncThread_.join();
+    }
+    if (snapshotThread_.joinable()) {
+        snapshotThread_.join();
+    }
+
+    // Final sync and close WAL
+    if (wal_) {
+        wal_->sync();
+        wal_->close();
+    }
+
+    spdlog::info("Persistence manager stopped");
+}
+
+bool PersistenceManager::recover(MemoryStore& store) {
+    spdlog::info("Starting recovery...");
+
+    // Store reference for potential snapshot during recovery
+    {
+        std::lock_guard<std::mutex> lock(storeMutex_);
+        currentStore_ = &store;
+    }
+
+    try {
+        // Load latest snapshot
+        uint64_t snapshotSequence = snapshotMgr_->loadLatestSnapshot(store);
+        spdlog::info("Loaded snapshot at sequence {}", snapshotSequence);
+
+        // Open WAL for replay
+        if (!wal_->open()) {
+            spdlog::error("Failed to open WAL for recovery");
+            return false;
+        }
+
+        // Replay WAL entries after snapshot
+        uint64_t replayedCount = wal_->replay(snapshotSequence, [&store](const WalEntry& entry) {
+            switch (entry.opType) {
+                case WalOpType::INSERT:
+                case WalOpType::UPDATE:
+                case WalOpType::UPSERT:
+                    if (entry.data) {
+                        Document doc = Document::fromJson(*entry.data);
+                        store.loadDocument(entry.collection, doc);
+                    }
+                    break;
+
+                case WalOpType::DELETE:
+                    store.remove(entry.collection, entry.documentId);
+                    break;
+
+                case WalOpType::CREATE_COLLECTION:
+                    if (entry.collectionOptions) {
+                        store.createCollection(entry.collection, *entry.collectionOptions);
+                    }
+                    break;
+
+                case WalOpType::DROP_COLLECTION:
+                    store.dropCollection(entry.collection);
+                    break;
+
+                case WalOpType::SET_ADD:
+                    if (entry.data && entry.data->contains("member")) {
+                        store.setAdd(entry.collection, entry.documentId,
+                                    (*entry.data)["member"].get<std::string>());
+                    }
+                    break;
+
+                case WalOpType::SET_REMOVE:
+                    if (entry.data && entry.data->contains("member")) {
+                        store.setRemove(entry.collection, entry.documentId,
+                                       (*entry.data)["member"].get<std::string>());
+                    }
+                    break;
+            }
+        });
+
+        spdlog::info("Replayed {} WAL entries", replayedCount);
+
+        // Update last snapshot sequence
+        {
+            std::lock_guard<std::mutex> lock(statsMutex_);
+            lastSnapshotSequence_ = snapshotSequence;
+        }
+
+        wal_->close();  // Close for now, will be reopened by start()
+
+        spdlog::info("Recovery complete, store has {} documents in {} collections",
+                     store.getStats().totalDocuments, store.getStats().totalCollections);
+
+        return true;
+
+    } catch (const std::exception& e) {
+        spdlog::error("Recovery failed: {}", e.what());
+        return false;
+    }
+}
+
+void PersistenceManager::logInsert(const std::string& collection, const Document& doc) {
+    if (!running_.load()) return;
+    uint64_t seq = wal_->append(WriteAheadLog::makeInsertEntry(collection, doc));
+    updateCollectionSequence(collection, seq);
+}
+
+void PersistenceManager::logUpdate(const std::string& collection, const Document& doc) {
+    if (!running_.load()) return;
+    uint64_t seq = wal_->append(WriteAheadLog::makeUpdateEntry(collection, doc));
+    updateCollectionSequence(collection, seq);
+}
+
+void PersistenceManager::logDelete(const std::string& collection, const std::string& id) {
+    if (!running_.load()) return;
+    uint64_t seq = wal_->append(WriteAheadLog::makeDeleteEntry(collection, id));
+    updateCollectionSequence(collection, seq);
+}
+
+void PersistenceManager::logUpsert(const std::string& collection, const Document& doc) {
+    if (!running_.load()) return;
+    uint64_t seq = wal_->append(WriteAheadLog::makeUpsertEntry(collection, doc));
+    updateCollectionSequence(collection, seq);
+}
+
+void PersistenceManager::logCreateCollection(const std::string& collection, const CollectionOptions& options) {
+    if (!running_.load()) return;
+    uint64_t seq = wal_->append(WriteAheadLog::makeCreateCollectionEntry(collection, options));
+    updateCollectionSequence(collection, seq);
+}
+
+void PersistenceManager::logDropCollection(const std::string& collection) {
+    if (!running_.load()) return;
+    wal_->append(WriteAheadLog::makeDropCollectionEntry(collection));
+    // Remove collection from tracking
+    std::lock_guard<std::mutex> lock(collectionSeqMutex_);
+    collectionSequences_.erase(collection);
+}
+
+void PersistenceManager::forceSnapshot(const MemoryStore& store) {
+    {
+        std::lock_guard<std::mutex> lock(storeMutex_);
+        currentStore_ = &store;
+    }
+
+    createSnapshot();
+}
+
+uint64_t PersistenceManager::currentWalSequence() const {
+    return wal_ ? wal_->currentSequence() : 0;
+}
+
+uint64_t PersistenceManager::walSizeBytes() const {
+    return wal_ ? wal_->totalSizeBytes() : 0;
+}
+
+PersistenceManager::Stats PersistenceManager::getStats() const {
+    Stats stats;
+    stats.walSequence = currentWalSequence();
+    stats.walSizeBytes = walSizeBytes();
+
+    auto snapshots = snapshotMgr_->listSnapshots();
+    stats.snapshotCount = snapshots.size();
+
+    {
+        std::lock_guard<std::mutex> lock(statsMutex_);
+        stats.lastSnapshotSequence = lastSnapshotSequence_;
+        stats.walEntriesSinceSnapshot = stats.walSequence - lastSnapshotSequence_;
+    }
+
+    if (!snapshots.empty()) {
+        auto info = snapshotMgr_->getSnapshotInfo(snapshots.front());
+        if (info) {
+            stats.lastSnapshotTimestamp = info->timestamp;
+        }
+    }
+
+    return stats;
+}
+
+void PersistenceManager::setSnapshotCallback(SnapshotCallback callback) {
+    snapshotCallback_ = std::move(callback);
+}
+
+uint64_t PersistenceManager::replayWal(uint64_t fromSequence, uint32_t limit,
+                                        std::function<void(const WalEntry&)> callback) {
+    if (!wal_) {
+        return 0;
+    }
+
+    uint64_t count = 0;
+    wal_->replay(fromSequence, [&count, limit, &callback](const WalEntry& entry) {
+        if (count < limit) {
+            callback(entry);
+            ++count;
+        }
+    });
+
+    return count;
+}
+
+void PersistenceManager::walSyncLoop() {
+    while (running_.load()) {
+        std::this_thread::sleep_for(std::chrono::milliseconds(config_.walSyncIntervalMs));
+
+        if (running_.load() && wal_) {
+            wal_->sync();
+        }
+    }
+}
+
+void PersistenceManager::snapshotLoop() {
+    while (running_.load()) {
+        std::unique_lock<std::mutex> lock(snapshotMutex_);
+
+        // Wait for snapshot interval or explicit request
+        auto waitUntil = std::chrono::steady_clock::now() +
+                         std::chrono::seconds(config_.snapshotIntervalSec);
+
+        snapshotCv_.wait_until(lock, waitUntil, [this] {
+            return !running_.load() || snapshotRequested_ || shouldSnapshot();
+        });
+
+        if (!running_.load()) {
+            break;
+        }
+
+        snapshotRequested_ = false;
+        lock.unlock();
+
+        if (shouldSnapshot()) {
+            createSnapshot();
+        }
+    }
+}
+
+void PersistenceManager::createSnapshot() {
+    const MemoryStore* store = nullptr;
+    {
+        std::lock_guard<std::mutex> lock(storeMutex_);
+        store = currentStore_;
+    }
+
+    if (!store) {
+        spdlog::warn("Cannot create snapshot: no store reference");
+        return;
+    }
+
+    try {
+        uint64_t walSeq = wal_->currentSequence();
+        auto snapshotPath = snapshotMgr_->createSnapshot(*store, walSeq);
+
+        {
+            std::lock_guard<std::mutex> lock(statsMutex_);
+            lastSnapshotSequence_ = walSeq;
+            lastSnapshotTime_ = std::chrono::steady_clock::now();
+        }
+
+        // Truncate old WAL entries
+        wal_->truncateBefore(walSeq);
+
+        spdlog::info("Created snapshot at sequence {}: {}",
+                     walSeq, snapshotPath.string());
+
+        if (snapshotCallback_) {
+            snapshotCallback_(snapshotPath);
+        }
+
+    } catch (const std::exception& e) {
+        spdlog::error("Failed to create snapshot: {}", e.what());
+    }
+}
+
+bool PersistenceManager::shouldSnapshot() const {
+    // Check WAL size
+    uint64_t walSize = walSizeBytes();
+    if (walSize >= config_.maxWalSizeMb * 1024 * 1024) {
+        return true;
+    }
+
+    // Check time since last snapshot
+    auto now = std::chrono::steady_clock::now();
+    auto elapsed = std::chrono::duration_cast<std::chrono::seconds>(now - lastSnapshotTime_).count();
+    if (elapsed >= static_cast<int64_t>(config_.snapshotIntervalSec)) {
+        return true;
+    }
+
+    return false;
+}
+
+void PersistenceManager::updateCollectionSequence(const std::string& collection, uint64_t sequence) {
+    std::lock_guard<std::mutex> lock(collectionSeqMutex_);
+    collectionSequences_[collection] = sequence;
+}
+
+std::unordered_map<std::string, uint64_t> PersistenceManager::getCollectionSequences() const {
+    std::lock_guard<std::mutex> lock(collectionSeqMutex_);
+    return collectionSequences_;
+}
+
+} // namespace smartbotic::database

service/src/persistence/persistence_manager.hpp

@@ -0,0 +1,208 @@
+#pragma once
+
+#include "wal.hpp"
+#include "snapshot.hpp"
+#include "../memory_store.hpp"
+
+#include <atomic>
+#include <chrono>
+#include <condition_variable>
+#include <filesystem>
+#include <functional>
+#include <mutex>
+#include <thread>
+#include <unordered_map>
+
+namespace smartbotic::database {
+
+/**
+ * Persistence manager coordinates WAL and snapshots for durable storage.
+ *
+ * Responsibilities:
+ * - Log all mutations to WAL before applying to memory
+ * - Periodically create snapshots
+ * - Recover state on startup (load snapshot + replay WAL)
+ * - Ensure all data is persisted on shutdown
+ */
+class PersistenceManager {
+public:
+    struct Config {
+        std::filesystem::path dataDir;
+        uint32_t walSyncIntervalMs = 100;      // fsync WAL every 100ms
+        uint32_t snapshotIntervalSec = 3600;   // Snapshot every hour
+        uint64_t maxWalSizeMb = 100;           // Trigger snapshot at this WAL size
+        bool compressionEnabled = true;
+        uint32_t maxSnapshots = 5;
+    };
+
+    explicit PersistenceManager(Config config);
+    ~PersistenceManager();
+
+    // Non-copyable
+    PersistenceManager(const PersistenceManager&) = delete;
+    PersistenceManager& operator=(const PersistenceManager&) = delete;
+
+    /**
+     * Start the persistence manager.
+     * Opens WAL and starts background threads.
+     */
+    bool start();
+
+    /**
+     * Stop the persistence manager.
+     * Ensures all data is persisted before returning.
+     */
+    void stop();
+
+    /**
+     * Check if running.
+     */
+    [[nodiscard]] bool isRunning() const { return running_.load(); }
+
+    /**
+     * Recover state into the memory store.
+     * Should be called before start() to load existing data.
+     * @return true if recovery succeeded (even if no data to recover)
+     */
+    bool recover(MemoryStore& store);
+
+    /**
+     * Log an insert operation.
+     */
+    void logInsert(const std::string& collection, const Document& doc);
+
+    /**
+     * Log an update operation.
+     */
+    void logUpdate(const std::string& collection, const Document& doc);
+
+    /**
+     * Log a delete operation.
+     */
+    void logDelete(const std::string& collection, const std::string& id);
+
+    /**
+     * Log an upsert operation.
+     */
+    void logUpsert(const std::string& collection, const Document& doc);
+
+    /**
+     * Log collection creation.
+     */
+    void logCreateCollection(const std::string& collection, const CollectionOptions& options);
+
+    /**
+     * Log collection drop.
+     */
+    void logDropCollection(const std::string& collection);
+
+    /**
+     * Force an immediate snapshot.
+     * Blocks until snapshot is complete.
+     */
+    void forceSnapshot(const MemoryStore& store);
+
+    /**
+     * Get current WAL sequence.
+     */
+    [[nodiscard]] uint64_t currentWalSequence() const;
+
+    /**
+     * Get total WAL size in bytes.
+     */
+    [[nodiscard]] uint64_t walSizeBytes() const;
+
+    /**
+     * Get persistence statistics.
+     */
+    struct Stats {
+        uint64_t walSequence = 0;
+        uint64_t walSizeBytes = 0;
+        uint64_t snapshotCount = 0;
+        uint64_t lastSnapshotSequence = 0;
+        uint64_t lastSnapshotTimestamp = 0;
+        uint64_t walEntriesSinceSnapshot = 0;
+    };
+    [[nodiscard]] Stats getStats() const;
+
+    /**
+     * Set callback for snapshot completion.
+     */
+    using SnapshotCallback = std::function<void(const std::filesystem::path&)>;
+    void setSnapshotCallback(SnapshotCallback callback);
+
+    /**
+     * Replay WAL entries for replication.
+     * @param fromSequence Start replaying from this sequence (exclusive)
+     * @param limit Maximum number of entries to replay
+     * @param callback Called for each entry
+     * @return Number of entries replayed
+     */
+    uint64_t replayWal(uint64_t fromSequence, uint32_t limit,
+                       std::function<void(const WalEntry&)> callback);
+
+    /**
+     * Get the last sequence number for each collection.
+     * Used for per-collection sync tracking.
+     */
+    [[nodiscard]] std::unordered_map<std::string, uint64_t> getCollectionSequences() const;
+
+private:
+    /**
+     * Update collection sequence tracking.
+     */
+    void updateCollectionSequence(const std::string& collection, uint64_t sequence);
+    /**
+     * WAL sync thread loop.
+     */
+    void walSyncLoop();
+
+    /**
+     * Snapshot thread loop.
+     */
+    void snapshotLoop();
+
+    /**
+     * Create a snapshot (called by snapshotLoop or forceSnapshot).
+     */
+    void createSnapshot();
+
+    /**
+     * Check if snapshot is needed based on WAL size or time.
+     */
+    [[nodiscard]] bool shouldSnapshot() const;
+
+    Config config_;
+    std::atomic<bool> running_{false};
+
+    // WAL and snapshot managers
+    std::unique_ptr<WriteAheadLog> wal_;
+    std::unique_ptr<SnapshotManager> snapshotMgr_;
+
+    // Reference to memory store (set during snapshot)
+    const MemoryStore* currentStore_ = nullptr;
+    mutable std::mutex storeMutex_;
+
+    // Background threads
+    std::thread walSyncThread_;
+    std::thread snapshotThread_;
+
+    // Snapshot control
+    std::mutex snapshotMutex_;
+    std::condition_variable snapshotCv_;
+    bool snapshotRequested_ = false;
+
+    // Statistics
+    mutable std::mutex statsMutex_;
+    uint64_t lastSnapshotSequence_ = 0;
+    std::chrono::steady_clock::time_point lastSnapshotTime_;
+
+    // Callbacks
+    SnapshotCallback snapshotCallback_;
+
+    // Per-collection sequence tracking
+    mutable std::mutex collectionSeqMutex_;
+    std::unordered_map<std::string, uint64_t> collectionSequences_;
+};
+
+} // namespace smartbotic::database

service/src/persistence/snapshot.cpp

@@ -0,0 +1,442 @@
+#include "snapshot.hpp"
+#include "wal.hpp"
+
+#include <algorithm>
+#include <chrono>
+#include <cstring>
+#include <fstream>
+#include <iomanip>
+#include <sstream>
+
+#ifndef DISABLE_LZ4
+#include <lz4.h>
+#endif
+
+namespace smartbotic::database {
+
+SnapshotManager::SnapshotManager(Config config)
+    : config_(std::move(config)) {
+    // Create snapshot directory if it doesn't exist
+    std::error_code ec;
+    std::filesystem::create_directories(config_.snapshotDir, ec);
+}
+
+std::filesystem::path SnapshotManager::createSnapshot(const MemoryStore& store, uint64_t walSequence) {
+    // Serialize store data
+    uint64_t docCount = 0;
+    uint32_t collCount = 0;
+    std::vector<uint8_t> uncompressedData = serializeStore(store, docCount, collCount);
+
+    // Compress if enabled
+    std::vector<uint8_t> bodyData;
+    uint32_t compressionType = 0;
+
+    if (config_.compressionEnabled) {
+        bodyData = compress(uncompressedData);
+        if (!bodyData.empty()) {
+            compressionType = 1;  // LZ4
+        } else {
+            bodyData = std::move(uncompressedData);
+        }
+    } else {
+        bodyData = std::move(uncompressedData);
+    }
+
+    // Build header
+    SnapshotHeader header;
+    header.timestamp = static_cast<uint64_t>(
+        std::chrono::duration_cast<std::chrono::milliseconds>(
+            std::chrono::system_clock::now().time_since_epoch()
+        ).count()
+    );
+    header.walSequence = walSequence;
+    header.documentCount = docCount;
+    header.collectionCount = collCount;
+    header.uncompressedSize = uncompressedData.size();
+    header.compressedSize = bodyData.size();
+    header.compressionType = compressionType;
+
+    // Calculate header checksum (excluding the checksum field itself)
+    header.headerChecksum = crc32::calculate(&header, sizeof(header) - sizeof(header.headerChecksum));
+
+    // Generate filename and write
+    std::filesystem::path snapshotPath = config_.snapshotDir / generateFilename();
+
+    std::ofstream file(snapshotPath, std::ios::binary);
+    if (!file) {
+        throw std::runtime_error("Failed to create snapshot file: " + snapshotPath.string());
+    }
+
+    // Write header
+    file.write(reinterpret_cast<const char*>(&header), sizeof(header));
+
+    // Write body
+    file.write(reinterpret_cast<const char*>(bodyData.data()),
+               static_cast<std::streamsize>(bodyData.size()));
+
+    // Write body checksum
+    uint32_t bodyChecksum = crc32::calculate(bodyData.data(), bodyData.size());
+    file.write(reinterpret_cast<const char*>(&bodyChecksum), sizeof(bodyChecksum));
+
+    file.close();
+
+    // Cleanup old snapshots
+    cleanupOldSnapshots();
+
+    return snapshotPath;
+}
+
+uint64_t SnapshotManager::loadLatestSnapshot(MemoryStore& store) {
+    auto snapshots = listSnapshots();
+    if (snapshots.empty()) {
+        return 0;
+    }
+
+    return loadSnapshot(snapshots.front(), store);
+}
+
+uint64_t SnapshotManager::loadSnapshot(const std::filesystem::path& path, MemoryStore& store) {
+    std::ifstream file(path, std::ios::binary);
+    if (!file) {
+        throw std::runtime_error("Failed to open snapshot file: " + path.string());
+    }
+
+    // Read header
+    SnapshotHeader header;
+    file.read(reinterpret_cast<char*>(&header), sizeof(header));
+    if (!file) {
+        throw std::runtime_error("Failed to read snapshot header");
+    }
+
+    // Verify magic
+    if (std::memcmp(header.magic, "CALSNAP1", 8) != 0) {
+        throw std::runtime_error("Invalid snapshot file magic");
+    }
+
+    // Verify header checksum
+    uint32_t expectedHeaderCrc = crc32::calculate(&header, sizeof(header) - sizeof(header.headerChecksum));
+    if (header.headerChecksum != expectedHeaderCrc) {
+        throw std::runtime_error("Snapshot header checksum mismatch");
+    }
+
+    // Read body
+    std::vector<uint8_t> bodyData(header.compressedSize);
+    file.read(reinterpret_cast<char*>(bodyData.data()),
+              static_cast<std::streamsize>(bodyData.size()));
+    if (!file) {
+        throw std::runtime_error("Failed to read snapshot body");
+    }
+
+    // Read and verify body checksum
+    uint32_t storedBodyCrc;
+    file.read(reinterpret_cast<char*>(&storedBodyCrc), sizeof(storedBodyCrc));
+    uint32_t calculatedBodyCrc = crc32::calculate(bodyData.data(), bodyData.size());
+    if (storedBodyCrc != calculatedBodyCrc) {
+        throw std::runtime_error("Snapshot body checksum mismatch");
+    }
+
+    // Decompress if needed
+    std::vector<uint8_t> uncompressedData;
+    if (header.compressionType == 1) {
+        uncompressedData = decompress(bodyData, header.uncompressedSize);
+        if (uncompressedData.empty()) {
+            throw std::runtime_error("Failed to decompress snapshot");
+        }
+    } else {
+        uncompressedData = std::move(bodyData);
+    }
+
+    // Clear store and deserialize
+    store.clear();
+    deserializeStore(uncompressedData, store);
+
+    return header.walSequence;
+}
+
+std::vector<std::filesystem::path> SnapshotManager::listSnapshots() const {
+    std::vector<std::pair<uint64_t, std::filesystem::path>> snapshots;
+
+    std::error_code ec;
+    for (const auto& entry : std::filesystem::directory_iterator(config_.snapshotDir, ec)) {
+        if (entry.is_regular_file()) {
+            const auto& path = entry.path();
+            if (path.filename().string().starts_with("snapshot-") &&
+                path.extension() == ".dat") {
+                // Read header to get timestamp
+                std::ifstream file(path, std::ios::binary);
+                if (file) {
+                    SnapshotHeader header;
+                    file.read(reinterpret_cast<char*>(&header), sizeof(header));
+                    if (file && std::memcmp(header.magic, "CALSNAP1", 8) == 0) {
+                        snapshots.emplace_back(header.timestamp, path);
+                    }
+                }
+            }
+        }
+    }
+
+    // Sort by timestamp descending (newest first)
+    std::sort(snapshots.begin(), snapshots.end(),
+              [](const auto& a, const auto& b) { return a.first > b.first; });
+
+    std::vector<std::filesystem::path> result;
+    result.reserve(snapshots.size());
+    for (const auto& [_, path] : snapshots) {
+        result.push_back(path);
+    }
+
+    return result;
+}
+
+std::optional<SnapshotHeader> SnapshotManager::getSnapshotInfo(const std::filesystem::path& path) const {
+    std::ifstream file(path, std::ios::binary);
+    if (!file) {
+        return std::nullopt;
+    }
+
+    SnapshotHeader header;
+    file.read(reinterpret_cast<char*>(&header), sizeof(header));
+    if (!file) {
+        return std::nullopt;
+    }
+
+    if (std::memcmp(header.magic, "CALSNAP1", 8) != 0) {
+        return std::nullopt;
+    }
+
+    return header;
+}
+
+void SnapshotManager::cleanupOldSnapshots() {
+    auto snapshots = listSnapshots();
+
+    // Delete snapshots beyond the max count
+    while (snapshots.size() > config_.maxSnapshots) {
+        const auto& oldestPath = snapshots.back();
+        std::error_code ec;
+        std::filesystem::remove(oldestPath, ec);
+        snapshots.pop_back();
+    }
+}
+
+bool SnapshotManager::verifySnapshot(const std::filesystem::path& path) const {
+    try {
+        std::ifstream file(path, std::ios::binary);
+        if (!file) {
+            return false;
+        }
+
+        // Read and verify header
+        SnapshotHeader header;
+        file.read(reinterpret_cast<char*>(&header), sizeof(header));
+        if (!file) {
+            return false;
+        }
+
+        if (std::memcmp(header.magic, "CALSNAP1", 8) != 0) {
+            return false;
+        }
+
+        uint32_t expectedHeaderCrc = crc32::calculate(&header, sizeof(header) - sizeof(header.headerChecksum));
+        if (header.headerChecksum != expectedHeaderCrc) {
+            return false;
+        }
+
+        // Read and verify body
+        std::vector<uint8_t> bodyData(header.compressedSize);
+        file.read(reinterpret_cast<char*>(bodyData.data()),
+                  static_cast<std::streamsize>(bodyData.size()));
+        if (!file) {
+            return false;
+        }
+
+        uint32_t storedBodyCrc;
+        file.read(reinterpret_cast<char*>(&storedBodyCrc), sizeof(storedBodyCrc));
+        uint32_t calculatedBodyCrc = crc32::calculate(bodyData.data(), bodyData.size());
+
+        return storedBodyCrc == calculatedBodyCrc;
+    } catch (...) {
+        return false;
+    }
+}
+
+std::string SnapshotManager::generateFilename() const {
+    auto now = std::chrono::system_clock::now();
+    auto time = std::chrono::system_clock::to_time_t(now);
+    auto tm = *std::localtime(&time);
+
+    std::ostringstream oss;
+    oss << "snapshot-"
+        << std::put_time(&tm, "%Y%m%d-%H%M%S")
+        << ".dat";
+
+    return oss.str();
+}
+
+std::vector<uint8_t> SnapshotManager::compress(const std::vector<uint8_t>& data) const {
+#ifndef DISABLE_LZ4
+    if (data.empty()) {
+        return {};
+    }
+
+    int maxCompressedSize = LZ4_compressBound(static_cast<int>(data.size()));
+    std::vector<uint8_t> compressed(maxCompressedSize);
+
+    int compressedSize = LZ4_compress_default(
+        reinterpret_cast<const char*>(data.data()),
+        reinterpret_cast<char*>(compressed.data()),
+        static_cast<int>(data.size()),
+        maxCompressedSize
+    );
+
+    if (compressedSize <= 0) {
+        return {};  // Compression failed
+    }
+
+    compressed.resize(compressedSize);
+    return compressed;
+#else
+    return {};  // Compression disabled
+#endif
+}
+
+std::vector<uint8_t> SnapshotManager::decompress(const std::vector<uint8_t>& data,
+                                                  uint64_t uncompressedSize) const {
+#ifndef DISABLE_LZ4
+    if (data.empty() || uncompressedSize == 0) {
+        return {};
+    }
+
+    std::vector<uint8_t> decompressed(uncompressedSize);
+
+    int decompressedSize = LZ4_decompress_safe(
+        reinterpret_cast<const char*>(data.data()),
+        reinterpret_cast<char*>(decompressed.data()),
+        static_cast<int>(data.size()),
+        static_cast<int>(uncompressedSize)
+    );
+
+    if (decompressedSize < 0 || static_cast<uint64_t>(decompressedSize) != uncompressedSize) {
+        return {};  // Decompression failed
+    }
+
+    return decompressed;
+#else
+    return {};  // Compression disabled
+#endif
+}
+
+std::vector<uint8_t> SnapshotManager::serializeStore(const MemoryStore& store,
+                                                      uint64_t& docCount,
+                                                      uint32_t& collCount) const {
+    std::vector<uint8_t> result;
+    docCount = 0;
+    collCount = 0;
+
+    auto collections = store.getAllCollectionsWithOptions();
+    collCount = static_cast<uint32_t>(collections.size());
+
+    for (const auto& [name, options] : collections) {
+        // Write collection name
+        uint16_t nameLen = static_cast<uint16_t>(name.size());
+        result.push_back(static_cast<uint8_t>(nameLen & 0xFF));
+        result.push_back(static_cast<uint8_t>((nameLen >> 8) & 0xFF));
+        result.insert(result.end(), name.begin(), name.end());
+
+        // Write collection options
+        std::string optionsJson = options.toJson().dump();
+        uint32_t optionsLen = static_cast<uint32_t>(optionsJson.size());
+        for (int i = 0; i < 4; ++i) {
+            result.push_back(static_cast<uint8_t>((optionsLen >> (i * 8)) & 0xFF));
+        }
+        result.insert(result.end(), optionsJson.begin(), optionsJson.end());
+
+        // Get all documents in collection
+        auto docs = store.getAllDocuments(name);
+        uint64_t collDocCount = docs.size();
+        docCount += collDocCount;
+
+        // Write document count
+        for (int i = 0; i < 8; ++i) {
+            result.push_back(static_cast<uint8_t>((collDocCount >> (i * 8)) & 0xFF));
+        }
+
+        // Write each document
+        for (const auto& doc : docs) {
+            std::string docJson = doc.toJson().dump();
+            uint32_t docLen = static_cast<uint32_t>(docJson.size());
+            for (int i = 0; i < 4; ++i) {
+                result.push_back(static_cast<uint8_t>((docLen >> (i * 8)) & 0xFF));
+            }
+            result.insert(result.end(), docJson.begin(), docJson.end());
+        }
+    }
+
+    return result;
+}
+
+void SnapshotManager::deserializeStore(const std::vector<uint8_t>& data, MemoryStore& store) const {
+    size_t offset = 0;
+
+    while (offset < data.size()) {
+        // Read collection name
+        if (offset + 2 > data.size()) break;
+        uint16_t nameLen = static_cast<uint16_t>(data[offset]) |
+                          (static_cast<uint16_t>(data[offset + 1]) << 8);
+        offset += 2;
+
+        if (offset + nameLen > data.size()) break;
+        std::string collName(reinterpret_cast<const char*>(&data[offset]), nameLen);
+        offset += nameLen;
+
+        // Read collection options
+        if (offset + 4 > data.size()) break;
+        uint32_t optionsLen = 0;
+        for (int i = 0; i < 4; ++i) {
+            optionsLen |= static_cast<uint32_t>(data[offset++]) << (i * 8);
+        }
+
+        if (offset + optionsLen > data.size()) break;
+        std::string optionsJson(reinterpret_cast<const char*>(&data[offset]), optionsLen);
+        offset += optionsLen;
+
+        CollectionOptions options;
+        try {
+            options = CollectionOptions::fromJson(nlohmann::json::parse(optionsJson));
+        } catch (const nlohmann::json::exception&) {
+            // Use default options
+        }
+
+        // Create collection
+        store.createCollection(collName, options);
+
+        // Read document count
+        if (offset + 8 > data.size()) break;
+        uint64_t docCount = 0;
+        for (int i = 0; i < 8; ++i) {
+            docCount |= static_cast<uint64_t>(data[offset++]) << (i * 8);
+        }
+
+        // Read each document
+        for (uint64_t i = 0; i < docCount; ++i) {
+            if (offset + 4 > data.size()) break;
+            uint32_t docLen = 0;
+            for (int j = 0; j < 4; ++j) {
+                docLen |= static_cast<uint32_t>(data[offset++]) << (j * 8);
+            }
+
+            if (offset + docLen > data.size()) break;
+            std::string docJson(reinterpret_cast<const char*>(&data[offset]), docLen);
+            offset += docLen;
+
+            try {
+                Document doc = Document::fromJson(nlohmann::json::parse(docJson));
+                store.loadDocument(collName, doc);
+            } catch (const nlohmann::json::exception&) {
+                // Skip invalid document
+            }
+        }
+    }
+}
+
+} // namespace smartbotic::database

service/src/persistence/snapshot.hpp

@@ -0,0 +1,150 @@
+#pragma once
+
+#include "../document.hpp"
+#include "../memory_store.hpp"
+
+#include <filesystem>
+#include <functional>
+#include <optional>
+#include <string>
+#include <vector>
+
+namespace smartbotic::database {
+
+/**
+ * Snapshot file format:
+ *
+ * Header (64 bytes):
+ *   - Magic: "CALSNAP1" (8 bytes)
+ *   - Version: uint32 (4 bytes)
+ *   - Timestamp: uint64 (8 bytes)
+ *   - WAL Sequence: uint64 (8 bytes) - Sequence at snapshot time
+ *   - Document Count: uint64 (8 bytes)
+ *   - Collection Count: uint32 (4 bytes)
+ *   - Uncompressed Size: uint64 (8 bytes)
+ *   - Compressed Size: uint64 (8 bytes)
+ *   - Compression Type: uint32 (4 bytes) - 0=none, 1=lz4
+ *   - Header Checksum: uint32 (4 bytes)
+ *
+ * Body (compressed):
+ *   For each collection:
+ *     - Collection name length: uint16
+ *     - Collection name: bytes
+ *     - Collection options JSON length: uint32
+ *     - Collection options JSON: bytes
+ *     - Document count: uint64
+ *     For each document:
+ *       - Document JSON length: uint32
+ *       - Document JSON: bytes
+ *
+ * Footer:
+ *   - Body checksum: uint32
+ */
+
+struct SnapshotHeader {
+    char magic[8] = {'C', 'A', 'L', 'S', 'N', 'A', 'P', '1'};
+    uint32_t version = 1;
+    uint64_t timestamp = 0;
+    uint64_t walSequence = 0;
+    uint64_t documentCount = 0;
+    uint32_t collectionCount = 0;
+    uint64_t uncompressedSize = 0;
+    uint64_t compressedSize = 0;
+    uint32_t compressionType = 0;  // 0=none, 1=lz4
+    uint32_t headerChecksum = 0;
+};
+
+/**
+ * Snapshot manager for creating and loading snapshots.
+ */
+class SnapshotManager {
+public:
+    struct Config {
+        std::filesystem::path snapshotDir;
+        bool compressionEnabled = true;
+        uint32_t maxSnapshots = 5;  // Keep last N snapshots
+    };
+
+    explicit SnapshotManager(Config config);
+    ~SnapshotManager() = default;
+
+    // Non-copyable
+    SnapshotManager(const SnapshotManager&) = delete;
+    SnapshotManager& operator=(const SnapshotManager&) = delete;
+
+    /**
+     * Create a snapshot from the current state of the memory store.
+     * @param store The memory store to snapshot
+     * @param walSequence Current WAL sequence number
+     * @return Path to the created snapshot file
+     */
+    std::filesystem::path createSnapshot(const MemoryStore& store, uint64_t walSequence);
+
+    /**
+     * Load the most recent snapshot into the memory store.
+     * @param store The memory store to load into
+     * @return The WAL sequence at which the snapshot was taken (or 0 if no snapshot)
+     */
+    uint64_t loadLatestSnapshot(MemoryStore& store);
+
+    /**
+     * Load a specific snapshot into the memory store.
+     * @param path Path to the snapshot file
+     * @param store The memory store to load into
+     * @return The WAL sequence at which the snapshot was taken
+     */
+    uint64_t loadSnapshot(const std::filesystem::path& path, MemoryStore& store);
+
+    /**
+     * Get list of available snapshots sorted by timestamp (newest first).
+     */
+    [[nodiscard]] std::vector<std::filesystem::path> listSnapshots() const;
+
+    /**
+     * Get info about a snapshot file.
+     */
+    [[nodiscard]] std::optional<SnapshotHeader> getSnapshotInfo(const std::filesystem::path& path) const;
+
+    /**
+     * Delete old snapshots, keeping only the most recent ones.
+     */
+    void cleanupOldSnapshots();
+
+    /**
+     * Verify a snapshot file's integrity.
+     */
+    [[nodiscard]] bool verifySnapshot(const std::filesystem::path& path) const;
+
+private:
+    /**
+     * Generate a filename for a new snapshot.
+     */
+    [[nodiscard]] std::string generateFilename() const;
+
+    /**
+     * Compress data using LZ4.
+     */
+    [[nodiscard]] std::vector<uint8_t> compress(const std::vector<uint8_t>& data) const;
+
+    /**
+     * Decompress data using LZ4.
+     */
+    [[nodiscard]] std::vector<uint8_t> decompress(const std::vector<uint8_t>& data,
+                                                   uint64_t uncompressedSize) const;
+
+    /**
+     * Serialize store data to bytes.
+     */
+    [[nodiscard]] std::vector<uint8_t> serializeStore(const MemoryStore& store,
+                                                       uint64_t& docCount,
+                                                       uint32_t& collCount) const;
+
+    /**
+     * Deserialize bytes into store.
+     */
+    void deserializeStore(const std::vector<uint8_t>& data, MemoryStore& store) const;
+
+    Config config_;
+};
+
+} // namespace smartbotic::database

service/src/persistence/wal.cpp

@@ -0,0 +1,597 @@
+#include "wal.hpp"
+
+#include <algorithm>
+#include <chrono>
+#include <cstring>
+#include <iomanip>
+#include <sstream>
+
+namespace smartbotic::database {
+
+// ===== CRC32 Implementation =====
+
+namespace crc32 {
+
+// CRC32 lookup table (polynomial 0xEDB88320)
+static const uint32_t table[256] = {
+    0x00000000, 0x77073096, 0xEE0E612C, 0x990951BA, 0x076DC419, 0x706AF48F, 0xE963A535, 0x9E6495A3,
+    0x0EDB8832, 0x79DCB8A4, 0xE0D5E91E, 0x97D2D988, 0x09B64C2B, 0x7EB17CBD, 0xE7B82D07, 0x90BF1D91,
+    0x1DB71064, 0x6AB020F2, 0xF3B97148, 0x84BE41DE, 0x1ADAD47D, 0x6DDDE4EB, 0xF4D4B551, 0x83D385C7,
+    0x136C9856, 0x646BA8C0, 0xFD62F97A, 0x8A65C9EC, 0x14015C4F, 0x63066CD9, 0xFA0F3D63, 0x8D080DF5,
+    0x3B6E20C8, 0x4C69105E, 0xD56041E4, 0xA2677172, 0x3C03E4D1, 0x4B04D447, 0xD20D85FD, 0xA50AB56B,
+    0x35B5A8FA, 0x42B2986C, 0xDBBBC9D6, 0xACBCF940, 0x32D86CE3, 0x45DF5C75, 0xDCD60DCF, 0xABD13D59,
+    0x26D930AC, 0x51DE003A, 0xC8D75180, 0xBFD06116, 0x21B4F4B5, 0x56B3C423, 0xCFBA9599, 0xB8BDA50F,
+    0x2802B89E, 0x5F058808, 0xC60CD9B2, 0xB10BE924, 0x2F6F7C87, 0x58684C11, 0xC1611DAB, 0xB6662D3D,
+    0x76DC4190, 0x01DB7106, 0x98D220BC, 0xEFD5102A, 0x71B18589, 0x06B6B51F, 0x9FBFE4A5, 0xE8B8D433,
+    0x7807C9A2, 0x0F00F934, 0x9609A88E, 0xE10E9818, 0x7F6A0DBB, 0x086D3D2D, 0x91646C97, 0xE6635C01,
+    0x6B6B51F4, 0x1C6C6162, 0x856530D8, 0xF262004E, 0x6C0695ED, 0x1B01A57B, 0x8208F4C1, 0xF50FC457,
+    0x65B0D9C6, 0x12B7E950, 0x8BBEB8EA, 0xFCB9887C, 0x62DD1DDF, 0x15DA2D49, 0x8CD37CF3, 0xFBD44C65,
+    0x4DB26158, 0x3AB551CE, 0xA3BC0074, 0xD4BB30E2, 0x4ADFA541, 0x3DD895D7, 0xA4D1C46D, 0xD3D6F4FB,
+    0x4369E96A, 0x346ED9FC, 0xAD678846, 0xDA60B8D0, 0x44042D73, 0x33031DE5, 0xAA0A4C5F, 0xDD0D7CC9,
+    0x5005713C, 0x270241AA, 0xBE0B1010, 0xC90C2086, 0x5768B525, 0x206F85B3, 0xB966D409, 0xCE61E49F,
+    0x5EDEF90E, 0x29D9C998, 0xB0D09822, 0xC7D7A8B4, 0x59B33D17, 0x2EB40D81, 0xB7BD5C3B, 0xC0BA6CAD,
+    0xEDB88320, 0x9ABFB3B6, 0x03B6E20C, 0x74B1D29A, 0xEAD54739, 0x9DD277AF, 0x04DB2615, 0x73DC1683,
+    0xE3630B12, 0x94643B84, 0x0D6D6A3E, 0x7A6A5AA8, 0xE40ECF0B, 0x9309FF9D, 0x0A00AE27, 0x7D079EB1,
+    0xF00F9344, 0x8708A3D2, 0x1E01F268, 0x6906C2FE, 0xF762575D, 0x806567CB, 0x196C3671, 0x6E6B06E7,
+    0xFED41B76, 0x89D32BE0, 0x10DA7A5A, 0x67DD4ACC, 0xF9B9DF6F, 0x8EBEEFF9, 0x17B7BE43, 0x60B08ED5,
+    0xD6D6A3E8, 0xA1D1937E, 0x38D8C2C4, 0x4FDFF252, 0xD1BB67F1, 0xA6BC5767, 0x3FB506DD, 0x48B2364B,
+    0xD80D2BDA, 0xAF0A1B4C, 0x36034AF6, 0x41047A60, 0xDF60EFC3, 0xA867DF55, 0x316E8EEF, 0x4669BE79,
+    0xCB61B38C, 0xBC66831A, 0x256FD2A0, 0x5268E236, 0xCC0C7795, 0xBB0B4703, 0x220216B9, 0x5505262F,
+    0xC5BA3BBE, 0xB2BD0B28, 0x2BB45A92, 0x5CB36A04, 0xC2D7FFA7, 0xB5D0CF31, 0x2CD99E8B, 0x5BDEAE1D,
+    0x9B64C2B0, 0xEC63F226, 0x756AA39C, 0x026D930A, 0x9C0906A9, 0xEB0E363F, 0x72076785, 0x05005713,
+    0x95BF4A82, 0xE2B87A14, 0x7BB12BAE, 0x0CB61B38, 0x92D28E9B, 0xE5D5BE0D, 0x7CDCEFB7, 0x0BDBDF21,
+    0x86D3D2D4, 0xF1D4E242, 0x68DDB3F8, 0x1FDA836E, 0x81BE16CD, 0xF6B9265B, 0x6FB077E1, 0x18B74777,
+    0x88085AE6, 0xFF0F6A70, 0x66063BCA, 0x11010B5C, 0x8F659EFF, 0xF862AE69, 0x616BFFD3, 0x166CCF45,
+    0xA00AE278, 0xD70DD2EE, 0x4E048354, 0x3903B3C2, 0xA7672661, 0xD06016F7, 0x4969474D, 0x3E6E77DB,
+    0xAED16A4A, 0xD9D65ADC, 0x40DF0B66, 0x37D83BF0, 0xA9BCAE53, 0xDEBB9EC5, 0x47B2CF7F, 0x30B5FFE9,
+    0xBDBDF21C, 0xCABAC28A, 0x53B39330, 0x24B4A3A6, 0xBAD03605, 0xCDD706B3, 0x54DE5729, 0x23D967BF,
+    0xB3667A2E, 0xC4614AB8, 0x5D681B02, 0x2A6F2B94, 0xB40BBE37, 0xC30C8EA1, 0x5A05DF1B, 0x2D02EF8D
+};
+
+uint32_t calculate(const void* data, size_t length) {
+    uint32_t crc = 0xFFFFFFFF;
+    const uint8_t* bytes = static_cast<const uint8_t*>(data);
+    for (size_t i = 0; i < length; ++i) {
+        crc = table[(crc ^ bytes[i]) & 0xFF] ^ (crc >> 8);
+    }
+    return crc ^ 0xFFFFFFFF;
+}
+
+uint32_t calculate(const std::string& str) {
+    return calculate(str.data(), str.size());
+}
+
+} // namespace crc32
+
+// ===== WalEntry Implementation =====
+
+std::vector<uint8_t> WalEntry::serialize() const {
+    std::vector<uint8_t> result;
+
+    // Reserve space for length prefix (will be set at the end)
+    result.resize(4);
+
+    // Write sequence (8 bytes)
+    for (int i = 0; i < 8; ++i) {
+        result.push_back(static_cast<uint8_t>((sequence >> (i * 8)) & 0xFF));
+    }
+
+    // Write timestamp (8 bytes)
+    for (int i = 0; i < 8; ++i) {
+        result.push_back(static_cast<uint8_t>((timestamp >> (i * 8)) & 0xFF));
+    }
+
+    // Write opType (1 byte)
+    result.push_back(static_cast<uint8_t>(opType));
+
+    // Write collection (2 bytes length + data)
+    uint16_t collLen = static_cast<uint16_t>(collection.size());
+    result.push_back(static_cast<uint8_t>(collLen & 0xFF));
+    result.push_back(static_cast<uint8_t>((collLen >> 8) & 0xFF));
+    result.insert(result.end(), collection.begin(), collection.end());
+
+    // Write documentId (2 bytes length + data)
+    uint16_t docIdLen = static_cast<uint16_t>(documentId.size());
+    result.push_back(static_cast<uint8_t>(docIdLen & 0xFF));
+    result.push_back(static_cast<uint8_t>((docIdLen >> 8) & 0xFF));
+    result.insert(result.end(), documentId.begin(), documentId.end());
+
+    // Write data flag and data (1 byte flag + 4 bytes length + data)
+    if (data) {
+        result.push_back(1);
+        std::string dataStr = data->dump();
+        uint32_t dataLen = static_cast<uint32_t>(dataStr.size());
+        for (int i = 0; i < 4; ++i) {
+            result.push_back(static_cast<uint8_t>((dataLen >> (i * 8)) & 0xFF));
+        }
+        result.insert(result.end(), dataStr.begin(), dataStr.end());
+    } else {
+        result.push_back(0);
+    }
+
+    // Write collectionOptions flag and data
+    if (collectionOptions) {
+        result.push_back(1);
+        std::string optsStr = collectionOptions->toJson().dump();
+        uint32_t optsLen = static_cast<uint32_t>(optsStr.size());
+        for (int i = 0; i < 4; ++i) {
+            result.push_back(static_cast<uint8_t>((optsLen >> (i * 8)) & 0xFF));
+        }
+        result.insert(result.end(), optsStr.begin(), optsStr.end());
+    } else {
+        result.push_back(0);
+    }
+
+    // Calculate and append checksum (excluding length prefix and checksum itself)
+    uint32_t crc = crc32::calculate(result.data() + 4, result.size() - 4);
+    for (int i = 0; i < 4; ++i) {
+        result.push_back(static_cast<uint8_t>((crc >> (i * 8)) & 0xFF));
+    }
+
+    // Set length prefix (total size minus 4 bytes for length prefix)
+    uint32_t totalLen = static_cast<uint32_t>(result.size() - 4);
+    result[0] = static_cast<uint8_t>(totalLen & 0xFF);
+    result[1] = static_cast<uint8_t>((totalLen >> 8) & 0xFF);
+    result[2] = static_cast<uint8_t>((totalLen >> 16) & 0xFF);
+    result[3] = static_cast<uint8_t>((totalLen >> 24) & 0xFF);
+
+    return result;
+}
+
+std::optional<WalEntry> WalEntry::deserialize(const std::vector<uint8_t>& data) {
+    if (data.size() < 4) {
+        return std::nullopt;
+    }
+
+    size_t offset = 0;
+
+    // Read length prefix
+    uint32_t length = 0;
+    for (int i = 0; i < 4; ++i) {
+        length |= static_cast<uint32_t>(data[offset++]) << (i * 8);
+    }
+
+    if (data.size() < length + 4) {
+        return std::nullopt;
+    }
+
+    // Verify checksum
+    uint32_t storedCrc = 0;
+    for (int i = 0; i < 4; ++i) {
+        storedCrc |= static_cast<uint32_t>(data[data.size() - 4 + i]) << (i * 8);
+    }
+    uint32_t calculatedCrc = crc32::calculate(data.data() + 4, length - 4);
+    if (storedCrc != calculatedCrc) {
+        return std::nullopt;  // Checksum mismatch
+    }
+
+    WalEntry entry;
+
+    // Read sequence
+    for (int i = 0; i < 8; ++i) {
+        entry.sequence |= static_cast<uint64_t>(data[offset++]) << (i * 8);
+    }
+
+    // Read timestamp
+    for (int i = 0; i < 8; ++i) {
+        entry.timestamp |= static_cast<uint64_t>(data[offset++]) << (i * 8);
+    }
+
+    // Read opType
+    entry.opType = static_cast<WalOpType>(data[offset++]);
+
+    // Read collection
+    uint16_t collLen = 0;
+    collLen |= static_cast<uint16_t>(data[offset++]);
+    collLen |= static_cast<uint16_t>(data[offset++]) << 8;
+    entry.collection = std::string(reinterpret_cast<const char*>(&data[offset]), collLen);
+    offset += collLen;
+
+    // Read documentId
+    uint16_t docIdLen = 0;
+    docIdLen |= static_cast<uint16_t>(data[offset++]);
+    docIdLen |= static_cast<uint16_t>(data[offset++]) << 8;
+    entry.documentId = std::string(reinterpret_cast<const char*>(&data[offset]), docIdLen);
+    offset += docIdLen;
+
+    // Read data
+    uint8_t hasData = data[offset++];
+    if (hasData) {
+        uint32_t dataLen = 0;
+        for (int i = 0; i < 4; ++i) {
+            dataLen |= static_cast<uint32_t>(data[offset++]) << (i * 8);
+        }
+        std::string dataStr(reinterpret_cast<const char*>(&data[offset]), dataLen);
+        offset += dataLen;
+        try {
+            entry.data = nlohmann::json::parse(dataStr);
+        } catch (const nlohmann::json::exception&) {
+            return std::nullopt;
+        }
+    }
+
+    // Read collectionOptions
+    uint8_t hasOpts = data[offset++];
+    if (hasOpts) {
+        uint32_t optsLen = 0;
+        for (int i = 0; i < 4; ++i) {
+            optsLen |= static_cast<uint32_t>(data[offset++]) << (i * 8);
+        }
+        std::string optsStr(reinterpret_cast<const char*>(&data[offset]), optsLen);
+        try {
+            entry.collectionOptions = CollectionOptions::fromJson(nlohmann::json::parse(optsStr));
+        } catch (const nlohmann::json::exception&) {
+            return std::nullopt;
+        }
+    }
+
+    entry.checksum = storedCrc;
+    return entry;
+}
+
+uint32_t WalEntry::calculateChecksum() const {
+    auto serialized = serialize();
+    // Exclude length prefix and checksum
+    return crc32::calculate(serialized.data() + 4, serialized.size() - 8);
+}
+
+// ===== WriteAheadLog Implementation =====
+
+WriteAheadLog::WriteAheadLog(Config config)
+    : config_(std::move(config)) {
+}
+
+WriteAheadLog::~WriteAheadLog() {
+    close();
+}
+
+bool WriteAheadLog::open() {
+    std::lock_guard<std::mutex> lock(writeMutex_);
+
+    if (isOpen_.load()) {
+        return true;
+    }
+
+    // Create WAL directory if it doesn't exist
+    std::error_code ec;
+    std::filesystem::create_directories(config_.walDir, ec);
+    if (ec) {
+        return false;
+    }
+
+    // Find existing WAL files and determine highest sequence
+    auto walFiles = getWalFiles();
+    if (!walFiles.empty()) {
+        // Read the last file to get the highest sequence
+        for (auto it = walFiles.rbegin(); it != walFiles.rend(); ++it) {
+            std::ifstream file(*it, std::ios::binary);
+            if (!file) continue;
+
+            auto header = readHeader(file);
+            if (!header) continue;
+
+            // Read entries to find highest sequence
+            while (file) {
+                uint32_t length;
+                file.read(reinterpret_cast<char*>(&length), 4);
+                if (!file || length == 0) break;
+
+                std::vector<uint8_t> entryData(length + 4);
+                std::memcpy(entryData.data(), &length, 4);
+                file.read(reinterpret_cast<char*>(entryData.data() + 4), length);
+                if (!file) break;
+
+                auto entry = WalEntry::deserialize(entryData);
+                if (entry && entry->sequence > sequence_.load()) {
+                    sequence_.store(entry->sequence);
+                }
+            }
+        }
+    }
+
+    // Open current WAL file for appending
+    auto currentPath = currentFilePath();
+    bool fileExists = std::filesystem::exists(currentPath);
+
+    currentFile_.open(currentPath, std::ios::binary | std::ios::app);
+    if (!currentFile_) {
+        return false;
+    }
+
+    if (!fileExists) {
+        // Write header for new file
+        currentFileStartSequence_ = sequence_.load() + 1;
+        writeHeader();
+    } else {
+        // Read existing header
+        std::ifstream readFile(currentPath, std::ios::binary);
+        auto header = readHeader(readFile);
+        if (header) {
+            currentFileStartSequence_ = header->startSequence;
+        }
+    }
+
+    currentFileSize_ = std::filesystem::file_size(currentPath);
+    isOpen_.store(true);
+    return true;
+}
+
+void WriteAheadLog::close() {
+    std::lock_guard<std::mutex> lock(writeMutex_);
+
+    if (!isOpen_.load()) {
+        return;
+    }
+
+    if (currentFile_.is_open()) {
+        currentFile_.flush();
+        currentFile_.close();
+    }
+
+    isOpen_.store(false);
+}
+
+uint64_t WriteAheadLog::append(WalEntry entry) {
+    std::lock_guard<std::mutex> lock(writeMutex_);
+
+    if (!isOpen_.load() || !currentFile_.is_open()) {
+        return 0;
+    }
+
+    // Assign sequence number and timestamp
+    entry.sequence = ++sequence_;
+    entry.timestamp = static_cast<uint64_t>(
+        std::chrono::duration_cast<std::chrono::milliseconds>(
+            std::chrono::system_clock::now().time_since_epoch()
+        ).count()
+    );
+
+    // Serialize and write
+    auto data = entry.serialize();
+    currentFile_.write(reinterpret_cast<const char*>(data.data()), static_cast<std::streamsize>(data.size()));
+    currentFileSize_ += data.size();
+
+    if (config_.syncOnWrite) {
+        currentFile_.flush();
+        sync();
+    }
+
+    // Rotate if file is too large
+    if (currentFileSize_ >= config_.maxFileSizeBytes) {
+        rotate();
+    }
+
+    return entry.sequence;
+}
+
+void WriteAheadLog::sync() {
+    if (currentFile_.is_open()) {
+        currentFile_.flush();
+        // Note: For true durability, we'd use fsync() here
+        // std::filesystem doesn't provide this, would need OS-specific code
+    }
+}
+
+uint64_t WriteAheadLog::replay(uint64_t fromSequence, std::function<void(const WalEntry&)> callback) {
+    auto walFiles = getWalFiles();
+    uint64_t count = 0;
+
+    for (const auto& file : walFiles) {
+        count += replayFile(file, fromSequence, callback);
+    }
+
+    return count;
+}
+
+uint64_t WriteAheadLog::totalSizeBytes() const {
+    uint64_t total = 0;
+    auto walFiles = getWalFiles();
+    for (const auto& file : walFiles) {
+        std::error_code ec;
+        total += std::filesystem::file_size(file, ec);
+    }
+    return total;
+}
+
+void WriteAheadLog::truncateBefore(uint64_t sequence) {
+    std::lock_guard<std::mutex> lock(writeMutex_);
+
+    auto walFiles = getWalFiles();
+    for (const auto& file : walFiles) {
+        // Read header to get start sequence
+        std::ifstream readFile(file, std::ios::binary);
+        auto header = readHeader(readFile);
+        readFile.close();
+
+        if (header && header->startSequence < sequence) {
+            // Check if all entries in this file are before the sequence
+            bool canDelete = true;
+            uint64_t maxSeqInFile = 0;
+
+            std::ifstream scanFile(file, std::ios::binary);
+            scanFile.seekg(sizeof(Header));
+
+            while (scanFile) {
+                uint32_t length;
+                scanFile.read(reinterpret_cast<char*>(&length), 4);
+                if (!scanFile || length == 0) break;
+
+                std::vector<uint8_t> entryData(length + 4);
+                std::memcpy(entryData.data(), &length, 4);
+                scanFile.read(reinterpret_cast<char*>(entryData.data() + 4), length);
+                if (!scanFile) break;
+
+                auto entry = WalEntry::deserialize(entryData);
+                if (entry) {
+                    maxSeqInFile = std::max(maxSeqInFile, entry->sequence);
+                }
+            }
+
+            if (maxSeqInFile < sequence) {
+                // Safe to delete this file
+                std::error_code ec;
+                std::filesystem::remove(file, ec);
+            }
+        }
+    }
+}
+
+WalEntry WriteAheadLog::makeInsertEntry(const std::string& collection, const Document& doc) {
+    WalEntry entry;
+    entry.opType = WalOpType::INSERT;
+    entry.collection = collection;
+    entry.documentId = doc.id;
+    entry.data = doc.toJson();
+    return entry;
+}
+
+WalEntry WriteAheadLog::makeUpdateEntry(const std::string& collection, const Document& doc) {
+    WalEntry entry;
+    entry.opType = WalOpType::UPDATE;
+    entry.collection = collection;
+    entry.documentId = doc.id;
+    entry.data = doc.toJson();
+    return entry;
+}
+
+WalEntry WriteAheadLog::makeDeleteEntry(const std::string& collection, const std::string& id) {
+    WalEntry entry;
+    entry.opType = WalOpType::DELETE;
+    entry.collection = collection;
+    entry.documentId = id;
+    return entry;
+}
+
+WalEntry WriteAheadLog::makeUpsertEntry(const std::string& collection, const Document& doc) {
+    WalEntry entry;
+    entry.opType = WalOpType::UPSERT;
+    entry.collection = collection;
+    entry.documentId = doc.id;
+    entry.data = doc.toJson();
+    return entry;
+}
+
+WalEntry WriteAheadLog::makeCreateCollectionEntry(const std::string& collection, const CollectionOptions& options) {
+    WalEntry entry;
+    entry.opType = WalOpType::CREATE_COLLECTION;
+    entry.collection = collection;
+    entry.collectionOptions = options;
+    return entry;
+}
+
+WalEntry WriteAheadLog::makeDropCollectionEntry(const std::string& collection) {
+    WalEntry entry;
+    entry.opType = WalOpType::DROP_COLLECTION;
+    entry.collection = collection;
+    return entry;
+}
+
+std::filesystem::path WriteAheadLog::currentFilePath() const {
+    return config_.walDir / "wal-current.log";
+}
+
+std::vector<std::filesystem::path> WriteAheadLog::getWalFiles() const {
+    std::vector<std::filesystem::path> files;
+
+    std::error_code ec;
+    for (const auto& entry : std::filesystem::directory_iterator(config_.walDir, ec)) {
+        if (entry.is_regular_file()) {
+            const auto& path = entry.path();
+            if (path.filename().string().starts_with("wal-") &&
+                path.extension() == ".log") {
+                files.push_back(path);
+            }
+        }
+    }
+
+    // Sort by filename (which includes sequence number for archived files)
+    std::sort(files.begin(), files.end());
+    return files;
+}
+
+void WriteAheadLog::rotate() {
+    if (currentFile_.is_open()) {
+        currentFile_.flush();
+        currentFile_.close();
+    }
+
+    // Rename current file with sequence range
+    auto currentPath = currentFilePath();
+    std::ostringstream oss;
+    oss << "wal-" << std::setfill('0') << std::setw(16) << currentFileStartSequence_
+        << "-" << std::setw(16) << sequence_.load() << ".log";
+    auto archivePath = config_.walDir / oss.str();
+
+    std::error_code ec;
+    std::filesystem::rename(currentPath, archivePath, ec);
+
+    // Open new current file
+    currentFileStartSequence_ = sequence_.load() + 1;
+    currentFile_.open(currentPath, std::ios::binary | std::ios::app);
+    if (currentFile_) {
+        writeHeader();
+        currentFileSize_ = sizeof(Header);
+    }
+}
+
+std::optional<WriteAheadLog::Header> WriteAheadLog::readHeader(std::ifstream& file) {
+    Header header;
+    file.read(reinterpret_cast<char*>(&header), sizeof(Header));
+    if (!file) {
+        return std::nullopt;
+    }
+
+    // Verify magic
+    if (std::memcmp(header.magic, "CALWAL01", 8) != 0) {
+        return std::nullopt;
+    }
+
+    return header;
+}
+
+void WriteAheadLog::writeHeader() {
+    Header header;
+    std::memcpy(header.magic, "CALWAL01", 8);
+    header.startSequence = currentFileStartSequence_;
+    currentFile_.write(reinterpret_cast<const char*>(&header), sizeof(Header));
+    currentFile_.flush();
+}
+
+uint64_t WriteAheadLog::replayFile(const std::filesystem::path& path, uint64_t fromSequence,
+                                    const std::function<void(const WalEntry&)>& callback) {
+    std::ifstream file(path, std::ios::binary);
+    if (!file) {
+        return 0;
+    }
+
+    auto header = readHeader(file);
+    if (!header) {
+        return 0;
+    }
+
+    uint64_t count = 0;
+
+    while (file) {
+        // Read entry length
+        uint32_t length;
+        file.read(reinterpret_cast<char*>(&length), 4);
+        if (!file || length == 0) break;
+
+        // Read entry data
+        std::vector<uint8_t> entryData(length + 4);
+        std::memcpy(entryData.data(), &length, 4);
+        file.read(reinterpret_cast<char*>(entryData.data() + 4), length);
+        if (!file) break;
+
+        // Deserialize and invoke callback
+        auto entry = WalEntry::deserialize(entryData);
+        if (entry && entry->sequence > fromSequence) {
+            callback(*entry);
+            count++;
+        }
+    }
+
+    return count;
+}
+
+} // namespace smartbotic::database

service/src/persistence/wal.hpp

@@ -0,0 +1,187 @@
+#pragma once
+
+#include "../document.hpp"
+
+#include <atomic>
+#include <cstdint>
+#include <filesystem>
+#include <fstream>
+#include <functional>
+#include <mutex>
+#include <optional>
+#include <string>
+#include <vector>
+
+namespace smartbotic::database {
+
+/**
+ * Operation type for WAL entries.
+ */
+enum class WalOpType : uint8_t {
+    INSERT = 1,
+    UPDATE = 2,
+    DELETE = 3,
+    UPSERT = 4,
+    CREATE_COLLECTION = 5,
+    DROP_COLLECTION = 6,
+    SET_ADD = 7,
+    SET_REMOVE = 8
+};
+
+/**
+ * WAL entry structure.
+ * Binary format for efficient serialization.
+ */
+struct WalEntry {
+    uint64_t sequence = 0;           // Monotonic sequence number
+    uint64_t timestamp = 0;          // Wall clock time (ms)
+    WalOpType opType = WalOpType::INSERT;
+    std::string collection;
+    std::string documentId;
+    std::optional<nlohmann::json> data;  // For INSERT/UPDATE/UPSERT
+    std::optional<CollectionOptions> collectionOptions;  // For CREATE_COLLECTION
+    uint32_t checksum = 0;           // CRC32 for integrity
+
+    // Serialize entry to binary
+    [[nodiscard]] std::vector<uint8_t> serialize() const;
+
+    // Deserialize entry from binary
+    static std::optional<WalEntry> deserialize(const std::vector<uint8_t>& data);
+
+    // Calculate CRC32 checksum
+    [[nodiscard]] uint32_t calculateChecksum() const;
+};
+
+/**
+ * Write-Ahead Log for durability.
+ *
+ * All mutations are written to the WAL before being applied to memory.
+ * The WAL ensures durability by persisting operations to disk.
+ *
+ * File format:
+ * - Header (16 bytes): magic, version, sequence
+ * - Entries: [length(4) | entry_data | crc32(4)] ...
+ */
+class WriteAheadLog {
+public:
+    struct Config {
+        std::filesystem::path walDir;
+        uint32_t syncIntervalMs = 100;        // fsync interval
+        uint64_t maxFileSizeBytes = 100ULL * 1024 * 1024;  // 100MB max per file
+        bool syncOnWrite = false;             // fsync after each write (slow but safe)
+    };
+
+    explicit WriteAheadLog(Config config);
+    ~WriteAheadLog();
+
+    // Non-copyable, non-movable
+    WriteAheadLog(const WriteAheadLog&) = delete;
+    WriteAheadLog& operator=(const WriteAheadLog&) = delete;
+    WriteAheadLog(WriteAheadLog&&) = delete;
+    WriteAheadLog& operator=(WriteAheadLog&&) = delete;
+
+    /**
+     * Open the WAL for writing.
+     * Creates the WAL directory if it doesn't exist.
+     */
+    bool open();
+
+    /**
+     * Close the WAL, ensuring all data is flushed.
+     */
+    void close();
+
+    /**
+     * Append an entry to the WAL.
+     * @return The sequence number assigned to the entry
+     */
+    uint64_t append(WalEntry entry);
+
+    /**
+     * Force sync to disk.
+     */
+    void sync();
+
+    /**
+     * Replay WAL entries from a sequence number.
+     * @param fromSequence Start replaying from this sequence (exclusive)
+     * @param callback Called for each entry
+     * @return Number of entries replayed
+     */
+    uint64_t replay(uint64_t fromSequence, std::function<void(const WalEntry&)> callback);
+
+    /**
+     * Get the current sequence number.
+     */
+    [[nodiscard]] uint64_t currentSequence() const { return sequence_.load(); }
+
+    /**
+     * Get the total size of all WAL files.
+     */
+    [[nodiscard]] uint64_t totalSizeBytes() const;
+
+    /**
+     * Truncate WAL files before a sequence number.
+     * Used after snapshot to remove old entries.
+     */
+    void truncateBefore(uint64_t sequence);
+
+    /**
+     * Check if WAL is open.
+     */
+    [[nodiscard]] bool isOpen() const { return isOpen_.load(); }
+
+    // Helper methods for creating entries
+    static WalEntry makeInsertEntry(const std::string& collection, const Document& doc);
+    static WalEntry makeUpdateEntry(const std::string& collection, const Document& doc);
+    static WalEntry makeDeleteEntry(const std::string& collection, const std::string& id);
+    static WalEntry makeUpsertEntry(const std::string& collection, const Document& doc);
+    static WalEntry makeCreateCollectionEntry(const std::string& collection, const CollectionOptions& options);
+    static WalEntry makeDropCollectionEntry(const std::string& collection);
+
+private:
+    // WAL file header
+    struct Header {
+        char magic[8] = {'C', 'A', 'L', 'W', 'A', 'L', '0', '1'};
+        uint64_t startSequence = 0;
+    };
+
+    // Get current WAL file path
+    [[nodiscard]] std::filesystem::path currentFilePath() const;
+
+    // Get all WAL file paths sorted by sequence
+    [[nodiscard]] std::vector<std::filesystem::path> getWalFiles() const;
+
+    // Rotate to a new WAL file
+    void rotate();
+
+    // Read header from file
+    static std::optional<Header> readHeader(std::ifstream& file);
+
+    // Write header to file
+    void writeHeader();
+
+    // Read entries from a file
+    uint64_t replayFile(const std::filesystem::path& path, uint64_t fromSequence,
+                        const std::function<void(const WalEntry&)>& callback);
+
+    Config config_;
+    std::atomic<bool> isOpen_{false};
+    std::atomic<uint64_t> sequence_{0};
+
+    std::ofstream currentFile_;
+    uint64_t currentFileSize_ = 0;
+    uint64_t currentFileStartSequence_ = 0;
+
+    mutable std::mutex writeMutex_;
+};
+
+/**
+ * CRC32 utility functions.
+ */
+namespace crc32 {
+    uint32_t calculate(const void* data, size_t length);
+    uint32_t calculate(const std::string& str);
+}
+
+} // namespace smartbotic::database

service/src/replication/conflict_resolver.cpp

@@ -0,0 +1,140 @@
+#include "conflict_resolver.hpp"
+
+#include <spdlog/spdlog.h>
+
+namespace smartbotic::database {
+
+ConflictResolver::ConflictResolver(Config config)
+    : config_(std::move(config))
+{
+}
+
+ConflictResolver::~ConflictResolver() = default;
+
+std::optional<Document> ConflictResolver::resolve(
+    const Document& local,
+    const Document& remote
+) const {
+    if (!isConflict(local, remote)) {
+        // No conflict - remote is strictly newer
+        return remote;
+    }
+
+    switch (config_.defaultStrategy) {
+        case Strategy::LAST_WRITER_WINS:
+            return resolveLastWriterWins(local, remote);
+
+        case Strategy::FIRST_WRITER_WINS:
+            return resolveFirstWriterWins(local, remote);
+
+        case Strategy::MERGE:
+            return resolveMerge(local, remote);
+
+        case Strategy::MANUAL:
+            // Requires manual resolution
+            spdlog::warn("Conflict requires manual resolution: {}/{}",
+                        local.collection, local.id);
+            return std::nullopt;
+    }
+
+    return std::nullopt;
+}
+
+bool ConflictResolver::isConflict(
+    const Document& local,
+    const Document& remote
+) const {
+    // Conflict occurs when:
+    // 1. Same document (collection + id)
+    // 2. Different versions based on different ancestors
+    // For simplicity, we detect conflict when versions diverge
+
+    if (local.collection != remote.collection || local.id != remote.id) {
+        return false;  // Not the same document
+    }
+
+    // If local and remote have different node origins and overlapping versions
+    // this is a conflict
+    if (local.nodeId != remote.nodeId && local.version == remote.version) {
+        return true;
+    }
+
+    // If both were updated at similar times from different nodes
+    if (local.nodeId != remote.nodeId &&
+        std::abs(static_cast<int64_t>(local.updatedAt) - static_cast<int64_t>(remote.updatedAt)) < 1000) {
+        return true;
+    }
+
+    return false;
+}
+
+Document ConflictResolver::resolveLastWriterWins(
+    const Document& local,
+    const Document& remote
+) const {
+    // The document with the most recent update time wins
+    if (local.updatedAt >= remote.updatedAt) {
+        spdlog::debug("Conflict resolved (LWW): local wins ({} >= {})",
+                     local.updatedAt, remote.updatedAt);
+        return local;
+    } else {
+        spdlog::debug("Conflict resolved (LWW): remote wins ({} > {})",
+                     remote.updatedAt, local.updatedAt);
+        return remote;
+    }
+}
+
+Document ConflictResolver::resolveFirstWriterWins(
+    const Document& local,
+    const Document& remote
+) const {
+    // The document with the earlier creation time wins
+    if (local.createdAt <= remote.createdAt) {
+        spdlog::debug("Conflict resolved (FWW): local wins ({} <= {})",
+                     local.createdAt, remote.createdAt);
+        return local;
+    } else {
+        spdlog::debug("Conflict resolved (FWW): remote wins ({} < {})",
+                     remote.createdAt, local.createdAt);
+        return remote;
+    }
+}
+
+std::optional<Document> ConflictResolver::resolveMerge(
+    const Document& local,
+    const Document& remote
+) const {
+    // Attempt to merge the two documents
+    // This is a simplified merge that combines non-conflicting fields
+
+    try {
+        Document merged = local;
+        merged.version = std::max(local.version, remote.version) + 1;
+        merged.updatedAt = std::max(local.updatedAt, remote.updatedAt);
+
+        // Merge JSON data
+        // For each field in remote that doesn't exist in local, add it
+        for (const auto& [key, value] : remote.data.items()) {
+            if (!local.data.contains(key)) {
+                merged.data[key] = value;
+            } else if (local.data[key] != value) {
+                // Field exists in both with different values
+                // For now, use the most recently updated value
+                if (remote.updatedAt > local.updatedAt) {
+                    merged.data[key] = value;
+                }
+            }
+        }
+
+        spdlog::debug("Conflict resolved (merge): combined {} and {} fields",
+                     local.data.size(), remote.data.size());
+
+        return merged;
+
+    } catch (const std::exception& e) {
+        spdlog::error("Merge failed: {}", e.what());
+        return std::nullopt;
+    }
+}
+
+} // namespace smartbotic::database

service/src/replication/conflict_resolver.hpp

@@ -0,0 +1,67 @@
+#pragma once
+
+#include "../document.hpp"
+
+#include <string>
+
+namespace smartbotic::database {
+
+/**
+ * Resolves conflicts when the same document is modified on multiple nodes.
+ */
+class ConflictResolver {
+public:
+    enum class Strategy {
+        LAST_WRITER_WINS,   // Most recent update wins (by timestamp)
+        FIRST_WRITER_WINS,  // First update wins (by version)
+        MERGE,              // Attempt to merge changes
+        MANUAL              // Require manual resolution
+    };
+
+    struct Config {
+        Strategy defaultStrategy;
+
+        Config() : defaultStrategy(Strategy::LAST_WRITER_WINS) {}
+    };
+
+    explicit ConflictResolver(Config config = Config{});
+    ~ConflictResolver();
+
+    /**
+     * Resolve conflict between local and remote document versions.
+     * @param local The local document
+     * @param remote The remote document
+     * @return The resolved document (winner), or std::nullopt if manual resolution needed
+     */
+    [[nodiscard]] std::optional<Document> resolve(
+        const Document& local,
+        const Document& remote
+    ) const;
+
+    /**
+     * Check if two documents are in conflict.
+     */
+    [[nodiscard]] bool isConflict(
+        const Document& local,
+        const Document& remote
+    ) const;
+
+    /**
+     * Get the current resolution strategy.
+     */
+    [[nodiscard]] Strategy getStrategy() const { return config_.defaultStrategy; }
+
+    /**
+     * Set the resolution strategy.
+     */
+    void setStrategy(Strategy strategy) { config_.defaultStrategy = strategy; }
+
+private:
+    Document resolveLastWriterWins(const Document& local, const Document& remote) const;
+    Document resolveFirstWriterWins(const Document& local, const Document& remote) const;
+    std::optional<Document> resolveMerge(const Document& local, const Document& remote) const;
+
+    Config config_;
+};
+
+} // namespace smartbotic::database

service/src/replication/replication_manager.cpp

@@ -0,0 +1,274 @@
+#include "replication_manager.hpp"
+#include "conflict_resolver.hpp"
+
+#include <spdlog/spdlog.h>
+
+#include <chrono>
+
+namespace smartbotic::database {
+
+// Namespace alias for proto types
+namespace pb = smartbotic::databasepb;
+
+ReplicationManager::ReplicationManager(Config config)
+    : config_(std::move(config))
+{
+}
+
+ReplicationManager::~ReplicationManager() {
+    stop();
+}
+
+void ReplicationManager::setEntryApplyCallback(EntryApplyCallback callback) {
+    entryApplyCallback_ = std::move(callback);
+}
+
+void ReplicationManager::setSequence(uint64_t sequence) {
+    currentSequence_.store(sequence);
+    if (syncProtocol_) {
+        syncProtocol_->setLocalSequence(sequence);
+    }
+}
+
+uint64_t ReplicationManager::getSequence() const {
+    return currentSequence_.load();
+}
+
+void ReplicationManager::setLocalCollections(const std::vector<std::string>& collections) {
+    if (syncProtocol_) {
+        syncProtocol_->setLocalCollections(collections);
+    }
+}
+
+void ReplicationManager::start() {
+    if (running_.exchange(true)) {
+        return;  // Already running
+    }
+
+    spdlog::info("ReplicationManager started (node: {}, peers: {})",
+                 config_.nodeId, config_.peerAddresses.size());
+
+    // Start async apply thread first
+    applyRunning_ = true;
+    applyThread_ = std::thread(&ReplicationManager::applyLoop, this);
+
+    // Create and start SyncProtocol
+    if (!config_.peerAddresses.empty()) {
+        SyncProtocol::Config syncConfig;
+        syncConfig.nodeId = config_.nodeId;
+        syncConfig.peerAddresses = config_.peerAddresses;
+        syncConfig.reconnectIntervalMs = 5000;
+        syncConfig.heartbeatIntervalMs = 1000;
+
+        syncProtocol_ = std::make_unique<SyncProtocol>(syncConfig);
+        syncProtocol_->setLocalSequence(currentSequence_.load());
+        syncProtocol_->start([this](const pb::ReplicationEntry& entry) {
+            onEntryReceived(entry);
+        });
+    }
+
+    // Log configured peers
+    for (const auto& peer : config_.peerAddresses) {
+        spdlog::info("Configured peer: {}", peer);
+    }
+
+    // Start sync thread
+    syncThread_ = std::thread(&ReplicationManager::syncLoop, this);
+}
+
+void ReplicationManager::stop() {
+    if (!running_.exchange(false)) {
+        return;  // Not running
+    }
+
+    if (syncProtocol_) {
+        syncProtocol_->stop();
+    }
+
+    if (syncThread_.joinable()) {
+        syncThread_.join();
+    }
+
+    // Stop the async apply thread
+    applyRunning_ = false;
+    queueCv_.notify_all();
+    if (applyThread_.joinable()) {
+        applyThread_.join();
+    }
+
+    spdlog::info("ReplicationManager stopped");
+}
+
+void ReplicationManager::onEntryReceived(const pb::ReplicationEntry& entry) {
+    // Update sequence tracking
+    handleIncomingEntry(entry);
+
+    // Queue entry for async apply (non-blocking)
+    queueEntry(entry);
+}
+
+void ReplicationManager::queueEntry(const pb::ReplicationEntry& entry) {
+    std::lock_guard<std::mutex> lock(queueMutex_);
+    entryQueue_.push(entry);
+    queueCv_.notify_one();
+}
+
+void ReplicationManager::applyLoop() {
+    spdlog::debug("ReplicationManager apply loop started");
+
+    while (applyRunning_) {
+        std::vector<pb::ReplicationEntry> batch;
+
+        // Wait for entries and collect a batch
+        {
+            std::unique_lock<std::mutex> lock(queueMutex_);
+            queueCv_.wait(lock, [this] {
+                return !entryQueue_.empty() || !applyRunning_;
+            });
+
+            if (!applyRunning_ && entryQueue_.empty()) {
+                break;
+            }
+
+            // Collect batch of up to 100 entries
+            while (!entryQueue_.empty() && batch.size() < 100) {
+                batch.push_back(std::move(entryQueue_.front()));
+                entryQueue_.pop();
+            }
+        }
+
+        // Apply batch outside the lock
+        if (!batch.empty() && entryApplyCallback_) {
+            for (const auto& entry : batch) {
+                entryApplyCallback_(entry);
+            }
+            spdlog::debug("Applied batch of {} replicated entries", batch.size());
+        }
+    }
+
+    spdlog::debug("ReplicationManager apply loop stopped");
+}
+
+void ReplicationManager::handleIncomingEntry(const pb::ReplicationEntry& entry) {
+    uint64_t entrySeq = entry.sequence();
+
+    {
+        std::lock_guard lock(mutex_);
+
+        // Update our sequence if this is newer
+        if (entrySeq > currentSequence_) {
+            currentSequence_ = entrySeq;
+        }
+
+        // Update watermark for the source node
+        if (!entry.node_id().empty()) {
+            peerWatermarks_[entry.node_id()] = entrySeq;
+        }
+    }
+
+    spdlog::debug("Received push entry from {} (seq: {}, op: {}, coll: {})",
+                  entry.node_id(), entrySeq, static_cast<int>(entry.op()),
+                  entry.collection());
+
+    // Queue entry for application
+    queueEntry(entry);
+}
+
+void ReplicationManager::handleHeartbeat(const pb::Heartbeat& heartbeat) {
+    std::lock_guard lock(mutex_);
+
+    // Update peer's sequence
+    peerWatermarks_[heartbeat.node_id()] = heartbeat.sequence();
+
+    spdlog::trace("Received heartbeat from {} (seq: {})",
+                  heartbeat.node_id(), heartbeat.sequence());
+}
+
+void ReplicationManager::handleWatermarkUpdate(const pb::WatermarkUpdate& watermark) {
+    std::lock_guard lock(mutex_);
+
+    // Update watermark for the peer
+    peerWatermarks_[watermark.node_id()] = watermark.sequence();
+
+    spdlog::trace("Received watermark update from {} (seq: {})",
+                  watermark.node_id(), watermark.sequence());
+}
+
+void ReplicationManager::handleSyncRequest(
+    const pb::SyncRequest& request,
+    grpc::ServerReaderWriter<pb::ReplicationMessage, pb::ReplicationMessage>* stream
+) {
+    spdlog::debug("Handling sync request (from_seq: {})",
+                  request.from_sequence());
+
+    // Stream entries since the requested sequence
+    streamEntriesSince(request.from_sequence(), 1000,
+        [stream](const pb::ReplicationEntry& entry) {
+            pb::ReplicationMessage msg;
+            *msg.mutable_entry() = entry;
+            stream->Write(msg);
+        });
+
+    // Send watermark update
+    pb::ReplicationMessage watermarkMsg;
+    auto* wm = watermarkMsg.mutable_watermark();
+    wm->set_node_id(config_.nodeId);
+    wm->set_sequence(currentSequence_.load());
+    stream->Write(watermarkMsg);
+}
+
+void ReplicationManager::streamEntriesSince(
+    uint64_t fromSequence,
+    uint32_t limit,
+    std::function<void(const pb::ReplicationEntry&)> callback
+) {
+    // In a full implementation, this would read from the WAL
+    // For now, just log the request
+    spdlog::debug("Streaming entries since seq {} (limit: {})", fromSequence, limit);
+
+    // This would iterate through the WAL and call callback for each entry
+    // The actual implementation depends on integration with WAL
+}
+
+ReplicationManager::NodeStateInfo ReplicationManager::getNodeState() const {
+    std::lock_guard lock(mutex_);
+
+    NodeStateInfo state;
+    state.nodeId = config_.nodeId;
+    state.currentSequence = currentSequence_.load();
+    state.peerWatermarks = peerWatermarks_;
+
+    return state;
+}
+
+void ReplicationManager::queueForReplication(const pb::ReplicationEntry& entry) {
+    // Increment our sequence
+    uint64_t seq = ++currentSequence_;
+
+    spdlog::trace("Queued entry for replication (seq: {}, op: {})",
+                  seq, static_cast<int>(entry.op()));
+
+    // Broadcast to connected peers
+    if (syncProtocol_) {
+        // Create entry with updated sequence
+        pb::ReplicationEntry entryWithSeq = entry;
+        entryWithSeq.set_sequence(seq);
+        entryWithSeq.set_node_id(config_.nodeId);
+        syncProtocol_->broadcast(entryWithSeq);
+    }
+}
+
+void ReplicationManager::syncLoop() {
+    while (running_) {
+        // Periodic sync check
+        std::this_thread::sleep_for(std::chrono::milliseconds(config_.syncIntervalMs));
+
+        if (!running_) break;
+
+        // Send heartbeats to peers
+        // In a full implementation, this would send heartbeats via gRPC
+        spdlog::trace("Sync loop tick (seq: {})", currentSequence_.load());
+    }
+}
+
+} // namespace smartbotic::database

service/src/replication/replication_manager.hpp

@@ -0,0 +1,139 @@
+#pragma once
+
+#include "../document.hpp"
+#include "sync_protocol.hpp"
+
+#include <database.grpc.pb.h>
+
+#include <atomic>
+#include <condition_variable>
+#include <functional>
+#include <grpcpp/grpcpp.h>
+#include <memory>
+#include <mutex>
+#include <queue>
+#include <string>
+#include <thread>
+#include <unordered_map>
+#include <vector>
+
+namespace smartbotic::database {
+
+// Namespace alias for proto types
+namespace pb = smartbotic::databasepb;
+
+/**
+ * Replication manager for multi-master synchronization.
+ */
+class ReplicationManager {
+public:
+    struct Config {
+        std::string nodeId = "storage-1";
+        std::vector<std::string> peerAddresses;
+        uint32_t syncIntervalMs = 100;
+        std::string conflictResolution = "last_writer_wins";
+    };
+
+    struct NodeStateInfo {
+        std::string nodeId;
+        uint64_t currentSequence = 0;
+        std::unordered_map<std::string, uint64_t> peerWatermarks;
+    };
+
+    using EntryApplyCallback = std::function<void(const pb::ReplicationEntry&)>;
+
+    explicit ReplicationManager(Config config);
+    ~ReplicationManager();
+
+    /**
+     * Set callback to apply replicated entries to the store.
+     */
+    void setEntryApplyCallback(EntryApplyCallback callback);
+
+    // Lifecycle
+    void start();
+    void stop();
+
+    /**
+     * Handle incoming replication entry from a peer.
+     */
+    void handleIncomingEntry(const pb::ReplicationEntry& entry);
+
+    /**
+     * Handle heartbeat from a peer.
+     */
+    void handleHeartbeat(const pb::Heartbeat& heartbeat);
+
+    /**
+     * Handle watermark update from a peer.
+     */
+    void handleWatermarkUpdate(const pb::WatermarkUpdate& watermark);
+
+    /**
+     * Handle sync request from a peer.
+     */
+    void handleSyncRequest(
+        const pb::SyncRequest& request,
+        grpc::ServerReaderWriter<pb::ReplicationMessage, pb::ReplicationMessage>* stream
+    );
+
+    /**
+     * Stream entries since a sequence number.
+     */
+    void streamEntriesSince(
+        uint64_t fromSequence,
+        uint32_t limit,
+        std::function<void(const pb::ReplicationEntry&)> callback
+    );
+
+    /**
+     * Get current node state.
+     */
+    [[nodiscard]] NodeStateInfo getNodeState() const;
+
+    /**
+     * Queue an entry for replication to peers.
+     */
+    void queueForReplication(const pb::ReplicationEntry& entry);
+
+    /**
+     * Set the initial sequence number (from persistence).
+     */
+    void setSequence(uint64_t sequence);
+
+    /**
+     * Get current sequence number.
+     */
+    uint64_t getSequence() const;
+
+    /**
+     * Update local collections for new collection discovery.
+     */
+    void setLocalCollections(const std::vector<std::string>& collections);
+
+private:
+    void syncLoop();
+    void applyLoop();
+    void onEntryReceived(const pb::ReplicationEntry& entry);
+    void queueEntry(const pb::ReplicationEntry& entry);
+
+    Config config_;
+    std::atomic<bool> running_{false};
+    std::atomic<uint64_t> currentSequence_{0};
+
+    std::unordered_map<std::string, uint64_t> peerWatermarks_;
+    mutable std::mutex mutex_;
+
+    std::thread syncThread_;
+    std::unique_ptr<SyncProtocol> syncProtocol_;
+    EntryApplyCallback entryApplyCallback_;
+
+    // Async entry queue for non-blocking replication
+    std::queue<pb::ReplicationEntry> entryQueue_;
+    std::mutex queueMutex_;
+    std::condition_variable queueCv_;
+    std::thread applyThread_;
+    std::atomic<bool> applyRunning_{false};
+};
+
+} // namespace smartbotic::database

service/src/replication/sync_protocol.cpp

@@ -0,0 +1,400 @@
+#include "sync_protocol.hpp"
+
+#include <spdlog/spdlog.h>
+
+#include <chrono>
+
+namespace smartbotic::database {
+
+SyncProtocol::SyncProtocol(Config config)
+    : config_(std::move(config))
+{
+    // Initialize peer connections
+    for (const auto& address : config_.peerAddresses) {
+        auto peer = std::make_unique<PeerConnection>();
+        peer->address = address;
+        peers_.push_back(std::move(peer));
+    }
+}
+
+SyncProtocol::~SyncProtocol() {
+    stop();
+}
+
+void SyncProtocol::start(EntryCallback onEntry) {
+    if (running_.exchange(true)) {
+        return;
+    }
+
+    entryCallback_ = std::move(onEntry);
+
+    // Start connection management thread
+    connectionThread_ = std::thread(&SyncProtocol::connectionLoop, this);
+
+    spdlog::info("SyncProtocol started with {} peers", peers_.size());
+}
+
+void SyncProtocol::stop() {
+    if (!running_.exchange(false)) {
+        return;
+    }
+
+    // Wait for connection thread
+    if (connectionThread_.joinable()) {
+        connectionThread_.join();
+    }
+
+    // Disconnect all peers
+    {
+        std::lock_guard lock(mutex_);
+        for (auto& peer : peers_) {
+            peer->connected = false;
+            peer->stub.reset();
+            peer->channel.reset();
+        }
+    }
+
+    spdlog::info("SyncProtocol stopped");
+}
+
+void SyncProtocol::broadcast(const pb::ReplicationEntry& entry) {
+    std::lock_guard lock(mutex_);
+
+    for (auto& peer : peers_) {
+        if (!peer->connected || !peer->stub) {
+            continue;
+        }
+
+        // Send entry via unary RPC to each peer
+        // Note: This is a simpler approach than bidirectional streams
+        // For real-time performance, consider using async or streaming
+        try {
+            grpc::ClientContext context;
+            context.set_deadline(
+                std::chrono::system_clock::now() + std::chrono::seconds(5));
+
+            pb::ReplicationMessage msg;
+            *msg.mutable_entry() = entry;
+
+            // Use SyncStream to send the entry
+            // Note: For a full implementation, we'd maintain persistent streams
+            auto stream = peer->stub->SyncStream(&context);
+            if (stream->Write(msg)) {
+                spdlog::trace("Broadcast entry seq {} to peer {}",
+                             entry.sequence(), peer->address);
+            }
+            stream->WritesDone();
+            stream->Finish();
+        } catch (const std::exception& e) {
+            spdlog::debug("Failed to broadcast to peer {}: {}",
+                         peer->address, e.what());
+        }
+    }
+}
+
+uint64_t SyncProtocol::getPeerWatermark(const std::string& nodeId) const {
+    std::lock_guard lock(mutex_);
+
+    for (const auto& peer : peers_) {
+        if (peer->nodeId == nodeId) {
+            return peer->lastWatermark;
+        }
+    }
+
+    return 0;
+}
+
+bool SyncProtocol::isConnected(const std::string& nodeId) const {
+    std::lock_guard lock(mutex_);
+
+    for (const auto& peer : peers_) {
+        if (peer->nodeId == nodeId) {
+            return peer->connected;
+        }
+    }
+
+    return false;
+}
+
+size_t SyncProtocol::connectedPeerCount() const {
+    std::lock_guard lock(mutex_);
+
+    size_t count = 0;
+    for (const auto& peer : peers_) {
+        if (peer->connected) {
+            ++count;
+        }
+    }
+
+    return count;
+}
+
+void SyncProtocol::setLocalSequence(uint64_t sequence) {
+    localSequence_.store(sequence);
+}
+
+uint64_t SyncProtocol::getLocalSequence() const {
+    return localSequence_.load();
+}
+
+void SyncProtocol::setLocalCollections(const std::vector<std::string>& collections) {
+    std::lock_guard<std::mutex> lock(collectionsMutex_);
+    localCollections_.clear();
+    localCollections_.insert(collections.begin(), collections.end());
+}
+
+void SyncProtocol::connectionLoop() {
+    while (running_) {
+        // Collect peers that need syncing (don't hold mutex during sync)
+        // pair: peer, fromSequence, optional collection name (empty = all)
+        struct SyncTask {
+            PeerConnection* peer;
+            uint64_t fromSequence;
+            std::string collection;  // Empty means sync all from sequence
+        };
+        std::vector<SyncTask> syncTasks;
+
+        // Get local collections for new collection discovery
+        std::set<std::string> localColls;
+        {
+            std::lock_guard<std::mutex> lock(collectionsMutex_);
+            localColls = localCollections_;
+        }
+
+        // Try to connect to disconnected peers and check connected peers for new data
+        {
+            std::lock_guard lock(mutex_);
+            for (auto& peer : peers_) {
+                if (!peer->connected) {
+                    connectToPeer(*peer);
+                    // If connected and behind, mark for syncing outside mutex
+                    if (peer->connected && peer->lastWatermark > localSequence_.load()) {
+                        syncTasks.push_back({peer.get(), localSequence_.load(), ""});
+                    }
+                } else {
+                    // Already connected - check for new entries by polling GetNodeState
+                    pollPeerState(*peer);
+
+                    // Check if peer has collections we don't have
+                    // Just track them - they'll be synced as part of normal entry sync
+                    for (const auto& peerColl : peer->collections) {
+                        if (localColls.find(peerColl) == localColls.end()) {
+                            spdlog::info("Discovered new collection '{}' on peer {}",
+                                        peerColl, peer->nodeId);
+                            // Add to local set - entries for this collection will be
+                            // synced as part of normal sequence-based sync
+                            localColls.insert(peerColl);
+                            {
+                                std::lock_guard<std::mutex> lock(collectionsMutex_);
+                                localCollections_.insert(peerColl);
+                            }
+                        }
+                    }
+
+                    // Check if peer is ahead in overall sequence
+                    if (peer->connected && peer->lastWatermark > localSequence_.load()) {
+                        syncTasks.push_back({peer.get(), localSequence_.load(), ""});
+                    }
+                }
+            }
+        }
+
+        // Perform sync outside the mutex
+        for (auto& task : syncTasks) {
+            if (task.peer->connected && running_) {
+                if (task.collection.empty()) {
+                    spdlog::debug("Syncing from peer {} (local seq: {}, peer seq: {})",
+                                task.peer->nodeId, task.fromSequence, task.peer->lastWatermark);
+                } else {
+                    spdlog::debug("Syncing collection '{}' from peer {} (from seq: {})",
+                                task.collection, task.peer->nodeId, task.fromSequence);
+                }
+                pullEntriesFromPeer(*task.peer, task.fromSequence);
+            }
+        }
+
+        // Wait before next connection attempt / poll
+        std::this_thread::sleep_for(
+            std::chrono::milliseconds(config_.reconnectIntervalMs));
+    }
+}
+
+void SyncProtocol::connectToPeer(PeerConnection& peer) {
+    try {
+        // Create channel
+        grpc::ChannelArguments args;
+        args.SetInt(GRPC_ARG_KEEPALIVE_TIME_MS, 10000);
+        args.SetInt(GRPC_ARG_KEEPALIVE_TIMEOUT_MS, 5000);
+
+        peer.channel = grpc::CreateCustomChannel(
+            peer.address,
+            grpc::InsecureChannelCredentials(),
+            args
+        );
+
+        // Create stub
+        peer.stub = pb::StorageReplication::NewStub(peer.channel);
+
+        // Try to get node state to verify connection
+        grpc::ClientContext context;
+        context.set_deadline(
+            std::chrono::system_clock::now() + std::chrono::seconds(5));
+
+        pb::GetNodeStateRequest request;
+        pb::NodeState response;
+
+        auto status = peer.stub->GetNodeState(&context, request, &response);
+        if (status.ok()) {
+            peer.nodeId = response.node_id();
+            peer.lastWatermark = response.current_sequence();
+            peer.connected = true;
+
+            // Populate collections from peer
+            peer.collections.clear();
+            peer.collections.reserve(response.collections_size());
+            for (const auto& coll : response.collections()) {
+                peer.collections.push_back(coll);
+            }
+
+            // Populate per-collection sequences
+            peer.collectionSequences.clear();
+            for (const auto& [collName, seq] : response.collection_sequences()) {
+                peer.collectionSequences[collName] = seq;
+            }
+
+            spdlog::info("Connected to peer {} (node: {}, seq: {}, collections: {})",
+                        peer.address, peer.nodeId, peer.lastWatermark, peer.collections.size());
+            // Note: sync will be triggered by connectionLoop outside the mutex
+        } else {
+            spdlog::debug("Failed to connect to peer {}: {}",
+                         peer.address, status.error_message());
+            peer.stub.reset();
+            peer.channel.reset();
+        }
+
+    } catch (const std::exception& e) {
+        spdlog::debug("Exception connecting to peer {}: {}",
+                     peer.address, e.what());
+        peer.stub.reset();
+        peer.channel.reset();
+    }
+}
+
+void SyncProtocol::pullEntriesFromPeer(PeerConnection& peer, uint64_t fromSequence) {
+    if (!peer.stub) {
+        return;
+    }
+
+    uint64_t currentSeq = fromSequence;
+    uint64_t targetSeq = peer.lastWatermark;
+
+    while (running_ && currentSeq < targetSeq) {
+        try {
+            grpc::ClientContext context;
+            context.set_deadline(
+                std::chrono::system_clock::now() + std::chrono::seconds(60));
+
+            pb::GetEntriesRequest request;
+            request.set_from_sequence(currentSeq);
+            request.set_limit(5000); // Pull 5000 entries at a time
+
+            auto reader = peer.stub->GetEntriesSince(&context, request);
+
+            pb::ReplicationEntry entry;
+            uint64_t count = 0;
+            uint64_t lastSeq = currentSeq;
+
+            while (reader->Read(&entry)) {
+                if (entryCallback_) {
+                    entryCallback_(entry);
+                }
+                lastSeq = entry.sequence();
+                ++count;
+            }
+
+            auto status = reader->Finish();
+            if (status.ok()) {
+                spdlog::info("Pulled {} entries from peer {} (seq: {} -> {})",
+                            count, peer.nodeId, currentSeq, lastSeq);
+                if (lastSeq > localSequence_.load()) {
+                    localSequence_.store(lastSeq);
+                }
+                currentSeq = lastSeq;
+
+                // If we got no new entries, we're caught up
+                if (count == 0) {
+                    break;
+                }
+            } else {
+                spdlog::warn("Failed to pull entries from peer {}: {}",
+                            peer.nodeId, status.error_message());
+                break;
+            }
+
+        } catch (const std::exception& e) {
+            spdlog::warn("Exception pulling entries from peer {}: {}",
+                        peer.nodeId, e.what());
+            break;
+        }
+    }
+
+    spdlog::info("Sync complete with peer {}, local sequence: {}",
+                peer.nodeId, localSequence_.load());
+}
+
+void SyncProtocol::pollPeerState(PeerConnection& peer) {
+    if (!peer.stub) {
+        return;
+    }
+
+    try {
+        grpc::ClientContext context;
+        context.set_deadline(
+            std::chrono::system_clock::now() + std::chrono::seconds(2));
+
+        pb::GetNodeStateRequest request;
+        pb::NodeState response;
+
+        auto status = peer.stub->GetNodeState(&context, request, &response);
+        if (status.ok()) {
+            peer.lastWatermark = response.current_sequence();
+
+            // Update collections list from peer
+            peer.collections.clear();
+            peer.collections.reserve(response.collections_size());
+            for (const auto& coll : response.collections()) {
+                peer.collections.push_back(coll);
+            }
+
+            // Update per-collection sequences
+            peer.collectionSequences.clear();
+            for (const auto& [collName, seq] : response.collection_sequences()) {
+                peer.collectionSequences[collName] = seq;
+            }
+
+            spdlog::trace("Polled peer {} state: seq {}, collections: {}",
+                         peer.nodeId, peer.lastWatermark, peer.collections.size());
+        } else {
+            // Connection lost
+            spdlog::debug("Lost connection to peer {}: {}",
+                         peer.address, status.error_message());
+            peer.connected = false;
+            peer.stub.reset();
+            peer.channel.reset();
+        }
+    } catch (const std::exception& e) {
+        spdlog::debug("Exception polling peer {}: {}",
+                     peer.address, e.what());
+        peer.connected = false;
+        peer.stub.reset();
+        peer.channel.reset();
+    }
+}
+
+void SyncProtocol::handlePeerStream(PeerConnection& peer) {
+    // This would handle the bidirectional stream with a peer
+    // Reading incoming entries and forwarding to entryCallback_
+    // For now, this is a placeholder for the full implementation
+}
+
+} // namespace smartbotic::database

service/src/replication/sync_protocol.hpp

@@ -0,0 +1,120 @@
+#pragma once
+
+#include "../document.hpp"
+
+#include <database.grpc.pb.h>
+
+#include <grpcpp/grpcpp.h>
+
+#include <atomic>
+#include <functional>
+#include <memory>
+#include <mutex>
+#include <set>
+#include <string>
+#include <thread>
+#include <unordered_map>
+#include <vector>
+
+namespace smartbotic::database {
+
+// Namespace alias for proto types
+namespace pb = smartbotic::databasepb;
+
+/**
+ * Handles the sync protocol for replication between nodes.
+ * Manages bidirectional gRPC streams with peers.
+ */
+class SyncProtocol {
+public:
+    struct PeerConnection {
+        std::string address;
+        std::string nodeId;
+        std::shared_ptr<grpc::Channel> channel;
+        std::unique_ptr<pb::StorageReplication::Stub> stub;
+        std::atomic<bool> connected{false};
+        uint64_t lastWatermark = 0;
+        std::vector<std::string> collections;  // Collections on peer
+        std::unordered_map<std::string, uint64_t> collectionSequences;  // Per-collection sequences
+    };
+
+    using EntryCallback = std::function<void(const pb::ReplicationEntry&)>;
+
+    struct Config {
+        std::string nodeId;
+        std::vector<std::string> peerAddresses;
+        uint32_t reconnectIntervalMs = 5000;
+        uint32_t heartbeatIntervalMs = 1000;
+    };
+
+    explicit SyncProtocol(Config config);
+    ~SyncProtocol();
+
+    /**
+     * Start the sync protocol.
+     */
+    void start(EntryCallback onEntry);
+
+    /**
+     * Stop the sync protocol.
+     */
+    void stop();
+
+    /**
+     * Send an entry to all connected peers.
+     */
+    void broadcast(const pb::ReplicationEntry& entry);
+
+    /**
+     * Get current watermark for a peer.
+     */
+    uint64_t getPeerWatermark(const std::string& nodeId) const;
+
+    /**
+     * Check if connected to a specific peer.
+     */
+    bool isConnected(const std::string& nodeId) const;
+
+    /**
+     * Get count of connected peers.
+     */
+    size_t connectedPeerCount() const;
+
+    /**
+     * Set the local sequence for comparison during sync.
+     */
+    void setLocalSequence(uint64_t sequence);
+
+    /**
+     * Get the local sequence.
+     */
+    uint64_t getLocalSequence() const;
+
+    /**
+     * Set local collections for comparison during sync.
+     */
+    void setLocalCollections(const std::vector<std::string>& collections);
+
+private:
+    void pullEntriesFromPeer(PeerConnection& peer, uint64_t fromSequence);
+    void connectionLoop();
+    void connectToPeer(PeerConnection& peer);
+    void pollPeerState(PeerConnection& peer);
+    void handlePeerStream(PeerConnection& peer);
+
+    Config config_;
+    std::atomic<bool> running_{false};
+    std::atomic<uint64_t> localSequence_{0};
+
+    std::vector<std::unique_ptr<PeerConnection>> peers_;
+    mutable std::mutex mutex_;
+
+    std::thread connectionThread_;
+    EntryCallback entryCallback_;
+
+    // Local collections for new collection discovery
+    std::set<std::string> localCollections_;
+    std::mutex collectionsMutex_;
+};
+
+} // namespace smartbotic::database