Enhance README with comprehensive documentation

- Add client library usage examples
- Add gRPC API documentation
- Add complete migration operation reference
- Add deployment patterns (standalone, bundled, HA)
- Add configuration reference table
- Add health check examples
- Add systemd service example

README.md

@@ -6,7 +6,7 @@ A standalone key-value document database with gRPC API, designed for microservic
 
 - **Document Storage** - JSON document store with collections
 - **Health Check** - gRPC `HealthCheck()` RPC for service status verification
-- **Replication** - Multi-master replication via `StorageReplication` gRPC service
+- **Replication** - Multi-master replication via `DatabaseReplication` gRPC service
 - **Persistence** - WAL (Write-Ahead Log) + snapshots with LZ4 compression
 - **Encryption** - Field-level AES-256-GCM encryption for sensitive data
 - **Events** - Pub/sub event subscription for real-time updates
@@ -23,89 +23,406 @@ smartbotic-database/
 └── packaging/        # Docker build and systemd files
 ```
 
+### Component Separation
+
+| Component | Target | Description |
+|-----------|--------|-------------|
+| **Client Library** | `smartbotic-db-client` | Lightweight gRPC client, injectable via submodule |
+| **Database Service** | `smartbotic-database` | Full server with persistence, encryption, replication |
+| **Proto Library** | `smartbotic_db_proto` | Generated protobuf/gRPC code |
+
 ## Building
 
+### Prerequisites
+
+```bash
+# Debian/Ubuntu
+sudo apt install cmake ninja-build \
+    protobuf-compiler libprotobuf-dev \
+    libgrpc++-dev protobuf-compiler-grpc \
+    nlohmann-json3-dev libspdlog-dev libssl-dev liblz4-dev
+```
+
 ### Standalone Build
 
 ```bash
 cmake -B build -G Ninja
 ninja -C build
+
+# Run the database
+./build/service/smartbotic-database --config config/database.json
 ```
 
 ### As Embedded Submodule
 
-When used as a submodule, configure build options:
+When used as a git submodule in another project:
+
+```bash
+# Add submodule
+git submodule add ssh://git@git.smartbotics.ai:10022/fszontagh/smartbotic-database.git external/smartbotic-database
+```
+
+Configure in your CMakeLists.txt:
 
 ```cmake
-set(SMARTBOTIC_DB_BUILD_CLIENT ON)   # Always need client
-set(SMARTBOTIC_DB_BUILD_SERVICE ON)  # Optional: include service binary
+# Build options
+set(SMARTBOTIC_DB_BUILD_CLIENT ON CACHE BOOL "" FORCE)   # Always need client
+set(SMARTBOTIC_DB_BUILD_SERVICE ON CACHE BOOL "" FORCE)  # Optional: include service
+
 add_subdirectory(external/smartbotic-database)
+
+# Link to client library
+target_link_libraries(your-service PRIVATE smartbotic-db-client)
+```
+
+## Client Library Usage
+
+### Basic Operations
+
+```cpp
+#include <smartbotic/database/client.hpp>
+
+// Connect to database
+smartbotic::database::Client client("localhost:9004");
+
+// Create a document
+nlohmann::json user = {
+    {"name", "John Doe"},
+    {"email", "john@example.com"}
+};
+auto id = client.put("users", user);
+
+// Get a document
+auto doc = client.get("users", id);
+
+// Query documents
+smartbotic::database::QueryOptions opts;
+opts.filter = {{"name", "John Doe"}};
+auto results = client.query("users", opts);
+
+// Delete a document
+client.remove("users", id);
 ```
 
+### Working with Files
+
+```cpp
+// Upload a file
+std::vector<uint8_t> audioData = loadAudioFile("recording.wav");
+auto fileId = client.uploadFile("call-recordings", "recording.wav", "audio/wav", audioData);
+
+// Download a file
+auto [data, metadata] = client.downloadFile(fileId);
+
+// Delete a file
+client.deleteFile(fileId);
+```
+
+### Subscribing to Events
+
+```cpp
+// Subscribe to collection events
+client.subscribe("users", [](const smartbotic::database::Event& event) {
+    if (event.type == smartbotic::database::EventType::Created) {
+        std::cout << "New user created: " << event.documentId << std::endl;
+    }
+});
+```
+
+## gRPC API
+
+### DatabaseService
+
+| RPC | Description |
+|-----|-------------|
+| `HealthCheck` | Check service health and status |
+| `Get` | Retrieve document by ID |
+| `Put` | Create or update document |
+| `Delete` | Remove document by ID |
+| `Query` | Query documents with filters |
+| `Watch` | Subscribe to document changes |
+| `ListCollections` | List all collections |
+| `UploadFile` | Upload binary file (streaming) |
+| `DownloadFile` | Download binary file (streaming) |
+| `DeleteFile` | Delete file by ID |
+| `GetFileMetadata` | Get file metadata |
+
+### DatabaseReplication
+
+| RPC | Description |
+|-----|-------------|
+| `Sync` | Bidirectional sync stream |
+| `GetChanges` | Get changes since sequence number |
+| `ApplyChanges` | Apply changes from peer |
+
 ## Configuration
 
 ```json
 {
+  "log_level": "info",
   "database": {
     "bind_address": "0.0.0.0",
     "rpc_port": 9004,
+    "node_id": "db-node-1",
     "data_directory": "/var/lib/smartbotic/database",
     "migrations": {
       "enabled": true,
       "directory": "/etc/smartbotic/migrations",
-      "auto_apply": true
+      "auto_apply": true,
+      "fail_on_error": true
     },
     "persistence": {
       "wal_sync_interval_ms": 100,
-      "snapshot_interval_sec": 3600
+      "snapshot_interval_sec": 3600,
+      "compression": "lz4"
     },
     "encryption": {
       "enabled": true,
+      "key_file": "/var/lib/smartbotic/database/storage.key",
       "auto_generate_key": true
     },
+    "files": {
+      "max_file_size_mb": 500,
+      "allowed_types": ["audio/*", "application/octet-stream"],
+      "cleanup_orphans_interval_sec": 3600
+    },
     "replication": {
       "enabled": false,
-      "peers": []
+      "peers": ["db-node-2.example.com:9004"],
+      "conflict_resolution": "last_writer_wins"
     }
   }
 }
 ```
 
-## Migrations
+### Configuration Reference
+
+| Section | Option | Default | Description |
+|---------|--------|---------|-------------|
+| `database` | `bind_address` | `0.0.0.0` | Interface to bind gRPC server |
+| `database` | `rpc_port` | `9004` | gRPC server port |
+| `database` | `node_id` | `db-1` | Unique node identifier for replication |
+| `database` | `data_directory` | `/var/lib/smartbotic/database` | Data storage path |
+| `migrations` | `enabled` | `true` | Enable migration system |
+| `migrations` | `directory` | - | Path to migration files |
+| `migrations` | `auto_apply` | `true` | Auto-apply pending migrations on startup |
+| `migrations` | `fail_on_error` | `true` | Stop if migration fails |
+| `persistence` | `wal_sync_interval_ms` | `100` | WAL sync frequency |
+| `persistence` | `snapshot_interval_sec` | `3600` | Snapshot creation interval |
+| `persistence` | `compression` | `lz4` | Compression algorithm |
+| `encryption` | `enabled` | `false` | Enable field-level encryption |
+| `encryption` | `key_file` | - | Path to encryption key |
+| `encryption` | `auto_generate_key` | `false` | Generate key if missing |
+| `replication` | `enabled` | `false` | Enable multi-master replication |
+| `replication` | `peers` | `[]` | List of peer node addresses |
+| `replication` | `conflict_resolution` | `last_writer_wins` | Conflict resolution strategy |
+
+## Migration System
 
-Create JSON migration files in the migrations directory:
+### Creating Migrations
 
+Create JSON migration files in the migrations directory. Files are processed in alphabetical order (use numeric prefixes).
+
+**001_create_users.json:**
 ```json
 {
   "version": "001",
   "name": "create_users",
-  "description": "Create users collection",
+  "description": "Create users collection with encryption",
   "operations": [
     {
       "type": "create_collection",
       "collection": "users",
-      "options": {"encrypted": true}
+      "options": {
+        "encrypted": true,
+        "sensitive_fields": ["password_hash", "api_key"]
+      }
     }
   ]
 }
 ```
 
-Supported operations:
-- `create_collection` - Create collection if not exists
-- `drop_collection` - Drop collection (requires `confirm: true`)
-- `insert` - Insert document (fails if exists)
-- `insert_if_not_exists` - Insert only if ID doesn't exist
-- `upsert` - Insert or update document
-- `update` - Update existing document
-- `delete` - Delete document
+**002_seed_admin.json:**
+```json
+{
+  "version": "002",
+  "name": "seed_admin",
+  "description": "Create default admin user",
+  "operations": [
+    {
+      "type": "insert_if_not_exists",
+      "collection": "users",
+      "id": "admin",
+      "data": {
+        "username": "admin",
+        "role": "administrator",
+        "created_at": "{{NOW}}"
+      }
+    }
+  ]
+}
+```
+
+### Migration Operations
+
+| Operation | Description | Required Fields |
+|-----------|-------------|-----------------|
+| `create_collection` | Create collection if not exists | `collection`, `options` (optional) |
+| `drop_collection` | Drop collection | `collection`, `confirm: true` |
+| `insert` | Insert document (fails if exists) | `collection`, `id`, `data` |
+| `insert_if_not_exists` | Insert only if ID doesn't exist | `collection`, `id`, `data` |
+| `upsert` | Insert or update document | `collection`, `id`, `data` |
+| `update` | Update existing document | `collection`, `id`, `data` |
+| `delete` | Delete document | `collection`, `id` |
+
+### Migration Tracking
+
+Applied migrations are tracked in the `_migrations` collection:
+
+```json
+{
+  "id": "001_create_users",
+  "version": "001",
+  "name": "create_users",
+  "applied_at": "2024-01-15T10:30:00Z",
+  "status": "applied",
+  "checksum": "sha256:abc123..."
+}
+```
+
+### Template Variables
+
+| Variable | Description |
+|----------|-------------|
+| `{{NOW}}` | Current ISO 8601 timestamp |
+| `{{NODE_ID}}` | Database node identifier |
+| `{{VERSION}}` | Database version |
+
+## Deployment Patterns
+
+### Pattern 1: Standalone Database
+
+Deploy as a separate service:
+
+```
+┌──────────────────┐         ┌──────────────────┐
+│   Your App       │──gRPC──▶│ smartbotic-db    │
+│   (client lib)   │         │ (separate node)  │
+└──────────────────┘         └──────────────────┘
+```
+
+```bash
+# Build and package database separately
+cd packaging
+./build.sh
+# Deploy smartbotic-database-x.y.z.tar.gz to database servers
+```
+
+### Pattern 2: Bundled with Application
+
+Include database binary in your application package:
+
+```cmake
+# In your CMakeLists.txt
+set(SMARTBOTIC_DB_BUILD_SERVICE ON)
+add_subdirectory(external/smartbotic-database)
+```
+
+Both binaries deployed together:
+```
+/opt/myapp/
+├── bin/
+│   ├── myapp
+│   └── smartbotic-database
+└── etc/
+    └── database.json
+```
+
+### Pattern 3: High Availability
+
+Multiple database nodes with replication:
+
+```
+┌─────────────┐     ┌─────────────┐     ┌─────────────┐
+│  db-node-1  │◄───▶│  db-node-2  │◄───▶│  db-node-3  │
+│  (primary)  │     │  (replica)  │     │  (replica)  │
+└─────────────┘     └─────────────┘     └─────────────┘
+       ▲                  ▲                   ▲
+       │                  │                   │
+       └──────────────────┴───────────────────┘
+                    Load Balancer
+```
+
+Enable replication in config:
+```json
+{
+  "replication": {
+    "enabled": true,
+    "peers": ["db-node-2:9004", "db-node-3:9004"],
+    "conflict_resolution": "last_writer_wins"
+  }
+}
+```
+
+## Health Checking
+
+```bash
+# Using grpcurl
+grpcurl -plaintext localhost:9004 smartbotic.databasepb.DatabaseService/HealthCheck
+
+# Response
+{
+  "healthy": true,
+  "nodeId": "db-node-1",
+  "uptime": 86400,
+  "documentsCount": 15234,
+  "collectionsCount": 8,
+  "diskUsageBytes": 52428800
+}
+```
+
+## Systemd Service
+
+```ini
+# /etc/systemd/system/smartbotic-database.service
+[Unit]
+Description=Smartbotic Database Service
+After=network.target
+
+[Service]
+Type=simple
+User=smartbotic
+ExecStart=/opt/smartbotic/bin/smartbotic-database --config /etc/smartbotic/database.json
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target
+```
 
 ## CMake Targets
 
-| Target | Description |
-|--------|-------------|
-| `smartbotic_db_proto` | Generated protobuf/gRPC code |
-| `smartbotic-db-client` | Client library |
-| `smartbotic-database` | Service executable |
+| Target | Description | Build Option |
+|--------|-------------|--------------|
+| `smartbotic_db_proto` | Generated protobuf/gRPC code | Always |
+| `smartbotic-db-client` | Client library | `SMARTBOTIC_DB_BUILD_CLIENT=ON` |
+| `smartbotic-database` | Service executable | `SMARTBOTIC_DB_BUILD_SERVICE=ON` |
+
+## Modifying from Consumer Project
+
+When using as a submodule, you can modify the database code directly:
+
+```bash
+cd external/smartbotic-database
+# Make changes
+git add -A && git commit -m "Fix: description"
+git push origin main
+
+# Update submodule reference in parent project
+cd ../..
+git add external/smartbotic-database
+git commit -m "Update smartbotic-database submodule"
+```
 
 ## License