docs: design spec for vector storage (embedding similarity search)

Adds vector_dimension collection option, _vector document field,
SimilaritySearch gRPC RPC with SIMD-accelerated cosine similarity.
In-memory storage with WAL+snapshot persistence.

docs/2026-03-26-vector-storage-design.md

@@ -0,0 +1,160 @@
+# Vector Storage for smartbotic-database
+
+## Goal
+Add embedding vector storage and similarity search to smartbotic-database, enabling semantic search without external dependencies (sqlite-vec, FAISS, etc.).
+
+## Design
+
+### Collection Option
+New `vector_dimension` option at collection creation:
+
+```json
+{
+  "type": "create_collection",
+  "collection": "memories",
+  "options": {
+    "vector_dimension": 4096
+  }
+}
+```
+
+- Fixed dimension per collection (set at creation, immutable)
+- Validated on every document insert/update — mismatched dimensions are rejected
+- `0` = no vector support (default, backward compatible)
+
+### Document Vector Field
+Documents in vector-enabled collections can include a `_vector` field:
+
+```json
+{
+  "id": "memory-123",
+  "content": "The user prefers dark mode",
+  "tags": "preferences",
+  "_vector": [0.123, -0.456, 0.789, ...]
+}
+```
+
+- `_vector` is optional — documents without it are stored normally but excluded from similarity search
+- `_vector` must be a float array of exactly `vector_dimension` length
+- Stored in-memory as contiguous `std::vector<float>` alongside the JSON document
+- Persisted via WAL + snapshots (serialized as raw float bytes, not JSON)
+
+### New gRPC RPC
+
+```protobuf
+// In DatabaseService
+rpc SimilaritySearch(SimilaritySearchRequest) returns (SimilaritySearchResponse);
+
+message SimilaritySearchRequest {
+  string collection = 1;
+  repeated float query_vector = 2;
+  uint32 top_k = 3;
+  float min_score = 4;  // optional minimum cosine similarity threshold (0.0-1.0)
+}
+
+message SimilaritySearchResponse {
+  repeated SimilarityResult results = 1;
+}
+
+message SimilarityResult {
+  string id = 1;
+  float score = 2;
+  bytes data = 3;  // full JSON document (without _vector to save bandwidth)
+}
+```
+
+### Collection Options Proto Update
+
+```protobuf
+message CollectionOptions {
+  bool auto_create_id = 1;
+  uint32 default_ttl_seconds = 2;
+  bool encrypted = 3;
+  repeated string sensitive_fields = 4;
+  uint32 max_versions = 5;
+  uint32 vector_dimension = 6;  // NEW — 0 = disabled
+}
+```
+
+### Implementation Details
+
+#### Storage
+- Vectors stored as `std::vector<float>` in a parallel map: `collection → {doc_id → vector}`
+- Separate from JSON document data (not serialized as JSON array)
+- WAL entries for vector operations: `VEC_PUT(collection, id, float_bytes)`, `VEC_DELETE(collection, id)`
+- Snapshots include vector data as binary sections (LZ4 compressed)
+
+#### Similarity Search (Cosine)
+```
+score = dot(a, b) / (norm(a) * norm(b))
+```
+
+- Brute-force scan over all vectors in collection (simple, correct, fast for <100K documents)
+- SIMD acceleration:
+  - SSE4.1 baseline (128-bit, 4 floats at a time)
+  - AVX2 when available (256-bit, 8 floats at a time)
+  - Runtime detection via `__builtin_cpu_supports`
+- Pre-compute and cache vector norms (invalidate on update)
+- Return top-K results sorted by score descending
+- Optional `min_score` filter (skip results below threshold)
+
+#### Performance Expectations
+- 10K documents × 4096 dimensions: ~160MB memory, <10ms search
+- 100K documents × 4096 dimensions: ~1.6GB memory, <100ms search
+- For ShadowMan's use case (memories + skills, likely <1K documents): sub-millisecond
+
+#### Put/Delete Integration
+- `Put()` with `_vector` field: extract vector, validate dimension, store separately, strip from JSON before document storage
+- `Put()` without `_vector` on vector-enabled collection: store document without vector (excluded from similarity search)
+- `Delete()`: remove both document and vector
+- `Get()` response: does NOT include `_vector` (large, not useful for display). Use `SimilaritySearch` for vector operations.
+
+#### Migration Support
+- `create_collection` with `vector_dimension` option
+- `alter_collection` cannot change `vector_dimension` (would invalidate all vectors)
+- No `insert`/`upsert` with vectors in migrations (vectors come from runtime embedding generation)
+
+### Client Library Update
+
+```cpp
+// New method on Client
+struct SimilarityResult {
+    std::string id;
+    float score;
+    nlohmann::json data;
+};
+
+std::vector<SimilarityResult> similaritySearch(
+    const std::string& collection,
+    const std::vector<float>& query_vector,
+    uint32_t top_k,
+    float min_score = 0.0f);
+
+// Put with vector
+void put(const std::string& collection, nlohmann::json document,
+         const std::vector<float>& vector = {});
+```
+
+### Files Changed
+
+| File | Change |
+|---|---|
+| `proto/database.proto` | Add `vector_dimension` to CollectionOptions, add SimilaritySearch RPC |
+| `service/src/document.hpp` | Add `vector_dimension` to CollectionOptions struct |
+| `service/src/memory_store.hpp` | Add vector storage maps, similarity search method |
+| `service/src/memory_store.cpp` | Implement vector CRUD + cosine similarity with SIMD |
+| `service/src/database_grpc_impl.cpp` | Handle SimilaritySearch RPC, extract `_vector` on Put |
+| `service/src/wal.cpp` | Add VEC_PUT/VEC_DELETE WAL entries |
+| `service/src/snapshot.cpp` | Serialize/deserialize vectors in snapshots |
+| `client/src/client.cpp` | Add `similaritySearch()` and vector-aware `put()` |
+| `client/include/smartbotic/database/client.hpp` | Client API additions |
+
+### Testing
+
+- Unit test: cosine similarity correctness (known vectors with known scores)
+- Unit test: dimension validation (reject wrong-sized vectors)
+- Unit test: Put with vector, SimilaritySearch returns it
+- Unit test: Delete removes vector
+- Unit test: WAL replay preserves vectors
+- Unit test: Snapshot preserves vectors
+- Performance test: 10K vectors × 4096 dims, verify <10ms search