# Vector Storage for smartbotic-database

## Goal
Add embedding vector storage and similarity search to smartbotic-database, enabling semantic search without external dependencies (sqlite-vec, FAISS, etc.).

## Design

### Collection Option
New `vector_dimension` option at collection creation:

```json
{
  "type": "create_collection",
  "collection": "memories",
  "options": {
    "vector_dimension": 4096
  }
}
```

- Fixed dimension per collection (set at creation, immutable)
- Validated on every document insert/update — mismatched dimensions are rejected
- `0` = no vector support (default, backward compatible)

### Document Vector Field
Documents in vector-enabled collections can include a `_vector` field:

```json
{
  "id": "memory-123",
  "content": "The user prefers dark mode",
  "tags": "preferences",
  "_vector": [0.123, -0.456, 0.789, ...]
}
```

- `_vector` is optional — documents without it are stored normally but excluded from similarity search
- `_vector` must be a float array of exactly `vector_dimension` length
- Stored in-memory as contiguous `std::vector<float>` alongside the JSON document
- Persisted via WAL + snapshots (serialized as raw float bytes, not JSON)

### New gRPC RPC

```protobuf
// In DatabaseService
rpc SimilaritySearch(SimilaritySearchRequest) returns (SimilaritySearchResponse);

message SimilaritySearchRequest {
  string collection = 1;
  repeated float query_vector = 2;
  uint32 top_k = 3;
  float min_score = 4;  // optional minimum cosine similarity threshold (0.0-1.0)
}

message SimilaritySearchResponse {
  repeated SimilarityResult results = 1;
}

message SimilarityResult {
  string id = 1;
  float score = 2;
  bytes data = 3;  // full JSON document (without _vector to save bandwidth)
}
```

### Collection Options Proto Update

```protobuf
message CollectionOptions {
  bool auto_create_id = 1;
  uint32 default_ttl_seconds = 2;
  bool encrypted = 3;
  repeated string sensitive_fields = 4;
  uint32 max_versions = 5;
  uint32 vector_dimension = 6;  // NEW — 0 = disabled
}
```

### Implementation Details

#### Storage
- Vectors stored as `std::vector<float>` in a parallel map: `collection → {doc_id → vector}`
- Separate from JSON document data (not serialized as JSON array)
- WAL entries for vector operations: `VEC_PUT(collection, id, float_bytes)`, `VEC_DELETE(collection, id)`
- Snapshots include vector data as binary sections (LZ4 compressed)

#### Similarity Search (Cosine)
```
score = dot(a, b) / (norm(a) * norm(b))
```

- Brute-force scan over all vectors in collection (simple, correct, fast for <100K documents)
- SIMD acceleration:
  - SSE4.1 baseline (128-bit, 4 floats at a time)
  - AVX2 when available (256-bit, 8 floats at a time)
  - Runtime detection via `__builtin_cpu_supports`
- Pre-compute and cache vector norms (invalidate on update)
- Return top-K results sorted by score descending
- Optional `min_score` filter (skip results below threshold)

#### Performance Expectations
- 10K documents × 4096 dimensions: ~160MB memory, <10ms search
- 100K documents × 4096 dimensions: ~1.6GB memory, <100ms search
- For ShadowMan's use case (memories + skills, likely <1K documents): sub-millisecond

#### Put/Delete Integration
- `Put()` with `_vector` field: extract vector, validate dimension, store separately, strip from JSON before document storage
- `Put()` without `_vector` on vector-enabled collection: store document without vector (excluded from similarity search)
- `Delete()`: remove both document and vector
- `Get()` response: does NOT include `_vector` (large, not useful for display). Use `SimilaritySearch` for vector operations.

#### Migration Support
- `create_collection` with `vector_dimension` option
- `alter_collection` cannot change `vector_dimension` (would invalidate all vectors)
- No `insert`/`upsert` with vectors in migrations (vectors come from runtime embedding generation)

### Client Library Update

```cpp
// New method on Client
struct SimilarityResult {
    std::string id;
    float score;
    nlohmann::json data;
};

std::vector<SimilarityResult> similaritySearch(
    const std::string& collection,
    const std::vector<float>& query_vector,
    uint32_t top_k,
    float min_score = 0.0f);

// Put with vector
void put(const std::string& collection, nlohmann::json document,
         const std::vector<float>& vector = {});
```

### Files Changed

| File | Change |
|---|---|
| `proto/database.proto` | Add `vector_dimension` to CollectionOptions, add SimilaritySearch RPC |
| `service/src/document.hpp` | Add `vector_dimension` to CollectionOptions struct |
| `service/src/memory_store.hpp` | Add vector storage maps, similarity search method |
| `service/src/memory_store.cpp` | Implement vector CRUD + cosine similarity with SIMD |
| `service/src/database_grpc_impl.cpp` | Handle SimilaritySearch RPC, extract `_vector` on Put |
| `service/src/wal.cpp` | Add VEC_PUT/VEC_DELETE WAL entries |
| `service/src/snapshot.cpp` | Serialize/deserialize vectors in snapshots |
| `client/src/client.cpp` | Add `similaritySearch()` and vector-aware `put()` |
| `client/include/smartbotic/database/client.hpp` | Client API additions |

### Testing

- Unit test: cosine similarity correctness (known vectors with known scores)
- Unit test: dimension validation (reject wrong-sized vectors)
- Unit test: Put with vector, SimilaritySearch returns it
- Unit test: Delete removes vector
- Unit test: WAL replay preserves vectors
- Unit test: Snapshot preserves vectors
- Performance test: 10K vectors × 4096 dims, verify <10ms search