Vector Storage for smartbotic-database

Goal

Add embedding vector storage and similarity search to smartbotic-database, enabling semantic search without external dependencies (sqlite-vec, FAISS, etc.).

Design

Collection Option

New vector_dimension option at collection creation:

{
  "type": "create_collection",
  "collection": "memories",
  "options": {
    "vector_dimension": 4096
  }
}

Fixed dimension per collection (set at creation, immutable)
Validated on every document insert/update — mismatched dimensions are rejected
0 = no vector support (default, backward compatible)

Document Vector Field

Documents in vector-enabled collections can include a _vector field:

{
  "id": "memory-123",
  "content": "The user prefers dark mode",
  "tags": "preferences",
  "_vector": [0.123, -0.456, 0.789, ...]
}

_vector is optional — documents without it are stored normally but excluded from similarity search
_vector must be a float array of exactly vector_dimension length
Stored in-memory as contiguous std::vector<float> alongside the JSON document
Persisted via WAL + snapshots (serialized as raw float bytes, not JSON)

New gRPC RPC

// In DatabaseService
rpc SimilaritySearch(SimilaritySearchRequest) returns (SimilaritySearchResponse);

message SimilaritySearchRequest {
  string collection = 1;
  repeated float query_vector = 2;
  uint32 top_k = 3;
  float min_score = 4;  // optional minimum cosine similarity threshold (0.0-1.0)
}

message SimilaritySearchResponse {
  repeated SimilarityResult results = 1;
}

message SimilarityResult {
  string id = 1;
  float score = 2;
  bytes data = 3;  // full JSON document (without _vector to save bandwidth)
}

Collection Options Proto Update

message CollectionOptions {
  bool auto_create_id = 1;
  uint32 default_ttl_seconds = 2;
  bool encrypted = 3;
  repeated string sensitive_fields = 4;
  uint32 max_versions = 5;
  uint32 vector_dimension = 6;  // NEW — 0 = disabled
}

Implementation Details

Storage

Vectors stored as std::vector<float> in a parallel map: collection → {doc_id → vector}
Separate from JSON document data (not serialized as JSON array)
WAL entries for vector operations: VEC_PUT(collection, id, float_bytes), VEC_DELETE(collection, id)
Snapshots include vector data as binary sections (LZ4 compressed)

Similarity Search (Cosine)

score = dot(a, b) / (norm(a) * norm(b))

Brute-force scan over all vectors in collection (simple, correct, fast for <100K documents)
SIMD acceleration:
- SSE4.1 baseline (128-bit, 4 floats at a time)
- AVX2 when available (256-bit, 8 floats at a time)
- Runtime detection via __builtin_cpu_supports
Pre-compute and cache vector norms (invalidate on update)
Return top-K results sorted by score descending
Optional min_score filter (skip results below threshold)

Performance Expectations

10K documents × 4096 dimensions: ~160MB memory, <10ms search
100K documents × 4096 dimensions: ~1.6GB memory, <100ms search
For ShadowMan's use case (memories + skills, likely <1K documents): sub-millisecond

Put/Delete Integration

Put() with _vector field: extract vector, validate dimension, store separately, strip from JSON before document storage
Put() without _vector on vector-enabled collection: store document without vector (excluded from similarity search)
Delete(): remove both document and vector
Get() response: does NOT include _vector (large, not useful for display). Use SimilaritySearch for vector operations.

Migration Support

create_collection with vector_dimension option
alter_collection cannot change vector_dimension (would invalidate all vectors)
No insert/upsert with vectors in migrations (vectors come from runtime embedding generation)

Client Library Update

// New method on Client
struct SimilarityResult {
    std::string id;
    float score;
    nlohmann::json data;
};

std::vector<SimilarityResult> similaritySearch(
    const std::string& collection,
    const std::vector<float>& query_vector,
    uint32_t top_k,
    float min_score = 0.0f);

// Put with vector
void put(const std::string& collection, nlohmann::json document,
         const std::vector<float>& vector = {});

Files Changed

File	Change
`proto/database.proto`	Add `vector_dimension` to CollectionOptions, add SimilaritySearch RPC
`service/src/document.hpp`	Add `vector_dimension` to CollectionOptions struct
`service/src/memory_store.hpp`	Add vector storage maps, similarity search method
`service/src/memory_store.cpp`	Implement vector CRUD + cosine similarity with SIMD
`service/src/database_grpc_impl.cpp`	Handle SimilaritySearch RPC, extract `_vector` on Put
`service/src/wal.cpp`	Add VEC_PUT/VEC_DELETE WAL entries
`service/src/snapshot.cpp`	Serialize/deserialize vectors in snapshots
`client/src/client.cpp`	Add `similaritySearch()` and vector-aware `put()`
`client/include/smartbotic/database/client.hpp`	Client API additions

Testing

Unit test: cosine similarity correctness (known vectors with known scores)
Unit test: dimension validation (reject wrong-sized vectors)
Unit test: Put with vector, SimilaritySearch returns it
Unit test: Delete removes vector
Unit test: WAL replay preserves vectors
Unit test: Snapshot preserves vectors
Performance test: 10K vectors × 4096 dims, verify <10ms search

2026-03-26-vector-storage-design.md 5.4 KB History Raw