|
|
@@ -267,3 +267,34 @@ RAG performance test
|
|
|
```
|
|
|
|
|
|
Larger samples produce essentially the same distribution — the variance is in upstream embedding tail, not in our client.
|
|
|
+
|
|
|
+## Implementation status (2026-06-04)
|
|
|
+
|
|
|
+All six options landed on branch `feat/embedding-perf-tuning`. Design spec:
|
|
|
+[`docs/superpowers/specs/2026-06-04-embedding-perf-design.md`](../superpowers/specs/2026-06-04-embedding-perf-design.md).
|
|
|
+
|
|
|
+- **Option 1 — LRU embedding cache:** `src/embedding_cache.{hpp,cpp}`, wired into the
|
|
|
+ shared `resolveVector()` (covers `/search` *and* `/vectors` insert). Keyed on
|
|
|
+ `(apiBase, model, text)` — apiBase included so a provider repoint can't serve
|
|
|
+ stale vectors. Bounded by entry count **and** byte budget; TTL; opt-in
|
|
|
+ trim+lowercase normalization.
|
|
|
+- **Option 2 — persistent client:** a per-origin **pool** of N keep-alive
|
|
|
+ `httplib::Client`s (`src/embeddings.cpp`), not a single client behind a mutex —
|
|
|
+ concurrent embeds run on independent sockets (verified by a max-in-flight test).
|
|
|
+- **Option 3 — configurable timeouts:** `embedding_connect_timeout_sec`,
|
|
|
+ `embedding_read_timeout_sec` (read default raised 30→60 s).
|
|
|
+- **Option 4 — telemetry:** `GET /api/v1/stats` now returns an `embedding` object
|
|
|
+ (hits/misses/hit_ratio/size/bytes/capacity).
|
|
|
+- **Option 5 — escape hatch:** `Cache-Control: no-store` on `/search` and
|
|
|
+ `/vectors` bypasses the cache (no read, no write).
|
|
|
+- **Option 6 — docs:** `query_vector` documented as the low-latency fast path in
|
|
|
+ `api/openapi.json` + `api/llms.txt`.
|
|
|
+
|
|
|
+New admin settings (all hot-reload, exposed in the web UI settings page):
|
|
|
+`embedding_cache_size`, `embedding_cache_ttl_sec`, `embedding_cache_max_bytes`,
|
|
|
+`embedding_cache_normalize`, `embedding_client_pool_size`, plus the two timeouts.
|
|
|
+
|
|
|
+**Re-measurement is pending** a controlled run against a live endpoint — the
|
|
|
+numbers above are the pre-change baseline. The cache's real-world win depends on
|
|
|
+the exact-text repeat rate of the workload; the new `/stats` hit-ratio is the
|
|
|
+intended way to measure it in production rather than asserting a speedup here.
|