소스 검색

docs(perf): record implementation status of the latency tunables

Fszontagh 1 개월 전
부모
커밋
d72e39a193
1개의 변경된 파일31개의 추가작업 그리고 0개의 파일을 삭제
  1. 31 0
      docs/performance/2026-06-04-search-latency-and-tunables.md

+ 31 - 0
docs/performance/2026-06-04-search-latency-and-tunables.md

@@ -267,3 +267,34 @@ RAG performance test
 ```
 
 Larger samples produce essentially the same distribution — the variance is in upstream embedding tail, not in our client.
+
+## Implementation status (2026-06-04)
+
+All six options landed on branch `feat/embedding-perf-tuning`. Design spec:
+[`docs/superpowers/specs/2026-06-04-embedding-perf-design.md`](../superpowers/specs/2026-06-04-embedding-perf-design.md).
+
+- **Option 1 — LRU embedding cache:** `src/embedding_cache.{hpp,cpp}`, wired into the
+  shared `resolveVector()` (covers `/search` *and* `/vectors` insert). Keyed on
+  `(apiBase, model, text)` — apiBase included so a provider repoint can't serve
+  stale vectors. Bounded by entry count **and** byte budget; TTL; opt-in
+  trim+lowercase normalization.
+- **Option 2 — persistent client:** a per-origin **pool** of N keep-alive
+  `httplib::Client`s (`src/embeddings.cpp`), not a single client behind a mutex —
+  concurrent embeds run on independent sockets (verified by a max-in-flight test).
+- **Option 3 — configurable timeouts:** `embedding_connect_timeout_sec`,
+  `embedding_read_timeout_sec` (read default raised 30→60 s).
+- **Option 4 — telemetry:** `GET /api/v1/stats` now returns an `embedding` object
+  (hits/misses/hit_ratio/size/bytes/capacity).
+- **Option 5 — escape hatch:** `Cache-Control: no-store` on `/search` and
+  `/vectors` bypasses the cache (no read, no write).
+- **Option 6 — docs:** `query_vector` documented as the low-latency fast path in
+  `api/openapi.json` + `api/llms.txt`.
+
+New admin settings (all hot-reload, exposed in the web UI settings page):
+`embedding_cache_size`, `embedding_cache_ttl_sec`, `embedding_cache_max_bytes`,
+`embedding_cache_normalize`, `embedding_client_pool_size`, plus the two timeouts.
+
+**Re-measurement is pending** a controlled run against a live endpoint — the
+numbers above are the pre-change baseline. The cache's real-world win depends on
+the exact-text repeat rate of the workload; the new `/stats` hit-ratio is the
+intended way to measure it in production rather than asserting a speedup here.