|
|
@@ -294,7 +294,34 @@ New admin settings (all hot-reload, exposed in the web UI settings page):
|
|
|
`embedding_cache_size`, `embedding_cache_ttl_sec`, `embedding_cache_max_bytes`,
|
|
|
`embedding_cache_normalize`, `embedding_client_pool_size`, plus the two timeouts.
|
|
|
|
|
|
-**Re-measurement is pending** a controlled run against a live endpoint — the
|
|
|
-numbers above are the pre-change baseline. The cache's real-world win depends on
|
|
|
-the exact-text repeat rate of the workload; the new `/stats` hit-ratio is the
|
|
|
-intended way to measure it in production rather than asserting a speedup here.
|
|
|
+### Measured after deployment (v0.1.5, 2026-06-04)
|
|
|
+
|
|
|
+`perf-test-rag.py` re-run against the upgraded prod
|
|
|
+(`https://rag001.smartbotics.ai`, `sungrow`/`datasheets`, `top_k=5`,
|
|
|
+`min_score=0.3`, warm-up disabled so the cold/hot split is clean; the embedding
|
|
|
+cache was empty at start — service freshly restarted).
|
|
|
+
|
|
|
+| run | what | p50 | mean | p95 | max | q/s | errors |
|
|
|
+|---|---|---|---|---|---|---|---|
|
|
|
+| cold, c=1 | 20 unique queries — all cache **misses** | 2355 ms | 3846 ms | 11134 ms | 14974 ms | 0.3 | 0/20 |
|
|
|
+| hot, c=1 | same 20 queries — all cache **hits** | 20 ms | 22 ms | 27 ms | 49 ms | 45.1 | 0/20 |
|
|
|
+| warm, c=5 | 60 runs, warm cache + client pool | 24 ms | 28 ms | 57 ms | 97 ms | 173.8 | 0/60 |
|
|
|
+| warm, c=10 | 60 runs, warm cache + client pool | 29 ms | 29 ms | 37 ms | 65 ms | 319.8 | 0/60 |
|
|
|
+
|
|
|
+Takeaways:
|
|
|
+
|
|
|
+- **Cache hit ≈ 118× faster than a miss** (p50 2355 ms → 20 ms). A hot query now
|
|
|
+ costs only the TLS hop + cosine search — matching the predicted "<50 ms".
|
|
|
+- **Cold path is unchanged** (still embedding-bound, same multi-second,
|
|
|
+ high-variance regime as the baseline) — the cache does not speed up first-time
|
|
|
+ queries, by design.
|
|
|
+- **Throughput at concurrency 10: ~320 q/s with 0 errors**, versus the pre-change
|
|
|
+ baseline's ~0.3 q/s and **17 % errors** at c=10. The per-origin client pool
|
|
|
+ scales cleanly (44.7 → 173.8 → 319.8 q/s at c=1/5/10) rather than serializing
|
|
|
+ or overwhelming the upstream — the configurable read timeout also eliminated the
|
|
|
+ timeout errors.
|
|
|
+- Samples remain small (20–60) — treat percentiles as directional; the cold-path
|
|
|
+ variance is upstream-embedding tail, per the original analysis.
|
|
|
+
|
|
|
+The realized production win depends on how often identical `query_text` recurs;
|
|
|
+watch `GET /api/v1/stats` → `embedding.cache_hit_ratio` to see it in the live mix.
|