Selaa lähdekoodia

docs(perf): record measured before/after from prod benchmark (v0.1.5)

Cache hit p50 ~118x faster than miss (2355ms -> 20ms); ~320 q/s at c=10 with
0 errors vs baseline ~0.3 q/s and 17% errors.
Fszontagh 1 kuukausi sitten
vanhempi
sitoutus
06e99e59e0
1 muutettua tiedostoa jossa 31 lisäystä ja 4 poistoa
  1. 31 4
      docs/performance/2026-06-04-search-latency-and-tunables.md

+ 31 - 4
docs/performance/2026-06-04-search-latency-and-tunables.md

@@ -294,7 +294,34 @@ New admin settings (all hot-reload, exposed in the web UI settings page):
 `embedding_cache_size`, `embedding_cache_ttl_sec`, `embedding_cache_max_bytes`,
 `embedding_cache_normalize`, `embedding_client_pool_size`, plus the two timeouts.
 
-**Re-measurement is pending** a controlled run against a live endpoint — the
-numbers above are the pre-change baseline. The cache's real-world win depends on
-the exact-text repeat rate of the workload; the new `/stats` hit-ratio is the
-intended way to measure it in production rather than asserting a speedup here.
+### Measured after deployment (v0.1.5, 2026-06-04)
+
+`perf-test-rag.py` re-run against the upgraded prod
+(`https://rag001.smartbotics.ai`, `sungrow`/`datasheets`, `top_k=5`,
+`min_score=0.3`, warm-up disabled so the cold/hot split is clean; the embedding
+cache was empty at start — service freshly restarted).
+
+| run | what | p50 | mean | p95 | max | q/s | errors |
+|---|---|---|---|---|---|---|---|
+| cold, c=1  | 20 unique queries — all cache **misses** | 2355 ms | 3846 ms | 11134 ms | 14974 ms | 0.3 | 0/20 |
+| hot, c=1   | same 20 queries — all cache **hits**     | 20 ms   | 22 ms   | 27 ms    | 49 ms    | 45.1 | 0/20 |
+| warm, c=5  | 60 runs, warm cache + client pool         | 24 ms   | 28 ms   | 57 ms    | 97 ms    | 173.8 | 0/60 |
+| warm, c=10 | 60 runs, warm cache + client pool         | 29 ms   | 29 ms   | 37 ms    | 65 ms    | 319.8 | 0/60 |
+
+Takeaways:
+
+- **Cache hit ≈ 118× faster than a miss** (p50 2355 ms → 20 ms). A hot query now
+  costs only the TLS hop + cosine search — matching the predicted "<50 ms".
+- **Cold path is unchanged** (still embedding-bound, same multi-second,
+  high-variance regime as the baseline) — the cache does not speed up first-time
+  queries, by design.
+- **Throughput at concurrency 10: ~320 q/s with 0 errors**, versus the pre-change
+  baseline's ~0.3 q/s and **17 % errors** at c=10. The per-origin client pool
+  scales cleanly (44.7 → 173.8 → 319.8 q/s at c=1/5/10) rather than serializing
+  or overwhelming the upstream — the configurable read timeout also eliminated the
+  timeout errors.
+- Samples remain small (20–60) — treat percentiles as directional; the cold-path
+  variance is upstream-embedding tail, per the original analysis.
+
+The realized production win depends on how often identical `query_text` recurs;
+watch `GET /api/v1/stats` → `embedding.cache_hit_ratio` to see it in the live mix.