docs(v1.7.0): conf.d + eviction & pressure + client retry (T13)

Comprehensive doc for v1.7.0's eviction-resilience work:

- Drop-in configuration section: RFC 7396 semantics, file naming,
  failure modes, logging
- Eviction & Memory Pressure section: four pressure levels, chunked
  eviction, per-collection priority, full config reference, events,
  client retry, observability, troubleshooting
- CLAUDE.md Key Features bullets for all three new subsystems

Written from the operator's perspective — what to put in conf.d,
what the logs mean, how to tune for a 2 GB shadowman workload.

CLAUDE.md

@@ -35,6 +35,9 @@ cmake --build build -j$(nproc) && ./build/tests/test_vector_storage
 - **Per-collection Timestamp Precision** — collection-level `timestamp_precision` config ("ms" default, "ns" for rapid-write collections). Stamps `_created_at`/`_updated_at` at the configured resolution, cached hot-path lookup. `configureCollection` RPC flips the config atomically; `migrateCollectionTimestamps` RPC converts existing data during maintenance windows (idempotent via 10^15 threshold).
 - **Durable Snapshots + Tiered Recovery** — atomic snapshot writer (`.tmp` + fsync + rename + post-write verification), loader fallback chain across snapshots, MySQL-style recovery modes (`normal` / `snapshot_fallback` / `wal_only` / `best_effort` / `force_empty`) selectable via `--recovery-mode` flag or `recovery.mode` config. Non-trivial recovery (fallback, WAL-only, forced empty) automatically enters **read-only mode** — operator must `smartbotic-db-cli unlock` or pass `--force-readwrite` to accept writes. Exit code `10` when recovery is refused.
 - **gRPC Concurrency Cap** — bounded inbound memory via gRPC `ResourceQuota` + per-RPC-type stream limits (`Subscribe`, `UploadFile`, `DownloadFile`). Configurable under `storage.grpc`. Excess streams get `RESOURCE_EXHAUSTED` with operator log.
+- **Drop-in config (conf.d)** — `/etc/smartbotic-database/conf.d/*.json` deep-merges over `config.json` using RFC 7396 semantics (arrays replace, objects merge). Lets consumer debs ship tuned settings without touching the upstream conffile. Strict on syntax errors (exit 11), lenient on unknown keys.
+- **Chunked pressure-aware eviction** — Eviction spreads work over multiple ticks with a default `eviction_chunk_size=1000`, `eviction_chunk_pause_ms=50`, `hot_write_floor_ms=30000`. Four pressure levels (normal/soft/hard/emergency) gate behavior: soft = trickle, hard = aggressive + `MEMORY_PRESSURE_HIGH` event, emergency = admission control rejects writes with `RESOURCE_EXHAUSTED`. Per-collection `memory_priority` (low/normal/high) biases selection. Quiesce skips docs with in-flight writes. WAL-fallback no longer holds per-collection mutex during disk I/O.
+- **GetMemoryStats RPC + client retry** — `Client::getMemoryStats()` returns per-collection stats + pressure level. Client library auto-retries write RPCs on `DEADLINE_EXCEEDED`/`RESOURCE_EXHAUSTED`/`UNAVAILABLE` with exponential backoff + jitter (`writeRetries=3`, `writeRetryBackoffMs=100`). Reads are not auto-retried.
 
 ## Packaging
 

docs/integration-guide.md

@@ -13,6 +13,8 @@ How to install, configure, and integrate smartbotic-database into your C++ proje
 - [Docker Build Integration](#docker-build-integration)
 - [CLI Administration](#cli-administration)
 - [Upgrading from Legacy Packages](#upgrading-from-legacy-packages)
+- [Drop-in configuration (conf.d)](#drop-in-configuration-confd)
+- [Eviction & Memory Pressure](#eviction--memory-pressure)
 
 ---
 
@@ -872,6 +874,229 @@ Under `storage.grpc` (added in v1.6.2):
 
 When a concurrency limit is hit, the server returns `RESOURCE_EXHAUSTED` immediately and logs `WARN` with the current / max counts. Clients should retry with backoff.
 
+### Drop-in configuration (conf.d)
+
+In addition to `/etc/smartbotic-database/config.json`, the server loads any `*.json` files from `/etc/smartbotic-database/conf.d/` and deep-merges them into the running config in lexicographic order.
+
+This lets consumer projects (shadowman-cpp, callerai, your own apps) ship their own tuning without touching the upstream `config.json` that `dpkg --configure` guards as a conffile.
+
+#### Merge semantics (RFC 7396)
+
+- **Objects** merge recursively: sibling fields not mentioned in a drop-in are preserved from the lower-priority layer.
+- **Arrays and scalars** are replaced entirely by a higher-priority layer.
+
+Example:
+
+```json
+// /etc/smartbotic-database/config.json
+{
+  "log_level": "info",
+  "storage": {
+    "rpc_port": 9004,
+    "memory": { "max_memory_mb": 512 }
+  }
+}
+
+// /etc/smartbotic-database/conf.d/50-shadowman.json  (shipped by shadowman-cpp's deb)
+{
+  "storage": {
+    "memory": { "max_memory_mb": 2048 }
+  }
+}
+
+// Effective merged config — sibling rpc_port preserved, max_memory_mb overridden
+{
+  "log_level": "info",
+  "storage": {
+    "rpc_port": 9004,
+    "memory": { "max_memory_mb": 2048 }
+  }
+}
+```
+
+#### File naming conventions
+
+Use numeric prefixes for deterministic merge order. Last writer wins.
+
+| File | Owner | Purpose |
+|------|-------|---------|
+| `00-defaults.json` | smartbotic-database (optional) | Baseline conservative tunings |
+| `50-<project>.json` | consumer project's deb postinst | Workload-tuned values |
+| `99-local.json` | operator | Site-specific overrides |
+
+Files are merged in filename-sort order, so `00-` applies before `50-` applies before `99-`.
+
+#### Failure modes
+
+- **Missing base `config.json`** — OK, treated as `{}`
+- **Missing `conf.d/` directory** — OK
+- **Syntax error in any file** — server refuses to start, exit code `11`, log identifies the file + parser location
+- **Unknown keys** — lenient. Individual config consumers may log `WARN` but don't block startup. This lets v1.x consumers drop a `conf.d/` file that mentions a key only v1.(x+1) knows about
+
+#### What gets logged
+
+At startup, the server prints each file as it merges it:
+
+```
+[INFO] Loading base config: /etc/smartbotic-database/config.json
+[INFO] Merging drop-in: /etc/smartbotic-database/conf.d/50-shadowman.json
+[INFO] Merging drop-in: /etc/smartbotic-database/conf.d/99-local.json
+```
+
+If a consumer wonders which tuning values are actually in effect, this is where to look.
+
+### Eviction & Memory Pressure
+
+smartbotic-database holds documents in memory for fast reads. When usage approaches the configured budget (`storage.memory.max_memory_mb`), eviction removes the least-useful docs to free space. Eviction is chunked, throttled, and hot-write-aware so it doesn't block foreground writes — a design requirement after the 2026-04-22 Zoe incident where one-shot eviction stalled LLM streaming writes long enough to miss their gRPC deadline.
+
+#### Pressure levels
+
+The server tracks four pressure levels based on current usage vs. `max_memory_mb`:
+
+| Level | Default % | Behavior |
+|-------|:-:|----------|
+| `normal` | < 70% | No eviction |
+| `soft` | 70–85% | Trickle eviction: one chunk per tick |
+| `hard` | 85–95% | Aggressive eviction: up to `max_eviction_passes_per_trigger / 2` chunks per tick. Fires `MEMORY_PRESSURE_HIGH` event (edge-triggered) |
+| `emergency` | ≥ 95% | All writes rejected with `RESOURCE_EXHAUSTED` until pressure drops below hard. Eviction runs at full rate |
+
+Query current state via `smartbotic-db-cli status` or the `GetMemoryStats` RPC.
+
+#### Chunked eviction
+
+Eviction never stops the world. Each tick (default every `eviction_check_interval_ms` = 5000ms, but reactive to pressure):
+
+1. Collects candidates — docs whose `updatedAt` is older than `hot_write_floor_ms` (30 s default) ago, in non-pinned collections
+2. Sorts by `(memory_priority, lastAccessedAt)` — low-priority + cold goes first
+3. Applies per-collection budget: no one collection contributes more than `(share × 1.5)` of a single chunk
+4. Evicts `eviction_chunk_size` docs (default 1000) in one pass
+5. Sleeps `eviction_chunk_pause_ms` (default 50 ms)
+6. Repeats up to `max_eviction_passes_per_trigger` times, stopping early if target reached
+
+Docs being actively written (quiesce) are skipped via an in-flight-write counter — eviction can't land mid-upsert.
+
+#### Per-collection priority
+
+Use `CollectionOptions.memory_priority` to bias eviction:
+
+| Value | Meaning |
+|-------|---------|
+| `low` | Evicted preferentially — cold archive collections |
+| `normal` (default) | Standard LRU weight |
+| `high` | Evicted last — hot operational collections |
+| (pinned=true on CollectionOptions) | Never evicted |
+
+Set via `createCollection` or migration:
+
+```json
+{
+  "type": "create_collection",
+  "collection": "messages",
+  "options": {
+    "memory_priority": "high"
+  }
+}
+```
+
+#### Config reference (`storage.memory`)
+
+| Setting | Default | Meaning |
+|---------|--------:|---------|
+| `max_memory_mb` | 512 | Memory budget before eviction kicks in |
+| `eviction_chunk_size` | 1000 | Docs evicted per chunk |
+| `eviction_chunk_pause_ms` | 50 | Sleep between chunks within one tick |
+| `max_eviction_passes_per_trigger` | 20 | Safety cap on chunks per tick |
+| `hot_write_floor_ms` | 30000 | Docs updated within this window are unevictable |
+| `memory_soft_percent` | 70 | Enter `soft` pressure |
+| `memory_hard_percent` | 85 | Enter `hard` pressure — fires PRESSURE_HIGH event |
+| `memory_emergency_percent` | 95 | Enter `emergency` — admission control rejects writes |
+| `eviction_burst_threshold` | 10000 | Fire `MEMORY_EVICTION_BURST` event if chunk ≥ this |
+| `eviction_target_percent` | 60 | Stop evicting once usage drops below this |
+| `eviction_check_interval_ms` | 5000 | Wake eviction loop this often |
+
+#### Events
+
+Subscribe to receive memory-related notifications:
+
+| Event type | When | Payload |
+|------------|------|---------|
+| `MEMORY_PRESSURE_HIGH` | Edge-triggered: pressure enters `hard` or `emergency` | `{pressure_level, pressure_percent, estimated_bytes, max_bytes}` |
+| `MEMORY_EVICTION_BURST` | An eviction pass touched ≥ `eviction_burst_threshold` docs | `{evicted_docs, bytes_freed, pressure_level}` |
+
+Both are system-level events (empty collection / id). Subscribers filter by event type.
+
+#### Client-side retry
+
+The client library (v1.7.0+) automatically retries write RPCs that return transient errors:
+
+| Status | Retry? |
+|--------|:-:|
+| `DEADLINE_EXCEEDED` | ✅ |
+| `RESOURCE_EXHAUSTED` (admission control / concurrency cap) | ✅ |
+| `UNAVAILABLE` | ✅ |
+| All others (INVALID_ARGUMENT, FAILED_PRECONDITION, etc.) | ❌ — final |
+
+Config under `Client::Config`:
+
+| Field | Default | Meaning |
+|-------|--------:|---------|
+| `writeRetries` | 3 | Max retry attempts |
+| `writeRetryBackoffMs` | 100 | Initial backoff (exponential: 100 → 200 → 400 …) |
+| `writeRetryMaxBackoffMs` | 5000 | Cap on any single backoff |
+| `writeRetryJitter` | 0.25 | ± 25% jitter |
+
+Writes wrapped: `insert`, `updateIfVersion`, `upsert`, `remove`, `patch`. Reads are NOT auto-retried — callers retry at their discretion.
+
+#### Observability
+
+`GetMemoryStats` RPC (and `Client::getMemoryStats()`) returns:
+
+- Total / max memory bytes, pressure percent + level
+- Per-collection: `document_count`, `estimated_bytes`, `evicted_stub_count`, `priority`
+- Last eviction: timestamp, docs evicted, bytes freed
+
+```cpp
+auto s = db.getMemoryStats();
+std::cout << "Memory: " << s.totalMemoryBytes << "/" << s.maxMemoryBytes
+          << " (" << s.pressurePercent << "% — " << s.pressureLevel << ")" << std::endl;
+for (const auto& c : s.collections) {
+    std::cout << "  " << c.collection << ": " << c.documentCount << " docs, "
+              << c.estimatedBytes << " bytes, priority=" << c.priority << std::endl;
+}
+```
+
+Also shown in `smartbotic-db-cli status` output (if the CLI exposes it; otherwise subscribers can consume the events).
+
+#### Recommended tunings
+
+- **Default (512 MB)** — development or small services with ~100 K docs
+- **2 GB** — production shadowman-cpp profile:
+
+```json
+// /etc/smartbotic-database/conf.d/50-shadowman.json
+{
+  "storage": {
+    "memory": {
+      "max_memory_mb": 2048,
+      "memory_soft_percent": 65,
+      "memory_hard_percent": 80,
+      "eviction_chunk_size": 500,
+      "hot_write_floor_ms": 60000
+    }
+  }
+}
+```
+
+- **8+ GB large** — consider raising `eviction_burst_threshold` to 50000 so you don't get event spam, and consider `hot_write_floor_ms: 120000` if your write paths are slow (e.g. STT-heavy callerai)
+
+#### Troubleshooting
+
+**"Client::insert failed: RESOURCE_EXHAUSTED (memory pressure emergency)"** — server is in emergency pressure, client is retrying. If this persists, raise `max_memory_mb` in a conf.d drop-in or lower `memory_emergency_percent`.
+
+**"Eviction: nothing evictable this pass"** — every candidate is either pinned, hot-written, or in-flight. Usually benign; server will re-try next tick. If persistent, consider reducing `hot_write_floor_ms` or flagging some collections as unpinned/low-priority.
+
+**Pressure stays at `hard` despite quiet writes** — check `eviction_target_percent`; eviction stops when usage drops below this. If you want eviction to drain further during idle, raise `max_memory_mb` or lower the target.
+
 ### Configuring for Your Project
 
 Consumer projects typically enable migrations by configuring the database via their own `postinst` script: