test(load): mixed workload + integrity + pinned-collection tests

Three new standalone load tests covering scenarios the original
load_test didn't touch:

  load_test_mixed     — concurrent reads + writes under pressure;
                        validates WAL-fallback async unlocking (T8) by
                        checking read p99 latency stays bounded while
                        writes + eviction happen
  load_test_integrity — byte-exact integrity of evicted-then-paged-in
                        docs; validates T8 hot-load path doesn't
                        corrupt data across the round trip
  load_test_pinned    — pinned=true collection protection under
                        sustained pressure on non-pinned collections

Each follows the same pattern as the original load_test: standalone
binary, connects to localhost:9005, runs a workload, prints PASS/FAIL.
Not wired into CTest — manual validation tools.

tests/load_test/README.md

@@ -1,8 +1,19 @@
-# Load test — eviction under memory pressure
+# Load tests — eviction under memory pressure
 
-Standalone binary that validates v1.7.0's chunked eviction + admission control + client retry against a real smartbotic-database instance.
+Standalone binaries that validate v1.7.0's chunked eviction + admission control + client retry against a real smartbotic-database instance.
 
-**Not a unit test** — runs a real server on a configurable port, hammers it with concurrent inserts, and reports success/failure counts + a pressure-level timeline.
+**Not unit tests** — each one runs a real server on a configurable port, drives a workload, and reports a PASS/FAIL verdict. Not wired into CTest — these are operator/developer validation tools, not CI.
+
+## Test suite
+
+Four load-test binaries:
+
+- `load_test` — pressure + retry validation (the original)
+- `load_test_mixed` — concurrent reads + writes under pressure
+- `load_test_integrity` — byte-exact integrity of evicted + paged-in docs
+- `load_test_pinned` — `pinned=true` collection protection
+
+All four talk to `localhost:9005` and use `Client::Config` with transport-level retries. They differ only in workload shape and verdict criteria.
 
 ## Prerequisites
 
@@ -85,3 +96,122 @@ Edit `config.json`:
 - **Slow drain**: raise `eviction_chunk_pause_ms` to `200` ms
 - **Admission control disabled**: raise `memory_emergency_percent` to `99`
 - **Bigger budget**: raise `max_memory_mb` to `256` to see if the chunked approach still works at scale
+
+---
+
+## Test 2 — Mixed read/write (`load_test_mixed`)
+
+Validates that reads stay responsive while writers + eviction are running. Three writer threads continuously insert 2 KB docs into a shared list of recent IDs; five reader threads pick random IDs from that list and call `get()`. Reads hit both live docs and evicted-stub IDs, exercising the WAL-fallback paging path.
+
+### Build
+
+```bash
+g++ -std=c++20 -O2 -I/usr/include load_test_mixed.cpp \
+    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
+    -lspdlog -lfmt -pthread -o load_test_mixed
+```
+
+### Run
+
+```bash
+# Same server setup as load_test:
+mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
+smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
+SERVER_PID=$!
+sleep 2
+
+./load_test_mixed [duration_sec=30] [writers=3] [readers=5] [doc_bytes=2048]
+
+kill $SERVER_PID
+```
+
+### PASS criteria
+
+- writer ops/s > 500
+- reader ops/s > 1000 (reads should be faster than writes — no eviction gate on reads)
+- reader p99 latency < 1000 ms
+- zero reader hard failures (RESOURCE_EXHAUSTED / timeout / other)
+
+`not-found` is counted separately from failures — it's a legitimate cold miss from a freshly-evicted stub where the WAL page hasn't been pulled yet, not a bug.
+
+---
+
+## Test 3 — Data integrity (`load_test_integrity`)
+
+Writes 10,000 docs with deterministic `makeContent(i)` bodies (~1.5 KB each, content includes the id so misdirected reads also fail), waits 3 s for eviction to settle, then reads every doc back and compares byte-for-byte. Runs sequentially — no concurrency on verify, by design.
+
+At 32 MB cap and ~15 MB of data plus overhead, eviction will fire; the verify phase exercises the WAL page-in path for every doc.
+
+### Build
+
+```bash
+g++ -std=c++20 -O2 -I/usr/include load_test_integrity.cpp \
+    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
+    -lspdlog -lfmt -pthread -o load_test_integrity
+```
+
+### Run
+
+Uses the same `config.json` as the base test.
+
+```bash
+mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
+smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
+SERVER_PID=$!
+sleep 2
+
+./load_test_integrity [num_docs=10000] [stabilize_sec=3]
+
+kill $SERVER_PID
+```
+
+### PASS criteria
+
+- All N docs retrievable
+- Every doc's `content` field byte-exact matches `makeContent(i)`
+- Zero mismatches, zero not-found, zero read errors
+
+If the summary says `no eviction stubs present`, the budget is too loose for your dataset size — either bump `num_docs` or lower `max_memory_mb` in the config so eviction actually fires during the run.
+
+---
+
+## Test 4 — Pinned collection (`load_test_pinned`)
+
+Confirms that `pinned=true` collections are never evicted, even under sustained pressure on other collections. Needs a different server config because the `settings` collection is created pinned by a migration — `createCollection` via the client API doesn't expose `pinned` yet.
+
+### Build
+
+```bash
+g++ -std=c++20 -O2 -I/usr/include load_test_pinned.cpp \
+    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
+    -lspdlog -lfmt -pthread -o load_test_pinned
+```
+
+### Run
+
+```bash
+# 1. Set up data dir + migrations dir + copy the migration file.
+mkdir -p /tmp/smartbotic-loadtest-pinned/{data,files,migrations}
+cp migrations/001_create_pinned_settings.json /tmp/smartbotic-loadtest-pinned/migrations/
+
+# 2. Start the server with the pinned-config. Migration auto-applies on startup.
+smartbotic-database --config ./config_pinned.json > /tmp/smartbotic-loadtest-pinned/server.log 2>&1 &
+SERVER_PID=$!
+sleep 2
+
+# 3. Run the test.
+./load_test_pinned [duration_sec=30] [bulk_writers=4] [num_settings=200]
+
+# 4. Stop.
+kill $SERVER_PID
+```
+
+### PASS criteria
+
+- `settings.documentCount == 200` (all pinned docs still present)
+- `settings.evictedStubCount == 0` (pinned docs never evicted)
+- `bulk.evictedStubCount > 0` (proves pressure actually fired — defensive sanity check)
+- All 200 settings docs round-trip with byte-exact content
+- No sample during the run ever saw settings shrink
+
+If `bulk.evictedStubCount` is `0`, either the run was too short, or the bulk workload produced less than ~32 MB — raise `duration_sec` or `bulk_writers`.

tests/load_test/config_pinned.json

@@ -0,0 +1,58 @@
+{
+  "log_level": "info",
+  "storage": {
+    "bind_address": "127.0.0.1",
+    "rpc_port": 9005,
+    "node_id": "loadtest-pinned",
+    "data_directory": "/tmp/smartbotic-loadtest-pinned/data",
+    "memory": {
+      "max_memory_mb": 32,
+      "eviction_threshold_percent": 80,
+      "eviction_target_percent": 50,
+      "eviction_check_interval_ms": 200,
+      "eviction_chunk_size": 500,
+      "eviction_chunk_pause_ms": 30,
+      "max_eviction_passes_per_trigger": 15,
+      "hot_write_floor_ms": 2000,
+      "memory_soft_percent": 60,
+      "memory_hard_percent": 80,
+      "memory_emergency_percent": 92,
+      "eviction_burst_threshold": 2000
+    },
+    "persistence": {
+      "wal_sync_interval_ms": 100,
+      "snapshot_interval_sec": 3600,
+      "compression": "lz4",
+      "snapshots": {
+        "validate_after_write": true,
+        "cleanup_only_if_verified": true
+      },
+      "recovery": {
+        "mode": "normal",
+        "auto_escalate": true,
+        "allow_empty_on_fresh_install": true
+      }
+    },
+    "encryption": {
+      "enabled": false
+    },
+    "migrations": {
+      "enabled": true,
+      "directory": "/tmp/smartbotic-loadtest-pinned/migrations",
+      "auto_apply": true
+    },
+    "files": {
+      "max_file_size_mb": 100
+    },
+    "replication": {
+      "enabled": false
+    },
+    "grpc": {
+      "max_receive_message_size_mb": 100,
+      "max_send_message_size_mb": 100,
+      "resource_quota_memory_mb": 64,
+      "max_concurrent_subscribe_streams": 50,
+      "max_concurrent_file_streams": 10
+    }
+  }
+}

tests/load_test/load_test_integrity.cpp

@@ -0,0 +1,266 @@
+// Test 3 — Data integrity across eviction + paging
+//
+// Writes N deterministically-generated docs with known content, waits
+// long enough for eviction to drain memory, then reads every doc back
+// and byte-compares to the expected content. Any mismatch proves the
+// evict-then-page-in path corrupts data; any not-found proves an
+// evicted stub lost its ID pointer.
+//
+// PASS criteria:
+//   - all N docs retrievable
+//   - every doc's `content` field byte-exact matches makeContent(i)
+//   - zero mismatches, zero not-found
+
+#include <smartbotic/database/client.hpp>
+
+#include <atomic>
+#include <chrono>
+#include <cstdio>
+#include <iomanip>
+#include <iostream>
+#include <mutex>
+#include <string>
+#include <thread>
+#include <vector>
+
+using namespace smartbotic::database;
+using Clock = std::chrono::steady_clock;
+
+// Deterministic ~1.5 KB content per-id. Pattern chosen so byte corruption is
+// easy to spot — includes the id itself so misdirected reads fail too.
+static std::string makeContent(int i) {
+    std::string tag = "doc-" + std::to_string(i) + "-check";
+    std::string pad;
+    pad.reserve(100 * 10);
+    for (int k = 0; k < 100; ++k) {
+        char buf[32];
+        std::snprintf(buf, sizeof(buf), "%08d-", i * 1000 + k);
+        pad += buf;
+    }
+    return tag + "-" + pad;
+}
+
+int main(int argc, char* argv[]) {
+    int numDocs = 10000;
+    int stabilizeSec = 3;
+    if (argc > 1) numDocs = std::stoi(argv[1]);
+    if (argc > 2) stabilizeSec = std::stoi(argv[2]);   // hidden knob for faster CI
+
+    std::cout << "=== smartbotic-database v1.7.0 load test — data integrity ===\n"
+              << "Server:        localhost:9005\n"
+              << "Docs:          " << numDocs << "\n"
+              << "Doc content:   deterministic ~1.5 KB, includes id\n"
+              << "Stabilize:     " << stabilizeSec << "s\n"
+              << "Memory cap:    32 MB (configured in server)\n"
+              << "\n";
+
+    Client::Config cfg;
+    cfg.address = "localhost:9005";
+    cfg.timeoutMs = 5000;
+    cfg.writeRetries = 8;
+    cfg.writeRetryBackoffMs = 50;
+    cfg.writeRetryMaxBackoffMs = 2000;
+
+    Client db(cfg);
+    if (!db.connect()) {
+        std::cerr << "connect failed\n";
+        return 1;
+    }
+
+    if (!db.createCollection("integrity")) {
+        std::cout << "(collection integrity already exists, continuing)\n";
+    }
+
+    // -------- Phase 1: Insert --------
+    std::cout << "Phase 1: inserting " << numDocs << " docs...\n";
+    auto t0 = Clock::now();
+
+    std::atomic<uint64_t> inserted{0};
+    std::atomic<uint64_t> insertFails{0};
+    std::atomic<bool> phase1Done{false};
+
+    std::thread progress([&]() {
+        while (!phase1Done.load()) {
+            std::this_thread::sleep_for(std::chrono::milliseconds(500));
+            if (phase1Done.load()) break;
+            auto s = db.getMemoryStats();
+            auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(
+                Clock::now() - t0).count();
+            std::cout << "  [t=" << std::setw(5) << elapsed << "ms] "
+                      << "pressure=" << std::setw(9) << s.pressureLevel
+                      << " " << std::setw(3) << s.pressurePercent << "%"
+                      << " inserted=" << std::setw(6) << inserted.load()
+                      << "/" << numDocs
+                      << " fails=" << insertFails.load()
+                      << "\n";
+            std::cout.flush();
+        }
+    });
+
+    auto writer = [&](int from, int to) {
+        Client::Config wcfg = cfg;
+        Client wdb(wcfg);
+        if (!wdb.connect()) return;
+        for (int i = from; i < to; i++) {
+            std::string id = "integ-" + std::to_string(i);
+            nlohmann::json doc{
+                {"i", i},
+                {"content", makeContent(i)}
+            };
+            try {
+                wdb.insert("integrity", doc, id);
+                inserted.fetch_add(1, std::memory_order_relaxed);
+            } catch (const std::exception&) {
+                insertFails.fetch_add(1, std::memory_order_relaxed);
+            }
+        }
+    };
+
+    int half = numDocs / 2;
+    std::thread t1(writer, 0, half);
+    std::thread t2(writer, half, numDocs);
+    t1.join();
+    t2.join();
+    phase1Done.store(true);
+    progress.join();
+
+    auto t1end = Clock::now();
+    auto phase1Ms = std::chrono::duration_cast<std::chrono::milliseconds>(t1end - t0).count();
+    std::cout << "Phase 1 done: " << inserted.load() << " inserted, "
+              << insertFails.load() << " failed, " << phase1Ms << "ms\n\n";
+
+    if (insertFails.load() > 0) {
+        std::cerr << "FAIL — phase 1 had insert failures; cannot assess integrity\n";
+        return 2;
+    }
+
+    // -------- Phase 2: Stabilize --------
+    std::cout << "Phase 2: stabilizing for " << stabilizeSec << "s...\n";
+    auto phase2Start = Clock::now();
+    for (int i = 0; i < stabilizeSec * 2; i++) {
+        std::this_thread::sleep_for(std::chrono::milliseconds(500));
+        auto s = db.getMemoryStats();
+        auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(
+            Clock::now() - phase2Start).count();
+        uint64_t docs = 0, evicted = 0;
+        for (const auto& c : s.collections) {
+            if (c.collection == "integrity") {
+                docs = c.documentCount;
+                evicted = c.evictedStubCount;
+            }
+        }
+        std::cout << "  [t=" << std::setw(5) << elapsed << "ms] "
+                  << "pressure=" << std::setw(9) << s.pressureLevel
+                  << " " << std::setw(3) << s.pressurePercent << "%"
+                  << " live=" << std::setw(6) << docs
+                  << " evicted=" << std::setw(6) << evicted
+                  << "\n";
+        std::cout.flush();
+    }
+    std::cout << "\n";
+
+    // -------- Phase 3: Verify --------
+    std::cout << "Phase 3: verifying all " << numDocs << " docs...\n";
+    auto phase3Start = Clock::now();
+
+    uint64_t matched = 0;
+    uint64_t mismatched = 0;
+    uint64_t notFound = 0;
+    uint64_t readError = 0;
+    std::vector<int> firstMismatches;
+    std::vector<int> firstNotFound;
+
+    for (int i = 0; i < numDocs; i++) {
+        std::string id = "integ-" + std::to_string(i);
+        try {
+            auto doc = db.get("integrity", id);
+            if (!doc) {
+                notFound++;
+                if (firstNotFound.size() < 10) firstNotFound.push_back(i);
+                continue;
+            }
+            std::string expected = makeContent(i);
+            if (!doc->contains("content")) {
+                mismatched++;
+                if (firstMismatches.size() < 10) firstMismatches.push_back(i);
+                continue;
+            }
+            std::string got = (*doc)["content"].get<std::string>();
+            if (got == expected) {
+                matched++;
+            } else {
+                mismatched++;
+                if (firstMismatches.size() < 10) firstMismatches.push_back(i);
+            }
+        } catch (const std::exception&) {
+            readError++;
+        }
+
+        if (i > 0 && i % 1000 == 0) {
+            auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(
+                Clock::now() - phase3Start).count();
+            std::cout << "  [t=" << std::setw(5) << elapsed << "ms] "
+                      << "verified=" << std::setw(6) << i << "/" << numDocs
+                      << " matched=" << matched
+                      << " mismatch=" << mismatched
+                      << " not-found=" << notFound
+                      << " errors=" << readError
+                      << "\n";
+            std::cout.flush();
+        }
+    }
+    auto phase3End = Clock::now();
+    auto phase3Ms = std::chrono::duration_cast<std::chrono::milliseconds>(
+        phase3End - phase3Start).count();
+
+    std::cout << "\n=== Summary ===\n";
+    std::cout << "Phase 1 (insert):     " << phase1Ms << "ms\n";
+    std::cout << "Phase 3 (verify):     " << phase3Ms << "ms\n";
+    std::cout << "Total docs:           " << numDocs << "\n";
+    std::cout << "  Matched content:    " << matched << "\n";
+    std::cout << "  Mismatched:         " << mismatched << "\n";
+    std::cout << "  Not found:          " << notFound << "\n";
+    std::cout << "  Read errors:        " << readError << "\n";
+
+    if (!firstMismatches.empty()) {
+        std::cout << "\nFirst mismatches (ids): ";
+        for (int i : firstMismatches) std::cout << "integ-" << i << " ";
+        std::cout << "\n";
+    }
+    if (!firstNotFound.empty()) {
+        std::cout << "First not-found (ids): ";
+        for (int i : firstNotFound) std::cout << "integ-" << i << " ";
+        std::cout << "\n";
+    }
+
+    auto finalStats = db.getMemoryStats();
+    uint64_t finalDocs = 0, finalEvicted = 0;
+    for (const auto& c : finalStats.collections) {
+        if (c.collection == "integrity") {
+            finalDocs = c.documentCount;
+            finalEvicted = c.evictedStubCount;
+        }
+    }
+    std::cout << "\nFinal state:\n"
+              << "  Pressure:           " << finalStats.pressureLevel
+              << " (" << finalStats.pressurePercent << "%)\n"
+              << "  Live docs:          " << finalDocs << "\n"
+              << "  Evicted stubs:      " << finalEvicted << "\n";
+
+    // Eviction must have fired for this test to mean anything.
+    if (finalEvicted == 0) {
+        std::cout << "\nWARNING: no eviction stubs present — test may not have exercised "
+                  << "the evict-then-page-in path. Consider a tighter memory budget.\n";
+    }
+
+    bool pass = (mismatched == 0) && (notFound == 0) && (readError == 0)
+             && (matched == static_cast<uint64_t>(numDocs));
+
+    std::cout << "\nVerdict: "
+              << (pass
+                  ? "PASS — every doc retrieved with byte-exact content"
+                  : "FAIL — data lost or corrupted across eviction boundary")
+              << "\n";
+
+    return pass ? 0 : 2;
+}

tests/load_test/load_test_mixed.cpp

@@ -0,0 +1,333 @@
+// Test 2 — Mixed read/write workload under memory pressure
+//
+// Verifies that reads can progress while writes + eviction are running.
+// This mirrors the shadowman-cpp agent loop: it continuously reads chat
+// history while chunks of the new assistant turn stream in. If WAL-fallback
+// pages serialize behind write locks (pre-T8 behaviour), read p99 spikes.
+//
+// PASS criteria:
+//   - writer ops/s > 500
+//   - reader ops/s > 1000
+//   - reader p99 latency < 1000 ms
+//   - zero hard reader failures (RESOURCE_EXHAUSTED / timeout)
+//   - not-found is tolerated — it's a cold cache miss, not a bug
+
+#include <smartbotic/database/client.hpp>
+
+#include <algorithm>
+#include <atomic>
+#include <chrono>
+#include <iomanip>
+#include <iostream>
+#include <mutex>
+#include <random>
+#include <string>
+#include <thread>
+#include <vector>
+
+using namespace smartbotic::database;
+using Clock = std::chrono::steady_clock;
+
+// Histogram covers 0..10000 ms in 100 ms buckets (100 buckets) + overflow.
+static constexpr size_t HIST_BUCKETS = 100;   // 0..9999 ms in 100 ms slices
+static constexpr uint64_t HIST_BUCKET_MS = 100;
+
+struct Counters {
+    std::atomic<uint64_t> writerInserts{0};
+    std::atomic<uint64_t> writerFails{0};
+
+    std::atomic<uint64_t> readerOps{0};
+    std::atomic<uint64_t> readerNotFound{0};
+    std::atomic<uint64_t> readerTimeouts{0};
+    std::atomic<uint64_t> readerResourceExhausted{0};
+    std::atomic<uint64_t> readerOtherErrors{0};
+
+    std::atomic<uint64_t> latencyHist[HIST_BUCKETS + 1] = {};   // [HIST_BUCKETS] = overflow
+
+    std::atomic<bool> stop{false};
+};
+
+// Recently-inserted IDs, capped at 1000. Shared between writers and readers.
+struct RecentIds {
+    std::mutex mu;
+    std::vector<std::string> ids;
+    size_t maxSize = 1000;
+
+    void push(const std::string& id) {
+        std::lock_guard<std::mutex> lk(mu);
+        if (ids.size() >= maxSize) {
+            // Drop oldest (simple O(n) shift — cheap relative to gRPC).
+            ids.erase(ids.begin());
+        }
+        ids.push_back(id);
+    }
+
+    std::optional<std::string> sample(std::mt19937& rng) {
+        std::lock_guard<std::mutex> lk(mu);
+        if (ids.empty()) return std::nullopt;
+        std::uniform_int_distribution<size_t> dist(0, ids.size() - 1);
+        return ids[dist(rng)];
+    }
+};
+
+static void recordLatency(Counters& c, int64_t micros) {
+    int64_t ms = micros / 1000;
+    size_t bucket = ms < 0 ? 0 : static_cast<size_t>(ms) / HIST_BUCKET_MS;
+    if (bucket >= HIST_BUCKETS) bucket = HIST_BUCKETS;   // overflow slot
+    c.latencyHist[bucket].fetch_add(1, std::memory_order_relaxed);
+}
+
+static uint64_t percentile(const Counters& c, double p) {
+    uint64_t total = 0;
+    for (size_t i = 0; i <= HIST_BUCKETS; i++) {
+        total += c.latencyHist[i].load(std::memory_order_relaxed);
+    }
+    if (total == 0) return 0;
+    uint64_t target = static_cast<uint64_t>(std::ceil(total * p));
+    uint64_t running = 0;
+    for (size_t i = 0; i <= HIST_BUCKETS; i++) {
+        running += c.latencyHist[i].load(std::memory_order_relaxed);
+        if (running >= target) {
+            if (i == HIST_BUCKETS) return HIST_BUCKETS * HIST_BUCKET_MS;   // >=10s
+            return (i + 1) * HIST_BUCKET_MS;   // upper bound of this bucket
+        }
+    }
+    return HIST_BUCKETS * HIST_BUCKET_MS;
+}
+
+int main(int argc, char* argv[]) {
+    int durationSec = 30;
+    int writers = 3;
+    int readers = 5;
+    int docBytes = 2048;
+    if (argc > 1) durationSec = std::stoi(argv[1]);
+    if (argc > 2) writers = std::stoi(argv[2]);
+    if (argc > 3) readers = std::stoi(argv[3]);
+    if (argc > 4) docBytes = std::stoi(argv[4]);
+
+    std::cout << "=== smartbotic-database v1.7.0 load test — mixed read/write ===\n"
+              << "Server:        localhost:9005\n"
+              << "Duration:      " << durationSec << "s\n"
+              << "Writers:       " << writers << "\n"
+              << "Readers:       " << readers << "\n"
+              << "Doc size:      " << docBytes << " bytes\n"
+              << "Memory cap:    32 MB (configured in server)\n"
+              << "\n";
+
+    Client::Config cfg;
+    cfg.address = "localhost:9005";
+    cfg.timeoutMs = 3000;
+    cfg.writeRetries = 5;
+    cfg.writeRetryBackoffMs = 50;
+    cfg.writeRetryMaxBackoffMs = 1000;
+
+    Client db(cfg);
+    if (!db.connect()) {
+        std::cerr << "connect failed\n";
+        return 1;
+    }
+
+    if (!db.createCollection("mixed")) {
+        std::cout << "(collection mixed already exists, continuing)\n";
+    }
+
+    Counters counters;
+    RecentIds recent;
+    recent.maxSize = 1000;
+
+    auto startTime = Clock::now();
+    std::string pad(docBytes, 'x');
+
+    // Writers
+    std::vector<std::thread> writerThreads;
+    for (int w = 0; w < writers; w++) {
+        writerThreads.emplace_back([&, w]() {
+            Client::Config wcfg = cfg;
+            Client wdb(wcfg);
+            if (!wdb.connect()) return;
+
+            uint64_t seq = 0;
+            while (!counters.stop.load(std::memory_order_relaxed)) {
+                std::string id = "w" + std::to_string(w) + "-" + std::to_string(seq);
+                try {
+                    nlohmann::json doc{
+                        {"worker", w},
+                        {"seq", seq},
+                        {"pad", pad}
+                    };
+                    wdb.insert("mixed", doc, id);
+                    counters.writerInserts.fetch_add(1, std::memory_order_relaxed);
+                    recent.push(id);
+                } catch (const std::exception&) {
+                    counters.writerFails.fetch_add(1, std::memory_order_relaxed);
+                }
+                seq++;
+            }
+        });
+    }
+
+    // Readers
+    std::vector<std::thread> readerThreads;
+    for (int r = 0; r < readers; r++) {
+        readerThreads.emplace_back([&, r]() {
+            Client::Config rcfg = cfg;
+            Client rdb(rcfg);
+            if (!rdb.connect()) return;
+
+            std::mt19937 rng(std::random_device{}() + r);
+
+            while (!counters.stop.load(std::memory_order_relaxed)) {
+                auto idOpt = recent.sample(rng);
+                if (!idOpt) {
+                    // Writers haven't produced anything yet — spin briefly.
+                    std::this_thread::sleep_for(std::chrono::milliseconds(1));
+                    continue;
+                }
+
+                auto t0 = Clock::now();
+                try {
+                    auto doc = rdb.get("mixed", *idOpt);
+                    auto dt = std::chrono::duration_cast<std::chrono::microseconds>(
+                        Clock::now() - t0).count();
+                    recordLatency(counters, dt);
+
+                    if (!doc) {
+                        // Cold miss — eviction may have dropped the body; ID is still
+                        // in our recent list. Not a failure.
+                        counters.readerNotFound.fetch_add(1, std::memory_order_relaxed);
+                    }
+                    counters.readerOps.fetch_add(1, std::memory_order_relaxed);
+                } catch (const std::exception& e) {
+                    auto dt = std::chrono::duration_cast<std::chrono::microseconds>(
+                        Clock::now() - t0).count();
+                    recordLatency(counters, dt);
+                    std::string msg = e.what();
+                    if (msg.find("RESOURCE_EXHAUSTED") != std::string::npos) {
+                        counters.readerResourceExhausted.fetch_add(1, std::memory_order_relaxed);
+                    } else if (msg.find("Deadline") != std::string::npos ||
+                               msg.find("DEADLINE") != std::string::npos) {
+                        counters.readerTimeouts.fetch_add(1, std::memory_order_relaxed);
+                    } else {
+                        counters.readerOtherErrors.fetch_add(1, std::memory_order_relaxed);
+                    }
+                }
+            }
+        });
+    }
+
+    // Stats sampler
+    std::thread statsThread([&]() {
+        uint64_t lastWriterInserts = 0;
+        uint64_t lastReaderOps = 0;
+        auto lastT = Clock::now();
+        while (!counters.stop.load(std::memory_order_relaxed)) {
+            std::this_thread::sleep_for(std::chrono::milliseconds(500));
+            auto s = db.getMemoryStats();
+            auto now = Clock::now();
+            auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(
+                now - startTime).count();
+            auto delta = std::chrono::duration_cast<std::chrono::milliseconds>(
+                now - lastT).count();
+            lastT = now;
+
+            uint64_t curInserts = counters.writerInserts.load();
+            uint64_t curReaderOps = counters.readerOps.load();
+            uint64_t insertRate = delta > 0 ? (curInserts - lastWriterInserts) * 1000 / delta : 0;
+            uint64_t readRate = delta > 0 ? (curReaderOps - lastReaderOps) * 1000 / delta : 0;
+            lastWriterInserts = curInserts;
+            lastReaderOps = curReaderOps;
+
+            std::cout << "[t=" << std::setw(5) << elapsed << "ms] "
+                      << "pressure=" << std::setw(9) << s.pressureLevel
+                      << " " << std::setw(3) << s.pressurePercent << "%"
+                      << " inserts=" << std::setw(7) << curInserts
+                      << " (" << std::setw(5) << insertRate << "/s)"
+                      << " reads=" << std::setw(7) << curReaderOps
+                      << " (" << std::setw(5) << readRate << "/s)"
+                      << " not-found=" << counters.readerNotFound.load()
+                      << "\n";
+            std::cout.flush();
+        }
+    });
+
+    std::this_thread::sleep_for(std::chrono::seconds(durationSec));
+    counters.stop.store(true);
+    for (auto& t : writerThreads) t.join();
+    for (auto& t : readerThreads) t.join();
+    statsThread.join();
+
+    auto endTime = Clock::now();
+    auto totalMs = std::chrono::duration_cast<std::chrono::milliseconds>(endTime - startTime).count();
+
+    uint64_t totalInserts = counters.writerInserts.load();
+    uint64_t totalWriteFails = counters.writerFails.load();
+    uint64_t totalReads = counters.readerOps.load();
+    uint64_t notFound = counters.readerNotFound.load();
+    uint64_t rTimeouts = counters.readerTimeouts.load();
+    uint64_t rResourceExh = counters.readerResourceExhausted.load();
+    uint64_t rOther = counters.readerOtherErrors.load();
+    uint64_t rHardFails = rTimeouts + rResourceExh + rOther;
+
+    uint64_t p50 = percentile(counters, 0.50);
+    uint64_t p99 = percentile(counters, 0.99);
+
+    uint64_t writerRate = totalMs > 0 ? (totalInserts * 1000 / totalMs) : 0;
+    uint64_t readerRate = totalMs > 0 ? (totalReads * 1000 / totalMs) : 0;
+
+    auto finalStats = db.getMemoryStats();
+    uint64_t finalDocs = 0, finalEvicted = 0;
+    for (const auto& c : finalStats.collections) {
+        if (c.collection == "mixed") {
+            finalDocs = c.documentCount;
+            finalEvicted = c.evictedStubCount;
+        }
+    }
+
+    std::cout << "\n=== Summary ===\n";
+    std::cout << "Duration:                 " << totalMs << "ms\n";
+    std::cout << "\n-- Writers --\n";
+    std::cout << "Successful inserts:       " << totalInserts
+              << " (" << writerRate << " ops/s)\n";
+    std::cout << "Failed inserts:           " << totalWriteFails << "\n";
+    std::cout << "\n-- Readers --\n";
+    std::cout << "Total reads:              " << totalReads
+              << " (" << readerRate << " ops/s)\n";
+    std::cout << "Not-found (cold miss):    " << notFound
+              << " (" << (totalReads > 0 ? notFound * 100 / totalReads : 0) << "%)\n";
+    std::cout << "Hard failures:            " << rHardFails << "\n";
+    std::cout << "  RESOURCE_EXHAUSTED:     " << rResourceExh << "\n";
+    std::cout << "  DEADLINE_EXCEEDED:      " << rTimeouts << "\n";
+    std::cout << "  Other:                  " << rOther << "\n";
+    std::cout << "Latency p50:              ~" << p50 << " ms\n";
+    std::cout << "Latency p99:              ~" << p99 << " ms\n";
+    std::cout << "\n-- Final state --\n";
+    std::cout << "Pressure:                 " << finalStats.pressureLevel
+              << " (" << finalStats.pressurePercent << "%)\n";
+    std::cout << "Live docs:                " << finalDocs << "\n";
+    std::cout << "Evicted stubs:            " << finalEvicted << "\n";
+    std::cout << "\n";
+
+    // Verdict
+    bool passWriter = writerRate > 500;
+    bool passReader = readerRate > 1000;
+    bool passLatency = p99 < 1000;
+    bool passFails = rHardFails == 0;
+
+    std::cout << "PASS criteria:\n";
+    std::cout << "  writer ops/s > 500:       " << (passWriter ? "PASS" : "FAIL")
+              << " (actual " << writerRate << ")\n";
+    std::cout << "  reader ops/s > 1000:      " << (passReader ? "PASS" : "FAIL")
+              << " (actual " << readerRate << ")\n";
+    std::cout << "  reader p99 < 1000ms:      " << (passLatency ? "PASS" : "FAIL")
+              << " (actual ~" << p99 << "ms)\n";
+    std::cout << "  zero reader hard fails:   " << (passFails ? "PASS" : "FAIL")
+              << " (actual " << rHardFails << ")\n";
+    std::cout << "\n";
+    std::cout << "Verdict: "
+              << (passWriter && passReader && passLatency && passFails
+                  ? "PASS — reads stay responsive while writes + eviction run"
+                  : "FAIL — see criteria above")
+              << "\n";
+
+    return (passWriter && passReader && passLatency && passFails) ? 0 : 2;
+}

tests/load_test/load_test_pinned.cpp

@@ -0,0 +1,287 @@
+// Test 4 — Pinned collection protection under sustained pressure
+//
+// The `settings` collection is created by the 001 migration with
+// pinned=true. We seed it with 200 documents (~400 KB total, well
+// under the 32 MB cap), then hammer a separate `bulk` collection with
+// 4 writer threads for the full duration. After the run we verify:
+//
+//   * settings still has all 200 docs live
+//   * settings has zero evicted stubs
+//   * bulk has >0 evicted stubs (proves the pressure was real)
+//   * every settings doc still round-trips with byte-exact content
+//
+// PASS iff all four hold.
+
+#include <smartbotic/database/client.hpp>
+
+#include <atomic>
+#include <chrono>
+#include <cstdio>
+#include <iomanip>
+#include <iostream>
+#include <string>
+#include <thread>
+#include <vector>
+
+using namespace smartbotic::database;
+using Clock = std::chrono::steady_clock;
+
+static std::string settingsContent(int i) {
+    // 2 KB of deterministic content — small enough that 200 fit easily, large
+    // enough to be noticed by eviction's size accounting.
+    std::string tag = "setting-" + std::to_string(i) + "-pin";
+    std::string pad;
+    pad.reserve(2000);
+    for (int k = 0; k < 100; ++k) {
+        char buf[24];
+        std::snprintf(buf, sizeof(buf), "%06d-%06d-", i, k);
+        pad += buf;
+    }
+    return tag + "-" + pad;
+}
+
+int main(int argc, char* argv[]) {
+    int durationSec = 30;
+    int bulkWriters = 4;
+    int numSettings = 200;
+    if (argc > 1) durationSec = std::stoi(argv[1]);
+    if (argc > 2) bulkWriters = std::stoi(argv[2]);
+    if (argc > 3) numSettings = std::stoi(argv[3]);
+
+    std::cout << "=== smartbotic-database v1.7.0 load test — pinned collection ===\n"
+              << "Server:        localhost:9005  (config_pinned.json)\n"
+              << "Duration:      " << durationSec << "s\n"
+              << "Settings docs: " << numSettings << " x 2 KB (pinned=true)\n"
+              << "Bulk writers:  " << bulkWriters << " (non-pinned)\n"
+              << "Memory cap:    32 MB (configured in server)\n"
+              << "\n";
+
+    Client::Config cfg;
+    cfg.address = "localhost:9005";
+    cfg.timeoutMs = 3000;
+    cfg.writeRetries = 5;
+    cfg.writeRetryBackoffMs = 50;
+    cfg.writeRetryMaxBackoffMs = 1000;
+
+    Client db(cfg);
+    if (!db.connect()) {
+        std::cerr << "connect failed\n";
+        return 1;
+    }
+
+    // The settings collection must already exist (created by migration).
+    auto cols = db.listCollections();
+    bool hasSettings = false;
+    for (const auto& c : cols) if (c == "settings") hasSettings = true;
+    if (!hasSettings) {
+        std::cerr << "FAIL — 'settings' collection not present. Did the migration run?\n"
+                  << "       Make sure config_pinned.json points at the migrations dir "
+                  << "and that 001_create_pinned_settings.json is there.\n";
+        return 1;
+    }
+    std::cout << "OK — 'settings' collection exists (created by migration).\n";
+
+    // Bulk collection for the pressure workload.
+    if (!db.createCollection("bulk")) {
+        std::cout << "(collection bulk already exists, continuing)\n";
+    }
+
+    // -------- Seed settings --------
+    std::cout << "\nSeeding " << numSettings << " pinned settings docs...\n";
+    for (int i = 0; i < numSettings; i++) {
+        std::string id = "setting-" + std::to_string(i);
+        nlohmann::json doc{
+            {"i", i},
+            {"content", settingsContent(i)}
+        };
+        try {
+            db.upsert("settings", doc, id);
+        } catch (const std::exception& e) {
+            std::cerr << "FAIL — could not seed settings/" << id << ": " << e.what() << "\n";
+            return 1;
+        }
+    }
+
+    // Confirm count.
+    {
+        auto s = db.getMemoryStats();
+        uint64_t setDocs = 0;
+        for (const auto& c : s.collections) {
+            if (c.collection == "settings") setDocs = c.documentCount;
+        }
+        if (setDocs != static_cast<uint64_t>(numSettings)) {
+            std::cerr << "FAIL — seeded " << numSettings << " but see " << setDocs
+                      << " in getMemoryStats\n";
+            return 1;
+        }
+        std::cout << "Seed OK: settings has " << setDocs << " live docs.\n";
+    }
+
+    // -------- Pressure phase --------
+    std::cout << "\nStarting bulk-writer pressure for " << durationSec << "s...\n";
+
+    std::atomic<uint64_t> bulkInserts{0};
+    std::atomic<uint64_t> bulkFails{0};
+    std::atomic<bool> stop{false};
+
+    auto startTime = Clock::now();
+    std::string pad(4096, 'x');   // 4 KB docs so we churn hard
+
+    std::vector<std::thread> workers;
+    for (int w = 0; w < bulkWriters; w++) {
+        workers.emplace_back([&, w]() {
+            Client::Config wcfg = cfg;
+            Client wdb(wcfg);
+            if (!wdb.connect()) return;
+            uint64_t seq = 0;
+            while (!stop.load(std::memory_order_relaxed)) {
+                nlohmann::json doc{
+                    {"worker", w},
+                    {"seq", seq},
+                    {"pad", pad}
+                };
+                std::string id = "bulk-" + std::to_string(w) + "-" + std::to_string(seq);
+                try {
+                    wdb.insert("bulk", doc, id);
+                    bulkInserts.fetch_add(1, std::memory_order_relaxed);
+                } catch (const std::exception&) {
+                    bulkFails.fetch_add(1, std::memory_order_relaxed);
+                }
+                seq++;
+            }
+        });
+    }
+
+    // Progress sampler — also watches settings count to flag regression live.
+    std::atomic<bool> sawSettingsShrink{false};
+    std::thread sampler([&]() {
+        while (!stop.load(std::memory_order_relaxed)) {
+            std::this_thread::sleep_for(std::chrono::milliseconds(500));
+            auto s = db.getMemoryStats();
+            auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(
+                Clock::now() - startTime).count();
+
+            uint64_t setDocs = 0, setEvicted = 0;
+            uint64_t bulkDocs = 0, bulkEvicted = 0;
+            for (const auto& c : s.collections) {
+                if (c.collection == "settings") {
+                    setDocs = c.documentCount;
+                    setEvicted = c.evictedStubCount;
+                } else if (c.collection == "bulk") {
+                    bulkDocs = c.documentCount;
+                    bulkEvicted = c.evictedStubCount;
+                }
+            }
+            if (setDocs < static_cast<uint64_t>(numSettings) || setEvicted > 0) {
+                sawSettingsShrink.store(true);
+            }
+
+            std::cout << "[t=" << std::setw(5) << elapsed << "ms] "
+                      << "pressure=" << std::setw(9) << s.pressureLevel
+                      << " " << std::setw(3) << s.pressurePercent << "%"
+                      << " settings(live/evicted)=" << setDocs << "/" << setEvicted
+                      << " bulk(live/evicted)=" << bulkDocs << "/" << bulkEvicted
+                      << " bulk_inserts=" << bulkInserts.load()
+                      << "\n";
+            std::cout.flush();
+        }
+    });
+
+    std::this_thread::sleep_for(std::chrono::seconds(durationSec));
+    stop.store(true);
+    for (auto& t : workers) t.join();
+    sampler.join();
+
+    auto endTime = Clock::now();
+    auto totalMs = std::chrono::duration_cast<std::chrono::milliseconds>(endTime - startTime).count();
+
+    // -------- Final verification --------
+    auto finalStats = db.getMemoryStats();
+    uint64_t setDocs = 0, setEvicted = 0;
+    uint64_t bulkDocs = 0, bulkEvicted = 0;
+    for (const auto& c : finalStats.collections) {
+        if (c.collection == "settings") {
+            setDocs = c.documentCount;
+            setEvicted = c.evictedStubCount;
+        } else if (c.collection == "bulk") {
+            bulkDocs = c.documentCount;
+            bulkEvicted = c.evictedStubCount;
+        }
+    }
+
+    // Also round-trip read all 200 settings docs.
+    uint64_t settingsMatched = 0;
+    uint64_t settingsMismatch = 0;
+    uint64_t settingsMissing = 0;
+    for (int i = 0; i < numSettings; i++) {
+        std::string id = "setting-" + std::to_string(i);
+        try {
+            auto doc = db.get("settings", id);
+            if (!doc) {
+                settingsMissing++;
+                continue;
+            }
+            std::string expected = settingsContent(i);
+            if (doc->contains("content") && (*doc)["content"].get<std::string>() == expected) {
+                settingsMatched++;
+            } else {
+                settingsMismatch++;
+            }
+        } catch (const std::exception&) {
+            settingsMissing++;
+        }
+    }
+
+    std::cout << "\n=== Summary ===\n";
+    std::cout << "Duration:                  " << totalMs << "ms\n";
+    std::cout << "Bulk inserts:              " << bulkInserts.load()
+              << " (" << (totalMs > 0 ? bulkInserts.load() * 1000 / totalMs : 0) << " ops/s)\n";
+    std::cout << "Bulk insert failures:      " << bulkFails.load() << "\n";
+    std::cout << "\n-- Final collection state --\n";
+    std::cout << "settings.documentCount:    " << setDocs
+              << " (expected " << numSettings << ")\n";
+    std::cout << "settings.evictedStubCount: " << setEvicted
+              << " (expected 0)\n";
+    std::cout << "bulk.documentCount:        " << bulkDocs << "\n";
+    std::cout << "bulk.evictedStubCount:     " << bulkEvicted
+              << " (expected > 0 — proves pressure fired)\n";
+    std::cout << "Pressure level at end:     " << finalStats.pressureLevel
+              << " (" << finalStats.pressurePercent << "%)\n";
+    std::cout << "\n-- Settings round-trip --\n";
+    std::cout << "Matched content:           " << settingsMatched << "/" << numSettings << "\n";
+    std::cout << "Mismatched content:        " << settingsMismatch << "\n";
+    std::cout << "Missing from read:         " << settingsMissing << "\n";
+    if (sawSettingsShrink.load()) {
+        std::cout << "\nNote: sampler saw settings shrink mid-run — pinned guarantee broken.\n";
+    }
+
+    bool passSettingsCount = (setDocs == static_cast<uint64_t>(numSettings));
+    bool passSettingsStubs = (setEvicted == 0);
+    bool passBulkPressure = (bulkEvicted > 0);
+    bool passRoundTrip = (settingsMatched == static_cast<uint64_t>(numSettings))
+                      && (settingsMismatch == 0)
+                      && (settingsMissing == 0);
+    bool passSnapshot = !sawSettingsShrink.load();
+
+    std::cout << "\nPASS criteria:\n";
+    std::cout << "  settings live == " << numSettings << ":   "
+              << (passSettingsCount ? "PASS" : "FAIL") << "\n";
+    std::cout << "  settings evicted == 0:      "
+              << (passSettingsStubs ? "PASS" : "FAIL") << "\n";
+    std::cout << "  bulk evicted > 0:           "
+              << (passBulkPressure ? "PASS" : "FAIL") << "\n";
+    std::cout << "  settings round-trip clean:  "
+              << (passRoundTrip ? "PASS" : "FAIL") << "\n";
+    std::cout << "  no shrink during run:       "
+              << (passSnapshot ? "PASS" : "FAIL") << "\n";
+
+    bool allPass = passSettingsCount && passSettingsStubs && passBulkPressure
+                && passRoundTrip && passSnapshot;
+    std::cout << "\nVerdict: "
+              << (allPass
+                  ? "PASS — pinned=true holds under sustained non-pinned pressure"
+                  : "FAIL — pinned guarantee violated")
+              << "\n";
+
+    return allPass ? 0 : 2;
+}

tests/load_test/migrations/001_create_pinned_settings.json

@@ -0,0 +1,18 @@
+{
+  "version": "001",
+  "name": "create_pinned_settings",
+  "description": "Load-test fixture: create a 'settings' collection marked pinned=true so the eviction loop must leave it alone under pressure.",
+  "operations": [
+    {
+      "type": "create_collection",
+      "collection": "settings"
+    },
+    {
+      "type": "alter_collection",
+      "collection": "settings",
+      "options": {
+        "pinned": true
+      }
+    }
+  ]
+}