# Load tests — eviction under memory pressure

Standalone binaries that validate v1.7.0's chunked eviction + admission control + client retry against a real smartbotic-database instance.

**Not unit tests** — each one runs a real server on a configurable port, drives a workload, and reports a PASS/FAIL verdict. Not wired into CTest — these are operator/developer validation tools, not CI.

## Test suite

Four load-test binaries:

- `load_test` — pressure + retry validation (the original)
- `load_test_mixed` — concurrent reads + writes under pressure
- `load_test_integrity` — byte-exact integrity of evicted + paged-in docs
- `load_test_pinned` — `pinned=true` collection protection

All four talk to `localhost:9005` and use `Client::Config` with transport-level retries. They differ only in workload shape and verdict criteria.

## Prerequisites

- `smartbotic-database` package installed (for the server binary)
- `libsmartbotic-db-client-dev` installed (for headers + shared lib)
- GCC with C++20 support

## Build

```bash
cd tests/load_test
g++ -std=c++20 -O2 -I/usr/include load_test.cpp \
    -lsmartbotic-db-client \
    $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread \
    -o load_test
```

## Run

Uses port `9005` and data dir `/tmp/smartbotic-loadtest/` to avoid conflicting with any existing instance on the default port `9004`.

```bash
# 1. Start the test server with its own config (memory capped at 32 MB)
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

# 2. Run the load test — default: 20s, 4 writers, 2 KB docs
./load_test [duration_seconds] [writer_threads] [doc_bytes]

# Example — 30s with 8 writers:
./load_test 30 8 2048

# 3. Stop the server
kill $SERVER_PID

# 4. Inspect eviction events
grep -E "Eviction|MEMORY_" /tmp/smartbotic-loadtest/server.log
```

## What to look for

Sample output from a successful run (32 MB cap, 20s, 4 writers, 2 KB docs):

```
[t=18504ms] pressure=     hard  84% mem=27584KB live=  5276 evicted= 41580 inserts=  46856 fails=    0

=== Summary ===
Duration:                 20162ms
Successful inserts:       51444 (2551 ops/s)
Failed (after retries):   0
Pressure timeline:
  normal:  1.5s
  soft:    5.0s
  hard:    9.0s
  emergency: 2.0s
  Transitions: 19

Verdict: PASS — client-side retry absorbed all transient errors
```

## Key signals

| Signal | What it means |
|--------|---------------|
| `Eviction chunk: N docs evicted, X bytes freed` | T4 chunked eviction is working |
| `Eviction: nothing evictable this pass, backing off` | T4 hot-write floor is protecting in-flight upserts |
| `Emitted MEMORY_PRESSURE_HIGH` | T10 edge-triggered event fired |
| `Insert rejected: memory pressure emergency` | T7 admission control engaged |
| `Client::insert ...; retrying in Nms (attempt K/5)` | T11 client retry kicked in |
| `Successful inserts: N == attempts, Failed: 0` | No writes lost under pressure |

## Tuning for different scenarios

Edit `config.json`:

- **Burst write stress**: lower `hot_write_floor_ms` to `500` ms so eviction can actually drop recent writes
- **Slow drain**: raise `eviction_chunk_pause_ms` to `200` ms
- **Admission control disabled**: raise `memory_emergency_percent` to `99`
- **Bigger budget**: raise `max_memory_mb` to `256` to see if the chunked approach still works at scale

---

## Test 2 — Mixed read/write (`load_test_mixed`)

Validates that reads stay responsive while writers + eviction are running. Three writer threads continuously insert 2 KB docs into a shared list of recent IDs; five reader threads pick random IDs from that list and call `get()`. Reads hit both live docs and evicted-stub IDs, exercising the WAL-fallback paging path.

### Build

```bash
g++ -std=c++20 -O2 -I/usr/include load_test_mixed.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_mixed
```

### Run

```bash
# Same server setup as load_test:
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

./load_test_mixed [duration_sec=30] [writers=3] [readers=5] [doc_bytes=2048]

kill $SERVER_PID
```

### PASS criteria

- writer ops/s > 500
- reader ops/s > 1000 (reads should be faster than writes — no eviction gate on reads)
- reader p99 latency < 1000 ms
- zero reader hard failures (RESOURCE_EXHAUSTED / timeout / other)

`not-found` is counted separately from failures — it's a legitimate cold miss from a freshly-evicted stub where the WAL page hasn't been pulled yet, not a bug.

---

## Test 3 — Data integrity (`load_test_integrity`)

Writes 10,000 docs with deterministic `makeContent(i)` bodies (~1.5 KB each, content includes the id so misdirected reads also fail), waits 3 s for eviction to settle, then reads every doc back and compares byte-for-byte. Runs sequentially — no concurrency on verify, by design.

At 32 MB cap and ~15 MB of data plus overhead, eviction will fire; the verify phase exercises the WAL page-in path for every doc.

### Build

```bash
g++ -std=c++20 -O2 -I/usr/include load_test_integrity.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_integrity
```

### Run

Uses the same `config.json` as the base test.

```bash
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

./load_test_integrity [num_docs=10000] [stabilize_sec=3]

kill $SERVER_PID
```

### PASS criteria

- All N docs retrievable
- Every doc's `content` field byte-exact matches `makeContent(i)`
- Zero mismatches, zero not-found, zero read errors

If the summary says `no eviction stubs present`, the budget is too loose for your dataset size — either bump `num_docs` or lower `max_memory_mb` in the config so eviction actually fires during the run.

---

## Test 4 — Pinned collection (`load_test_pinned`)

Confirms that `pinned=true` collections are never evicted, even under sustained pressure on other collections. Needs a different server config because the `settings` collection is created pinned by a migration — `createCollection` via the client API doesn't expose `pinned` yet.

### Build

```bash
g++ -std=c++20 -O2 -I/usr/include load_test_pinned.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_pinned
```

### Run

```bash
# 1. Set up data dir + migrations dir + copy the migration file.
mkdir -p /tmp/smartbotic-loadtest-pinned/{data,files,migrations}
cp migrations/001_create_pinned_settings.json /tmp/smartbotic-loadtest-pinned/migrations/

# 2. Start the server with the pinned-config. Migration auto-applies on startup.
smartbotic-database --config ./config_pinned.json > /tmp/smartbotic-loadtest-pinned/server.log 2>&1 &
SERVER_PID=$!
sleep 2

# 3. Run the test.
./load_test_pinned [duration_sec=30] [bulk_writers=4] [num_settings=200]

# 4. Stop.
kill $SERVER_PID
```

### PASS criteria

- `settings.documentCount == 200` (all pinned docs still present)
- `settings.evictedStubCount == 0` (pinned docs never evicted)
- `bulk.evictedStubCount > 0` (proves pressure actually fired — defensive sanity check)
- All 200 settings docs round-trip with byte-exact content
- No sample during the run ever saw settings shrink

If `bulk.evictedStubCount` is `0`, either the run was too short, or the bulk workload produced less than ~32 MB — raise `duration_sec` or `bulk_writers`.

---

## Test 5 — Crash recovery mid-eviction (`run_crash_test.sh`)

The most complex scenario — starts a server, starts an inserter, waits until eviction has started firing, `kill -9`s both, restarts the server with `--force-readwrite`, and verifies every successfully-logged insert is still retrievable byte-exactly.

This is the one that validates v1.6.1 (snapshot durability + recovery) + v1.7.0 (eviction) working together under a hard failure mode.

### Build

    cd tests/load_test
    g++ -std=c++20 -O2 -I/usr/include load_test_crash_insert.cpp \
        -lsmartbotic-db-client $(pkg-config --libs grpc++) \
        -lspdlog -lfmt -pthread -o load_test_crash_insert
    g++ -std=c++20 -O2 -I/usr/include load_test_crash_verify.cpp \
        -lsmartbotic-db-client $(pkg-config --libs grpc++) \
        -lspdlog -lfmt -pthread -o load_test_crash_verify
    chmod +x run_crash_test.sh

### Run

    ./run_crash_test.sh

Exit code:
- 0 — every logged insert recovered byte-exactly (PASS)
- non-zero — either the server refused to restart or content mismatched (see script output)

### What it does under the hood

1. Wipes `/tmp/smartbotic-loadtest-crash/`, makes fresh dirs
2. Writes `config_crash.json` based on `config.json` with the crash data dir
3. Starts server, begins inserting 20k deterministic docs across 3 writer threads
4. Waits for the first "Eviction chunk" log line, then sleeps 2 s more to ensure we're in the middle of an eviction storm
5. `kill -9` the inserter + the server
6. Restarts the server with `--force-readwrite` (accepts auto-readonly non-trivial-recovery state)
7. Reads every ID that the inserter successfully logged before the crash; verifies content matches `makeContent(i)`

### PASS criteria

- Every ID in `/tmp/smartbotic-loadtest-crash/inserted.log` is retrievable
- Every content byte-exact matches its expected value
- Zero not-found, zero mismatches

### If the server refuses to restart (exit code 10)

The recovery system decided the snapshot+WAL combination was unrecoverable. `run_crash_test.sh` then dumps the last 30 lines of `server2.log` — look for `RECOVERY REFUSED`. Escalate manually:

    smartbotic-database --config /tmp/smartbotic-loadtest-crash/config_crash.json \
        --recovery-mode=snapshot_fallback --force-readwrite

---

## Test 6 — Replication sanity check (`run_replication_test.sh`)

Spawns two local server instances (leader on 9005, follower on 9006), each configured with the other as a replication peer, inserts 1000 docs on the leader, then polls the follower every 200 ms until it sees them. Reports catch-up time and content integrity.

This is a smoke test — it does NOT exercise replication at real-deployment scale. It verifies the replication mechanism still fires alongside v1.7.0's eviction / admission-control / client-retry changes. Because v1.7.0 didn't touch replication code, the expected outcome is "no regression" — if the mechanism breaks here, bisect toward something we did change (WAL ordering, store callbacks).

### Build

    cd tests/load_test
    g++ -std=c++20 -O2 -I/usr/include load_test_replication.cpp \
        -lsmartbotic-db-client $(pkg-config --libs grpc++) \
        -lspdlog -lfmt -pthread -o load_test_replication
    chmod +x run_replication_test.sh

### Run

    ./run_replication_test.sh [num_docs=1000] [max_wait_sec=30]

### Configs

- `config_replica_leader.json` — port 9005, node_id `replica-leader`, replication enabled, peer = `127.0.0.1:9006`, data dir `/tmp/smartbotic-loadtest-replica-leader/data`
- `config_replica_follower.json` — port 9006, node_id `replica-follower`, replication enabled, peer = `127.0.0.1:9005`, data dir `/tmp/smartbotic-loadtest-replica-follower/data`

Both configs use symmetric multi-master peering (as the replication code supports) — each side lists the other in `peers`. Inserts are performed only on the leader in this test; the follower is a pure consumer, but could also accept writes.

### PASS criteria

- All N docs visible on the follower
- Zero content mismatches (the body field matches what the leader inserted)
- Catch-up time < `max_wait_sec` seconds

Exit codes:

- `0` — PASS (full catch-up, zero corruption)
- `2` — PARTIAL (some docs replicated, no corruption, did not fully catch up in time)
- `1` — FAIL (either nothing replicated OR content mismatches observed)

### Known quirk found while writing this test (pre-v1.7.0)

A first run of this test against v1.7.0 produced **FAIL — 200 docs replicated but zero content matches**. The follower's copy of each document contained only the system fields (`_id`, `_version`, `_created_at`, `_updated_at`, etc.), with the user-supplied payload fields missing.

Root cause is in the leader-side broadcast code at `service/src/database_service.cpp:515`:

    if (doc) entry.set_data(doc->data.dump());

This serializes only the document's `data` field. The follower-side apply code at `service/src/database_service.cpp:612` expects the full Document envelope:

    auto json = nlohmann::json::parse(entry.data());
    Document doc = Document::fromJson(json);   // looks for "data", "version", etc.

`Document::fromJson` reads `j.value("data", {})` — i.e. it expects the payload nested under `data`, not at the top level. So the follower effectively discards the body.

`git blame` shows this mismatch predates v1.7.0 (commit `a96c168`, 2026-02-05). It is a latent bug in the push path: inserts/updates are getting replicated structurally (the system fields propagate) but with an empty body on the follower.

The pull path (`GetEntriesSince` → WAL) uses a different code path — it serializes the WAL entry's `data` field directly, so pull-based catch-up may or may not exhibit the same issue depending on WAL format. Not investigated here.

This test catches the bug but is deliberately not wired to fix it — the goal of Test 6 is v1.7.0 sanity, and the bug is unrelated to the eviction refactor. File a separate ticket to either:

1. On the leader, use `doc->toJson().dump()` instead of `doc->data.dump()`, OR
2. On the follower, parse `entry.data()` as the raw document body and construct the Document from `entry`'s other fields (`collection`, `document_id`, `node_id`).

### If replication silently doesn't happen

Check both server logs for the peer handshake:

    "Connected to peer 127.0.0.1:9006 (node: replica-follower, ...)"
    "Connected to peer 127.0.0.1:9005 (node: replica-leader, ...)"

Without both lines, peering never established — likely a firewall or port conflict. `ss -tlnp | grep 900[56]` will confirm both are listening.

---

## Test 8 — Replicated eviction (`run_replica_eviction_test.sh`)

The follower has a deliberately tighter memory cap than the leader. The leader ingests enough data that the follower cannot hold it all in RAM, so the follower must:

1. Accept every replication entry (no `RESOURCE_EXHAUSTED` on the apply path — that would break sync forever)
2. Evict older docs to make room
3. Still serve every replicated doc correctly on `get()`, paging evicted ones back in from the WAL

This test catches bugs that only appear under the combined replicate + evict + page-in round trip:

- follower's replication-apply accidentally gated by admission control
- evicted-on-follower docs unreadable because replication took a shortcut past the normal WAL-write path
- content corruption across the `replicate -> evict -> page-in` cycle

### Build

    cd tests/load_test
    g++ -std=c++20 -O2 -I/usr/include load_test_replica_eviction.cpp \
        -lsmartbotic-db-client $(pkg-config --libs grpc++) \
        -lspdlog -lfmt -pthread -o load_test_replica_eviction
    chmod +x run_replica_eviction_test.sh

### Run

    ./run_replica_eviction_test.sh [num_docs=5000] [stabilize_sec=5]

### Configs

- `config_replica_leader_128mb.json` — port 9005, `max_memory_mb: 128` (plenty of room), replication enabled, peer = `127.0.0.1:9006`, data dir `/tmp/smartbotic-loadtest-replica-eviction-leader/data`
- `config_replica_follower_16mb.json` — port 9006, `max_memory_mb: 16` (tight — forces eviction), soft/hard/emergency = 50/70/90, replication enabled, peer = `127.0.0.1:9005`, data dir `/tmp/smartbotic-loadtest-replica-eviction-follower/data`

### Workload

- 5000 docs x ~2 KB (each has a 1900-byte `pad` field) inserted on the leader
- In memory terms (~2.5x overhead per v1.7.0 `estimateDocumentSize`) = ~25 MB on each side
- Leader (128 MB): no pressure
- Follower (16 MB): ~150% over cap -> eviction fires continuously

### PASS criteria

- All 5000 docs readable on the follower (live-map + WAL-fallback both exercised)
- Zero content mismatches, zero read errors
- `evictedStubCount > 0` (proves eviction fired — if not, the test is INCONCLUSIVE, raise `num_docs` or lower follower cap)

### Exit codes

- `0` — PASS (all docs replicated + byte-exact, eviction fired on follower)
- `1` — FAIL (missing docs or content corruption)
- `2` — INCONCLUSIVE (eviction never fired — follower had room to spare)

### Known bug: evicted-on-follower docs not recoverable (as of v1.7.1)

First run of this test against v1.7.1 produced **FAIL — all 5000 docs replicated (follower shows 5000 evicted stubs), but zero readable on follower**. Server log shows `"Failed to recover evicted document replica_evict/repl-<i>"` for every id.

Root cause is in `DatabaseService::applyReplicatedEntry` at `service/src/database_service.cpp:623`:

    // Use loadDocument to bypass normal callbacks (avoid re-replication)
    store_->loadDocument(entry.collection(), doc);

`MemoryStore::loadDocument` only inserts the doc into the in-memory map — it deliberately skips the usual callback chain so the follower does not re-broadcast. But that chain is also what appends the doc to the **local** WAL. So on the follower the doc lives only in RAM.

When eviction then fires on the follower, the doc is dropped to an evicted-stub. On read, the recovery callback calls `persistence_->loadDocument()` which scans the follower's WAL — and finds nothing, because replication never wrote it there. Result: permanent data loss the instant the follower evicts, even though the stub still reports the doc as "accounted for".

Fix options (not made in this changeset — the test documents the bug):

1. In `applyReplicatedEntry`, after `store_->loadDocument(...)`, append to the local WAL (e.g. `persistence_->logInsert/logUpdate` depending on op), so evicted-on-follower docs have a durable backing store.
2. Teach the eviction callback to fall back to pulling from a peer when the local WAL has no entry — more complex, adds a network hop on cold read.

### Attempted fix (option 1) and why it failed

A v1.7.2 attempt added `persistence_->logInsert(collection, doc)` directly inside `applyReplicatedEntry` after the `loadDocument` call. The fix compiled cleanly but **caused infinite replication echo amplification** when run:

```
[23:30:14] Memory check: 1954 MB estimated (100%, pressure=emergency),
           408121 docs, 30747 evicted
[23:30:14] Pulled 5000 entries from peer replica-leader-128
           (seq: 211866 -> 216867)
```

The leader's WAL reached 216867+ entries after inserting only 5000. Root cause is at `service/src/database_grpc_impl.cpp:1561` — the `GetEntriesSince` server handler hard-codes the response's `node_id` to its OWN `nodeState.nodeId`, not the entry's original origin. So when follower's WAL is streamed back to the leader, every entry claims to have originated on the follower. The leader then can't recognize its own original writes and re-applies them, re-logs to its WAL, re-streams them to the follower, and the cycle continues.

The proper fix requires:

1. Adding a `nodeId` field to `struct WalEntry` (`service/src/persistence/wal.hpp:37`) — **WAL on-disk format change, breaks backward compat without a version bump**
2. Plumbing the origin nodeId through `persistence_->logInsert/logUpdate/logDelete`
3. Using `walEntry.nodeId` (not `nodeState.nodeId`) in `GetEntriesSince`
4. The sync protocol on the sending side skipping entries whose `node_id` equals the peer being sent to

This is a v1.8.0-class change that warrants its own design + format migration path. For v1.7.x the documented limitation stands: **replicated docs on a follower that get evicted are lost. Size the follower's `max_memory_mb` to hold the working set until a proper replication overhaul ships.**

### Test output the bug produces

    Phase 2: waiting 5s for follower to stabilize...
      [t+2000ms] follower pressure=emergency (100%) live=0 evicted=5000
      ...
    Phase 3: reading all 5000 docs from follower...
      not-found id=repl-0
      not-found id=repl-1
      ...
    Summary: Found: 0, Not found: 5000
    Verdict: FAIL

And in the follower server log:

    [warning] Failed to recover evicted document replica_evict/repl-4999

This confirms replication worked (5000 stubs exist on the follower and their IDs match what the leader inserted) — but the evicted-stub recovery path has nothing to page in from.