# Load tests — eviction under memory pressure

Standalone binaries that validate v1.7.0's chunked eviction + admission control + client retry against a real smartbotic-database instance.

**Not unit tests** — each one runs a real server on a configurable port, drives a workload, and reports a PASS/FAIL verdict. Not wired into CTest — these are operator/developer validation tools, not CI.

## Test suite

Four load-test binaries:

- `load_test` — pressure + retry validation (the original)
- `load_test_mixed` — concurrent reads + writes under pressure
- `load_test_integrity` — byte-exact integrity of evicted + paged-in docs
- `load_test_pinned` — `pinned=true` collection protection

All four talk to `localhost:9005` and use `Client::Config` with transport-level retries. They differ only in workload shape and verdict criteria.

## Prerequisites

- `smartbotic-database` package installed (for the server binary)
- `libsmartbotic-db-client-dev` installed (for headers + shared lib)
- GCC with C++20 support

## Build

```bash
cd tests/load_test
g++ -std=c++20 -O2 -I/usr/include load_test.cpp \
    -lsmartbotic-db-client \
    $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread \
    -o load_test
```

## Run

Uses port `9005` and data dir `/tmp/smartbotic-loadtest/` to avoid conflicting with any existing instance on the default port `9004`.

```bash
# 1. Start the test server with its own config (memory capped at 32 MB)
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

# 2. Run the load test — default: 20s, 4 writers, 2 KB docs
./load_test [duration_seconds] [writer_threads] [doc_bytes]

# Example — 30s with 8 writers:
./load_test 30 8 2048

# 3. Stop the server
kill $SERVER_PID

# 4. Inspect eviction events
grep -E "Eviction|MEMORY_" /tmp/smartbotic-loadtest/server.log
```

## What to look for

Sample output from a successful run (32 MB cap, 20s, 4 writers, 2 KB docs):

```
[t=18504ms] pressure=     hard  84% mem=27584KB live=  5276 evicted= 41580 inserts=  46856 fails=    0

=== Summary ===
Duration:                 20162ms
Successful inserts:       51444 (2551 ops/s)
Failed (after retries):   0
Pressure timeline:
  normal:  1.5s
  soft:    5.0s
  hard:    9.0s
  emergency: 2.0s
  Transitions: 19

Verdict: PASS — client-side retry absorbed all transient errors
```

## Key signals

| Signal | What it means |
|--------|---------------|
| `Eviction chunk: N docs evicted, X bytes freed` | T4 chunked eviction is working |
| `Eviction: nothing evictable this pass, backing off` | T4 hot-write floor is protecting in-flight upserts |
| `Emitted MEMORY_PRESSURE_HIGH` | T10 edge-triggered event fired |
| `Insert rejected: memory pressure emergency` | T7 admission control engaged |
| `Client::insert ...; retrying in Nms (attempt K/5)` | T11 client retry kicked in |
| `Successful inserts: N == attempts, Failed: 0` | No writes lost under pressure |

## Tuning for different scenarios

Edit `config.json`:

- **Burst write stress**: lower `hot_write_floor_ms` to `500` ms so eviction can actually drop recent writes
- **Slow drain**: raise `eviction_chunk_pause_ms` to `200` ms
- **Admission control disabled**: raise `memory_emergency_percent` to `99`
- **Bigger budget**: raise `max_memory_mb` to `256` to see if the chunked approach still works at scale

---

## Test 2 — Mixed read/write (`load_test_mixed`)

Validates that reads stay responsive while writers + eviction are running. Three writer threads continuously insert 2 KB docs into a shared list of recent IDs; five reader threads pick random IDs from that list and call `get()`. Reads hit both live docs and evicted-stub IDs, exercising the WAL-fallback paging path.

### Build

```bash
g++ -std=c++20 -O2 -I/usr/include load_test_mixed.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_mixed
```

### Run

```bash
# Same server setup as load_test:
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

./load_test_mixed [duration_sec=30] [writers=3] [readers=5] [doc_bytes=2048]

kill $SERVER_PID
```

### PASS criteria

- writer ops/s > 500
- reader ops/s > 1000 (reads should be faster than writes — no eviction gate on reads)
- reader p99 latency < 1000 ms
- zero reader hard failures (RESOURCE_EXHAUSTED / timeout / other)

`not-found` is counted separately from failures — it's a legitimate cold miss from a freshly-evicted stub where the WAL page hasn't been pulled yet, not a bug.

---

## Test 3 — Data integrity (`load_test_integrity`)

Writes 10,000 docs with deterministic `makeContent(i)` bodies (~1.5 KB each, content includes the id so misdirected reads also fail), waits 3 s for eviction to settle, then reads every doc back and compares byte-for-byte. Runs sequentially — no concurrency on verify, by design.

At 32 MB cap and ~15 MB of data plus overhead, eviction will fire; the verify phase exercises the WAL page-in path for every doc.

### Build

```bash
g++ -std=c++20 -O2 -I/usr/include load_test_integrity.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_integrity
```

### Run

Uses the same `config.json` as the base test.

```bash
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

./load_test_integrity [num_docs=10000] [stabilize_sec=3]

kill $SERVER_PID
```

### PASS criteria

- All N docs retrievable
- Every doc's `content` field byte-exact matches `makeContent(i)`
- Zero mismatches, zero not-found, zero read errors

If the summary says `no eviction stubs present`, the budget is too loose for your dataset size — either bump `num_docs` or lower `max_memory_mb` in the config so eviction actually fires during the run.

---

## Test 4 — Pinned collection (`load_test_pinned`)

Confirms that `pinned=true` collections are never evicted, even under sustained pressure on other collections. Needs a different server config because the `settings` collection is created pinned by a migration — `createCollection` via the client API doesn't expose `pinned` yet.

### Build

```bash
g++ -std=c++20 -O2 -I/usr/include load_test_pinned.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_pinned
```

### Run

```bash
# 1. Set up data dir + migrations dir + copy the migration file.
mkdir -p /tmp/smartbotic-loadtest-pinned/{data,files,migrations}
cp migrations/001_create_pinned_settings.json /tmp/smartbotic-loadtest-pinned/migrations/

# 2. Start the server with the pinned-config. Migration auto-applies on startup.
smartbotic-database --config ./config_pinned.json > /tmp/smartbotic-loadtest-pinned/server.log 2>&1 &
SERVER_PID=$!
sleep 2

# 3. Run the test.
./load_test_pinned [duration_sec=30] [bulk_writers=4] [num_settings=200]

# 4. Stop.
kill $SERVER_PID
```

### PASS criteria

- `settings.documentCount == 200` (all pinned docs still present)
- `settings.evictedStubCount == 0` (pinned docs never evicted)
- `bulk.evictedStubCount > 0` (proves pressure actually fired — defensive sanity check)
- All 200 settings docs round-trip with byte-exact content
- No sample during the run ever saw settings shrink

If `bulk.evictedStubCount` is `0`, either the run was too short, or the bulk workload produced less than ~32 MB — raise `duration_sec` or `bulk_writers`.

---

## Test 5 — Crash recovery mid-eviction (`run_crash_test.sh`)

The most complex scenario — starts a server, starts an inserter, waits until eviction has started firing, `kill -9`s both, restarts the server with `--force-readwrite`, and verifies every successfully-logged insert is still retrievable byte-exactly.

This is the one that validates v1.6.1 (snapshot durability + recovery) + v1.7.0 (eviction) working together under a hard failure mode.

### Build

    cd tests/load_test
    g++ -std=c++20 -O2 -I/usr/include load_test_crash_insert.cpp \
        -lsmartbotic-db-client $(pkg-config --libs grpc++) \
        -lspdlog -lfmt -pthread -o load_test_crash_insert
    g++ -std=c++20 -O2 -I/usr/include load_test_crash_verify.cpp \
        -lsmartbotic-db-client $(pkg-config --libs grpc++) \
        -lspdlog -lfmt -pthread -o load_test_crash_verify
    chmod +x run_crash_test.sh

### Run

    ./run_crash_test.sh

Exit code:
- 0 — every logged insert recovered byte-exactly (PASS)
- non-zero — either the server refused to restart or content mismatched (see script output)

### What it does under the hood

1. Wipes `/tmp/smartbotic-loadtest-crash/`, makes fresh dirs
2. Writes `config_crash.json` based on `config.json` with the crash data dir
3. Starts server, begins inserting 20k deterministic docs across 3 writer threads
4. Waits for the first "Eviction chunk" log line, then sleeps 2 s more to ensure we're in the middle of an eviction storm
5. `kill -9` the inserter + the server
6. Restarts the server with `--force-readwrite` (accepts auto-readonly non-trivial-recovery state)
7. Reads every ID that the inserter successfully logged before the crash; verifies content matches `makeContent(i)`

### PASS criteria

- Every ID in `/tmp/smartbotic-loadtest-crash/inserted.log` is retrievable
- Every content byte-exact matches its expected value
- Zero not-found, zero mismatches

### If the server refuses to restart (exit code 10)

The recovery system decided the snapshot+WAL combination was unrecoverable. `run_crash_test.sh` then dumps the last 30 lines of `server2.log` — look for `RECOVERY REFUSED`. Escalate manually:

    smartbotic-database --config /tmp/smartbotic-loadtest-crash/config_crash.json \
        --recovery-mode=snapshot_fallback --force-readwrite

---

## Test 6 — Replication sanity check (`run_replication_test.sh`)

Spawns two local server instances (leader on 9005, follower on 9006), each configured with the other as a replication peer, inserts 1000 docs on the leader, then polls the follower every 200 ms until it sees them. Reports catch-up time and content integrity.

This is a smoke test — it does NOT exercise replication at real-deployment scale. It verifies the replication mechanism still fires alongside v1.7.0's eviction / admission-control / client-retry changes. Because v1.7.0 didn't touch replication code, the expected outcome is "no regression" — if the mechanism breaks here, bisect toward something we did change (WAL ordering, store callbacks).

### Build

    cd tests/load_test
    g++ -std=c++20 -O2 -I/usr/include load_test_replication.cpp \
        -lsmartbotic-db-client $(pkg-config --libs grpc++) \
        -lspdlog -lfmt -pthread -o load_test_replication
    chmod +x run_replication_test.sh

### Run

    ./run_replication_test.sh [num_docs=1000] [max_wait_sec=30]

### Configs

- `config_replica_leader.json` — port 9005, node_id `replica-leader`, replication enabled, peer = `127.0.0.1:9006`, data dir `/tmp/smartbotic-loadtest-replica-leader/data`
- `config_replica_follower.json` — port 9006, node_id `replica-follower`, replication enabled, peer = `127.0.0.1:9005`, data dir `/tmp/smartbotic-loadtest-replica-follower/data`

Both configs use symmetric multi-master peering (as the replication code supports) — each side lists the other in `peers`. Inserts are performed only on the leader in this test; the follower is a pure consumer, but could also accept writes.

### PASS criteria

- All N docs visible on the follower
- Zero content mismatches (the body field matches what the leader inserted)
- Catch-up time < `max_wait_sec` seconds

Exit codes:

- `0` — PASS (full catch-up, zero corruption)
- `2` — PARTIAL (some docs replicated, no corruption, did not fully catch up in time)
- `1` — FAIL (either nothing replicated OR content mismatches observed)

### Known quirk found while writing this test (pre-v1.7.0)

A first run of this test against v1.7.0 produced **FAIL — 200 docs replicated but zero content matches**. The follower's copy of each document contained only the system fields (`_id`, `_version`, `_created_at`, `_updated_at`, etc.), with the user-supplied payload fields missing.

Root cause is in the leader-side broadcast code at `service/src/database_service.cpp:515`:

    if (doc) entry.set_data(doc->data.dump());

This serializes only the document's `data` field. The follower-side apply code at `service/src/database_service.cpp:612` expects the full Document envelope:

    auto json = nlohmann::json::parse(entry.data());
    Document doc = Document::fromJson(json);   // looks for "data", "version", etc.

`Document::fromJson` reads `j.value("data", {})` — i.e. it expects the payload nested under `data`, not at the top level. So the follower effectively discards the body.

`git blame` shows this mismatch predates v1.7.0 (commit `a96c168`, 2026-02-05). It is a latent bug in the push path: inserts/updates are getting replicated structurally (the system fields propagate) but with an empty body on the follower.

The pull path (`GetEntriesSince` → WAL) uses a different code path — it serializes the WAL entry's `data` field directly, so pull-based catch-up may or may not exhibit the same issue depending on WAL format. Not investigated here.

This test catches the bug but is deliberately not wired to fix it — the goal of Test 6 is v1.7.0 sanity, and the bug is unrelated to the eviction refactor. File a separate ticket to either:

1. On the leader, use `doc->toJson().dump()` instead of `doc->data.dump()`, OR
2. On the follower, parse `entry.data()` as the raw document body and construct the Document from `entry`'s other fields (`collection`, `document_id`, `node_id`).

### If replication silently doesn't happen

Check both server logs for the peer handshake:

    "Connected to peer 127.0.0.1:9006 (node: replica-follower, ...)"
    "Connected to peer 127.0.0.1:9005 (node: replica-leader, ...)"

Without both lines, peering never established — likely a firewall or port conflict. `ss -tlnp | grep 900[56]` will confirm both are listening.