# Load tests — eviction under memory pressure

Standalone binaries that validate v1.7.0's chunked eviction + admission control + client retry against a real smartbotic-database instance.

**Not unit tests** — each one runs a real server on a configurable port, drives a workload, and reports a PASS/FAIL verdict. Not wired into CTest — these are operator/developer validation tools, not CI.

## Test suite

Four load-test binaries:

- `load_test` — pressure + retry validation (the original)
- `load_test_mixed` — concurrent reads + writes under pressure
- `load_test_integrity` — byte-exact integrity of evicted + paged-in docs
- `load_test_pinned` — `pinned=true` collection protection

All four talk to `localhost:9005` and use `Client::Config` with transport-level retries. They differ only in workload shape and verdict criteria.

## Prerequisites

- `smartbotic-database` package installed (for the server binary)
- `libsmartbotic-db-client-dev` installed (for headers + shared lib)
- GCC with C++20 support

## Build

```bash
cd tests/load_test
g++ -std=c++20 -O2 -I/usr/include load_test.cpp \
    -lsmartbotic-db-client \
    $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread \
    -o load_test
```

## Run

Uses port `9005` and data dir `/tmp/smartbotic-loadtest/` to avoid conflicting with any existing instance on the default port `9004`.

```bash
# 1. Start the test server with its own config (memory capped at 32 MB)
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

# 2. Run the load test — default: 20s, 4 writers, 2 KB docs
./load_test [duration_seconds] [writer_threads] [doc_bytes]

# Example — 30s with 8 writers:
./load_test 30 8 2048

# 3. Stop the server
kill $SERVER_PID

# 4. Inspect eviction events
grep -E "Eviction|MEMORY_" /tmp/smartbotic-loadtest/server.log
```

## What to look for

Sample output from a successful run (32 MB cap, 20s, 4 writers, 2 KB docs):

```
[t=18504ms] pressure=     hard  84% mem=27584KB live=  5276 evicted= 41580 inserts=  46856 fails=    0

=== Summary ===
Duration:                 20162ms
Successful inserts:       51444 (2551 ops/s)
Failed (after retries):   0
Pressure timeline:
  normal:  1.5s
  soft:    5.0s
  hard:    9.0s
  emergency: 2.0s
  Transitions: 19

Verdict: PASS — client-side retry absorbed all transient errors
```

## Key signals

| Signal | What it means |
|--------|---------------|
| `Eviction chunk: N docs evicted, X bytes freed` | T4 chunked eviction is working |
| `Eviction: nothing evictable this pass, backing off` | T4 hot-write floor is protecting in-flight upserts |
| `Emitted MEMORY_PRESSURE_HIGH` | T10 edge-triggered event fired |
| `Insert rejected: memory pressure emergency` | T7 admission control engaged |
| `Client::insert ...; retrying in Nms (attempt K/5)` | T11 client retry kicked in |
| `Successful inserts: N == attempts, Failed: 0` | No writes lost under pressure |

## Tuning for different scenarios

Edit `config.json`:

- **Burst write stress**: lower `hot_write_floor_ms` to `500` ms so eviction can actually drop recent writes
- **Slow drain**: raise `eviction_chunk_pause_ms` to `200` ms
- **Admission control disabled**: raise `memory_emergency_percent` to `99`
- **Bigger budget**: raise `max_memory_mb` to `256` to see if the chunked approach still works at scale

---

## Test 2 — Mixed read/write (`load_test_mixed`)

Validates that reads stay responsive while writers + eviction are running. Three writer threads continuously insert 2 KB docs into a shared list of recent IDs; five reader threads pick random IDs from that list and call `get()`. Reads hit both live docs and evicted-stub IDs, exercising the WAL-fallback paging path.

### Build

```bash
g++ -std=c++20 -O2 -I/usr/include load_test_mixed.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_mixed
```

### Run

```bash
# Same server setup as load_test:
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

./load_test_mixed [duration_sec=30] [writers=3] [readers=5] [doc_bytes=2048]

kill $SERVER_PID
```

### PASS criteria

- writer ops/s > 500
- reader ops/s > 1000 (reads should be faster than writes — no eviction gate on reads)
- reader p99 latency < 1000 ms
- zero reader hard failures (RESOURCE_EXHAUSTED / timeout / other)

`not-found` is counted separately from failures — it's a legitimate cold miss from a freshly-evicted stub where the WAL page hasn't been pulled yet, not a bug.

---

## Test 3 — Data integrity (`load_test_integrity`)

Writes 10,000 docs with deterministic `makeContent(i)` bodies (~1.5 KB each, content includes the id so misdirected reads also fail), waits 3 s for eviction to settle, then reads every doc back and compares byte-for-byte. Runs sequentially — no concurrency on verify, by design.

At 32 MB cap and ~15 MB of data plus overhead, eviction will fire; the verify phase exercises the WAL page-in path for every doc.

### Build

```bash
g++ -std=c++20 -O2 -I/usr/include load_test_integrity.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_integrity
```

### Run

Uses the same `config.json` as the base test.

```bash
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

./load_test_integrity [num_docs=10000] [stabilize_sec=3]

kill $SERVER_PID
```

### PASS criteria

- All N docs retrievable
- Every doc's `content` field byte-exact matches `makeContent(i)`
- Zero mismatches, zero not-found, zero read errors

If the summary says `no eviction stubs present`, the budget is too loose for your dataset size — either bump `num_docs` or lower `max_memory_mb` in the config so eviction actually fires during the run.

---

## Test 4 — Pinned collection (`load_test_pinned`)

Confirms that `pinned=true` collections are never evicted, even under sustained pressure on other collections. Needs a different server config because the `settings` collection is created pinned by a migration — `createCollection` via the client API doesn't expose `pinned` yet.

### Build

```bash
g++ -std=c++20 -O2 -I/usr/include load_test_pinned.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_pinned
```

### Run

```bash
# 1. Set up data dir + migrations dir + copy the migration file.
mkdir -p /tmp/smartbotic-loadtest-pinned/{data,files,migrations}
cp migrations/001_create_pinned_settings.json /tmp/smartbotic-loadtest-pinned/migrations/

# 2. Start the server with the pinned-config. Migration auto-applies on startup.
smartbotic-database --config ./config_pinned.json > /tmp/smartbotic-loadtest-pinned/server.log 2>&1 &
SERVER_PID=$!
sleep 2

# 3. Run the test.
./load_test_pinned [duration_sec=30] [bulk_writers=4] [num_settings=200]

# 4. Stop.
kill $SERVER_PID
```

### PASS criteria

- `settings.documentCount == 200` (all pinned docs still present)
- `settings.evictedStubCount == 0` (pinned docs never evicted)
- `bulk.evictedStubCount > 0` (proves pressure actually fired — defensive sanity check)
- All 200 settings docs round-trip with byte-exact content
- No sample during the run ever saw settings shrink

If `bulk.evictedStubCount` is `0`, either the run was too short, or the bulk workload produced less than ~32 MB — raise `duration_sec` or `bulk_writers`.