smartbotic-database/tests/load_test/README.md

Load tests — eviction under memory pressure

Standalone binaries that validate v1.7.0's chunked eviction + admission control + client retry against a real smartbotic-database instance.

Not unit tests — each one runs a real server on a configurable port, drives a workload, and reports a PASS/FAIL verdict. Not wired into CTest — these are operator/developer validation tools, not CI.

Test suite

Four load-test binaries:

• load_test — pressure + retry validation (the original)
• load_test_mixed — concurrent reads + writes under pressure
• load_test_integrity — byte-exact integrity of evicted + paged-in docs
• load_test_pinned — pinned=true collection protection

All four talk to localhost:9005 and use Client::Config with transport-level retries. They differ only in workload shape and verdict criteria.

Prerequisites

• smartbotic-database package installed (for the server binary)
• libsmartbotic-db-client-dev installed (for headers + shared lib)
• GCC with C++20 support

Build

cd tests/load_test
g++ -std=c++20 -O2 -I/usr/include load_test.cpp \
    -lsmartbotic-db-client \
    $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread \
    -o load_test

Run

Uses port 9005 and data dir /tmp/smartbotic-loadtest/ to avoid conflicting with any existing instance on the default port 9004.

# 1. Start the test server with its own config (memory capped at 32 MB)
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

# 2. Run the load test — default: 20s, 4 writers, 2 KB docs
./load_test [duration_seconds] [writer_threads] [doc_bytes]

# Example — 30s with 8 writers:
./load_test 30 8 2048

# 3. Stop the server
kill $SERVER_PID

# 4. Inspect eviction events
grep -E "Eviction|MEMORY_" /tmp/smartbotic-loadtest/server.log

What to look for

Sample output from a successful run (32 MB cap, 20s, 4 writers, 2 KB docs):

[t=18504ms] pressure=     hard  84% mem=27584KB live=  5276 evicted= 41580 inserts=  46856 fails=    0

=== Summary ===
Duration:                 20162ms
Successful inserts:       51444 (2551 ops/s)
Failed (after retries):   0
Pressure timeline:
  normal:  1.5s
  soft:    5.0s
  hard:    9.0s
  emergency: 2.0s
  Transitions: 19

Verdict: PASS — client-side retry absorbed all transient errors

Key signals

Signal	What it means
Eviction chunk: N docs evicted, X bytes freed	T4 chunked eviction is working
Eviction: nothing evictable this pass, backing off	T4 hot-write floor is protecting in-flight upserts
Emitted MEMORY_PRESSURE_HIGH	T10 edge-triggered event fired
Insert rejected: memory pressure emergency	T7 admission control engaged
Client::insert ...; retrying in Nms (attempt K/5)	T11 client retry kicked in
Successful inserts: N == attempts, Failed: 0	No writes lost under pressure

Tuning for different scenarios

Edit config.json:

• Burst write stress: lower hot_write_floor_ms to 500 ms so eviction can actually drop recent writes
• Slow drain: raise eviction_chunk_pause_ms to 200 ms
• Admission control disabled: raise memory_emergency_percent to 99
• Bigger budget: raise max_memory_mb to 256 to see if the chunked approach still works at scale

---

Test 2 — Mixed read/write (load_test_mixed)

Validates that reads stay responsive while writers + eviction are running. Three writer threads continuously insert 2 KB docs into a shared list of recent IDs; five reader threads pick random IDs from that list and call get(). Reads hit both live docs and evicted-stub IDs, exercising the WAL-fallback paging path.

Build

g++ -std=c++20 -O2 -I/usr/include load_test_mixed.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_mixed

Run

# Same server setup as load_test:
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

./load_test_mixed [duration_sec=30] [writers=3] [readers=5] [doc_bytes=2048]

kill $SERVER_PID

PASS criteria

• writer ops/s > 500
• reader ops/s > 1000 (reads should be faster than writes — no eviction gate on reads)
• reader p99 latency < 1000 ms
• zero reader hard failures (RESOURCE_EXHAUSTED / timeout / other)

not-found is counted separately from failures — it's a legitimate cold miss from a freshly-evicted stub where the WAL page hasn't been pulled yet, not a bug.

---

Test 3 — Data integrity (load_test_integrity)

Writes 10,000 docs with deterministic makeContent(i) bodies (~1.5 KB each, content includes the id so misdirected reads also fail), waits 3 s for eviction to settle, then reads every doc back and compares byte-for-byte. Runs sequentially — no concurrency on verify, by design.

At 32 MB cap and ~15 MB of data plus overhead, eviction will fire; the verify phase exercises the WAL page-in path for every doc.

Build

g++ -std=c++20 -O2 -I/usr/include load_test_integrity.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_integrity

Run

Uses the same config.json as the base test.

mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

./load_test_integrity [num_docs=10000] [stabilize_sec=3]

kill $SERVER_PID

PASS criteria

• All N docs retrievable
• Every doc's content field byte-exact matches makeContent(i)
• Zero mismatches, zero not-found, zero read errors

If the summary says no eviction stubs present, the budget is too loose for your dataset size — either bump num_docs or lower max_memory_mb in the config so eviction actually fires during the run.

---

Test 4 — Pinned collection (load_test_pinned)

Confirms that pinned=true collections are never evicted, even under sustained pressure on other collections. Needs a different server config because the settings collection is created pinned by a migration — createCollection via the client API doesn't expose pinned yet.

Build

g++ -std=c++20 -O2 -I/usr/include load_test_pinned.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_pinned

Run

# 1. Set up data dir + migrations dir + copy the migration file.
mkdir -p /tmp/smartbotic-loadtest-pinned/{data,files,migrations}
cp migrations/001_create_pinned_settings.json /tmp/smartbotic-loadtest-pinned/migrations/

# 2. Start the server with the pinned-config. Migration auto-applies on startup.
smartbotic-database --config ./config_pinned.json > /tmp/smartbotic-loadtest-pinned/server.log 2>&1 &
SERVER_PID=$!
sleep 2

# 3. Run the test.
./load_test_pinned [duration_sec=30] [bulk_writers=4] [num_settings=200]

# 4. Stop.
kill $SERVER_PID

PASS criteria

• settings.documentCount == 200 (all pinned docs still present)
• settings.evictedStubCount == 0 (pinned docs never evicted)
• bulk.evictedStubCount > 0 (proves pressure actually fired — defensive sanity check)
• All 200 settings docs round-trip with byte-exact content
• No sample during the run ever saw settings shrink

If bulk.evictedStubCount is 0, either the run was too short, or the bulk workload produced less than ~32 MB — raise duration_sec or bulk_writers.