smartbotic-database/tests/load_test/README.md

Load tests — eviction under memory pressure

Standalone binaries that validate v1.7.0's chunked eviction + admission control + client retry against a real smartbotic-database instance.

Not unit tests — each one runs a real server on a configurable port, drives a workload, and reports a PASS/FAIL verdict. Not wired into CTest — these are operator/developer validation tools, not CI.

Test suite

Four load-test binaries:

• load_test — pressure + retry validation (the original)
• load_test_mixed — concurrent reads + writes under pressure
• load_test_integrity — byte-exact integrity of evicted + paged-in docs
• load_test_pinned — pinned=true collection protection

All four talk to localhost:9005 and use Client::Config with transport-level retries. They differ only in workload shape and verdict criteria.

Prerequisites

• smartbotic-database package installed (for the server binary)
• libsmartbotic-db-client-dev installed (for headers + shared lib)
• GCC with C++20 support

Build

cd tests/load_test
g++ -std=c++20 -O2 -I/usr/include load_test.cpp \
    -lsmartbotic-db-client \
    $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread \
    -o load_test

Run

Uses port 9005 and data dir /tmp/smartbotic-loadtest/ to avoid conflicting with any existing instance on the default port 9004.

# 1. Start the test server with its own config (memory capped at 32 MB)
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

# 2. Run the load test — default: 20s, 4 writers, 2 KB docs
./load_test [duration_seconds] [writer_threads] [doc_bytes]

# Example — 30s with 8 writers:
./load_test 30 8 2048

# 3. Stop the server
kill $SERVER_PID

# 4. Inspect eviction events
grep -E "Eviction|MEMORY_" /tmp/smartbotic-loadtest/server.log

What to look for

Sample output from a successful run (32 MB cap, 20s, 4 writers, 2 KB docs):

[t=18504ms] pressure=     hard  84% mem=27584KB live=  5276 evicted= 41580 inserts=  46856 fails=    0

=== Summary ===
Duration:                 20162ms
Successful inserts:       51444 (2551 ops/s)
Failed (after retries):   0
Pressure timeline:
  normal:  1.5s
  soft:    5.0s
  hard:    9.0s
  emergency: 2.0s
  Transitions: 19

Verdict: PASS — client-side retry absorbed all transient errors

Key signals

Signal	What it means
Eviction chunk: N docs evicted, X bytes freed	T4 chunked eviction is working
Eviction: nothing evictable this pass, backing off	T4 hot-write floor is protecting in-flight upserts
Emitted MEMORY_PRESSURE_HIGH	T10 edge-triggered event fired
Insert rejected: memory pressure emergency	T7 admission control engaged
Client::insert ...; retrying in Nms (attempt K/5)	T11 client retry kicked in
Successful inserts: N == attempts, Failed: 0	No writes lost under pressure

Tuning for different scenarios

Edit config.json:

• Burst write stress: lower hot_write_floor_ms to 500 ms so eviction can actually drop recent writes
• Slow drain: raise eviction_chunk_pause_ms to 200 ms
• Admission control disabled: raise memory_emergency_percent to 99
• Bigger budget: raise max_memory_mb to 256 to see if the chunked approach still works at scale

---

Test 2 — Mixed read/write (load_test_mixed)

Validates that reads stay responsive while writers + eviction are running. Three writer threads continuously insert 2 KB docs into a shared list of recent IDs; five reader threads pick random IDs from that list and call get(). Reads hit both live docs and evicted-stub IDs, exercising the WAL-fallback paging path.

Build

g++ -std=c++20 -O2 -I/usr/include load_test_mixed.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_mixed

Run

# Same server setup as load_test:
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

./load_test_mixed [duration_sec=30] [writers=3] [readers=5] [doc_bytes=2048]

kill $SERVER_PID

PASS criteria

• writer ops/s > 500
• reader ops/s > 1000 (reads should be faster than writes — no eviction gate on reads)
• reader p99 latency < 1000 ms
• zero reader hard failures (RESOURCE_EXHAUSTED / timeout / other)

not-found is counted separately from failures — it's a legitimate cold miss from a freshly-evicted stub where the WAL page hasn't been pulled yet, not a bug.

---

Test 3 — Data integrity (load_test_integrity)

Writes 10,000 docs with deterministic makeContent(i) bodies (~1.5 KB each, content includes the id so misdirected reads also fail), waits 3 s for eviction to settle, then reads every doc back and compares byte-for-byte. Runs sequentially — no concurrency on verify, by design.

At 32 MB cap and ~15 MB of data plus overhead, eviction will fire; the verify phase exercises the WAL page-in path for every doc.

Build

g++ -std=c++20 -O2 -I/usr/include load_test_integrity.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_integrity

Run

Uses the same config.json as the base test.

mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

./load_test_integrity [num_docs=10000] [stabilize_sec=3]

kill $SERVER_PID

PASS criteria

• All N docs retrievable
• Every doc's content field byte-exact matches makeContent(i)
• Zero mismatches, zero not-found, zero read errors

If the summary says no eviction stubs present, the budget is too loose for your dataset size — either bump num_docs or lower max_memory_mb in the config so eviction actually fires during the run.

---

Test 4 — Pinned collection (load_test_pinned)

Confirms that pinned=true collections are never evicted, even under sustained pressure on other collections. Needs a different server config because the settings collection is created pinned by a migration — createCollection via the client API doesn't expose pinned yet.

Build

g++ -std=c++20 -O2 -I/usr/include load_test_pinned.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_pinned

Run

# 1. Set up data dir + migrations dir + copy the migration file.
mkdir -p /tmp/smartbotic-loadtest-pinned/{data,files,migrations}
cp migrations/001_create_pinned_settings.json /tmp/smartbotic-loadtest-pinned/migrations/

# 2. Start the server with the pinned-config. Migration auto-applies on startup.
smartbotic-database --config ./config_pinned.json > /tmp/smartbotic-loadtest-pinned/server.log 2>&1 &
SERVER_PID=$!
sleep 2

# 3. Run the test.
./load_test_pinned [duration_sec=30] [bulk_writers=4] [num_settings=200]

# 4. Stop.
kill $SERVER_PID

PASS criteria

• settings.documentCount == 200 (all pinned docs still present)
• settings.evictedStubCount == 0 (pinned docs never evicted)
• bulk.evictedStubCount > 0 (proves pressure actually fired — defensive sanity check)
• All 200 settings docs round-trip with byte-exact content
• No sample during the run ever saw settings shrink

If bulk.evictedStubCount is 0, either the run was too short, or the bulk workload produced less than ~32 MB — raise duration_sec or bulk_writers.

---

Test 5 — Crash recovery mid-eviction (run_crash_test.sh)

The most complex scenario — starts a server, starts an inserter, waits until eviction has started firing, kill -9s both, restarts the server with --force-readwrite, and verifies every successfully-logged insert is still retrievable byte-exactly.

This is the one that validates v1.6.1 (snapshot durability + recovery) + v1.7.0 (eviction) working together under a hard failure mode.

Build

cd tests/load_test
g++ -std=c++20 -O2 -I/usr/include load_test_crash_insert.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_crash_insert
g++ -std=c++20 -O2 -I/usr/include load_test_crash_verify.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_crash_verify
chmod +x run_crash_test.sh

Run

./run_crash_test.sh

Exit code:

• 0 — every logged insert recovered byte-exactly (PASS)
• non-zero — either the server refused to restart or content mismatched (see script output)

What it does under the hood

1. Wipes /tmp/smartbotic-loadtest-crash/, makes fresh dirs
2. Writes config_crash.json based on config.json with the crash data dir
3. Starts server, begins inserting 20k deterministic docs across 3 writer threads
4. Waits for the first "Eviction chunk" log line, then sleeps 2 s more to ensure we're in the middle of an eviction storm
5. kill -9 the inserter + the server
6. Restarts the server with --force-readwrite (accepts auto-readonly non-trivial-recovery state)
7. Reads every ID that the inserter successfully logged before the crash; verifies content matches makeContent(i)

PASS criteria

• Every ID in /tmp/smartbotic-loadtest-crash/inserted.log is retrievable
• Every content byte-exact matches its expected value
• Zero not-found, zero mismatches

If the server refuses to restart (exit code 10)

The recovery system decided the snapshot+WAL combination was unrecoverable. run_crash_test.sh then dumps the last 30 lines of server2.log — look for RECOVERY REFUSED. Escalate manually:

smartbotic-database --config /tmp/smartbotic-loadtest-crash/config_crash.json \
    --recovery-mode=snapshot_fallback --force-readwrite

---

Test 6 — Replication sanity check (run_replication_test.sh)

Spawns two local server instances (leader on 9005, follower on 9006), each configured with the other as a replication peer, inserts 1000 docs on the leader, then polls the follower every 200 ms until it sees them. Reports catch-up time and content integrity.

This is a smoke test — it does NOT exercise replication at real-deployment scale. It verifies the replication mechanism still fires alongside v1.7.0's eviction / admission-control / client-retry changes. Because v1.7.0 didn't touch replication code, the expected outcome is "no regression" — if the mechanism breaks here, bisect toward something we did change (WAL ordering, store callbacks).

Build

cd tests/load_test
g++ -std=c++20 -O2 -I/usr/include load_test_replication.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_replication
chmod +x run_replication_test.sh

Run

./run_replication_test.sh [num_docs=1000] [max_wait_sec=30]

Configs

• config_replica_leader.json — port 9005, node_id replica-leader, replication enabled, peer = 127.0.0.1:9006, data dir /tmp/smartbotic-loadtest-replica-leader/data
• config_replica_follower.json — port 9006, node_id replica-follower, replication enabled, peer = 127.0.0.1:9005, data dir /tmp/smartbotic-loadtest-replica-follower/data

Both configs use symmetric multi-master peering (as the replication code supports) — each side lists the other in peers. Inserts are performed only on the leader in this test; the follower is a pure consumer, but could also accept writes.

PASS criteria

• All N docs visible on the follower
• Zero content mismatches (the body field matches what the leader inserted)
• Catch-up time < max_wait_sec seconds

Exit codes:

• 0 — PASS (full catch-up, zero corruption)
• 2 — PARTIAL (some docs replicated, no corruption, did not fully catch up in time)
• 1 — FAIL (either nothing replicated OR content mismatches observed)

Known quirk found while writing this test (pre-v1.7.0)

A first run of this test against v1.7.0 produced FAIL — 200 docs replicated but zero content matches. The follower's copy of each document contained only the system fields (_id, _version, _created_at, _updated_at, etc.), with the user-supplied payload fields missing.

Root cause is in the leader-side broadcast code at service/src/database_service.cpp:515:

if (doc) entry.set_data(doc->data.dump());

This serializes only the document's data field. The follower-side apply code at service/src/database_service.cpp:612 expects the full Document envelope:

auto json = nlohmann::json::parse(entry.data());
Document doc = Document::fromJson(json);   // looks for "data", "version", etc.

Document::fromJson reads j.value("data", {}) — i.e. it expects the payload nested under data, not at the top level. So the follower effectively discards the body.

git blame shows this mismatch predates v1.7.0 (commit a96c168, 2026-02-05). It is a latent bug in the push path: inserts/updates are getting replicated structurally (the system fields propagate) but with an empty body on the follower.

The pull path (GetEntriesSince → WAL) uses a different code path — it serializes the WAL entry's data field directly, so pull-based catch-up may or may not exhibit the same issue depending on WAL format. Not investigated here.

This test catches the bug but is deliberately not wired to fix it — the goal of Test 6 is v1.7.0 sanity, and the bug is unrelated to the eviction refactor. File a separate ticket to either:

1. On the leader, use doc->toJson().dump() instead of doc->data.dump(), OR
2. On the follower, parse entry.data() as the raw document body and construct the Document from entry's other fields (collection, document_id, node_id).

If replication silently doesn't happen

Check both server logs for the peer handshake:

"Connected to peer 127.0.0.1:9006 (node: replica-follower, ...)"
"Connected to peer 127.0.0.1:9005 (node: replica-leader, ...)"

Without both lines, peering never established — likely a firewall or port conflict. ss -tlnp | grep 900[56] will confirm both are listening.