smartbotic-database/tests/load_test

Load tests — eviction under memory pressure

Standalone binaries that validate v1.7.0's chunked eviction + admission control + client retry against a real smartbotic-database instance.

Not unit tests — each one runs a real server on a configurable port, drives a workload, and reports a PASS/FAIL verdict. Not wired into CTest — these are operator/developer validation tools, not CI.

Test suite

Four load-test binaries:

• load_test — pressure + retry validation (the original)
• load_test_mixed — concurrent reads + writes under pressure
• load_test_integrity — byte-exact integrity of evicted + paged-in docs
• load_test_pinned — pinned=true collection protection

All four talk to localhost:9005 and use Client::Config with transport-level retries. They differ only in workload shape and verdict criteria.

Prerequisites

• smartbotic-database package installed (for the server binary)
• libsmartbotic-db-client-dev installed (for headers + shared lib)
• GCC with C++20 support

Build

cd tests/load_test
g++ -std=c++20 -O2 -I/usr/include load_test.cpp \
    -lsmartbotic-db-client \
    $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread \
    -o load_test

Run

Uses port 9005 and data dir /tmp/smartbotic-loadtest/ to avoid conflicting with any existing instance on the default port 9004.

# 1. Start the test server with its own config (memory capped at 32 MB)
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

# 2. Run the load test — default: 20s, 4 writers, 2 KB docs
./load_test [duration_seconds] [writer_threads] [doc_bytes]

# Example — 30s with 8 writers:
./load_test 30 8 2048

# 3. Stop the server
kill $SERVER_PID

# 4. Inspect eviction events
grep -E "Eviction|MEMORY_" /tmp/smartbotic-loadtest/server.log

What to look for

Sample output from a successful run (32 MB cap, 20s, 4 writers, 2 KB docs):

[t=18504ms] pressure=     hard  84% mem=27584KB live=  5276 evicted= 41580 inserts=  46856 fails=    0

=== Summary ===
Duration:                 20162ms
Successful inserts:       51444 (2551 ops/s)
Failed (after retries):   0
Pressure timeline:
  normal:  1.5s
  soft:    5.0s
  hard:    9.0s
  emergency: 2.0s
  Transitions: 19

Verdict: PASS — client-side retry absorbed all transient errors

Key signals

Signal	What it means
Eviction chunk: N docs evicted, X bytes freed	T4 chunked eviction is working
Eviction: nothing evictable this pass, backing off	T4 hot-write floor is protecting in-flight upserts
Emitted MEMORY_PRESSURE_HIGH	T10 edge-triggered event fired
Insert rejected: memory pressure emergency	T7 admission control engaged
Client::insert ...; retrying in Nms (attempt K/5)	T11 client retry kicked in
Successful inserts: N == attempts, Failed: 0	No writes lost under pressure

Tuning for different scenarios

Edit config.json:

• Burst write stress: lower hot_write_floor_ms to 500 ms so eviction can actually drop recent writes
• Slow drain: raise eviction_chunk_pause_ms to 200 ms
• Admission control disabled: raise memory_emergency_percent to 99
• Bigger budget: raise max_memory_mb to 256 to see if the chunked approach still works at scale

---

Test 2 — Mixed read/write (load_test_mixed)

Validates that reads stay responsive while writers + eviction are running. Three writer threads continuously insert 2 KB docs into a shared list of recent IDs; five reader threads pick random IDs from that list and call get(). Reads hit both live docs and evicted-stub IDs, exercising the WAL-fallback paging path.

Build

g++ -std=c++20 -O2 -I/usr/include load_test_mixed.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_mixed

Run

# Same server setup as load_test:
mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

./load_test_mixed [duration_sec=30] [writers=3] [readers=5] [doc_bytes=2048]

kill $SERVER_PID

PASS criteria

• writer ops/s > 500
• reader ops/s > 1000 (reads should be faster than writes — no eviction gate on reads)
• reader p99 latency < 1000 ms
• zero reader hard failures (RESOURCE_EXHAUSTED / timeout / other)

not-found is counted separately from failures — it's a legitimate cold miss from a freshly-evicted stub where the WAL page hasn't been pulled yet, not a bug.

---

Test 3 — Data integrity (load_test_integrity)

Writes 10,000 docs with deterministic makeContent(i) bodies (~1.5 KB each, content includes the id so misdirected reads also fail), waits 3 s for eviction to settle, then reads every doc back and compares byte-for-byte. Runs sequentially — no concurrency on verify, by design.

At 32 MB cap and ~15 MB of data plus overhead, eviction will fire; the verify phase exercises the WAL page-in path for every doc.

Build

g++ -std=c++20 -O2 -I/usr/include load_test_integrity.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_integrity

Run

Uses the same config.json as the base test.

mkdir -p /tmp/smartbotic-loadtest/data /tmp/smartbotic-loadtest/files
smartbotic-database --config ./config.json > /tmp/smartbotic-loadtest/server.log 2>&1 &
SERVER_PID=$!
sleep 2

./load_test_integrity [num_docs=10000] [stabilize_sec=3]

kill $SERVER_PID

PASS criteria

• All N docs retrievable
• Every doc's content field byte-exact matches makeContent(i)
• Zero mismatches, zero not-found, zero read errors

If the summary says no eviction stubs present, the budget is too loose for your dataset size — either bump num_docs or lower max_memory_mb in the config so eviction actually fires during the run.

---

Test 4 — Pinned collection (load_test_pinned)

Confirms that pinned=true collections are never evicted, even under sustained pressure on other collections. Needs a different server config because the settings collection is created pinned by a migration — createCollection via the client API doesn't expose pinned yet.

Build

g++ -std=c++20 -O2 -I/usr/include load_test_pinned.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_pinned

Run

# 1. Set up data dir + migrations dir + copy the migration file.
mkdir -p /tmp/smartbotic-loadtest-pinned/{data,files,migrations}
cp migrations/001_create_pinned_settings.json /tmp/smartbotic-loadtest-pinned/migrations/

# 2. Start the server with the pinned-config. Migration auto-applies on startup.
smartbotic-database --config ./config_pinned.json > /tmp/smartbotic-loadtest-pinned/server.log 2>&1 &
SERVER_PID=$!
sleep 2

# 3. Run the test.
./load_test_pinned [duration_sec=30] [bulk_writers=4] [num_settings=200]

# 4. Stop.
kill $SERVER_PID

PASS criteria

• settings.documentCount == 200 (all pinned docs still present)
• settings.evictedStubCount == 0 (pinned docs never evicted)
• bulk.evictedStubCount > 0 (proves pressure actually fired — defensive sanity check)
• All 200 settings docs round-trip with byte-exact content
• No sample during the run ever saw settings shrink

If bulk.evictedStubCount is 0, either the run was too short, or the bulk workload produced less than ~32 MB — raise duration_sec or bulk_writers.

---

Test 5 — Crash recovery mid-eviction (run_crash_test.sh)

The most complex scenario — starts a server, starts an inserter, waits until eviction has started firing, kill -9s both, restarts the server with --force-readwrite, and verifies every successfully-logged insert is still retrievable byte-exactly.

This is the one that validates v1.6.1 (snapshot durability + recovery) + v1.7.0 (eviction) working together under a hard failure mode.

Build

cd tests/load_test
g++ -std=c++20 -O2 -I/usr/include load_test_crash_insert.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_crash_insert
g++ -std=c++20 -O2 -I/usr/include load_test_crash_verify.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_crash_verify
chmod +x run_crash_test.sh

Run

./run_crash_test.sh

Exit code:

• 0 — every logged insert recovered byte-exactly (PASS)
• non-zero — either the server refused to restart or content mismatched (see script output)

What it does under the hood

1. Wipes /tmp/smartbotic-loadtest-crash/, makes fresh dirs
2. Writes config_crash.json based on config.json with the crash data dir
3. Starts server, begins inserting 20k deterministic docs across 3 writer threads
4. Waits for the first "Eviction chunk" log line, then sleeps 2 s more to ensure we're in the middle of an eviction storm
5. kill -9 the inserter + the server
6. Restarts the server with --force-readwrite (accepts auto-readonly non-trivial-recovery state)
7. Reads every ID that the inserter successfully logged before the crash; verifies content matches makeContent(i)

PASS criteria

• Every ID in /tmp/smartbotic-loadtest-crash/inserted.log is retrievable
• Every content byte-exact matches its expected value
• Zero not-found, zero mismatches

If the server refuses to restart (exit code 10)

The recovery system decided the snapshot+WAL combination was unrecoverable. run_crash_test.sh then dumps the last 30 lines of server2.log — look for RECOVERY REFUSED. Escalate manually:

smartbotic-database --config /tmp/smartbotic-loadtest-crash/config_crash.json \
    --recovery-mode=snapshot_fallback --force-readwrite

---

Test 6 — Replication sanity check (run_replication_test.sh)

Spawns two local server instances (leader on 9005, follower on 9006), each configured with the other as a replication peer, inserts 1000 docs on the leader, then polls the follower every 200 ms until it sees them. Reports catch-up time and content integrity.

This is a smoke test — it does NOT exercise replication at real-deployment scale. It verifies the replication mechanism still fires alongside v1.7.0's eviction / admission-control / client-retry changes. Because v1.7.0 didn't touch replication code, the expected outcome is "no regression" — if the mechanism breaks here, bisect toward something we did change (WAL ordering, store callbacks).

Build

cd tests/load_test
g++ -std=c++20 -O2 -I/usr/include load_test_replication.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_replication
chmod +x run_replication_test.sh

Run

./run_replication_test.sh [num_docs=1000] [max_wait_sec=30]

Configs

• config_replica_leader.json — port 9005, node_id replica-leader, replication enabled, peer = 127.0.0.1:9006, data dir /tmp/smartbotic-loadtest-replica-leader/data
• config_replica_follower.json — port 9006, node_id replica-follower, replication enabled, peer = 127.0.0.1:9005, data dir /tmp/smartbotic-loadtest-replica-follower/data

Both configs use symmetric multi-master peering (as the replication code supports) — each side lists the other in peers. Inserts are performed only on the leader in this test; the follower is a pure consumer, but could also accept writes.

PASS criteria

• All N docs visible on the follower
• Zero content mismatches (the body field matches what the leader inserted)
• Catch-up time < max_wait_sec seconds

Exit codes:

• 0 — PASS (full catch-up, zero corruption)
• 2 — PARTIAL (some docs replicated, no corruption, did not fully catch up in time)
• 1 — FAIL (either nothing replicated OR content mismatches observed)

Known quirk found while writing this test (pre-v1.7.0)

A first run of this test against v1.7.0 produced FAIL — 200 docs replicated but zero content matches. The follower's copy of each document contained only the system fields (_id, _version, _created_at, _updated_at, etc.), with the user-supplied payload fields missing.

Root cause is in the leader-side broadcast code at service/src/database_service.cpp:515:

if (doc) entry.set_data(doc->data.dump());

This serializes only the document's data field. The follower-side apply code at service/src/database_service.cpp:612 expects the full Document envelope:

auto json = nlohmann::json::parse(entry.data());
Document doc = Document::fromJson(json);   // looks for "data", "version", etc.

Document::fromJson reads j.value("data", {}) — i.e. it expects the payload nested under data, not at the top level. So the follower effectively discards the body.

git blame shows this mismatch predates v1.7.0 (commit a96c168, 2026-02-05). It is a latent bug in the push path: inserts/updates are getting replicated structurally (the system fields propagate) but with an empty body on the follower.

The pull path (GetEntriesSince → WAL) uses a different code path — it serializes the WAL entry's data field directly, so pull-based catch-up may or may not exhibit the same issue depending on WAL format. Not investigated here.

This test catches the bug but is deliberately not wired to fix it — the goal of Test 6 is v1.7.0 sanity, and the bug is unrelated to the eviction refactor. File a separate ticket to either:

1. On the leader, use doc->toJson().dump() instead of doc->data.dump(), OR
2. On the follower, parse entry.data() as the raw document body and construct the Document from entry's other fields (collection, document_id, node_id).

If replication silently doesn't happen

Check both server logs for the peer handshake:

"Connected to peer 127.0.0.1:9006 (node: replica-follower, ...)"
"Connected to peer 127.0.0.1:9005 (node: replica-leader, ...)"

Without both lines, peering never established — likely a firewall or port conflict. ss -tlnp | grep 900[56] will confirm both are listening.

---

Test 8 — Replicated eviction (run_replica_eviction_test.sh)

The follower has a deliberately tighter memory cap than the leader. The leader ingests enough data that the follower cannot hold it all in RAM, so the follower must:

1. Accept every replication entry (no RESOURCE_EXHAUSTED on the apply path — that would break sync forever)
2. Evict older docs to make room
3. Still serve every replicated doc correctly on get(), paging evicted ones back in from the WAL

This test catches bugs that only appear under the combined replicate + evict + page-in round trip:

• follower's replication-apply accidentally gated by admission control
• evicted-on-follower docs unreadable because replication took a shortcut past the normal WAL-write path
• content corruption across the replicate -> evict -> page-in cycle

Build

cd tests/load_test
g++ -std=c++20 -O2 -I/usr/include load_test_replica_eviction.cpp \
    -lsmartbotic-db-client $(pkg-config --libs grpc++) \
    -lspdlog -lfmt -pthread -o load_test_replica_eviction
chmod +x run_replica_eviction_test.sh

Run

./run_replica_eviction_test.sh [num_docs=5000] [stabilize_sec=5]

Configs

• config_replica_leader_128mb.json — port 9005, max_memory_mb: 128 (plenty of room), replication enabled, peer = 127.0.0.1:9006, data dir /tmp/smartbotic-loadtest-replica-eviction-leader/data
• config_replica_follower_16mb.json — port 9006, max_memory_mb: 16 (tight — forces eviction), soft/hard/emergency = 50/70/90, replication enabled, peer = 127.0.0.1:9005, data dir /tmp/smartbotic-loadtest-replica-eviction-follower/data

Workload

• 5000 docs x ~2 KB (each has a 1900-byte pad field) inserted on the leader
• In memory terms (~2.5x overhead per v1.7.0 estimateDocumentSize) = ~25 MB on each side
• Leader (128 MB): no pressure
• Follower (16 MB): ~150% over cap -> eviction fires continuously

PASS criteria

• All 5000 docs readable on the follower (live-map + WAL-fallback both exercised)
• Zero content mismatches, zero read errors
• evictedStubCount > 0 (proves eviction fired — if not, the test is INCONCLUSIVE, raise num_docs or lower follower cap)

Exit codes

• 0 — PASS (all docs replicated + byte-exact, eviction fired on follower)
• 1 — FAIL (missing docs or content corruption)
• 2 — INCONCLUSIVE (eviction never fired — follower had room to spare)

Known bug: evicted-on-follower docs not recoverable (as of v1.7.1)

First run of this test against v1.7.1 produced FAIL — all 5000 docs replicated (follower shows 5000 evicted stubs), but zero readable on follower. Server log shows "Failed to recover evicted document replica_evict/repl-<i>" for every id.

Root cause is in DatabaseService::applyReplicatedEntry at service/src/database_service.cpp:623:

// Use loadDocument to bypass normal callbacks (avoid re-replication)
store_->loadDocument(entry.collection(), doc);

MemoryStore::loadDocument only inserts the doc into the in-memory map — it deliberately skips the usual callback chain so the follower does not re-broadcast. But that chain is also what appends the doc to the local WAL. So on the follower the doc lives only in RAM.

When eviction then fires on the follower, the doc is dropped to an evicted-stub. On read, the recovery callback calls persistence_->loadDocument() which scans the follower's WAL — and finds nothing, because replication never wrote it there. Result: permanent data loss the instant the follower evicts, even though the stub still reports the doc as "accounted for".

Fix options (not made in this changeset — the test documents the bug):

1. In applyReplicatedEntry, after store_->loadDocument(...), append to the local WAL (e.g. persistence_->logInsert/logUpdate depending on op), so evicted-on-follower docs have a durable backing store.
2. Teach the eviction callback to fall back to pulling from a peer when the local WAL has no entry — more complex, adds a network hop on cold read.

Attempted fix (option 1) and why it failed

A v1.7.2 attempt added persistence_->logInsert(collection, doc) directly inside applyReplicatedEntry after the loadDocument call. The fix compiled cleanly but caused infinite replication echo amplification when run:

[23:30:14] Memory check: 1954 MB estimated (100%, pressure=emergency),
           408121 docs, 30747 evicted
[23:30:14] Pulled 5000 entries from peer replica-leader-128
           (seq: 211866 -> 216867)

The leader's WAL reached 216867+ entries after inserting only 5000. Root cause is at service/src/database_grpc_impl.cpp:1561 — the GetEntriesSince server handler hard-codes the response's node_id to its OWN nodeState.nodeId, not the entry's original origin. So when follower's WAL is streamed back to the leader, every entry claims to have originated on the follower. The leader then can't recognize its own original writes and re-applies them, re-logs to its WAL, re-streams them to the follower, and the cycle continues.

The proper fix requires:

1. Adding a nodeId field to struct WalEntry (service/src/persistence/wal.hpp:37) — WAL on-disk format change, breaks backward compat without a version bump
2. Plumbing the origin nodeId through persistence_->logInsert/logUpdate/logDelete
3. Using walEntry.nodeId (not nodeState.nodeId) in GetEntriesSince
4. The sync protocol on the sending side skipping entries whose node_id equals the peer being sent to

This is a v1.8.0-class change that warrants its own design + format migration path. For v1.7.x the documented limitation stands: replicated docs on a follower that get evicted are lost. Size the follower's max_memory_mb to hold the working set until a proper replication overhaul ships.

Test output the bug produces

Phase 2: waiting 5s for follower to stabilize...
  [t+2000ms] follower pressure=emergency (100%) live=0 evicted=5000
  ...
Phase 3: reading all 5000 docs from follower...
  not-found id=repl-0
  not-found id=repl-1
  ...
Summary: Found: 0, Not found: 5000
Verdict: FAIL

And in the follower server log:

[warning] Failed to recover evicted document replica_evict/repl-4999

This confirms replication worked (5000 stubs exist on the follower and their IDs match what the leader inserted) — but the evicted-stub recovery path has nothing to page in from.