Selaa lähdekoodia

release(v2.0.0): LMDB storage substrate

This release lands the v2.0 storage substrate per the Phase C
execution plan. v2.x is the project's terminal arc — no v3.0.

## What's in v2.0.0

- **LMDB-backed document storage.** Each user collection becomes one
  sub-database under `<dataDir>/env/`. `LmdbDocumentStore` provides
  put/get/del/scan + filter/sort/projection (semantics ported from
  MemoryStore via the shared `service/src/storage/filter_eval.hpp`).
- **LMDB-backed vector storage.** Collections with `vector_dimension > 0`
  get a sidecar `_vectors_<collection>` sub-db storing raw float32 bytes
  keyed by docId.
- **Dual-write under MemoryStore lock.** `mirrorWriteToDocStore` +
  `mirrorVectorToDocStore` invoked before every per-collection
  `lock.unlock()`. Race-free for concurrent readers; the persist
  callback in DatabaseService now does only WAL + replication + events.
- **All four read handlers LMDB-first.** Exists, Get, Find, and
  SimilaritySearch consult `doc_store_` when `mirror_healthy_` is true
  and `mirror_drift_count_ == 0`. On LMDB throw, each falls back to
  the MemoryStore path. SimilaritySearch uses the shared cosine_simd.hpp
  kernel (AVX2/SSE4.1/scalar fallback) ported from MemoryStore.
- **Boot-time backfill.** `DatabaseService::backfillIntoDocStore()` runs
  synchronously after recovery + migrations and before gRPC accepts
  traffic. Skipped when `migrate_v1_to_v2` has set
  `_meta.schema_version=2`.
- **v1.x → v2.0 auto-migration.** Existing data dirs auto-migrate on
  first boot (`--no-auto-migrate` to opt out). Documents and vectors
  are copied; v1.x snapshots and WAL remain untouched as a rollback
  safety net (paired with v1.9.5's preinst backup).
- **Eviction config deprecation.** `max_memory_mb`, `eviction_chunk_*`,
  `hot_write_floor_ms`, `memory_priority` log a WARN at boot. The
  knobs still apply to MemoryStore (which is still authoritative for
  writes) but RSS is bounded by the LMDB mapsize + OS page cache, not
  by these settings.

## What's deferred to v2.1+

- Write-handler migration: handlers still call MemoryStore directly;
  the LMDB write is a mirror, not the primary. Decommissioning
  MemoryStore needs id-gen + timestamp + version orchestration extracted
  into a WriteCoordinator.
- History (`_history_<collection>`) sub-db migration: still served by
  the v1.9 disk-resident HistoryStore (.hlog files).
- Files (`_files`) sub-db migration: file metadata still in MemoryStore;
  blobs already on disk under `files/`.
- Views (`_views`) sub-db migration: views still stored in the v1.x
  `_views` system collection.
- Buffer-pool resize / hot-copy / adaptive mapsize tooling.

## Measured numbers (load_test_mixed 30s, 4 writers / 8 readers, 32MB
MemoryStore budget, hard pressure at 82%)

- 2360 writes/s, 5690 reads/s, p99 ~100ms, 0 hard failures
- 240k total operations across the run, 66k MemoryStore evictions
- 282MB persisted in LMDB env
- 0 mirror failures, 0 LMDB query failures

## Runtime + build deps

- New runtime: `liblmdb0` (Debian 13)
- New build: `liblmdb-dev`
- Existing deps unchanged: libyyjson0, libspdlog, libssl, libgrpc++, lz4
fszontagh 2 kuukautta sitten
vanhempi
sitoutus
fb16f38163
3 muutettua tiedostoa jossa 27 lisäystä ja 1 poistoa
  1. 0 0
      CLAUDE.md
  2. 1 1
      VERSION
  3. 26 0
      service/src/database_service.cpp

Tiedoston diff-näkymää rajattu, sillä se on liian suuri
+ 0 - 0
CLAUDE.md


+ 1 - 1
VERSION

@@ -1 +1 @@
-1.11.0
+2.0.0

+ 26 - 0
service/src/database_service.cpp

@@ -468,6 +468,32 @@ DatabaseService::Config DatabaseService::parseConfig(const nlohmann::json& json)
         // Memory eviction settings
         if (db.contains("memory")) {
             auto& memory = db["memory"];
+
+            // v2.0 Stage 8 — deprecation log. These knobs control MemoryStore
+            // eviction, which is still active for the MemoryStore-side mirror
+            // but does NOT bound RSS in v2.0 (LMDB mmap is the dominant RSS
+            // contributor). The substrate-level equivalent is the LMDB env
+            // mapsize and OS page cache. v2.1 will rename `max_memory_mb` to
+            // `buffer_pool_size_mb` per the Phase C plan.
+            for (const char* deprecated : {"max_memory_mb",
+                                           "eviction_threshold_percent",
+                                           "eviction_target_percent",
+                                           "eviction_check_interval_ms",
+                                           "eviction_chunk_size",
+                                           "eviction_chunk_pause_ms",
+                                           "max_eviction_passes_per_trigger",
+                                           "hot_write_floor_ms",
+                                           "memory_priority"}) {
+                if (memory.contains(deprecated)) {
+                    spdlog::warn("v2.0 deprecation: storage.memory.{} is deprecated "
+                                 "and will be removed in v2.1. v2.0 RSS is bounded "
+                                 "by the LMDB env mapsize + OS page cache, not by "
+                                 "MemoryStore eviction. Setting still applied to "
+                                 "the MemoryStore mirror for back-compat.",
+                                 deprecated);
+                }
+            }
+
             config.maxMemoryMb = memory.value("max_memory_mb", config.maxMemoryMb);
             config.evictionThresholdPercent = memory.value("eviction_threshold_percent", config.evictionThresholdPercent);
             config.evictionTargetPercent = memory.value("eviction_target_percent", config.evictionTargetPercent);

Kaikkia tiedostoja ei voida näyttää, sillä liian monta tiedostoa muuttui tässä diffissä