#1 [BUG] Project collection registry lost on restart — collections appear empty/404 (data survives)

Version: 0.1.6 · deployment https://rag001.smartbotics.ai

Symptom

After a server restart, every project's collections vanish from the API even though the data is still on disk:

• GET /api/v1/projects/{project}/collections → {"collections": []}
• GET .../collections/{c}/documents → 404 {"error":{"code":"not_found","message":"no such collection"}}

Observed across all projects at once (default, sungrow, toloajto) — not project-specific. Meanwhile API keys and projects survive the restart.

Why it's misleading

The underlying DB-layer collections are not actually lost — only the per-project registry index is. Proof: issuing any management call rebuilds it. When I POST .../collections to recreate them, 7 of 8 returned 422 "collection already exists" and immediately reappeared with their document counts intact; only the one genuinely-missing collection got recreated.

Root cause (suspected)

• The per-project registry lives in qualify(project, "vectorapi_collections") (src/collection_registry.hpp). On restart this index is empty/stale, while the actual qualify(project, <name>) collections persist.
• vectorapi_keys survives (and the admin key is re-bootstrapped from SMARTBOTIC_VECTORAPI_KEY at src/main.cpp:61), so persistence is inconsistent between the key store and the per-project collection registry.
• This is the desync already flagged in-code: see the "DB v2.3.1 Stage F caveat" comment in src/handlers/collections.cpp:28-32. The POST-collection handler calls d->registry.ensure(project) (collections.cpp:14), which is what rebuilds the index from the surviving DB-layer collections — but nothing triggers that on startup.

Impact

Harmless pre-launch (empty designs/quotes). Post-launch this is data-integrity-critical: customer-saved designs and submitted quotes will survive on disk but become invisible to the app after any restart/redeploy until a management call happens to run registry.ensure(). Reads via scoped/public keys (which only do document ops, never management) will never trigger the rebuild, so a client app stays broken indefinitely.

Proposed fix

1. Minimal / safe: on startup, enumerate all known projects and call registry.ensure(project) for each, rebuilding every index from the DB-layer collections. (Need a reliable project list at boot — e.g. derive from existing qualify(*, …) collections or a projects registry; verify the enumeration source.)
2. Root-cause: make the per-project *_vectorapi_collections registry persist across restarts the same way vectorapi_keys does, and investigate why one persists and the other doesn't (same DbGateway).

Consider a regression test: seed a collection → simulate restart (drop the in-memory/registry state, keep the data store) → assert GET /collections still lists it.

Workaround (until fixed)

Hit any management endpoint per project after a restart (e.g. GET /api/v1/projects/{p}/collections with an admin key, or recreate one collection) to force registry.ensure() and rebuild the index.

Root cause confirmed and fixed (commit 7e22bbb).

Why keys survive but the registry didn't: KeyStore::bootstrap() re-opens vectorapi_keys on every startup (getCollectionInfo → createCollection), so the DB surfaces it via find() after a restart. The collection registry has the same ensure() primitive, but startup (main.cpp) only ran it for the default project — named projects (sungrow, toloajto, …) were never re-opened until a management call lazily ran ensure(). Their per-project vectorapi_collections index therefore stayed invisible to find()/list() after a restart, while the underlying data (and get-by-id) survived. That matches every reported symptom: GET /collections empty, document ops 404, recreate → 422 "already exists", any management call fixing it.

Fix: added CollectionRegistry::bootstrap(projects) and call it for every project from db.listProjects() at startup (mirrors the key-store pattern). Regression guard added in tests/test_registry.cpp.

This implements proposed fix #1. A true restart-survival assertion can't run in ctest (the lost state lives in the DB server's catalog, not the app), so it will be verified on prod by restarting after deploy.

Pending: release + deploy to rag001 — will verify the symptom is gone post-restart, then close.

Verified fixed on prod (rag001) — closing.

Deployed the fix and did a full cold restart (restarted smartbotic-database and smartbotic-vectorapi), then confirmed every project's registry survived:

toloajto:vectorapi_collections (8 docs)
sungrow:vectorapi_collections  (2 docs)
sungrow:datasheets             (165 docs)   # data intact

Startup log: opened collection registry for 3 project(s) at startup; /readyz OK. Before the fix, named-project collections returned empty after a restart.

Heads-up / incident note (unrelated to this bug): the first deploy crash-looped (SIGSEGV at startup) because vectorapi had been built against the 2.3.x libsmartbotic-db-client while the prod DB was upgraded to 2.4.0 — an ABI mismatch that stayed latent until the process restarted. Fixed by rebuilding vectorapi against the 2.4.0 client lib and shipping 0.1.7-2. (Lesson: rebuild + redeploy consumers right after a DB client-lib major bump.)

Fixed in commit 7e22bbb (released as 0.1.7, packaged 0.1.7-2).