Parcourir la source

docs(config): document per-collection timestamp precision

- Collection Configuration section with ns vs ms rationale
- configureCollection usage + warning that it doesn't auto-migrate
- migrateCollectionTimestamps workflow + idempotency semantics
- CLAUDE.md Key Features bullet
fszontagh il y a 3 mois
Parent
commit
18571feb1f
2 fichiers modifiés avec 94 ajouts et 0 suppressions
  1. 1 0
      CLAUDE.md
  2. 93 0
      docs/integration-guide.md

+ 1 - 0
CLAUDE.md

@@ -32,6 +32,7 @@ cmake --build build -j$(nproc) && ./build/tests/test_vector_storage
 - **Vector Storage** — Embedding vectors stored alongside documents with SIMD-accelerated (AVX2/SSE4.1) cosine similarity search via `SimilaritySearch` RPC. Vectors use `_vector` document field, stored in parallel float arrays, persisted through WAL (`VEC_PUT`/`VEC_DELETE`) and snapshots (v3 format). Collection `vector_dimension` option is immutable after creation.
 - **Atomic Partial Updates** — `PatchDocument` RPC merges fields into existing documents atomically (server-side, within collection lock). Client `patch()` method. `update()` uses automatic optimistic locking with retry. See `docs/integration-guide.md` for concurrency patterns.
 - **Views** — Read-only named projections over collections. Include/exclude field lists (dot-notation for nested paths), baked-in filters with all operators (EQ/NE/GT/GTE/LT/LTE/IN/CONTAINS/EXISTS/REGEX/SEARCH) AND-merged with caller filters, optional default sort that caller can override. Stored in `_views` system collection, created via `create_view` migration op or `createView` RPC. Writes on view names are rejected.
+- **Per-collection Timestamp Precision** — collection-level `timestamp_precision` config ("ms" default, "ns" for rapid-write collections). Stamps `_created_at`/`_updated_at` at the configured resolution, cached hot-path lookup. `configureCollection` RPC flips the config atomically; `migrateCollectionTimestamps` RPC converts existing data during maintenance windows (idempotent via 10^15 threshold).
 
 ## Packaging
 

+ 93 - 0
docs/integration-guide.md

@@ -334,6 +334,99 @@ The `op` field in `where` filters accepts either a string name (`EQ`, `NE`, `GT`
 
 The view's filters act as a **security enforcement boundary**. A consumer querying `active_admins` literally cannot see inactive users — the filter is applied server-side, after authentication, before projection. This is how SQL views with `WHERE` clauses work, and how row-level security works in most databases. Any consumer filter is ANDed on top, narrowing further.
 
+### Collection Configuration
+
+Per-collection runtime settings. Currently supports timestamp precision; extensible for future knobs.
+
+#### Timestamp precision
+
+Every document carries `_created_at` and `_updated_at` — int64 timestamps since epoch. By default these are **milliseconds**. Collections that receive rapid-fire writes (LLM streaming chunks, agent tool loops) often see ties at ms resolution that make `ORDER BY _created_at` ambiguous. Configure those collections for **nanoseconds** to get unique, strictly ordered timestamps.
+
+```cpp
+using Cfg = smartbotic::database::Client::CollectionConfig;
+
+// Default: ms (no configuration needed)
+auto id = db.insert("logs", {{"msg", "..."}});
+// doc["_created_at"] is ~13 digits, milliseconds since epoch
+
+// Switch a collection to nanoseconds
+db.configureCollection("llm_tokens", Cfg{"ns"});
+
+// Future inserts into llm_tokens get ns-precision timestamps
+auto id2 = db.insert("llm_tokens", {{"token", "hello"}});
+// doc["_created_at"] is ~19 digits, nanoseconds since epoch
+```
+
+#### Reading the config
+
+```cpp
+auto cfg = db.getCollectionConfig("llm_tokens");
+std::cout << cfg.timestampPrecision << std::endl;  // "ns"
+
+bool explicit_set = db.hasCollectionConfig("llm_tokens");  // true
+bool default_only = db.hasCollectionConfig("logs");        // false (still ms default)
+```
+
+#### What configureCollection does NOT do
+
+**It does not convert existing timestamps.** Flipping a collection from `ms` to `ns` only affects *future* inserts. Pre-existing docs keep whatever precision they had. This keeps the config flip atomic and reversible.
+
+Consequences of mixed precision within one collection:
+- `int64` ordering within the collection is still consistent (ms values are always < 10^15, ns values are always > 10^15, so ns docs sort after ms docs). Queries don't break.
+- But the ns-labeled docs appear to be in the future relative to the ms-labeled docs unless you convert.
+
+#### Migrating existing timestamps
+
+When you're ready to convert legacy data, run the migration helper during a planned maintenance window:
+
+```cpp
+auto result = db.migrateCollectionTimestamps("llm_tokens",
+                                              /*fromPrecision=*/"ms",
+                                              /*toPrecision=*/"ns");
+if (result.success) {
+    std::cout << "migrated " << result.rowsMigrated
+              << " rows, skipped " << result.rowsSkipped << std::endl;
+} else {
+    std::cerr << "failed: " << result.error << std::endl;
+}
+```
+
+Semantics:
+- `ms → ns`: multiply `_created_at` / `_updated_at` by 10^6
+- `ns → ms`: divide by 10^6 (truncation — loses sub-ms precision)
+- **Idempotent and resumable.** Rows that are already in the target range are detected via a 10^15 threshold and skipped. If the migration is interrupted (network drop, deadline), rerun it — already-converted rows will count as `rows_skipped`.
+- `from == to` is a no-op success.
+- **Metadata-only** — document version numbers are NOT incremented, WAL is not appended, history is not updated. This is an infrastructure rewrite, not a user-visible change.
+- Deadline: the client's RPC timeout for this call is 10 minutes — large collections may take a while.
+
+#### Recommended workflow for switching precision on production data
+
+1. Schedule a maintenance window (writes against the collection may pause briefly if needed).
+2. Call `configureCollection(name, {ns})` — atomic config flip; future writes immediately use ns.
+3. Call `migrateCollectionTimestamps(name, "ms", "ns")` — converts existing docs.
+4. Verify with a sample read that `_created_at` values are now in the ns range.
+5. Resume normal traffic.
+
+Between steps 2 and 3, new writes are ns and old docs are ms — this is temporary but normal. Ordering still works because the 10^15 threshold cleanly separates the two ranges.
+
+#### Via migrations
+
+To set timestamp precision declaratively, call `db.configureCollection()` from your consumer project's startup code. The `configure_collection` migration operation will be available in a future version.
+
+```json
+{
+  "version": "021",
+  "name": "llm_tokens_ns_precision",
+  "operations": [
+    {
+      "type": "configure_collection",
+      "collection": "llm_tokens",
+      "config": { "timestamp_precision": "ns" }
+    }
+  ]
+}
+```
+
 ### Querying
 
 ```cpp