feat: document metadata on the row, ordered by the server - and a precision bug I caused

The document list showed a name and an id. When it was written, by whom,
and at which version were in a panel that only appeared once a document
was opened, so a collection of ten thousand executions gave no way to
tell which row was recent without opening them one at a time. Each row
carries its updated time, its created time when they differ, its version
and who touched it.

Ordering is the server's, newest-updated first. The endpoint takes
sortBy and sortDir, restricted to _updated_at and _created_at - every
document has them, and an arbitrary field name would order by something
half the documents do not carry.

That restriction matters more than it looks: sorting in the browser
orders the twenty-five rows it happens to hold, which over four hundred
pages is not "latest first" at all. The executions page was already doing
the right thing for newest and oldest; its longest/shortest sort cannot
be, because duration is computed and for a running row depends on the
current time. That limitation is now written where somebody will read it
rather than left to be discovered.

**And the sort is what exposed a bug I had caused.** Asking for newest
first returned nanosecond values, oldest first returned milliseconds -
the same field holding two magnitudes. My disable-versioning sweep built
a CollectionConfig, set versioningEnabled, and left the rest "unset". It
is not unset: timestampPrecision defaults to "ms" in that struct, and
only an EMPTY string leaves the collection alone. Sixteen collections
went from ns to ms, and every document written afterwards carried
milliseconds where the older ones carry nanoseconds.

Nothing in our own code broke, because TimeUtils::documentStamp decides
by magnitude rather than dividing blindly. What broke is anything that
compares the raw number - which is exactly what a server-side sort does,
and what a range query would do.

Precision is restored on all sixteen, the 813 documents written in
milliseconds have been rewritten in nanoseconds, and the script that did
it now clears timestampPrecision explicitly with the reason next to it.
The sweep found far more than the 14 a 200-row sample suggested, which is
why it swept rather than trusted the sample.

70 passed, 0 failed.

deploy/zeus/maintenance/disable-versioning.cpp

@@ -35,7 +35,15 @@ int main() {
         for (auto& k : keep) if (n == k) must_keep = true;
         if (must_keep) { printf("  %-28s left ON (its history is read)\n", n.c_str()); kept++; continue; }
         if (!on) { skipped++; continue; }
-        Client::CollectionConfig want;          // precision left unset, so untouched
+        Client::CollectionConfig want;
+        // timestampPrecision has to be cleared explicitly. The struct defaults it
+        // to "ms", and only an EMPTY string leaves the collection's setting
+        // alone - so constructing this and setting one field silently rewrites
+        // the other. It did: sixteen collections went from ns to ms, and every
+        // document written afterwards carried milliseconds in a field where the
+        // older ones carry nanoseconds, which makes any numeric sort or range
+        // over _created_at and _updated_at wrong.
+        want.timestampPrecision = "";
         want.versioningEnabled = false;
         if (c.configureCollection(n, want)) { printf("  %-28s versioning off\n", n.c_str()); off++; }
         else printf("  %-28s FAILED\n", n.c_str());

deploy/zeus/maintenance/restore-timestamp-precision.cpp

@@ -0,0 +1,33 @@
+// Put the timestamp precision back to nanoseconds.
+//
+// My disable-versioning sweep sent CollectionConfig with only versioningEnabled
+// set, on the assumption that the rest was "unset". It is not: the struct's
+// timestampPrecision defaults to "ms", and the header says plainly that an
+// EMPTY string is what leaves it alone. So every collection it touched was
+// switched from ns to ms, and documents written after that carry milliseconds in
+// a field where every older document carries nanoseconds - which makes any
+// numeric sort or range over _created_at/_updated_at wrong.
+#include <smartbotic/database/client.hpp>
+#include <cstdio>
+using namespace smartbotic::database;
+int main() {
+    Client::Config cfg; cfg.address="192.168.2.150:9004"; cfg.project="smartbotic-automation";
+    Client c(cfg);
+    if (!c.connect()) return 1;
+    int fixed=0, ok=0;
+    for (auto& n : c.listCollections()) {
+        auto name = n.rfind(':')==std::string::npos ? n : n.substr(n.rfind(':')+1);
+        auto cur = c.getCollectionConfig(name);
+        if (cur.timestampPrecision == "ns") { ok++; continue; }
+        Client::CollectionConfig want;
+        want.timestampPrecision = "ns";   // versioningEnabled left unset, which DOES leave it alone
+        if (c.configureCollection(name, want)) {
+            printf("  %-28s ms -> ns\n", name.c_str());
+            fixed++;
+        } else {
+            printf("  %-28s FAILED\n", name.c_str());
+        }
+    }
+    printf("restored %d, already ns %d\n", fixed, ok);
+    return 0;
+}

src/webserver/api/database_controller.cpp

@@ -364,6 +364,25 @@ void DatabaseController::getDocuments(const httplib::Request& req, httplib::Resp
     options.page = page;
     options.page_size = page_size;
 
+    // Order across the whole collection, not just the page the client happens to
+    // hold. Sorting 25 rows in the browser puts the newest of those 25 on top,
+    // which over 406 pages of executions is not the same thing at all.
+    //
+    // Restricted to the two metadata stamps on purpose: every document has them,
+    // they are what somebody means by "latest", and an arbitrary field name here
+    // would sort by something half the documents do not carry.
+    if (req.has_param("sortBy")) {
+        const std::string sort_by = req.get_param_value("sortBy");
+        if (sort_by == "_updated_at" || sort_by == "_created_at") {
+            const bool ascending = req.has_param("sortDir")
+                                   && req.get_param_value("sortDir") == "asc";
+            options.sorts.emplace_back(sort_by, ascending);
+        } else {
+            sendError(res, "sortBy must be _updated_at or _created_at", 400);
+            return;
+        }
+    }
+
     auto result = storage_.query(collection, options);
 
     if (result.failed()) {

webui/src/api/workflows.ts

@@ -401,9 +401,23 @@ export const databaseApi = {
     return response.data
   },
 
-  getDocuments: async (collection: string, page = 1, pageSize = 50) => {
+  /**
+   * A page of documents, ordered by the server.
+   *
+   * sortBy takes `_updated_at` or `_created_at` only - every document has them,
+   * and they are what "latest" means. Ordering has to happen server-side: a
+   * client that sorts the page it holds is ordering 25 rows out of ten thousand,
+   * which is not the same answer.
+   */
+  getDocuments: async (
+    collection: string,
+    page = 1,
+    pageSize = 50,
+    sortBy?: '_updated_at' | '_created_at',
+    sortDir: 'asc' | 'desc' = 'desc'
+  ) => {
     const response = await api.get(`/database/collections/${collection}/documents`, {
-      params: { page, pageSize },
+      params: { page, pageSize, ...(sortBy ? { sortBy, sortDir } : {}) },
     })
     return response.data
   },

webui/src/pages/DatabasePage.tsx

@@ -79,6 +79,8 @@ export default function DatabasePage() {
   const [showNewCollectionModal, setShowNewCollectionModal] = useState(false)
   const [showDeleteModal, setShowDeleteModal] = useState(false)
   const [documentsToDelete, setDocumentsToDelete] = useState<string[]>([])
+  const [sortField, setSortField] = useState<'_updated_at' | '_created_at'>('_updated_at')
+  const [sortDescending, setSortDescending] = useState(true)
   const [newCollectionName, setNewCollectionName] = useState('')
   const [collectionSettings, setCollectionSettings] = useState<CollectionSettings>({})
   const [showUnsavedWarning, setShowUnsavedWarning] = useState(false)
@@ -123,8 +125,10 @@ export default function DatabasePage() {
   }, [collectionsData])
 
   const { data: documentsData, isLoading: documentsLoading, error: documentsError } = useQuery({
-    queryKey: ['database-documents', selectedCollection, currentPage],
-    queryFn: () => databaseApi.getDocuments(selectedCollection!, currentPage, PAGE_SIZE),
+    queryKey: ['database-documents', selectedCollection, currentPage, sortField, sortDescending],
+    queryFn: () =>
+      databaseApi.getDocuments(selectedCollection!, currentPage, PAGE_SIZE, sortField,
+                               sortDescending ? 'desc' : 'asc'),
     enabled: !!selectedCollection,
     // A collection the server will not open is not going to open on the third
     // try either, and retrying a refusal only delays saying so.
@@ -446,7 +450,24 @@ export default function DatabasePage() {
     return { systemCollections: system, customCollections: custom }
   }, [collections])
 
-  // Filter documents by search term
+  // The database writes these in nanoseconds. Dividing blindly would be wrong
+  // for anything that ever wrote milliseconds, so the magnitude decides - the
+  // same rule the server uses.
+  const NANOSECOND_THRESHOLD = 100000000000000
+  const stampMs = (raw?: number) => {
+    if (!raw || raw <= 0) return 0
+    return raw > NANOSECOND_THRESHOLD ? Math.floor(raw / 1000000) : raw
+  }
+
+  // Newest first, by when it was last written.
+  //
+  // The list arrived in whatever order the database returned it, which for a
+  // collection of ten thousand executions means the useful rows are wherever
+  // they happen to be. Sorting is over the page the server sent, so it orders
+  // what is on screen rather than the whole collection.
+  // Order comes from the server, so the page is a page of the whole collection
+  // rather than the newest of whatever 25 rows arrived. The search below still
+  // narrows what is on screen - it is a filter over the page, and says so.
   const filteredDocuments = documents.filter((doc) => {
     if (!searchTerm) return true
     const docStr = JSON.stringify(doc).toLowerCase()
@@ -634,10 +655,36 @@ export default function DatabasePage() {
                   type="text"
                   value={searchTerm}
                   onChange={(e) => setSearchTerm(e.target.value)}
-                  placeholder="Search documents..."
+                  placeholder="Search this page..."
                   className="w-full pl-9 pr-3 py-1.5 text-sm border border-gray-300 dark:border-slate-600 rounded-lg bg-white dark:bg-slate-700 text-gray-900 dark:text-gray-100 placeholder-gray-400 dark:placeholder-gray-500 focus:ring-2 focus:ring-primary-500 focus:border-primary-500"
                 />
               </div>
+              {/* Which stamp to order by, and which way. Newest-updated first by
+                  default, because that is the row somebody has come to look at.
+                  It orders the page the server sent, not the whole collection -
+                  said here rather than left to be inferred from a page of 25
+                  that does not start where the previous one ended. */}
+              <div className="mt-2 flex items-center gap-2 text-xs text-gray-500 dark:text-gray-400">
+                <span>Order by</span>
+                <select
+                  value={sortField}
+                  onChange={(e) => setSortField(e.target.value as '_updated_at' | '_created_at')}
+                  className="px-2 py-1 border border-gray-300 dark:border-slate-600 rounded bg-white dark:bg-slate-700 text-gray-900 dark:text-gray-100"
+                >
+                  <option value="_updated_at">updated</option>
+                  <option value="_created_at">created</option>
+                </select>
+                <button
+                  onClick={() => setSortDescending((v) => !v)}
+                  className="px-2 py-1 border border-gray-300 dark:border-slate-600 rounded hover:bg-gray-50 dark:hover:bg-slate-700"
+                  title={sortDescending ? 'Newest first' : 'Oldest first'}
+                >
+                  {sortDescending ? 'newest first' : 'oldest first'}
+                </button>
+                <span className="ml-auto text-gray-400 dark:text-gray-500">
+                  whole collection
+                </span>
+              </div>
             </div>
             {/* Selection info bar */}
             {selectedDocuments.size > 0 && (
@@ -687,6 +734,26 @@ export default function DatabasePage() {
                         {doc.name || doc.username || doc._id}
                       </div>
                       <div className="text-xs text-gray-500 dark:text-gray-400 truncate">{doc._id}</div>
+                      {/* When it was written and by whom, on the row itself. The
+                          panel that showed this only appeared once a document
+                          was opened, so a list of ten thousand executions gave
+                          no way to tell which was recent without opening them
+                          one at a time. */}
+                      <div className="mt-0.5 flex items-center gap-2 text-[11px] text-gray-400 dark:text-gray-500">
+                        {stampMs(doc._updated_at) > 0 && (
+                          <span title={`Updated ${formatTimestamp(stampMs(doc._updated_at))}`}>
+                            {formatTimestamp(stampMs(doc._updated_at))}
+                          </span>
+                        )}
+                        {stampMs(doc._created_at) > 0 &&
+                          stampMs(doc._created_at) !== stampMs(doc._updated_at) && (
+                          <span title={`Created ${formatTimestamp(stampMs(doc._created_at))}`}>
+                            · created {formatTimestamp(stampMs(doc._created_at))}
+                          </span>
+                        )}
+                        {doc._version != null && <span>· v{doc._version}</span>}
+                        {doc._updated_by && <span className="truncate">· {doc._updated_by}</span>}
+                      </div>
                     </div>
                   </button>
                 ))

webui/src/pages/ExecutionsPage.tsx

@@ -254,6 +254,13 @@ export default function ExecutionsPage() {
 
   const executions = useMemo(() => {
     const rows = data?.executions || []
+    // Newest and oldest are sorted by the server, over the whole collection -
+    // see sortOrder above. Longest and shortest cannot be: duration is computed
+    // from startedAt and finishedAt, and for a run still going it depends on the
+    // current time, so there is no stored field to order by.
+    //
+    // This therefore orders the twenty rows on screen, not the collection. It is
+    // the longest of this page, which is not the same as the longest run.
     if (sort !== 'longest' && sort !== 'shortest') {
       return rows
     }