Kaynağa Gözat

fix: stop retrying an Ollama refusal three times

A 403 was retried like a hiccup. The weekly allowance was spent, so each
attempt spent more of an allowance that was already gone, backed off between
them, and reported "failed after 3 attempt(s)" - which reads as something
flaky that might work next time and sends whoever gets the alert looking for
a fault that is not there.

Statuses are now classified by whether trying again could possibly help.
Ollama documents 400, 404, 429, 500 and 502. It does not document 401 or 403,
but the cloud endpoint returns both - 401 with no key, 403 when the key is
accepted and the account is refused, which is what an exhausted allowance
looks like.

  not retried   400 bad request, 401 unauthorized, 403 refused, 404 no model
  retried       429 rate limited, 500, 502, transport failures and timeouts

429 stays retried deliberately: a rate limit is exactly what backing off is
for, unlike an allowance that resets weekly.

The wording follows the same distinction. A refusal reports "ollama-chat was
refused: ..." rather than counting attempts, so the alert says what happened.

Checked against the live account while its quota was still spent: 403, 404 and
401 each fail on the first attempt with no backoff, and a transport failure
still takes three attempts across 2s + 4s of backoff. Node suite 92/0.

Left alone: the ten OpenAI-compatible sibling nodes retry indiscriminately in
the same way, and Retry-After is not honoured on a 429 - the docs do not
mention the header and I have not seen one sent.
fszontagh 1 ay önce
ebeveyn
işleme
ada32ee78a
1 değiştirilmiş dosya ile 46 ekleme ve 3 silme
  1. 46 3
      nodes/ai/ollama-chat.js

+ 46 - 3
nodes/ai/ollama-chat.js

@@ -271,6 +271,18 @@ function buildHeaders(config) {
     return headers;
 }
 
+// Statuses no amount of retrying can get past.
+//
+// 429 is deliberately NOT here: a rate limit is exactly what backing off is
+// for. Neither are 500 and 502 - Ollama's cloud returns those transiently, and
+// spacing the retries out recovers from them.
+const PERMANENT_STATUSES = [
+    400,  // bad request - the same payload will be just as bad next time
+    401,  // unauthorized - no key, or one the server will not accept
+    403,  // refused: no subscription for this model, or the allowance is spent
+    404   // no such model
+];
+
 function callOllama(config, base64, headers) {
     const messages = [];
     if (config.systemPrompt && String(config.systemPrompt).trim().length > 0) {
@@ -317,7 +329,21 @@ function callOllama(config, base64, headers) {
 
     if (response.status < 200 || response.status >= 300) {
         const detail = typeof response.data === 'string' ? response.data : JSON.stringify(response.data);
-        throw new Error('Ollama HTTP ' + response.status + ': ' + detail);
+        const error = new Error('Ollama HTTP ' + response.status + ': ' + detail);
+        // Whether trying again could possibly help.
+        //
+        // Ollama documents 400, 404, 429, 500 and 502; 401 and 403 are not in
+        // the documentation but both come back from the cloud endpoint - 401
+        // with no key, 403 when the key is fine and the account is refused,
+        // which is what an exhausted weekly allowance looks like.
+        //
+        // Nothing in the permanent set changes because we ask again a few
+        // seconds later: the request is malformed, the model does not exist,
+        // the key is wrong, or the plan says no. Retrying those only delays
+        // the report and spends more of whatever ran out.
+        error.permanent = PERMANENT_STATUSES.indexOf(response.status) !== -1;
+        error.status = response.status;
+        throw error;
     }
 
     const body = typeof response.data === 'string' ? JSON.parse(response.data) : response.data;
@@ -363,6 +389,7 @@ module.exports = {
         let parsed = null;
         let lastError = '';
         let used = 0;
+        let permanent = false;
 
         for (let attempt = 1; attempt <= attempts; attempt++) {
             used = attempt;
@@ -376,6 +403,14 @@ module.exports = {
             } catch (err) {
                 lastError = err && err.message ? err.message : String(err);
                 parsed = null;
+                if (err && err.permanent === true) {
+                    // Said once, and said as what it is. Reporting "failed after
+                    // 3 attempts" for a refusal invites the reader to wonder
+                    // what was flaky, when nothing was.
+                    smartbotic.log.warn('ollama-chat: ' + lastError);
+                    permanent = true;
+                    break;
+                }
                 smartbotic.log.warn('ollama-chat: attempt ' + attempt + ' of ' + attempts + ' failed: ' + lastError);
                 if (attempt < attempts) {
                     // Exponential backoff. Ollama's cloud tier returns transient 500s far
@@ -394,10 +429,18 @@ module.exports = {
         const passImage = config.passthroughImage === true;
 
         if (lastError) {
+            // A refusal is reported as a refusal. "Failed after 3 attempts"
+            // reads as something flaky that might work next time, which sends
+            // whoever gets the alert looking for a fault that is not there -
+            // the answer to a spent allowance or a wrong key is not to run it
+            // again.
+            const summary = permanent
+                ? 'ollama-chat was refused: ' + lastError
+                : 'ollama-chat failed after ' + used + ' attempt(s): ' + lastError;
             if (config.skipOnError !== true) {
-                throw new Error('ollama-chat failed after ' + used + ' attempt(s): ' + lastError);
+                throw new Error(summary);
             }
-            smartbotic.log.warn('ollama-chat: skipping after ' + used + ' attempt(s)');
+            smartbotic.log.warn('ollama-chat: skipping - ' + summary);
             return {
                 success: false,
                 error: lastError,