فهرست منبع

fix(runner): re-register with the webserver after it restarts

The runner registered once at startup. The registry lives in webserver
memory, so a webserver restart forgot every runner while the runners
themselves stayed healthy: they kept reconnecting their node-sync stream and
kept heartbeating, so nothing looked wrong from their side. Meanwhile the
webserver had no runners, and every execution failed with "no runners
available" until someone restarted the runners by hand. A scheduled workflow
simply stopped running, logging one error per tick.

The webserver already answers 404 to a heartbeat from a runner it does not
know, but the heartbeat loop only inspected the curl transport result and
ignored the HTTP status, so that signal was thrown away. It now treats a
failed or rejected heartbeat as a lost registration and re-registers, which
closes the gap within one heartbeat interval.

registerWithWebServer reports whether it actually succeeded rather than
whether curl managed to send the request, so a rejected registration is no
longer logged as success. A registered flag keeps the logs to one line per
state change instead of one per interval.

Verified by restarting the webserver under a live runner: heartbeat failed at
18:45:43, re-registered at 18:45:53, runner online again with no intervention.
fszontagh 1 ماه پیش
والد
کامیت
e0d32e0561
2فایلهای تغییر یافته به همراه53 افزوده شده و 8 حذف شده
  1. 45 7
      src/runner/runner_service.cpp
  2. 8 1
      src/runner/runner_service.hpp

+ 45 - 7
src/runner/runner_service.cpp

@@ -578,9 +578,9 @@ void RunnerService::start() {
     running_ = true;
 
     // Register with webserver
-    registerWithWebServer();
+    registered_ = registerWithWebServer();
 
-    // Start heartbeat
+    // Start heartbeat, which also re-registers whenever the webserver forgets us
     heartbeat_thread_ = std::thread(&RunnerService::heartbeatLoop, this);
 
     LOG_INFO("Runner service {} started", config_.runner_id);
@@ -619,7 +619,7 @@ void RunnerService::stop() {
     LOG_INFO("Runner service stopped");
 }
 
-void RunnerService::registerWithWebServer() {
+bool RunnerService::registerWithWebServer() {
     // Use HTTP to register with webserver
     nlohmann::json body;
     body["id"] = config_.runner_id;
@@ -640,7 +640,7 @@ void RunnerService::registerWithWebServer() {
     CURL* curl = curl_easy_init();
     if (!curl) {
         LOG_WARN("Failed to initialize curl for registration");
-        return;
+        return false;
     }
 
     std::string response_data;
@@ -664,14 +664,26 @@ void RunnerService::registerWithWebServer() {
 
     CURLcode res = curl_easy_perform(curl);
 
+    long http_code = 0;
     if (res == CURLE_OK) {
+        curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code);
+    }
+
+    // A transport-level success is not enough: the webserver can still reject the
+    // registration, and reporting that as success would hide a runner that is not
+    // actually reachable for work.
+    const bool ok = (res == CURLE_OK && http_code >= 200 && http_code < 300);
+    if (ok) {
         LOG_INFO("Runner registered with webserver");
-    } else {
+    } else if (res != CURLE_OK) {
         LOG_WARN("Failed to register with webserver: {}", curl_easy_strerror(res));
+    } else {
+        LOG_WARN("Webserver rejected registration: HTTP {}", http_code);
     }
 
     curl_slist_free_all(headers);
     curl_easy_cleanup(curl);
+    return ok;
 }
 
 void RunnerService::heartbeatLoop() {
@@ -728,12 +740,38 @@ void RunnerService::heartbeatLoop() {
 
         CURLcode res = curl_easy_perform(curl);
 
-        if (res != CURLE_OK) {
-            LOG_WARN("Heartbeat failed: {}", curl_easy_strerror(res));
+        long http_code = 0;
+        if (res == CURLE_OK) {
+            curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code);
         }
 
         curl_slist_free_all(headers);
         curl_easy_cleanup(curl);
+
+        if (res == CURLE_OK && http_code >= 200 && http_code < 300) {
+            if (!registered_) {
+                LOG_INFO("Reconnected to webserver");
+                registered_ = true;
+            }
+            continue;
+        }
+
+        // The registry lives in webserver memory, so a webserver restart forgets
+        // this runner while the runner itself stays healthy. It answers 404 for an
+        // unknown runner; without re-registering here the runner would stay
+        // invisible and every execution would fail with "no runners available".
+        if (registered_) {
+            if (res != CURLE_OK) {
+                LOG_WARN("Heartbeat failed: {}", curl_easy_strerror(res));
+            } else {
+                LOG_WARN("Heartbeat rejected: HTTP {}", http_code);
+            }
+            registered_ = false;
+        }
+
+        if (registerWithWebServer()) {
+            registered_ = true;
+        }
     }
 }
 

+ 8 - 1
src/runner/runner_service.hpp

@@ -105,7 +105,9 @@ public:
     NodeRegistry& registry() { return *registry_; }
 
 private:
-    void registerWithWebServer();
+    // Returns false when the webserver could not be reached or rejected the
+    // registration, so callers can retry.
+    bool registerWithWebServer();
     void heartbeatLoop();
     void unregisterFromWebServer();
     RunnerMetrics collectMetrics();
@@ -122,6 +124,11 @@ private:
     std::thread heartbeat_thread_;
     std::atomic<bool> running_{false};
 
+    // Whether the webserver currently knows about this runner. Drives
+    // re-registration from the heartbeat loop and keeps the logs to one line per
+    // state change rather than one per interval.
+    std::atomic<bool> registered_{false};
+
     // Shutdown signaling
     std::mutex shutdown_mutex_;
     std::condition_variable shutdown_cv_;