|
|
@@ -1,411 +1,348 @@
|
|
|
# Single-Node Re-Execution - Design
|
|
|
|
|
|
-Date: 2026-08-12
|
|
|
-
|
|
|
-## Goal
|
|
|
+Date: 2026-08-12 (revised same day)
|
|
|
+
|
|
|
+## What changed since the first version, and why
|
|
|
+
|
|
|
+The first version of this document proposed building single-node
|
|
|
+re-execution on top of `POST /api/v1/nodes/:type/options` - the inline,
|
|
|
+unpersisted one-node workflow the editor uses to populate a node's dropdown
|
|
|
+options - as the closest existing scaffold, and designed a new `rerun`
|
|
|
+endpoint, a new `sideEffects` node field, and a new persisted `input` field
|
|
|
+around it.
|
|
|
+
|
|
|
+That premise was wrong. The editor already has single-node re-execution,
|
|
|
+built and shipped, not proposed:
|
|
|
+
|
|
|
+- `workflowsApi.executeToNode` (`webui/src/api/workflows.ts:252`) sends
|
|
|
+ `_targetNodeId` in the trigger data to the normal `execute` endpoint.
|
|
|
+- `WorkflowEditorPage.tsx`'s `executeNode` callback (starts line 1070, calls
|
|
|
+ `executeToNode` at line 1127) builds a `cachedOutputs` map of
|
|
|
+ `nodeId -> { output, configHash }` for every upstream node and sends it
|
|
|
+ alongside the target.
|
|
|
+- `WorkflowEngine::execute` (`src/runner/workflow_engine.cpp:350-353`) reads
|
|
|
+ `_targetNodeId` and `_cachedOutputs` off the real trigger data (no inline
|
|
|
+ workflow, no separate endpoint - this is an ordinary execution of the real
|
|
|
+ workflow, just told where to stop and what it can skip).
|
|
|
+- The normal graph walk (`workflow_engine.cpp:788-825`) and the loop-body
|
|
|
+ walk (`workflow_engine.cpp:2660-2677`, `2963-2967`) both check
|
|
|
+ `cached_outputs` for every upstream node: if a node's current config
|
|
|
+ still hashes to what was cached, its stored output is substituted and the
|
|
|
+ node is **not executed at all** - no network call, no side effect, no
|
|
|
+ wait. Only nodes whose config changed, and the target node itself, run
|
|
|
+ for real.
|
|
|
+
|
|
|
+So the feature this document set out to design - "re-run one node with the
|
|
|
+current draft config, without waiting for the whole slow chain again" - is
|
|
|
+mostly built already. Revising this document rather than starting a new one
|
|
|
+because the investigation that follows (what `cachedOutputs` does and does
|
|
|
+not cover) is what actually determines what's left to build, and that
|
|
|
+investigation supersedes essentially every conclusion in the first version:
|
|
|
+the recommended endpoint, the recommended persisted `input` field, and the
|
|
|
+"what already exists" section built around `nodes/:type/options` are all
|
|
|
+withdrawn below. What survives from the first version - the scratch-result
|
|
|
+placement, the no-history-mutation stance, the excluded downstream-rerun
|
|
|
+scope - is repeated here with the same reasoning, not rewritten for its own
|
|
|
+sake.
|
|
|
+
|
|
|
+The `sideEffects` node-module field is being implemented right now by
|
|
|
+another agent, in parallel with this revision. It is written up below as
|
|
|
+something arriving, not something this design still needs to propose.
|
|
|
+
|
|
|
+## Goal (unchanged)
|
|
|
|
|
|
Let somebody re-run one node of a finished execution - in place, in the
|
|
|
editor, with the current draft config - without re-running the whole
|
|
|
workflow. The motivating incident: a refactor broke a pipeline in three
|
|
|
places, each hidden behind the last, and each fix was verified by re-running
|
|
|
the entire workflow (minutes of image generation and a published blog post)
|
|
|
-to check what amounted to a one-line expression change. n8n calls this
|
|
|
-"execute node" or "pin data + re-run"; the ask here is the same shape.
|
|
|
-
|
|
|
-Six questions have no obvious answer and are treated in order: where the
|
|
|
-input comes from, which iteration, where the result goes, what to do about
|
|
|
-side effects, what "the fixed node" means, and what ships first.
|
|
|
+to check what amounted to a one-line expression change.
|
|
|
|
|
|
## What already exists
|
|
|
|
|
|
-Checked against the code before this design was written.
|
|
|
-
|
|
|
-### The stored execution record has no input
|
|
|
-
|
|
|
-`NodeExecutionResult` (`src/runner/workflow_engine.hpp:73-107`) does carry an
|
|
|
-`input` field, and `executeNode` populates it at
|
|
|
-`src/runner/workflow_engine.cpp:1412` (`result.input = input;`). But
|
|
|
-`ExecutionResult::toJson()` - the function that turns the in-memory result
|
|
|
-into the document written to the `executions` collection - drops it on the
|
|
|
-floor. `workflow_engine.cpp:271-298`:
|
|
|
-
|
|
|
-```cpp
|
|
|
-nr["nodeId"] = result.node_id;
|
|
|
-nr["status"] = nodeStatusToString(result.status);
|
|
|
-nr["startedAt"] = result.started_at;
|
|
|
-nr["finishedAt"] = result.finished_at;
|
|
|
-// Note: input is intentionally not stored to avoid data duplication
|
|
|
-// Each node's input can be reconstructed from upstream node outputs + connections
|
|
|
-nr["output"] = keep_full_outputs ? result.output : truncateLargeValues(result.output);
|
|
|
-nr["error"] = result.error;
|
|
|
-nr["retryCount"] = result.retry_count;
|
|
|
-if (result.loop_node_id) nr["loopNodeId"] = *result.loop_node_id;
|
|
|
-if (result.loop_iteration) nr["loopIteration"] = *result.loop_iteration;
|
|
|
-```
|
|
|
-
|
|
|
-`webui/src/api/workflows.ts:462-473` mirrors this exactly - the TypeScript
|
|
|
-`NodeExecution` type declares `input?: any` but the backend never sends it, so
|
|
|
-the field is always `undefined` in the editor today. This is a deliberate,
|
|
|
-documented decision in both languages, not an oversight: storing input was
|
|
|
-judged to duplicate the upstream node's output.
|
|
|
-
|
|
|
-That decision is precisely what blocks replay. The comment's premise -
|
|
|
-"reconstructed from upstream node outputs + connections" - is available for
|
|
|
-the *first* node the user wants to re-run, since the immediate upstream
|
|
|
-node's `output` **is** stored. It stops being available the moment
|
|
|
-reconstruction requires re-running expression evaluation, merging multiple
|
|
|
-upstream branches, or replaying loop-item slicing, none of which is
|
|
|
-persisted anywhere. Section 1 below covers what to do about this.
|
|
|
-
|
|
|
-### `loopNodeId` / `loopIteration` already identify one iteration
|
|
|
-
|
|
|
-Confirmed at `workflow_engine.cpp:292-297`. Absence of both fields means "not
|
|
|
-in a loop" - it is not a sentinel value
|
|
|
-(`workflow_engine.hpp:84-93`). Inside a loop body, node ids carry an
|
|
|
-`_iter_<n>` suffix in the in-memory `node_results` map
|
|
|
-(`workflow_engine.cpp:242-244`), and a plain-id "mirror" copy is filtered out
|
|
|
-of the persisted `nodeExecutions` array via `is_loop_mirror`
|
|
|
-(`workflow_engine.cpp:258-260`) so it never double-counts as an execution.
|
|
|
-The array is sorted by `(started_at, seq)` (`workflow_engine.cpp:246-269`),
|
|
|
-so multiple iterations of the same node appear as distinct, ordered entries,
|
|
|
-each carrying its own `loopIteration`. This is exactly what the ask needs:
|
|
|
-"this node, as it ran in iteration 4" is already an addressable record via
|
|
|
-`(nodeId, loopNodeId, loopIteration)`.
|
|
|
-
|
|
|
-### There is a working pattern for running one node in isolation - but not with real input
|
|
|
-
|
|
|
-`POST /api/v1/nodes/:type/options` (`src/webserver/api/node_controller.cpp:26-127`)
|
|
|
-already executes a single node outside of any saved workflow. It builds an
|
|
|
-inline, unpersisted one-node workflow document in-process
|
|
|
-(`node_controller.cpp:63-74`):
|
|
|
-
|
|
|
-```cpp
|
|
|
-nlohmann::json node;
|
|
|
-node["id"] = "options";
|
|
|
-node["type"] = node_type;
|
|
|
-node["name"] = "Options";
|
|
|
-node["config"] = config;
|
|
|
-node["position"] = {{"x", 0}, {"y", 0}};
|
|
|
-
|
|
|
-nlohmann::json inline_workflow;
|
|
|
-inline_workflow["name"] = "options:" + node_type;
|
|
|
-inline_workflow["nodes"] = nlohmann::json::array({node});
|
|
|
-inline_workflow["connections"] = nlohmann::json::array();
|
|
|
-inline_workflow["settings"] = nlohmann::json::object();
|
|
|
-```
|
|
|
-
|
|
|
-sends it via gRPC as `ExecuteWorkflowRequest.inline_workflow` with
|
|
|
-`wait_for_completion = true`, and the runner
|
|
|
-(`src/runner/runner_service.cpp:45-88`) parses it, stamps the real
|
|
|
-`workflow_id` onto it so credential/storage scoping still resolves, and runs
|
|
|
-it through the ordinary `engine_.execute(...)` - the same path a real
|
|
|
-workflow run takes. The inline workflow is never persisted
|
|
|
-(`runner_service.cpp:50-52`). The webserver then pulls the single
|
|
|
-`nodeExecutions` entry with `nodeId == "options"` back out of the returned
|
|
|
-`ExecutionResult` and hands its `output` to the caller
|
|
|
-(`node_controller.cpp:106-118`).
|
|
|
-
|
|
|
-The gap: this endpoint hardcodes `input = {}`
|
|
|
-(`node_controller.cpp:61-62`, "It runs with empty input, which is all a
|
|
|
-listing needs") and there is no request field for caller-supplied input. It
|
|
|
-solves "run this node type with this config" but not "run this node with
|
|
|
-this config against this input."
|
|
|
-
|
|
|
-One level down, `executeNode`
|
|
|
-(`src/runner/workflow_engine.hpp:319-323`, body from
|
|
|
-`workflow_engine.cpp:1405`) is the function both the normal graph walk
|
|
|
-(`workflow_engine.cpp:873`) and the loop-body walk
|
|
|
-(`workflow_engine.cpp:3089`, `executeLoopBody`) call per node. It takes an
|
|
|
-already-resolved `input` verbatim (`ctx.input = input;` at
|
|
|
-`workflow_engine.cpp:1509`) and does not touch `ExecutionResult::node_results`
|
|
|
-or any prior-node-output map - all input resolution happens in the caller,
|
|
|
-before `executeNode` is invoked. It needs a `Workflow` object only for `id`
|
|
|
-(storage/credential namespacing) and `settings` (storage permissions,
|
|
|
-retention) - not the real persisted workflow, as the inline-options path
|
|
|
-already proves by handing it a synthetic one-node `Workflow`. In short:
|
|
|
-**`executeNode` has no structural obstacle to being called with an arbitrary
|
|
|
-saved input outside of a full graph walk.** The inline-workflow-over-gRPC
|
|
|
-mechanism is the right scaffold to extend, not a new one to invent.
|
|
|
-
|
|
|
-`GET /api/v1/executions/:id/retry` exists as a route
|
|
|
-(`execution_controller.cpp:445-458`) but is an unfinished stub - it reads the
|
|
|
-old execution, does nothing with it, and always reports fake success. It is
|
|
|
-not a foothold for this feature; treat it as dead code to eventually
|
|
|
-implement or remove, out of scope here.
|
|
|
-
|
|
|
-### The editor holds one draft config, scoped to whichever modal is open
|
|
|
-
|
|
|
-There is no zustand store or persistent per-node draft abstraction. In
|
|
|
-`webui/src/pages/WorkflowEditorPage.tsx`, `editingConfig`
|
|
|
-(line 377, `useState<Record<string, any>>({})`) is seeded from the in-canvas
|
|
|
-ReactFlow node's current data when its config modal opens
|
|
|
-(`openNodeConfig`, lines 1901-1923, specifically `setEditingConfig(node.data.config || {})`
|
|
|
-at line 1918), lives only while that modal is open, and is written back into
|
|
|
-the canvas's `nodes` array on Save (`saveNodeConfig`, lines 2337-2362) - not
|
|
|
-yet to the backend; that is a separate, later "Save workflow" action gated by
|
|
|
-`hasChanges`. There is no ready-made "get current draft config for node X"
|
|
|
-call outside the modal's own lifecycle.
|
|
|
-
|
|
|
-### No node declares side effects
|
|
|
-
|
|
|
-Grepping the repo for `sideEffects` / `SideEffect` returns nothing. The
|
|
|
-parsed node metadata, `NodeDefinition`
|
|
|
-(`src/runner/node_registry.hpp:29-48`), has `category` (a free-text UI
|
|
|
-grouping tag, not a safety signal) and `is_trigger` (means "starts a
|
|
|
-workflow," not "safe to re-run") and nothing else resembling idempotency or
|
|
|
-danger. The module interface documented in `docs/nodes.md:7-12,59-70` is
|
|
|
-exactly `configSchema`, `inputSchema`, `outputSchema`, `execute` - no
|
|
|
-optional fifth field for this today.
|
|
|
-
|
|
|
-### Execution records are replaced whole, never patched
|
|
|
-
|
|
|
-`workflow_engine.cpp:2438-2494`: on persist, the engine does
|
|
|
-`storage_.get("executions", id)` to check existence, then either
|
|
|
-`storage_.upsert(...)` (refresh TTL on a re-paused Waiting execution) or
|
|
|
-`storage_.update("executions", result.execution_id, result.toJson())` -
|
|
|
-always the full document, rebuilt from the complete in-memory `node_results`
|
|
|
-map. There is no code path that patches one entry inside a stored
|
|
|
-`nodeExecutions` array. The one existing precedent for "the record changes
|
|
|
-after its first write" is a paused (`EXECUTION_STATUS_WAITING`) execution
|
|
|
-resuming and running more nodes - and even then the whole document is
|
|
|
-regenerated and replaced, not edited in place. There is no precedent at all
|
|
|
-for rewriting a record to claim something different happened than what was
|
|
|
-originally observed. That absence of precedent is itself an argument (see
|
|
|
-Question 3): grafting a re-run's result into the original record would be a
|
|
|
-new, unprecedented kind of mutation, not an extension of an existing one.
|
|
|
+### `executeToNode` + `cachedOutputs` already skip the upstream chain, for real
|
|
|
+
|
|
|
+`executeNode` in `WorkflowEditorPage.tsx` (line 1070) walks the canvas graph
|
|
|
+backward from the target node (`collectUpstreamNodes`, inline in the same
|
|
|
+function) and, for every upstream node it finds, looks up
|
|
|
+`lastExecutionResults[nodeId] ?? pinnedNodeData[nodeId]` and pairs it with a
|
|
|
+hash of that node's *current* config (`hashConfig`, line 1062:
|
|
|
+`JSON.stringify(config || {})`). `lastExecutionResults` is not scoped to the
|
|
|
+current editor session only - it is seeded on load (lines 849-870) from the
|
|
|
+five most recent stored executions, merging each node's output across them
|
|
|
+so a freshly opened editor still has something to send. This whole map goes
|
|
|
+out as `_cachedOutputs` on `executeToNode` (`workflows.ts:252-264`).
|
|
|
+
|
|
|
+On the runner side, `configMatchesHash` (`workflow_engine.cpp:165`) is the
|
|
|
+other half of the check: for every upstream node in the walk, if
|
|
|
+`cached_outputs` has an entry and its `configHash` matches the node's
|
|
|
+current config, the node's stored output is substituted directly into
|
|
|
+`node_result.output` and `from_cache` is set true (`workflow_engine.cpp:
|
|
|
+788-825`) - the node is never run. The same check, same function, runs
|
|
|
+again for loop-body nodes (`workflow_engine.cpp:2963-2967`). This is not a
|
|
|
+reimplementation of the graph walk in the webserver (which the first
|
|
|
+version's option (b) rightly worried about as duplicated logic) - it *is*
|
|
|
+the graph walk, the same one a full run takes, just fed a cached value
|
|
|
+instead of a freshly computed one at each upstream node. Expression
|
|
|
+evaluation, branch merging, and loop-item slicing for the *target* node all
|
|
|
+still happen through the engine's normal code, because the target node
|
|
|
+itself is deliberately never served from cache
|
|
|
+(`node_id != target_node_id` guard, `workflow_engine.cpp:791`).
|
|
|
+
|
|
|
+Net effect: for the motivating incident's slow chain, re-running the fixed
|
|
|
+node today already skips every unchanged upstream node's real work -
|
|
|
+including the minutes-long image generation and the blog publish - as long
|
|
|
+as those nodes' configs are unchanged, which they are when only the failing
|
|
|
+node was edited. **This is the headline finding**: the expensive thing the
|
|
|
+first version worried about re-running is, in the common case, already not
|
|
|
+re-run.
|
|
|
+
|
|
|
+### `nodes/:type/options` is a different, narrower mechanism - not the scaffold to build on
|
|
|
+
|
|
|
+`POST /api/v1/nodes/:type/options` (`src/webserver/api/node_controller.cpp:
|
|
|
+26-127`) builds an inline, unpersisted one-node workflow and runs it with
|
|
|
+`input = {}` hardcoded. It exists to populate dropdown options for a node
|
|
|
+type in isolation and has no notion of upstream data at all. It is a
|
|
|
+legitimate pattern for "run this node type with this config and no input,"
|
|
|
+but `executeToNode` already solves the harder and more relevant problem -
|
|
|
+"run this node with this config against real upstream data" - by reusing
|
|
|
+the real execution path instead. Nothing here needs cloning.
|
|
|
+
|
|
|
+### `loopNodeId` / `loopIteration` already identify one iteration - in storage, not yet in re-run
|
|
|
+
|
|
|
+Confirmed unchanged from the original investigation: each persisted
|
|
|
+`nodeExecutions` entry carries `loopNodeId` / `loopIteration`
|
|
|
+(`workflow_engine.cpp:292-297` in the prior numbering; the fields are
|
|
|
+written wherever a node result is appended from inside a loop body), absent
|
|
|
+together means "not in a loop," and multiple iterations of the same node
|
|
|
+appear as distinct, ordered entries. The editor already tracks this per row
|
|
|
+- `selectedIterations` (`WorkflowEditorPage.tsx:413`) is state for browsing
|
|
|
+which iteration of a node's output is currently shown, populated from the
|
|
|
+engine-supplied `loopIteration` field (`WorkflowEditorPage.tsx:476-478`).
|
|
|
+So the UI already has "iteration 4 of node X" as an addressable, displayed
|
|
|
+concept. What it does not have is a way to feed that iteration back into a
|
|
|
+re-run.
|
|
|
+
|
|
|
+### The single-node loop path always replays iteration 0, never a chosen iteration
|
|
|
+
|
|
|
+This is the first genuine gap. `executeLoopBody`'s `single_node_mode`
|
|
|
+(`workflow_engine.cpp:2660-2677`) trims the loop body to the target node and
|
|
|
+its in-body upstream, then unconditionally does `ctx.items.resize(1)` -
|
|
|
+whatever the loop's item collection is, only its first element survives.
|
|
|
+There is no parameter anywhere in `_targetNodeId` / `_cachedOutputs` for
|
|
|
+"start at item N" or "use this specific item value." A user looking at
|
|
|
+iteration 4's failed output and re-running the fixed node from the editor
|
|
|
+today would silently get iteration 0's item instead, which for a feed-based
|
|
|
+loop or an image batch is not the item they were debugging.
|
|
|
+
|
|
|
+### `cachedOutputs` is populated from "recent, merged," not "this exact execution"
|
|
|
+
|
|
|
+The second gap. `lastExecutionResults` is seeded by merging outputs across
|
|
|
+the last 5 executions (`WorkflowEditorPage.tsx:820-870`, `mergeOutputs`),
|
|
|
+favoring the newest non-empty value per node - a heuristic tuned for "what
|
|
|
+does this node usually produce," not "what did it produce in the specific
|
|
|
+execution and iteration I am looking at right now." Re-running from a
|
|
|
+specific historical execution's detail view today sends whatever is
|
|
|
+currently cached in editor state, which is usually close but not
|
|
|
+guaranteed to be that execution's actual upstream values (e.g. if a later
|
|
|
+run changed an upstream node's output shape in the meantime, or if the
|
|
|
+execution being inspected is older than the 5 seeded).
|
|
|
+
|
|
|
+### Node inputs are still not persisted - and, revisited, do not need to be
|
|
|
+
|
|
|
+`ExecutionResult::toJson()` still drops the in-memory `NodeExecutionResult
|
|
|
+::input` field when writing to the `executions` collection, with the
|
|
|
+comment "input is intentionally not stored to avoid data duplication...
|
|
|
+reconstructed from upstream node outputs + connections." The first version
|
|
|
+of this document concluded that comment was wrong for anything beyond the
|
|
|
+immediate upstream node, specifically calling out loop-item slicing as a
|
|
|
+case the comment's premise doesn't cover.
|
|
|
+
|
|
|
+Revisiting that with `cachedOutputs` in view: it's wrong the other
|
|
|
+direction. The loop node's own stored output already retains the *full*
|
|
|
+item collection it iterated over, not just the merged result. Nothing in
|
|
|
+`executeLoopBody` strips `_items` or `_isLoop` out of `node_result.output`
|
|
|
+before it is written into `result.node_results[node_id]`
|
|
|
+(`workflow_engine.cpp:926-936`) - only the per-iteration *mirror* copies of
|
|
|
+body nodes are filtered from the persisted array
|
|
|
+(`is_loop_mirror` filtering, unchanged from the original investigation), not
|
|
|
+the loop node's own record. So `_items[4]` - the exact value iteration 4's
|
|
|
+body received - is already sitting in the stored execution document, on the
|
|
|
+loop node's own `output`, today, for every execution that has already run.
|
|
|
+Combined with `cachedOutputs` sourced correctly (see Gap 2 above) and a
|
|
|
+target-iteration parameter (Gap 1), the engine can recompute the *exact*
|
|
|
+input the target node received in that exact iteration, through its own
|
|
|
+real expression-evaluation and merge code - not an approximation, and not a
|
|
|
+second engine. Persisting `input` would add nothing this doesn't already
|
|
|
+give, for the loop case that was the original document's strongest argument
|
|
|
+for persisting it.
|
|
|
+
|
|
|
+For the non-loop case, the immediately upstream node's stored `output` was
|
|
|
+already sufficient (the original document said so too) - and that's exactly
|
|
|
+what `cachedOutputs`, sourced from the specific execution, already exposes.
|
|
|
+
|
|
|
+## The storage question, revisited
|
|
|
+
|
|
|
+The original document's option (a) - add `input` to `toJson()`, symmetric
|
|
|
+with `output` - is withdrawn. Two independent things changed the answer:
|
|
|
+
|
|
|
+1. It isn't needed. See above: `cachedOutputs` plus the loop node's own
|
|
|
+ already-stored `_items` reconstructs the exact input for both the
|
|
|
+ plain-DAG case and the loop-iteration case, using the real engine code
|
|
|
+ path, with no new persisted field.
|
|
|
+2. The cost was understated even on its own terms. Execution documents are
|
|
|
+ already large - the biggest measured so far is 336 KB - and listing them
|
|
|
+ costs roughly 30 ms per document; a page of them can approach or exceed
|
|
|
+ the gRPC message limit. Persisting `input` next to `output` for every
|
|
|
+ node execution would push documents with many nodes or large payloads
|
|
|
+ toward roughly double their current size (input is usually close in size
|
|
|
+ to the producing node's output, since it's often that output verbatim or
|
|
|
+ a shallow transform of it), which worsens exactly the listing-cost and
|
|
|
+ page-size ceiling that retention and TTL work already treats as a
|
|
|
+ constraint on document size.
|
|
|
+
|
|
|
+Recommendation: **do not persist input, for any node, in any form** - not
|
|
|
+for every node, not only for failed nodes, not truncated. The zero-cost
|
|
|
+option is also the correct one here, which is unusual enough to state
|
|
|
+plainly rather than default to the "safe middle" of "only failed nodes" or
|
|
|
+"truncate like output does." If a future need surfaces that
|
|
|
+`cachedOutputs` + stored `_items` genuinely cannot cover - something that
|
|
|
+consumes upstream data through a path this design didn't find - that is a
|
|
|
+new, narrower case to design against with its own concrete size numbers,
|
|
|
+not a reason to persist input broadly now.
|
|
|
|
|
|
## 1. Where does the input come from
|
|
|
|
|
|
-The stored record has no input (see above), so replay cannot simply read a
|
|
|
-field. Three options:
|
|
|
-
|
|
|
-**(a) Start persisting input.** Add `input` back to `toJson()`, symmetric
|
|
|
-with `output`, subject to the same `truncateLargeValues` treatment for large
|
|
|
-payloads. This makes replay exact - literally the JSON the node received,
|
|
|
-including anything upstream expression evaluation, branch merging, or loop
|
|
|
-slicing produced - and it costs nothing at execution time (the value already
|
|
|
-exists in `NodeExecutionResult::input`, per `workflow_engine.cpp:1412`; the
|
|
|
-change is at serialization). The "avoid duplication" comment undersells the
|
|
|
-cost, too: for most nodes, input is a reference to the immediately upstream
|
|
|
-node's output, so the duplication is a second copy of already-small JSON.
|
|
|
-Only nodes with unusually large input (e.g. a node fed a big prior output
|
|
|
-directly, or a loop item that assembled a large object) would see the
|
|
|
-truncation path trigger, exactly as output already does.
|
|
|
-
|
|
|
-**(b) Reconstruct at re-run time from upstream outputs + connections.** This
|
|
|
-is what the existing comment gestures at. It requires re-running the
|
|
|
-workflow's expression evaluator against the *stored* upstream outputs and
|
|
|
-the *saved* connection graph, entirely in the webserver or a new runner
|
|
|
-path, duplicating logic that today only exists inside the graph walk in
|
|
|
-`workflow_engine.cpp`. It breaks down for anything the walk resolved that
|
|
|
-isn't a pure function of upstream `output` values - loop-item slicing for a
|
|
|
-node inside a loop body is the clearest case, since a loop iteration's input
|
|
|
-is a slice of the loop's collection at a particular index, not something
|
|
|
-visible in any single upstream node's stored output as a self-contained
|
|
|
-value.
|
|
|
-
|
|
|
-**(c) Require the caller (the UI) to supply input explicitly**, sourced from
|
|
|
-whatever the `ExecutionResultsPanel` already has loaded (the upstream node's
|
|
|
-stored `output`, or the target node's own last `output` if the user is
|
|
|
-happy replaying "whatever ran last time" without re-deriving it). This is
|
|
|
-what n8n effectively does with "pinned data."
|
|
|
-
|
|
|
-Recommendation: **(a)**. It is the smallest change, it is exact rather than
|
|
|
-approximate, and it turns "which iteration" (Question 2) and "the fixed
|
|
|
-node" (Question 5) into simple record lookups instead of a second expression
|
|
|
-engine. (b) is real work for a weaker guarantee - it re-derives something
|
|
|
-that was already computed once and thrown away. (c) is a fallback worth
|
|
|
-keeping in the design as what the UI falls back to for *executions recorded
|
|
|
-before this change ships* (their records will never have `input`, since
|
|
|
-nothing rewrites history - see Question 3), but should not be the primary
|
|
|
-mechanism for new executions once (a) lands.
|
|
|
-
|
|
|
-Concretely: add `input` to `ExecutionResult::toJson()` at
|
|
|
-`workflow_engine.cpp:271-298` alongside `output`, add the corresponding
|
|
|
-field to `webui/src/api/workflows.ts`'s `NodeExecution` (already declared,
|
|
|
-just unused), and it flows through `GET /api/v1/executions/:id`
|
|
|
-(`execution_controller.cpp:327-343`) unchanged, since that endpoint already
|
|
|
-returns the whole stored document verbatim.
|
|
|
+Answered above: from the engine's normal graph walk, fed `cachedOutputs`
|
|
|
+for every node whose config is unchanged since that output was produced.
|
|
|
+No stored `input` field, no reconstruction logic outside the engine.
|
|
|
|
|
|
## 2. Which iteration
|
|
|
|
|
|
-`(nodeId, loopNodeId, loopIteration)` already uniquely addresses one
|
|
|
-execution record, per the confirmation above. The re-run request should
|
|
|
-carry these three fields (loopNodeId/loopIteration omitted for a node not in
|
|
|
-a loop), the webserver looks up the matching entry in the stored
|
|
|
-`nodeExecutions` array of the given execution, and reads its `input` (once
|
|
|
-Question 1's change ships). The `ExecutionResultsPanel`, which the task
|
|
|
-states is already the surface where a user browses iterations, is the
|
|
|
-natural place to put a "Re-run this iteration" action per visible node
|
|
|
-entry - it already has the exact triple in hand for whatever row is being
|
|
|
-displayed.
|
|
|
-
|
|
|
-## 3. Where does the result go
|
|
|
-
|
|
|
-Not into the stored execution record. Two independent reasons converge here:
|
|
|
-the task's own framing (amending history makes the record untrustworthy,
|
|
|
-and this project had an incident today from exactly that kind of record
|
|
|
-asserting something unobserved), and the codebase's own precedent
|
|
|
-(Section "Execution records are replaced whole, never patched" above) - the
|
|
|
-system has never mutated a record to claim something different happened,
|
|
|
-and the one case that looks like mutation (a resumed Waiting execution)
|
|
|
-actually appends real new observations, it doesn't retcon old ones.
|
|
|
-
|
|
|
-The result of a re-run is a **scratch result**, not a history record. Its
|
|
|
-home and lifetime:
|
|
|
-
|
|
|
-- **Home**: a new, unpersisted response returned directly by the re-run
|
|
|
- endpoint - not written to the `executions` collection at all. This
|
|
|
- mirrors the existing `nodes/:type/options` pattern exactly (inline
|
|
|
- workflow, `wait_for_completion`, read the one `nodeExecutions` entry back
|
|
|
- out of the gRPC response, discard the rest) - nothing there touches
|
|
|
- storage today, and this feature shouldn't start.
|
|
|
-- **Where it's shown**: the editor keeps it as local component state
|
|
|
- attached to the node being re-run (comparable to how `editingConfig` is
|
|
|
- scoped to whichever modal is open) - e.g. a "last re-run result" panel
|
|
|
- next to or replacing the historical output view for that node, visibly
|
|
|
- labeled as a re-run rather than folded into the historical output so a
|
|
|
- user can never mistake it for what actually happened during the real
|
|
|
- execution.
|
|
|
-- **Lifetime**: it lives exactly as long as the editor tab/session does.
|
|
|
- Navigating away, refreshing, or closing the node's panel discards it.
|
|
|
- This is honest about what it is - a verification aid, not a record - and
|
|
|
- it avoids inventing a second persistence tier (a "scratch executions"
|
|
|
- collection, TTL policy, cleanup job) for a feature that exists to answer
|
|
|
- "did my one-line fix work" in the time it takes to click a button. If
|
|
|
- usage shows people want to keep or compare re-run results across
|
|
|
- sessions, that is a deliberate follow-up with its own retention design,
|
|
|
- not a default.
|
|
|
+Two concrete, small changes, not a new endpoint:
|
|
|
+
|
|
|
+- **Scope `cachedOutputs` to the execution being inspected.** When the
|
|
|
+ re-run is triggered from a specific execution's detail view (as opposed
|
|
|
+ to "just try the workflow as currently drafted"), build the map from that
|
|
|
+ execution's own `nodeExecutions` entries - matching `loopNodeId` /
|
|
|
+ `loopIteration` where the target node is inside a loop - rather than the
|
|
|
+ 5-execution merge `executeNode` uses today. The 5-execution merge remains
|
|
|
+ the right default for "no specific execution in mind, just re-verify
|
|
|
+ against whatever's recent."
|
|
|
+- **Carry the target iteration to the runner.** Add a field alongside
|
|
|
+ `_targetNodeId` - e.g. `_targetLoopIteration` - and thread it into
|
|
|
+ `single_node_mode` (`workflow_engine.cpp:2660-2677`) so it selects
|
|
|
+ `ctx.items[iteration]` instead of unconditionally `resize(1)` down to
|
|
|
+ item 0. Falls back to the current "first item" behavior when omitted, so
|
|
|
+ nothing about today's un-iterated re-run changes.
|
|
|
+
|
|
|
+`selectedIterations` (`WorkflowEditorPage.tsx:413`) already holds, per node,
|
|
|
+which iteration the user is currently looking at - it is the natural source
|
|
|
+for this field. This is the one piece that turns "re-run this node" into
|
|
|
+"re-run this node as it ran in iteration 4," which was not expressible
|
|
|
+before today (per-iteration `loopIteration` tagging on execution records is
|
|
|
+newly available) and is genuinely new work, not something already built.
|
|
|
+
|
|
|
+## 3. Where does the result go (unchanged from the first version)
|
|
|
+
|
|
|
+Not into the stored execution record. `WorkflowEditorPage.tsx`'s
|
|
|
+`executeNode` already treats a targeted run's result as transient editor
|
|
|
+state - `executionState` set from WebSocket events keyed off the returned
|
|
|
+`executionId`, discarded on navigation - not merged back into any stored
|
|
|
+document. `ExecutionResult` persistence (`workflow_engine.cpp`'s
|
|
|
+`upsert`/`update` on the `executions` collection) always writes the whole
|
|
|
+document rebuilt from the complete in-memory `node_results` map; there is
|
|
|
+no path that patches one entry inside a stored `nodeExecutions` array, and
|
|
|
+this design should not add one. A single-node re-run through
|
|
|
+`executeToNode` already produces its own real (if narrow) execution record
|
|
|
+- that's fine as an audit trail of "a test run happened," and is exactly
|
|
|
+how it works today - but its outputs should never be spliced into the
|
|
|
+*original* execution being investigated. The UI already respects this by
|
|
|
+keeping the re-run's result under its own `executionId`; the requirement
|
|
|
+going forward is just to keep it visibly labeled as a re-run when shown
|
|
|
+next to the original execution's per-node output, not silently swapped in.
|
|
|
|
|
|
## 4. Side effects
|
|
|
|
|
|
-Three options were named in the task; take them in order.
|
|
|
-
|
|
|
-**Do nothing beyond a warning.** Cheapest, ships immediately, matches n8n's
|
|
|
-own choice. Cost: the warning is generic ("this node may have side
|
|
|
-effects") because nothing today lets a node say whether it has any, so
|
|
|
-either every re-run gets the same boilerplate warning (which people learn to
|
|
|
-click through, defeating the point) or the warning is skipped entirely,
|
|
|
-in which case it isn't really a warning.
|
|
|
-
|
|
|
-**Let a node declare `sideEffects: true`.** Requires: a new optional field
|
|
|
-on the module interface documented in `docs/nodes.md`, parsing it into
|
|
|
-`NodeDefinition` (`node_registry.hpp:29-48`) alongside `is_trigger`, and
|
|
|
-exposing it through `NodeDefinition::toJson()` to the editor. Cost is real
|
|
|
-but small - one field, one parse site, one serialization site - and it's the
|
|
|
-only option that lets the warning be specific ("Publish To Blog will publish
|
|
|
-again") instead of generic, and lets the UI visually distinguish
|
|
|
-side-effect-free nodes (most `set-fields`, `condition`, `code` nodes) from
|
|
|
-ones that aren't, which is exactly the distinction that would have let the
|
|
|
-user in the motivating incident re-run the condition node and the broken
|
|
|
-call node freely while getting a pointed warning only when they got near
|
|
|
-the loop's actual side-effecting step.
|
|
|
-
|
|
|
-**Dry-run.** Would require every node author to implement a second code
|
|
|
-path (`dryRun` alongside `execute`), or the runner to intercept and no-op
|
|
|
-specific known-dangerous operations (HTTP calls, storage writes) generically
|
|
|
-- neither is realistic given nodes are arbitrary JS running in QuickJS with
|
|
|
-direct access to `fetch`/storage/credential APIs (per `docs/nodes.md`).
|
|
|
-This is real work with no existing scaffold, and it silently produces wrong
|
|
|
-verification results for any node whose "dry" path diverges from its real
|
|
|
-path (e.g. a node whose real behavior depends on a live external response).
|
|
|
-Rule out for this feature.
|
|
|
-
|
|
|
-**Recommendation: declare-and-warn** (the second option), not "do nothing."
|
|
|
-The cost is one boolean field end to end - genuinely small next to the other
|
|
|
-two - and it directly serves the stated goal: letting someone re-run a
|
|
|
-condition or a call node with confidence while being stopped short of
|
|
|
-blindly re-running `Publish To Blog`. Node authors who don't set the field
|
|
|
-default to "unknown," which the UI should render as a mild, generic caution
|
|
|
-(not silence, not a blocking confirmation) - so the field is additive and
|
|
|
-nothing regresses for the 80-plus nodes that exist today until each is
|
|
|
-reviewed and tagged. Tagging existing nodes is a follow-up, not part of this
|
|
|
-design; it does not block ship.
|
|
|
-
|
|
|
-## 5. What counts as "the fixed node"
|
|
|
-
|
|
|
-Explicit design decision: **re-run uses the current editor draft config
|
|
|
-(`editingConfig`, whatever is in the modal right now, unsaved or not) with
|
|
|
-the recorded input from Question 1/2.** Not the config the workflow last
|
|
|
-saved, not the config from the original execution - the live draft, because
|
|
|
-the entire point is verifying an edit before committing it.
|
|
|
-
|
|
|
-Consequence worth stating plainly, because it's easy to get wrong: **the
|
|
|
-re-run does not touch the node's own input.** If the edit being tested would
|
|
|
-itself change what upstream nodes produce - because the node under test is
|
|
|
-also, say, a condition whose branch choice determines what an earlier node
|
|
|
-in the graph would have received - a single-node re-run cannot see that.
|
|
|
-Input is fixed at "what this node received last time"; only the node's own
|
|
|
-processing of that fixed input is being re-verified. This is exactly right
|
|
|
-for the motivating incident's second and third bugs (a stale-path
|
|
|
-expression, a reference to a deleted node) - both are properties of one
|
|
|
-node's own config against its recorded input - and it is exactly wrong for
|
|
|
-any question of the shape "would earlier nodes behave differently." That
|
|
|
-second question needs "re-run this node and everything downstream" (or
|
|
|
-upstream), which is out of scope - see Question 6.
|
|
|
+`sideEffects` is being implemented on the node module interface by another
|
|
|
+agent, in parallel with this document. Once it lands, the re-run action
|
|
|
+already present in the editor (`executeNode` / the "run to this node"
|
|
|
+affordance) should read it for the target node and any node whose cache
|
|
|
+entry was rejected (config changed, so it will actually execute) and show a
|
|
|
+specific warning - "this will really call X again" - rather than a generic
|
|
|
+one. Nodes that don't declare it render the existing mild, generic caution.
|
|
|
+No further design needed here; this section exists so a reader of this
|
|
|
+document knows the flag is inbound rather than still open.
|
|
|
+
|
|
|
+## 5. What counts as "the fixed node" (unchanged from the first version)
|
|
|
+
|
|
|
+Re-run uses the current editor draft config with the input the node
|
|
|
+received (now: reconstructed live through the engine and `cachedOutputs`,
|
|
|
+per iteration where relevant - not a stored field). Not the last-saved
|
|
|
+config, not the original execution's config - the live draft, because the
|
|
|
+point is verifying an edit before committing it. And, as before: the re-run
|
|
|
+does not touch the node's own input. If the edit under test would itself
|
|
|
+change what upstream nodes produce, a single-node re-run can't see that -
|
|
|
+out of scope, unchanged from the first version's Question 5.
|
|
|
|
|
|
## 6. Scope
|
|
|
|
|
|
**Ship:**
|
|
|
-- Persist `input` in `NodeExecutionResult`'s serialized form
|
|
|
- (`workflow_engine.cpp:271-298`), threaded through to the TS type already
|
|
|
- declaring it unused.
|
|
|
-- A new endpoint, e.g. `POST /api/v1/executions/:id/nodes/:nodeId/rerun`,
|
|
|
- accepting `{ config, loopNodeId?, loopIteration? }`, looking up the
|
|
|
- matching stored `nodeExecutions` entry for its `input`, and reusing the
|
|
|
- inline-workflow-over-gRPC mechanism from `node_controller.cpp:26-127` -
|
|
|
- same pattern, but with the looked-up `input` instead of hardcoded `{}`,
|
|
|
- and the caller-supplied `config` instead of whatever the saved workflow
|
|
|
- document has. Never writes to the `executions` collection.
|
|
|
-- Editor: a "Re-run this node" action wherever a node's result is already
|
|
|
- shown per-iteration (`ExecutionResultsPanel`), using the currently open
|
|
|
- `editingConfig` when the node's modal is open, falling back to the
|
|
|
- canvas's last-saved config otherwise. Result renders as a clearly-labeled
|
|
|
- scratch panel, kept only for the editor session (Question 3).
|
|
|
-- The `sideEffects` boolean on the node module interface and a generic
|
|
|
- warning in the re-run UI when it's `true` or unset (Question 4). Tagging
|
|
|
- existing nodes with it is explicitly follow-up work, not blocking.
|
|
|
-
|
|
|
-**Deliberately excluded:**
|
|
|
-- Re-running a node plus everything downstream of it. This is a different
|
|
|
- and larger feature: it needs a resumable partial graph walk seeded with
|
|
|
- one real result and continuing through the normal engine, decisions about
|
|
|
- whether *that* continuation writes a real execution record or another
|
|
|
- scratch result, and its own side-effect story multiplied across however
|
|
|
- many downstream nodes are side-effecting. It shares scaffolding with this
|
|
|
- design (the same `input`-persistence change, the same inline-execution
|
|
|
- mechanism generalized to more than one node) but is not needed to solve
|
|
|
- the motivating problem, where each of the three bugs was diagnosable and
|
|
|
- fixable one node at a time.
|
|
|
-- Tagging the 80-plus existing nodes with `sideEffects`.
|
|
|
-- Dry-run of any kind.
|
|
|
-- A "keep this scratch result" / compare-across-sessions feature.
|
|
|
-- Fixing or implementing `POST /api/v1/executions/:id/retry`
|
|
|
+- Scope `cachedOutputs` to the specific execution being inspected when a
|
|
|
+ re-run is launched from that execution's detail view, instead of always
|
|
|
+ using the 5-execution merge (`WorkflowEditorPage.tsx:820-870`,
|
|
|
+ `executeNode` at line 1070). The 5-execution merge stays as the default
|
|
|
+ for a re-run with no specific execution context.
|
|
|
+- Add a target-iteration field (e.g. `_targetLoopIteration`) alongside
|
|
|
+ `_targetNodeId` / `_cachedOutputs` in the trigger data
|
|
|
+ (`workflow_engine.cpp:350` on), and use it in `executeLoopBody`'s
|
|
|
+ `single_node_mode` (`workflow_engine.cpp:2660-2677`) to select the
|
|
|
+ intended item instead of always item 0. Source it from
|
|
|
+ `selectedIterations` (`WorkflowEditorPage.tsx:413`) in the editor.
|
|
|
+- Wire the `sideEffects` flag (arriving from other in-progress work) into
|
|
|
+ the existing re-run affordance's warning, once it lands.
|
|
|
+
|
|
|
+**Deliberately not doing:**
|
|
|
+- Persisting `input` on `NodeExecutionResult` / `ExecutionResult::toJson()`,
|
|
|
+ in any form - full, failed-only, or truncated. See the storage section
|
|
|
+ above.
|
|
|
+- A new `rerun` endpoint or any reuse of `nodes/:type/options` - the
|
|
|
+ existing `execute` + `_targetNodeId` + `_cachedOutputs` path already
|
|
|
+ covers this; there is nothing to add a parallel mechanism for.
|
|
|
+- Re-running a node plus everything downstream of it - a different, larger
|
|
|
+ feature (needs a resumable partial graph walk continuing through the
|
|
|
+ normal engine, and its own decision on whether that continuation writes
|
|
|
+ a real record or a scratch one). Not needed for the motivating incident,
|
|
|
+ where each bug was diagnosable one node at a time.
|
|
|
+- Tagging existing nodes with `sideEffects` - the flag's existence is
|
|
|
+ in-flight work; tagging the ~80 existing nodes with it is a follow-up.
|
|
|
+- Mutating or patching a stored execution record in any way.
|
|
|
+- Fixing or implementing `GET /api/v1/executions/:id/retry`
|
|
|
(`execution_controller.cpp:445-458`) - unrelated stub, whole-execution
|
|
|
- scope, left as-is.
|
|
|
+ scope, unchanged from the first version.
|
|
|
|
|
|
## Summary of file:line touch points for implementation
|
|
|
|
|
|
-- `src/runner/workflow_engine.hpp:73-107` - `NodeExecutionResult` (input
|
|
|
- already present in memory).
|
|
|
-- `src/runner/workflow_engine.cpp:271-298` - `toJson()`, add `input` next to
|
|
|
- `output`.
|
|
|
-- `src/runner/workflow_engine.cpp:319-323`, `1405` on - `executeNode`,
|
|
|
- callable as-is with a saved input.
|
|
|
-- `src/runner/runner_service.cpp:45-88` - inline-workflow gRPC handling to
|
|
|
- extend/reuse.
|
|
|
-- `src/webserver/api/node_controller.cpp:26-127` - the pattern to clone for
|
|
|
- the new rerun endpoint.
|
|
|
-- `src/webserver/api/execution_controller.cpp:327-343` - existing detail
|
|
|
- fetch, unchanged, now carries `input` once Question 1 ships.
|
|
|
-- `src/runner/node_registry.hpp:29-48` - `NodeDefinition`, add
|
|
|
- `sideEffects`.
|
|
|
-- `docs/nodes.md:7-12,59-70` - module interface doc, add the optional
|
|
|
- `sideEffects` field.
|
|
|
-- `webui/src/api/workflows.ts:462-473` - `NodeExecution` type, `input`
|
|
|
- field already declared, wire it up.
|
|
|
-- `webui/src/pages/WorkflowEditorPage.tsx:377,1901-1923,2337-2362` -
|
|
|
- `editingConfig`, source of "the fixed node"'s config for a re-run.
|
|
|
-- `webui/src/components/workflow/ExecutionResultsPanel.tsx` - where the
|
|
|
- re-run action and its scratch result should surface, since it already
|
|
|
- displays per-iteration node results.
|
|
|
+- `webui/src/pages/WorkflowEditorPage.tsx:820-870` - seeding of
|
|
|
+ `lastExecutionResults`; needs an execution-scoped variant for Question 2.
|
|
|
+- `webui/src/pages/WorkflowEditorPage.tsx:1070-1144` - `executeNode`,
|
|
|
+ where `cachedOutputs` is built and `executeToNode` is called; add
|
|
|
+ execution-scoped cache sourcing and the target-iteration field.
|
|
|
+- `webui/src/pages/WorkflowEditorPage.tsx:413`, `476-478` -
|
|
|
+ `selectedIterations`, source for the target-iteration field.
|
|
|
+- `webui/src/api/workflows.ts:252-264` - `executeToNode`; add an optional
|
|
|
+ target-iteration parameter alongside `_targetNodeId` / `_cachedOutputs`.
|
|
|
+- `src/runner/workflow_engine.cpp:350-353` on - `execute()`'s trigger-data
|
|
|
+ parsing; add the target-iteration field next to `_targetNodeId` /
|
|
|
+ `_cachedOutputs`.
|
|
|
+- `src/runner/workflow_engine.cpp:2660-2677` - `executeLoopBody`'s
|
|
|
+ `single_node_mode`; replace unconditional `ctx.items.resize(1)` with
|
|
|
+ selection by target iteration when supplied.
|
|
|
+- `src/runner/workflow_engine.cpp:788-825`, `2963-2967` -
|
|
|
+ `configMatchesHash`-gated cache substitution, top-level and loop-body;
|
|
|
+ unchanged, this is the mechanism being extended, not rebuilt.
|
|
|
+- `src/runner/workflow_engine.cpp:926-936` - loop node's own stored
|
|
|
+ `output`, confirmed to retain `_items` untouched; the reason persisting
|
|
|
+ `input` is not needed for the loop case.
|
|
|
+- Node module `sideEffects` field and its surfacing in the re-run UI -
|
|
|
+ tracked as arriving from other in-progress work, not part of this scope.
|