# Creating Workflow Nodes

This guide explains how to create custom workflow nodes for SmartBotic.

## Node File Structure

Nodes are JavaScript modules located in `nodes/<category>/` directories. Each node file must export:

- `configSchema` - JSON Schema for node configuration
- `inputSchema` - JSON Schema for input data
- `outputSchema` - JSON Schema for output data
- `execute` - Async function that performs the node's action

## Basic Template

```javascript
/**
 * @node my-node-id
 * @name My Node Name
 * @category my-category
 * @version 1.0.0
 * @description What this node does
 * @icon icon-name
 */

const configSchema = {
    type: 'object',
    properties: {
        myOption: {
            type: 'string',
            title: 'My Option',
            description: 'Description of this option',
            default: 'default-value'
        }
    },
    required: ['myOption']
};

const inputSchema = {
    type: 'object',
    properties: {
        data: {
            type: 'any',
            description: 'Input data'
        }
    }
};

const outputSchema = {
    type: 'object',
    properties: {
        result: {
            type: 'string',
            description: 'Output result'
        }
    }
};

module.exports = {
    configSchema,
    inputSchema,
    outputSchema,

    async execute(config, input, context) {
        // Your node logic here
        return {
            result: 'success'
        };
    }
};
```

## JSDoc Metadata Tags

The comment block at the top of the file defines node metadata:

| Tag | Required | Description |
|-----|----------|-------------|
| `@node` | Yes | Unique node identifier (kebab-case) |
| `@name` | Yes | Display name shown in UI |
| `@category` | Yes | Category for grouping (e.g., `http`, `data`, `email`) |
| `@version` | Yes | Semantic version (e.g., `1.0.0`) |
| `@description` | Yes | Brief description of node functionality. Must be a single line - the parser captures only the first line of a multi-line `@description` |
| `@icon` | No | Lucide icon name (e.g., `globe`, `mail`, `database`) |
| `@trigger` | No | Add this tag if the node is a trigger (starts workflows) |

## Configuration Schema

The `configSchema` defines what options users can configure in the node editor.

A top-level property's `default` is applied automatically whenever the
stored config omits that key - both when the workflow is saved and again
before the node executes - so `execute()` can rely on the key being
present even for a workflow saved before the field existed. A default is
resolved through the same expression evaluation as any other stored
config value, so a `default:` containing `{{ }}` will be evaluated as an
expression rather than kept as a literal string. No current node relies on
this; keep it in mind if you're the first to try it.

### Supported Field Types

```javascript
// String field
myString: {
    type: 'string',
    title: 'Label',
    description: 'Help text',
    default: 'default value'
}

// Number field
myNumber: {
    type: 'number',
    title: 'Count',
    default: 10
}

// Boolean field
myBoolean: {
    type: 'boolean',
    title: 'Enable Feature',
    default: false
}

// Dropdown (enum)
myChoice: {
    type: 'string',
    title: 'Select Option',
    enum: ['option1', 'option2', 'option3'],
    default: 'option1'
}

// Object field
myHeaders: {
    type: 'object',
    title: 'Headers',
    additionalProperties: { type: 'string' }
}
```

### Dynamic Options (Credentials)

To create a credential selector:

```javascript
credentialId: {
    type: 'string',
    title: 'Credential',
    description: 'Select authentication credential',
    dynamicOptions: {
        source: 'credentials',
        filter: { type: 'bearer' }  // Filter by credential type
    }
}
```

Available credential types: `bearer`, `api_key`, `basic`, `imap`

## Custom Outputs

Define multiple output ports:

```javascript
const outputs = [
    { name: 'success', displayName: 'Success', type: 'object', color: '#10b981' },
    { name: 'error', displayName: 'Error', type: 'object', color: '#ef4444' }
];

module.exports = {
    configSchema,
    inputSchema,
    outputSchema,
    outputs,
    execute
};
```

## Outputs That Come From Config

A node whose output count depends on how it is configured declares
`dynamicOutputs` beside its static `outputs`:

```javascript
const outputs = [
    { name: 'fallback', displayName: 'Fallback', type: 'any', color: '#6b7280' }
];

const dynamicOutputs = {
    from: 'rules',
    namePrefix: 'case',
    labelFrom: 'label',
    color: '#3b82f6'
};

module.exports = { configSchema, inputSchema, outputSchema, outputs, dynamicOutputs, execute };
```

For each entry in the placed node's `config.rules`, the editor draws a port named
`case0`, `case1` and so on, labelled from that entry's `label` field, followed by
the static outputs. Port names are positional, so an edge survives editing a
rule's label or value. Deleting a rule drops the edges hanging off the port that
went with it.

At run time the node routes with `_activeBranch`, exactly as a fixed-port node
does - the branch name is just computed:

```javascript
return { _activeBranch: 'case' + i, ['case' + i]: data };
```

## Pausing for an Answer

A node pauses its execution by returning a `_pause` marker, the way a branching
node returns `_activeBranch`:

```javascript
return {
    token: token,
    reason: 'Approve the refund',
    expiresAt: Date.now() + 86400000,
    _pause: { token: token, reason: 'Approve the refund', expiresAt: expiresAt }
};
```

The engine stops the walk, stores the execution as `waiting` with everything
computed so far, and returns. Nothing downstream runs. The marker is stripped
from the stored output, so a reader sees the request rather than the mechanism.

Answering it continues the run:

```
POST /api/v1/executions/{id}/resume
{ "token": "...", "approved": true, "data": { "note": "looks fine" } }
```

The paused node's output becomes that payload, and the walk continues from
there. Nodes that already ran are not run again. The workflow is rebuilt from
the snapshot stored with the execution, not from the workflow as it stands now,
because it may have been edited while the approval waited.

`GET /api/v1/executions/pending` lists executions waiting for an answer. It
deliberately omits the token: listing is a weaker permission than approving.

Two limits are deliberate. A pause inside a Loop body cannot be resumed, because
loop iteration state is not part of the stored execution, so a node must refuse
to pause there rather than record something unanswerable. And a webhook cannot
wait for an approval - the HTTP request is still open and its deadline is
35 seconds - so a webhook-triggered workflow that pauses returns a 202 with
`{ "executionId": "...", "status": "waiting" }` rather than blocking for an
answer that has nowhere to arrive on this connection. Answer it the same way
as any other paused execution, with `POST /api/v1/executions/{id}/resume`.

## Answering a Webhook

A webhook-triggered workflow returns the last node's output as JSON by default.
To control the response, return a `_webhookResponse` marker from any node:

```javascript
return {
    _webhookResponse: {
        status: 201,
        headers: { 'Content-Type': 'application/json' },
        body: { id: created.id }
    }
};
```

Any node may set it, not only the last one to run, so adding a node to the end
of a workflow cannot silently change what its API returns. If several set it,
the last one wins and the runner logs that it happened.

## Multiple Inputs

A node that joins two branches names its inputs:

```javascript
const inputs = [
    { name: 'input1', displayName: 'Input 1', type: 'any', required: false },
    { name: 'input2', displayName: 'Input 2', type: 'any', required: false }
];

module.exports = { configSchema, inputSchema, outputSchema, inputs, execute };
```

Each named input becomes its own target handle, and arrives in `execute` under
that name - `input.input1`, `input.input2`. A node that declares no `inputs`
keeps the single `data` handle it has always had.

Note the difference between a node that sets `_activeBranch` and one that does
not: with the marker, exactly one output carries data and everything downstream
of the others is skipped. Without it, every named output the node returns is
live at once, which is how `filter` sends kept and discarded items down two
paths in the same run.

## Conventions These Nodes Follow

Four rules emerged while building the Tier 1 set. New nodes should follow them.

**Fail loudly rather than drop data.** A node that cannot do what was asked
throws, with a message naming what it received. `set-fields` throws when
keep-all mode is handed an array instead of an object; its dotted paths refuse
to overwrite a value they would otherwise destroy; `merge` throws when an item
lacks the key it was told to combine on. The alternative - returning an empty
object, or bucketing everything under the string `undefined` - produces a
workflow that keeps running and quietly produces wrong data.

**Guard a configurable output name against your own reserved keys.** A node that
returns fixed keys alongside a user-named field must reject a name that would
collide, throwing early:

```javascript
if (outputField === 'count' || outputField === 'groups' || (groupBy && outputField === 'key')) {
    throw new Error('Aggregate: outputField cannot be "' + outputField +
        '", which is a reserved output name for this node. Pick another name.');
}
```

`aggregate`, `sort-limit-dedupe` and `datetime` all need this. `template` and
`json` do not, because they return exactly one key and there is nothing to
collide with - do not add the guard where it protects nothing. When a node
builds its reserved-key list dynamically, as `aggregate` does for `key`
(only reserved once `groupBy` is set, since that is the only time it is
written), check every mode the node has, not just the always-on keys.

**Config values arrive already evaluated.** The engine resolves `{{...}}` in
node config before `execute()` runs, and a config string that is exactly one
expression keeps its native type. So a field typed as a string in the schema can
legitimately hold an array at run time, which is why several nodes begin with
`if (Array.isArray(inputField))`. That branch is not dead code. Nodes must not
re-interpolate config values. `template` is the one deliberate exception: its
`templateSource: 'input'` mode reads template text out of the input DATA at run
time, which the engine never walks, so that text still has its `{{...}}`
placeholders intact when `execute()` sees it, and calling
`smartbotic.utils.interpolate` on it is genuine work, not a re-interpolation of
something the engine already resolved.

**Read paths with the shared helper.** Use
`smartbotic.utils.getFieldValue(data, path)` rather than writing a private path
walker. Two older nodes, `if-condition` and `loop`, still carry their own
divergent copies - `loop` silently skips a leading `data.` segment and
`if-condition` does not - and that inconsistency is exactly what the shared
helper exists to stop spreading.

## Available APIs

Inside `execute()`, you have access to the `smartbotic` global object:

### Logging

```javascript
smartbotic.log.debug('Debug message');
smartbotic.log.info('Info message');
smartbotic.log.warn('Warning message');
smartbotic.log.error('Error message');
```

### HTTP Requests

```javascript
const response = smartbotic.http.request({
    method: 'GET',  // GET, POST, PUT, PATCH, DELETE
    url: 'https://api.example.com/data',
    headers: { 'Authorization': 'Bearer token' },
    body: JSON.stringify({ key: 'value' }),
    timeout: 30000,
    followRedirects: true
});

// Response: { status, headers, data, dataBase64 }
```

#### Sending and receiving binary

`body` is a text field. It is read as a UTF-8 string, so any byte sequence that
is not valid UTF-8 is replaced on the way out - an image sent that way uploads
successfully and arrives corrupt. There are two binary-safe paths instead.

For an API that takes a bare binary payload, such as an S3 PUT or the AT
Protocol blob upload, use `bodyBase64`. It is decoded to raw bytes and sent as
the whole body, and it wins over `body` when both are given:

```javascript
const file = smartbotic.storage.downloadFile(fileId);

smartbotic.http.request({
    method: 'POST',
    url: 'https://bsky.social/xrpc/com.atproto.repo.uploadBlob',
    headers: { 'Content-Type': 'image/jpeg' },
    bodyBase64: file.data
});
```

For an API that expects a form upload, use `files` with `formData`, which builds
a multipart body - see `nodes/integration/telegram-send.js`.

On the way back, `data` is also UTF-8 text and cannot carry binary. Read
`dataBase64` for image or file downloads.

One trap worth knowing: `utils.base64Encode` encodes a JavaScript string as
UTF-8, so it is not a byte-preserving round trip for arbitrary bytes you build
in JavaScript with `String.fromCharCode`. Anything at or above 0x80 becomes two
bytes. Binary should come from the file store, where it never passes through a
JavaScript string, rather than being assembled by hand.

### Storage (Database)

```javascript
// Insert document
const result = smartbotic.storage.insert('collection', { key: 'value' });
// Returns: { success, id }

// Get document
const doc = smartbotic.storage.get('collection', 'document-id');
// Returns: { found, document }

// Query documents
const results = smartbotic.storage.query('collection', { field: 'value' });
// Returns: { success, documents }

// Update document
smartbotic.storage.update('collection', 'document-id', { key: 'new-value' });

// Delete document
smartbotic.storage.delete('collection', 'document-id');
```

`http-request` (Store Download) and `imap-extract-attachments` (Store in
Database) both write through `smartbotic.storage.insert` with a TTL. Both
default that TTL field to 24 hours, so downloaded files and extracted
attachments stored by those nodes now expire and are auto-deleted a day
after they're stored, unless the workflow sets the TTL field to `0`
explicitly - `0` means keep forever. Any workflow that was relying on
permanent storage under an unset TTL field needs that `0` set explicitly,
or it will start losing data a day later.

### Filesystem

```javascript
// Read a file
const read = smartbotic.fs.readFile('/tmp/example.txt', 'base64');
// { success, data } - data is base64-encoded when the encoding argument is 'base64'

// Write a file
smartbotic.fs.writeFile('/tmp/example.txt', base64Data, 'base64');

// Check existence
const there = smartbotic.fs.exists('/tmp/example.txt');

// Stat a file
const info = smartbotic.fs.stat('/tmp/example.txt');
// { success, size, mtime, isDirectory } - mtime is milliseconds since the epoch

// Create a directory
smartbotic.fs.mkdir('/tmp/example-dir');

// Delete a file
smartbotic.fs.unlink('/tmp/example.txt');

// List a directory, one level
const listing = smartbotic.fs.readdir('/var/spool/incoming');
// { success: true, entries: [{ name, path, size, modifiedAt, isDirectory }] }
```

The `fs` API applies no path restrictions. Every call takes an arbitrary
absolute path and acts on it with the runner process's own permissions, the
same trust level a `code` node or `process.exec` already has.

### Credentials

```javascript
const auth = smartbotic.credentials.get(credentialId);
if (auth.success) {
    // auth.headerName = 'Authorization' or 'X-API-Key'
    // auth.headerValue = 'Bearer <token>' or '<api-key>'
    headers[auth.headerName] = auth.headerValue;
}
```

### Sending Email

```javascript
const result = smartbotic.smtp.send({
    credentialId: config.credentialId,  // an smtp credential, or the imap one for the same mailbox
    to: ['someone@example.com'],        // a single address or an array
    cc: [],
    bcc: [],
    subject: 'Subject line',
    body: 'Message text',
    html: false,                        // true sends the body as text/html
    replyTo: '',
    from: '',                           // overrides the sender on the credential
    fromName: '',
    attachments: [
        { filename: 'note.txt', mimeType: 'text/plain', contentBase64: '...' }
    ]
});
// result.messageId - the Message-ID the mail went out with
// result.accepted  - how many addresses it was submitted for
```

An SMTP credential holds `host`, `port`, `username`, `password`, `security`
(`starttls`, `ssl` or `none`), and optionally `from_address` and `from_name`.
An IMAP credential is accepted in its place: the same account is reached on the
submission port with STARTTLS, so a mailbox stored for reading mail does not
have to be entered again to send it.

### Utilities

```javascript
// Generate UUID
const id = smartbotic.utils.uuid();

// Sleep (pause execution)
smartbotic.utils.sleep(1000);  // milliseconds, max 300000 (5 min)

// Template interpolation
const result = smartbotic.utils.interpolate('Hello {{name}}', { name: 'World' });

// Base64 encoding/decoding
const encoded = smartbotic.utils.base64Encode('data');
const decoded = smartbotic.utils.base64Decode(encoded);

// SHA256 hash
const hash = smartbotic.utils.sha256('data');  // Returns hex string

// Object utilities
const picked = smartbotic.utils.pick(obj, ['key1', 'key2']);
const omitted = smartbotic.utils.omit(obj, ['unwantedKey']);

// Read a nested value by dotted path
const city = smartbotic.utils.getFieldValue(input, 'data.user.address.city');
const second = smartbotic.utils.getFieldValue(input, 'data.items[1].id');
const howMany = smartbotic.utils.getFieldValue(input, 'data.items.length');
```

Missing paths return `undefined`. Arrays are reached with `key[0]` or a bare
numeric segment, and `.length` works on an array. The path is taken literally,
with no special handling of a leading `data.` segment.

## Important Guidelines

### Indentation

Use **4 spaces** for indentation (consistent with existing nodes).

### Avoid Inline Comments

Do NOT use `//` comments inside schema definitions. The parser may incorrectly interpret them:

```javascript
// BAD - comment may break parsing
const configSchema = {
    type: 'object',
    properties: {
        url: {
            type: 'string',
            default: 'http://localhost:3000'  // This is the default  <- AVOID
        }
    }
};

// GOOD - no inline comments in schema
const configSchema = {
    type: 'object',
    properties: {
        url: {
            type: 'string',
            default: 'http://localhost:3000'
        }
    }
};
```

### String Quotes

Use **single quotes** for string values in schemas. The parser converts them to JSON:

```javascript
// GOOD
title: 'My Title'

// AVOID - embedded quotes can cause issues
description: 'Use "quotes" carefully'  // May break parsing
```

### Error Handling

**IMPORTANT**: Throw errors instead of returning `success: false`. This ensures the workflow fails properly:

```javascript
async execute(config, input, context) {
    // Validate required inputs
    if (!config.requiredField) {
        throw new Error('Required field is missing');
    }

    // Make API call
    const response = smartbotic.http.request({
        method: 'GET',
        url: config.url,
        headers: headers
    });

    // Check response status - throw on error
    if (response.status < 200 || response.status >= 300) {
        const errorMsg = typeof response.data === 'string'
            ? response.data
            : JSON.stringify(response.data);
        throw new Error(`API error (${response.status}): ${errorMsg}`);
    }

    // Return data on success (no try/catch needed for most cases)
    return {
        data: response.data
    };
}
```

**Do NOT** catch errors just to return `{ success: false }` - let them propagate so the workflow fails.

## Migrating Nodes to Database

After creating or modifying node files, migrate them to the running database:

```bash
# Get authentication token
TOKEN=$(curl -s http://localhost:8090/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "admin"}' | jq -r '.accessToken')

# Migrate nodes
curl -X POST http://localhost:8090/api/v1/nodes/migrate \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"nodesPath": "./nodes"}'
```

Runners automatically receive node updates via gRPC streaming.

## Example: HTTP Download Node

```javascript
/**
 * @node download-file
 * @name Download File
 * @category http
 * @version 1.0.0
 * @description Download a file from URL
 * @icon download
 */

const configSchema = {
    type: 'object',
    properties: {
        url: {
            type: 'string',
            title: 'URL',
            description: 'URL to download from'
        },
        timeout: {
            type: 'number',
            title: 'Timeout (ms)',
            default: 30000
        }
    },
    required: ['url']
};

const inputSchema = {
    type: 'object',
    properties: {
        url: { type: 'string', description: 'Override URL from input' }
    }
};

const outputSchema = {
    type: 'object',
    properties: {
        success: { type: 'boolean' },
        data: { type: 'string' },
        contentType: { type: 'string' }
    }
};

module.exports = {
    configSchema,
    inputSchema,
    outputSchema,

    async execute(config, input, context) {
        const url = input.url || config.url;

        smartbotic.log.info(`Downloading from ${url}`);

        try {
            const response = smartbotic.http.request({
                method: 'GET',
                url: url,
                timeout: config.timeout
            });

            if (response.status < 200 || response.status >= 300) {
                throw new Error(`HTTP ${response.status}`);
            }

            return {
                success: true,
                data: response.data,
                contentType: response.headers['content-type'] || ''
            };
        } catch (error) {
            smartbotic.log.error(`Download failed: ${error.message}`);
            return {
                success: false,
                error: error.message
            };
        }
    }
};
```

## Trigger Nodes

Trigger nodes start workflow executions. Add `@trigger` to the JSDoc:

```javascript
/**
 * @node my-trigger
 * @name My Trigger
 * @category triggers
 * @version 1.0.0
 * @description Triggers workflow on event
 * @icon zap
 * @trigger
 */
```

Triggers typically don't have input schemas (they generate initial data).

## Chat nodes for OpenAI-compatible providers

OpenRouter, Together, Groq, DeepSeek, Mistral, xAI, Fireworks, Perplexity,
DeepInfra and OpenAI itself all accept the same chat request and return the
same reply. One implementation covers all of them, in
`scripts/gen-openai-compatible-nodes.py`, which writes one node per provider.
A node cannot require another node's file, so the code is generated rather
than shared at runtime - edit the generator, run it, commit the diff.

Adding a provider is a row in the `PROVIDERS` table: its id, display name,
base URL, a sensible default model, and where its keys come from. Set
`no_model_list` when the provider publishes no `/models` endpoint, and the
model field becomes a plain box rather than an empty dropdown.

Each node registers a credential type of its own, so a key for one provider
is never offered to another - and any of them will still take a plain bearer
credential, because the shape is the same.

The base URL is a setting, so any of these nodes also reaches a proxy, a
gateway or a self-hosted service that speaks the same API.

## Retrying an HTTP request

`smartbotic.http.request` takes retry options, so a node does not need its own
loop and its own sleep:

```javascript
smartbotic.http.request({
  method: 'GET', url: feedUrl, timeout: 30000,
  retries: 2,               // extra attempts, 0 (off) unless asked for
  retryDelayMs: 2000,       // doubles each attempt
  retryMaxDelayMs: 30000,   // and is capped here
  retryOnStatus: [429, 503] // defaults to 408, 425, 429, 500, 502, 503, 504
});
```

Retried: a timeout, a refused connection, a name that would not resolve, and
the statuses above. Not retried: a 404 or a 401 - those are answers, and asking
again does not change them. A `Retry-After` header from the server wins over the
backoff, capped at `retryMaxDelayMs`, because being told to wait and then not
waiting is how a rate limit becomes a ban.

It is off by default on purpose. A POST that timed out may already have been
received at the far end, and repeating it would do the thing twice - so the
caller opts in where repeating is safe.