smartbotic

SmartBotic

A powerful workflow automation and execution platform built with a modern microservices architecture.

Overview

SmartBotic is a distributed system for creating, managing, and executing complex workflows. It features a high-performance C++20 backend with gRPC communication, an in-memory database with persistence, and a modern React/TypeScript frontend for workflow design.

Key Features

• Microservices Architecture: Three independent services (Database, WebServer, Runner) that scale independently
• Visual Workflow Editor: React-based UI with drag-and-drop workflow design using ReactFlow
• JavaScript Node System: Extensible workflow nodes written in JavaScript, executed in QuickJS
• Real-time Updates: WebSocket support for live workflow execution monitoring
• Distributed Execution: Multiple runner instances for parallel workflow execution
• Persistent Storage: In-memory database with Write-Ahead Log (WAL) and snapshot support
• Hot Reload: Node changes are automatically propagated to runners without restart
• JWT Authentication: Secure API access with token-based authentication
• RESTful API: Comprehensive HTTP API for all operations

Architecture

SmartBotic consists of three microservices:

┌─────────────────┐
│   WebUI         │  React/TypeScript Frontend
│   (Port 3000)   │  Workflow Editor, Dashboard
└────────┬────────┘
         │ HTTP/WS
         ↓
┌─────────────────┐
│   WebServer     │  HTTP REST API, WebSocket
│   (Port 8090)   │  JWT Auth, Runner Registry
└────┬───────┬────┘
     │       │ gRPC
     ↓       ↓
┌─────────┐ ┌──────────┐
│Database │ │ Runner(s)│  Workflow Execution
│(Pt 9004)│ │(Pt 9011) │  QuickJS Engine
│ external│ └──────────┘
└─────────┘

Services

1. Database (external - not built from this repo)

   • The standalone smartbotic-database daemon, developed at ssh://git@git.smartbotics.ai:10022/fszontagh/smartbotic-database.git
   • Installed from the SmartBotics APT repository as smartbotic-database; this repo links against its libsmartbotic-db-client-dev package
   • Document store with gRPC API, WAL durability, snapshots, and encryption
   • Configured independently at /etc/smartbotic-database/config.json

2. WebServer Service (src/webserver/)

   • HTTP REST API for workflows, nodes, and executions
   • WebSocket endpoint for real-time updates
   • JWT-based authentication
   • Runner registry and load balancing
   • Static file serving for the frontend

3. Runner Service (src/runner/)

   • Executes workflows using QuickJS JavaScript engine
   • Supports hot-reload of node definitions
   • Concurrent execution with configurable limits
   • Context injection for node execution (credentials, logging, etc.)

Prerequisites

C++ Backend

• Compiler: GCC 11+ or Clang 14+ (C++20 support required)
• CMake: 3.20 or higher
• Dependencies:
  • gRPC and Protocol Buffers
  • OpenSSL
  • spdlog (logging)
  • nlohmann/json (JSON parsing)
  • cpp-httplib (HTTP server)
  • QuickJS (JavaScript engine)
  • bcrypt (password hashing)
  • libcurl

Frontend

• Node.js: 18.x or higher
• npm: 9.x or higher

Building

Backend (C++)

# Create build directory
mkdir build && cd build

# Debug build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make -j$(nproc)

# Release build (optimized)
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j$(nproc)

# With Address Sanitizer (for debugging memory issues)
cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_ASAN=ON ..
make -j$(nproc)

Binaries will be created in the build/ directory:

• smartbotic-webserver
• smartbotic-runner

Frontend (WebUI)

cd webui

# Install dependencies
npm install

# Development server (port 3000, proxies API to localhost:8090)
npm run dev

# Production build
npm run build

# Lint
npm run lint

The production build outputs to webui/dist/ and is served by the WebServer service.

Running

Services must be started in order due to dependencies:

Manual Start

# 1. Start the external database daemon (must be first)
sudo systemctl start smartbotic-database

# 2. Start WebServer (depends on Database)
./build/smartbotic-webserver

# 3. Start Runner(s) (depends on Database and WebServer)
./build/smartbotic-runner

Using systemd

# Start all services
systemctl --user start smartbotic.target

# Check status
systemctl --user status smartbotic-*

# View logs
journalctl --user -u smartbotic-* -f

# Stop all services
systemctl --user stop smartbotic.target

Unit files are provided in the systemd/ directory.

Default Ports

• WebUI Dev Server: 3000
• WebServer HTTP API: 8090
• WebServer gRPC: 9002
• Database gRPC: 9004 (external daemon)
• Runner gRPC: 9011

Default Credentials

• Username: admin
• Password: admin

⚠️ Change these credentials in production!

Configuration

Services load JSON configuration files from the config/ directory:

• webserver.json - HTTP port, JWT settings, CORS, runner load balancing, database_address
• runner.json - Runner ID, max concurrent executions, node hot reload, database_address

The database daemon is configured separately at /etc/smartbotic-database/config.json, which is owned by the upstream smartbotic-database package.

Configuration supports environment variable substitution using ${VAR_NAME:default} syntax.

Database connection

Both service configs expose the same two database settings, so the host, port and namespace are configurable everywhere without editing a file:

Key	Default	Env override	Purpose
database_address	localhost:9004	DATABASE_ADDRESS	Host and port of the smartbotic-database daemon
database_project	smartbotic-automation	DATABASE_PROJECT	Multi-tenant namespace within that daemon

A single smartbotic-database instance is designed to serve multiple projects. database_project selects which namespace this deployment owns - the client sends every collection as <project>:<collection>, so two projects sharing one daemon never see each other's collections. Leave it at smartbotic-automation unless you are running more than one independent SmartBotic deployment against the same database, in which case give each one a distinct value.

Do not set database_project to default: that is the shared back-compat namespace, where unrelated projects would collide.

Example: config/webserver.json

{
  "http_port": 8090,
  "node_sync_port": 9012,
  "credential_service_port": 9013,
  "static_files_path": "${WEBUI_PATH:./webui/dist}",
  "database_address": "${DATABASE_ADDRESS:localhost:9004}",
  "database_project": "${DATABASE_PROJECT:smartbotic-automation}",
  "auth": {
    "jwt_secret": "${JWT_SECRET:dev-secret-change-in-production}",
    "access_token_lifetime_sec": 900,
    "refresh_token_lifetime_sec": 604800
  }
}

Development

Workflow Nodes

Workflow nodes are JavaScript modules located in the nodes/ directory. Each node defines:

• Configuration schema (static settings)
• Input schema (data from previous nodes)
• Output schema (data passed to next nodes)
• Execute function (async JavaScript code)

See docs/nodes.md for a complete guide on creating nodes.

See docs/node-roadmap.md for nodes worth building next.

Example Node

// nodes/my-node.js
module.exports = {
  configSchema: {
    type: "object",
    properties: {
      message: { type: "string", default: "Hello" }
    }
  },
  inputSchema: {
    type: "object",
    properties: {
      name: { type: "string" }
    }
  },
  outputSchema: {
    type: "object",
    properties: {
      greeting: { type: "string" }
    }
  },
  execute: async (config, input, context) => {
    return {
      greeting: `${config.message}, ${input.name}!`
    };
  }
};

Migrating Nodes to Database

After creating nodes in nodes/, migrate them to the running database:

# 1. Get JWT token
TOKEN=$(curl -s http://localhost:8090/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "admin"}' | jq -r '.accessToken')

# 2. Migrate nodes
curl -X POST http://localhost:8090/api/v1/nodes/migrate \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"nodesPath": "./nodes"}'

# 3. Verify
curl http://localhost:8090/api/v1/nodes \
  -H "Authorization: Bearer $TOKEN"

Runners automatically receive updates via gRPC streaming - no restart required!

API Examples

# Login
curl -X POST http://localhost:8090/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "admin"}'

# List workflows
curl http://localhost:8090/api/v1/workflows \
  -H "Authorization: Bearer $TOKEN"

# Create workflow
curl -X POST http://localhost:8090/api/v1/workflows \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My Workflow",
    "description": "Test workflow",
    "definition": {...}
  }'

# Execute workflow
curl -X POST http://localhost:8090/api/v1/workflows/{id}/execute \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"input": {}}'

WebSocket API

Connect to ws://localhost:8090/api/v1/ws?token={JWT_TOKEN} for real-time updates:

const ws = new WebSocket(`ws://localhost:8090/api/v1/ws?token=${token}`);

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  console.log('Update:', data);
};

Project Structure

smartbotic/
├── build/              # Build output directory
├── cmake/              # CMake modules
├── config/             # Runtime configuration files
├── docs/               # Documentation
│   └── nodes.md        # Node development guide
├── lib/                # Shared C++ libraries
│   ├── common/         # Utilities (UUID, time, string)
│   ├── config/         # Configuration loader
│   ├── credentials/    # Credential management
│   ├── crypto/         # Encryption (AES-GCM)
│   ├── logging/        # Logging wrapper
│   └── storage/        # Database client
├── nodes/              # Workflow node definitions (JS)
├── proto/              # Protocol Buffer definitions
├── src/                # Microservice implementations
│   ├── database/       # Database service
│   ├── runner/         # Runner service
│   └── webserver/      # WebServer service
├── systemd/            # systemd unit files
├── webui/              # React frontend
│   ├── src/
│   │   ├── components/ # React components
│   │   ├── pages/      # Page components
│   │   ├── stores/     # Zustand state management
│   │   └── types/      # TypeScript types
│   └── dist/           # Production build output
├── CLAUDE.md           # AI assistant guidance
└── CMakeLists.txt      # Root CMake configuration

C++ Standards

• C++20 required (strict compliance, no extensions)
• Compiler warnings: -Wall -Wextra -Wpedantic
• Modern practices: RAII, smart pointers, move semantics
• Error handling: Exceptions for exceptional cases, error codes for expected failures

Frontend Stack

• React 18.2 with TypeScript
• State Management: Zustand + React Query
• UI: TailwindCSS + Lucide icons
• Workflow Editor: ReactFlow
• Build Tool: Vite
• Code Quality: ESLint + TypeScript strict mode

Testing

# C++ (if tests are configured)
cd build
ctest

# Frontend
cd webui
npm run lint

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Follow the existing code style and architecture
4. Ensure all services build successfully
5. Test your changes thoroughly
6. Commit your changes (git commit -m 'feat: add amazing feature')
7. Push to the branch (git push origin feature/amazing-feature)
8. Open a Pull Request

Commit Message Convention

Follow conventional commits:

• feat: - New feature
• fix: - Bug fix
• docs: - Documentation changes
• refactor: - Code refactoring
• test: - Adding tests
• chore: - Maintenance tasks

Troubleshooting

Services won't start

• Ensure ports 8090, 9011, 9012, 9013 are available, and that the external database daemon is reachable on 9004 (systemctl status smartbotic-database)
• Start services in order: Database → WebServer → Runner
• Check logs for detailed error messages

Build errors

• Verify C++20 compiler support
• Install all required dependencies
• Clear build directory and rebuild: rm -rf build && mkdir build && cd build && cmake ..

Frontend proxy issues

• Ensure WebServer is running on port 8090
• Check webui/vite.config.ts proxy configuration

Node execution failures

• Verify node definitions have all required fields
• Check QuickJS syntax compatibility (ES2020)
• Review runner logs for JavaScript errors

Performance

• Database: In-memory storage with microsecond latency
• WebServer: Async I/O with cpp-httplib
• Runner: Configurable concurrent execution limit
• Frontend: Code splitting and lazy loading

Security

• JWT-based authentication with configurable expiration
• Password hashing with bcrypt
• Credential encryption with AES-256-GCM
• CORS configuration in WebServer
• Input validation on all API endpoints

⚠️ Production Checklist:

• Change default admin credentials
• Set strong JWT secret
• Configure HTTPS/TLS
• Enable firewall rules
• Regular security updates

License

[License information not specified - please add LICENSE file]

Support

For issues, questions, or contributions, please use the project's issue tracker.

---

Built with ❤️ using C++20, React, and modern cloud-native practices.