|
@@ -0,0 +1,453 @@
|
|
|
|
|
+# SmartBotic
|
|
|
|
|
+
|
|
|
|
|
+A powerful workflow automation and execution platform built with a modern microservices architecture.
|
|
|
|
|
+
|
|
|
|
|
+## Overview
|
|
|
|
|
+
|
|
|
|
|
+SmartBotic is a distributed system for creating, managing, and executing complex workflows. It features a high-performance C++20 backend with gRPC communication, an in-memory database with persistence, and a modern React/TypeScript frontend for workflow design.
|
|
|
|
|
+
|
|
|
|
|
+### Key Features
|
|
|
|
|
+
|
|
|
|
|
+- **Microservices Architecture**: Three independent services (Database, WebServer, Runner) that scale independently
|
|
|
|
|
+- **Visual Workflow Editor**: React-based UI with drag-and-drop workflow design using ReactFlow
|
|
|
|
|
+- **JavaScript Node System**: Extensible workflow nodes written in JavaScript, executed in QuickJS
|
|
|
|
|
+- **Real-time Updates**: WebSocket support for live workflow execution monitoring
|
|
|
|
|
+- **Distributed Execution**: Multiple runner instances for parallel workflow execution
|
|
|
|
|
+- **Persistent Storage**: In-memory database with Write-Ahead Log (WAL) and snapshot support
|
|
|
|
|
+- **Hot Reload**: Node changes are automatically propagated to runners without restart
|
|
|
|
|
+- **JWT Authentication**: Secure API access with token-based authentication
|
|
|
|
|
+- **RESTful API**: Comprehensive HTTP API for all operations
|
|
|
|
|
+
|
|
|
|
|
+## Architecture
|
|
|
|
|
+
|
|
|
|
|
+SmartBotic consists of three microservices:
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+┌─────────────────┐
|
|
|
|
|
+│ WebUI │ React/TypeScript Frontend
|
|
|
|
|
+│ (Port 3000) │ Workflow Editor, Dashboard
|
|
|
|
|
+└────────┬────────┘
|
|
|
|
|
+ │ HTTP/WS
|
|
|
|
|
+ ↓
|
|
|
|
|
+┌─────────────────┐
|
|
|
|
|
+│ WebServer │ HTTP REST API, WebSocket
|
|
|
|
|
+│ (Port 8090) │ JWT Auth, Runner Registry
|
|
|
|
|
+└────┬───────┬────┘
|
|
|
|
|
+ │ │ gRPC
|
|
|
|
|
+ ↓ ↓
|
|
|
|
|
+┌─────────┐ ┌──────────┐
|
|
|
|
|
+│Database │ │ Runner(s)│ Workflow Execution
|
|
|
|
|
+│(Pt 9010)│ │(Pt 9011) │ QuickJS Engine
|
|
|
|
|
+└─────────┘ └──────────┘
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Services
|
|
|
|
|
+
|
|
|
|
|
+1. **Database Service** (`src/database/`)
|
|
|
|
|
+ - In-memory key-value storage with gRPC API
|
|
|
|
|
+ - Write-Ahead Log (WAL) for durability
|
|
|
|
|
+ - Periodic snapshots for recovery
|
|
|
|
|
+ - Prefix-based queries and watch support
|
|
|
|
|
+
|
|
|
|
|
+2. **WebServer Service** (`src/webserver/`)
|
|
|
|
|
+ - HTTP REST API for workflows, nodes, and executions
|
|
|
|
|
+ - WebSocket endpoint for real-time updates
|
|
|
|
|
+ - JWT-based authentication
|
|
|
|
|
+ - Runner registry and load balancing
|
|
|
|
|
+ - Static file serving for the frontend
|
|
|
|
|
+
|
|
|
|
|
+3. **Runner Service** (`src/runner/`)
|
|
|
|
|
+ - Executes workflows using QuickJS JavaScript engine
|
|
|
|
|
+ - Supports hot-reload of node definitions
|
|
|
|
|
+ - Concurrent execution with configurable limits
|
|
|
|
|
+ - Context injection for node execution (credentials, logging, etc.)
|
|
|
|
|
+
|
|
|
|
|
+## Prerequisites
|
|
|
|
|
+
|
|
|
|
|
+### C++ Backend
|
|
|
|
|
+
|
|
|
|
|
+- **Compiler**: GCC 11+ or Clang 14+ (C++20 support required)
|
|
|
|
|
+- **CMake**: 3.20 or higher
|
|
|
|
|
+- **Dependencies**:
|
|
|
|
|
+ - gRPC and Protocol Buffers
|
|
|
|
|
+ - OpenSSL
|
|
|
|
|
+ - spdlog (logging)
|
|
|
|
|
+ - nlohmann/json (JSON parsing)
|
|
|
|
|
+ - cpp-httplib (HTTP server)
|
|
|
|
|
+ - QuickJS (JavaScript engine)
|
|
|
|
|
+ - bcrypt (password hashing)
|
|
|
|
|
+ - libcurl
|
|
|
|
|
+
|
|
|
|
|
+### Frontend
|
|
|
|
|
+
|
|
|
|
|
+- **Node.js**: 18.x or higher
|
|
|
|
|
+- **npm**: 9.x or higher
|
|
|
|
|
+
|
|
|
|
|
+## Building
|
|
|
|
|
+
|
|
|
|
|
+### Backend (C++)
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# Create build directory
|
|
|
|
|
+mkdir build && cd build
|
|
|
|
|
+
|
|
|
|
|
+# Debug build
|
|
|
|
|
+cmake -DCMAKE_BUILD_TYPE=Debug ..
|
|
|
|
|
+make -j$(nproc)
|
|
|
|
|
+
|
|
|
|
|
+# Release build (optimized)
|
|
|
|
|
+cmake -DCMAKE_BUILD_TYPE=Release ..
|
|
|
|
|
+make -j$(nproc)
|
|
|
|
|
+
|
|
|
|
|
+# With Address Sanitizer (for debugging memory issues)
|
|
|
|
|
+cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_ASAN=ON ..
|
|
|
|
|
+make -j$(nproc)
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Binaries will be created in the `build/` directory:
|
|
|
|
|
+- `smartbotic-database`
|
|
|
|
|
+- `smartbotic-webserver`
|
|
|
|
|
+- `smartbotic-runner`
|
|
|
|
|
+
|
|
|
|
|
+### Frontend (WebUI)
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+cd webui
|
|
|
|
|
+
|
|
|
|
|
+# Install dependencies
|
|
|
|
|
+npm install
|
|
|
|
|
+
|
|
|
|
|
+# Development server (port 3000, proxies API to localhost:8090)
|
|
|
|
|
+npm run dev
|
|
|
|
|
+
|
|
|
|
|
+# Production build
|
|
|
|
|
+npm run build
|
|
|
|
|
+
|
|
|
|
|
+# Lint
|
|
|
|
|
+npm run lint
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+The production build outputs to `webui/dist/` and is served by the WebServer service.
|
|
|
|
|
+
|
|
|
|
|
+## Running
|
|
|
|
|
+
|
|
|
|
|
+Services must be started in order due to dependencies:
|
|
|
|
|
+
|
|
|
|
|
+### Manual Start
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# 1. Start Database service (must be first)
|
|
|
|
|
+./build/smartbotic-database
|
|
|
|
|
+
|
|
|
|
|
+# 2. Start WebServer (depends on Database)
|
|
|
|
|
+./build/smartbotic-webserver
|
|
|
|
|
+
|
|
|
|
|
+# 3. Start Runner(s) (depends on Database and WebServer)
|
|
|
|
|
+./build/smartbotic-runner
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Using systemd
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# Start all services
|
|
|
|
|
+systemctl --user start smartbotic.target
|
|
|
|
|
+
|
|
|
|
|
+# Check status
|
|
|
|
|
+systemctl --user status smartbotic-*
|
|
|
|
|
+
|
|
|
|
|
+# View logs
|
|
|
|
|
+journalctl --user -u smartbotic-* -f
|
|
|
|
|
+
|
|
|
|
|
+# Stop all services
|
|
|
|
|
+systemctl --user stop smartbotic.target
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Unit files are provided in the `systemd/` directory.
|
|
|
|
|
+
|
|
|
|
|
+### Default Ports
|
|
|
|
|
+
|
|
|
|
|
+- **WebUI Dev Server**: 3000
|
|
|
|
|
+- **WebServer HTTP API**: 8090
|
|
|
|
|
+- **WebServer gRPC**: 9002
|
|
|
|
|
+- **Database gRPC**: 9010
|
|
|
|
|
+- **Runner gRPC**: 9011
|
|
|
|
|
+
|
|
|
|
|
+### Default Credentials
|
|
|
|
|
+
|
|
|
|
|
+- **Username**: `admin`
|
|
|
|
|
+- **Password**: `admin`
|
|
|
|
|
+
|
|
|
|
|
+⚠️ **Change these credentials in production!**
|
|
|
|
|
+
|
|
|
|
|
+## Configuration
|
|
|
|
|
+
|
|
|
|
|
+Services load JSON configuration files from the `config/` directory:
|
|
|
|
|
+
|
|
|
|
|
+- `database.json` - Storage settings, WAL/snapshot intervals
|
|
|
|
|
+- `webserver.json` - HTTP port, JWT settings, CORS, runner load balancing
|
|
|
|
|
+- `runner.json` - Runner ID, max concurrent executions, node hot reload
|
|
|
|
|
+
|
|
|
|
|
+Configuration supports environment variable substitution using `${VAR_NAME:default}` syntax.
|
|
|
|
|
+
|
|
|
|
|
+### Example: `config/webserver.json`
|
|
|
|
|
+
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "httpPort": 8090,
|
|
|
|
|
+ "grpcPort": 9002,
|
|
|
|
|
+ "databaseAddress": "localhost:9010",
|
|
|
|
|
+ "jwtSecret": "${JWT_SECRET:your-secret-key-change-in-production}",
|
|
|
|
|
+ "jwtExpiration": 3600,
|
|
|
|
|
+ "staticFilesPath": "./webui/dist"
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+## Development
|
|
|
|
|
+
|
|
|
|
|
+### Workflow Nodes
|
|
|
|
|
+
|
|
|
|
|
+Workflow nodes are JavaScript modules located in the `nodes/` directory. Each node defines:
|
|
|
|
|
+- Configuration schema (static settings)
|
|
|
|
|
+- Input schema (data from previous nodes)
|
|
|
|
|
+- Output schema (data passed to next nodes)
|
|
|
|
|
+- Execute function (async JavaScript code)
|
|
|
|
|
+
|
|
|
|
|
+**See [docs/nodes.md](docs/nodes.md) for a complete guide on creating nodes.**
|
|
|
|
|
+
|
|
|
|
|
+#### Example Node
|
|
|
|
|
+
|
|
|
|
|
+```javascript
|
|
|
|
|
+// nodes/my-node.js
|
|
|
|
|
+module.exports = {
|
|
|
|
|
+ configSchema: {
|
|
|
|
|
+ type: "object",
|
|
|
|
|
+ properties: {
|
|
|
|
|
+ message: { type: "string", default: "Hello" }
|
|
|
|
|
+ }
|
|
|
|
|
+ },
|
|
|
|
|
+ inputSchema: {
|
|
|
|
|
+ type: "object",
|
|
|
|
|
+ properties: {
|
|
|
|
|
+ name: { type: "string" }
|
|
|
|
|
+ }
|
|
|
|
|
+ },
|
|
|
|
|
+ outputSchema: {
|
|
|
|
|
+ type: "object",
|
|
|
|
|
+ properties: {
|
|
|
|
|
+ greeting: { type: "string" }
|
|
|
|
|
+ }
|
|
|
|
|
+ },
|
|
|
|
|
+ execute: async (config, input, context) => {
|
|
|
|
|
+ return {
|
|
|
|
|
+ greeting: `${config.message}, ${input.name}!`
|
|
|
|
|
+ };
|
|
|
|
|
+ }
|
|
|
|
|
+};
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Migrating Nodes to Database
|
|
|
|
|
+
|
|
|
|
|
+After creating nodes in `nodes/`, migrate them to the running database:
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# 1. Get JWT token
|
|
|
|
|
+TOKEN=$(curl -s http://localhost:8090/api/v1/auth/login \
|
|
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
|
|
+ -d '{"username": "admin", "password": "admin"}' | jq -r '.accessToken')
|
|
|
|
|
+
|
|
|
|
|
+# 2. Migrate nodes
|
|
|
|
|
+curl -X POST http://localhost:8090/api/v1/nodes/migrate \
|
|
|
|
|
+ -H "Authorization: Bearer $TOKEN" \
|
|
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
|
|
+ -d '{"nodesPath": "./nodes"}'
|
|
|
|
|
+
|
|
|
|
|
+# 3. Verify
|
|
|
|
|
+curl http://localhost:8090/api/v1/nodes \
|
|
|
|
|
+ -H "Authorization: Bearer $TOKEN"
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Runners automatically receive updates via gRPC streaming - no restart required!
|
|
|
|
|
+
|
|
|
|
|
+### API Examples
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# Login
|
|
|
|
|
+curl -X POST http://localhost:8090/api/v1/auth/login \
|
|
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
|
|
+ -d '{"username": "admin", "password": "admin"}'
|
|
|
|
|
+
|
|
|
|
|
+# List workflows
|
|
|
|
|
+curl http://localhost:8090/api/v1/workflows \
|
|
|
|
|
+ -H "Authorization: Bearer $TOKEN"
|
|
|
|
|
+
|
|
|
|
|
+# Create workflow
|
|
|
|
|
+curl -X POST http://localhost:8090/api/v1/workflows \
|
|
|
|
|
+ -H "Authorization: Bearer $TOKEN" \
|
|
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
|
|
+ -d '{
|
|
|
|
|
+ "name": "My Workflow",
|
|
|
|
|
+ "description": "Test workflow",
|
|
|
|
|
+ "definition": {...}
|
|
|
|
|
+ }'
|
|
|
|
|
+
|
|
|
|
|
+# Execute workflow
|
|
|
|
|
+curl -X POST http://localhost:8090/api/v1/workflows/{id}/execute \
|
|
|
|
|
+ -H "Authorization: Bearer $TOKEN" \
|
|
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
|
|
+ -d '{"input": {}}'
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### WebSocket API
|
|
|
|
|
+
|
|
|
|
|
+Connect to `ws://localhost:8090/api/v1/ws?token={JWT_TOKEN}` for real-time updates:
|
|
|
|
|
+
|
|
|
|
|
+```javascript
|
|
|
|
|
+const ws = new WebSocket(`ws://localhost:8090/api/v1/ws?token=${token}`);
|
|
|
|
|
+
|
|
|
|
|
+ws.onmessage = (event) => {
|
|
|
|
|
+ const data = JSON.parse(event.data);
|
|
|
|
|
+ console.log('Update:', data);
|
|
|
|
|
+};
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+## Project Structure
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+smartbotic/
|
|
|
|
|
+├── build/ # Build output directory
|
|
|
|
|
+├── cmake/ # CMake modules
|
|
|
|
|
+├── config/ # Runtime configuration files
|
|
|
|
|
+├── docs/ # Documentation
|
|
|
|
|
+│ └── nodes.md # Node development guide
|
|
|
|
|
+├── lib/ # Shared C++ libraries
|
|
|
|
|
+│ ├── common/ # Utilities (UUID, time, string)
|
|
|
|
|
+│ ├── config/ # Configuration loader
|
|
|
|
|
+│ ├── credentials/ # Credential management
|
|
|
|
|
+│ ├── crypto/ # Encryption (AES-GCM)
|
|
|
|
|
+│ ├── logging/ # Logging wrapper
|
|
|
|
|
+│ └── storage/ # Database client
|
|
|
|
|
+├── nodes/ # Workflow node definitions (JS)
|
|
|
|
|
+├── proto/ # Protocol Buffer definitions
|
|
|
|
|
+├── src/ # Microservice implementations
|
|
|
|
|
+│ ├── database/ # Database service
|
|
|
|
|
+│ ├── runner/ # Runner service
|
|
|
|
|
+│ └── webserver/ # WebServer service
|
|
|
|
|
+├── systemd/ # systemd unit files
|
|
|
|
|
+├── webui/ # React frontend
|
|
|
|
|
+│ ├── src/
|
|
|
|
|
+│ │ ├── components/ # React components
|
|
|
|
|
+│ │ ├── pages/ # Page components
|
|
|
|
|
+│ │ ├── stores/ # Zustand state management
|
|
|
|
|
+│ │ └── types/ # TypeScript types
|
|
|
|
|
+│ └── dist/ # Production build output
|
|
|
|
|
+├── CLAUDE.md # AI assistant guidance
|
|
|
|
|
+└── CMakeLists.txt # Root CMake configuration
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+## C++ Standards
|
|
|
|
|
+
|
|
|
|
|
+- **C++20** required (strict compliance, no extensions)
|
|
|
|
|
+- Compiler warnings: `-Wall -Wextra -Wpedantic`
|
|
|
|
|
+- Modern practices: RAII, smart pointers, move semantics
|
|
|
|
|
+- Error handling: Exceptions for exceptional cases, error codes for expected failures
|
|
|
|
|
+
|
|
|
|
|
+## Frontend Stack
|
|
|
|
|
+
|
|
|
|
|
+- **React** 18.2 with TypeScript
|
|
|
|
|
+- **State Management**: Zustand + React Query
|
|
|
|
|
+- **UI**: TailwindCSS + Lucide icons
|
|
|
|
|
+- **Workflow Editor**: ReactFlow
|
|
|
|
|
+- **Build Tool**: Vite
|
|
|
|
|
+- **Code Quality**: ESLint + TypeScript strict mode
|
|
|
|
|
+
|
|
|
|
|
+## Testing
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# C++ (if tests are configured)
|
|
|
|
|
+cd build
|
|
|
|
|
+ctest
|
|
|
|
|
+
|
|
|
|
|
+# Frontend
|
|
|
|
|
+cd webui
|
|
|
|
|
+npm run lint
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+## Contributing
|
|
|
|
|
+
|
|
|
|
|
+1. Fork the repository
|
|
|
|
|
+2. Create a feature branch (`git checkout -b feature/amazing-feature`)
|
|
|
|
|
+3. Follow the existing code style and architecture
|
|
|
|
|
+4. Ensure all services build successfully
|
|
|
|
|
+5. Test your changes thoroughly
|
|
|
|
|
+6. Commit your changes (`git commit -m 'feat: add amazing feature'`)
|
|
|
|
|
+7. Push to the branch (`git push origin feature/amazing-feature`)
|
|
|
|
|
+8. Open a Pull Request
|
|
|
|
|
+
|
|
|
|
|
+### Commit Message Convention
|
|
|
|
|
+
|
|
|
|
|
+Follow conventional commits:
|
|
|
|
|
+- `feat:` - New feature
|
|
|
|
|
+- `fix:` - Bug fix
|
|
|
|
|
+- `docs:` - Documentation changes
|
|
|
|
|
+- `refactor:` - Code refactoring
|
|
|
|
|
+- `test:` - Adding tests
|
|
|
|
|
+- `chore:` - Maintenance tasks
|
|
|
|
|
+
|
|
|
|
|
+## Troubleshooting
|
|
|
|
|
+
|
|
|
|
|
+### Services won't start
|
|
|
|
|
+
|
|
|
|
|
+- Ensure ports 8090, 9002, 9010, 9011 are available
|
|
|
|
|
+- Start services in order: Database → WebServer → Runner
|
|
|
|
|
+- Check logs for detailed error messages
|
|
|
|
|
+
|
|
|
|
|
+### Build errors
|
|
|
|
|
+
|
|
|
|
|
+- Verify C++20 compiler support
|
|
|
|
|
+- Install all required dependencies
|
|
|
|
|
+- Clear build directory and rebuild: `rm -rf build && mkdir build && cd build && cmake ..`
|
|
|
|
|
+
|
|
|
|
|
+### Frontend proxy issues
|
|
|
|
|
+
|
|
|
|
|
+- Ensure WebServer is running on port 8090
|
|
|
|
|
+- Check `webui/vite.config.ts` proxy configuration
|
|
|
|
|
+
|
|
|
|
|
+### Node execution failures
|
|
|
|
|
+
|
|
|
|
|
+- Verify node definitions have all required fields
|
|
|
|
|
+- Check QuickJS syntax compatibility (ES2020)
|
|
|
|
|
+- Review runner logs for JavaScript errors
|
|
|
|
|
+
|
|
|
|
|
+## Performance
|
|
|
|
|
+
|
|
|
|
|
+- **Database**: In-memory storage with microsecond latency
|
|
|
|
|
+- **WebServer**: Async I/O with cpp-httplib
|
|
|
|
|
+- **Runner**: Configurable concurrent execution limit
|
|
|
|
|
+- **Frontend**: Code splitting and lazy loading
|
|
|
|
|
+
|
|
|
|
|
+## Security
|
|
|
|
|
+
|
|
|
|
|
+- JWT-based authentication with configurable expiration
|
|
|
|
|
+- Password hashing with bcrypt
|
|
|
|
|
+- Credential encryption with AES-256-GCM
|
|
|
|
|
+- CORS configuration in WebServer
|
|
|
|
|
+- Input validation on all API endpoints
|
|
|
|
|
+
|
|
|
|
|
+⚠️ **Production Checklist**:
|
|
|
|
|
+- [ ] Change default admin credentials
|
|
|
|
|
+- [ ] Set strong JWT secret
|
|
|
|
|
+- [ ] Configure HTTPS/TLS
|
|
|
|
|
+- [ ] Enable firewall rules
|
|
|
|
|
+- [ ] Regular security updates
|
|
|
|
|
+
|
|
|
|
|
+## License
|
|
|
|
|
+
|
|
|
|
|
+[License information not specified - please add LICENSE file]
|
|
|
|
|
+
|
|
|
|
|
+## Support
|
|
|
|
|
+
|
|
|
|
|
+For issues, questions, or contributions, please use the project's issue tracker.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+**Built with ❤️ using C++20, React, and modern cloud-native practices.**
|