#3 feat: webhooks

When sending a new website to scrape (initially adding), allow to specify a webhook which is called at every automated action

• Implement a /webhok endpoint where webhooks can be managed per site (replace, delete, disable)

Webhooks always must be called except when not added, or disabled.

webhook must POST the event which is triggered with corresponding informations (for example a content changed on a store's page)

@fszontagh I'm starting work on this issue now!

I'll implement the webhook feature with the following approach:

1. Add webhook field to the site model (store webhook URL, enabled status)
2. Create /webhook endpoint for managing webhooks per site (add/replace, delete, disable/enable)
3. Integrate webhook POST calls into the automated scraping actions
4. Ensure webhooks are called with appropriate event data (e.g., content changes, scraping status)

Let me explore the codebase first to understand the current architecture.

Codebase Analysis Complete

I've finished exploring the codebase. Here's what I found:

Current Stack:

• TypeScript/Node.js with Express.js
• SQLite database (better-sqlite3)
• Job queue system with event emitters
• Automated scheduling with node-cron

Implementation Plan:

1. Database Changes

• Add webhooks table to store webhook configurations per site
• Fields: id, shop_id, url, enabled, secret (for verification), created_at

2. New Service Layer

• Create src/webhooks/WebhookManager.ts to handle webhook delivery
• Use axios for POST requests to webhook URLs
• Implement retry logic and error handling

3. API Endpoints (as specified in issue)

• POST /api/shops/:id/webhooks - Add/replace webhook
• DELETE /api/shops/:id/webhooks - Delete webhook
• PATCH /api/shops/:id/webhooks - Enable/disable webhook
• GET /api/shops/:id/webhooks - Get webhook status

4. Event Integration Points

I'll trigger webhooks at these events:

• scrape_started - When job begins processing
• scrape_completed - When scraping finishes successfully
• scrape_failed - When scraping fails with error
• content_changed - When content hash differs (most important!)
• schedule_created - When next scrape is scheduled

5. Webhook Payload Format

{
  "event": "content_changed",
  "timestamp": "2025-11-14T10:30:00Z",
  "shop_id": "uuid",
  "shop_url": "https://example.com",
  "data": {
    "content_type": "shipping",
    "url": "https://example.com/shipping",
    "changed": true,
    "old_hash": "abc123",
    "new_hash": "def456"
  }
}

Starting implementation now...

✅ Implementation Complete!

I've successfully implemented the webhook feature as requested. Here's what has been delivered:

🗄️ Database Changes

• Added webhooks table to store webhook configurations per shop
• Added webhook_deliveries table to track delivery attempts and status
• Extended ShopDatabase class with webhook management methods

🔧 Core Service

Created WebhookManager service (src/webhooks/WebhookManager.ts) with:

• Webhook delivery with automatic retry logic (3 attempts with exponential backoff)
• HMAC-SHA256 signature support for webhook security verification
• Delivery logging for monitoring and debugging
• Helper methods for each event type

🌐 API Endpoints

Implemented complete webhook management API as specified:

• POST /api/shops/:id/webhooks - Create or replace webhook for a site
• GET /api/shops/:id/webhooks - Get webhook configuration and recent deliveries
• PATCH /api/shops/:id/webhooks - Enable/disable webhook
• DELETE /api/shops/:id/webhooks - Delete webhook

📡 Event Integration

Webhooks are triggered automatically for all these events:

1. scrape_started - When a scraping job begins processing
2. scrape_completed - When scraping finishes successfully (includes metrics)
3. scrape_failed - When scraping fails with error details
4. content_changed - When content hash differs from previous scrape (most important!)
5. schedule_created - When next automated scrape is scheduled

📦 Webhook Payload Format

{
  "event": "content_changed",
  "timestamp": "2025-11-14T10:30:00Z",
  "shop_id": "uuid",
  "shop_url": "https://example.com",
  "data": {
    "content_type": "shipping",
    "url": "https://example.com/shipping",
    "old_hash": "abc123",
    "new_hash": "def456"
  }
}

🔒 Security

• Optional webhook secret for HMAC-SHA256 signature verification
• Signature sent in X-Webhook-Signature header as sha256=<hash>
• Receivers can verify authenticity using WebhookManager.verifySignature()

✅ Status

• ✅ Code compiles without errors
• ✅ All changes committed to git
• ✅ Pushed to remote repository
• ✅ Ready for testing

The webhook system is now fully operational and will POST events to configured webhook URLs as specified in the issue requirements!