API Documentation

All endpoints (except /health) require Bearer authentication using the Authorization header:

Authorization: Bearer <your-api-key>

Health Check
Jobs
Shops

Health Check

Check the health status of the application.

Endpoint: GET /health

Authentication: Not required

Response:

{
  "status": "ok",
  "timestamp": "2025-01-01T12:00:00.000Z",
  "queue_stats": {
    "total": 10,
    "pending": 2,
    "running": 1,
    "completed": 6,
    "failed": 1
  }
}

Jobs

Create Job

Create a new scraping job for a webshop.

Endpoint: POST /api/jobs

Request Body:

{
  "url": "https://example-shop.com"
}

Response: 201 Created

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "shop_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "pending",
  "message": "Job created successfully"
}

Error Responses:

400 Bad Request - Missing or invalid URL
500 Internal Server Error - Server error

Example:

curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer your-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'

Get Job Status

Get the status and result of a specific job.

Endpoint: GET /api/jobs/:id

URL Parameters:

id - Job UUID

Response:

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "created_at": "2025-01-01T12:00:00.000Z",
  "updated_at": "2025-01-01T12:05:00.000Z",
  "result": {
    "initial_sitemap": "https://example-shop.com/sitemap.xml",
    "shipping_informations": [
      {
        "url": "https://example-shop.com/shipping",
        "content": "# Shipping Information\n\n..."
      }
    ],
    "contacts": [
      {
        "url": "https://example-shop.com/contact",
        "content": "# Contact Us\n\n..."
      }
    ],
    "terms_of_conditions": [
      {
        "url": "https://example-shop.com/terms",
        "content": "# Terms and Conditions\n\n..."
      }
    ],
    "faq": [
      {
        "url": "https://example-shop.com/faq",
        "content": "# FAQ\n\n..."
      }
    ]
  }
}

Status Values:

pending - Job is waiting in queue
running - Job is currently being processed
completed - Job finished successfully
failed - Job failed (includes error field)

Error Responses:

404 Not Found - Job not found
500 Internal Server Error - Server error

Example:

curl http://localhost:3000/api/jobs/a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
  -H "Authorization: Bearer your-secret-key"

List All Jobs

Get a list of all jobs with queue statistics.

Endpoint: GET /api/jobs

Response:

{
  "stats": {
    "total": 10,
    "pending": 2,
    "running": 1,
    "completed": 6,
    "failed": 1
  },
  "jobs": [
    {
      "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "status": "completed",
      "sitemap_url": "https://example-shop.com",
      "created_at": "2025-01-01T12:00:00.000Z",
      "updated_at": "2025-01-01T12:05:00.000Z"
    }
  ]
}

Example:

curl http://localhost:3000/api/jobs \
  -H "Authorization: Bearer your-secret-key"

Shops

List All Shops

Get a list of all shops with their analytics.

Endpoint: GET /api/shops

Response:

{
  "shops": [
    {
      "id": "550e8400-e29b-41d4-a716-446655440000",
      "url": "https://example-shop.com",
      "sitemap_url": "https://example-shop.com/sitemap.xml",
      "webshop_type": "shopify",
      "created_at": "2025-01-01T10:00:00.000Z",
      "updated_at": "2025-01-01T12:00:00.000Z",
      "analytics": {
        "total_scrapes": 5,
        "last_scraped_at": "2025-01-01T12:00:00.000Z",
        "next_scrape_at": "2025-01-02T12:00:00.000Z",
        "total_urls_found": 42,
        "average_scrape_time": 5432.5,
        "average_page_scrape_time": 234.8
      }
    }
  ]
}

Error Responses:

503 Service Unavailable - Database not available
500 Internal Server Error - Server error

Example:

curl http://localhost:3000/api/shops \
  -H "Authorization: Bearer your-secret-key"

Get Shop Details

Get detailed information about a specific shop including analytics, content metadata, scrape history, and scheduled jobs.

Endpoint: GET /api/shops/:id

URL Parameters:

id - Shop UUID

Response:

{
  "shop": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "url": "https://example-shop.com",
    "sitemap_url": "https://example-shop.com/sitemap.xml",
    "webshop_type": "shopify",
    "created_at": "2025-01-01T10:00:00.000Z",
    "updated_at": "2025-01-01T12:00:00.000Z"
  },
  "analytics": {
    "total_scrapes": 5,
    "last_scraped_at": "2025-01-01T12:00:00.000Z",
    "next_scrape_at": "2025-01-02T12:00:00.000Z",
    "total_urls_found": 42,
    "average_scrape_time": 5432.5,
    "average_page_scrape_time": 234.8
  },
  "content_metadata": {
    "shipping_informations": [
      {
        "url": "https://example-shop.com/shipping",
        "changed": false,
        "last_updated": "2025-01-01T12:00:00.000Z"
      }
    ],
    "contacts": [
      {
        "url": "https://example-shop.com/contact",
        "changed": true,
        "last_updated": "2025-01-01T12:00:00.000Z"
      }
    ],
    "terms_of_conditions": [],
    "faq": []
  },
  "scrape_history": [
    {
      "id": "history-uuid-1",
      "job_id": "job-uuid-1",
      "started_at": "2025-01-01T12:00:00.000Z",
      "completed_at": "2025-01-01T12:05:00.000Z",
      "status": "completed",
      "error": null,
      "scrape_time_ms": 5432,
      "urls_found": 42
    }
  ],
  "scheduled_jobs": [
    {
      "id": "schedule-uuid-1",
      "next_run_at": "2025-01-02T12:00:00.000Z",
      "frequency": "daily",
      "last_modified": "2025-01-01T10:00:00.000Z",
      "status": "pending",
      "enabled": true,
      "created_at": "2025-01-01T10:00:00.000Z"
    }
  ]
}

Notes:

This endpoint returns content metadata only (URLs, change status, last updated)
To get full content, use the /api/shops/:id/results endpoint
changed flag indicates if content has changed since the previous scrape

Error Responses:

404 Not Found - Shop not found
503 Service Unavailable - Database not available
500 Internal Server Error - Server error

Example:

curl http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer your-secret-key"

Get Shop Results

Get shop scraping results with full content. Supports filtering by date range, content type, and limit.

Endpoint: GET /api/shops/:id/results

URL Parameters:

id - Shop UUID

Query Parameters:

limit (optional) - Maximum number of results to return
date_from (optional) - Filter results from this date (ISO 8601 format)
date_to (optional) - Filter results to this date (ISO 8601 format)
content_type (optional) - Filter by content type: shipping, contacts, terms, or faq

Response:

{
  "shop_id": "550e8400-e29b-41d4-a716-446655440000",
  "filters": {
    "limit": 10,
    "date_from": "2025-01-01",
    "date_to": null,
    "content_type": null
  },
  "results": {
    "shipping_informations": [
      {
        "url": "https://example-shop.com/shipping",
        "content": "# Shipping Information\n\nDetailed shipping content...",
        "changed": false,
        "last_updated": "2025-01-01T12:00:00.000Z"
      },
      {
        "url": "https://example-shop.com/delivery-info",
        "content": "# Delivery Information\n\nDetailed delivery content...",
        "changed": true,
        "last_updated": "2025-01-01T12:00:00.000Z"
      }
    ],
    "contacts": [
      {
        "url": "https://example-shop.com/contact",
        "content": "# Contact Us\n\nEmail: contact@example.com\nPhone: +1234567890",
        "changed": true,
        "last_updated": "2025-01-01T12:00:00.000Z"
      }
    ],
    "terms_of_conditions": [
      {
        "url": "https://example-shop.com/terms",
        "content": "# Terms and Conditions\n\nDetailed terms...",
        "changed": false,
        "last_updated": "2025-01-01T12:00:00.000Z"
      }
    ],
    "faq": [
      {
        "url": "https://example-shop.com/faq",
        "content": "# Frequently Asked Questions\n\nQ: Question 1?\nA: Answer 1...",
        "changed": false,
        "last_updated": "2025-01-01T12:00:00.000Z"
      }
    ]
  }
}

Notes:

All content categories use arrays to support multiple pages per category
changed flag indicates if content has changed since the previous scrape
Content is in Markdown format, converted from HTML

Error Responses:

404 Not Found - Shop not found
503 Service Unavailable - Database not available
500 Internal Server Error - Server error

Examples:

Get all results:

curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results" \
  -H "Authorization: Bearer your-secret-key"

Get limited results:

curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?limit=10" \
  -H "Authorization: Bearer your-secret-key"

Get results from a specific date:

curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?date_from=2025-01-01" \
  -H "Authorization: Bearer your-secret-key"

Get only shipping information:

curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?content_type=shipping" \
  -H "Authorization: Bearer your-secret-key"

Get results with date range and limit:

curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?date_from=2025-01-01&date_to=2025-01-31&limit=50" \
  -H "Authorization: Bearer your-secret-key"

Enable/Disable Schedule

Enable or disable scheduled scraping for a shop.

Endpoint: PATCH /api/shops/:id/schedule

URL Parameters:

id - Shop UUID

Request Body:

{
  "enabled": true
}

Response:

{
  "shop_id": "550e8400-e29b-41d4-a716-446655440000",
  "schedule_enabled": true,
  "message": "Schedule enabled successfully"
}

Error Responses:

400 Bad Request - Missing or invalid enabled field
404 Not Found - Shop not found
503 Service Unavailable - Database not available
500 Internal Server Error - Server error

Examples:

Enable scheduling:

curl -X PATCH http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/schedule \
  -H "Authorization: Bearer your-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true}'

Disable scheduling:

curl -X PATCH http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/schedule \
  -H "Authorization: Bearer your-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

Delete Shop

Delete a shop and all related data (analytics, scrape history, content, scheduled jobs).

Endpoint: DELETE /api/shops/:id

URL Parameters:

id - Shop UUID

Response:

{
  "message": "Shop and all related data deleted successfully",
  "shop_id": "550e8400-e29b-41d4-a716-446655440000"
}

Notes:

This operation is irreversible
Deletes all associated data:
- Shop record
- Analytics
- Scrape history
- All content records
- Scheduled jobs

Error Responses:

404 Not Found - Shop not found
503 Service Unavailable - Database not available
500 Internal Server Error - Server error

Example:

curl -X DELETE http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer your-secret-key"

Error Handling

All endpoints return standard HTTP status codes:

200 OK - Request successful
201 Created - Resource created successfully
400 Bad Request - Invalid request parameters
404 Not Found - Resource not found
500 Internal Server Error - Server error
503 Service Unavailable - Service (e.g., database) unavailable

Error responses include a JSON body with an error field:

{
  "error": "Description of the error"
}

Content Types

The scraper categorizes content into four types:

shipping (shipping_informations) - Shipping, delivery, and logistics information
contacts - Contact information (email, phone, addresses)
terms (terms_of_conditions) - Terms of service, privacy policy, legal information
faq - Frequently asked questions and help pages

Each category can contain multiple pages (returned as arrays).

Scheduled Scraping

The application automatically schedules scraping based on sitemap rules:

Frequency rules from sitemap: always, hourly, daily, weekly, monthly, yearly
Last modified dates from sitemap
Schedules can be enabled/disabled per shop
Scheduler checks for due jobs every minute
After each successful scrape, the next scrape is automatically scheduled

Content Change Detection

The system uses MD5 hashing to detect content changes:

Each content piece is hashed when scraped
Compared with the previous version's hash
changed flag set to true if hashes differ
Change detection works for all content types
Useful for monitoring when shop policies or information are updated

API.md 13 KB

文件歷史 原始文件

API Documentation

Table of Contents

Health Check

Jobs

Create Job

Get Job Status

List All Jobs

Shops

List All Shops

Get Shop Details

Get Shop Results

Enable/Disable Schedule

Delete Shop

Error Handling

Content Types

Scheduled Scraping

Content Change Detection

API.md 13 KB 文件歷史 原始文件

API Documentation

Table of Contents

Health Check

Jobs

Create Job

Get Job Status

List All Jobs

Shops

List All Shops

Get Shop Details

Get Shop Results

Enable/Disable Schedule

Delete Shop

Error Handling

Content Types

Scheduled Scraping

Content Change Detection

API.md 13 KB

文件歷史原始文件