# API Documentation

All endpoints (except `/health`) require Bearer authentication using the `Authorization` header:

```
Authorization: Bearer <your-api-key>
```

## Table of Contents

- [Health Check](#health-check)
- [Jobs](#jobs)
  - [Create Job](#create-job)
  - [Get Job Status](#get-job-status)
  - [List All Jobs](#list-all-jobs)
- [Shops](#shops)
  - [List All Shops](#list-all-shops)
  - [Get Shop Details](#get-shop-details)
  - [Get Shop Results](#get-shop-results)
  - [Enable/Disable Schedule](#enabledisable-schedule)
  - [Delete Shop](#delete-shop)

---

## Health Check

Check the health status of the application.

**Endpoint:** `GET /health`

**Authentication:** Not required

**Response:**
```json
{
  "status": "ok",
  "timestamp": "2025-01-01T12:00:00.000Z",
  "queue_stats": {
    "total": 10,
    "pending": 2,
    "running": 1,
    "completed": 6,
    "failed": 1
  }
}
```

---

## Jobs

### Create Job

Create a new scraping job for a webshop.

**Endpoint:** `POST /api/jobs`

**Request Body:**
```json
{
  "url": "https://example-shop.com"
}
```

**Response:** `201 Created`
```json
{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "shop_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "pending",
  "message": "Job created successfully"
}
```

**Error Responses:**
- `400 Bad Request` - Missing or invalid URL
- `500 Internal Server Error` - Server error

**Example:**
```bash
curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer your-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'
```

---

### Get Job Status

Get the status and result of a specific job.

**Endpoint:** `GET /api/jobs/:id`

**URL Parameters:**
- `id` - Job UUID

**Response:**
```json
{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "created_at": "2025-01-01T12:00:00.000Z",
  "updated_at": "2025-01-01T12:05:00.000Z",
  "result": {
    "initial_sitemap": "https://example-shop.com/sitemap.xml",
    "shipping_informations": [
      {
        "url": "https://example-shop.com/shipping",
        "content": "# Shipping Information\n\n..."
      }
    ],
    "contacts": [
      {
        "url": "https://example-shop.com/contact",
        "content": "# Contact Us\n\n..."
      }
    ],
    "terms_of_conditions": [
      {
        "url": "https://example-shop.com/terms",
        "content": "# Terms and Conditions\n\n..."
      }
    ],
    "faq": [
      {
        "url": "https://example-shop.com/faq",
        "content": "# FAQ\n\n..."
      }
    ]
  }
}
```

**Status Values:**
- `pending` - Job is waiting in queue
- `running` - Job is currently being processed
- `completed` - Job finished successfully
- `failed` - Job failed (includes `error` field)

**Error Responses:**
- `404 Not Found` - Job not found
- `500 Internal Server Error` - Server error

**Example:**
```bash
curl http://localhost:3000/api/jobs/a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
  -H "Authorization: Bearer your-secret-key"
```

---

### List All Jobs

Get a list of all jobs with queue statistics.

**Endpoint:** `GET /api/jobs`

**Response:**
```json
{
  "stats": {
    "total": 10,
    "pending": 2,
    "running": 1,
    "completed": 6,
    "failed": 1
  },
  "jobs": [
    {
      "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "status": "completed",
      "sitemap_url": "https://example-shop.com",
      "created_at": "2025-01-01T12:00:00.000Z",
      "updated_at": "2025-01-01T12:05:00.000Z"
    }
  ]
}
```

**Example:**
```bash
curl http://localhost:3000/api/jobs \
  -H "Authorization: Bearer your-secret-key"
```

---

## Shops

### List All Shops

Get a list of all shops with their analytics.

**Endpoint:** `GET /api/shops`

**Response:**
```json
{
  "shops": [
    {
      "id": "550e8400-e29b-41d4-a716-446655440000",
      "url": "https://example-shop.com",
      "sitemap_url": "https://example-shop.com/sitemap.xml",
      "webshop_type": "shopify",
      "created_at": "2025-01-01T10:00:00.000Z",
      "updated_at": "2025-01-01T12:00:00.000Z",
      "analytics": {
        "total_scrapes": 5,
        "last_scraped_at": "2025-01-01T12:00:00.000Z",
        "next_scrape_at": "2025-01-02T12:00:00.000Z",
        "total_urls_found": 42,
        "average_scrape_time": 5432.5,
        "average_page_scrape_time": 234.8
      }
    }
  ]
}
```

**Error Responses:**
- `503 Service Unavailable` - Database not available
- `500 Internal Server Error` - Server error

**Example:**
```bash
curl http://localhost:3000/api/shops \
  -H "Authorization: Bearer your-secret-key"
```

---

### Get Shop Details

Get detailed information about a specific shop including analytics, content metadata, scrape history, and scheduled jobs.

**Endpoint:** `GET /api/shops/:id`

**URL Parameters:**
- `id` - Shop UUID

**Response:**
```json
{
  "shop": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "url": "https://example-shop.com",
    "sitemap_url": "https://example-shop.com/sitemap.xml",
    "webshop_type": "shopify",
    "created_at": "2025-01-01T10:00:00.000Z",
    "updated_at": "2025-01-01T12:00:00.000Z"
  },
  "analytics": {
    "total_scrapes": 5,
    "last_scraped_at": "2025-01-01T12:00:00.000Z",
    "next_scrape_at": "2025-01-02T12:00:00.000Z",
    "total_urls_found": 42,
    "average_scrape_time": 5432.5,
    "average_page_scrape_time": 234.8
  },
  "content_metadata": {
    "shipping_informations": [
      {
        "url": "https://example-shop.com/shipping",
        "changed": false,
        "last_updated": "2025-01-01T12:00:00.000Z"
      }
    ],
    "contacts": [
      {
        "url": "https://example-shop.com/contact",
        "changed": true,
        "last_updated": "2025-01-01T12:00:00.000Z"
      }
    ],
    "terms_of_conditions": [],
    "faq": []
  },
  "scrape_history": [
    {
      "id": "history-uuid-1",
      "job_id": "job-uuid-1",
      "started_at": "2025-01-01T12:00:00.000Z",
      "completed_at": "2025-01-01T12:05:00.000Z",
      "status": "completed",
      "error": null,
      "scrape_time_ms": 5432,
      "urls_found": 42
    }
  ],
  "scheduled_jobs": [
    {
      "id": "schedule-uuid-1",
      "next_run_at": "2025-01-02T12:00:00.000Z",
      "frequency": "daily",
      "last_modified": "2025-01-01T10:00:00.000Z",
      "status": "pending",
      "enabled": true,
      "created_at": "2025-01-01T10:00:00.000Z"
    }
  ]
}
```

**Notes:**
- This endpoint returns content metadata only (URLs, change status, last updated)
- To get full content, use the `/api/shops/:id/results` endpoint
- `changed` flag indicates if content has changed since the previous scrape

**Error Responses:**
- `404 Not Found` - Shop not found
- `503 Service Unavailable` - Database not available
- `500 Internal Server Error` - Server error

**Example:**
```bash
curl http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer your-secret-key"
```

---

### Get Shop Results

Get shop scraping results with full content. Supports filtering by date range, content type, and limit.

**Endpoint:** `GET /api/shops/:id/results`

**URL Parameters:**
- `id` - Shop UUID

**Query Parameters:**
- `limit` (optional) - Maximum number of results to return
- `date_from` (optional) - Filter results from this date (ISO 8601 format)
- `date_to` (optional) - Filter results to this date (ISO 8601 format)
- `content_type` (optional) - Filter by content type: `shipping`, `contacts`, `terms`, or `faq`

**Response:**
```json
{
  "shop_id": "550e8400-e29b-41d4-a716-446655440000",
  "filters": {
    "limit": 10,
    "date_from": "2025-01-01",
    "date_to": null,
    "content_type": null
  },
  "results": {
    "shipping_informations": [
      {
        "url": "https://example-shop.com/shipping",
        "content": "# Shipping Information\n\nDetailed shipping content...",
        "changed": false,
        "last_updated": "2025-01-01T12:00:00.000Z"
      },
      {
        "url": "https://example-shop.com/delivery-info",
        "content": "# Delivery Information\n\nDetailed delivery content...",
        "changed": true,
        "last_updated": "2025-01-01T12:00:00.000Z"
      }
    ],
    "contacts": [
      {
        "url": "https://example-shop.com/contact",
        "content": "# Contact Us\n\nEmail: contact@example.com\nPhone: +1234567890",
        "changed": true,
        "last_updated": "2025-01-01T12:00:00.000Z"
      }
    ],
    "terms_of_conditions": [
      {
        "url": "https://example-shop.com/terms",
        "content": "# Terms and Conditions\n\nDetailed terms...",
        "changed": false,
        "last_updated": "2025-01-01T12:00:00.000Z"
      }
    ],
    "faq": [
      {
        "url": "https://example-shop.com/faq",
        "content": "# Frequently Asked Questions\n\nQ: Question 1?\nA: Answer 1...",
        "changed": false,
        "last_updated": "2025-01-01T12:00:00.000Z"
      }
    ]
  }
}
```

**Notes:**
- All content categories use arrays to support multiple pages per category
- `changed` flag indicates if content has changed since the previous scrape
- Content is in Markdown format, converted from HTML

**Error Responses:**
- `404 Not Found` - Shop not found
- `503 Service Unavailable` - Database not available
- `500 Internal Server Error` - Server error

**Examples:**

Get all results:
```bash
curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results" \
  -H "Authorization: Bearer your-secret-key"
```

Get limited results:
```bash
curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?limit=10" \
  -H "Authorization: Bearer your-secret-key"
```

Get results from a specific date:
```bash
curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?date_from=2025-01-01" \
  -H "Authorization: Bearer your-secret-key"
```

Get only shipping information:
```bash
curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?content_type=shipping" \
  -H "Authorization: Bearer your-secret-key"
```

Get results with date range and limit:
```bash
curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?date_from=2025-01-01&date_to=2025-01-31&limit=50" \
  -H "Authorization: Bearer your-secret-key"
```

---

### Enable/Disable Schedule

Enable or disable scheduled scraping for a shop.

**Endpoint:** `PATCH /api/shops/:id/schedule`

**URL Parameters:**
- `id` - Shop UUID

**Request Body:**
```json
{
  "enabled": true
}
```

**Response:**
```json
{
  "shop_id": "550e8400-e29b-41d4-a716-446655440000",
  "schedule_enabled": true,
  "message": "Schedule enabled successfully"
}
```

**Error Responses:**
- `400 Bad Request` - Missing or invalid `enabled` field
- `404 Not Found` - Shop not found
- `503 Service Unavailable` - Database not available
- `500 Internal Server Error` - Server error

**Examples:**

Enable scheduling:
```bash
curl -X PATCH http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/schedule \
  -H "Authorization: Bearer your-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true}'
```

Disable scheduling:
```bash
curl -X PATCH http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/schedule \
  -H "Authorization: Bearer your-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'
```

---

### Delete Shop

Delete a shop and all related data (analytics, scrape history, content, scheduled jobs).

**Endpoint:** `DELETE /api/shops/:id`

**URL Parameters:**
- `id` - Shop UUID

**Response:**
```json
{
  "message": "Shop and all related data deleted successfully",
  "shop_id": "550e8400-e29b-41d4-a716-446655440000"
}
```

**Notes:**
- This operation is irreversible
- Deletes all associated data:
  - Shop record
  - Analytics
  - Scrape history
  - All content records
  - Scheduled jobs

**Error Responses:**
- `404 Not Found` - Shop not found
- `503 Service Unavailable` - Database not available
- `500 Internal Server Error` - Server error

**Example:**
```bash
curl -X DELETE http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer your-secret-key"
```

---

## Error Handling

All endpoints return standard HTTP status codes:

- `200 OK` - Request successful
- `201 Created` - Resource created successfully
- `400 Bad Request` - Invalid request parameters
- `404 Not Found` - Resource not found
- `500 Internal Server Error` - Server error
- `503 Service Unavailable` - Service (e.g., database) unavailable

Error responses include a JSON body with an `error` field:

```json
{
  "error": "Description of the error"
}
```

## Content Types

The scraper categorizes content into four types:

1. **shipping** (shipping_informations) - Shipping, delivery, and logistics information
2. **contacts** - Contact information (email, phone, addresses)
3. **terms** (terms_of_conditions) - Terms of service, privacy policy, legal information
4. **faq** - Frequently asked questions and help pages

Each category can contain multiple pages (returned as arrays).

## Scheduled Scraping

The application automatically schedules scraping based on sitemap rules:

- **Frequency rules** from sitemap: `always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly`
- **Last modified dates** from sitemap
- Schedules can be enabled/disabled per shop
- Scheduler checks for due jobs every minute
- After each successful scrape, the next scrape is automatically scheduled

## Content Change Detection

The system uses MD5 hashing to detect content changes:

- Each content piece is hashed when scraped
- Compared with the previous version's hash
- `changed` flag set to `true` if hashes differ
- Change detection works for all content types
- Useful for monitoring when shop policies or information are updated