|
@@ -0,0 +1,573 @@
|
|
|
|
|
+# API Documentation
|
|
|
|
|
+
|
|
|
|
|
+All endpoints (except `/health`) require Bearer authentication using the `Authorization` header:
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+Authorization: Bearer <your-api-key>
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+## Table of Contents
|
|
|
|
|
+
|
|
|
|
|
+- [Health Check](#health-check)
|
|
|
|
|
+- [Jobs](#jobs)
|
|
|
|
|
+ - [Create Job](#create-job)
|
|
|
|
|
+ - [Get Job Status](#get-job-status)
|
|
|
|
|
+ - [List All Jobs](#list-all-jobs)
|
|
|
|
|
+- [Shops](#shops)
|
|
|
|
|
+ - [List All Shops](#list-all-shops)
|
|
|
|
|
+ - [Get Shop Details](#get-shop-details)
|
|
|
|
|
+ - [Get Shop Results](#get-shop-results)
|
|
|
|
|
+ - [Enable/Disable Schedule](#enabledisable-schedule)
|
|
|
|
|
+ - [Delete Shop](#delete-shop)
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Health Check
|
|
|
|
|
+
|
|
|
|
|
+Check the health status of the application.
|
|
|
|
|
+
|
|
|
|
|
+**Endpoint:** `GET /health`
|
|
|
|
|
+
|
|
|
|
|
+**Authentication:** Not required
|
|
|
|
|
+
|
|
|
|
|
+**Response:**
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "status": "ok",
|
|
|
|
|
+ "timestamp": "2025-01-01T12:00:00.000Z",
|
|
|
|
|
+ "queue_stats": {
|
|
|
|
|
+ "total": 10,
|
|
|
|
|
+ "pending": 2,
|
|
|
|
|
+ "running": 1,
|
|
|
|
|
+ "completed": 6,
|
|
|
|
|
+ "failed": 1
|
|
|
|
|
+ }
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Jobs
|
|
|
|
|
+
|
|
|
|
|
+### Create Job
|
|
|
|
|
+
|
|
|
|
|
+Create a new scraping job for a webshop.
|
|
|
|
|
+
|
|
|
|
|
+**Endpoint:** `POST /api/jobs`
|
|
|
|
|
+
|
|
|
|
|
+**Request Body:**
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "url": "https://example-shop.com"
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Response:** `201 Created`
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
|
|
|
|
+ "shop_id": "550e8400-e29b-41d4-a716-446655440000",
|
|
|
|
|
+ "status": "pending",
|
|
|
|
|
+ "message": "Job created successfully"
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Error Responses:**
|
|
|
|
|
+- `400 Bad Request` - Missing or invalid URL
|
|
|
|
|
+- `500 Internal Server Error` - Server error
|
|
|
|
|
+
|
|
|
|
|
+**Example:**
|
|
|
|
|
+```bash
|
|
|
|
|
+curl -X POST http://localhost:3000/api/jobs \
|
|
|
|
|
+ -H "Authorization: Bearer your-secret-key" \
|
|
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
|
|
+ -d '{"url": "https://example-shop.com"}'
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+### Get Job Status
|
|
|
|
|
+
|
|
|
|
|
+Get the status and result of a specific job.
|
|
|
|
|
+
|
|
|
|
|
+**Endpoint:** `GET /api/jobs/:id`
|
|
|
|
|
+
|
|
|
|
|
+**URL Parameters:**
|
|
|
|
|
+- `id` - Job UUID
|
|
|
|
|
+
|
|
|
|
|
+**Response:**
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
|
|
|
|
+ "status": "completed",
|
|
|
|
|
+ "created_at": "2025-01-01T12:00:00.000Z",
|
|
|
|
|
+ "updated_at": "2025-01-01T12:05:00.000Z",
|
|
|
|
|
+ "result": {
|
|
|
|
|
+ "initial_sitemap": "https://example-shop.com/sitemap.xml",
|
|
|
|
|
+ "shipping_informations": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "url": "https://example-shop.com/shipping",
|
|
|
|
|
+ "content": "# Shipping Information\n\n..."
|
|
|
|
|
+ }
|
|
|
|
|
+ ],
|
|
|
|
|
+ "contacts": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "url": "https://example-shop.com/contact",
|
|
|
|
|
+ "content": "# Contact Us\n\n..."
|
|
|
|
|
+ }
|
|
|
|
|
+ ],
|
|
|
|
|
+ "terms_of_conditions": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "url": "https://example-shop.com/terms",
|
|
|
|
|
+ "content": "# Terms and Conditions\n\n..."
|
|
|
|
|
+ }
|
|
|
|
|
+ ],
|
|
|
|
|
+ "faq": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "url": "https://example-shop.com/faq",
|
|
|
|
|
+ "content": "# FAQ\n\n..."
|
|
|
|
|
+ }
|
|
|
|
|
+ ]
|
|
|
|
|
+ }
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Status Values:**
|
|
|
|
|
+- `pending` - Job is waiting in queue
|
|
|
|
|
+- `running` - Job is currently being processed
|
|
|
|
|
+- `completed` - Job finished successfully
|
|
|
|
|
+- `failed` - Job failed (includes `error` field)
|
|
|
|
|
+
|
|
|
|
|
+**Error Responses:**
|
|
|
|
|
+- `404 Not Found` - Job not found
|
|
|
|
|
+- `500 Internal Server Error` - Server error
|
|
|
|
|
+
|
|
|
|
|
+**Example:**
|
|
|
|
|
+```bash
|
|
|
|
|
+curl http://localhost:3000/api/jobs/a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
|
|
|
|
|
+ -H "Authorization: Bearer your-secret-key"
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+### List All Jobs
|
|
|
|
|
+
|
|
|
|
|
+Get a list of all jobs with queue statistics.
|
|
|
|
|
+
|
|
|
|
|
+**Endpoint:** `GET /api/jobs`
|
|
|
|
|
+
|
|
|
|
|
+**Response:**
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "stats": {
|
|
|
|
|
+ "total": 10,
|
|
|
|
|
+ "pending": 2,
|
|
|
|
|
+ "running": 1,
|
|
|
|
|
+ "completed": 6,
|
|
|
|
|
+ "failed": 1
|
|
|
|
|
+ },
|
|
|
|
|
+ "jobs": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
|
|
|
|
+ "status": "completed",
|
|
|
|
|
+ "sitemap_url": "https://example-shop.com",
|
|
|
|
|
+ "created_at": "2025-01-01T12:00:00.000Z",
|
|
|
|
|
+ "updated_at": "2025-01-01T12:05:00.000Z"
|
|
|
|
|
+ }
|
|
|
|
|
+ ]
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Example:**
|
|
|
|
|
+```bash
|
|
|
|
|
+curl http://localhost:3000/api/jobs \
|
|
|
|
|
+ -H "Authorization: Bearer your-secret-key"
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Shops
|
|
|
|
|
+
|
|
|
|
|
+### List All Shops
|
|
|
|
|
+
|
|
|
|
|
+Get a list of all shops with their analytics.
|
|
|
|
|
+
|
|
|
|
|
+**Endpoint:** `GET /api/shops`
|
|
|
|
|
+
|
|
|
|
|
+**Response:**
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "shops": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "id": "550e8400-e29b-41d4-a716-446655440000",
|
|
|
|
|
+ "url": "https://example-shop.com",
|
|
|
|
|
+ "sitemap_url": "https://example-shop.com/sitemap.xml",
|
|
|
|
|
+ "webshop_type": "shopify",
|
|
|
|
|
+ "created_at": "2025-01-01T10:00:00.000Z",
|
|
|
|
|
+ "updated_at": "2025-01-01T12:00:00.000Z",
|
|
|
|
|
+ "analytics": {
|
|
|
|
|
+ "total_scrapes": 5,
|
|
|
|
|
+ "last_scraped_at": "2025-01-01T12:00:00.000Z",
|
|
|
|
|
+ "next_scrape_at": "2025-01-02T12:00:00.000Z",
|
|
|
|
|
+ "total_urls_found": 42,
|
|
|
|
|
+ "average_scrape_time": 5432.5,
|
|
|
|
|
+ "average_page_scrape_time": 234.8
|
|
|
|
|
+ }
|
|
|
|
|
+ }
|
|
|
|
|
+ ]
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Error Responses:**
|
|
|
|
|
+- `503 Service Unavailable` - Database not available
|
|
|
|
|
+- `500 Internal Server Error` - Server error
|
|
|
|
|
+
|
|
|
|
|
+**Example:**
|
|
|
|
|
+```bash
|
|
|
|
|
+curl http://localhost:3000/api/shops \
|
|
|
|
|
+ -H "Authorization: Bearer your-secret-key"
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+### Get Shop Details
|
|
|
|
|
+
|
|
|
|
|
+Get detailed information about a specific shop including analytics, content metadata, scrape history, and scheduled jobs.
|
|
|
|
|
+
|
|
|
|
|
+**Endpoint:** `GET /api/shops/:id`
|
|
|
|
|
+
|
|
|
|
|
+**URL Parameters:**
|
|
|
|
|
+- `id` - Shop UUID
|
|
|
|
|
+
|
|
|
|
|
+**Response:**
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "shop": {
|
|
|
|
|
+ "id": "550e8400-e29b-41d4-a716-446655440000",
|
|
|
|
|
+ "url": "https://example-shop.com",
|
|
|
|
|
+ "sitemap_url": "https://example-shop.com/sitemap.xml",
|
|
|
|
|
+ "webshop_type": "shopify",
|
|
|
|
|
+ "created_at": "2025-01-01T10:00:00.000Z",
|
|
|
|
|
+ "updated_at": "2025-01-01T12:00:00.000Z"
|
|
|
|
|
+ },
|
|
|
|
|
+ "analytics": {
|
|
|
|
|
+ "total_scrapes": 5,
|
|
|
|
|
+ "last_scraped_at": "2025-01-01T12:00:00.000Z",
|
|
|
|
|
+ "next_scrape_at": "2025-01-02T12:00:00.000Z",
|
|
|
|
|
+ "total_urls_found": 42,
|
|
|
|
|
+ "average_scrape_time": 5432.5,
|
|
|
|
|
+ "average_page_scrape_time": 234.8
|
|
|
|
|
+ },
|
|
|
|
|
+ "content_metadata": {
|
|
|
|
|
+ "shipping_informations": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "url": "https://example-shop.com/shipping",
|
|
|
|
|
+ "changed": false,
|
|
|
|
|
+ "last_updated": "2025-01-01T12:00:00.000Z"
|
|
|
|
|
+ }
|
|
|
|
|
+ ],
|
|
|
|
|
+ "contacts": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "url": "https://example-shop.com/contact",
|
|
|
|
|
+ "changed": true,
|
|
|
|
|
+ "last_updated": "2025-01-01T12:00:00.000Z"
|
|
|
|
|
+ }
|
|
|
|
|
+ ],
|
|
|
|
|
+ "terms_of_conditions": [],
|
|
|
|
|
+ "faq": []
|
|
|
|
|
+ },
|
|
|
|
|
+ "scrape_history": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "id": "history-uuid-1",
|
|
|
|
|
+ "job_id": "job-uuid-1",
|
|
|
|
|
+ "started_at": "2025-01-01T12:00:00.000Z",
|
|
|
|
|
+ "completed_at": "2025-01-01T12:05:00.000Z",
|
|
|
|
|
+ "status": "completed",
|
|
|
|
|
+ "error": null,
|
|
|
|
|
+ "scrape_time_ms": 5432,
|
|
|
|
|
+ "urls_found": 42
|
|
|
|
|
+ }
|
|
|
|
|
+ ],
|
|
|
|
|
+ "scheduled_jobs": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "id": "schedule-uuid-1",
|
|
|
|
|
+ "next_run_at": "2025-01-02T12:00:00.000Z",
|
|
|
|
|
+ "frequency": "daily",
|
|
|
|
|
+ "last_modified": "2025-01-01T10:00:00.000Z",
|
|
|
|
|
+ "status": "pending",
|
|
|
|
|
+ "enabled": true,
|
|
|
|
|
+ "created_at": "2025-01-01T10:00:00.000Z"
|
|
|
|
|
+ }
|
|
|
|
|
+ ]
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Notes:**
|
|
|
|
|
+- This endpoint returns content metadata only (URLs, change status, last updated)
|
|
|
|
|
+- To get full content, use the `/api/shops/:id/results` endpoint
|
|
|
|
|
+- `changed` flag indicates if content has changed since the previous scrape
|
|
|
|
|
+
|
|
|
|
|
+**Error Responses:**
|
|
|
|
|
+- `404 Not Found` - Shop not found
|
|
|
|
|
+- `503 Service Unavailable` - Database not available
|
|
|
|
|
+- `500 Internal Server Error` - Server error
|
|
|
|
|
+
|
|
|
|
|
+**Example:**
|
|
|
|
|
+```bash
|
|
|
|
|
+curl http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000 \
|
|
|
|
|
+ -H "Authorization: Bearer your-secret-key"
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+### Get Shop Results
|
|
|
|
|
+
|
|
|
|
|
+Get shop scraping results with full content. Supports filtering by date range, content type, and limit.
|
|
|
|
|
+
|
|
|
|
|
+**Endpoint:** `GET /api/shops/:id/results`
|
|
|
|
|
+
|
|
|
|
|
+**URL Parameters:**
|
|
|
|
|
+- `id` - Shop UUID
|
|
|
|
|
+
|
|
|
|
|
+**Query Parameters:**
|
|
|
|
|
+- `limit` (optional) - Maximum number of results to return
|
|
|
|
|
+- `date_from` (optional) - Filter results from this date (ISO 8601 format)
|
|
|
|
|
+- `date_to` (optional) - Filter results to this date (ISO 8601 format)
|
|
|
|
|
+- `content_type` (optional) - Filter by content type: `shipping`, `contacts`, `terms`, or `faq`
|
|
|
|
|
+
|
|
|
|
|
+**Response:**
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "shop_id": "550e8400-e29b-41d4-a716-446655440000",
|
|
|
|
|
+ "filters": {
|
|
|
|
|
+ "limit": 10,
|
|
|
|
|
+ "date_from": "2025-01-01",
|
|
|
|
|
+ "date_to": null,
|
|
|
|
|
+ "content_type": null
|
|
|
|
|
+ },
|
|
|
|
|
+ "results": {
|
|
|
|
|
+ "shipping_informations": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "url": "https://example-shop.com/shipping",
|
|
|
|
|
+ "content": "# Shipping Information\n\nDetailed shipping content...",
|
|
|
|
|
+ "changed": false,
|
|
|
|
|
+ "last_updated": "2025-01-01T12:00:00.000Z"
|
|
|
|
|
+ },
|
|
|
|
|
+ {
|
|
|
|
|
+ "url": "https://example-shop.com/delivery-info",
|
|
|
|
|
+ "content": "# Delivery Information\n\nDetailed delivery content...",
|
|
|
|
|
+ "changed": true,
|
|
|
|
|
+ "last_updated": "2025-01-01T12:00:00.000Z"
|
|
|
|
|
+ }
|
|
|
|
|
+ ],
|
|
|
|
|
+ "contacts": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "url": "https://example-shop.com/contact",
|
|
|
|
|
+ "content": "# Contact Us\n\nEmail: contact@example.com\nPhone: +1234567890",
|
|
|
|
|
+ "changed": true,
|
|
|
|
|
+ "last_updated": "2025-01-01T12:00:00.000Z"
|
|
|
|
|
+ }
|
|
|
|
|
+ ],
|
|
|
|
|
+ "terms_of_conditions": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "url": "https://example-shop.com/terms",
|
|
|
|
|
+ "content": "# Terms and Conditions\n\nDetailed terms...",
|
|
|
|
|
+ "changed": false,
|
|
|
|
|
+ "last_updated": "2025-01-01T12:00:00.000Z"
|
|
|
|
|
+ }
|
|
|
|
|
+ ],
|
|
|
|
|
+ "faq": [
|
|
|
|
|
+ {
|
|
|
|
|
+ "url": "https://example-shop.com/faq",
|
|
|
|
|
+ "content": "# Frequently Asked Questions\n\nQ: Question 1?\nA: Answer 1...",
|
|
|
|
|
+ "changed": false,
|
|
|
|
|
+ "last_updated": "2025-01-01T12:00:00.000Z"
|
|
|
|
|
+ }
|
|
|
|
|
+ ]
|
|
|
|
|
+ }
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Notes:**
|
|
|
|
|
+- All content categories use arrays to support multiple pages per category
|
|
|
|
|
+- `changed` flag indicates if content has changed since the previous scrape
|
|
|
|
|
+- Content is in Markdown format, converted from HTML
|
|
|
|
|
+
|
|
|
|
|
+**Error Responses:**
|
|
|
|
|
+- `404 Not Found` - Shop not found
|
|
|
|
|
+- `503 Service Unavailable` - Database not available
|
|
|
|
|
+- `500 Internal Server Error` - Server error
|
|
|
|
|
+
|
|
|
|
|
+**Examples:**
|
|
|
|
|
+
|
|
|
|
|
+Get all results:
|
|
|
|
|
+```bash
|
|
|
|
|
+curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results" \
|
|
|
|
|
+ -H "Authorization: Bearer your-secret-key"
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Get limited results:
|
|
|
|
|
+```bash
|
|
|
|
|
+curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?limit=10" \
|
|
|
|
|
+ -H "Authorization: Bearer your-secret-key"
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Get results from a specific date:
|
|
|
|
|
+```bash
|
|
|
|
|
+curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?date_from=2025-01-01" \
|
|
|
|
|
+ -H "Authorization: Bearer your-secret-key"
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Get only shipping information:
|
|
|
|
|
+```bash
|
|
|
|
|
+curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?content_type=shipping" \
|
|
|
|
|
+ -H "Authorization: Bearer your-secret-key"
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Get results with date range and limit:
|
|
|
|
|
+```bash
|
|
|
|
|
+curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?date_from=2025-01-01&date_to=2025-01-31&limit=50" \
|
|
|
|
|
+ -H "Authorization: Bearer your-secret-key"
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+### Enable/Disable Schedule
|
|
|
|
|
+
|
|
|
|
|
+Enable or disable scheduled scraping for a shop.
|
|
|
|
|
+
|
|
|
|
|
+**Endpoint:** `PATCH /api/shops/:id/schedule`
|
|
|
|
|
+
|
|
|
|
|
+**URL Parameters:**
|
|
|
|
|
+- `id` - Shop UUID
|
|
|
|
|
+
|
|
|
|
|
+**Request Body:**
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "enabled": true
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Response:**
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "shop_id": "550e8400-e29b-41d4-a716-446655440000",
|
|
|
|
|
+ "schedule_enabled": true,
|
|
|
|
|
+ "message": "Schedule enabled successfully"
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Error Responses:**
|
|
|
|
|
+- `400 Bad Request` - Missing or invalid `enabled` field
|
|
|
|
|
+- `404 Not Found` - Shop not found
|
|
|
|
|
+- `503 Service Unavailable` - Database not available
|
|
|
|
|
+- `500 Internal Server Error` - Server error
|
|
|
|
|
+
|
|
|
|
|
+**Examples:**
|
|
|
|
|
+
|
|
|
|
|
+Enable scheduling:
|
|
|
|
|
+```bash
|
|
|
|
|
+curl -X PATCH http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/schedule \
|
|
|
|
|
+ -H "Authorization: Bearer your-secret-key" \
|
|
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
|
|
+ -d '{"enabled": true}'
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Disable scheduling:
|
|
|
|
|
+```bash
|
|
|
|
|
+curl -X PATCH http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/schedule \
|
|
|
|
|
+ -H "Authorization: Bearer your-secret-key" \
|
|
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
|
|
+ -d '{"enabled": false}'
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+### Delete Shop
|
|
|
|
|
+
|
|
|
|
|
+Delete a shop and all related data (analytics, scrape history, content, scheduled jobs).
|
|
|
|
|
+
|
|
|
|
|
+**Endpoint:** `DELETE /api/shops/:id`
|
|
|
|
|
+
|
|
|
|
|
+**URL Parameters:**
|
|
|
|
|
+- `id` - Shop UUID
|
|
|
|
|
+
|
|
|
|
|
+**Response:**
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "message": "Shop and all related data deleted successfully",
|
|
|
|
|
+ "shop_id": "550e8400-e29b-41d4-a716-446655440000"
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**Notes:**
|
|
|
|
|
+- This operation is irreversible
|
|
|
|
|
+- Deletes all associated data:
|
|
|
|
|
+ - Shop record
|
|
|
|
|
+ - Analytics
|
|
|
|
|
+ - Scrape history
|
|
|
|
|
+ - All content records
|
|
|
|
|
+ - Scheduled jobs
|
|
|
|
|
+
|
|
|
|
|
+**Error Responses:**
|
|
|
|
|
+- `404 Not Found` - Shop not found
|
|
|
|
|
+- `503 Service Unavailable` - Database not available
|
|
|
|
|
+- `500 Internal Server Error` - Server error
|
|
|
|
|
+
|
|
|
|
|
+**Example:**
|
|
|
|
|
+```bash
|
|
|
|
|
+curl -X DELETE http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000 \
|
|
|
|
|
+ -H "Authorization: Bearer your-secret-key"
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## Error Handling
|
|
|
|
|
+
|
|
|
|
|
+All endpoints return standard HTTP status codes:
|
|
|
|
|
+
|
|
|
|
|
+- `200 OK` - Request successful
|
|
|
|
|
+- `201 Created` - Resource created successfully
|
|
|
|
|
+- `400 Bad Request` - Invalid request parameters
|
|
|
|
|
+- `404 Not Found` - Resource not found
|
|
|
|
|
+- `500 Internal Server Error` - Server error
|
|
|
|
|
+- `503 Service Unavailable` - Service (e.g., database) unavailable
|
|
|
|
|
+
|
|
|
|
|
+Error responses include a JSON body with an `error` field:
|
|
|
|
|
+
|
|
|
|
|
+```json
|
|
|
|
|
+{
|
|
|
|
|
+ "error": "Description of the error"
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+## Content Types
|
|
|
|
|
+
|
|
|
|
|
+The scraper categorizes content into four types:
|
|
|
|
|
+
|
|
|
|
|
+1. **shipping** (shipping_informations) - Shipping, delivery, and logistics information
|
|
|
|
|
+2. **contacts** - Contact information (email, phone, addresses)
|
|
|
|
|
+3. **terms** (terms_of_conditions) - Terms of service, privacy policy, legal information
|
|
|
|
|
+4. **faq** - Frequently asked questions and help pages
|
|
|
|
|
+
|
|
|
|
|
+Each category can contain multiple pages (returned as arrays).
|
|
|
|
|
+
|
|
|
|
|
+## Scheduled Scraping
|
|
|
|
|
+
|
|
|
|
|
+The application automatically schedules scraping based on sitemap rules:
|
|
|
|
|
+
|
|
|
|
|
+- **Frequency rules** from sitemap: `always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly`
|
|
|
|
|
+- **Last modified dates** from sitemap
|
|
|
|
|
+- Schedules can be enabled/disabled per shop
|
|
|
|
|
+- Scheduler checks for due jobs every minute
|
|
|
|
|
+- After each successful scrape, the next scrape is automatically scheduled
|
|
|
|
|
+
|
|
|
|
|
+## Content Change Detection
|
|
|
|
|
+
|
|
|
|
|
+The system uses MD5 hashing to detect content changes:
|
|
|
|
|
+
|
|
|
|
|
+- Each content piece is hashed when scraped
|
|
|
|
|
+- Compared with the previous version's hash
|
|
|
|
|
+- `changed` flag set to `true` if hashes differ
|
|
|
|
|
+- Change detection works for all content types
|
|
|
|
|
+- Useful for monitoring when shop policies or information are updated
|