docs: rewrite README and docs from source of truth

Delete stale docs (API_REFERENCE, QDRANT_IMPLEMENTATION, SETUP_GUIDE) that
had drifted from the code — wrong auth header, fabricated rate limiting,
wrong MCP tool names, wrong Qdrant endpoint method/path, missing
QDRANT_ENABLED env var.

Rewrite README.md, docs/API.md and docs/SETUP.md against the real source:
- Bearer-only auth (matches src/api/middleware/auth.ts)
- Actual MCP tool names: list_contents, shipping_search, contacts_search,
  terms_search, faq_search
- Full endpoint surface map extracted from src/api/components/*
- Env var table matches src/config/index.ts exactly
- Delegate full endpoint reference to the live /doc and /doc/openapi so
  docs can no longer drift from code

README.md

@@ -1,208 +1,147 @@
 # Webshop Scraper
 
-A TypeScript-based web scraper for extracting information from webshops with persistent storage, scheduled scraping, and change detection.
+A TypeScript service that scrapes shipping, contact, terms, and FAQ pages from webshops (ShopRenter, WooCommerce, Shopify), stores them in SQLite with change detection, and optionally indexes them into Qdrant for semantic search via MCP tools.
 
 ## Features
 
-- **Multi-webshop Support**: ShopRenter, WooCommerce, and Shopify
-- **Persistent Storage**: SQLite database with UUID tracking and analytics
-- **Scheduled Scraping**: Automated scraping based on sitemap frequency rules
-- **Content Change Detection**: SHA-256 hash-based change tracking for all content
-- **Vector Search Integration**: Qdrant-powered semantic search with OpenRouter embeddings
-- **MCP Tools**: Model Context Protocol integration for LLM applications
-- **REST API**: Full CRUD operations for shops and scraping jobs
-- **High Performance**: Optimized database queries with 40-90% faster response times
-- **HTTP Compression**: Automatic gzip/brotli compression (60-80% smaller responses)
-- **Pagination Support**: Efficient pagination for large datasets
-- **Web Interface**: Complete administrative dashboard with real-time monitoring
-- **Bearer Authentication**: Secure API access
-- **Job Queue**: Configurable concurrency control
-- **Webhook Support**: Real-time notifications for scraping events
-- **Systemd Integration**: Install as a system service
-
-## Quick Start
-
-### Installation
-
-1. Clone and install:
+- **Multi-platform scraping** — ShopRenter, WooCommerce, Shopify
+- **Sitemap-driven** — parses sitemaps, classifies pages into `shipping` / `contacts` / `terms` / `faq`
+- **Scheduled scraping** — automatic re-scrape based on sitemap `changefreq` rules, toggleable per shop
+- **Change detection** — SHA-256 content hashing; unchanged pages are not re-processed
+- **Persistent storage** — SQLite (`better-sqlite3`) with shop analytics and scrape history
+- **Optional vector search** — Qdrant + OpenRouter embeddings, one collection per `{custom_id}-{category}`
+- **MCP server** — Streamable HTTP MCP endpoint at `/mcp` exposing `list_contents`, `shipping_search`, `contacts_search`, `terms_search`, `faq_search`
+- **Self-documenting REST API** — endpoint metadata is rendered as HTML at `/doc` and OpenAPI 3.0 at `/doc/openapi`
+- **Bearer authentication** for all `/api/*` endpoints
+- **Webhooks** per shop — `scrape_started` / `scrape_completed` / `scrape_failed`
+- **gzip/brotli compression** on all responses over 1 KB
+- **Web UI** served at `/ui`
+- **systemd installer** under `scripts/`
+
+## Quick start
+
 ```bash
 git clone <repository-url>
 cd webshop-scraper
 npm install
-```
-
-2. Build:
-```bash
 npm run build
+cp .env.example .env   # then edit API_KEY and anything else you need
+npm start
 ```
 
-3. Configure environment:
-```bash
-# Basic Configuration
-API_KEY=your-secret-key
-PORT=3000
-MAX_CONCURRENT_JOBS=3
-
-# Logging (Optional)
-LOG_HTTP_VERBOSE=true  # Enable detailed HTTP logs with User-Agent and content-length
+Then open:
 
-# Qdrant Vector Search (Optional)
-QDRANT_API_URL=http://localhost:6333
-QDRANT_API_KEY=your-qdrant-key
+- `http://localhost:3000/doc` — live API reference (HTML)
+- `http://localhost:3000/ui` — web dashboard
+- `http://localhost:3000/health` — liveness and queue stats (no auth)
 
-# OpenRouter for Embeddings (Optional)
-OPENROUTER_API_KEY=your-openrouter-key
-OPENROUTER_MODEL=openai/text-embedding-3-large
+## Minimal configuration
 
-# MCP Integration (Optional)
-MCP_ENABLED=true
-MCP_TRANSPORT=stdio
-```
+Only `API_KEY` is required. See [`docs/SETUP.md`](./docs/SETUP.md) for the full environment variable reference.
 
-4. Start:
 ```bash
-npm start
+API_KEY=change-me
+PORT=3000
+MAX_CONCURRENT_JOBS=3
 ```
 
-### As a systemd service
+For Qdrant + MCP, add:
 
 ```bash
-sudo ./scripts/install.sh
+QDRANT_ENABLED=true
+QDRANT_API_URL=http://localhost:6333
+OPENROUTER_API_KEY=sk-or-v1-...
+MCP_ENABLED=true
 ```
 
-### Reset and Restore
+## Authentication
 
-Reset the system to a clean state (creates backup first):
-```bash
-sudo ./scripts/reset.sh
-```
+Every `/api/*` endpoint requires a Bearer token matching `API_KEY`:
 
-Restore from a previous backup:
 ```bash
-sudo ./scripts/restore.sh
+curl http://localhost:3000/api/shops \
+  -H "Authorization: Bearer $API_KEY"
 ```
 
-**What gets backed up:**
-- SQLite database (data/shops.db)
-- Environment configuration (.env)
-- Service logs from journalctl
-
-Backups are stored in `/var/backups/webshop-scraper/` with timestamps.
-
-## Vector Search & MCP Integration
-
-The system includes optional Qdrant vector search capabilities for semantic search of scraped content:
-
-- **Automatic Embedding**: Content is automatically embedded using OpenRouter's text-embedding-3-large model
-- **Semantic Search**: 4 MCP tools provide semantic search capabilities for LLM applications
-- **Real-time Management**: Web interface for enabling/disabling and monitoring embeddings
-- **Error Tracking**: Comprehensive error logging and debugging capabilities
-
-### MCP Tools Available
-
-- `search_shipping_info(shop_id, query, limit?)` - Search shipping and delivery information
-- `search_contact_info(shop_id, query, limit?)` - Search contact and support information
-- `search_terms_info(shop_id, query, limit?)` - Search terms of service and policies
-- `search_faq_info(shop_id, query, limit?)` - Search frequently asked questions
-
-See the [Qdrant Setup Guide](./docs/SETUP_GUIDE.md) for complete configuration instructions.
+`/health`, `/doc*`, `/ui/*`, and `/mcp*` are public.
 
-## API Documentation
-
-See [API Documentation](./docs/API.md) for complete endpoint reference.
-See [Qdrant API Reference](./docs/API_REFERENCE.md) for vector search endpoints.
-
-### Quick Examples
+## Usage examples
 
 **Create a scraping job:**
+
 ```bash
 curl -X POST http://localhost:3000/api/jobs \
-  -H "Authorization: Bearer your-secret-key" \
+  -H "Authorization: Bearer $API_KEY" \
   -H "Content-Type: application/json" \
   -d '{"url": "https://example-shop.com"}'
 ```
 
-**Get shop information:**
+**Get shop details:**
+
 ```bash
-curl http://localhost:3000/api/shops/:id \
-  -H "Authorization: Bearer your-secret-key"
+curl http://localhost:3000/api/shops/<id> \
+  -H "Authorization: Bearer $API_KEY"
 ```
 
-**Get shop results:**
+**Get shop results filtered to shipping content, last 10 entries:**
+
 ```bash
-curl "http://localhost:3000/api/shops/:id/results?limit=10" \
-  -H "Authorization: Bearer your-secret-key"
+curl "http://localhost:3000/api/shops/<id>/results?content_type=shipping&limit=10" \
+  -H "Authorization: Bearer $API_KEY"
 ```
 
-## Requirements
+`<id>` can be either the system-generated UUID or the optional `custom_id` you assigned to the shop.
+
+## MCP tools
+
+When `MCP_ENABLED=true` and `QDRANT_ENABLED=true`, the server mounts an MCP Streamable HTTP endpoint at `/mcp`. Five tools are registered:
 
-- Node.js v16 or higher
-- npm
-- Linux with systemd (for service installation)
+- `list_contents(shop_id)` — enumerate available pages per category, tagged with the next tool to use
+- `shipping_search(shop_id, query, limit?)`
+- `contacts_search(shop_id, query, limit?)`
+- `terms_search(shop_id, query, limit?)`
+- `faq_search(shop_id, query, limit?)`
+
+All tools take the shop's `custom_id` as `shop_id`. Tools return a friendly "Search not available for this shop" message when the shop is unknown or Qdrant is not enabled on it.
+
+## Documentation
+
+- [`docs/API.md`](./docs/API.md) — API concepts, auth, endpoint surface map, MCP tool inventory. For the full endpoint reference use `/doc` on the running server.
+- [`docs/SETUP.md`](./docs/SETUP.md) — installation, environment variables, Qdrant/MCP setup, migration, troubleshooting.
 
 ## Development
 
 ```bash
-# Development mode
-npm run dev
+npm run dev     # run with ts-node, no build
+npm run watch   # tsc in watch mode
+npm run build   # build TS + web UI
+```
 
-# Watch mode
-npm run watch
+## Systemd service
 
-# Build
-npm run build
+```bash
+sudo ./scripts/install.sh   # install as service
+sudo ./scripts/reset.sh     # wipe to clean state (backs up first)
+sudo ./scripts/restore.sh   # restore from backup
 ```
 
-## Database
+Backups land in `/var/backups/webshop-scraper/`.
 
-Data is stored in `data/shops.db` (SQLite). The database includes:
-- Shop information with UUID tracking
-- Scrape analytics and history
-- Content with change detection
-- Scheduled jobs queue
+## Data
 
-## Logging
+SQLite lives in `data/shops.db`. Schema is applied at startup from `src/database/Database.ts`. It tracks shops (with UUID and optional `custom_id`), scrape history, per-page content with hashes, scheduled jobs, webhooks, and — when Qdrant is enabled — embedding state, deletion queue, error log, and MCP call logs.
 
-The system includes enhanced HTTP access logging with status codes, response times, and optional verbose mode.
+## Logs
 
-### Log Format
+HTTP access logs are emitted per request with status code icon, path, IP, response time, and (with `LOG_HTTP_VERBOSE=true`) user agent and content length.
 
-**Standard HTTP logs:**
 ```
 [INFO] [WebshopScraper] ✓ 200 GET /api/shops ::1 45ms
 [INFO] [WebshopScraper] ⚠️ 404 GET /api/shops/invalid ::1 12ms
 [INFO] [WebshopScraper] ❌ 500 POST /api/jobs ::1 234ms
 ```
 
-**Verbose HTTP logs** (with `LOG_HTTP_VERBOSE=true`):
-```
-[INFO] [WebshopScraper] ✓ 200 GET /api/shops ::1 45ms 2456b UA:"curl/7.81.0"
-```
-
-### Status Code Indicators
-- ✓ (200-299) - Success
-- ↪️ (300-399) - Redirect
-- ⚠️ (400-499) - Client error
-- ❌ (500-599) - Server error
-
-### View Logs
-
-Using journalctl (systemd):
-```bash
-# Follow logs in real-time
-sudo journalctl -u webshop-scraper -f
-
-# View logs with timestamps
-sudo journalctl -u webshop-scraper --no-pager
-
-# Filter by priority
-sudo journalctl -u webshop-scraper -p err  # Only errors
-```
-
-Using npm (development):
-```bash
-npm run dev  # Includes timestamps in output
-```
+Icons: `✓` 2xx, `↪️` 3xx, `⚠️` 4xx, `❌` 5xx.
 
 ## License
 

docs/API.md

@@ -1,636 +1,187 @@
-# API Documentation
+# API Overview
 
-All endpoints (except `/health`) require Bearer authentication using the `Authorization` header:
+This project ships a **self-documenting REST API**. Each endpoint carries its own metadata (parameters, request body, responses, auth requirement) and an endpoint registry generates both browsable HTML and an OpenAPI 3.0 spec at runtime.
 
-```
-Authorization: Bearer <your-api-key>
-```
-
-## Shop Identifiers
-
-All shop-related endpoints support dual identifier access:
-
-- **Internal UUID**: Auto-generated unique identifier assigned by the system (e.g., `550e8400-e29b-41d4-a716-446655440000`)
-- **Custom UUID**: Optional custom identifier set by the user (e.g., `cf1a0652-d4cd-4e6f-85a0-87d203578c35`)
-
-You can use either identifier in any shop endpoint URL parameter. Custom UUIDs are optional and must be in valid UUID format if provided.
-
-## Table of Contents
-
-- [Health Check](#health-check)
-- [Jobs](#jobs)
-  - [Create Job](#create-job)
-  - [Get Job Status](#get-job-status)
-  - [List All Jobs](#list-all-jobs)
-- [Shops](#shops)
-  - [List All Shops](#list-all-shops)
-  - [Get Shop Details](#get-shop-details)
-  - [Get Shop Results](#get-shop-results)
-  - [Set/Update Custom ID](#setupdate-custom-id)
-  - [Enable/Disable Schedule](#enabledisable-schedule)
-  - [Delete Shop](#delete-shop)
-
----
-
-## Health Check
-
-Check the health status of the application.
-
-**Endpoint:** `GET /health`
-
-**Authentication:** Not required
-
-**Response:**
-```json
-{
-  "status": "ok",
-  "timestamp": "2025-01-01T12:00:00.000Z",
-  "queue_stats": {
-    "total": 10,
-    "pending": 2,
-    "running": 1,
-    "completed": 6,
-    "failed": 1
-  }
-}
-```
-
----
-
-## Jobs
-
-### Create Job
-
-Create a new scraping job for a webshop.
-
-**Endpoint:** `POST /api/jobs`
-
-**Request Body:**
-```json
-{
-  "url": "https://example-shop.com",
-  "custom_id": "cf1a0652-d4cd-4e6f-85a0-87d203578c35"
-}
-```
-
-**Fields:**
-- `url` (required) - The main URL of the webshop to scrape
-- `custom_id` (optional) - Custom UUID identifier for the shop (must be unique)
-
-**Response:** `201 Created`
-```json
-{
-  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
-  "shop_id": "550e8400-e29b-41d4-a716-446655440000",
-  "status": "pending",
-  "message": "Job created successfully"
-}
-```
-
-**Error Responses:**
-- `400 Bad Request` - Missing or invalid URL, or invalid custom_id format
-- `409 Conflict` - custom_id is already in use
-- `500 Internal Server Error` - Server error
-
-**Example:**
-```bash
-curl -X POST http://localhost:3000/api/jobs \
-  -H "Authorization: Bearer your-secret-key" \
-  -H "Content-Type: application/json" \
-  -d '{"url": "https://example-shop.com"}'
-```
-
----
-
-### Get Job Status
-
-Get the status and result of a specific job.
-
-**Endpoint:** `GET /api/jobs/:id`
+**Always treat the running server as the source of truth.** This file covers the concepts — auth, conventions, shape of the surface — and points you at the live reference for the full endpoint list.
 
-**URL Parameters:**
-- `id` - Job UUID
+## Live reference
 
-**Response:**
-```json
-{
-  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
-  "status": "completed",
-  "created_at": "2025-01-01T12:00:00.000Z",
-  "updated_at": "2025-01-01T12:05:00.000Z",
-  "result": {
-    "initial_sitemap": "https://example-shop.com/sitemap.xml",
-    "shipping_informations": [
-      {
-        "url": "https://example-shop.com/shipping",
-        "content": "# Shipping Information\n\n..."
-      }
-    ],
-    "contacts": [
-      {
-        "url": "https://example-shop.com/contact",
-        "content": "# Contact Us\n\n..."
-      }
-    ],
-    "terms_of_conditions": [
-      {
-        "url": "https://example-shop.com/terms",
-        "content": "# Terms and Conditions\n\n..."
-      }
-    ],
-    "faq": [
-      {
-        "url": "https://example-shop.com/faq",
-        "content": "# FAQ\n\n..."
-      }
-    ]
-  }
-}
-```
-
-**Status Values:**
-- `pending` - Job is waiting in queue
-- `running` - Job is currently being processed
-- `completed` - Job finished successfully
-- `failed` - Job failed (includes `error` field)
+Once the server is running, three documentation endpoints are available (no auth required):
 
-**Error Responses:**
-- `404 Not Found` - Job not found
-- `500 Internal Server Error` - Server error
+| Endpoint | Returns |
+|---|---|
+| `GET /doc` | Interactive HTML documentation page |
+| `GET /doc/openapi` | OpenAPI 3.0 JSON specification |
+| `GET /doc/endpoints` | Compact JSON list of `{ method, path, summary, tags, requiresAuth }` |
 
-**Example:**
 ```bash
-curl http://localhost:3000/api/jobs/a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
-  -H "Authorization: Bearer your-secret-key"
-```
-
----
-
-### List All Jobs
-
-Get a list of all jobs with queue statistics.
-
-**Endpoint:** `GET /api/jobs`
-
-**Response:**
-```json
-{
-  "stats": {
-    "total": 10,
-    "pending": 2,
-    "running": 1,
-    "completed": 6,
-    "failed": 1
-  },
-  "jobs": [
-    {
-      "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
-      "status": "completed",
-      "sitemap_url": "https://example-shop.com",
-      "created_at": "2025-01-01T12:00:00.000Z",
-      "updated_at": "2025-01-01T12:05:00.000Z"
-    }
-  ]
-}
-```
+# Browse in a browser:
+open http://localhost:3000/doc
 
-**Example:**
-```bash
-curl http://localhost:3000/api/jobs \
-  -H "Authorization: Bearer your-secret-key"
+# Pipe the spec into anything OpenAPI-aware:
+curl http://localhost:3000/doc/openapi | jq .
 ```
 
----
-
-## Shops
-
-### List All Shops
+## Authentication
 
-Get a list of all shops with their analytics.
+All `/api/*` endpoints require **Bearer token authentication**. A handful of non-`/api` endpoints (`/health`, `/doc*`, `/ui/*`, `/mcp*`) are public.
 
-**Endpoint:** `GET /api/shops`
-
-**Response:**
-```json
-{
-  "shops": [
-    {
-      "id": "550e8400-e29b-41d4-a716-446655440000",
-      "custom_id": "cf1a0652-d4cd-4e6f-85a0-87d203578c35",
-      "url": "https://example-shop.com",
-      "sitemap_url": "https://example-shop.com/sitemap.xml",
-      "webshop_type": "shopify",
-      "created_at": "2025-01-01T10:00:00.000Z",
-      "updated_at": "2025-01-01T12:00:00.000Z",
-      "analytics": {
-        "total_scrapes": 5,
-        "last_scraped_at": "2025-01-01T12:00:00.000Z",
-        "next_scrape_at": "2025-01-02T12:00:00.000Z",
-        "total_urls_found": 42,
-        "average_scrape_time": 5432.5,
-        "average_page_scrape_time": 234.8
-      }
-    }
-  ]
-}
+```http
+Authorization: Bearer <API_KEY>
 ```
 
-**Error Responses:**
-- `503 Service Unavailable` - Database not available
-- `500 Internal Server Error` - Server error
+The token must match the `API_KEY` environment variable. Missing or malformed `Authorization` headers return `401`. See `src/api/middleware/auth.ts`.
 
-**Example:**
 ```bash
 curl http://localhost:3000/api/shops \
-  -H "Authorization: Bearer your-secret-key"
-```
-
----
-
-### Get Shop Details
-
-Get detailed information about a specific shop including analytics, content metadata, scrape history, and scheduled jobs.
-
-**Endpoint:** `GET /api/shops/:id`
-
-**URL Parameters:**
-- `id` - Shop identifier (can use either internal UUID or custom_id)
-
-**Response:**
-```json
-{
-  "shop": {
-    "id": "550e8400-e29b-41d4-a716-446655440000",
-    "custom_id": "cf1a0652-d4cd-4e6f-85a0-87d203578c35",
-    "url": "https://example-shop.com",
-    "sitemap_url": "https://example-shop.com/sitemap.xml",
-    "webshop_type": "shopify",
-    "created_at": "2025-01-01T10:00:00.000Z",
-    "updated_at": "2025-01-01T12:00:00.000Z"
-  },
-  "analytics": {
-    "total_scrapes": 5,
-    "last_scraped_at": "2025-01-01T12:00:00.000Z",
-    "next_scrape_at": "2025-01-02T12:00:00.000Z",
-    "total_urls_found": 42,
-    "average_scrape_time": 5432.5,
-    "average_page_scrape_time": 234.8
-  },
-  "content_metadata": {
-    "shipping_informations": [
-      {
-        "url": "https://example-shop.com/shipping",
-        "changed": false,
-        "last_updated": "2025-01-01T12:00:00.000Z"
-      }
-    ],
-    "contacts": [
-      {
-        "url": "https://example-shop.com/contact",
-        "changed": true,
-        "last_updated": "2025-01-01T12:00:00.000Z"
-      }
-    ],
-    "terms_of_conditions": [],
-    "faq": []
-  },
-  "scrape_history": [
-    {
-      "id": "history-uuid-1",
-      "job_id": "job-uuid-1",
-      "started_at": "2025-01-01T12:00:00.000Z",
-      "completed_at": "2025-01-01T12:05:00.000Z",
-      "status": "completed",
-      "error": null,
-      "scrape_time_ms": 5432,
-      "urls_found": 42
-    }
-  ],
-  "scheduled_jobs": [
-    {
-      "id": "schedule-uuid-1",
-      "next_run_at": "2025-01-02T12:00:00.000Z",
-      "frequency": "daily",
-      "last_modified": "2025-01-01T10:00:00.000Z",
-      "status": "pending",
-      "enabled": true,
-      "created_at": "2025-01-01T10:00:00.000Z"
-    }
-  ]
-}
-```
-
-**Notes:**
-- This endpoint returns content metadata only (URLs, change status, last updated)
-- To get full content, use the `/api/shops/:id/results` endpoint
-- `changed` flag indicates if content has changed since the previous scrape
-
-**Error Responses:**
-- `404 Not Found` - Shop not found
-- `503 Service Unavailable` - Database not available
-- `500 Internal Server Error` - Server error
-
-**Example:**
-```bash
-curl http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000 \
-  -H "Authorization: Bearer your-secret-key"
-```
-
----
-
-### Get Shop Results
-
-Get shop scraping results with full content. Supports filtering by date range, content type, and limit.
-
-**Endpoint:** `GET /api/shops/:id/results`
-
-**URL Parameters:**
-- `id` - Shop identifier (can use either internal UUID or custom_id)
-
-**Query Parameters:**
-- `limit` (optional) - Maximum number of results to return
-- `date_from` (optional) - Filter results from this date (ISO 8601 format)
-- `date_to` (optional) - Filter results to this date (ISO 8601 format)
-- `content_type` (optional) - Filter by content type: `shipping`, `contacts`, `terms`, or `faq`
-
-**Response:**
-```json
-{
-  "shop_id": "550e8400-e29b-41d4-a716-446655440000",
-  "filters": {
-    "limit": 10,
-    "date_from": "2025-01-01",
-    "date_to": null,
-    "content_type": null
-  },
-  "results": {
-    "shipping_informations": [
-      {
-        "url": "https://example-shop.com/shipping",
-        "content": "# Shipping Information\n\nDetailed shipping content...",
-        "changed": false,
-        "last_updated": "2025-01-01T12:00:00.000Z"
-      },
-      {
-        "url": "https://example-shop.com/delivery-info",
-        "content": "# Delivery Information\n\nDetailed delivery content...",
-        "changed": true,
-        "last_updated": "2025-01-01T12:00:00.000Z"
-      }
-    ],
-    "contacts": [
-      {
-        "url": "https://example-shop.com/contact",
-        "content": "# Contact Us\n\nEmail: contact@example.com\nPhone: +1234567890",
-        "changed": true,
-        "last_updated": "2025-01-01T12:00:00.000Z"
-      }
-    ],
-    "terms_of_conditions": [
-      {
-        "url": "https://example-shop.com/terms",
-        "content": "# Terms and Conditions\n\nDetailed terms...",
-        "changed": false,
-        "last_updated": "2025-01-01T12:00:00.000Z"
-      }
-    ],
-    "faq": [
-      {
-        "url": "https://example-shop.com/faq",
-        "content": "# Frequently Asked Questions\n\nQ: Question 1?\nA: Answer 1...",
-        "changed": false,
-        "last_updated": "2025-01-01T12:00:00.000Z"
-      }
-    ]
-  }
-}
-```
-
-**Notes:**
-- All content categories use arrays to support multiple pages per category
-- `changed` flag indicates if content has changed since the previous scrape
-- Content is in Markdown format, converted from HTML
-
-**Error Responses:**
-- `404 Not Found` - Shop not found
-- `503 Service Unavailable` - Database not available
-- `500 Internal Server Error` - Server error
-
-**Examples:**
-
-Get all results:
-```bash
-curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results" \
-  -H "Authorization: Bearer your-secret-key"
-```
-
-Get limited results:
-```bash
-curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?limit=10" \
-  -H "Authorization: Bearer your-secret-key"
-```
-
-Get results from a specific date:
-```bash
-curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?date_from=2025-01-01" \
-  -H "Authorization: Bearer your-secret-key"
-```
-
-Get only shipping information:
-```bash
-curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?content_type=shipping" \
-  -H "Authorization: Bearer your-secret-key"
-```
-
-Get results with date range and limit:
-```bash
-curl "http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/results?date_from=2025-01-01&date_to=2025-01-31&limit=50" \
-  -H "Authorization: Bearer your-secret-key"
-```
-
----
-
-### Set/Update Custom ID
-
-Set or update the custom ID for a shop.
-
-**Endpoint:** `PATCH /api/shops/:id/custom-id`
-
-**URL Parameters:**
-- `id` - Shop identifier (can use either internal UUID or custom_id)
-
-**Request Body:**
-```json
-{
-  "custom_id": "cf1a0652-d4cd-4e6f-85a0-87d203578c35"
-}
-```
-
-**Fields:**
-- `custom_id` - Custom UUID identifier for the shop (set to `null` to remove)
-
-**Response:**
-```json
-{
-  "shop_id": "550e8400-e29b-41d4-a716-446655440000",
-  "custom_id": "cf1a0652-d4cd-4e6f-85a0-87d203578c35",
-  "message": "Custom ID set to cf1a0652-d4cd-4e6f-85a0-87d203578c35"
-}
-```
-
-**Error Responses:**
-- `400 Bad Request` - Invalid custom_id format
-- `404 Not Found` - Shop not found
-- `409 Conflict` - custom_id is already in use
-- `503 Service Unavailable` - Database not available
-- `500 Internal Server Error` - Server error
-
-**Example:**
-```bash
-curl -X PATCH http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/custom-id \
-  -H "Authorization: Bearer your-secret-key" \
-  -H "Content-Type: application/json" \
-  -d '{"custom_id": "cf1a0652-d4cd-4e6f-85a0-87d203578c35"}'
-```
-
----
-
-### Enable/Disable Schedule
-
-Enable or disable scheduled scraping for a shop.
-
-**Endpoint:** `PATCH /api/shops/:id/schedule`
-
-**URL Parameters:**
-- `id` - Shop identifier (can use either internal UUID or custom_id)
-
-**Request Body:**
-```json
-{
-  "enabled": true
-}
-```
-
-**Response:**
-```json
-{
-  "shop_id": "550e8400-e29b-41d4-a716-446655440000",
-  "schedule_enabled": true,
-  "message": "Schedule enabled successfully"
-}
-```
-
-**Error Responses:**
-- `400 Bad Request` - Missing or invalid `enabled` field
-- `404 Not Found` - Shop not found
-- `503 Service Unavailable` - Database not available
-- `500 Internal Server Error` - Server error
-
-**Examples:**
-
-Enable scheduling:
-```bash
-curl -X PATCH http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/schedule \
-  -H "Authorization: Bearer your-secret-key" \
-  -H "Content-Type: application/json" \
-  -d '{"enabled": true}'
-```
-
-Disable scheduling:
-```bash
-curl -X PATCH http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000/schedule \
-  -H "Authorization: Bearer your-secret-key" \
-  -H "Content-Type: application/json" \
-  -d '{"enabled": false}'
-```
-
----
-
-### Delete Shop
-
-Delete a shop and all related data (analytics, scrape history, content, scheduled jobs).
-
-**Endpoint:** `DELETE /api/shops/:id`
-
-**URL Parameters:**
-- `id` - Shop identifier (can use either internal UUID or custom_id)
-
-**Response:**
-```json
-{
-  "message": "Shop and all related data deleted successfully",
-  "shop_id": "550e8400-e29b-41d4-a716-446655440000"
-}
-```
-
-**Notes:**
-- This operation is irreversible
-- Deletes all associated data:
-  - Shop record
-  - Analytics
-  - Scrape history
-  - All content records
-  - Scheduled jobs
-
-**Error Responses:**
-- `404 Not Found` - Shop not found
-- `503 Service Unavailable` - Database not available
-- `500 Internal Server Error` - Server error
-
-**Example:**
-```bash
-curl -X DELETE http://localhost:3000/api/shops/550e8400-e29b-41d4-a716-446655440000 \
-  -H "Authorization: Bearer your-secret-key"
-```
-
----
-
-## Error Handling
-
-All endpoints return standard HTTP status codes:
-
-- `200 OK` - Request successful
-- `201 Created` - Resource created successfully
-- `400 Bad Request` - Invalid request parameters
-- `404 Not Found` - Resource not found
-- `500 Internal Server Error` - Server error
-- `503 Service Unavailable` - Service (e.g., database) unavailable
-
-Error responses include a JSON body with an `error` field:
+  -H "Authorization: Bearer $API_KEY"
+```
+
+## Endpoint surface (high-level)
+
+The registry is composed from endpoint components under `src/api/components/`. Grouped by tag:
+
+| Tag | Component | What it covers |
+|---|---|---|
+| Health | `HealthEndpoint.ts` | `/health` — public liveness + queue stats |
+| Jobs | `JobsEndpoint.ts` | Create and inspect scraping jobs |
+| Shops | `ShopsEndpoint.ts` | CRUD + analytics + schedule toggle + custom_id + Qdrant flag |
+| Content | `ContentEndpoint.ts` | Per-shop scraped content (enable/disable pages) |
+| Custom URLs | `CustomUrlsEndpoint.ts` | Manually add URLs the sitemap crawler would miss |
+| Webhooks | `WebhooksEndpoint.ts` | Per-shop webhook CRUD (scrape_started / completed / failed) |
+| Qdrant | `QdrantEndpoint.ts` | Vector DB management, per-shop embedding lifecycle, v1→v2 migration |
+| MCP | `McpEndpoint.ts` | MCP call logs and analytics |
+| Documentation | `DocumentationEndpoint.ts` | `/doc`, `/doc/openapi`, `/doc/endpoints` |
+
+Use `GET /doc/endpoints` for the exact, always-up-to-date list. A summarised snapshot at the time of writing:
+
+### Public
+- `GET /health`
+- `GET /doc`, `GET /doc/openapi`, `GET /doc/endpoints`
+- `POST /mcp`, `GET /mcp`, `DELETE /mcp`, `GET /mcp/health` *(only when MCP is enabled)*
+
+### Jobs
+- `POST /api/jobs` — create a scraping job from a URL
+- `GET /api/jobs` — list jobs with queue stats
+- `GET /api/jobs/:id` — job status and result
+
+### Shops
+- `GET /api/shops` — list with analytics
+- `GET /api/shops/deleted` — soft-deleted shops
+- `GET /api/shops/:id` — detail, supports either internal UUID or `custom_id`
+- `GET /api/shops/:id/results` — scraped content with filters
+- `PATCH /api/shops/:id/schedule` — enable/disable scheduled scraping
+- `PATCH /api/shops/:id/custom-id` — set/update custom_id
+- `PATCH /api/shops/:id/qdrant` — enable/disable embeddings (also see `PUT /api/shops/:shopId/qdrant/toggle`)
+- `DELETE /api/shops/:id` — soft delete
+- `DELETE /api/shops/:id/force` — hard delete
+
+### Shop content
+- `GET /api/shops/:id/content`
+- `PATCH /api/shops/:id/content/:contentId`
+
+### Shop custom URLs
+- `POST /api/shops/:id/custom-urls`
+- `GET /api/shops/:id/custom-urls`
+- `PATCH /api/shops/:id/custom-urls/:customUrlId`
+- `DELETE /api/shops/:id/custom-urls/:customUrlId`
+
+### Shop webhooks
+- `POST /api/shops/:id/webhooks`
+- `GET /api/shops/:id/webhooks`
+- `PATCH /api/shops/:id/webhooks`
+- `DELETE /api/shops/:id/webhooks`
+
+### Qdrant (system-wide)
+- `GET /api/qdrant/stats`
+- `GET /api/qdrant/health`
+- `GET /api/qdrant/collections`
+- `DELETE /api/qdrant/collections/:collectionName`
+- `POST /api/qdrant/cleanup`
+
+### Qdrant (per shop)
+- `GET /api/qdrant/shops/:shopId`
+- `GET /api/qdrant/shops/:shopId/embeddings`
+- `GET /api/qdrant/shops/:shopId/errors`
+- `DELETE /api/qdrant/shops/:shopId/errors`
+- `POST /api/qdrant/shops/:shopId/schedule-deletion`
+- `GET /api/shops/:shopId/qdrant` — detailed Qdrant state
+- `PUT /api/shops/:shopId/qdrant/toggle` — enable/disable
+- `PUT /api/shops/:shopId/qdrant/custom-id` — set immutable custom id for the shop
+- `POST /api/shops/:shopId/qdrant/re-embed`
+- `POST /api/shops/:shopId/qdrant/sync`
+- `GET /api/shops/:shopId/qdrant/errors`
+- `DELETE /api/shops/:shopId/qdrant/errors`
+- `DELETE /api/shops/:shopId/qdrant/pending-embeddings`
+
+### Qdrant migration (v1 legacy → v2 consolidated)
+- `GET /api/shops/:shopId/qdrant/migration-status`
+- `POST /api/shops/:shopId/qdrant/migrate`
+- `POST /api/shops/:shopId/qdrant/cleanup-old-collections`
+
+### MCP logs & analytics
+- `GET /api/mcp/stats`
+- `GET /api/mcp/analytics`
+- `GET /api/mcp/logs`
+- `GET /api/mcp/logs/:logId`
+- `DELETE /api/mcp/logs/cleanup`
+- `GET /api/mcp/tools/stats`
+- `GET /api/mcp/shops/:shopId/logs`
+- `DELETE /api/mcp/shops/:shopId/logs`
+- `GET /api/shops/:shopId/mcp/analytics`
+
+For every endpoint above, `/doc/openapi` returns the full request/response schema.
+
+## Shop identifiers
+
+Every shop endpoint path parameter `:id` accepts **either**:
+
+- the internal UUID generated by the system, or
+- the `custom_id` if one is set.
+
+The lookup in `src/database/Database.ts` tries both. `custom_id` must be a valid UUID and becomes immutable once Qdrant is enabled for the shop.
+
+## Content categories
+
+Scraped pages are classified into four buckets, used consistently across the API, the DB, and the MCP tools:
+
+- `shipping` — shipping, delivery, logistics
+- `contacts` — contact info, support
+- `terms` — terms of service, privacy, legal
+- `faq` — frequently asked questions, help pages
+
+## Error responses
+
+Errors return a JSON body:
 
 ```json
-{
-  "error": "Description of the error"
-}
+{ "error": "Description of the error" }
 ```
 
-## Content Types
+Status codes follow standard REST semantics: `400` for bad input, `401` for bad auth, `404` for missing resources, `409` for conflicts (e.g. duplicate `custom_id`), `500` for server errors, `503` when the database or Qdrant is unavailable.
 
-The scraper categorizes content into four types:
+## MCP (Model Context Protocol)
 
-1. **shipping** (shipping_informations) - Shipping, delivery, and logistics information
-2. **contacts** - Contact information (email, phone, addresses)
-3. **terms** (terms_of_conditions) - Terms of service, privacy policy, legal information
-4. **faq** - Frequently asked questions and help pages
+When both `MCP_ENABLED=true` and `QDRANT_ENABLED=true`, the server mounts a **Streamable HTTP** MCP endpoint at `/mcp` (see `src/mcp/McpServer.ts`). Five tools are registered:
 
-Each category can contain multiple pages (returned as arrays).
+| Tool name | Purpose |
+|---|---|
+| `list_contents` | Enumerate available pages for a shop, grouped by category, each tagged with the search tool to use next. Call this first. |
+| `shipping_search` | Semantic search in shipping/delivery pages |
+| `contacts_search` | Semantic search in contact/support pages |
+| `terms_search` | Semantic search in terms/policy pages |
+| `faq_search` | Semantic search in FAQ/help pages |
 
-## Scheduled Scraping
+All tools require `shop_id` (the shop's `custom_id`). The four search tools additionally take `query` (required) and `limit` (optional, 1–20, default 10). If a shop does not exist or does not have Qdrant enabled, tools return `"Search not available for this shop"` rather than erroring.
 
-The application automatically schedules scraping based on sitemap rules:
+Tool schemas are declared in `src/mcp/tools/*.ts` and exposed to MCP clients via the standard `tools/list` request.
 
-- **Frequency rules** from sitemap: `always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly`
-- **Last modified dates** from sitemap
-- Schedules can be enabled/disabled per shop
-- Scheduler checks for due jobs every minute
-- After each successful scrape, the next scrape is automatically scheduled
+## Response compression
 
-## Content Change Detection
+All responses larger than 1 KB are automatically gzip/brotli-compressed by the `compression` middleware (`src/api/server.ts`). Set header `x-no-compression: 1` to opt out.
 
-The system uses MD5 hashing to detect content changes:
+## What this document deliberately does not include
 
-- Each content piece is hashed when scraped
-- Compared with the previous version's hash
-- `changed` flag set to `true` if hashes differ
-- Change detection works for all content types
-- Useful for monitoring when shop policies or information are updated
+- Full request/response JSON schemas — they live in `/doc/openapi`, generated from the endpoint metadata. Duplicating them here guarantees drift.
+- Rate limiting — the server does not implement any; if you put one in, document it here.
+- An SDK — there isn't one. Use any HTTP client with the Bearer token.

docs/API_REFERENCE.md

@@ -1,557 +0,0 @@
-# API Reference - Qdrant & MCP Endpoints
-
-This document provides detailed API reference for the Qdrant vector search and MCP (Model Context Protocol) endpoints.
-
-## Base URL
-
-All endpoints are relative to the API base URL: `http://localhost:3000` (or your configured host/port)
-
-## Authentication
-
-All API endpoints require authentication via the `X-API-Key` header:
-
-```http
-X-API-Key: your-api-key
-```
-
-## Qdrant Management Endpoints
-
-### Get Qdrant Statistics
-
-Get system-wide Qdrant statistics and metrics.
-
-```http
-GET /api/qdrant/stats
-```
-
-**Response:**
-```json
-{
-  "data": {
-    "total_collections": 15,
-    "total_vectors": 1250,
-    "enabled_shops": 8,
-    "recent_embeddings": 45,
-    "embedding_queue_size": 3
-  },
-  "success": true
-}
-```
-
-### Health Check
-
-Check the health status of Qdrant services.
-
-```http
-GET /api/qdrant/health
-```
-
-**Response:**
-```json
-{
-  "data": {
-    "healthy": true,
-    "services": {
-      "qdrant": true,
-      "embedding": true,
-      "database": true
-    },
-    "errors": []
-  },
-  "success": true
-}
-```
-
-### Get Shop Qdrant Information
-
-Get comprehensive Qdrant information for a specific shop.
-
-```http
-GET /api/qdrant/shops/:shopId
-```
-
-**Parameters:**
-- `shopId` (string, required): Shop identifier
-
-**Response:**
-```json
-{
-  "data": {
-    "shop": {
-      "id": "shop-123",
-      "custom_id": "shop123",
-      "url": "https://example-shop.com",
-      "qdrant_enabled": true,
-      "created_at": "2023-01-01T00:00:00Z",
-      "updated_at": "2023-01-01T00:00:00Z"
-    },
-    "embeddings": [
-      {
-        "id": "emb-456",
-        "shop_content_id": "content-789",
-        "collection_name": "shop123-shipping_1",
-        "point_id": "point-uuid",
-        "embedding_status": "completed",
-        "created_at": "2023-01-01T00:00:00Z",
-        "updated_at": "2023-01-01T00:00:00Z"
-      }
-    ],
-    "errors": [
-      {
-        "id": "err-123",
-        "shop_id": "shop-123",
-        "error_type": "embedding",
-        "error_message": "Rate limit exceeded",
-        "metadata": {},
-        "created_at": "2023-01-01T00:00:00Z"
-      }
-    ],
-    "collections": [
-      {
-        "collection_name": "shop123-shipping_1",
-        "vectors_count": 125,
-        "config": {
-          "vector_size": 3072,
-          "distance": "cosine"
-        }
-      }
-    ]
-  },
-  "success": true
-}
-```
-
-### Get Shop Embeddings
-
-Get all embeddings for a specific shop.
-
-```http
-GET /api/qdrant/shops/:shopId/embeddings
-```
-
-**Parameters:**
-- `shopId` (string, required): Shop identifier
-
-**Response:**
-```json
-{
-  "data": {
-    "embeddings": [
-      {
-        "id": "emb-456",
-        "shop_content_id": "content-789",
-        "collection_name": "shop123-shipping_1",
-        "point_id": "point-uuid",
-        "embedding_status": "completed",
-        "created_at": "2023-01-01T00:00:00Z",
-        "updated_at": "2023-01-01T00:00:00Z"
-      }
-    ]
-  },
-  "success": true
-}
-```
-
-### Get Shop Qdrant Errors
-
-Get all Qdrant errors for a specific shop.
-
-```http
-GET /api/qdrant/shops/:shopId/errors
-```
-
-**Parameters:**
-- `shopId` (string, required): Shop identifier
-
-**Response:**
-```json
-{
-  "data": {
-    "errors": [
-      {
-        "id": "err-123",
-        "shop_id": "shop-123",
-        "error_type": "embedding",
-        "error_message": "OpenRouter API rate limit exceeded",
-        "metadata": {
-          "content_id": "content-789",
-          "retry_count": 3
-        },
-        "created_at": "2023-01-01T00:00:00Z"
-      }
-    ]
-  },
-  "success": true
-}
-```
-
-### Clear Shop Qdrant Errors
-
-Clear all Qdrant errors for a specific shop.
-
-```http
-DELETE /api/qdrant/shops/:shopId/errors
-```
-
-**Parameters:**
-- `shopId` (string, required): Shop identifier
-
-**Response:**
-```json
-{
-  "data": {
-    "message": "Qdrant errors cleared successfully"
-  },
-  "success": true
-}
-```
-
-### Enable/Disable Qdrant for Shop
-
-Enable or disable Qdrant functionality for a specific shop.
-
-```http
-PATCH /api/shops/:shopId/qdrant
-```
-
-**Parameters:**
-- `shopId` (string, required): Shop identifier
-
-**Request Body:**
-```json
-{
-  "enabled": true
-}
-```
-
-**Response:**
-```json
-{
-  "data": {
-    "shop_id": "shop-123",
-    "qdrant_enabled": true,
-    "message": "Qdrant settings updated successfully"
-  },
-  "success": true
-}
-```
-
-### Trigger Cleanup
-
-Manually trigger Qdrant cleanup operations.
-
-```http
-POST /api/qdrant/cleanup
-```
-
-**Response:**
-```json
-{
-  "data": {
-    "message": "Cleanup process initiated successfully"
-  },
-  "success": true
-}
-```
-
-### Delete Collection
-
-Delete a specific Qdrant collection.
-
-```http
-DELETE /api/qdrant/collections/:collectionName
-```
-
-**Parameters:**
-- `collectionName` (string, required): Name of the collection to delete
-
-**Response:**
-```json
-{
-  "data": {
-    "message": "Collection deleted successfully"
-  },
-  "success": true
-}
-```
-
-### Schedule Shop Deletion
-
-Schedule a shop for Qdrant data deletion with 24-hour delay.
-
-```http
-POST /api/qdrant/shops/:shopId/schedule-deletion
-```
-
-**Parameters:**
-- `shopId` (string, required): Shop identifier
-
-**Response:**
-```json
-{
-  "data": {
-    "message": "Shop scheduled for Qdrant deletion",
-    "scheduled_at": "2023-01-02T00:00:00Z"
-  },
-  "success": true
-}
-```
-
-## MCP (Model Context Protocol) Endpoints
-
-### Get MCP Statistics
-
-Get system-wide MCP statistics and usage metrics.
-
-```http
-GET /api/mcp/stats
-```
-
-**Response:**
-```json
-{
-  "data": {
-    "total_calls": 1250,
-    "successful_calls": 1180,
-    "failed_calls": 70,
-    "average_response_time": 325,
-    "top_tools": [
-      {
-        "tool_name": "search_shipping_info",
-        "call_count": 450
-      },
-      {
-        "tool_name": "search_faq_info",
-        "call_count": 380
-      }
-    ],
-    "recent_activity": [
-      {
-        "id": "log-123",
-        "shop_id": "shop-123",
-        "tool_name": "search_shipping_info",
-        "query": "international delivery",
-        "response_time_ms": 285,
-        "results_count": 5,
-        "success": true,
-        "error_message": null,
-        "created_at": "2023-01-01T00:00:00Z"
-      }
-    ]
-  },
-  "success": true
-}
-```
-
-### Get Shop MCP Logs
-
-Get MCP usage logs for a specific shop.
-
-```http
-GET /api/mcp/shops/:shopId/logs
-```
-
-**Parameters:**
-- `shopId` (string, required): Shop identifier
-
-**Query Parameters:**
-- `limit` (number, optional): Number of logs to return (default: 50)
-- `success` (boolean, optional): Filter by success status
-- `tool_name` (string, optional): Filter by tool name
-
-**Response:**
-```json
-{
-  "data": {
-    "logs": [
-      {
-        "id": "log-123",
-        "shop_id": "shop-123",
-        "tool_name": "search_shipping_info",
-        "query": "international delivery times",
-        "response_time_ms": 285,
-        "results_count": 5,
-        "success": true,
-        "error_message": null,
-        "created_at": "2023-01-01T00:00:00Z"
-      },
-      {
-        "id": "log-124",
-        "shop_id": "shop-123",
-        "tool_name": "search_contact_info",
-        "query": "customer service phone",
-        "response_time_ms": 412,
-        "results_count": 3,
-        "success": true,
-        "error_message": null,
-        "created_at": "2023-01-01T00:05:00Z"
-      }
-    ]
-  },
-  "success": true
-}
-```
-
-### Clear Shop MCP Logs
-
-Clear all MCP logs for a specific shop.
-
-```http
-DELETE /api/mcp/shops/:shopId/logs
-```
-
-**Parameters:**
-- `shopId` (string, required): Shop identifier
-
-**Response:**
-```json
-{
-  "data": {
-    "message": "MCP logs cleared successfully"
-  },
-  "success": true
-}
-```
-
-### Get MCP Analytics
-
-Get comprehensive MCP analytics and insights.
-
-```http
-GET /api/mcp/analytics
-```
-
-**Response:**
-```json
-{
-  "data": {
-    "total_calls": 1250,
-    "successful_calls": 1180,
-    "failed_calls": 70,
-    "average_response_time": 325,
-    "shops_using_mcp": 8,
-    "top_tools": [
-      {
-        "tool_name": "search_shipping_info",
-        "call_count": 450
-      },
-      {
-        "tool_name": "search_faq_info",
-        "call_count": 380
-      },
-      {
-        "tool_name": "search_contact_info",
-        "call_count": 240
-      },
-      {
-        "tool_name": "search_terms_info",
-        "call_count": 180
-      }
-    ],
-    "recent_activity": [
-      {
-        "id": "log-123",
-        "shop_id": "shop-123",
-        "tool_name": "search_shipping_info",
-        "query": "international delivery",
-        "response_time_ms": 285,
-        "results_count": 5,
-        "success": true,
-        "error_message": null,
-        "created_at": "2023-01-01T00:00:00Z"
-      }
-    ]
-  },
-  "success": true
-}
-```
-
-## Error Responses
-
-All endpoints may return error responses in the following format:
-
-```json
-{
-  "error": "Error message description",
-  "success": false
-}
-```
-
-### Common HTTP Status Codes
-
-- `200 OK`: Request successful
-- `400 Bad Request`: Invalid request parameters
-- `401 Unauthorized`: Missing or invalid API key
-- `404 Not Found`: Resource not found
-- `500 Internal Server Error`: Server error
-
-### Error Types
-
-#### Qdrant Errors
-- `connection`: Qdrant service unavailable
-- `embedding`: OpenRouter API issues
-- `search`: Vector search failures
-- `collection`: Collection management errors
-
-#### MCP Errors
-- `tool_not_found`: Requested tool doesn't exist
-- `invalid_parameters`: Tool called with invalid parameters
-- `search_failed`: Vector search operation failed
-- `no_embeddings`: No embeddings found for shop
-
-## Rate Limiting
-
-API endpoints are subject to rate limiting:
-
-- **General endpoints**: 100 requests per minute per API key
-- **Search endpoints**: 50 requests per minute per API key
-- **Administrative endpoints**: 20 requests per minute per API key
-
-Rate limit headers are included in responses:
-
-```http
-X-RateLimit-Limit: 100
-X-RateLimit-Remaining: 95
-X-RateLimit-Reset: 1609459200
-```
-
-## Examples
-
-### Enable Qdrant for a Shop
-
-```bash
-curl -X PATCH "http://localhost:3000/api/shops/shop-123/qdrant" \
-  -H "X-API-Key: your-api-key" \
-  -H "Content-Type: application/json" \
-  -d '{"enabled": true}'
-```
-
-### Get Shop Qdrant Status
-
-```bash
-curl -X GET "http://localhost:3000/api/qdrant/shops/shop-123" \
-  -H "X-API-Key: your-api-key"
-```
-
-### Clear Shop Errors
-
-```bash
-curl -X DELETE "http://localhost:3000/api/qdrant/shops/shop-123/errors" \
-  -H "X-API-Key: your-api-key"
-```
-
-### Get MCP Analytics
-
-```bash
-curl -X GET "http://localhost:3000/api/mcp/analytics" \
-  -H "X-API-Key: your-api-key"
-```
-
-### Trigger System Cleanup
-
-```bash
-curl -X POST "http://localhost:3000/api/qdrant/cleanup" \
-  -H "X-API-Key: your-api-key"
-```
-
-This API reference provides comprehensive access to all Qdrant and MCP functionality, enabling full management and monitoring of the vector search system.

docs/QDRANT_IMPLEMENTATION.md

@@ -1,501 +0,0 @@
-# Qdrant Vector Search Implementation Guide
-
-This document provides comprehensive information about the Qdrant vector search implementation in the webshop scraper system.
-
-## Overview
-
-The Qdrant integration enables semantic search capabilities for scraped webshop content using text embeddings. The system automatically processes scraped content through OpenRouter's text-embedding-3-large model and stores vectors in Qdrant for efficient similarity search via MCP (Model Context Protocol) tools.
-
-## Architecture
-
-### Core Components
-
-1. **QdrantService** (`src/services/QdrantService.ts`)
-   - Manages Qdrant database connections and operations
-   - Handles collection creation and management
-   - Performs vector search operations
-
-2. **EmbeddingService** (`src/services/EmbeddingService.ts`)
-   - Integrates with OpenRouter API for text embeddings
-   - Uses text-embedding-3-large model (3072 dimensions)
-   - Handles text preprocessing and batching
-
-3. **QdrantEmbeddingWorkflow** (`src/services/QdrantEmbeddingWorkflow.ts`)
-   - Orchestrates the embedding pipeline
-   - Integrates with scraping workflow
-   - Manages error handling and retries
-
-4. **QdrantCleanupService** (`src/services/QdrantCleanupService.ts`)
-   - Handles automated cleanup operations
-   - Manages 24-hour deletion delays
-   - Processes deletion queues
-
-5. **MCP Server** (`src/mcp/McpServer.ts`)
-   - Provides semantic search tools for LLMs
-   - Implements 4 category-specific search tools
-   - Handles analytics and logging
-
-6. **SchedulerService** (`src/services/SchedulerService.ts`)
-   - Background task management
-   - Health monitoring
-   - Automated maintenance
-
-## Configuration
-
-### Environment Variables
-
-```bash
-# Qdrant Configuration
-QDRANT_API_URL=http://localhost:6333
-QDRANT_API_KEY=your-qdrant-api-key  # Optional
-
-# OpenRouter Configuration (for embeddings)
-OPENROUTER_API_KEY=your-openrouter-api-key
-OPENROUTER_MODEL=openai/text-embedding-3-large
-
-# MCP Configuration
-MCP_ENABLED=true
-MCP_TRANSPORT=stdio  # or http
-MCP_PORT=3001        # for HTTP transport
-```
-
-### Database Schema Extensions
-
-The implementation extends the existing SQLite database with new tables:
-
-```sql
--- Enable Qdrant per shop
-ALTER TABLE shops ADD COLUMN qdrant_enabled BOOLEAN DEFAULT FALSE;
-
--- Track embeddings
-CREATE TABLE shop_embeddings (
-  id TEXT PRIMARY KEY,
-  shop_content_id TEXT NOT NULL,
-  collection_name TEXT NOT NULL,
-  point_id TEXT NOT NULL,
-  embedding_status TEXT NOT NULL CHECK (embedding_status IN ('pending', 'completed', 'failed')),
-  created_at TEXT NOT NULL,
-  updated_at TEXT NOT NULL,
-  FOREIGN KEY (shop_content_id) REFERENCES shop_contents (id) ON DELETE CASCADE
-);
-
--- Deletion queue with 24h delays
-CREATE TABLE qdrant_deletion_queue (
-  id TEXT PRIMARY KEY,
-  shop_id TEXT NOT NULL,
-  custom_id TEXT NOT NULL,
-  status TEXT NOT NULL DEFAULT 'pending',
-  scheduled_deletion_at TEXT NOT NULL,
-  created_at TEXT NOT NULL,
-  updated_at TEXT NOT NULL,
-  error_message TEXT,
-  FOREIGN KEY (shop_id) REFERENCES shops (id) ON DELETE CASCADE
-);
-
--- Error tracking
-CREATE TABLE qdrant_errors (
-  id TEXT PRIMARY KEY,
-  shop_id TEXT NOT NULL,
-  error_type TEXT NOT NULL,
-  error_message TEXT NOT NULL,
-  metadata TEXT,
-  created_at TEXT NOT NULL,
-  FOREIGN KEY (shop_id) REFERENCES shops (id) ON DELETE CASCADE
-);
-
--- MCP analytics
-CREATE TABLE mcp_logs (
-  id TEXT PRIMARY KEY,
-  shop_id TEXT NOT NULL,
-  tool_name TEXT NOT NULL,
-  query TEXT NOT NULL,
-  response_time_ms INTEGER NOT NULL,
-  results_count INTEGER NOT NULL,
-  success BOOLEAN NOT NULL,
-  error_message TEXT,
-  created_at TEXT NOT NULL,
-  FOREIGN KEY (shop_id) REFERENCES shops (id) ON DELETE CASCADE
-);
-```
-
-## Collection Naming Convention
-
-### Current Structure (v2)
-
-Collections follow a simplified pattern with **one collection per category**:
-
-```
-{shop_custom_id}-{category}
-```
-
-**Examples:**
-- `shop123-shipping` - All shipping information URLs for shop123
-- `shop123-contacts` - All contact information URLs for shop123
-- `shop456-faq` - All FAQ URLs for shop456
-
-Within each collection, **each URL is stored as unique points** (one per content chunk). Point IDs are deterministic based on:
-```
-SHA256(url + content_hash + chunk_index)
-```
-
-This ensures:
-- **Uniqueness**: Different URLs have different point IDs
-- **Deduplication**: Same URL with same content = same point ID (upsert behavior)
-- **Change tracking**: Same URL with changed content = new point IDs
-
-**Categories:**
-- `shipping` - Shipping and delivery information
-- `contacts` - Contact information and support
-- `terms` - Terms of service and conditions
-- `faq` - Frequently asked questions
-
-### Legacy Structure (v1)
-
-The previous implementation used URL-specific collections:
-
-```
-{shop_custom_id}-{category}_{url_hash}
-```
-
-**Examples:**
-- `shop123-shipping_a1b2c3d4` - Single URL's shipping information
-- `shop123-contacts_e5f6g7h8` - Single URL's contact information
-
-**Migration**: Use the migration API endpoints to convert from legacy to current structure.
-
-### Benefits of v2 Structure
-
-1. **Eliminates URL Duplication**: The same URL appearing in multiple categories (e.g., a help page in both FAQ and Contacts) now gets stored as separate unique points rather than creating duplicate collections
-2. **Better Organization**: One collection per category instead of dozens of URL-specific collections
-3. **Improved Search Performance**: Single collection search is more efficient than aggregating results from multiple collections
-4. **Simpler Management**: Easier to monitor, backup, and maintain consolidated collections
-5. **Reduced Storage**: Less metadata overhead with fewer collections
-6. **Backward Compatible**: System automatically handles both old and new formats during transition
-
-## MCP Tools
-
-The system provides 4 semantic search tools accessible to LLMs:
-
-### Tool Signatures
-
-```typescript
-search_shipping_info(shop_id: string, query: string, limit?: number): Promise<SearchResult[]>
-search_contact_info(shop_id: string, query: string, limit?: number): Promise<SearchResult[]>
-search_terms_info(shop_id: string, query: string, limit?: number): Promise<SearchResult[]>
-search_faq_info(shop_id: string, query: string, limit?: number): Promise<SearchResult[]>
-```
-
-### Usage Examples
-
-```javascript
-// Search for shipping policies
-const results = await search_shipping_info("shop123", "international delivery times", 5);
-
-// Find contact information
-const contacts = await search_contact_info("shop123", "customer service phone", 3);
-
-// Look up return policies
-const terms = await search_terms_info("shop123", "return policy within 30 days", 5);
-
-// Find specific FAQ answers
-const faqs = await search_faq_info("shop123", "how to track my order", 10);
-```
-
-### Response Format
-
-```typescript
-interface SearchResult {
-  id: string;
-  score: number;  // Similarity score (0-1)
-  payload: {
-    shop_id: string;
-    content_id: string;
-    content_type: string;
-    url: string;
-    title?: string;
-    content_hash: string;
-    created_at: string;
-  };
-}
-```
-
-## API Endpoints
-
-### Qdrant Management
-
-- `GET /api/qdrant/stats` - System statistics
-- `GET /api/qdrant/health` - Health check
-- `GET /api/qdrant/shops/:shopId` - Shop Qdrant info
-- `GET /api/qdrant/shops/:shopId/embeddings` - Shop embeddings
-- `GET /api/qdrant/shops/:shopId/errors` - Shop errors
-- `DELETE /api/qdrant/shops/:shopId/errors` - Clear shop errors
-- `PATCH /api/shops/:shopId/qdrant` - Enable/disable Qdrant
-- `POST /api/qdrant/cleanup` - Trigger cleanup
-- `DELETE /api/qdrant/collections/:name` - Delete collection
-- `POST /api/qdrant/shops/:shopId/schedule-deletion` - Schedule deletion
-
-### Migration Management (v1 to v2)
-
-- `GET /api/shops/:shopId/qdrant/migration-status` - Check if shop needs migration
-- `POST /api/shops/:shopId/qdrant/migrate` - Migrate shop to new collection structure
-- `POST /api/shops/:shopId/qdrant/cleanup-old-collections` - Remove old collections after migration
-
-**Migration Workflow:**
-
-1. **Check Status**: `GET /api/shops/:shopId/qdrant/migration-status`
-   ```json
-   {
-     "needsMigration": true,
-     "oldCollections": 15,
-     "newCollections": 0,
-     "categories": [
-       {
-         "category": "faq",
-         "oldCollections": 5,
-         "newCollectionExists": false
-       }
-     ]
-   }
-   ```
-
-2. **Run Migration**: `POST /api/shops/:shopId/qdrant/migrate`
-   - Copies all points from old collections to new consolidated collections
-   - Preserves all metadata and vectors
-   - Creates one collection per category
-
-3. **Verify**: Test that search works with new collections
-
-4. **Cleanup (Dry Run)**: `POST /api/shops/:shopId/qdrant/cleanup-old-collections`
-   ```json
-   { "dryRun": true }
-   ```
-   Shows what would be deleted without actually deleting
-
-5. **Cleanup (Actual)**: `POST /api/shops/:shopId/qdrant/cleanup-old-collections`
-   ```json
-   { "dryRun": false }
-   ```
-   Permanently removes old collection structure
-
-### MCP Analytics
-
-- `GET /api/mcp/stats` - MCP statistics
-- `GET /api/mcp/shops/:shopId/logs` - Shop MCP logs
-- `DELETE /api/mcp/shops/:shopId/logs` - Clear shop logs
-- `GET /api/mcp/analytics` - System analytics
-
-## Workflow Integration
-
-### Automatic Embedding Process
-
-1. **Content Scraping**: WebshopScraper extracts content
-2. **Hash Calculation**: Content hash calculated for change detection
-3. **Qdrant Check**: Verify if shop has Qdrant enabled
-4. **Embedding Generation**: OpenRouter API processes text
-5. **Vector Storage**: Qdrant stores embeddings with metadata
-6. **Database Tracking**: Record embedding status and metadata
-7. **Webhook Notification**: Success/failure notifications sent
-
-### Change Detection
-
-The system tracks content changes using SHA-256 hashes:
-
-```typescript
-function calculateContentHash(content: string): string {
-  return crypto.createHash('sha256').update(content, 'utf8').digest('hex');
-}
-```
-
-Only changed content triggers re-embedding to optimize API usage.
-
-## Frontend Integration
-
-### Shop Detail Page
-
-The shop detail page includes a comprehensive Qdrant section:
-
-- **Toggle Switch**: Enable/disable Qdrant for the shop
-- **Embedding Status**: Real-time counts of completed/pending/failed embeddings
-- **Error Display**: Recent Qdrant errors with clear button
-- **Custom ID Warning**: Alerts when custom_id is required
-
-### Qdrant Management Page
-
-System-wide Qdrant management interface:
-
-- **Health Status**: Service connectivity and status
-- **Statistics**: Collections, vectors, enabled shops counts
-- **Administrative Actions**: Cleanup operations and health checks
-- **Configuration Info**: Environment setup guidance
-
-## Security Considerations
-
-### Custom ID Protection
-
-Once Qdrant is enabled for a shop, the custom_id becomes immutable to prevent collection orphaning:
-
-```typescript
-if (shop.qdrant_enabled && shop.custom_id !== newCustomId) {
-  throw new Error('Cannot modify custom_id when Qdrant is enabled');
-}
-```
-
-### API Key Management
-
-- OpenRouter API key stored securely in environment variables
-- Qdrant API key optional for development environments
-- MCP server uses authenticated endpoints
-
-### Data Isolation
-
-- Collections are isolated by shop custom_id
-- Deletion queues prevent accidental data loss
-- 24-hour deletion delays allow recovery
-
-## Monitoring and Analytics
-
-### Health Monitoring
-
-Continuous health checks monitor:
-
-- Qdrant service connectivity
-- OpenRouter API availability
-- Embedding service status
-- Database connectivity
-
-### MCP Analytics
-
-Comprehensive analytics track:
-
-- Tool usage patterns
-- Response times
-- Success/failure rates
-- Popular search queries
-- Shop usage statistics
-
-### Error Tracking
-
-Centralized error logging captures:
-
-- Embedding failures
-- API connectivity issues
-- Vector storage problems
-- MCP tool errors
-
-## Troubleshooting
-
-### Common Issues
-
-1. **Embedding Failures**
-   - Check OpenRouter API key validity
-   - Verify API rate limits
-   - Review content length restrictions
-
-2. **Qdrant Connection Issues**
-   - Verify QDRANT_API_URL configuration
-   - Check network connectivity
-   - Validate API key if using authentication
-
-3. **Collection Not Found**
-   - Ensure shop has custom_id set
-   - Check collection naming pattern
-   - Verify Qdrant is enabled for shop
-
-4. **MCP Tool Failures**
-   - Check shop_id parameter validity
-   - Verify embeddings exist for shop
-   - Review MCP server logs
-
-### Debugging Commands
-
-```bash
-# Check Qdrant health
-curl http://localhost:6333/collections
-
-# View embedding status
-sqlite3 database.db "SELECT * FROM shop_embeddings WHERE shop_content_id = 'content-id';"
-
-# Check MCP logs
-sqlite3 database.db "SELECT * FROM mcp_logs ORDER BY created_at DESC LIMIT 10;"
-
-# Review error logs
-sqlite3 database.db "SELECT * FROM qdrant_errors ORDER BY created_at DESC;"
-```
-
-## Performance Optimization
-
-### Embedding Efficiency
-
-- Batch processing for multiple content items
-- Content change detection prevents unnecessary re-embedding
-- Configurable delays between API calls
-
-### Vector Search
-
-- Optimized similarity search with configurable limits
-- Cached collection metadata
-- Efficient metadata filtering
-
-### Background Processing
-
-- Automated cleanup scheduling
-- Asynchronous embedding pipeline
-- Health monitoring with alerts
-
-## Migration and Deployment
-
-### Development Setup
-
-1. Install Qdrant locally or use cloud service
-2. Obtain OpenRouter API key
-3. Configure environment variables
-4. Run database migrations
-5. Start MCP server
-
-### Production Deployment
-
-1. Deploy Qdrant cluster with persistence
-2. Configure API keys and authentication
-3. Set up monitoring and alerting
-4. Schedule regular cleanup operations
-5. Monitor embedding costs and usage
-
-### Backup and Recovery
-
-- Regular Qdrant collection backups
-- Database backup includes embedding metadata
-- 24-hour deletion delays enable recovery
-- Collection export/import procedures
-
-## Cost Management
-
-### OpenRouter Usage
-
-- Text preprocessing reduces token count
-- Change detection prevents duplicate embeddings
-- Configurable embedding limits per shop
-
-### Qdrant Storage
-
-- Automatic cleanup of orphaned collections
-- Compression and optimization settings
-- Storage monitoring and alerts
-
-## Future Enhancements
-
-### Planned Features
-
-1. **Multi-language Support**: Language-specific embedding models
-2. **Semantic Clustering**: Content organization by topics
-3. **Advanced Analytics**: Usage patterns and optimization
-4. **Custom Embeddings**: Shop-specific fine-tuning
-5. **Real-time Updates**: Live embedding synchronization
-
-### Integration Opportunities
-
-1. **Search API**: Public search endpoints for shops
-2. **Recommendation Engine**: Product recommendations
-3. **Content Analysis**: Automated content insights
-4. **A/B Testing**: Search relevance optimization
-
-This implementation provides a robust, scalable foundation for semantic search capabilities while maintaining data integrity and providing comprehensive monitoring and management tools.

docs/SETUP.md

@@ -0,0 +1,235 @@
+# Setup Guide
+
+This document walks through installing, configuring, and running the webshop scraper — including the optional Qdrant vector search and MCP server.
+
+## Requirements
+
+- Node.js 16 or newer
+- npm
+- SQLite (ships via `better-sqlite3`, no external server needed)
+- Linux with systemd (only if you want to install it as a service)
+- Optionally: a Qdrant instance and an OpenRouter API key for semantic search
+
+## Install
+
+```bash
+git clone <repository-url>
+cd webshop-scraper
+npm install
+```
+
+The `npm install` step will also install dependencies for the web UI under `web/`.
+
+## Configure
+
+Copy `.env.example` to `.env` and fill in values:
+
+```bash
+cp .env.example .env
+```
+
+### Environment variables
+
+All environment variables read by the code live in `src/config/index.ts`. Nothing else is consulted.
+
+| Variable | Required | Default | Purpose |
+|---|---|---|---|
+| `API_KEY` | **yes** | — | Bearer token required for every `/api/*` endpoint. Startup fails without it. |
+| `HOST` | no | `0.0.0.0` | HTTP bind address. Use `127.0.0.1` to only accept local connections. |
+| `PORT` | no | `3000` | HTTP port. |
+| `MAX_CONCURRENT_JOBS` | no | `3` | Parallel scraping worker slots. |
+| `LOG_HTTP_VERBOSE` | no | `false` | When `true`, HTTP access logs include user agent and content length. |
+| `QDRANT_ENABLED` | no | `false` | Master switch for the Qdrant integration. Must be `true` for embeddings and MCP search to work. |
+| `QDRANT_API_URL` | no | `http://localhost:6333` | Qdrant HTTP endpoint. |
+| `QDRANT_API_KEY` | no | *(empty)* | Qdrant API key; leave empty for unauthenticated local instances. |
+| `QDRANT_DELETION_DELAY_HOURS` | no | `24` | Grace period before queued Qdrant deletions actually run. |
+| `OPENROUTER_API_KEY` | no | *(empty)* | Required if `QDRANT_ENABLED=true`; used to generate embeddings. |
+| `MCP_ENABLED` | no | `false` | Mount the MCP Streamable HTTP endpoints at `/mcp`. Also requires `QDRANT_ENABLED=true`. |
+| `MCP_PORT` | no | `3001` | Reserved port (the MCP server currently mounts onto the main HTTP server under `/mcp`). |
+
+Example minimal `.env`:
+
+```bash
+API_KEY=change-me
+HOST=0.0.0.0
+PORT=3000
+MAX_CONCURRENT_JOBS=3
+```
+
+Example `.env` with Qdrant + MCP:
+
+```bash
+API_KEY=change-me
+PORT=3000
+
+QDRANT_ENABLED=true
+QDRANT_API_URL=http://localhost:6333
+QDRANT_API_KEY=
+QDRANT_DELETION_DELAY_HOURS=24
+
+OPENROUTER_API_KEY=sk-or-v1-...
+
+MCP_ENABLED=true
+```
+
+## Build
+
+```bash
+npm run build
+```
+
+This compiles TypeScript (`tsc`) and then runs `npm run build:web` to build the bundled web UI under `web/`. The web UI is served at `/ui` from the main HTTP server.
+
+## Run
+
+Ask the user before starting the server — don't start it yourself.
+
+```bash
+# Production
+npm start
+
+# Development (ts-node, no build required)
+npm run dev
+
+# TypeScript watch mode
+npm run watch
+```
+
+Once up:
+
+- API: `http://<host>:<port>`
+- Web UI: `http://<host>:<port>/ui`
+- Live API docs: `http://<host>:<port>/doc`
+- OpenAPI spec: `http://<host>:<port>/doc/openapi`
+
+## Install as a systemd service
+
+```bash
+sudo ./scripts/install.sh
+```
+
+## Reset and restore
+
+Both scripts live in `scripts/` and create timestamped backups under `/var/backups/webshop-scraper/`.
+
+```bash
+sudo ./scripts/reset.sh    # wipe back to a clean state (backs up first)
+sudo ./scripts/restore.sh  # restore from a previous backup
+```
+
+A backup contains:
+
+- `data/shops.db` (SQLite database)
+- `.env`
+- Service logs from `journalctl`
+
+## Qdrant vector search (optional)
+
+The scraper can embed scraped content and push it to a Qdrant collection so downstream systems (or the MCP tools) can do semantic search over it. Everything is gated on `QDRANT_ENABLED=true`.
+
+### 1. Run Qdrant
+
+```bash
+# Local Docker instance with persistent storage
+docker run -p 6333:6333 -v $(pwd)/qdrant_storage:/qdrant/storage qdrant/qdrant
+```
+
+Or use Qdrant Cloud and point `QDRANT_API_URL` / `QDRANT_API_KEY` at your cluster.
+
+### 2. Get an OpenRouter API key
+
+The `EmbeddingService` (`src/services/EmbeddingService.ts`) calls OpenRouter through the `openai` client. Sign up at <https://openrouter.ai>, create a key, and put it in `OPENROUTER_API_KEY`.
+
+### 3. Turn on the integration
+
+Set `QDRANT_ENABLED=true` in `.env` and restart the server.
+
+### 4. Enable Qdrant per shop
+
+Embedding happens per shop — you opt each shop in.
+
+1. Make sure the shop has a `custom_id` set (`PATCH /api/shops/:id/custom-id`). Once Qdrant is enabled for a shop, its `custom_id` becomes immutable — this prevents orphaned collections.
+2. Toggle Qdrant on via either the web UI or the API:
+   ```bash
+   curl -X PUT http://localhost:3000/api/shops/<shopId>/qdrant/toggle \
+     -H "Authorization: Bearer $API_KEY" \
+     -H "Content-Type: application/json" \
+     -d '{"enabled": true}'
+   ```
+3. The next successful scrape of that shop triggers embedding generation via `QdrantEmbeddingWorkflow`.
+
+### 5. Collection layout
+
+- One collection per `{custom_id}-{category}`, e.g. `shop123-shipping`, `shop123-faq`.
+- Categories: `shipping`, `contacts`, `terms`, `faq`.
+- Each page is stored as one or more chunks (via `ChunkingService`); point IDs are deterministic: `SHA256(url + content_hash + chunk_index)`. Same URL + same content ⇒ same point ID, so re-embedding is a no-op; content changes yield a fresh point.
+
+Verified in `src/services/QdrantEmbeddingWorkflow.ts` and `src/services/QdrantService.ts`.
+
+### 6. Legacy → current migration
+
+If you have data from a previous version that used per-URL collections (`{custom_id}-{category}_{url_hash}`), migrate using the per-shop endpoints:
+
+```bash
+# 1. See what would happen
+curl -H "Authorization: Bearer $API_KEY" \
+  http://localhost:3000/api/shops/<shopId>/qdrant/migration-status
+
+# 2. Copy points into the new consolidated collections
+curl -X POST -H "Authorization: Bearer $API_KEY" \
+  http://localhost:3000/api/shops/<shopId>/qdrant/migrate
+
+# 3. Dry-run cleanup of old collections
+curl -X POST -H "Authorization: Bearer $API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"dryRun": true}' \
+  http://localhost:3000/api/shops/<shopId>/qdrant/cleanup-old-collections
+
+# 4. Actually delete the old collections
+curl -X POST -H "Authorization: Bearer $API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"dryRun": false}' \
+  http://localhost:3000/api/shops/<shopId>/qdrant/cleanup-old-collections
+```
+
+## MCP (Model Context Protocol)
+
+When `MCP_ENABLED=true` **and** `QDRANT_ENABLED=true`, `src/api/server.ts` mounts the MCP Streamable HTTP transport at `/mcp`. Five tools are exposed; see `docs/API.md` for the list and the shapes.
+
+Point any MCP client at:
+
+```
+http://<host>:<port>/mcp
+```
+
+The server uses the stateless mode of `StreamableHTTPServerTransport`, so each request is self-contained.
+
+Quick health check:
+
+```bash
+curl http://localhost:3000/mcp/health
+```
+
+## Troubleshooting
+
+| Symptom | Where to look |
+|---|---|
+| `API_KEY environment variable is required` at startup | `.env` missing or not loaded; `API_KEY` must be set. |
+| `401` on every request | Missing `Authorization: Bearer ...` header, wrong token, or using a different header (the server only accepts Bearer). |
+| Shop exists but MCP tools return "Search not available" | Either `QDRANT_ENABLED=false` or the shop's `qdrant_enabled` flag is false or the shop has no `custom_id`. Check `GET /api/shops/:shopId/qdrant`. |
+| `qdrant_errors` piling up | Inspect via `GET /api/shops/:shopId/qdrant/errors`; the likely culprits are OpenRouter quota, wrong `OPENROUTER_API_KEY`, or unreachable `QDRANT_API_URL`. |
+| Embeddings are not being created after scrape | Check the shop is toggled on (`PUT /api/shops/:shopId/qdrant/toggle`) and re-run with `POST /api/shops/:shopId/qdrant/re-embed` or `/sync`. |
+
+For logs:
+
+```bash
+# systemd
+sudo journalctl -u webshop-scraper -f
+
+# development
+npm run dev
+```
+
+## Database
+
+SQLite file at `data/shops.db`. Schema lives in `src/database/Database.ts` and is applied on startup — no manual migration step. Qdrant-related tables (`shop_embeddings`, `qdrant_deletion_queue`, `qdrant_errors`, `mcp_logs`) are created automatically whether or not Qdrant is enabled.

docs/SETUP_GUIDE.md

@@ -1,340 +0,0 @@
-# Qdrant Vector Search Setup Guide
-
-This guide provides step-by-step instructions to set up and configure the Qdrant vector search integration.
-
-## Prerequisites
-
-- Node.js 18+
-- SQLite database (existing webshop scraper database)
-- OpenRouter API account
-- Qdrant instance (local or cloud)
-
-## Quick Setup
-
-### 1. Install Dependencies
-
-The required dependencies are already included in `package.json`:
-
-```bash
-npm install
-```
-
-**New Dependencies Added:**
-- `@qdrant/js-client-rest` - Qdrant JavaScript client
-- `openai` - OpenRouter API client (OpenAI-compatible)
-- `@modelcontextprotocol/sdk` - MCP server implementation
-
-### 2. Environment Configuration
-
-Create or update your `.env` file with Qdrant and OpenRouter settings:
-
-```bash
-# Existing configuration
-API_KEY=your-existing-api-key
-PORT=3000
-
-# Qdrant Configuration
-QDRANT_API_URL=http://localhost:6333
-QDRANT_API_KEY=                    # Optional for local development
-
-# OpenRouter Configuration (for embeddings)
-OPENROUTER_API_KEY=your-openrouter-api-key
-OPENROUTER_MODEL=openai/text-embedding-3-large
-
-# MCP Configuration
-MCP_ENABLED=true
-MCP_TRANSPORT=stdio                # or 'http'
-MCP_PORT=3001                      # for HTTP transport only
-```
-
-### 3. Qdrant Setup
-
-#### Option A: Local Docker Instance
-
-```bash
-# Pull and run Qdrant
-docker run -p 6333:6333 qdrant/qdrant
-
-# Or with persistence
-docker run -p 6333:6333 -v $(pwd)/qdrant_storage:/qdrant/storage qdrant/qdrant
-```
-
-#### Option B: Qdrant Cloud
-
-1. Sign up at [Qdrant Cloud](https://cloud.qdrant.io/)
-2. Create a cluster
-3. Get your API URL and key
-4. Update environment variables:
-
-```bash
-QDRANT_API_URL=https://your-cluster.qdrant.io
-QDRANT_API_KEY=your-api-key
-```
-
-### 4. OpenRouter API Setup
-
-1. Sign up at [OpenRouter](https://openrouter.ai/)
-2. Create an API key
-3. Add to environment variables:
-
-```bash
-OPENROUTER_API_KEY=sk-or-v1-your-api-key
-```
-
-**Note:** The system uses the `openai/text-embedding-3-large` model which provides 3072-dimensional embeddings optimized for semantic search.
-
-### 5. Database Migrations
-
-The database schema extensions are automatically applied when the application starts. No manual migration is required.
-
-**New Tables Created:**
-- `shop_embeddings` - Embedding tracking
-- `qdrant_deletion_queue` - Deletion management
-- `qdrant_errors` - Error logging
-- `mcp_logs` - MCP analytics
-
-### 6. Start the Application
-
-```bash
-# Development mode
-npm run dev
-
-# Production mode
-npm start
-```
-
-**Services Started:**
-- Main API server (port 3000)
-- MCP server (stdio transport by default)
-- Background scheduler for cleanup and monitoring
-
-## Configuration Options
-
-### Qdrant Settings
-
-```typescript
-// Default configuration in src/config/index.ts
-qdrant: {
-  apiUrl: process.env.QDRANT_API_URL || 'http://localhost:6333',
-  apiKey: process.env.QDRANT_API_KEY,
-  vectorSize: 3072,
-  distance: 'cosine'
-}
-```
-
-### Embedding Settings
-
-```typescript
-// OpenRouter configuration
-openRouter: {
-  apiKey: process.env.OPENROUTER_API_KEY!,
-  model: process.env.OPENROUTER_MODEL || 'openai/text-embedding-3-large',
-  maxTokens: 8192,
-  batchSize: 10
-}
-```
-
-### MCP Settings
-
-```typescript
-// MCP server configuration
-mcp: {
-  enabled: process.env.MCP_ENABLED === 'true',
-  transport: process.env.MCP_TRANSPORT as 'stdio' | 'http',
-  port: parseInt(process.env.MCP_PORT || '3001'),
-  maxResults: 50
-}
-```
-
-## Verification Steps
-
-### 1. Check Service Health
-
-```bash
-# Check main application
-curl http://localhost:3000/health
-
-# Check Qdrant
-curl http://localhost:6333/collections
-```
-
-### 2. Verify API Endpoints
-
-```bash
-# Check Qdrant health via API
-curl -H "X-API-Key: your-api-key" http://localhost:3000/api/qdrant/health
-
-# Get Qdrant statistics
-curl -H "X-API-Key: your-api-key" http://localhost:3000/api/qdrant/stats
-```
-
-### 3. Test MCP Tools
-
-The MCP server provides 4 semantic search tools:
-- `search_shipping_info`
-- `search_contact_info`
-- `search_terms_info`
-- `search_faq_info`
-
-MCP tools are accessed via the stdio transport by LLM applications.
-
-## Usage Workflow
-
-### 1. Enable Qdrant for a Shop
-
-1. Navigate to the shop detail page in the web interface
-2. Ensure the shop has a custom_id set
-3. Toggle the Qdrant switch to "Enabled"
-4. The system will automatically start embedding content during the next scrape
-
-### 2. Monitor Embedding Progress
-
-- View embedding status on the shop detail page
-- Check the Qdrant management page for system-wide statistics
-- Review error logs if embedding fails
-
-### 3. Use Semantic Search
-
-Once embeddings are created, LLM applications can use MCP tools:
-
-```javascript
-// Example: Search for shipping information
-const results = await search_shipping_info("shop123", "international delivery", 5);
-```
-
-## Troubleshooting
-
-### Common Issues
-
-1. **"Qdrant service unavailable"**
-   - Verify Qdrant is running: `curl http://localhost:6333/collections`
-   - Check QDRANT_API_URL in environment variables
-   - Ensure network connectivity
-
-2. **"OpenRouter API key invalid"**
-   - Verify API key format: `sk-or-v1-...`
-   - Check account balance and rate limits
-   - Test API key: `curl -H "Authorization: Bearer $OPENROUTER_API_KEY" https://openrouter.ai/api/v1/models`
-
-3. **"Custom ID required"**
-   - Set a custom_id for the shop before enabling Qdrant
-   - Custom ID cannot be changed once Qdrant is enabled
-   - Use a descriptive, unique identifier
-
-4. **"Embedding failed"**
-   - Check content length (max ~8K tokens)
-   - Verify OpenRouter API rate limits
-   - Review error logs in the UI or database
-
-### Debug Commands
-
-```bash
-# Check application logs
-docker logs webshop-scraper
-
-# Query embedding status
-sqlite3 database.db "SELECT * FROM shop_embeddings ORDER BY created_at DESC LIMIT 10;"
-
-# Check for errors
-sqlite3 database.db "SELECT * FROM qdrant_errors ORDER BY created_at DESC LIMIT 10;"
-
-# View MCP usage
-sqlite3 database.db "SELECT * FROM mcp_logs ORDER BY created_at DESC LIMIT 10;"
-```
-
-### Log Locations
-
-- Application logs: Console output
-- Database errors: `qdrant_errors` table
-- MCP analytics: `mcp_logs` table
-- Webhook deliveries: `webhook_deliveries` table
-
-## Performance Tuning
-
-### Embedding Optimization
-
-```typescript
-// Adjust in src/services/EmbeddingService.ts
-const config = {
-  batchSize: 5,           // Process N items at once
-  delayMs: 100,          // Delay between API calls
-  maxRetries: 3,         // Retry failed embeddings
-  maxTokens: 8192        // Content truncation limit
-};
-```
-
-### Qdrant Optimization
-
-```typescript
-// Collection settings in src/services/QdrantService.ts
-const collectionConfig = {
-  vectors: {
-    size: 3072,
-    distance: 'Cosine'
-  },
-  optimizers_config: {
-    default_segment_number: 2
-  },
-  replication_factor: 1
-};
-```
-
-### Background Tasks
-
-```typescript
-// Scheduler intervals in src/services/SchedulerService.ts
-const intervals = {
-  cleanup: 24 * 60 * 60 * 1000,      // 24 hours
-  healthCheck: 5 * 60 * 1000,        // 5 minutes
-  deletionQueue: 60 * 60 * 1000,     // 1 hour
-  logCleanup: 7 * 24 * 60 * 60 * 1000 // 7 days
-};
-```
-
-## Security Considerations
-
-### API Keys
-
-- Store API keys in environment variables only
-- Never commit API keys to version control
-- Use different keys for development and production
-- Monitor API usage and costs
-
-### Data Protection
-
-- Qdrant collections are isolated by shop custom_id
-- 24-hour deletion delays prevent accidental data loss
-- Custom_id becomes immutable when Qdrant is enabled
-- Error logs contain no sensitive information
-
-### Network Security
-
-- Use HTTPS in production
-- Configure Qdrant authentication for production
-- Limit API access by IP if required
-- Monitor for unusual usage patterns
-
-## Cost Management
-
-### OpenRouter Costs
-
-- Text embedding costs: ~$0.00002 per 1K tokens
-- Average content size: 2-4K tokens per page
-- Typical cost: $0.04-$0.16 per 1K pages
-- Change detection prevents re-embedding unchanged content
-
-### Qdrant Storage
-
-- Local instance: Free (your storage costs)
-- Qdrant Cloud: Varies by cluster size and usage
-- Regular cleanup removes orphaned collections
-- Monitor storage usage in production
-
-### Monitoring
-
-- Track embedding counts and costs
-- Set up alerts for unusual API usage
-- Monitor Qdrant storage growth
-- Regular cleanup schedules
-
-This setup guide provides everything needed to get the Qdrant vector search integration running successfully. The system is designed to be robust and production-ready while maintaining ease of setup and management.