# API Overview

This project ships a **self-documenting REST API**. Each endpoint carries its own metadata (parameters, request body, responses, auth requirement) and an endpoint registry generates both browsable HTML and an OpenAPI 3.0 spec at runtime.

**Always treat the running server as the source of truth.** This file covers the concepts — auth, conventions, shape of the surface — and points you at the live reference for the full endpoint list.

## Live reference

Once the server is running, three documentation endpoints are available (no auth required):

| Endpoint | Returns |
|---|---|
| `GET /doc` | Interactive HTML documentation page |
| `GET /doc/openapi` | OpenAPI 3.0 JSON specification |
| `GET /doc/endpoints` | Compact JSON list of `{ method, path, summary, tags, requiresAuth }` |

```bash
# Browse in a browser:
open http://localhost:3000/doc

# Pipe the spec into anything OpenAPI-aware:
curl http://localhost:3000/doc/openapi | jq .
```

## Authentication

All `/api/*` endpoints require **Bearer token authentication**. A handful of non-`/api` endpoints (`/health`, `/doc*`, `/ui/*`, `/mcp*`) are public.

```http
Authorization: Bearer <API_KEY>
```

The token must match the `API_KEY` environment variable. Missing or malformed `Authorization` headers return `401`. See `src/api/middleware/auth.ts`.

```bash
curl http://localhost:3000/api/sites \
  -H "Authorization: Bearer $API_KEY"
```

## Endpoint surface (high-level)

The registry is composed from endpoint components under `src/api/components/`. Grouped by tag:

| Tag | Component | What it covers |
|---|---|---|
| Health | `HealthEndpoint.ts` | `/health` — public liveness + queue stats |
| Jobs | `JobsEndpoint.ts` | Create and inspect scraping jobs |
| Sites | `SitesEndpoint.ts` | CRUD + analytics + schedule toggle + custom_id + Qdrant flag |
| Content | `ContentEndpoint.ts` | Per-site scraped content (enable/disable pages) |
| Custom URLs | `CustomUrlsEndpoint.ts` | Manually add URLs the sitemap crawler would miss |
| Webhooks | `WebhooksEndpoint.ts` | Per-site webhook CRUD (scrape_started / completed / failed) |
| Search | `SearchEndpoint.ts` | Semantic search over scraped content (REST wrapper over Qdrant) |
| Qdrant | `QdrantEndpoint.ts` | Vector DB management, per-site embedding lifecycle, v1→v2 migration |
| MCP | `McpEndpoint.ts` | MCP call logs and analytics |
| Documentation | `DocumentationEndpoint.ts` | `/doc`, `/doc/openapi`, `/doc/endpoints` |

Use `GET /doc/endpoints` for the exact, always-up-to-date list. A summarised snapshot at the time of writing:

### Public
- `GET /health`
- `GET /doc`, `GET /doc/openapi`, `GET /doc/endpoints`
- `POST /mcp`, `GET /mcp`, `DELETE /mcp`, `GET /mcp/health` *(only when MCP is enabled)*

### Jobs
- `POST /api/jobs` — create a scraping job from a URL
- `GET /api/jobs` — list jobs with queue stats
- `GET /api/jobs/:id` — job status and result

### Sites
- `GET /api/sites` — list with analytics
- `GET /api/sites/deleted` — soft-deleted sites
- `GET /api/sites/:id` — detail, supports either internal UUID or `custom_id`
- `GET /api/sites/:id/results` — scraped content with filters
- `PATCH /api/sites/:id/schedule` — enable/disable scheduled scraping
- `PATCH /api/sites/:id/custom-id` — set/update custom_id
- `PATCH /api/sites/:id/qdrant` — enable/disable embeddings (also see `PUT /api/sites/:siteId/qdrant/toggle`)
- `DELETE /api/sites/:id` — soft delete
- `DELETE /api/sites/:id/force` — hard delete

### Site content
- `GET /api/sites/:id/content`
- `PATCH /api/sites/:id/content/:contentId`

### Site custom URLs
- `POST /api/sites/:id/custom-urls`
- `GET /api/sites/:id/custom-urls`
- `PATCH /api/sites/:id/custom-urls/:customUrlId`
- `DELETE /api/sites/:id/custom-urls/:customUrlId`

### Site webhooks
- `POST /api/sites/:id/webhooks`
- `GET /api/sites/:id/webhooks`
- `PATCH /api/sites/:id/webhooks`
- `DELETE /api/sites/:id/webhooks`

### Semantic search
- `POST /api/sites/:id/search` — see [Semantic search](#semantic-search) below

### Qdrant (system-wide)
- `GET /api/qdrant/stats`
- `GET /api/qdrant/health`
- `GET /api/qdrant/collections`
- `DELETE /api/qdrant/collections/:collectionName`
- `POST /api/qdrant/cleanup`

### Qdrant (per site)
- `GET /api/qdrant/sites/:siteId`
- `GET /api/qdrant/sites/:siteId/embeddings`
- `GET /api/qdrant/sites/:siteId/errors`
- `DELETE /api/qdrant/sites/:siteId/errors`
- `POST /api/qdrant/sites/:siteId/schedule-deletion`
- `GET /api/sites/:siteId/qdrant` — detailed Qdrant state
- `PUT /api/sites/:siteId/qdrant/toggle` — enable/disable
- `PUT /api/sites/:siteId/qdrant/custom-id` — set immutable custom id for the site
- `POST /api/sites/:siteId/qdrant/re-embed`
- `POST /api/sites/:siteId/qdrant/sync`
- `GET /api/sites/:siteId/qdrant/errors`
- `DELETE /api/sites/:siteId/qdrant/errors`
- `DELETE /api/sites/:siteId/qdrant/pending-embeddings`

### Qdrant migration (v1 legacy → v2 consolidated)
- `GET /api/sites/:siteId/qdrant/migration-status`
- `POST /api/sites/:siteId/qdrant/migrate`
- `POST /api/sites/:siteId/qdrant/cleanup-old-collections`

### MCP logs & analytics
- `GET /api/mcp/stats`
- `GET /api/mcp/analytics`
- `GET /api/mcp/logs`
- `GET /api/mcp/logs/:logId`
- `DELETE /api/mcp/logs/cleanup`
- `GET /api/mcp/tools/stats`
- `GET /api/mcp/sites/:siteId/logs`
- `DELETE /api/mcp/sites/:siteId/logs`
- `GET /api/sites/:siteId/mcp/analytics`

For every endpoint above, `/doc/openapi` returns the full request/response schema.

## Site identifiers

Every site endpoint path parameter `:id` accepts **either**:

- the internal UUID generated by the system, or
- the `custom_id` if one is set.

The lookup in `src/database/SiteDatabase.ts` tries both. `custom_id` must be a valid UUID and becomes immutable once Qdrant is enabled for the site.

## Content categories

Scraped pages are classified into four buckets, used consistently across the API, the DB, and the MCP tools:

- `shipping` — shipping, delivery, logistics
- `contacts` — contact info, support
- `terms` — terms of service, privacy, legal
- `faq` — frequently asked questions, help pages

## Error responses

Errors return a JSON body:

```json
{ "error": "Description of the error" }
```

Status codes follow standard REST semantics: `400` for bad input, `401` for bad auth, `404` for missing resources, `409` for conflicts (e.g. duplicate `custom_id`), `500` for server errors, `503` when the database or Qdrant is unavailable.

## Semantic search

`POST /api/sites/:id/search` is a thin REST wrapper over the Qdrant vector search that powers the MCP tools. Use it when you want semantic (sentence/keyword) search from a plain HTTP client without speaking MCP.

**Preconditions** (enforced by the endpoint):

- `QDRANT_ENABLED=true` on the server and `OPENROUTER_API_KEY` is set.
- The site has a `custom_id` and is toggled on via `PUT /api/sites/:id/qdrant/toggle`.
- The site has been scraped at least once so embeddings exist.

**Path parameter**

- `:id` — either the internal UUID or the site's `custom_id`.

**Query string**

- `format=json` *(default)* — structured JSON.
- `format=text` — plain text, one block per result, suited for voice/LLM pipelines.

**Request body**

```json
{
  "q": "international delivery times",
  "category": "shipping",
  "limit": 5
}
```

| Field | Required | Description |
|---|---|---|
| `q` | yes | Natural-language query. |
| `category` | no | One of `shipping`, `contacts`, `terms`, `faq`. Omit to search all four in parallel and get a merged, score-ranked list. |
| `limit` | no | 1–20, default 10. Applies to the final merged list. |

**Example — search a single category:**

```bash
curl -X POST http://localhost:3000/api/sites/shop123/search \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q":"express shipping to Germany","category":"shipping","limit":5}'
```

**Example — search everything, text output:**

```bash
curl -X POST "http://localhost:3000/api/sites/shop123/search?format=text" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q":"how do I return a defective product"}'
```

**JSON response shape:**

```json
{
  "site_id": "shop123",
  "query": "express shipping to Germany",
  "category": "shipping",
  "total": 3,
  "results": [
    {
      "score": 0.87,
      "content_type": "shipping",
      "url": "https://example-shop.com/shipping",
      "title": "Shipping policy",
      "content_id": "…",
      "snippet": "We offer express delivery to Germany within 2-3 business days…"
    }
  ]
}
```

**Text response shape:**

```
Shipping policy
We offer express delivery to Germany within 2-3 business days…

Delivery FAQ
For express shipments, tracking numbers are issued within…
```

**Status codes**

| Code | Meaning |
|---|---|
| `200` | Results (possibly empty). |
| `400` | Missing `q`, invalid `category`, or `limit` out of range. |
| `404` | Site not found. |
| `409` | Site has no `custom_id`, or `qdrant_enabled` is false on the site. |
| `503` | Qdrant disabled server-wide, or embedding service not configured. |

**Logging**

Each REST search is written to `mcp_logs` with `tool_name: "rest_search"` so it shows up alongside MCP tool calls in the analytics endpoints — useful for comparing traffic sources.

## MCP (Model Context Protocol)

When both `MCP_ENABLED=true` and `QDRANT_ENABLED=true`, the server mounts a **Streamable HTTP** MCP endpoint at `/mcp` (see `src/mcp/McpServer.ts`). Five tools are registered:

| Tool name | Purpose |
|---|---|
| `list_contents` | Enumerate available pages for a site, grouped by category, each tagged with the search tool to use next. Call this first. |
| `shipping_search` | Semantic search in shipping/delivery pages |
| `contacts_search` | Semantic search in contact/support pages |
| `terms_search` | Semantic search in terms/policy pages |
| `faq_search` | Semantic search in FAQ/help pages |

All tools accept both `site_id` and `shop_id` (legacy alias) as the parameter name, mapped to the site's `custom_id`. The four search tools additionally take `query` (required) and `limit` (optional, 1–20, default 10). If a site does not exist or does not have Qdrant enabled, tools return `"Search not available for this site"` rather than erroring.

Tool schemas are declared in `src/mcp/tools/*.ts` and exposed to MCP clients via the standard `tools/list` request.

## Response compression

All responses larger than 1 KB are automatically gzip/brotli-compressed by the `compression` middleware (`src/api/server.ts`). Set header `x-no-compression: 1` to opt out.

## Backward compatibility

All `/api/sites/*` endpoints are also available under their previous `/api/shops/*` paths. Legacy routes behave identically but return responses with the old key names (`shop_id`, `webshop_type`, etc.) so existing consumers continue working without changes.

- New code should use `/api/sites/*`.
- Existing consumers using `/api/shops/*` continue working — no migration required.
- MCP tools accept both `site_id` and `shop_id` as the identifier parameter name.

The legacy routes will be maintained for the foreseeable future. When they are eventually deprecated, a deprecation notice will be added here and in the `/doc` output first.

## What this document deliberately does not include

- Full request/response JSON schemas — they live in `/doc/openapi`, generated from the endpoint metadata. Duplicating them here guarantees drift.
- Rate limiting — the server does not implement any; if you put one in, document it here.
- An SDK — there isn't one. Use any HTTP client with the Bearer token.