# API Overview

This project ships a **self-documenting REST API**. Each endpoint carries its own metadata (parameters, request body, responses, auth requirement) and an endpoint registry generates both browsable HTML and an OpenAPI 3.0 spec at runtime.

**Always treat the running server as the source of truth.** This file covers the concepts — auth, conventions, shape of the surface — and points you at the live reference for the full endpoint list.

## Live reference

Once the server is running, three documentation endpoints are available (no auth required):

| Endpoint | Returns |
|---|---|
| `GET /doc` | Interactive HTML documentation page |
| `GET /doc/openapi` | OpenAPI 3.0 JSON specification |
| `GET /doc/endpoints` | Compact JSON list of `{ method, path, summary, tags, requiresAuth }` |

```bash
# Browse in a browser:
open http://localhost:3000/doc

# Pipe the spec into anything OpenAPI-aware:
curl http://localhost:3000/doc/openapi | jq .
```

## Authentication

All `/api/*` endpoints require **Bearer token authentication**. A handful of non-`/api` endpoints (`/health`, `/doc*`, `/ui/*`, `/mcp*`) are public.

```http
Authorization: Bearer <API_KEY>
```

The token must match the `API_KEY` environment variable. Missing or malformed `Authorization` headers return `401`. See `src/api/middleware/auth.ts`.

```bash
curl http://localhost:3000/api/shops \
  -H "Authorization: Bearer $API_KEY"
```

## Endpoint surface (high-level)

The registry is composed from endpoint components under `src/api/components/`. Grouped by tag:

| Tag | Component | What it covers |
|---|---|---|
| Health | `HealthEndpoint.ts` | `/health` — public liveness + queue stats |
| Jobs | `JobsEndpoint.ts` | Create and inspect scraping jobs |
| Shops | `ShopsEndpoint.ts` | CRUD + analytics + schedule toggle + custom_id + Qdrant flag |
| Content | `ContentEndpoint.ts` | Per-shop scraped content (enable/disable pages) |
| Custom URLs | `CustomUrlsEndpoint.ts` | Manually add URLs the sitemap crawler would miss |
| Webhooks | `WebhooksEndpoint.ts` | Per-shop webhook CRUD (scrape_started / completed / failed) |
| Qdrant | `QdrantEndpoint.ts` | Vector DB management, per-shop embedding lifecycle, v1→v2 migration |
| MCP | `McpEndpoint.ts` | MCP call logs and analytics |
| Documentation | `DocumentationEndpoint.ts` | `/doc`, `/doc/openapi`, `/doc/endpoints` |

Use `GET /doc/endpoints` for the exact, always-up-to-date list. A summarised snapshot at the time of writing:

### Public
- `GET /health`
- `GET /doc`, `GET /doc/openapi`, `GET /doc/endpoints`
- `POST /mcp`, `GET /mcp`, `DELETE /mcp`, `GET /mcp/health` *(only when MCP is enabled)*

### Jobs
- `POST /api/jobs` — create a scraping job from a URL
- `GET /api/jobs` — list jobs with queue stats
- `GET /api/jobs/:id` — job status and result

### Shops
- `GET /api/shops` — list with analytics
- `GET /api/shops/deleted` — soft-deleted shops
- `GET /api/shops/:id` — detail, supports either internal UUID or `custom_id`
- `GET /api/shops/:id/results` — scraped content with filters
- `PATCH /api/shops/:id/schedule` — enable/disable scheduled scraping
- `PATCH /api/shops/:id/custom-id` — set/update custom_id
- `PATCH /api/shops/:id/qdrant` — enable/disable embeddings (also see `PUT /api/shops/:shopId/qdrant/toggle`)
- `DELETE /api/shops/:id` — soft delete
- `DELETE /api/shops/:id/force` — hard delete

### Shop content
- `GET /api/shops/:id/content`
- `PATCH /api/shops/:id/content/:contentId`

### Shop custom URLs
- `POST /api/shops/:id/custom-urls`
- `GET /api/shops/:id/custom-urls`
- `PATCH /api/shops/:id/custom-urls/:customUrlId`
- `DELETE /api/shops/:id/custom-urls/:customUrlId`

### Shop webhooks
- `POST /api/shops/:id/webhooks`
- `GET /api/shops/:id/webhooks`
- `PATCH /api/shops/:id/webhooks`
- `DELETE /api/shops/:id/webhooks`

### Qdrant (system-wide)
- `GET /api/qdrant/stats`
- `GET /api/qdrant/health`
- `GET /api/qdrant/collections`
- `DELETE /api/qdrant/collections/:collectionName`
- `POST /api/qdrant/cleanup`

### Qdrant (per shop)
- `GET /api/qdrant/shops/:shopId`
- `GET /api/qdrant/shops/:shopId/embeddings`
- `GET /api/qdrant/shops/:shopId/errors`
- `DELETE /api/qdrant/shops/:shopId/errors`
- `POST /api/qdrant/shops/:shopId/schedule-deletion`
- `GET /api/shops/:shopId/qdrant` — detailed Qdrant state
- `PUT /api/shops/:shopId/qdrant/toggle` — enable/disable
- `PUT /api/shops/:shopId/qdrant/custom-id` — set immutable custom id for the shop
- `POST /api/shops/:shopId/qdrant/re-embed`
- `POST /api/shops/:shopId/qdrant/sync`
- `GET /api/shops/:shopId/qdrant/errors`
- `DELETE /api/shops/:shopId/qdrant/errors`
- `DELETE /api/shops/:shopId/qdrant/pending-embeddings`

### Qdrant migration (v1 legacy → v2 consolidated)
- `GET /api/shops/:shopId/qdrant/migration-status`
- `POST /api/shops/:shopId/qdrant/migrate`
- `POST /api/shops/:shopId/qdrant/cleanup-old-collections`

### MCP logs & analytics
- `GET /api/mcp/stats`
- `GET /api/mcp/analytics`
- `GET /api/mcp/logs`
- `GET /api/mcp/logs/:logId`
- `DELETE /api/mcp/logs/cleanup`
- `GET /api/mcp/tools/stats`
- `GET /api/mcp/shops/:shopId/logs`
- `DELETE /api/mcp/shops/:shopId/logs`
- `GET /api/shops/:shopId/mcp/analytics`

For every endpoint above, `/doc/openapi` returns the full request/response schema.

## Shop identifiers

Every shop endpoint path parameter `:id` accepts **either**:

- the internal UUID generated by the system, or
- the `custom_id` if one is set.

The lookup in `src/database/Database.ts` tries both. `custom_id` must be a valid UUID and becomes immutable once Qdrant is enabled for the shop.

## Content categories

Scraped pages are classified into four buckets, used consistently across the API, the DB, and the MCP tools:

- `shipping` — shipping, delivery, logistics
- `contacts` — contact info, support
- `terms` — terms of service, privacy, legal
- `faq` — frequently asked questions, help pages

## Error responses

Errors return a JSON body:

```json
{ "error": "Description of the error" }
```

Status codes follow standard REST semantics: `400` for bad input, `401` for bad auth, `404` for missing resources, `409` for conflicts (e.g. duplicate `custom_id`), `500` for server errors, `503` when the database or Qdrant is unavailable.

## MCP (Model Context Protocol)

When both `MCP_ENABLED=true` and `QDRANT_ENABLED=true`, the server mounts a **Streamable HTTP** MCP endpoint at `/mcp` (see `src/mcp/McpServer.ts`). Five tools are registered:

| Tool name | Purpose |
|---|---|
| `list_contents` | Enumerate available pages for a shop, grouped by category, each tagged with the search tool to use next. Call this first. |
| `shipping_search` | Semantic search in shipping/delivery pages |
| `contacts_search` | Semantic search in contact/support pages |
| `terms_search` | Semantic search in terms/policy pages |
| `faq_search` | Semantic search in FAQ/help pages |

All tools require `shop_id` (the shop's `custom_id`). The four search tools additionally take `query` (required) and `limit` (optional, 1–20, default 10). If a shop does not exist or does not have Qdrant enabled, tools return `"Search not available for this shop"` rather than erroring.

Tool schemas are declared in `src/mcp/tools/*.ts` and exposed to MCP clients via the standard `tools/list` request.

## Response compression

All responses larger than 1 KB are automatically gzip/brotli-compressed by the `compression` middleware (`src/api/server.ts`). Set header `x-no-compression: 1` to opt out.

## What this document deliberately does not include

- Full request/response JSON schemas — they live in `/doc/openapi`, generated from the endpoint metadata. Duplicating them here guarantees drift.
- Rate limiting — the server does not implement any; if you put one in, document it here.
- An SDK — there isn't one. Use any HTTP client with the Bearer token.