API Overview

This project ships a self-documenting REST API. Each endpoint carries its own metadata (parameters, request body, responses, auth requirement) and an endpoint registry generates both browsable HTML and an OpenAPI 3.0 spec at runtime.

Always treat the running server as the source of truth. This file covers the concepts — auth, conventions, shape of the surface — and points you at the live reference for the full endpoint list.

Live reference

Once the server is running, three documentation endpoints are available (no auth required):

Endpoint	Returns
`GET /doc`	Interactive HTML documentation page
`GET /doc/openapi`	OpenAPI 3.0 JSON specification
`GET /doc/endpoints`	Compact JSON list of `{ method, path, summary, tags, requiresAuth }`

# Browse in a browser:
open http://localhost:3000/doc

# Pipe the spec into anything OpenAPI-aware:
curl http://localhost:3000/doc/openapi | jq .

Authentication

All /api/* endpoints require Bearer token authentication. A handful of non-/api endpoints (/health, /doc*, /ui/*, /mcp*) are public.

Authorization: Bearer <API_KEY>

The token must match the API_KEY environment variable. Missing or malformed Authorization headers return 401. See src/api/middleware/auth.ts.

curl http://localhost:3000/api/shops \
  -H "Authorization: Bearer $API_KEY"

Endpoint surface (high-level)

The registry is composed from endpoint components under src/api/components/. Grouped by tag:

Tag	Component	What it covers
Health	`HealthEndpoint.ts`	`/health` — public liveness + queue stats
Jobs	`JobsEndpoint.ts`	Create and inspect scraping jobs
Shops	`ShopsEndpoint.ts`	CRUD + analytics + schedule toggle + custom_id + Qdrant flag
Content	`ContentEndpoint.ts`	Per-shop scraped content (enable/disable pages)
Custom URLs	`CustomUrlsEndpoint.ts`	Manually add URLs the sitemap crawler would miss
Webhooks	`WebhooksEndpoint.ts`	Per-shop webhook CRUD (scrape_started / completed / failed)
Search	`SearchEndpoint.ts`	Semantic search over scraped content (REST wrapper over Qdrant)
Qdrant	`QdrantEndpoint.ts`	Vector DB management, per-shop embedding lifecycle, v1→v2 migration
MCP	`McpEndpoint.ts`	MCP call logs and analytics
Documentation	`DocumentationEndpoint.ts`	`/doc`, `/doc/openapi`, `/doc/endpoints`

Use GET /doc/endpoints for the exact, always-up-to-date list. A summarised snapshot at the time of writing:

Public

GET /health
GET /doc, GET /doc/openapi, GET /doc/endpoints
POST /mcp, GET /mcp, DELETE /mcp, GET /mcp/health (only when MCP is enabled)

Jobs

POST /api/jobs — create a scraping job from a URL
GET /api/jobs — list jobs with queue stats
GET /api/jobs/:id — job status and result

Shops

GET /api/shops — list with analytics
GET /api/shops/deleted — soft-deleted shops
GET /api/shops/:id — detail, supports either internal UUID or custom_id
GET /api/shops/:id/results — scraped content with filters
PATCH /api/shops/:id/schedule — enable/disable scheduled scraping
PATCH /api/shops/:id/custom-id — set/update custom_id
PATCH /api/shops/:id/qdrant — enable/disable embeddings (also see PUT /api/shops/:shopId/qdrant/toggle)
DELETE /api/shops/:id — soft delete
DELETE /api/shops/:id/force — hard delete

Shop content

GET /api/shops/:id/content
PATCH /api/shops/:id/content/:contentId

Shop custom URLs

POST /api/shops/:id/custom-urls
GET /api/shops/:id/custom-urls
PATCH /api/shops/:id/custom-urls/:customUrlId
DELETE /api/shops/:id/custom-urls/:customUrlId

Shop webhooks

POST /api/shops/:id/webhooks
GET /api/shops/:id/webhooks
PATCH /api/shops/:id/webhooks
DELETE /api/shops/:id/webhooks

Semantic search

POST /api/shops/:id/search — see Semantic search below

Qdrant (system-wide)

GET /api/qdrant/stats
GET /api/qdrant/health
GET /api/qdrant/collections
DELETE /api/qdrant/collections/:collectionName
POST /api/qdrant/cleanup

Qdrant (per shop)

GET /api/qdrant/shops/:shopId
GET /api/qdrant/shops/:shopId/embeddings
GET /api/qdrant/shops/:shopId/errors
DELETE /api/qdrant/shops/:shopId/errors
POST /api/qdrant/shops/:shopId/schedule-deletion
GET /api/shops/:shopId/qdrant — detailed Qdrant state
PUT /api/shops/:shopId/qdrant/toggle — enable/disable
PUT /api/shops/:shopId/qdrant/custom-id — set immutable custom id for the shop
POST /api/shops/:shopId/qdrant/re-embed
POST /api/shops/:shopId/qdrant/sync
GET /api/shops/:shopId/qdrant/errors
DELETE /api/shops/:shopId/qdrant/errors
DELETE /api/shops/:shopId/qdrant/pending-embeddings

Qdrant migration (v1 legacy → v2 consolidated)

GET /api/shops/:shopId/qdrant/migration-status
POST /api/shops/:shopId/qdrant/migrate
POST /api/shops/:shopId/qdrant/cleanup-old-collections

MCP logs & analytics

GET /api/mcp/stats
GET /api/mcp/analytics
GET /api/mcp/logs
GET /api/mcp/logs/:logId
DELETE /api/mcp/logs/cleanup
GET /api/mcp/tools/stats
GET /api/mcp/shops/:shopId/logs
DELETE /api/mcp/shops/:shopId/logs
GET /api/shops/:shopId/mcp/analytics

For every endpoint above, /doc/openapi returns the full request/response schema.

Shop identifiers

Every shop endpoint path parameter :id accepts either:

the internal UUID generated by the system, or
the custom_id if one is set.

The lookup in src/database/Database.ts tries both. custom_id must be a valid UUID and becomes immutable once Qdrant is enabled for the shop.

Content categories

Scraped pages are classified into four buckets, used consistently across the API, the DB, and the MCP tools:

shipping — shipping, delivery, logistics
contacts — contact info, support
terms — terms of service, privacy, legal
faq — frequently asked questions, help pages

Error responses

Errors return a JSON body:

{ "error": "Description of the error" }

Status codes follow standard REST semantics: 400 for bad input, 401 for bad auth, 404 for missing resources, 409 for conflicts (e.g. duplicate custom_id), 500 for server errors, 503 when the database or Qdrant is unavailable.

Semantic search

POST /api/shops/:id/search is a thin REST wrapper over the Qdrant vector search that powers the MCP tools. Use it when you want semantic (sentence/keyword) search from a plain HTTP client without speaking MCP.

Preconditions (enforced by the endpoint):

QDRANT_ENABLED=true on the server and OPENROUTER_API_KEY is set.
The shop has a custom_id and is toggled on via PUT /api/shops/:id/qdrant/toggle.
The shop has been scraped at least once so embeddings exist.

Path parameter

:id — either the internal UUID or the shop's custom_id.

Query string

format=json (default) — structured JSON.
format=text — plain text, one block per result, suited for voice/LLM pipelines.

Request body

{
  "q": "international delivery times",
  "category": "shipping",
  "limit": 5
}

Field	Required	Description
`q`	yes	Natural-language query.
`category`	no	One of `shipping`, `contacts`, `terms`, `faq`. Omit to search all four in parallel and get a merged, score-ranked list.
`limit`	no	1–20, default 10. Applies to the final merged list.

Example — search a single category:

curl -X POST http://localhost:3000/api/shops/shop123/search \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q":"express shipping to Germany","category":"shipping","limit":5}'

Example — search everything, text output:

curl -X POST "http://localhost:3000/api/shops/shop123/search?format=text" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q":"how do I return a defective product"}'

JSON response shape:

{
  "shop_id": "shop123",
  "query": "express shipping to Germany",
  "category": "shipping",
  "total": 3,
  "results": [
    {
      "score": 0.87,
      "content_type": "shipping",
      "url": "https://example-shop.com/shipping",
      "title": "Shipping policy",
      "content_id": "…",
      "snippet": "We offer express delivery to Germany within 2-3 business days…"
    }
  ]
}

Text response shape:

Shipping policy
We offer express delivery to Germany within 2-3 business days…

Delivery FAQ
For express shipments, tracking numbers are issued within…

Status codes

Code	Meaning
`200`	Results (possibly empty).
`400`	Missing `q`, invalid `category`, or `limit` out of range.
`404`	Shop not found.
`409`	Shop has no `custom_id`, or `qdrant_enabled` is false on the shop.
`503`	Qdrant disabled server-wide, or embedding service not configured.

Logging

Each REST search is written to mcp_logs with tool_name: "rest_search" so it shows up alongside MCP tool calls in the analytics endpoints — useful for comparing traffic sources.

MCP (Model Context Protocol)

When both MCP_ENABLED=true and QDRANT_ENABLED=true, the server mounts a Streamable HTTP MCP endpoint at /mcp (see src/mcp/McpServer.ts). Five tools are registered:

Tool name	Purpose
`list_contents`	Enumerate available pages for a shop, grouped by category, each tagged with the search tool to use next. Call this first.
`shipping_search`	Semantic search in shipping/delivery pages
`contacts_search`	Semantic search in contact/support pages
`terms_search`	Semantic search in terms/policy pages
`faq_search`	Semantic search in FAQ/help pages

All tools require shop_id (the shop's custom_id). The four search tools additionally take query (required) and limit (optional, 1–20, default 10). If a shop does not exist or does not have Qdrant enabled, tools return "Search not available for this shop" rather than erroring.

Tool schemas are declared in src/mcp/tools/*.ts and exposed to MCP clients via the standard tools/list request.

Response compression

All responses larger than 1 KB are automatically gzip/brotli-compressed by the compression middleware (src/api/server.ts). Set header x-no-compression: 1 to opt out.

What this document deliberately does not include

Full request/response JSON schemas — they live in /doc/openapi, generated from the endpoint metadata. Duplicating them here guarantees drift.
Rate limiting — the server does not implement any; if you put one in, document it here.
An SDK — there isn't one. Use any HTTP client with the Bearer token.

API.md 11 KB История Исходник

API Overview

Live reference

Authentication

Endpoint surface (high-level)

Public

Jobs

Shops

Shop content

Shop custom URLs

Shop webhooks

Semantic search

Qdrant (system-wide)

Qdrant (per shop)

Qdrant migration (v1 legacy → v2 consolidated)

MCP logs & analytics

Shop identifiers

Content categories

Error responses

Semantic search

MCP (Model Context Protocol)

Response compression

What this document deliberately does not include

API.md 11 KB

История Исходник