API Overview

This project ships a self-documenting REST API. Each endpoint carries its own metadata (parameters, request body, responses, auth requirement) and an endpoint registry generates both browsable HTML and an OpenAPI 3.0 spec at runtime.

Always treat the running server as the source of truth. This file covers the concepts — auth, conventions, shape of the surface — and points you at the live reference for the full endpoint list.

Live reference

Once the server is running, three documentation endpoints are available (no auth required):

Endpoint	Returns
`GET /doc`	Interactive HTML documentation page
`GET /doc/openapi`	OpenAPI 3.0 JSON specification
`GET /doc/endpoints`	Compact JSON list of `{ method, path, summary, tags, requiresAuth }`

# Browse in a browser:
open http://localhost:3000/doc

# Pipe the spec into anything OpenAPI-aware:
curl http://localhost:3000/doc/openapi | jq .

Authentication

All /api/* endpoints require Bearer token authentication. A handful of non-/api endpoints (/health, /doc*, /ui/*, /mcp*) are public.

Authorization: Bearer <API_KEY>

The token must match the API_KEY environment variable. Missing or malformed Authorization headers return 401. See src/api/middleware/auth.ts.

curl http://localhost:3000/api/sites \
  -H "Authorization: Bearer $API_KEY"

Endpoint surface (high-level)

The registry is composed from endpoint components under src/api/components/. Grouped by tag:

Tag	Component	What it covers
Health	`HealthEndpoint.ts`	`/health` — public liveness + queue stats
Jobs	`JobsEndpoint.ts`	Create and inspect scraping jobs
Sites	`SitesEndpoint.ts`	CRUD + analytics + schedule toggle + custom_id + Qdrant flag
Content	`ContentEndpoint.ts`	Per-site scraped content (enable/disable pages)
Custom URLs	`CustomUrlsEndpoint.ts`	Manually add URLs the sitemap crawler would miss
Webhooks	`WebhooksEndpoint.ts`	Per-site webhook CRUD (scrape_started / completed / failed)
Search	`SearchEndpoint.ts`	Semantic search over scraped content (REST wrapper over Qdrant)
URL Exclusions	`UrlExclusionsEndpoint.ts`	Per-site URL exclusion patterns (glob-based)
Qdrant	`QdrantEndpoint.ts`	Vector DB management, per-site embedding lifecycle, v1→v2 migration
MCP	`McpEndpoint.ts`	MCP call logs and analytics
Documentation	`DocumentationEndpoint.ts`	`/doc`, `/doc/openapi`, `/doc/endpoints`

Use GET /doc/endpoints for the exact, always-up-to-date list. A summarised snapshot at the time of writing:

Public

GET /health
GET /doc, GET /doc/openapi, GET /doc/endpoints
POST /mcp, GET /mcp, DELETE /mcp, GET /mcp/health (only when MCP is enabled)

Jobs

POST /api/jobs — create a scraping job from a URL
GET /api/jobs — list jobs with queue stats
GET /api/jobs/:id — job status and result

Sites

POST /api/sites/detect — auto-detect site type, platform, and sitemap URL from a given URL
GET /api/sites — list with analytics
GET /api/sites/deleted — soft-deleted sites
GET /api/sites/:id — detail, supports either internal UUID or custom_id
GET /api/sites/:id/results — scraped content with filters
PATCH /api/sites/:id/schedule — enable/disable scheduled scraping
PATCH /api/sites/:id/custom-id — set/update custom_id
PATCH /api/sites/:id/settings — update site settings (site_type, site_platform, custom_sitemap_url, max_crawl_depth, max_pages)
PATCH /api/sites/:id/qdrant — enable/disable embeddings (also see PUT /api/sites/:siteId/qdrant/toggle)
DELETE /api/sites/:id — soft delete
DELETE /api/sites/:id/force — hard delete

Site content

GET /api/sites/:id/content
PATCH /api/sites/:id/content/:contentId

Site custom URLs

POST /api/sites/:id/custom-urls
GET /api/sites/:id/custom-urls
PATCH /api/sites/:id/custom-urls/:customUrlId
DELETE /api/sites/:id/custom-urls/:customUrlId

Site URL exclusions

POST /api/sites/:siteId/url-exclusions — add a glob pattern to exclude URLs from scraping
GET /api/sites/:siteId/url-exclusions — list exclusion patterns for a site
DELETE /api/sites/:siteId/url-exclusions/:exclusionId — delete an exclusion pattern

Site webhooks

POST /api/sites/:id/webhooks
GET /api/sites/:id/webhooks
PATCH /api/sites/:id/webhooks
DELETE /api/sites/:id/webhooks

Semantic search

POST /api/sites/:id/search — see Semantic search below

Qdrant (system-wide)

GET /api/qdrant/stats
GET /api/qdrant/health
GET /api/qdrant/collections
DELETE /api/qdrant/collections/:collectionName
POST /api/qdrant/cleanup

Qdrant (per site)

GET /api/qdrant/sites/:siteId
GET /api/qdrant/sites/:siteId/embeddings
GET /api/qdrant/sites/:siteId/errors
DELETE /api/qdrant/sites/:siteId/errors
POST /api/qdrant/sites/:siteId/schedule-deletion
GET /api/sites/:siteId/qdrant — detailed Qdrant state
PUT /api/sites/:siteId/qdrant/toggle — enable/disable
PUT /api/sites/:siteId/qdrant/custom-id — set immutable custom id for the site
POST /api/sites/:siteId/qdrant/re-embed
POST /api/sites/:siteId/qdrant/sync
GET /api/sites/:siteId/qdrant/errors
DELETE /api/sites/:siteId/qdrant/errors
DELETE /api/sites/:siteId/qdrant/pending-embeddings

Qdrant migration (v1 legacy → v2 consolidated)

GET /api/sites/:siteId/qdrant/migration-status
POST /api/sites/:siteId/qdrant/migrate
POST /api/sites/:siteId/qdrant/cleanup-old-collections

MCP logs & analytics

GET /api/mcp/stats
GET /api/mcp/analytics
GET /api/mcp/logs
GET /api/mcp/logs/:logId
DELETE /api/mcp/logs/cleanup
GET /api/mcp/tools/stats
GET /api/mcp/sites/:siteId/logs
DELETE /api/mcp/sites/:siteId/logs
GET /api/sites/:siteId/mcp/analytics

For every endpoint above, /doc/openapi returns the full request/response schema.

Site identifiers

Every site endpoint path parameter :id accepts either:

the internal UUID generated by the system, or
the custom_id if one is set.

The lookup in src/database/SiteDatabase.ts tries both. custom_id must be a valid UUID and becomes immutable once Qdrant is enabled for the site.

Content categories

Scraped pages are classified into 13 buckets, used consistently across the API, the DB, and the MCP tools:

shipping — shipping, delivery, logistics
contacts — contact info, support
terms — terms of service, privacy, legal
faq — frequently asked questions, help pages
services — services offered
about — about the company/organization
team — team members, staff pages
blog — blog posts, articles, news
pricing — pricing, plans, rates
testimonials — reviews, testimonials, case studies
gallery — image galleries, portfolios
landing — landing pages, home page
other — pages that don't fit the above categories

Error responses

Errors return a JSON body:

{ "error": "Description of the error" }

Status codes follow standard REST semantics: 400 for bad input, 401 for bad auth, 404 for missing resources, 409 for conflicts (e.g. duplicate custom_id), 500 for server errors, 503 when the database or Qdrant is unavailable.

Semantic search

POST /api/sites/:id/search is a thin REST wrapper over the Qdrant vector search that powers the MCP tools. Use it when you want semantic (sentence/keyword) search from a plain HTTP client without speaking MCP.

Preconditions (enforced by the endpoint):

QDRANT_ENABLED=true on the server and OPENROUTER_API_KEY is set.
The site has a custom_id and is toggled on via PUT /api/sites/:id/qdrant/toggle.
The site has been scraped at least once so embeddings exist.

Path parameter

:id — either the internal UUID or the site's custom_id.

Query string

format=json (default) — structured JSON.
format=text — plain text, one block per result, suited for voice/LLM pipelines.

Request body

{
  "q": "international delivery times",
  "category": "shipping",
  "limit": 5
}

Field	Required	Description
`q`	yes	Natural-language query.
`category`	no	One of `shipping`, `contacts`, `terms`, `faq`, `services`, `about`, `team`, `blog`, `pricing`, `testimonials`, `gallery`, `landing`, `other`. Omit to search all 13 in parallel and get a merged, score-ranked list.
`limit`	no	1–20, default 10. Applies to the final merged list.

Example — search a single category:

curl -X POST http://localhost:3000/api/sites/shop123/search \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q":"express shipping to Germany","category":"shipping","limit":5}'

Example — search everything, text output:

curl -X POST "http://localhost:3000/api/sites/shop123/search?format=text" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q":"how do I return a defective product"}'

JSON response shape:

{
  "site_id": "shop123",
  "query": "express shipping to Germany",
  "category": "shipping",
  "total": 3,
  "results": [
    {
      "score": 0.87,
      "content_type": "shipping",
      "url": "https://example-shop.com/shipping",
      "title": "Shipping policy",
      "content_id": "…",
      "snippet": "We offer express delivery to Germany within 2-3 business days…"
    }
  ]
}

Text response shape:

Shipping policy
We offer express delivery to Germany within 2-3 business days…

Delivery FAQ
For express shipments, tracking numbers are issued within…

Status codes

Code	Meaning
`200`	Results (possibly empty).
`400`	Missing `q`, invalid `category`, or `limit` out of range.
`404`	Site not found.
`409`	Site has no `custom_id`, or `qdrant_enabled` is false on the site.
`503`	Qdrant disabled server-wide, or embedding service not configured.

Logging

Each REST search is written to mcp_logs with tool_name: "rest_search" so it shows up alongside MCP tool calls in the analytics endpoints — useful for comparing traffic sources.

MCP (Model Context Protocol)

When both MCP_ENABLED=true and QDRANT_ENABLED=true, the server mounts a Streamable HTTP MCP endpoint at /mcp (see src/mcp/McpServer.ts). 14 tools are registered:

Tool name	Purpose
`list_contents`	Enumerate available pages for a site, grouped by category, each tagged with the search tool to use next. Call this first.
`shipping_search`	Semantic search in shipping/delivery pages
`contacts_search`	Semantic search in contact/support pages
`terms_search`	Semantic search in terms/policy pages
`faq_search`	Semantic search in FAQ/help pages
`services_search`	Semantic search in services pages
`about_search`	Semantic search in about pages
`team_search`	Semantic search in team/staff pages
`blog_search`	Semantic search in blog/article pages
`pricing_search`	Semantic search in pricing/plans pages
`testimonials_search`	Semantic search in testimonial/review pages
`gallery_search`	Semantic search in gallery/portfolio pages
`landing_search`	Semantic search in landing/home pages
`other_search`	Semantic search in uncategorized pages

All tools accept both site_id and shop_id (legacy alias) as the parameter name, mapped to the site's custom_id. The 13 search tools additionally take query (required) and limit (optional, 1–20, default 10). If a site does not exist or does not have Qdrant enabled, tools return "Search not available for this site" rather than erroring.

Tool schemas are declared in src/mcp/tools/*.ts and exposed to MCP clients via the standard tools/list request.

Response compression

All responses larger than 1 KB are automatically gzip/brotli-compressed by the compression middleware (src/api/server.ts). Set header x-no-compression: 1 to opt out.

Backward compatibility

All /api/sites/* endpoints are also available under their previous /api/shops/* paths. Legacy routes behave identically but return responses with the old key names (shop_id, webshop_type, etc.) so existing consumers continue working without changes.

New code should use /api/sites/*.
Existing consumers using /api/shops/* continue working — no migration required.
MCP tools accept both site_id and shop_id as the identifier parameter name.

The legacy routes will be maintained for the foreseeable future. When they are eventually deprecated, a deprecation notice will be added here and in the /doc output first.

What this document deliberately does not include

Full request/response JSON schemas — they live in /doc/openapi, generated from the endpoint metadata. Duplicating them here guarantees drift.
Rate limiting — the server does not implement any; if you put one in, document it here.
An SDK — there isn't one. Use any HTTP client with the Bearer token.

API.md 13 KB История Исходник

API Overview

Live reference

Authentication

Endpoint surface (high-level)

Public

Jobs

Sites

Site content

Site custom URLs

Site URL exclusions

Site webhooks

Semantic search

Qdrant (system-wide)

Qdrant (per site)

Qdrant migration (v1 legacy → v2 consolidated)

MCP logs & analytics

Site identifiers

Content categories

Error responses

Semantic search

MCP (Model Context Protocol)

Response compression

Backward compatibility

What this document deliberately does not include

API.md 13 KB

История Исходник