|
|
@@ -33,7 +33,7 @@ Authorization: Bearer <API_KEY>
|
|
|
The token must match the `API_KEY` environment variable. Missing or malformed `Authorization` headers return `401`. See `src/api/middleware/auth.ts`.
|
|
|
|
|
|
```bash
|
|
|
-curl http://localhost:3000/api/shops \
|
|
|
+curl http://localhost:3000/api/sites \
|
|
|
-H "Authorization: Bearer $API_KEY"
|
|
|
```
|
|
|
|
|
|
@@ -45,12 +45,12 @@ The registry is composed from endpoint components under `src/api/components/`. G
|
|
|
|---|---|---|
|
|
|
| Health | `HealthEndpoint.ts` | `/health` — public liveness + queue stats |
|
|
|
| Jobs | `JobsEndpoint.ts` | Create and inspect scraping jobs |
|
|
|
-| Shops | `ShopsEndpoint.ts` | CRUD + analytics + schedule toggle + custom_id + Qdrant flag |
|
|
|
-| Content | `ContentEndpoint.ts` | Per-shop scraped content (enable/disable pages) |
|
|
|
+| Sites | `SitesEndpoint.ts` | CRUD + analytics + schedule toggle + custom_id + Qdrant flag |
|
|
|
+| Content | `ContentEndpoint.ts` | Per-site scraped content (enable/disable pages) |
|
|
|
| Custom URLs | `CustomUrlsEndpoint.ts` | Manually add URLs the sitemap crawler would miss |
|
|
|
-| Webhooks | `WebhooksEndpoint.ts` | Per-shop webhook CRUD (scrape_started / completed / failed) |
|
|
|
+| Webhooks | `WebhooksEndpoint.ts` | Per-site webhook CRUD (scrape_started / completed / failed) |
|
|
|
| Search | `SearchEndpoint.ts` | Semantic search over scraped content (REST wrapper over Qdrant) |
|
|
|
-| Qdrant | `QdrantEndpoint.ts` | Vector DB management, per-shop embedding lifecycle, v1→v2 migration |
|
|
|
+| Qdrant | `QdrantEndpoint.ts` | Vector DB management, per-site embedding lifecycle, v1→v2 migration |
|
|
|
| MCP | `McpEndpoint.ts` | MCP call logs and analytics |
|
|
|
| Documentation | `DocumentationEndpoint.ts` | `/doc`, `/doc/openapi`, `/doc/endpoints` |
|
|
|
|
|
|
@@ -66,35 +66,35 @@ Use `GET /doc/endpoints` for the exact, always-up-to-date list. A summarised sna
|
|
|
- `GET /api/jobs` — list jobs with queue stats
|
|
|
- `GET /api/jobs/:id` — job status and result
|
|
|
|
|
|
-### Shops
|
|
|
-- `GET /api/shops` — list with analytics
|
|
|
-- `GET /api/shops/deleted` — soft-deleted shops
|
|
|
-- `GET /api/shops/:id` — detail, supports either internal UUID or `custom_id`
|
|
|
-- `GET /api/shops/:id/results` — scraped content with filters
|
|
|
-- `PATCH /api/shops/:id/schedule` — enable/disable scheduled scraping
|
|
|
-- `PATCH /api/shops/:id/custom-id` — set/update custom_id
|
|
|
-- `PATCH /api/shops/:id/qdrant` — enable/disable embeddings (also see `PUT /api/shops/:shopId/qdrant/toggle`)
|
|
|
-- `DELETE /api/shops/:id` — soft delete
|
|
|
-- `DELETE /api/shops/:id/force` — hard delete
|
|
|
-
|
|
|
-### Shop content
|
|
|
-- `GET /api/shops/:id/content`
|
|
|
-- `PATCH /api/shops/:id/content/:contentId`
|
|
|
-
|
|
|
-### Shop custom URLs
|
|
|
-- `POST /api/shops/:id/custom-urls`
|
|
|
-- `GET /api/shops/:id/custom-urls`
|
|
|
-- `PATCH /api/shops/:id/custom-urls/:customUrlId`
|
|
|
-- `DELETE /api/shops/:id/custom-urls/:customUrlId`
|
|
|
-
|
|
|
-### Shop webhooks
|
|
|
-- `POST /api/shops/:id/webhooks`
|
|
|
-- `GET /api/shops/:id/webhooks`
|
|
|
-- `PATCH /api/shops/:id/webhooks`
|
|
|
-- `DELETE /api/shops/:id/webhooks`
|
|
|
+### Sites
|
|
|
+- `GET /api/sites` — list with analytics
|
|
|
+- `GET /api/sites/deleted` — soft-deleted sites
|
|
|
+- `GET /api/sites/:id` — detail, supports either internal UUID or `custom_id`
|
|
|
+- `GET /api/sites/:id/results` — scraped content with filters
|
|
|
+- `PATCH /api/sites/:id/schedule` — enable/disable scheduled scraping
|
|
|
+- `PATCH /api/sites/:id/custom-id` — set/update custom_id
|
|
|
+- `PATCH /api/sites/:id/qdrant` — enable/disable embeddings (also see `PUT /api/sites/:siteId/qdrant/toggle`)
|
|
|
+- `DELETE /api/sites/:id` — soft delete
|
|
|
+- `DELETE /api/sites/:id/force` — hard delete
|
|
|
+
|
|
|
+### Site content
|
|
|
+- `GET /api/sites/:id/content`
|
|
|
+- `PATCH /api/sites/:id/content/:contentId`
|
|
|
+
|
|
|
+### Site custom URLs
|
|
|
+- `POST /api/sites/:id/custom-urls`
|
|
|
+- `GET /api/sites/:id/custom-urls`
|
|
|
+- `PATCH /api/sites/:id/custom-urls/:customUrlId`
|
|
|
+- `DELETE /api/sites/:id/custom-urls/:customUrlId`
|
|
|
+
|
|
|
+### Site webhooks
|
|
|
+- `POST /api/sites/:id/webhooks`
|
|
|
+- `GET /api/sites/:id/webhooks`
|
|
|
+- `PATCH /api/sites/:id/webhooks`
|
|
|
+- `DELETE /api/sites/:id/webhooks`
|
|
|
|
|
|
### Semantic search
|
|
|
-- `POST /api/shops/:id/search` — see [Semantic search](#semantic-search) below
|
|
|
+- `POST /api/sites/:id/search` — see [Semantic search](#semantic-search) below
|
|
|
|
|
|
### Qdrant (system-wide)
|
|
|
- `GET /api/qdrant/stats`
|
|
|
@@ -103,25 +103,25 @@ Use `GET /doc/endpoints` for the exact, always-up-to-date list. A summarised sna
|
|
|
- `DELETE /api/qdrant/collections/:collectionName`
|
|
|
- `POST /api/qdrant/cleanup`
|
|
|
|
|
|
-### Qdrant (per shop)
|
|
|
-- `GET /api/qdrant/shops/:shopId`
|
|
|
-- `GET /api/qdrant/shops/:shopId/embeddings`
|
|
|
-- `GET /api/qdrant/shops/:shopId/errors`
|
|
|
-- `DELETE /api/qdrant/shops/:shopId/errors`
|
|
|
-- `POST /api/qdrant/shops/:shopId/schedule-deletion`
|
|
|
-- `GET /api/shops/:shopId/qdrant` — detailed Qdrant state
|
|
|
-- `PUT /api/shops/:shopId/qdrant/toggle` — enable/disable
|
|
|
-- `PUT /api/shops/:shopId/qdrant/custom-id` — set immutable custom id for the shop
|
|
|
-- `POST /api/shops/:shopId/qdrant/re-embed`
|
|
|
-- `POST /api/shops/:shopId/qdrant/sync`
|
|
|
-- `GET /api/shops/:shopId/qdrant/errors`
|
|
|
-- `DELETE /api/shops/:shopId/qdrant/errors`
|
|
|
-- `DELETE /api/shops/:shopId/qdrant/pending-embeddings`
|
|
|
+### Qdrant (per site)
|
|
|
+- `GET /api/qdrant/sites/:siteId`
|
|
|
+- `GET /api/qdrant/sites/:siteId/embeddings`
|
|
|
+- `GET /api/qdrant/sites/:siteId/errors`
|
|
|
+- `DELETE /api/qdrant/sites/:siteId/errors`
|
|
|
+- `POST /api/qdrant/sites/:siteId/schedule-deletion`
|
|
|
+- `GET /api/sites/:siteId/qdrant` — detailed Qdrant state
|
|
|
+- `PUT /api/sites/:siteId/qdrant/toggle` — enable/disable
|
|
|
+- `PUT /api/sites/:siteId/qdrant/custom-id` — set immutable custom id for the site
|
|
|
+- `POST /api/sites/:siteId/qdrant/re-embed`
|
|
|
+- `POST /api/sites/:siteId/qdrant/sync`
|
|
|
+- `GET /api/sites/:siteId/qdrant/errors`
|
|
|
+- `DELETE /api/sites/:siteId/qdrant/errors`
|
|
|
+- `DELETE /api/sites/:siteId/qdrant/pending-embeddings`
|
|
|
|
|
|
### Qdrant migration (v1 legacy → v2 consolidated)
|
|
|
-- `GET /api/shops/:shopId/qdrant/migration-status`
|
|
|
-- `POST /api/shops/:shopId/qdrant/migrate`
|
|
|
-- `POST /api/shops/:shopId/qdrant/cleanup-old-collections`
|
|
|
+- `GET /api/sites/:siteId/qdrant/migration-status`
|
|
|
+- `POST /api/sites/:siteId/qdrant/migrate`
|
|
|
+- `POST /api/sites/:siteId/qdrant/cleanup-old-collections`
|
|
|
|
|
|
### MCP logs & analytics
|
|
|
- `GET /api/mcp/stats`
|
|
|
@@ -130,20 +130,20 @@ Use `GET /doc/endpoints` for the exact, always-up-to-date list. A summarised sna
|
|
|
- `GET /api/mcp/logs/:logId`
|
|
|
- `DELETE /api/mcp/logs/cleanup`
|
|
|
- `GET /api/mcp/tools/stats`
|
|
|
-- `GET /api/mcp/shops/:shopId/logs`
|
|
|
-- `DELETE /api/mcp/shops/:shopId/logs`
|
|
|
-- `GET /api/shops/:shopId/mcp/analytics`
|
|
|
+- `GET /api/mcp/sites/:siteId/logs`
|
|
|
+- `DELETE /api/mcp/sites/:siteId/logs`
|
|
|
+- `GET /api/sites/:siteId/mcp/analytics`
|
|
|
|
|
|
For every endpoint above, `/doc/openapi` returns the full request/response schema.
|
|
|
|
|
|
-## Shop identifiers
|
|
|
+## Site identifiers
|
|
|
|
|
|
-Every shop endpoint path parameter `:id` accepts **either**:
|
|
|
+Every site endpoint path parameter `:id` accepts **either**:
|
|
|
|
|
|
- the internal UUID generated by the system, or
|
|
|
- the `custom_id` if one is set.
|
|
|
|
|
|
-The lookup in `src/database/Database.ts` tries both. `custom_id` must be a valid UUID and becomes immutable once Qdrant is enabled for the shop.
|
|
|
+The lookup in `src/database/SiteDatabase.ts` tries both. `custom_id` must be a valid UUID and becomes immutable once Qdrant is enabled for the site.
|
|
|
|
|
|
## Content categories
|
|
|
|
|
|
@@ -166,17 +166,17 @@ Status codes follow standard REST semantics: `400` for bad input, `401` for bad
|
|
|
|
|
|
## Semantic search
|
|
|
|
|
|
-`POST /api/shops/:id/search` is a thin REST wrapper over the Qdrant vector search that powers the MCP tools. Use it when you want semantic (sentence/keyword) search from a plain HTTP client without speaking MCP.
|
|
|
+`POST /api/sites/:id/search` is a thin REST wrapper over the Qdrant vector search that powers the MCP tools. Use it when you want semantic (sentence/keyword) search from a plain HTTP client without speaking MCP.
|
|
|
|
|
|
**Preconditions** (enforced by the endpoint):
|
|
|
|
|
|
- `QDRANT_ENABLED=true` on the server and `OPENROUTER_API_KEY` is set.
|
|
|
-- The shop has a `custom_id` and is toggled on via `PUT /api/shops/:id/qdrant/toggle`.
|
|
|
-- The shop has been scraped at least once so embeddings exist.
|
|
|
+- The site has a `custom_id` and is toggled on via `PUT /api/sites/:id/qdrant/toggle`.
|
|
|
+- The site has been scraped at least once so embeddings exist.
|
|
|
|
|
|
**Path parameter**
|
|
|
|
|
|
-- `:id` — either the internal UUID or the shop's `custom_id`.
|
|
|
+- `:id` — either the internal UUID or the site's `custom_id`.
|
|
|
|
|
|
**Query string**
|
|
|
|
|
|
@@ -202,7 +202,7 @@ Status codes follow standard REST semantics: `400` for bad input, `401` for bad
|
|
|
**Example — search a single category:**
|
|
|
|
|
|
```bash
|
|
|
-curl -X POST http://localhost:3000/api/shops/shop123/search \
|
|
|
+curl -X POST http://localhost:3000/api/sites/shop123/search \
|
|
|
-H "Authorization: Bearer $API_KEY" \
|
|
|
-H "Content-Type: application/json" \
|
|
|
-d '{"q":"express shipping to Germany","category":"shipping","limit":5}'
|
|
|
@@ -211,7 +211,7 @@ curl -X POST http://localhost:3000/api/shops/shop123/search \
|
|
|
**Example — search everything, text output:**
|
|
|
|
|
|
```bash
|
|
|
-curl -X POST "http://localhost:3000/api/shops/shop123/search?format=text" \
|
|
|
+curl -X POST "http://localhost:3000/api/sites/shop123/search?format=text" \
|
|
|
-H "Authorization: Bearer $API_KEY" \
|
|
|
-H "Content-Type: application/json" \
|
|
|
-d '{"q":"how do I return a defective product"}'
|
|
|
@@ -221,7 +221,7 @@ curl -X POST "http://localhost:3000/api/shops/shop123/search?format=text" \
|
|
|
|
|
|
```json
|
|
|
{
|
|
|
- "shop_id": "shop123",
|
|
|
+ "site_id": "shop123",
|
|
|
"query": "express shipping to Germany",
|
|
|
"category": "shipping",
|
|
|
"total": 3,
|
|
|
@@ -254,8 +254,8 @@ For express shipments, tracking numbers are issued within…
|
|
|
|---|---|
|
|
|
| `200` | Results (possibly empty). |
|
|
|
| `400` | Missing `q`, invalid `category`, or `limit` out of range. |
|
|
|
-| `404` | Shop not found. |
|
|
|
-| `409` | Shop has no `custom_id`, or `qdrant_enabled` is false on the shop. |
|
|
|
+| `404` | Site not found. |
|
|
|
+| `409` | Site has no `custom_id`, or `qdrant_enabled` is false on the site. |
|
|
|
| `503` | Qdrant disabled server-wide, or embedding service not configured. |
|
|
|
|
|
|
**Logging**
|
|
|
@@ -268,13 +268,13 @@ When both `MCP_ENABLED=true` and `QDRANT_ENABLED=true`, the server mounts a **St
|
|
|
|
|
|
| Tool name | Purpose |
|
|
|
|---|---|
|
|
|
-| `list_contents` | Enumerate available pages for a shop, grouped by category, each tagged with the search tool to use next. Call this first. |
|
|
|
+| `list_contents` | Enumerate available pages for a site, grouped by category, each tagged with the search tool to use next. Call this first. |
|
|
|
| `shipping_search` | Semantic search in shipping/delivery pages |
|
|
|
| `contacts_search` | Semantic search in contact/support pages |
|
|
|
| `terms_search` | Semantic search in terms/policy pages |
|
|
|
| `faq_search` | Semantic search in FAQ/help pages |
|
|
|
|
|
|
-All tools require `shop_id` (the shop's `custom_id`). The four search tools additionally take `query` (required) and `limit` (optional, 1–20, default 10). If a shop does not exist or does not have Qdrant enabled, tools return `"Search not available for this shop"` rather than erroring.
|
|
|
+All tools accept both `site_id` and `shop_id` (legacy alias) as the parameter name, mapped to the site's `custom_id`. The four search tools additionally take `query` (required) and `limit` (optional, 1–20, default 10). If a site does not exist or does not have Qdrant enabled, tools return `"Search not available for this site"` rather than erroring.
|
|
|
|
|
|
Tool schemas are declared in `src/mcp/tools/*.ts` and exposed to MCP clients via the standard `tools/list` request.
|
|
|
|
|
|
@@ -282,6 +282,16 @@ Tool schemas are declared in `src/mcp/tools/*.ts` and exposed to MCP clients via
|
|
|
|
|
|
All responses larger than 1 KB are automatically gzip/brotli-compressed by the `compression` middleware (`src/api/server.ts`). Set header `x-no-compression: 1` to opt out.
|
|
|
|
|
|
+## Backward compatibility
|
|
|
+
|
|
|
+All `/api/sites/*` endpoints are also available under their previous `/api/shops/*` paths. Legacy routes behave identically but return responses with the old key names (`shop_id`, `webshop_type`, etc.) so existing consumers continue working without changes.
|
|
|
+
|
|
|
+- New code should use `/api/sites/*`.
|
|
|
+- Existing consumers using `/api/shops/*` continue working — no migration required.
|
|
|
+- MCP tools accept both `site_id` and `shop_id` as the identifier parameter name.
|
|
|
+
|
|
|
+The legacy routes will be maintained for the foreseeable future. When they are eventually deprecated, a deprecation notice will be added here and in the `/doc` output first.
|
|
|
+
|
|
|
## What this document deliberately does not include
|
|
|
|
|
|
- Full request/response JSON schemas — they live in `/doc/openapi`, generated from the endpoint metadata. Duplicating them here guarantees drift.
|