docs: update all docs for universal scraper features

README.md

@@ -1,16 +1,20 @@
 # Web Scraper
 
-A TypeScript service that scrapes shipping, contact, terms, and FAQ pages from sites (ShopRenter, WooCommerce, Shopify), stores them in SQLite with change detection, and optionally indexes them into Qdrant for semantic search via MCP tools.
+A TypeScript service that scrapes and classifies content from any website — webshops (ShopRenter, WooCommerce, Shopify) and generic websites alike. Content is stored in SQLite with change detection and optionally indexed into Qdrant for semantic search via MCP tools or REST API.
 
 ## Features
 
-- **Multi-platform scraping** — ShopRenter, WooCommerce, Shopify
-- **Sitemap-driven** — parses sitemaps, classifies pages into `shipping` / `contacts` / `terms` / `faq`
+- **Universal scraping** — supports two site types: `webshop` (ShopRenter, WooCommerce, Shopify) and `website` (any generic site)
+- **13 content categories** — `shipping`, `contacts`, `terms`, `faq`, `services`, `about`, `team`, `blog`, `pricing`, `testimonials`, `gallery`, `landing`, `other`
+- **Smart content discovery** — sitemap-driven for webshops; automatic link discovery engine (BFS crawl) for sites without a sitemap, with configurable `max_crawl_depth` and `max_pages`
+- **URL exclusions** — glob patterns to skip unwanted URLs during scraping
+- **Auto-detect** — `POST /api/sites/detect` probes a URL to determine site type, platform, and sitemap
+- **Custom sitemap** — override the auto-detected sitemap with your own URL
 - **Scheduled scraping** — automatic re-scrape based on sitemap `changefreq` rules, toggleable per site
 - **Change detection** — SHA-256 content hashing; unchanged pages are not re-processed
 - **Persistent storage** — SQLite (`better-sqlite3`) with site analytics and scrape history
 - **Optional vector search** — Qdrant + OpenRouter embeddings, one collection per `{custom_id}-{category}`
-- **MCP server** — Streamable HTTP MCP endpoint at `/mcp` exposing `list_contents`, `shipping_search`, `contacts_search`, `terms_search`, `faq_search`
+- **MCP server** — Streamable HTTP MCP endpoint at `/mcp` exposing 14 tools: `list_contents` plus a `*_search` tool for each of the 13 content categories
 - **Self-documenting REST API** — endpoint metadata is rendered as HTML at `/doc` and OpenAPI 3.0 at `/doc/openapi`
 - **Bearer authentication** for all `/api/*` endpoints
 - **Webhooks** per site — `scrape_started` / `scrape_completed` / `scrape_failed`
@@ -99,19 +103,28 @@ curl -X POST http://localhost:3000/api/sites/<id>/search \
   -d '{"q":"do you ship to Germany","limit":5}'
 ```
 
-Omit `category` to search all four content buckets in parallel. Add `?format=text` for voice/LLM output.
+Omit `category` to search all 13 content buckets in parallel. Add `?format=text` for voice/LLM output.
 
 `<id>` can be either the system-generated UUID or the optional `custom_id` you assigned to the site.
 
 ## MCP tools
 
-When `MCP_ENABLED=true` and `QDRANT_ENABLED=true`, the server mounts an MCP Streamable HTTP endpoint at `/mcp`. Five tools are registered:
+When `MCP_ENABLED=true` and `QDRANT_ENABLED=true`, the server mounts an MCP Streamable HTTP endpoint at `/mcp`. 14 tools are registered:
 
 - `list_contents(site_id)` — enumerate available pages per category, tagged with the next tool to use
 - `shipping_search(site_id, query, limit?)`
 - `contacts_search(site_id, query, limit?)`
 - `terms_search(site_id, query, limit?)`
 - `faq_search(site_id, query, limit?)`
+- `services_search(site_id, query, limit?)`
+- `about_search(site_id, query, limit?)`
+- `team_search(site_id, query, limit?)`
+- `blog_search(site_id, query, limit?)`
+- `pricing_search(site_id, query, limit?)`
+- `testimonials_search(site_id, query, limit?)`
+- `gallery_search(site_id, query, limit?)`
+- `landing_search(site_id, query, limit?)`
+- `other_search(site_id, query, limit?)`
 
 All tools accept both `site_id` and `shop_id` (legacy alias) as the parameter name, mapped to the site's `custom_id`. Tools return a friendly "Search not available for this site" message when the site is unknown or Qdrant is not enabled on it.
 

docs/API.md

@@ -50,6 +50,7 @@ The registry is composed from endpoint components under `src/api/components/`. G
 | Custom URLs | `CustomUrlsEndpoint.ts` | Manually add URLs the sitemap crawler would miss |
 | Webhooks | `WebhooksEndpoint.ts` | Per-site webhook CRUD (scrape_started / completed / failed) |
 | Search | `SearchEndpoint.ts` | Semantic search over scraped content (REST wrapper over Qdrant) |
+| URL Exclusions | `UrlExclusionsEndpoint.ts` | Per-site URL exclusion patterns (glob-based) |
 | Qdrant | `QdrantEndpoint.ts` | Vector DB management, per-site embedding lifecycle, v1→v2 migration |
 | MCP | `McpEndpoint.ts` | MCP call logs and analytics |
 | Documentation | `DocumentationEndpoint.ts` | `/doc`, `/doc/openapi`, `/doc/endpoints` |
@@ -67,12 +68,14 @@ Use `GET /doc/endpoints` for the exact, always-up-to-date list. A summarised sna
 - `GET /api/jobs/:id` — job status and result
 
 ### Sites
+- `POST /api/sites/detect` — auto-detect site type, platform, and sitemap URL from a given URL
 - `GET /api/sites` — list with analytics
 - `GET /api/sites/deleted` — soft-deleted sites
 - `GET /api/sites/:id` — detail, supports either internal UUID or `custom_id`
 - `GET /api/sites/:id/results` — scraped content with filters
 - `PATCH /api/sites/:id/schedule` — enable/disable scheduled scraping
 - `PATCH /api/sites/:id/custom-id` — set/update custom_id
+- `PATCH /api/sites/:id/settings` — update site settings (site_type, site_platform, custom_sitemap_url, max_crawl_depth, max_pages)
 - `PATCH /api/sites/:id/qdrant` — enable/disable embeddings (also see `PUT /api/sites/:siteId/qdrant/toggle`)
 - `DELETE /api/sites/:id` — soft delete
 - `DELETE /api/sites/:id/force` — hard delete
@@ -87,6 +90,11 @@ Use `GET /doc/endpoints` for the exact, always-up-to-date list. A summarised sna
 - `PATCH /api/sites/:id/custom-urls/:customUrlId`
 - `DELETE /api/sites/:id/custom-urls/:customUrlId`
 
+### Site URL exclusions
+- `POST /api/sites/:siteId/url-exclusions` — add a glob pattern to exclude URLs from scraping
+- `GET /api/sites/:siteId/url-exclusions` — list exclusion patterns for a site
+- `DELETE /api/sites/:siteId/url-exclusions/:exclusionId` — delete an exclusion pattern
+
 ### Site webhooks
 - `POST /api/sites/:id/webhooks`
 - `GET /api/sites/:id/webhooks`
@@ -147,12 +155,21 @@ The lookup in `src/database/SiteDatabase.ts` tries both. `custom_id` must be a v
 
 ## Content categories
 
-Scraped pages are classified into four buckets, used consistently across the API, the DB, and the MCP tools:
+Scraped pages are classified into 13 buckets, used consistently across the API, the DB, and the MCP tools:
 
 - `shipping` — shipping, delivery, logistics
 - `contacts` — contact info, support
 - `terms` — terms of service, privacy, legal
 - `faq` — frequently asked questions, help pages
+- `services` — services offered
+- `about` — about the company/organization
+- `team` — team members, staff pages
+- `blog` — blog posts, articles, news
+- `pricing` — pricing, plans, rates
+- `testimonials` — reviews, testimonials, case studies
+- `gallery` — image galleries, portfolios
+- `landing` — landing pages, home page
+- `other` — pages that don't fit the above categories
 
 ## Error responses
 
@@ -196,7 +213,7 @@ Status codes follow standard REST semantics: `400` for bad input, `401` for bad
 | Field | Required | Description |
 |---|---|---|
 | `q` | yes | Natural-language query. |
-| `category` | no | One of `shipping`, `contacts`, `terms`, `faq`. Omit to search all four in parallel and get a merged, score-ranked list. |
+| `category` | no | One of `shipping`, `contacts`, `terms`, `faq`, `services`, `about`, `team`, `blog`, `pricing`, `testimonials`, `gallery`, `landing`, `other`. Omit to search all 13 in parallel and get a merged, score-ranked list. |
 | `limit` | no | 1–20, default 10. Applies to the final merged list. |
 
 **Example — search a single category:**
@@ -264,7 +281,7 @@ Each REST search is written to `mcp_logs` with `tool_name: "rest_search"` so it
 
 ## MCP (Model Context Protocol)
 
-When both `MCP_ENABLED=true` and `QDRANT_ENABLED=true`, the server mounts a **Streamable HTTP** MCP endpoint at `/mcp` (see `src/mcp/McpServer.ts`). Five tools are registered:
+When both `MCP_ENABLED=true` and `QDRANT_ENABLED=true`, the server mounts a **Streamable HTTP** MCP endpoint at `/mcp` (see `src/mcp/McpServer.ts`). 14 tools are registered:
 
 | Tool name | Purpose |
 |---|---|
@@ -273,8 +290,17 @@ When both `MCP_ENABLED=true` and `QDRANT_ENABLED=true`, the server mounts a **St
 | `contacts_search` | Semantic search in contact/support pages |
 | `terms_search` | Semantic search in terms/policy pages |
 | `faq_search` | Semantic search in FAQ/help pages |
-
-All tools accept both `site_id` and `shop_id` (legacy alias) as the parameter name, mapped to the site's `custom_id`. The four search tools additionally take `query` (required) and `limit` (optional, 1–20, default 10). If a site does not exist or does not have Qdrant enabled, tools return `"Search not available for this site"` rather than erroring.
+| `services_search` | Semantic search in services pages |
+| `about_search` | Semantic search in about pages |
+| `team_search` | Semantic search in team/staff pages |
+| `blog_search` | Semantic search in blog/article pages |
+| `pricing_search` | Semantic search in pricing/plans pages |
+| `testimonials_search` | Semantic search in testimonial/review pages |
+| `gallery_search` | Semantic search in gallery/portfolio pages |
+| `landing_search` | Semantic search in landing/home pages |
+| `other_search` | Semantic search in uncategorized pages |
+
+All tools accept both `site_id` and `shop_id` (legacy alias) as the parameter name, mapped to the site's `custom_id`. The 13 search tools additionally take `query` (required) and `limit` (optional, 1–20, default 10). If a site does not exist or does not have Qdrant enabled, tools return `"Search not available for this site"` rather than erroring.
 
 Tool schemas are declared in `src/mcp/tools/*.ts` and exposed to MCP clients via the standard `tools/list` request.
 

docs/SETUP.md

@@ -72,6 +72,28 @@ OPENROUTER_API_KEY=sk-or-v1-...
 MCP_ENABLED=true
 ```
 
+## Site types
+
+The scraper supports two site types, set when a job is created or updated via `PATCH /api/sites/:id/settings`:
+
+- **`webshop`** — e-commerce sites built on ShopRenter, WooCommerce, or Shopify. The scraper uses the platform's sitemap to discover pages and classifies them into content categories.
+- **`website`** — any generic website. When no sitemap is found, the scraper falls back to the link discovery engine (see below).
+
+When you create a job with `POST /api/jobs`, the system auto-detects the site type and platform via `POST /api/sites/detect`. You can override the detected values with `PATCH /api/sites/:id/settings`.
+
+## Link discovery
+
+When a site has no sitemap (common for generic websites), the scraper uses a BFS-based link discovery engine that crawls same-origin links starting from the site's base URL. Two settings control the crawl:
+
+| Setting | Default | Description |
+|---|---|---|
+| `max_crawl_depth` | `3` | Maximum link-follow depth from the start URL. |
+| `max_pages` | `200` | Maximum number of pages to visit. |
+
+Both can be configured per site via `PATCH /api/sites/:id/settings`. Discovered URLs are then classified into content categories the same way sitemap URLs are.
+
+You can also add **URL exclusion patterns** (`POST /api/sites/:siteId/url-exclusions`) using glob syntax (e.g. `*/admin/*`, `*/wp-json/*`) to skip unwanted URLs during scraping.
+
 ## Build
 
 ```bash
@@ -161,7 +183,7 @@ Embedding happens per site — you opt each site in.
 ### 5. Collection layout
 
 - One collection per `{custom_id}-{category}`, e.g. `shop123-shipping`, `shop123-faq`.
-- Categories: `shipping`, `contacts`, `terms`, `faq`.
+- Categories: `shipping`, `contacts`, `terms`, `faq`, `services`, `about`, `team`, `blog`, `pricing`, `testimonials`, `gallery`, `landing`, `other`.
 - Each page is stored as one or more chunks (via `ChunkingService`); point IDs are deterministic: `SHA256(url + content_hash + chunk_index)`. Same URL + same content ⇒ same point ID, so re-embedding is a no-op; content changes yield a fresh point.
 
 Verified in `src/services/QdrantEmbeddingWorkflow.ts` and `src/services/QdrantService.ts`.
@@ -194,7 +216,7 @@ curl -X POST -H "Authorization: Bearer $API_KEY" \
 
 ## MCP (Model Context Protocol)
 
-When `MCP_ENABLED=true` **and** `QDRANT_ENABLED=true`, `src/api/server.ts` mounts the MCP Streamable HTTP transport at `/mcp`. Five tools are exposed; see `docs/API.md` for the list and the shapes.
+When `MCP_ENABLED=true` **and** `QDRANT_ENABLED=true`, `src/api/server.ts` mounts the MCP Streamable HTTP transport at `/mcp`. 14 tools are exposed (1 listing + 13 search tools, one per content category); see `docs/API.md` for the list and the shapes.
 
 Point any MCP client at: