docs: flesh out CLAUDE.md with architecture, conventions, and gotchas

CLAUDE.md

@@ -1 +1,126 @@
-- Never start the server, aks the user
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Working with this repo
+
+- **Never start the server yourself — ask the user.** `npm start` / `npm run dev` are long-running and the user runs them manually.
+- The live server is the source of truth for API surface. Browse `http://localhost:3000/doc` (HTML) or `/doc/openapi` (OpenAPI 3.0 JSON) instead of duplicating route docs in code/notes.
+
+## Commands
+
+```bash
+npm install        # also runs `npm install` inside web/
+npm run build      # tsc → dist/, then `cd web && npm run build` → dist/web/
+npm run dev        # ts-node, no build
+npm run watch      # tsc -w
+npm run dev:web    # vite dev server for the UI only
+```
+
+There is no test runner, lint, or formatter wired up in `package.json` — don't invent one.
+
+`npm run build` runs `npm run generate:version` first (`scripts/generate-version.js`), which writes `src/version.ts` from `git rev-parse`. The file is gitignored and regenerated each build; it powers the `User-Agent` header (`ShopCallCrawler/<version>+<hash>`).
+
+Systemd install/reset/restore lives in `scripts/install.sh`, `scripts/reset.sh`, `scripts/restore.sh` — install copies the built `dist/` to `/opt/webshop-scraper`, reads `.env` from the project dir, and creates a `webshop-scraper` systemd unit.
+
+## Architecture
+
+A TypeScript Express service that scrapes any website (webshops or generic), classifies content into 13 fixed categories, stores in SQLite, optionally embeds into Qdrant for semantic search, and exposes the result via REST + MCP.
+
+### Single-process pipeline
+
+`src/index.ts` wires everything in one process:
+
+```
+HTTP request → JobQueue (in-memory, EventEmitter, max N concurrent)
+                 │
+                 ▼  emits 'process-job'
+              WebScraper.scrape(baseUrl, siteId)
+                 ├─ SitemapParser: detect platform + resolve sitemap (custom → robots.txt → fallback)
+                 ├─ LinkDiscovery: BFS same-origin crawl, capped by site.max_crawl_depth / max_pages
+                 ├─ filter via UrlExclusions (glob patterns) and disabled-content URLs
+                 ├─ merge in CustomUrls (manually added)
+                 ├─ ContentExtractor: fetch + turndown → markdown
+                 ├─ SiteDatabase.saveContent: SHA-256 hash, skip-if-unchanged
+                 └─ QdrantEmbeddingWorkflow: chunk → embed via OpenRouter → upsert
+              ScrapeScheduler (node-cron, every minute): re-queues jobs from `scheduled_jobs`
+```
+
+There is no message broker — `JobQueue` is a plain `EventEmitter` over an in-memory `Map`. `index.ts` listens for `process-job` and runs the scrape. Jobs do not survive a process restart; `scheduled_jobs` (DB) is what survives.
+
+### Site types and platforms
+
+Two **site types** drive scraping behavior:
+- `webshop` — ShopRenter, WooCommerce, Shopify. Sitemap-driven.
+- `website` — anything else. Falls back to BFS link discovery when no sitemap is found.
+
+Four **site platforms**: `shoprenter`, `woocommerce`, `shopify`, `generic`. Detected by `detectPlatform()` in `src/scraper/SitemapParser.ts` (hostname patterns + homepage HTML sniffing for WooCommerce).
+
+### 13 content categories (canonical list)
+
+`shipping`, `contacts`, `terms`, `faq`, `services`, `about`, `team`, `blog`, `pricing`, `testimonials`, `gallery`, `landing`, `other`. Defined in `src/types/index.ts` (`ContentType`, `ALL_CONTENT_TYPES`) and used identically across DB columns, REST responses, Qdrant collection names (`{custom_id}-{category}`), and MCP tool names (`{category}_search`). Adding a 14th category means updating types, classifier, MCP tool registry (`src/mcp/McpServer.ts`), and Qdrant collection management.
+
+### Self-documenting API
+
+Don't add routes by calling `app.get(...)` directly. The pattern:
+
+1. Each domain lives in `src/api/components/*Endpoint.ts`, extends `BaseEndpointComponent` from `src/api/core/types.ts`.
+2. `getEndpoints()` returns `EndpointMethod[]` with handler **and** JSON Schema documentation (parameters, requestBody, responses, `requiresAuth`).
+3. `src/api/routes.ts` instantiates components, registers them with `EndpointRegistry`, and the registry builds the Express router *and* the `/doc` + `/doc/openapi` outputs from the same metadata.
+4. `requiresAuth: true` makes `authMiddleware` (Bearer token vs `API_KEY`) prepend automatically. `/health`, `/doc*`, `/ui/*`, `/mcp*` are public.
+
+When adding an endpoint: edit the relevant `*Endpoint.ts`, add to the array, fill out the schema. Don't add a separate doc — the OpenAPI spec generates from the schema you wrote.
+
+### Legacy `shops` ↔ `sites` compatibility (important)
+
+The project was originally "webshop scraper" (shops). It was renamed to "sites" but **both surfaces are live**:
+
+- **Database**: `migrateShopsToSites()` in `src/database/Database.ts` renames tables/columns on first boot. The DB file is still `data/shops.db` (preserved for back-compat); fresh installs would create `sites.db` but existing deployments keep `shops.db`. Don't rename it.
+- **REST**: `routes.ts` clones every `/api/sites/*` (and `/api/qdrant/sites/*`, `/api/mcp/sites/*`) route into `/api/shops/*` with `:siteId` → `:id`. The clones run through `legacyShopAlias` middleware (`src/api/middleware/legacyShopAlias.ts`) which monkey-patches `res.json()` to recursively rename keys: `site_id → shop_id`, `sites → shops`, `site_type → webshop_type`, etc. New code should use the site-based names; legacy callers keep working unchanged.
+- **MCP tools**: accept both `site_id` and `shop_id` as the parameter name.
+
+If you add a new `site_*` key in a response and legacy clients need it, add it to `KEY_MAP` in `legacyShopAlias.ts`.
+
+### Identifiers
+
+Sites have **two** identifiers:
+- `id` — internal UUID, generated.
+- `custom_id` — optional client-provided UUID. Becomes **immutable once Qdrant is enabled** for that site (would orphan collections otherwise). Qdrant collection names are derived from `custom_id`, not `id`.
+
+`SiteDatabase.getSiteByAnyId(id)` accepts either. REST path params named `:id` or `:siteId` accept either.
+
+### Qdrant + embeddings
+
+Optional, gated on `QDRANT_ENABLED=true` server-wide **and** `qdrant_enabled=true` per-site. Pipeline:
+
+1. After `saveContent` writes a row, `QdrantEmbeddingWorkflow.processContentBatch()` runs.
+2. `ChunkingService` splits content into chunks.
+3. `EmbeddingService` calls OpenRouter (uses the `openai` client pointed at OpenRouter) for vectors.
+4. `QdrantService` upserts into collection `{custom_id}-{category}`. Point ID is deterministic: `SHA256(url + content_hash + chunk_index)` — same URL + same content ⇒ same point ID, so re-embedding is a no-op; content changes yield a fresh point and orphan the old one.
+5. Deletions are deferred — they go into `qdrant_deletion_queue` and `QdrantCleanupService` actually deletes them after `QDRANT_DELETION_DELAY_HOURS` (default 24).
+6. There is a v1→v2 migration path (`QdrantMigrationService`) from the old per-URL collection layout (`{custom_id}-{category}_{url_hash}`) to the consolidated layout.
+
+Failures are logged into the `qdrant_errors` table, surfaced via `GET /api/sites/:siteId/qdrant/errors`.
+
+### MCP server
+
+Mounted at `/mcp` (Streamable HTTP, stateless) only when **both** `MCP_ENABLED=true` and `QDRANT_ENABLED=true`. `src/mcp/McpServer.ts` registers 14 tools (1 `list_contents` + 13 `*_search`, one per category). Tool classes live in `src/mcp/tools/*.ts`, all extending `BaseTool`. Adding a tool: create the class, register it in `initializeTools()`, and (if it's a new category) update everything in the "13 content categories" section above. Every tool call is logged to `mcp_logs` via `McpLogger`; REST `/api/sites/:id/search` also logs to `mcp_logs` with `tool_name: "rest_search"` so analytics show traffic from both transports together.
+
+If the site is unknown or `qdrant_enabled` is false, tools return `"Search not available for this site"` rather than erroring — preserve this behavior; agents downstream rely on it.
+
+### Database
+
+SQLite via `better-sqlite3`. Schema applied at startup from `src/database/Database.ts` — no separate migration step, the constructor runs `migrateShopsToSites()`, `migrateSiteTypeToPlatform()`, then `initializeTables()`. File path: `data/shops.db` if it exists, else `data/sites.db`.
+
+`saveContent` deduplicates **across content types**: if the same `content_hash` already exists for the site in another category, the new row is skipped (returns `{ skipped: true }`) and no embedding/webhook is fired. This is intentional — pages that classify into multiple categories should only be embedded once.
+
+### Webhooks
+
+Per-site events fired by `WebhookManager`: `scrape_started`, `scrape_completed`, `scrape_failed`, `content_changed`. Deliveries are recorded in `webhook_deliveries` (success/failed + response code).
+
+## Conventions
+
+- Path-style references: use `file_path:line_number` so the user can jump.
+- Logs use the singleton `logger` from `src/utils/logger.ts`. HTTP access logs are emitted from `server.ts` with icon-prefixed status (`✓` 2xx, `↪️` 3xx, `⚠️` 4xx, `❌` 5xx); set `LOG_HTTP_VERBOSE=true` to include UA + content length.
+- `src/version.ts` is gitignored and machine-generated — don't edit it, don't import build metadata from anywhere else.
+- The web UI (`web/`) is a separate Vite + Tailwind project built into `dist/web` and served at `/ui` from the Node server. It has its own `package.json` and `tsconfig.json`.