webshop-scraper/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Working with this repo

• Never start the server yourself — ask the user. npm start / npm run dev are long-running and the user runs them manually.
• The live server is the source of truth for API surface. Browse http://localhost:3000/doc (HTML) or /doc/openapi (OpenAPI 3.0 JSON) instead of duplicating route docs in code/notes.

Commands

npm install        # also runs `npm install` inside web/
npm run build      # tsc → dist/, then `cd web && npm run build` → dist/web/
npm run dev        # ts-node, no build
npm run watch      # tsc -w
npm run dev:web    # vite dev server for the UI only

There is no test runner, lint, or formatter wired up in package.json — don't invent one.

npm run build runs npm run generate:version first (scripts/generate-version.js), which writes src/version.ts from git rev-parse. The file is gitignored and regenerated each build; it powers the User-Agent header (ShopCallCrawler/<version>+<hash>).

Systemd install/reset/restore lives in scripts/install.sh, scripts/reset.sh, scripts/restore.sh — install copies the built dist/ to /opt/webshop-scraper, reads .env from the project dir, and creates a webshop-scraper systemd unit.

Architecture

A TypeScript Express service that scrapes any website (webshops or generic), classifies content into 13 fixed categories, stores in SQLite, optionally embeds into Qdrant for semantic search, and exposes the result via REST + MCP.

Single-process pipeline

src/index.ts wires everything in one process:

HTTP request → JobQueue (in-memory, EventEmitter, max N concurrent)
                 │
                 ▼  emits 'process-job'
              WebScraper.scrape(baseUrl, siteId)
                 ├─ SitemapParser: detect platform + resolve sitemap (custom → robots.txt → fallback)
                 ├─ LinkDiscovery: BFS same-origin crawl, capped by site.max_crawl_depth / max_pages
                 ├─ filter via UrlExclusions (glob patterns) and disabled-content URLs
                 ├─ merge in CustomUrls (manually added)
                 ├─ ContentExtractor: fetch + turndown → markdown
                 ├─ SiteDatabase.saveContent: SHA-256 hash, skip-if-unchanged
                 └─ QdrantEmbeddingWorkflow: chunk → embed via OpenRouter → upsert
              ScrapeScheduler (node-cron, every minute): re-queues jobs from `scheduled_jobs`

There is no message broker — JobQueue is a plain EventEmitter over an in-memory Map. index.ts listens for process-job and runs the scrape. Jobs do not survive a process restart; scheduled_jobs (DB) is what survives.

Site types and platforms

Two site types drive scraping behavior:

• webshop — ShopRenter, WooCommerce, Shopify. Sitemap-driven.
• website — anything else. Falls back to BFS link discovery when no sitemap is found.

Four site platforms: shoprenter, woocommerce, shopify, generic. Detected by detectPlatform() in src/scraper/SitemapParser.ts (hostname patterns + homepage HTML sniffing for WooCommerce).

13 content categories (canonical list)

shipping, contacts, terms, faq, services, about, team, blog, pricing, testimonials, gallery, landing, other. Defined in src/types/index.ts (ContentType, ALL_CONTENT_TYPES) and used identically across DB columns, REST responses, Qdrant collection names ({custom_id}-{category}), and MCP tool names ({category}_search). Adding a 14th category means updating types, classifier, MCP tool registry (src/mcp/McpServer.ts), and Qdrant collection management.

Self-documenting API

Don't add routes by calling app.get(...) directly. The pattern:

1. Each domain lives in src/api/components/*Endpoint.ts, extends BaseEndpointComponent from src/api/core/types.ts.
2. getEndpoints() returns EndpointMethod[] with handler and JSON Schema documentation (parameters, requestBody, responses, requiresAuth).
3. src/api/routes.ts instantiates components, registers them with EndpointRegistry, and the registry builds the Express router and the /doc + /doc/openapi outputs from the same metadata.
4. requiresAuth: true makes authMiddleware (Bearer token vs API_KEY) prepend automatically. /health, /doc*, /ui/*, /mcp* are public.

When adding an endpoint: edit the relevant *Endpoint.ts, add to the array, fill out the schema. Don't add a separate doc — the OpenAPI spec generates from the schema you wrote.

Legacy shops ↔ sites compatibility (important)

The project was originally "webshop scraper" (shops). It was renamed to "sites" but both surfaces are live:

• Database: migrateShopsToSites() in src/database/Database.ts renames tables/columns on first boot. The DB file is still data/shops.db (preserved for back-compat); fresh installs would create sites.db but existing deployments keep shops.db. Don't rename it.
• REST: routes.ts clones every /api/sites/* (and /api/qdrant/sites/*, /api/mcp/sites/*) route into /api/shops/* with :siteId → :id. The clones run through legacyShopAlias middleware (src/api/middleware/legacyShopAlias.ts) which monkey-patches res.json() to recursively rename keys: site_id → shop_id, sites → shops, site_type → webshop_type, etc. New code should use the site-based names; legacy callers keep working unchanged.
• MCP tools: accept both site_id and shop_id as the parameter name.

If you add a new site_* key in a response and legacy clients need it, add it to KEY_MAP in legacyShopAlias.ts.

Identifiers

Sites have two identifiers:

• id — internal UUID, generated.
• custom_id — optional client-provided UUID. Becomes immutable once Qdrant is enabled for that site (would orphan collections otherwise). Qdrant collection names are derived from custom_id, not id.

SiteDatabase.getSiteByAnyId(id) accepts either. REST path params named :id or :siteId accept either.

Qdrant + embeddings

Optional, gated on QDRANT_ENABLED=true server-wide and qdrant_enabled=true per-site. Pipeline:

1. After saveContent writes a row, QdrantEmbeddingWorkflow.processContentBatch() runs.
2. ChunkingService splits content into chunks.
3. EmbeddingService calls OpenRouter (uses the openai client pointed at OpenRouter) for vectors.
4. QdrantService upserts into collection {custom_id}-{category}. Point ID is deterministic: SHA256(url + content_hash + chunk_index) — same URL + same content ⇒ same point ID, so re-embedding is a no-op; content changes yield a fresh point and orphan the old one.
5. Deletions are deferred — they go into qdrant_deletion_queue and QdrantCleanupService actually deletes them after QDRANT_DELETION_DELAY_HOURS (default 24).
6. There is a v1→v2 migration path (QdrantMigrationService) from the old per-URL collection layout ({custom_id}-{category}_{url_hash}) to the consolidated layout.

Failures are logged into the qdrant_errors table, surfaced via GET /api/sites/:siteId/qdrant/errors.

MCP server

Mounted at /mcp (Streamable HTTP, stateless) only when both MCP_ENABLED=true and QDRANT_ENABLED=true. src/mcp/McpServer.ts registers 14 tools (1 list_contents + 13 *_search, one per category). Tool classes live in src/mcp/tools/*.ts, all extending BaseTool. Adding a tool: create the class, register it in initializeTools(), and (if it's a new category) update everything in the "13 content categories" section above. Every tool call is logged to mcp_logs via McpLogger; REST /api/sites/:id/search also logs to mcp_logs with tool_name: "rest_search" so analytics show traffic from both transports together.

If the site is unknown or qdrant_enabled is false, tools return "Search not available for this site" rather than erroring — preserve this behavior; agents downstream rely on it.

Database

SQLite via better-sqlite3. Schema applied at startup from src/database/Database.ts — no separate migration step, the constructor runs migrateShopsToSites(), migrateSiteTypeToPlatform(), then initializeTables(). File path: data/shops.db if it exists, else data/sites.db.

saveContent deduplicates across content types: if the same content_hash already exists for the site in another category, the new row is skipped (returns { skipped: true }) and no embedding/webhook is fired. This is intentional — pages that classify into multiple categories should only be embedded once.

Webhooks

Per-site events fired by WebhookManager: scrape_started, scrape_completed, scrape_failed, content_changed. Deliveries are recorded in webhook_deliveries (success/failed + response code).

Conventions

• Path-style references: use file_path:line_number so the user can jump.
• Logs use the singleton logger from src/utils/logger.ts. HTTP access logs are emitted from server.ts with icon-prefixed status (✓ 2xx, ↪️ 3xx, ⚠️ 4xx, ❌ 5xx); set LOG_HTTP_VERBOSE=true to include UA + content length.
• src/version.ts is gitignored and machine-generated — don't edit it, don't import build metadata from anywhere else.
• The web UI (web/) is a separate Vite + Tailwind project built into dist/web and served at /ui from the Node server. It has its own package.json and tsconfig.json.