webshop-scraper/docs/SETUP.md

Setup Guide

This document walks through installing, configuring, and running the web scraper — including the optional Qdrant vector search and MCP server.

Requirements

• Node.js 16 or newer
• npm
• SQLite (ships via better-sqlite3, no external server needed)
• Linux with systemd (only if you want to install it as a service)
• Optionally: a Qdrant instance and an OpenRouter API key for semantic search

Install

git clone <repository-url>
cd webshop-scraper
npm install

The npm install step will also install dependencies for the web UI under web/.

Configure

Copy .env.example to .env and fill in values:

cp .env.example .env

Environment variables

All environment variables read by the code live in src/config/index.ts. Nothing else is consulted.

Variable	Required	Default	Purpose
API_KEY	yes	—	Bearer token required for every /api/* endpoint. Startup fails without it.
HOST	no	0.0.0.0	HTTP bind address. Use 127.0.0.1 to only accept local connections.
PORT	no	3000	HTTP port.
MAX_CONCURRENT_JOBS	no	3	Parallel scraping worker slots.
LOG_HTTP_VERBOSE	no	false	When true, HTTP access logs include user agent and content length.
QDRANT_ENABLED	no	false	Master switch for the Qdrant integration. Must be true for embeddings and MCP search to work.
QDRANT_API_URL	no	http://localhost:6333	Qdrant HTTP endpoint.
QDRANT_API_KEY	no	(empty)	Qdrant API key; leave empty for unauthenticated local instances.
QDRANT_DELETION_DELAY_HOURS	no	24	Grace period before queued Qdrant deletions actually run.
OPENROUTER_API_KEY	no	(empty)	Required if QDRANT_ENABLED=true; used to generate embeddings.
MCP_ENABLED	no	false	Mount the MCP Streamable HTTP endpoints at /mcp. Also requires QDRANT_ENABLED=true.
MCP_PORT	no	3001	Reserved port (the MCP server currently mounts onto the main HTTP server under /mcp).

Example minimal .env:

API_KEY=change-me
HOST=0.0.0.0
PORT=3000
MAX_CONCURRENT_JOBS=3

Example .env with Qdrant + MCP:

API_KEY=change-me
PORT=3000

QDRANT_ENABLED=true
QDRANT_API_URL=http://localhost:6333
QDRANT_API_KEY=
QDRANT_DELETION_DELAY_HOURS=24

OPENROUTER_API_KEY=sk-or-v1-...

MCP_ENABLED=true

Site types

The scraper supports two site types, set when a job is created or updated via PATCH /api/sites/:id/settings:

• webshop — e-commerce sites built on ShopRenter, WooCommerce, or Shopify. The scraper uses the platform's sitemap to discover pages and classifies them into content categories.
• website — any generic website. When no sitemap is found, the scraper falls back to the link discovery engine (see below).

When you create a job with POST /api/jobs, the system auto-detects the site type and platform via POST /api/sites/detect. You can override the detected values with PATCH /api/sites/:id/settings.

Link discovery

When a site has no sitemap (common for generic websites), the scraper uses a BFS-based link discovery engine that crawls same-origin links starting from the site's base URL. Two settings control the crawl:

Setting	Default	Description
max_crawl_depth	3	Maximum link-follow depth from the start URL.
max_pages	200	Maximum number of pages to visit.

Both can be configured per site via PATCH /api/sites/:id/settings. Discovered URLs are then classified into content categories the same way sitemap URLs are.

You can also add URL exclusion patterns (POST /api/sites/:siteId/url-exclusions) using glob syntax (e.g. */admin/*, */wp-json/*) to skip unwanted URLs during scraping.

Build

npm run build

This compiles TypeScript (tsc) and then runs npm run build:web to build the bundled web UI under web/. The web UI is served at /ui from the main HTTP server.

Run

Ask the user before starting the server — don't start it yourself.

# Production
npm start

# Development (ts-node, no build required)
npm run dev

# TypeScript watch mode
npm run watch

Once up:

• API: http://<host>:<port>
• Web UI: http://<host>:<port>/ui
• Live API docs: http://<host>:<port>/doc
• OpenAPI spec: http://<host>:<port>/doc/openapi

Install as a systemd service

sudo ./scripts/install.sh

Reset and restore

Both scripts live in scripts/ and create timestamped backups under /var/backups/webshop-scraper/.

sudo ./scripts/reset.sh    # wipe back to a clean state (backs up first)
sudo ./scripts/restore.sh  # restore from a previous backup

A backup contains:

• data/shops.db (SQLite database)
• .env
• Service logs from journalctl

Qdrant vector search (optional)

The scraper can embed scraped content and push it to a Qdrant collection so downstream systems (or the MCP tools) can do semantic search over it. Everything is gated on QDRANT_ENABLED=true.

1. Run Qdrant

# Local Docker instance with persistent storage
docker run -p 6333:6333 -v $(pwd)/qdrant_storage:/qdrant/storage qdrant/qdrant

Or use Qdrant Cloud and point QDRANT_API_URL / QDRANT_API_KEY at your cluster.

2. Get an OpenRouter API key

The EmbeddingService (src/services/EmbeddingService.ts) calls OpenRouter through the openai client. Sign up at https://openrouter.ai, create a key, and put it in OPENROUTER_API_KEY.

3. Turn on the integration

Set QDRANT_ENABLED=true in .env and restart the server.

4. Enable Qdrant per site

Embedding happens per site — you opt each site in.

1. Make sure the site has a custom_id set (PATCH /api/sites/:id/custom-id). Once Qdrant is enabled for a site, its custom_id becomes immutable — this prevents orphaned collections.

2. Toggle Qdrant on via either the web UI or the API:

   curl -X PUT http://localhost:3000/api/sites/<siteId>/qdrant/toggle \
    -H "Authorization: Bearer $API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"enabled": true}'
   

3. The next successful scrape of that site triggers embedding generation via QdrantEmbeddingWorkflow.

5. Collection layout

• One collection per {custom_id}-{category}, e.g. shop123-shipping, shop123-faq.
• Categories: shipping, contacts, terms, faq, services, about, team, blog, pricing, testimonials, gallery, landing, other.
• Each page is stored as one or more chunks (via ChunkingService); point IDs are deterministic: SHA256(url + content_hash + chunk_index). Same URL + same content ⇒ same point ID, so re-embedding is a no-op; content changes yield a fresh point.

Verified in src/services/QdrantEmbeddingWorkflow.ts and src/services/QdrantService.ts.

6. Legacy → current migration

If you have data from a previous version that used per-URL collections ({custom_id}-{category}_{url_hash}), migrate using the per-site endpoints:

# 1. See what would happen
curl -H "Authorization: Bearer $API_KEY" \
  http://localhost:3000/api/sites/<siteId>/qdrant/migration-status

# 2. Copy points into the new consolidated collections
curl -X POST -H "Authorization: Bearer $API_KEY" \
  http://localhost:3000/api/sites/<siteId>/qdrant/migrate

# 3. Dry-run cleanup of old collections
curl -X POST -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dryRun": true}' \
  http://localhost:3000/api/sites/<siteId>/qdrant/cleanup-old-collections

# 4. Actually delete the old collections
curl -X POST -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dryRun": false}' \
  http://localhost:3000/api/sites/<siteId>/qdrant/cleanup-old-collections

MCP (Model Context Protocol)

When MCP_ENABLED=true and QDRANT_ENABLED=true, src/api/server.ts mounts the MCP Streamable HTTP transport at /mcp. 14 tools are exposed (1 listing + 13 search tools, one per content category); see docs/API.md for the list and the shapes.

Point any MCP client at:

http://<host>:<port>/mcp

The server uses the stateless mode of StreamableHTTPServerTransport, so each request is self-contained.

Quick health check:

curl http://localhost:3000/mcp/health

Troubleshooting

Symptom	Where to look
API_KEY environment variable is required at startup	.env missing or not loaded; API_KEY must be set.
401 on every request	Missing Authorization: Bearer ... header, wrong token, or using a different header (the server only accepts Bearer).
Site exists but MCP tools return "Search not available"	Either QDRANT_ENABLED=false or the site's qdrant_enabled flag is false or the site has no custom_id. Check GET /api/sites/:siteId/qdrant.
qdrant_errors piling up	Inspect via GET /api/sites/:siteId/qdrant/errors; the likely culprits are OpenRouter quota, wrong OPENROUTER_API_KEY, or unreachable QDRANT_API_URL.
Embeddings are not being created after scrape	Check the site is toggled on (PUT /api/sites/:siteId/qdrant/toggle) and re-run with POST /api/sites/:siteId/qdrant/re-embed or /sync.

For logs:

# systemd
sudo journalctl -u webshop-scraper -f

# development
npm run dev

Database

SQLite file at data/shops.db. Schema lives in src/database/SiteDatabase.ts and is applied on startup — no manual migration step. Qdrant-related tables (shop_embeddings, qdrant_deletion_queue, qdrant_errors, mcp_logs) are created automatically whether or not Qdrant is enabled.