webshop-scraper

Web Scraper

A TypeScript service that scrapes shipping, contact, terms, and FAQ pages from sites (ShopRenter, WooCommerce, Shopify), stores them in SQLite with change detection, and optionally indexes them into Qdrant for semantic search via MCP tools.

Features

• Multi-platform scraping — ShopRenter, WooCommerce, Shopify
• Sitemap-driven — parses sitemaps, classifies pages into shipping / contacts / terms / faq
• Scheduled scraping — automatic re-scrape based on sitemap changefreq rules, toggleable per site
• Change detection — SHA-256 content hashing; unchanged pages are not re-processed
• Persistent storage — SQLite (better-sqlite3) with site analytics and scrape history
• Optional vector search — Qdrant + OpenRouter embeddings, one collection per {custom_id}-{category}
• MCP server — Streamable HTTP MCP endpoint at /mcp exposing list_contents, shipping_search, contacts_search, terms_search, faq_search
• Self-documenting REST API — endpoint metadata is rendered as HTML at /doc and OpenAPI 3.0 at /doc/openapi
• Bearer authentication for all /api/* endpoints
• Webhooks per site — scrape_started / scrape_completed / scrape_failed
• gzip/brotli compression on all responses over 1 KB
• Web UI served at /ui
• systemd installer under scripts/

Quick start

git clone <repository-url>
cd webshop-scraper
npm install
npm run build
cp .env.example .env   # then edit API_KEY and anything else you need
npm start

Then open:

• http://localhost:3000/doc — live API reference (HTML)
• http://localhost:3000/ui — web dashboard
• http://localhost:3000/health — liveness and queue stats (no auth)

Minimal configuration

Only API_KEY is required. See docs/SETUP.md for the full environment variable reference.

API_KEY=change-me
PORT=3000
MAX_CONCURRENT_JOBS=3

For Qdrant + MCP, add:

QDRANT_ENABLED=true
QDRANT_API_URL=http://localhost:6333
OPENROUTER_API_KEY=sk-or-v1-...
MCP_ENABLED=true

Authentication

Every /api/* endpoint requires a Bearer token matching API_KEY:

curl http://localhost:3000/api/sites \
  -H "Authorization: Bearer $API_KEY"

/health, /doc*, /ui/*, and /mcp* are public.

Usage examples

Create a scraping job:

curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'

Get site details:

curl http://localhost:3000/api/sites/<id> \
  -H "Authorization: Bearer $API_KEY"

Get site results filtered to shipping content, last 10 entries:

curl "http://localhost:3000/api/sites/<id>/results?content_type=shipping&limit=10" \
  -H "Authorization: Bearer $API_KEY"

Semantic search across scraped content (requires Qdrant + OpenRouter):

curl -X POST http://localhost:3000/api/sites/<id>/search \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q":"do you ship to Germany","limit":5}'

Omit category to search all four content buckets in parallel. Add ?format=text for voice/LLM output.

<id> can be either the system-generated UUID or the optional custom_id you assigned to the site.

MCP tools

When MCP_ENABLED=true and QDRANT_ENABLED=true, the server mounts an MCP Streamable HTTP endpoint at /mcp. Five tools are registered:

• list_contents(site_id) — enumerate available pages per category, tagged with the next tool to use
• shipping_search(site_id, query, limit?)
• contacts_search(site_id, query, limit?)
• terms_search(site_id, query, limit?)
• faq_search(site_id, query, limit?)

All tools accept both site_id and shop_id (legacy alias) as the parameter name, mapped to the site's custom_id. Tools return a friendly "Search not available for this site" message when the site is unknown or Qdrant is not enabled on it.

Documentation

• docs/API.md — API concepts, auth, endpoint surface map, MCP tool inventory. For the full endpoint reference use /doc on the running server.
• docs/SETUP.md — installation, environment variables, Qdrant/MCP setup, migration, troubleshooting.

Development

npm run dev     # run with ts-node, no build
npm run watch   # tsc in watch mode
npm run build   # build TS + web UI

Systemd service

sudo ./scripts/install.sh   # install as service
sudo ./scripts/reset.sh     # wipe to clean state (backs up first)
sudo ./scripts/restore.sh   # restore from backup

Backups land in /var/backups/webshop-scraper/.

Data

SQLite lives in data/shops.db. Schema is applied at startup from src/database/Database.ts. It tracks sites (with UUID and optional custom_id), scrape history, per-page content with hashes, scheduled jobs, webhooks, and — when Qdrant is enabled — embedding state, deletion queue, error log, and MCP call logs.

Logs

HTTP access logs are emitted per request with status code icon, path, IP, response time, and (with LOG_HTTP_VERBOSE=true) user agent and content length.

[INFO] [WebScraper] ✓ 200 GET /api/sites ::1 45ms
[INFO] [WebScraper] ⚠️ 404 GET /api/sites/invalid ::1 12ms
[INFO] [WebScraper] ❌ 500 POST /api/jobs ::1 234ms

Icons: ✓ 2xx, ↪️ 3xx, ⚠️ 4xx, ❌ 5xx.

License

ISC