webshop-scraper

Web Scraper

A TypeScript service that scrapes and classifies content from any website — webshops (ShopRenter, WooCommerce, Shopify) and generic websites alike. Content is stored in SQLite with change detection and optionally indexed into Qdrant for semantic search via MCP tools or REST API.

Features

• Universal scraping — supports two site types: webshop (ShopRenter, WooCommerce, Shopify) and website (any generic site)
• 13 content categories — shipping, contacts, terms, faq, services, about, team, blog, pricing, testimonials, gallery, landing, other
• Smart content discovery — sitemap-driven for webshops; automatic link discovery engine (BFS crawl) for sites without a sitemap, with configurable max_crawl_depth and max_pages
• URL exclusions — glob patterns to skip unwanted URLs during scraping
• Auto-detect — POST /api/sites/detect probes a URL to determine site type, platform, and sitemap
• Custom sitemap — override the auto-detected sitemap with your own URL
• Scheduled scraping — automatic re-scrape based on sitemap changefreq rules, toggleable per site
• Change detection — SHA-256 content hashing; unchanged pages are not re-processed
• Persistent storage — SQLite (better-sqlite3) with site analytics and scrape history
• Optional vector search — Qdrant + OpenRouter embeddings, one collection per {custom_id}-{category}
• MCP server — Streamable HTTP MCP endpoint at /mcp exposing 14 tools: list_contents plus a *_search tool for each of the 13 content categories
• Self-documenting REST API — endpoint metadata is rendered as HTML at /doc and OpenAPI 3.0 at /doc/openapi
• Bearer authentication for all /api/* endpoints
• Webhooks per site — scrape_started / scrape_completed / scrape_failed
• gzip/brotli compression on all responses over 1 KB
• Web UI served at /ui
• systemd installer under scripts/

Quick start

git clone <repository-url>
cd webshop-scraper
npm install
npm run build
cp .env.example .env   # then edit API_KEY and anything else you need
npm start

Then open:

• http://localhost:3000/doc — live API reference (HTML)
• http://localhost:3000/ui — web dashboard
• http://localhost:3000/health — liveness and queue stats (no auth)

Minimal configuration

Only API_KEY is required. See docs/SETUP.md for the full environment variable reference.

API_KEY=change-me
PORT=3000
MAX_CONCURRENT_JOBS=3

For Qdrant + MCP, add:

QDRANT_ENABLED=true
QDRANT_API_URL=http://localhost:6333
OPENROUTER_API_KEY=sk-or-v1-...
MCP_ENABLED=true

Authentication

Every /api/* endpoint requires a Bearer token matching API_KEY:

curl http://localhost:3000/api/sites \
  -H "Authorization: Bearer $API_KEY"

/health, /doc*, /ui/*, and /mcp* are public.

Usage examples

Create a scraping job:

curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'

Get site details:

curl http://localhost:3000/api/sites/<id> \
  -H "Authorization: Bearer $API_KEY"

Get site results filtered to shipping content, last 10 entries:

curl "http://localhost:3000/api/sites/<id>/results?content_type=shipping&limit=10" \
  -H "Authorization: Bearer $API_KEY"

Semantic search across scraped content (requires Qdrant + OpenRouter):

curl -X POST http://localhost:3000/api/sites/<id>/search \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q":"do you ship to Germany","limit":5}'

Omit category to search all 13 content buckets in parallel. Add ?format=text for voice/LLM output.

<id> can be either the system-generated UUID or the optional custom_id you assigned to the site.

MCP tools

When MCP_ENABLED=true and QDRANT_ENABLED=true, the server mounts an MCP Streamable HTTP endpoint at /mcp. 14 tools are registered:

• list_contents(site_id) — enumerate available pages per category, tagged with the next tool to use
• shipping_search(site_id, query, limit?)
• contacts_search(site_id, query, limit?)
• terms_search(site_id, query, limit?)
• faq_search(site_id, query, limit?)
• services_search(site_id, query, limit?)
• about_search(site_id, query, limit?)
• team_search(site_id, query, limit?)
• blog_search(site_id, query, limit?)
• pricing_search(site_id, query, limit?)
• testimonials_search(site_id, query, limit?)
• gallery_search(site_id, query, limit?)
• landing_search(site_id, query, limit?)
• other_search(site_id, query, limit?)

All tools accept both site_id and shop_id (legacy alias) as the parameter name, mapped to the site's custom_id. Tools return a friendly "Search not available for this site" message when the site is unknown or Qdrant is not enabled on it.

Documentation

• docs/API.md — API concepts, auth, endpoint surface map, MCP tool inventory. For the full endpoint reference use /doc on the running server.
• docs/SETUP.md — installation, environment variables, Qdrant/MCP setup, migration, troubleshooting.

Development

npm run dev     # run with ts-node, no build
npm run watch   # tsc in watch mode
npm run build   # build TS + web UI

Systemd service

sudo ./scripts/install.sh   # install as service
sudo ./scripts/reset.sh     # wipe to clean state (backs up first)
sudo ./scripts/restore.sh   # restore from backup

Backups land in /var/backups/webshop-scraper/.

Data

SQLite lives in data/shops.db. Schema is applied at startup from src/database/Database.ts. It tracks sites (with UUID and optional custom_id), scrape history, per-page content with hashes, scheduled jobs, webhooks, and — when Qdrant is enabled — embedding state, deletion queue, error log, and MCP call logs.

Logs

HTTP access logs are emitted per request with status code icon, path, IP, response time, and (with LOG_HTTP_VERBOSE=true) user agent and content length.

[INFO] [WebScraper] ✓ 200 GET /api/sites ::1 45ms
[INFO] [WebScraper] ⚠️ 404 GET /api/sites/invalid ::1 12ms
[INFO] [WebScraper] ❌ 500 POST /api/jobs ::1 234ms

Icons: ✓ 2xx, ↪️ 3xx, ⚠️ 4xx, ❌ 5xx.

License

ISC