Home Verkennen Help
Inloggen
fszontagh
/
webshop-scraper
1
0
Vork 0
Bestanden Issues 4 Pull-aanvragen 0 Wiki
Boom: b37492f482
Aftakkingen Labels
webshop-scraper
/
docs
/
SETUP.md

SETUP.md 7.9 KB
Geschiedenis Ruwe

Setup Guide

This document walks through installing, configuring, and running the webshop scraper — including the optional Qdrant vector search and MCP server.

Requirements

  • Node.js 16 or newer
  • npm
  • SQLite (ships via better-sqlite3, no external server needed)
  • Linux with systemd (only if you want to install it as a service)
  • Optionally: a Qdrant instance and an OpenRouter API key for semantic search

Install

git clone <repository-url>
cd webshop-scraper
npm install

The npm install step will also install dependencies for the web UI under web/.

Configure

Copy .env.example to .env and fill in values:

cp .env.example .env

Environment variables

All environment variables read by the code live in src/config/index.ts. Nothing else is consulted.

Variable Required Default Purpose
API_KEY yes — Bearer token required for every /api/* endpoint. Startup fails without it.
HOST no 0.0.0.0 HTTP bind address. Use 127.0.0.1 to only accept local connections.
PORT no 3000 HTTP port.
MAX_CONCURRENT_JOBS no 3 Parallel scraping worker slots.
LOG_HTTP_VERBOSE no false When true, HTTP access logs include user agent and content length.
QDRANT_ENABLED no false Master switch for the Qdrant integration. Must be true for embeddings and MCP search to work.
QDRANT_API_URL no http://localhost:6333 Qdrant HTTP endpoint.
QDRANT_API_KEY no (empty) Qdrant API key; leave empty for unauthenticated local instances.
QDRANT_DELETION_DELAY_HOURS no 24 Grace period before queued Qdrant deletions actually run.
OPENROUTER_API_KEY no (empty) Required if QDRANT_ENABLED=true; used to generate embeddings.
MCP_ENABLED no false Mount the MCP Streamable HTTP endpoints at /mcp. Also requires QDRANT_ENABLED=true.
MCP_PORT no 3001 Reserved port (the MCP server currently mounts onto the main HTTP server under /mcp).

Example minimal .env:

API_KEY=change-me
HOST=0.0.0.0
PORT=3000
MAX_CONCURRENT_JOBS=3

Example .env with Qdrant + MCP:

API_KEY=change-me
PORT=3000

QDRANT_ENABLED=true
QDRANT_API_URL=http://localhost:6333
QDRANT_API_KEY=
QDRANT_DELETION_DELAY_HOURS=24

OPENROUTER_API_KEY=sk-or-v1-...

MCP_ENABLED=true

Build

npm run build

This compiles TypeScript (tsc) and then runs npm run build:web to build the bundled web UI under web/. The web UI is served at /ui from the main HTTP server.

Run

Ask the user before starting the server — don't start it yourself.

# Production
npm start

# Development (ts-node, no build required)
npm run dev

# TypeScript watch mode
npm run watch

Once up:

  • API: http://<host>:<port>
  • Web UI: http://<host>:<port>/ui
  • Live API docs: http://<host>:<port>/doc
  • OpenAPI spec: http://<host>:<port>/doc/openapi

Install as a systemd service

sudo ./scripts/install.sh

Reset and restore

Both scripts live in scripts/ and create timestamped backups under /var/backups/webshop-scraper/.

sudo ./scripts/reset.sh    # wipe back to a clean state (backs up first)
sudo ./scripts/restore.sh  # restore from a previous backup

A backup contains:

  • data/shops.db (SQLite database)
  • .env
  • Service logs from journalctl

Qdrant vector search (optional)

The scraper can embed scraped content and push it to a Qdrant collection so downstream systems (or the MCP tools) can do semantic search over it. Everything is gated on QDRANT_ENABLED=true.

1. Run Qdrant

# Local Docker instance with persistent storage
docker run -p 6333:6333 -v $(pwd)/qdrant_storage:/qdrant/storage qdrant/qdrant

Or use Qdrant Cloud and point QDRANT_API_URL / QDRANT_API_KEY at your cluster.

2. Get an OpenRouter API key

The EmbeddingService (src/services/EmbeddingService.ts) calls OpenRouter through the openai client. Sign up at https://openrouter.ai, create a key, and put it in OPENROUTER_API_KEY.

3. Turn on the integration

Set QDRANT_ENABLED=true in .env and restart the server.

4. Enable Qdrant per shop

Embedding happens per shop — you opt each shop in.

  1. Make sure the shop has a custom_id set (PATCH /api/shops/:id/custom-id). Once Qdrant is enabled for a shop, its custom_id becomes immutable — this prevents orphaned collections.

  2. Toggle Qdrant on via either the web UI or the API:

    curl -X PUT http://localhost:3000/api/shops/<shopId>/qdrant/toggle \
 -H "Authorization: Bearer $API_KEY" \
 -H "Content-Type: application/json" \
 -d '{"enabled": true}'

  3. The next successful scrape of that shop triggers embedding generation via QdrantEmbeddingWorkflow.

5. Collection layout

  • One collection per {custom_id}-{category}, e.g. shop123-shipping, shop123-faq.
  • Categories: shipping, contacts, terms, faq.
  • Each page is stored as one or more chunks (via ChunkingService); point IDs are deterministic: SHA256(url + content_hash + chunk_index). Same URL + same content ⇒ same point ID, so re-embedding is a no-op; content changes yield a fresh point.

Verified in src/services/QdrantEmbeddingWorkflow.ts and src/services/QdrantService.ts.

6. Legacy → current migration

If you have data from a previous version that used per-URL collections ({custom_id}-{category}_{url_hash}), migrate using the per-shop endpoints:

# 1. See what would happen
curl -H "Authorization: Bearer $API_KEY" \
  http://localhost:3000/api/shops/<shopId>/qdrant/migration-status

# 2. Copy points into the new consolidated collections
curl -X POST -H "Authorization: Bearer $API_KEY" \
  http://localhost:3000/api/shops/<shopId>/qdrant/migrate

# 3. Dry-run cleanup of old collections
curl -X POST -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dryRun": true}' \
  http://localhost:3000/api/shops/<shopId>/qdrant/cleanup-old-collections

# 4. Actually delete the old collections
curl -X POST -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dryRun": false}' \
  http://localhost:3000/api/shops/<shopId>/qdrant/cleanup-old-collections

MCP (Model Context Protocol)

When MCP_ENABLED=true and QDRANT_ENABLED=true, src/api/server.ts mounts the MCP Streamable HTTP transport at /mcp. Five tools are exposed; see docs/API.md for the list and the shapes.

Point any MCP client at:

http://<host>:<port>/mcp

The server uses the stateless mode of StreamableHTTPServerTransport, so each request is self-contained.

Quick health check:

curl http://localhost:3000/mcp/health

Troubleshooting

Symptom Where to look
API_KEY environment variable is required at startup .env missing or not loaded; API_KEY must be set.
401 on every request Missing Authorization: Bearer ... header, wrong token, or using a different header (the server only accepts Bearer).
Shop exists but MCP tools return "Search not available" Either QDRANT_ENABLED=false or the shop's qdrant_enabled flag is false or the shop has no custom_id. Check GET /api/shops/:shopId/qdrant.
qdrant_errors piling up Inspect via GET /api/shops/:shopId/qdrant/errors; the likely culprits are OpenRouter quota, wrong OPENROUTER_API_KEY, or unreachable QDRANT_API_URL.
Embeddings are not being created after scrape Check the shop is toggled on (PUT /api/shops/:shopId/qdrant/toggle) and re-run with POST /api/shops/:shopId/qdrant/re-embed or /sync.

For logs:

# systemd
sudo journalctl -u webshop-scraper -f

# development
npm run dev

Database

SQLite file at data/shops.db. Schema lives in src/database/Database.ts and is applied on startup — no manual migration step. Qdrant-related tables (shop_embeddings, qdrant_deletion_queue, qdrant_errors, mcp_logs) are created automatically whether or not Qdrant is enabled.