# Webshop Scraper

A TypeScript service that scrapes shipping, contact, terms, and FAQ pages from webshops (ShopRenter, WooCommerce, Shopify), stores them in SQLite with change detection, and optionally indexes them into Qdrant for semantic search via MCP tools.

## Features

- **Multi-platform scraping** — ShopRenter, WooCommerce, Shopify
- **Sitemap-driven** — parses sitemaps, classifies pages into `shipping` / `contacts` / `terms` / `faq`
- **Scheduled scraping** — automatic re-scrape based on sitemap `changefreq` rules, toggleable per shop
- **Change detection** — SHA-256 content hashing; unchanged pages are not re-processed
- **Persistent storage** — SQLite (`better-sqlite3`) with shop analytics and scrape history
- **Optional vector search** — Qdrant + OpenRouter embeddings, one collection per `{custom_id}-{category}`
- **MCP server** — Streamable HTTP MCP endpoint at `/mcp` exposing `list_contents`, `shipping_search`, `contacts_search`, `terms_search`, `faq_search`
- **Self-documenting REST API** — endpoint metadata is rendered as HTML at `/doc` and OpenAPI 3.0 at `/doc/openapi`
- **Bearer authentication** for all `/api/*` endpoints
- **Webhooks** per shop — `scrape_started` / `scrape_completed` / `scrape_failed`
- **gzip/brotli compression** on all responses over 1 KB
- **Web UI** served at `/ui`
- **systemd installer** under `scripts/`

## Quick start

```bash
git clone <repository-url>
cd webshop-scraper
npm install
npm run build
cp .env.example .env   # then edit API_KEY and anything else you need
npm start
```

Then open:

- `http://localhost:3000/doc` — live API reference (HTML)
- `http://localhost:3000/ui` — web dashboard
- `http://localhost:3000/health` — liveness and queue stats (no auth)

## Minimal configuration

Only `API_KEY` is required. See [`docs/SETUP.md`](./docs/SETUP.md) for the full environment variable reference.

```bash
API_KEY=change-me
PORT=3000
MAX_CONCURRENT_JOBS=3
```

For Qdrant + MCP, add:

```bash
QDRANT_ENABLED=true
QDRANT_API_URL=http://localhost:6333
OPENROUTER_API_KEY=sk-or-v1-...
MCP_ENABLED=true
```

## Authentication

Every `/api/*` endpoint requires a Bearer token matching `API_KEY`:

```bash
curl http://localhost:3000/api/shops \
  -H "Authorization: Bearer $API_KEY"
```

`/health`, `/doc*`, `/ui/*`, and `/mcp*` are public.

## Usage examples

**Create a scraping job:**

```bash
curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'
```

**Get shop details:**

```bash
curl http://localhost:3000/api/shops/<id> \
  -H "Authorization: Bearer $API_KEY"
```

**Get shop results filtered to shipping content, last 10 entries:**

```bash
curl "http://localhost:3000/api/shops/<id>/results?content_type=shipping&limit=10" \
  -H "Authorization: Bearer $API_KEY"
```

**Semantic search across scraped content** (requires Qdrant + OpenRouter):

```bash
curl -X POST http://localhost:3000/api/shops/<id>/search \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q":"do you ship to Germany","limit":5}'
```

Omit `category` to search all four content buckets in parallel. Add `?format=text` for voice/LLM output.

`<id>` can be either the system-generated UUID or the optional `custom_id` you assigned to the shop.

## MCP tools

When `MCP_ENABLED=true` and `QDRANT_ENABLED=true`, the server mounts an MCP Streamable HTTP endpoint at `/mcp`. Five tools are registered:

- `list_contents(shop_id)` — enumerate available pages per category, tagged with the next tool to use
- `shipping_search(shop_id, query, limit?)`
- `contacts_search(shop_id, query, limit?)`
- `terms_search(shop_id, query, limit?)`
- `faq_search(shop_id, query, limit?)`

All tools take the shop's `custom_id` as `shop_id`. Tools return a friendly "Search not available for this shop" message when the shop is unknown or Qdrant is not enabled on it.

## Documentation

- [`docs/API.md`](./docs/API.md) — API concepts, auth, endpoint surface map, MCP tool inventory. For the full endpoint reference use `/doc` on the running server.
- [`docs/SETUP.md`](./docs/SETUP.md) — installation, environment variables, Qdrant/MCP setup, migration, troubleshooting.

## Development

```bash
npm run dev     # run with ts-node, no build
npm run watch   # tsc in watch mode
npm run build   # build TS + web UI
```

## Systemd service

```bash
sudo ./scripts/install.sh   # install as service
sudo ./scripts/reset.sh     # wipe to clean state (backs up first)
sudo ./scripts/restore.sh   # restore from backup
```

Backups land in `/var/backups/webshop-scraper/`.

## Data

SQLite lives in `data/shops.db`. Schema is applied at startup from `src/database/Database.ts`. It tracks shops (with UUID and optional `custom_id`), scrape history, per-page content with hashes, scheduled jobs, webhooks, and — when Qdrant is enabled — embedding state, deletion queue, error log, and MCP call logs.

## Logs

HTTP access logs are emitted per request with status code icon, path, IP, response time, and (with `LOG_HTTP_VERBOSE=true`) user agent and content length.

```
[INFO] [WebshopScraper] ✓ 200 GET /api/shops ::1 45ms
[INFO] [WebshopScraper] ⚠️ 404 GET /api/shops/invalid ::1 12ms
[INFO] [WebshopScraper] ❌ 500 POST /api/jobs ::1 234ms
```

Icons: `✓` 2xx, `↪️` 3xx, `⚠️` 4xx, `❌` 5xx.

## License

ISC