# Web Scraper

A TypeScript service that scrapes shipping, contact, terms, and FAQ pages from sites (ShopRenter, WooCommerce, Shopify), stores them in SQLite with change detection, and optionally indexes them into Qdrant for semantic search via MCP tools.

## Features

- **Multi-platform scraping** — ShopRenter, WooCommerce, Shopify
- **Sitemap-driven** — parses sitemaps, classifies pages into `shipping` / `contacts` / `terms` / `faq`
- **Scheduled scraping** — automatic re-scrape based on sitemap `changefreq` rules, toggleable per site
- **Change detection** — SHA-256 content hashing; unchanged pages are not re-processed
- **Persistent storage** — SQLite (`better-sqlite3`) with site analytics and scrape history
- **Optional vector search** — Qdrant + OpenRouter embeddings, one collection per `{custom_id}-{category}`
- **MCP server** — Streamable HTTP MCP endpoint at `/mcp` exposing `list_contents`, `shipping_search`, `contacts_search`, `terms_search`, `faq_search`
- **Self-documenting REST API** — endpoint metadata is rendered as HTML at `/doc` and OpenAPI 3.0 at `/doc/openapi`
- **Bearer authentication** for all `/api/*` endpoints
- **Webhooks** per site — `scrape_started` / `scrape_completed` / `scrape_failed`
- **gzip/brotli compression** on all responses over 1 KB
- **Web UI** served at `/ui`
- **systemd installer** under `scripts/`

## Quick start

```bash
git clone <repository-url>
cd webshop-scraper
npm install
npm run build
cp .env.example .env   # then edit API_KEY and anything else you need
npm start
```

Then open:

- `http://localhost:3000/doc` — live API reference (HTML)
- `http://localhost:3000/ui` — web dashboard
- `http://localhost:3000/health` — liveness and queue stats (no auth)

## Minimal configuration

Only `API_KEY` is required. See [`docs/SETUP.md`](./docs/SETUP.md) for the full environment variable reference.

```bash
API_KEY=change-me
PORT=3000
MAX_CONCURRENT_JOBS=3
```

For Qdrant + MCP, add:

```bash
QDRANT_ENABLED=true
QDRANT_API_URL=http://localhost:6333
OPENROUTER_API_KEY=sk-or-v1-...
MCP_ENABLED=true
```

## Authentication

Every `/api/*` endpoint requires a Bearer token matching `API_KEY`:

```bash
curl http://localhost:3000/api/sites \
  -H "Authorization: Bearer $API_KEY"
```

`/health`, `/doc*`, `/ui/*`, and `/mcp*` are public.

## Usage examples

**Create a scraping job:**

```bash
curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'
```

**Get site details:**

```bash
curl http://localhost:3000/api/sites/<id> \
  -H "Authorization: Bearer $API_KEY"
```

**Get site results filtered to shipping content, last 10 entries:**

```bash
curl "http://localhost:3000/api/sites/<id>/results?content_type=shipping&limit=10" \
  -H "Authorization: Bearer $API_KEY"
```

**Semantic search across scraped content** (requires Qdrant + OpenRouter):

```bash
curl -X POST http://localhost:3000/api/sites/<id>/search \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q":"do you ship to Germany","limit":5}'
```

Omit `category` to search all four content buckets in parallel. Add `?format=text` for voice/LLM output.

`<id>` can be either the system-generated UUID or the optional `custom_id` you assigned to the site.

## MCP tools

When `MCP_ENABLED=true` and `QDRANT_ENABLED=true`, the server mounts an MCP Streamable HTTP endpoint at `/mcp`. Five tools are registered:

- `list_contents(site_id)` — enumerate available pages per category, tagged with the next tool to use
- `shipping_search(site_id, query, limit?)`
- `contacts_search(site_id, query, limit?)`
- `terms_search(site_id, query, limit?)`
- `faq_search(site_id, query, limit?)`

All tools accept both `site_id` and `shop_id` (legacy alias) as the parameter name, mapped to the site's `custom_id`. Tools return a friendly "Search not available for this site" message when the site is unknown or Qdrant is not enabled on it.

## Documentation

- [`docs/API.md`](./docs/API.md) — API concepts, auth, endpoint surface map, MCP tool inventory. For the full endpoint reference use `/doc` on the running server.
- [`docs/SETUP.md`](./docs/SETUP.md) — installation, environment variables, Qdrant/MCP setup, migration, troubleshooting.

## Development

```bash
npm run dev     # run with ts-node, no build
npm run watch   # tsc in watch mode
npm run build   # build TS + web UI
```

## Systemd service

```bash
sudo ./scripts/install.sh   # install as service
sudo ./scripts/reset.sh     # wipe to clean state (backs up first)
sudo ./scripts/restore.sh   # restore from backup
```

Backups land in `/var/backups/webshop-scraper/`.

## Data

SQLite lives in `data/shops.db`. Schema is applied at startup from `src/database/Database.ts`. It tracks sites (with UUID and optional `custom_id`), scrape history, per-page content with hashes, scheduled jobs, webhooks, and — when Qdrant is enabled — embedding state, deletion queue, error log, and MCP call logs.

## Logs

HTTP access logs are emitted per request with status code icon, path, IP, response time, and (with `LOG_HTTP_VERBOSE=true`) user agent and content length.

```
[INFO] [WebScraper] ✓ 200 GET /api/sites ::1 45ms
[INFO] [WebScraper] ⚠️ 404 GET /api/sites/invalid ::1 12ms
[INFO] [WebScraper] ❌ 500 POST /api/jobs ::1 234ms
```

Icons: `✓` 2xx, `↪️` 3xx, `⚠️` 4xx, `❌` 5xx.

## License

ISC