# Web Scraper

A TypeScript service that scrapes and classifies content from any website — webshops (ShopRenter, WooCommerce, Shopify) and generic websites alike. Content is stored in SQLite with change detection and optionally indexed into Qdrant for semantic search via MCP tools or REST API.

## Features

- **Universal scraping** — supports two site types: `webshop` (ShopRenter, WooCommerce, Shopify) and `website` (any generic site)
- **13 content categories** — `shipping`, `contacts`, `terms`, `faq`, `services`, `about`, `team`, `blog`, `pricing`, `testimonials`, `gallery`, `landing`, `other`
- **Smart content discovery** — sitemap-driven for webshops; automatic link discovery engine (BFS crawl) for sites without a sitemap, with configurable `max_crawl_depth` and `max_pages`
- **URL exclusions** — glob patterns to skip unwanted URLs during scraping
- **Auto-detect** — `POST /api/sites/detect` probes a URL to determine site type, platform, and sitemap
- **Custom sitemap** — override the auto-detected sitemap with your own URL
- **Scheduled scraping** — automatic re-scrape based on sitemap `changefreq` rules, toggleable per site
- **Change detection** — SHA-256 content hashing; unchanged pages are not re-processed
- **Persistent storage** — SQLite (`better-sqlite3`) with site analytics and scrape history
- **Optional vector search** — Qdrant + OpenRouter embeddings, one collection per `{custom_id}-{category}`
- **MCP server** — Streamable HTTP MCP endpoint at `/mcp` exposing 14 tools: `list_contents` plus a `*_search` tool for each of the 13 content categories
- **Self-documenting REST API** — endpoint metadata is rendered as HTML at `/doc` and OpenAPI 3.0 at `/doc/openapi`
- **Bearer authentication** for all `/api/*` endpoints
- **Webhooks** per site — `scrape_started` / `scrape_completed` / `scrape_failed`
- **gzip/brotli compression** on all responses over 1 KB
- **Web UI** served at `/ui`
- **systemd installer** under `scripts/`

## Quick start

```bash
git clone <repository-url>
cd webshop-scraper
npm install
npm run build
cp .env.example .env   # then edit API_KEY and anything else you need
npm start
```

Then open:

- `http://localhost:3000/doc` — live API reference (HTML)
- `http://localhost:3000/ui` — web dashboard
- `http://localhost:3000/health` — liveness and queue stats (no auth)

## Minimal configuration

Only `API_KEY` is required. See [`docs/SETUP.md`](./docs/SETUP.md) for the full environment variable reference.

```bash
API_KEY=change-me
PORT=3000
MAX_CONCURRENT_JOBS=3
```

For Qdrant + MCP, add:

```bash
QDRANT_ENABLED=true
QDRANT_API_URL=http://localhost:6333
OPENROUTER_API_KEY=sk-or-v1-...
MCP_ENABLED=true
```

## Authentication

Every `/api/*` endpoint requires a Bearer token matching `API_KEY`:

```bash
curl http://localhost:3000/api/sites \
  -H "Authorization: Bearer $API_KEY"
```

`/health`, `/doc*`, `/ui/*`, and `/mcp*` are public.

## Usage examples

**Create a scraping job:**

```bash
curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'
```

**Get site details:**

```bash
curl http://localhost:3000/api/sites/<id> \
  -H "Authorization: Bearer $API_KEY"
```

**Get site results filtered to shipping content, last 10 entries:**

```bash
curl "http://localhost:3000/api/sites/<id>/results?content_type=shipping&limit=10" \
  -H "Authorization: Bearer $API_KEY"
```

**Semantic search across scraped content** (requires Qdrant + OpenRouter):

```bash
curl -X POST http://localhost:3000/api/sites/<id>/search \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q":"do you ship to Germany","limit":5}'
```

Omit `category` to search all 13 content buckets in parallel. Add `?format=text` for voice/LLM output.

`<id>` can be either the system-generated UUID or the optional `custom_id` you assigned to the site.

## MCP tools

When `MCP_ENABLED=true` and `QDRANT_ENABLED=true`, the server mounts an MCP Streamable HTTP endpoint at `/mcp`. 14 tools are registered:

- `list_contents(site_id)` — enumerate available pages per category, tagged with the next tool to use
- `shipping_search(site_id, query, limit?)`
- `contacts_search(site_id, query, limit?)`
- `terms_search(site_id, query, limit?)`
- `faq_search(site_id, query, limit?)`
- `services_search(site_id, query, limit?)`
- `about_search(site_id, query, limit?)`
- `team_search(site_id, query, limit?)`
- `blog_search(site_id, query, limit?)`
- `pricing_search(site_id, query, limit?)`
- `testimonials_search(site_id, query, limit?)`
- `gallery_search(site_id, query, limit?)`
- `landing_search(site_id, query, limit?)`
- `other_search(site_id, query, limit?)`

All tools accept both `site_id` and `shop_id` (legacy alias) as the parameter name, mapped to the site's `custom_id`. Tools return a friendly "Search not available for this site" message when the site is unknown or Qdrant is not enabled on it.

## Documentation

- [`docs/API.md`](./docs/API.md) — API concepts, auth, endpoint surface map, MCP tool inventory. For the full endpoint reference use `/doc` on the running server.
- [`docs/SETUP.md`](./docs/SETUP.md) — installation, environment variables, Qdrant/MCP setup, migration, troubleshooting.

## Development

```bash
npm run dev     # run with ts-node, no build
npm run watch   # tsc in watch mode
npm run build   # build TS + web UI
```

## Systemd service

```bash
sudo ./scripts/install.sh   # install as service
sudo ./scripts/reset.sh     # wipe to clean state (backs up first)
sudo ./scripts/restore.sh   # restore from backup
```

Backups land in `/var/backups/webshop-scraper/`.

## Data

SQLite lives in `data/shops.db`. Schema is applied at startup from `src/database/Database.ts`. It tracks sites (with UUID and optional `custom_id`), scrape history, per-page content with hashes, scheduled jobs, webhooks, and — when Qdrant is enabled — embedding state, deletion queue, error log, and MCP call logs.

## Logs

HTTP access logs are emitted per request with status code icon, path, IP, response time, and (with `LOG_HTTP_VERBOSE=true`) user agent and content length.

```
[INFO] [WebScraper] ✓ 200 GET /api/sites ::1 45ms
[INFO] [WebScraper] ⚠️ 404 GET /api/sites/invalid ::1 12ms
[INFO] [WebScraper] ❌ 500 POST /api/jobs ::1 234ms
```

Icons: `✓` 2xx, `↪️` 3xx, `⚠️` 4xx, `❌` 5xx.

## License

ISC