# Webshop Scraper

A TypeScript-based web scraper for extracting information from webshops with persistent storage, scheduled scraping, and change detection.

## Features

- **Multi-webshop Support**: ShopRenter, WooCommerce, and Shopify
- **Persistent Storage**: SQLite database with UUID tracking and analytics
- **Scheduled Scraping**: Automated scraping based on sitemap frequency rules
- **Content Change Detection**: SHA-256 hash-based change tracking for all content
- **Vector Search Integration**: Qdrant-powered semantic search with OpenRouter embeddings
- **MCP Tools**: Model Context Protocol integration for LLM applications
- **REST API**: Full CRUD operations for shops and scraping jobs
- **Web Interface**: Complete administrative dashboard with real-time monitoring
- **Bearer Authentication**: Secure API access
- **Job Queue**: Configurable concurrency control
- **Webhook Support**: Real-time notifications for scraping events
- **Systemd Integration**: Install as a system service

## Quick Start

### Installation

1. Clone and install:
```bash
git clone <repository-url>
cd webshop-scraper
npm install
```

2. Build:
```bash
npm run build
```

3. Configure environment:
```bash
# Basic Configuration
API_KEY=your-secret-key
PORT=3000
MAX_CONCURRENT_JOBS=3

# Qdrant Vector Search (Optional)
QDRANT_API_URL=http://localhost:6333
QDRANT_API_KEY=your-qdrant-key

# OpenRouter for Embeddings (Optional)
OPENROUTER_API_KEY=your-openrouter-key
OPENROUTER_MODEL=openai/text-embedding-3-large

# MCP Integration (Optional)
MCP_ENABLED=true
MCP_TRANSPORT=stdio
```

4. Start:
```bash
npm start
```

### As a systemd service

```bash
sudo ./scripts/install.sh
```

## Vector Search & MCP Integration

The system includes optional Qdrant vector search capabilities for semantic search of scraped content:

- **Automatic Embedding**: Content is automatically embedded using OpenRouter's text-embedding-3-large model
- **Semantic Search**: 4 MCP tools provide semantic search capabilities for LLM applications
- **Real-time Management**: Web interface for enabling/disabling and monitoring embeddings
- **Error Tracking**: Comprehensive error logging and debugging capabilities

### MCP Tools Available

- `search_shipping_info(shop_id, query, limit?)` - Search shipping and delivery information
- `search_contact_info(shop_id, query, limit?)` - Search contact and support information
- `search_terms_info(shop_id, query, limit?)` - Search terms of service and policies
- `search_faq_info(shop_id, query, limit?)` - Search frequently asked questions

See the [Qdrant Setup Guide](./docs/SETUP_GUIDE.md) for complete configuration instructions.

## API Documentation

See [API Documentation](./docs/API.md) for complete endpoint reference.
See [Qdrant API Reference](./docs/API_REFERENCE.md) for vector search endpoints.

### Quick Examples

**Create a scraping job:**
```bash
curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer your-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'
```

**Get shop information:**
```bash
curl http://localhost:3000/api/shops/:id \
  -H "Authorization: Bearer your-secret-key"
```

**Get shop results:**
```bash
curl "http://localhost:3000/api/shops/:id/results?limit=10" \
  -H "Authorization: Bearer your-secret-key"
```

## Requirements

- Node.js v16 or higher
- npm
- Linux with systemd (for service installation)

## Development

```bash
# Development mode
npm run dev

# Watch mode
npm run watch

# Build
npm run build
```

## Database

Data is stored in `data/shops.db` (SQLite). The database includes:
- Shop information with UUID tracking
- Scrape analytics and history
- Content with change detection
- Scheduled jobs queue

## Logging

View logs using journalctl:
```bash
sudo journalctl -u webshop-scraper -f
```

## License

ISC