# Webshop Scraper

A TypeScript-based web scraper for extracting information from webshops with persistent storage, scheduled scraping, and change detection.

## Features

- **Multi-webshop Support**: ShopRenter, WooCommerce, and Shopify
- **Persistent Storage**: SQLite database with UUID tracking and analytics
- **Scheduled Scraping**: Automated scraping based on sitemap frequency rules
- **Content Change Detection**: SHA-256 hash-based change tracking for all content
- **Vector Search Integration**: Qdrant-powered semantic search with OpenRouter embeddings
- **MCP Tools**: Model Context Protocol integration for LLM applications
- **REST API**: Full CRUD operations for shops and scraping jobs
- **High Performance**: Optimized database queries with 40-90% faster response times
- **HTTP Compression**: Automatic gzip/brotli compression (60-80% smaller responses)
- **Pagination Support**: Efficient pagination for large datasets
- **Web Interface**: Complete administrative dashboard with real-time monitoring
- **Bearer Authentication**: Secure API access
- **Job Queue**: Configurable concurrency control
- **Webhook Support**: Real-time notifications for scraping events
- **Systemd Integration**: Install as a system service

## Quick Start

### Installation

1. Clone and install:
```bash
git clone <repository-url>
cd webshop-scraper
npm install
```

2. Build:
```bash
npm run build
```

3. Configure environment:
```bash
# Basic Configuration
API_KEY=your-secret-key
PORT=3000
MAX_CONCURRENT_JOBS=3

# Logging (Optional)
LOG_HTTP_VERBOSE=true  # Enable detailed HTTP logs with User-Agent and content-length

# Qdrant Vector Search (Optional)
QDRANT_API_URL=http://localhost:6333
QDRANT_API_KEY=your-qdrant-key

# OpenRouter for Embeddings (Optional)
OPENROUTER_API_KEY=your-openrouter-key
OPENROUTER_MODEL=openai/text-embedding-3-large

# MCP Integration (Optional)
MCP_ENABLED=true
MCP_TRANSPORT=stdio
```

4. Start:
```bash
npm start
```

### As a systemd service

```bash
sudo ./scripts/install.sh
```

### Reset and Restore

Reset the system to a clean state (creates backup first):
```bash
sudo ./scripts/reset.sh
```

Restore from a previous backup:
```bash
sudo ./scripts/restore.sh
```

**What gets backed up:**
- SQLite database (data/shops.db)
- Environment configuration (.env)
- Service logs from journalctl

Backups are stored in `/var/backups/webshop-scraper/` with timestamps.

## Vector Search & MCP Integration

The system includes optional Qdrant vector search capabilities for semantic search of scraped content:

- **Automatic Embedding**: Content is automatically embedded using OpenRouter's text-embedding-3-large model
- **Semantic Search**: 4 MCP tools provide semantic search capabilities for LLM applications
- **Real-time Management**: Web interface for enabling/disabling and monitoring embeddings
- **Error Tracking**: Comprehensive error logging and debugging capabilities

### MCP Tools Available

- `search_shipping_info(shop_id, query, limit?)` - Search shipping and delivery information
- `search_contact_info(shop_id, query, limit?)` - Search contact and support information
- `search_terms_info(shop_id, query, limit?)` - Search terms of service and policies
- `search_faq_info(shop_id, query, limit?)` - Search frequently asked questions

See the [Qdrant Setup Guide](./docs/SETUP_GUIDE.md) for complete configuration instructions.

## API Documentation

See [API Documentation](./docs/API.md) for complete endpoint reference.
See [Qdrant API Reference](./docs/API_REFERENCE.md) for vector search endpoints.

### Quick Examples

**Create a scraping job:**
```bash
curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer your-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'
```

**Get shop information:**
```bash
curl http://localhost:3000/api/shops/:id \
  -H "Authorization: Bearer your-secret-key"
```

**Get shop results:**
```bash
curl "http://localhost:3000/api/shops/:id/results?limit=10" \
  -H "Authorization: Bearer your-secret-key"
```

## Requirements

- Node.js v16 or higher
- npm
- Linux with systemd (for service installation)

## Development

```bash
# Development mode
npm run dev

# Watch mode
npm run watch

# Build
npm run build
```

## Database

Data is stored in `data/shops.db` (SQLite). The database includes:
- Shop information with UUID tracking
- Scrape analytics and history
- Content with change detection
- Scheduled jobs queue

## Logging

The system includes enhanced HTTP access logging with status codes, response times, and optional verbose mode.

### Log Format

**Standard HTTP logs:**
```
[INFO] [WebshopScraper] ✓ 200 GET /api/shops ::1 45ms
[INFO] [WebshopScraper] ⚠️ 404 GET /api/shops/invalid ::1 12ms
[INFO] [WebshopScraper] ❌ 500 POST /api/jobs ::1 234ms
```

**Verbose HTTP logs** (with `LOG_HTTP_VERBOSE=true`):
```
[INFO] [WebshopScraper] ✓ 200 GET /api/shops ::1 45ms 2456b UA:"curl/7.81.0"
```

### Status Code Indicators
- ✓ (200-299) - Success
- ↪️ (300-399) - Redirect
- ⚠️ (400-499) - Client error
- ❌ (500-599) - Server error

### View Logs

Using journalctl (systemd):
```bash
# Follow logs in real-time
sudo journalctl -u webshop-scraper -f

# View logs with timestamps
sudo journalctl -u webshop-scraper --no-pager

# Filter by priority
sudo journalctl -u webshop-scraper -p err  # Only errors
```

Using npm (development):
```bash
npm run dev  # Includes timestamps in output
```

## License

ISC