# Webshop Scraper

A TypeScript-based web scraper for extracting information from webshops with persistent storage, scheduled scraping, and change detection.

## Features

- **Multi-webshop Support**: ShopRenter, WooCommerce, and Shopify
- **Persistent Storage**: SQLite database with UUID tracking and analytics
- **Scheduled Scraping**: Automated scraping based on sitemap frequency rules
- **Content Change Detection**: MD5 hash-based change tracking for all content
- **REST API**: Full CRUD operations for shops and scraping jobs
- **Bearer Authentication**: Secure API access
- **Job Queue**: Configurable concurrency control
- **Systemd Integration**: Install as a system service

## Quick Start

### Installation

1. Clone and install:
```bash
git clone <repository-url>
cd webshop-scraper
npm install
```

2. Build:
```bash
npm run build
```

3. Configure environment:
```bash
API_KEY=your-secret-key
PORT=3000
MAX_CONCURRENT_JOBS=3
```

4. Start:
```bash
npm start
```

### As a systemd service

```bash
sudo ./scripts/install.sh
```

## API Documentation

See [API Documentation](./docs/API.md) for complete endpoint reference.

### Quick Examples

**Create a scraping job:**
```bash
curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer your-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'
```

**Get shop information:**
```bash
curl http://localhost:3000/api/shops/:id \
  -H "Authorization: Bearer your-secret-key"
```

**Get shop results:**
```bash
curl "http://localhost:3000/api/shops/:id/results?limit=10" \
  -H "Authorization: Bearer your-secret-key"
```

## Requirements

- Node.js v16 or higher
- npm
- Linux with systemd (for service installation)

## Development

```bash
# Development mode
npm run dev

# Watch mode
npm run watch

# Build
npm run build
```

## Database

Data is stored in `data/shops.db` (SQLite). The database includes:
- Shop information with UUID tracking
- Scrape analytics and history
- Content with change detection
- Scheduled jobs queue

## Logging

View logs using journalctl:
```bash
sudo journalctl -u webshop-scraper -f
```

## License

ISC