webshop-scraper

Webshop Scraper

A TypeScript-based web scraper for extracting information from webshops with persistent storage, scheduled scraping, and change detection.

Features

• Multi-webshop Support: ShopRenter, WooCommerce, and Shopify
• Persistent Storage: SQLite database with UUID tracking and analytics
• Scheduled Scraping: Automated scraping based on sitemap frequency rules
• Content Change Detection: MD5 hash-based change tracking for all content
• REST API: Full CRUD operations for shops and scraping jobs
• Bearer Authentication: Secure API access
• Job Queue: Configurable concurrency control
• Systemd Integration: Install as a system service

Quick Start

Installation

1. Clone and install:

   git clone <repository-url>
   cd webshop-scraper
   npm install
   

2. Build:

   npm run build
   

3. Configure environment:

   API_KEY=your-secret-key
   PORT=3000
   MAX_CONCURRENT_JOBS=3
   

4. Start:

   npm start
   

As a systemd service

sudo ./scripts/install.sh

API Documentation

See API Documentation for complete endpoint reference.

Quick Examples

Create a scraping job:

curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer your-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'

Get shop information:

curl http://localhost:3000/api/shops/:id \
  -H "Authorization: Bearer your-secret-key"

Get shop results:

curl "http://localhost:3000/api/shops/:id/results?limit=10" \
  -H "Authorization: Bearer your-secret-key"

Requirements

• Node.js v16 or higher
• npm
• Linux with systemd (for service installation)

Development

# Development mode
npm run dev

# Watch mode
npm run watch

# Build
npm run build

Database

Data is stored in data/shops.db (SQLite). The database includes:

• Shop information with UUID tracking
• Scrape analytics and history
• Content with change detection
• Scheduled jobs queue

Logging

View logs using journalctl:

sudo journalctl -u webshop-scraper -f

License

ISC