webshop-scraper

Webshop Scraper

A TypeScript-based web scraper for extracting information from webshops with persistent storage, scheduled scraping, and change detection.

Features

• Multi-webshop Support: ShopRenter, WooCommerce, and Shopify
• Persistent Storage: SQLite database with UUID tracking and analytics
• Scheduled Scraping: Automated scraping based on sitemap frequency rules
• Content Change Detection: SHA-256 hash-based change tracking for all content
• Vector Search Integration: Qdrant-powered semantic search with OpenRouter embeddings
• MCP Tools: Model Context Protocol integration for LLM applications
• REST API: Full CRUD operations for shops and scraping jobs
• High Performance: Optimized database queries with 40-90% faster response times
• HTTP Compression: Automatic gzip/brotli compression (60-80% smaller responses)
• Pagination Support: Efficient pagination for large datasets
• Web Interface: Complete administrative dashboard with real-time monitoring
• Bearer Authentication: Secure API access
• Job Queue: Configurable concurrency control
• Webhook Support: Real-time notifications for scraping events
• Systemd Integration: Install as a system service

Quick Start

Installation

1. Clone and install:

   git clone <repository-url>
   cd webshop-scraper
   npm install
   

2. Build:

   npm run build
   

3. Configure environment:

   # Basic Configuration
   API_KEY=your-secret-key
   PORT=3000
   MAX_CONCURRENT_JOBS=3
   
   # Logging (Optional)
   LOG_HTTP_VERBOSE=true  # Enable detailed HTTP logs with User-Agent and content-length
   
   # Qdrant Vector Search (Optional)
   QDRANT_API_URL=http://localhost:6333
   QDRANT_API_KEY=your-qdrant-key
   
   # OpenRouter for Embeddings (Optional)
   OPENROUTER_API_KEY=your-openrouter-key
   OPENROUTER_MODEL=openai/text-embedding-3-large
   
   # MCP Integration (Optional)
   MCP_ENABLED=true
   MCP_TRANSPORT=stdio
   

4. Start:

   npm start
   

As a systemd service

sudo ./scripts/install.sh

Reset and Restore

Reset the system to a clean state (creates backup first):

sudo ./scripts/reset.sh

Restore from a previous backup:

sudo ./scripts/restore.sh

What gets backed up:

• SQLite database (data/shops.db)
• Environment configuration (.env)
• Service logs from journalctl

Backups are stored in /var/backups/webshop-scraper/ with timestamps.

Vector Search & MCP Integration

The system includes optional Qdrant vector search capabilities for semantic search of scraped content:

• Automatic Embedding: Content is automatically embedded using OpenRouter's text-embedding-3-large model
• Semantic Search: 4 MCP tools provide semantic search capabilities for LLM applications
• Real-time Management: Web interface for enabling/disabling and monitoring embeddings
• Error Tracking: Comprehensive error logging and debugging capabilities

MCP Tools Available

• search_shipping_info(shop_id, query, limit?) - Search shipping and delivery information
• search_contact_info(shop_id, query, limit?) - Search contact and support information
• search_terms_info(shop_id, query, limit?) - Search terms of service and policies
• search_faq_info(shop_id, query, limit?) - Search frequently asked questions

See the Qdrant Setup Guide for complete configuration instructions.

API Documentation

See API Documentation for complete endpoint reference. See Qdrant API Reference for vector search endpoints.

Quick Examples

Create a scraping job:

curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer your-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'

Get shop information:

curl http://localhost:3000/api/shops/:id \
  -H "Authorization: Bearer your-secret-key"

Get shop results:

curl "http://localhost:3000/api/shops/:id/results?limit=10" \
  -H "Authorization: Bearer your-secret-key"

Requirements

• Node.js v16 or higher
• npm
• Linux with systemd (for service installation)

Development

# Development mode
npm run dev

# Watch mode
npm run watch

# Build
npm run build

Database

Data is stored in data/shops.db (SQLite). The database includes:

• Shop information with UUID tracking
• Scrape analytics and history
• Content with change detection
• Scheduled jobs queue

Logging

The system includes enhanced HTTP access logging with status codes, response times, and optional verbose mode.

Log Format

Standard HTTP logs:

[INFO] [WebshopScraper] ✓ 200 GET /api/shops ::1 45ms
[INFO] [WebshopScraper] ⚠️ 404 GET /api/shops/invalid ::1 12ms
[INFO] [WebshopScraper] ❌ 500 POST /api/jobs ::1 234ms

Verbose HTTP logs (with LOG_HTTP_VERBOSE=true):

[INFO] [WebshopScraper] ✓ 200 GET /api/shops ::1 45ms 2456b UA:"curl/7.81.0"

Status Code Indicators

• ✓ (200-299) - Success
• ↪️ (300-399) - Redirect
• ⚠️ (400-499) - Client error
• ❌ (500-599) - Server error

View Logs

Using journalctl (systemd):

# Follow logs in real-time
sudo journalctl -u webshop-scraper -f

# View logs with timestamps
sudo journalctl -u webshop-scraper --no-pager

# Filter by priority
sudo journalctl -u webshop-scraper -p err  # Only errors

Using npm (development):

npm run dev  # Includes timestamps in output

License

ISC