webshop-scraper

Webshop Scraper

A TypeScript-based web scraper for extracting information from webshops (ShopRenter, WooCommerce, Shopify). The application provides a REST API for submitting scraping jobs and retrieving results.

Features

• Multi-webshop Support: Automatically detects and scrapes ShopRenter, WooCommerce, and Shopify stores
• REST API: HTTP API for job submission and status tracking
• Bearer Authentication: Secure API access with pre-shared key
• Job Queue: Prevents system overload and race conditions with configurable concurrency
• Content Extraction: Extracts shipping information, contacts, terms & conditions, and FAQ
• Markdown Conversion: Cleans HTML and converts to markdown format
• Systemd Integration: Install as a system service with proper logging
• Logging: Outputs to stdout/stderr for systemd/journalctl integration

Requirements

• Node.js (v16 or higher)
• npm
• Linux with systemd (for service installation)

Installation

As a systemd service (recommended)

1. Clone the repository:

   git clone <repository-url>
   cd webshop-scraper
   

2. Run the installation script:

   sudo ./scripts/install.sh
   

The script will:

• Prompt for the API key
• Ask for port number (default: 3000)
• Ask for max concurrent jobs (default: 3)
• Install dependencies
• Build the application
• Create a systemd service
• Start the service

Manual installation

1. Install dependencies:

   npm install
   

2. Build the application:

   npm run build
   

3. Create a .env file:

   cp .env.example .env
   

4. Edit .env and set your configuration:

   API_KEY=your-secret-key-here
   PORT=3000
   MAX_CONCURRENT_JOBS=3
   

5. Start the application:

   npm start
   

Usage

Starting the service

sudo systemctl start webshop-scraper

Stopping the service

sudo systemctl stop webshop-scraper

Viewing logs

sudo journalctl -u webshop-scraper -f

API Endpoints

All API endpoints (except /health) require Bearer authentication.

Health Check

GET /health

Example:

curl http://localhost:3000/health

Create Scraping Job

POST /api/jobs
Authorization: Bearer <your-api-key>
Content-Type: application/json

{
  "url": "https://example-shop.com"
}

Example:

curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer your-secret-key-here" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'

Response:

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "pending",
  "message": "Job created successfully"
}

Get Job Status

GET /api/jobs/:id
Authorization: Bearer <your-api-key>

Example:

curl http://localhost:3000/api/jobs/a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
  -H "Authorization: Bearer your-secret-key-here"

Response (when completed):

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "created_at": "2024-01-01T12:00:00.000Z",
  "updated_at": "2024-01-01T12:05:00.000Z",
  "result": {
    "initial_sitemap": "https://example-shop.com/sitemap.xml",
    "shipping_informations": {
      "url": "https://example-shop.com/shipping",
      "content": "# Shipping Information\n\n..."
    },
    "contacts": {
      "url": "https://example-shop.com/contact",
      "content": "# Contact Us\n\n..."
    },
    "terms_of_conditions": {
      "url": "https://example-shop.com/terms",
      "content": "# Terms and Conditions\n\n..."
    },
    "faq": {
      "url": "https://example-shop.com/faq",
      "content": "# FAQ\n\n..."
    }
  }
}

If a page type is not found, it will be null:

{
  "faq": null
}

List All Jobs

GET /api/jobs
Authorization: Bearer <your-api-key>

Example:

curl http://localhost:3000/api/jobs \
  -H "Authorization: Bearer your-secret-key-here"

Configuration

Configuration is done via environment variables:

• API_KEY: Pre-shared key for Bearer authentication (required)
• PORT: HTTP server port (default: 3000)
• MAX_CONCURRENT_JOBS: Maximum number of concurrent scraping jobs (default: 3)

Uninstallation

To remove the service and all files:

sudo ./scripts/uninstall.sh

Development

Run in development mode

npm run dev

Build

npm run build

Watch mode

npm run watch

Architecture

• REST API Server: Express.js server with Bearer authentication
• Job Queue: In-memory queue with concurrency control
• Sitemap Parser: Detects webshop type and parses sitemap.xml
• Content Extractor: Fetches pages and extracts main content
• HTML to Markdown: Converts HTML to clean markdown using Turndown

Logging

All logs are written to stdout/stderr and can be viewed using journalctl:

# Follow logs
sudo journalctl -u webshop-scraper -f

# View logs from today
sudo journalctl -u webshop-scraper --since today

# View last 100 lines
sudo journalctl -u webshop-scraper -n 100

License

ISC