feat: implement complete webshop scraper application #1

- Implement TypeScript-based web scraper for ShopRenter, WooCommerce, and Shopify
- Add REST API with Bearer token authentication
- Implement job queue system with concurrency control to prevent race conditions
- Add sitemap parser with webshop type detection
- Implement content extraction with HTML to Markdown conversion
- Create systemd install/uninstall scripts with environment variable configuration
- Add proper logging to stdout/stderr for systemd/journalctl integration
- Complete documentation in README

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

.env.example

@@ -0,0 +1,8 @@
+# API Key for Bearer authentication
+API_KEY=your-secret-key-here
+
+# HTTP server port
+PORT=3000
+
+# Maximum number of concurrent scraping jobs
+MAX_CONCURRENT_JOBS=3

.gitignore

@@ -28,3 +28,16 @@ build/Release
 # https://docs.npmjs.com/misc/faq#should-i-check-my-node-modules-folder-into-git
 node_modules
 
+# TypeScript
+dist/
+*.tsbuildinfo
+
+# Environment variables
+.env
+
+# Editor directories and files
+.vscode/
+.idea/
+*.swp
+*.swo
+*~

.mcp-gogs.json

@@ -0,0 +1,12 @@
+{
+  "mcpServers": {
+    "gogs": {
+      "type": "stdio",
+      "command": "node",
+      "args": [
+        "/home/claude/gogs-mcp/dist/index.js"
+      ],
+      "env": {}
+    }
+  }
+}

README.md

@@ -1,2 +1,262 @@
-# webshop-scraper
+# Webshop Scraper
 
+A TypeScript-based web scraper for extracting information from webshops (ShopRenter, WooCommerce, Shopify). The application provides a REST API for submitting scraping jobs and retrieving results.
+
+## Features
+
+- **Multi-webshop Support**: Automatically detects and scrapes ShopRenter, WooCommerce, and Shopify stores
+- **REST API**: HTTP API for job submission and status tracking
+- **Bearer Authentication**: Secure API access with pre-shared key
+- **Job Queue**: Prevents system overload and race conditions with configurable concurrency
+- **Content Extraction**: Extracts shipping information, contacts, terms & conditions, and FAQ
+- **Markdown Conversion**: Cleans HTML and converts to markdown format
+- **Systemd Integration**: Install as a system service with proper logging
+- **Logging**: Outputs to stdout/stderr for systemd/journalctl integration
+
+## Requirements
+
+- Node.js (v16 or higher)
+- npm
+- Linux with systemd (for service installation)
+
+## Installation
+
+### As a systemd service (recommended)
+
+1. Clone the repository:
+```bash
+git clone <repository-url>
+cd webshop-scraper
+```
+
+2. Run the installation script:
+```bash
+sudo ./scripts/install.sh
+```
+
+The script will:
+- Prompt for the API key
+- Ask for port number (default: 3000)
+- Ask for max concurrent jobs (default: 3)
+- Install dependencies
+- Build the application
+- Create a systemd service
+- Start the service
+
+### Manual installation
+
+1. Install dependencies:
+```bash
+npm install
+```
+
+2. Build the application:
+```bash
+npm run build
+```
+
+3. Create a `.env` file:
+```bash
+cp .env.example .env
+```
+
+4. Edit `.env` and set your configuration:
+```
+API_KEY=your-secret-key-here
+PORT=3000
+MAX_CONCURRENT_JOBS=3
+```
+
+5. Start the application:
+```bash
+npm start
+```
+
+## Usage
+
+### Starting the service
+
+```bash
+sudo systemctl start webshop-scraper
+```
+
+### Stopping the service
+
+```bash
+sudo systemctl stop webshop-scraper
+```
+
+### Viewing logs
+
+```bash
+sudo journalctl -u webshop-scraper -f
+```
+
+### API Endpoints
+
+All API endpoints (except `/health`) require Bearer authentication.
+
+#### Health Check
+
+```bash
+GET /health
+```
+
+Example:
+```bash
+curl http://localhost:3000/health
+```
+
+#### Create Scraping Job
+
+```bash
+POST /api/jobs
+Authorization: Bearer <your-api-key>
+Content-Type: application/json
+
+{
+  "url": "https://example-shop.com"
+}
+```
+
+Example:
+```bash
+curl -X POST http://localhost:3000/api/jobs \
+  -H "Authorization: Bearer your-secret-key-here" \
+  -H "Content-Type: application/json" \
+  -d '{"url": "https://example-shop.com"}'
+```
+
+Response:
+```json
+{
+  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
+  "status": "pending",
+  "message": "Job created successfully"
+}
+```
+
+#### Get Job Status
+
+```bash
+GET /api/jobs/:id
+Authorization: Bearer <your-api-key>
+```
+
+Example:
+```bash
+curl http://localhost:3000/api/jobs/a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
+  -H "Authorization: Bearer your-secret-key-here"
+```
+
+Response (when completed):
+```json
+{
+  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
+  "status": "completed",
+  "created_at": "2024-01-01T12:00:00.000Z",
+  "updated_at": "2024-01-01T12:05:00.000Z",
+  "result": {
+    "initial_sitemap": "https://example-shop.com/sitemap.xml",
+    "shipping_informations": {
+      "url": "https://example-shop.com/shipping",
+      "content": "# Shipping Information\n\n..."
+    },
+    "contacts": {
+      "url": "https://example-shop.com/contact",
+      "content": "# Contact Us\n\n..."
+    },
+    "terms_of_conditions": {
+      "url": "https://example-shop.com/terms",
+      "content": "# Terms and Conditions\n\n..."
+    },
+    "faq": {
+      "url": "https://example-shop.com/faq",
+      "content": "# FAQ\n\n..."
+    }
+  }
+}
+```
+
+If a page type is not found, it will be `null`:
+```json
+{
+  "faq": null
+}
+```
+
+#### List All Jobs
+
+```bash
+GET /api/jobs
+Authorization: Bearer <your-api-key>
+```
+
+Example:
+```bash
+curl http://localhost:3000/api/jobs \
+  -H "Authorization: Bearer your-secret-key-here"
+```
+
+## Configuration
+
+Configuration is done via environment variables:
+
+- `API_KEY`: Pre-shared key for Bearer authentication (required)
+- `PORT`: HTTP server port (default: 3000)
+- `MAX_CONCURRENT_JOBS`: Maximum number of concurrent scraping jobs (default: 3)
+
+## Uninstallation
+
+To remove the service and all files:
+
+```bash
+sudo ./scripts/uninstall.sh
+```
+
+## Development
+
+### Run in development mode
+
+```bash
+npm run dev
+```
+
+### Build
+
+```bash
+npm run build
+```
+
+### Watch mode
+
+```bash
+npm run watch
+```
+
+## Architecture
+
+- **REST API Server**: Express.js server with Bearer authentication
+- **Job Queue**: In-memory queue with concurrency control
+- **Sitemap Parser**: Detects webshop type and parses sitemap.xml
+- **Content Extractor**: Fetches pages and extracts main content
+- **HTML to Markdown**: Converts HTML to clean markdown using Turndown
+
+## Logging
+
+All logs are written to stdout/stderr and can be viewed using journalctl:
+
+```bash
+# Follow logs
+sudo journalctl -u webshop-scraper -f
+
+# View logs from today
+sudo journalctl -u webshop-scraper --since today
+
+# View last 100 lines
+sudo journalctl -u webshop-scraper -n 100
+```
+
+## License
+
+ISC

package.json

@@ -0,0 +1,37 @@
+{
+  "name": "webshop-scraper",
+  "version": "1.0.0",
+  "description": "A web scraper for extracting information from webshops (ShopRenter, WooCommerce, Shopify)",
+  "main": "dist/index.js",
+  "scripts": {
+    "build": "tsc",
+    "start": "node dist/index.js",
+    "dev": "ts-node src/index.ts",
+    "watch": "tsc -w"
+  },
+  "keywords": [
+    "scraper",
+    "webshop",
+    "shopify",
+    "woocommerce",
+    "shoprenter"
+  ],
+  "author": "",
+  "license": "ISC",
+  "dependencies": {
+    "express": "^4.18.2",
+    "axios": "^1.6.0",
+    "cheerio": "^1.0.0-rc.12",
+    "turndown": "^7.1.2",
+    "xml2js": "^0.6.2",
+    "uuid": "^9.0.1"
+  },
+  "devDependencies": {
+    "@types/express": "^4.17.20",
+    "@types/node": "^20.9.0",
+    "@types/uuid": "^9.0.7",
+    "@types/xml2js": "^0.4.13",
+    "typescript": "^5.2.2",
+    "ts-node": "^10.9.1"
+  }
+}

scripts/install.sh

@@ -0,0 +1,156 @@
+#!/bin/bash
+
+set -e
+
+echo "==================================="
+echo "Webshop Scraper Installation Script"
+echo "==================================="
+echo ""
+
+# Check if running as root
+if [ "$EUID" -ne 0 ]; then
+    echo "Error: Please run as root (use sudo)"
+    exit 1
+fi
+
+# Get the actual user who ran sudo
+ACTUAL_USER=${SUDO_USER:-$USER}
+ACTUAL_HOME=$(eval echo ~$ACTUAL_USER)
+
+# Installation directory
+INSTALL_DIR="/opt/webshop-scraper"
+SERVICE_NAME="webshop-scraper"
+
+echo "Installation directory: $INSTALL_DIR"
+echo "Service name: $SERVICE_NAME"
+echo ""
+
+# Prompt for API key
+echo "Please enter the API key for authentication:"
+read -s API_KEY
+echo ""
+
+if [ -z "$API_KEY" ]; then
+    echo "Error: API key cannot be empty"
+    exit 1
+fi
+
+# Prompt for port (optional)
+echo "Enter the HTTP port (default: 3000):"
+read PORT
+PORT=${PORT:-3000}
+
+# Prompt for max concurrent jobs (optional)
+echo "Enter the maximum number of concurrent jobs (default: 3):"
+read MAX_CONCURRENT_JOBS
+MAX_CONCURRENT_JOBS=${MAX_CONCURRENT_JOBS:-3}
+
+echo ""
+echo "Configuration:"
+echo "  Port: $PORT"
+echo "  Max Concurrent Jobs: $MAX_CONCURRENT_JOBS"
+echo ""
+
+# Create installation directory
+echo "Creating installation directory..."
+mkdir -p "$INSTALL_DIR"
+
+# Copy application files
+echo "Copying application files..."
+cp -r package.json tsconfig.json src "$INSTALL_DIR/"
+
+# Change to installation directory
+cd "$INSTALL_DIR"
+
+# Check if Node.js is installed
+if ! command -v node &> /dev/null; then
+    echo "Error: Node.js is not installed. Please install Node.js first."
+    exit 1
+fi
+
+# Check if npm is installed
+if ! command -v npm &> /dev/null; then
+    echo "Error: npm is not installed. Please install npm first."
+    exit 1
+fi
+
+# Install dependencies
+echo "Installing dependencies..."
+npm install --production
+
+# Build TypeScript
+echo "Building application..."
+npm run build
+
+# Create environment file
+echo "Creating environment file..."
+cat > "$INSTALL_DIR/.env" <<EOF
+API_KEY=$API_KEY
+PORT=$PORT
+MAX_CONCURRENT_JOBS=$MAX_CONCURRENT_JOBS
+EOF
+
+# Set permissions
+chmod 600 "$INSTALL_DIR/.env"
+chown -R $ACTUAL_USER:$ACTUAL_USER "$INSTALL_DIR"
+
+# Create systemd service file
+echo "Creating systemd service..."
+cat > "/etc/systemd/system/$SERVICE_NAME.service" <<EOF
+[Unit]
+Description=Webshop Scraper Service
+After=network.target
+
+[Service]
+Type=simple
+User=$ACTUAL_USER
+WorkingDirectory=$INSTALL_DIR
+EnvironmentFile=$INSTALL_DIR/.env
+ExecStart=/usr/bin/node $INSTALL_DIR/dist/index.js
+Restart=on-failure
+RestartSec=10
+
+# Logging
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=$SERVICE_NAME
+
+# Security
+NoNewPrivileges=true
+PrivateTmp=true
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+# Reload systemd
+echo "Reloading systemd..."
+systemctl daemon-reload
+
+# Enable service
+echo "Enabling service..."
+systemctl enable "$SERVICE_NAME"
+
+# Start service
+echo "Starting service..."
+systemctl start "$SERVICE_NAME"
+
+# Check status
+echo ""
+echo "==================================="
+echo "Installation completed successfully!"
+echo "==================================="
+echo ""
+echo "Service status:"
+systemctl status "$SERVICE_NAME" --no-pager || true
+echo ""
+echo "Useful commands:"
+echo "  Start service:   sudo systemctl start $SERVICE_NAME"
+echo "  Stop service:    sudo systemctl stop $SERVICE_NAME"
+echo "  Restart service: sudo systemctl restart $SERVICE_NAME"
+echo "  View logs:       sudo journalctl -u $SERVICE_NAME -f"
+echo "  Check status:    sudo systemctl status $SERVICE_NAME"
+echo ""
+echo "API is available at: http://localhost:$PORT"
+echo "Health check: http://localhost:$PORT/health"
+echo ""

scripts/uninstall.sh

@@ -0,0 +1,65 @@
+#!/bin/bash
+
+set -e
+
+echo "======================================"
+echo "Webshop Scraper Uninstallation Script"
+echo "======================================"
+echo ""
+
+# Check if running as root
+if [ "$EUID" -ne 0 ]; then
+    echo "Error: Please run as root (use sudo)"
+    exit 1
+fi
+
+INSTALL_DIR="/opt/webshop-scraper"
+SERVICE_NAME="webshop-scraper"
+
+# Confirm uninstallation
+echo "This will remove the Webshop Scraper service and all its files."
+echo "Are you sure you want to continue? (y/N)"
+read -r CONFIRM
+
+if [ "$CONFIRM" != "y" ] && [ "$CONFIRM" != "Y" ]; then
+    echo "Uninstallation cancelled."
+    exit 0
+fi
+
+echo ""
+echo "Uninstalling Webshop Scraper..."
+echo ""
+
+# Stop service if running
+if systemctl is-active --quiet "$SERVICE_NAME"; then
+    echo "Stopping service..."
+    systemctl stop "$SERVICE_NAME"
+fi
+
+# Disable service
+if systemctl is-enabled --quiet "$SERVICE_NAME" 2>/dev/null; then
+    echo "Disabling service..."
+    systemctl disable "$SERVICE_NAME"
+fi
+
+# Remove systemd service file
+if [ -f "/etc/systemd/system/$SERVICE_NAME.service" ]; then
+    echo "Removing systemd service file..."
+    rm -f "/etc/systemd/system/$SERVICE_NAME.service"
+fi
+
+# Reload systemd
+echo "Reloading systemd..."
+systemctl daemon-reload
+
+# Remove installation directory
+if [ -d "$INSTALL_DIR" ]; then
+    echo "Removing installation directory..."
+    rm -rf "$INSTALL_DIR"
+fi
+
+echo ""
+echo "======================================"
+echo "Uninstallation completed successfully!"
+echo "======================================"
+echo ""

src/api/middleware/auth.ts

@@ -0,0 +1,33 @@
+import { Request, Response, NextFunction } from 'express';
+import { logger } from '../../utils/logger';
+
+/**
+ * Bearer token authentication middleware
+ */
+export function authMiddleware(apiKey: string) {
+  return (req: Request, res: Response, next: NextFunction): void => {
+    const authHeader = req.headers.authorization;
+
+    if (!authHeader) {
+      logger.warn('Authentication failed: No authorization header');
+      res.status(401).json({ error: 'No authorization header provided' });
+      return;
+    }
+
+    const parts = authHeader.split(' ');
+    if (parts.length !== 2 || parts[0] !== 'Bearer') {
+      logger.warn('Authentication failed: Invalid authorization format');
+      res.status(401).json({ error: 'Invalid authorization format. Use: Bearer <token>' });
+      return;
+    }
+
+    const token = parts[1];
+    if (token !== apiKey) {
+      logger.warn('Authentication failed: Invalid token');
+      res.status(401).json({ error: 'Invalid API key' });
+      return;
+    }
+
+    next();
+  };
+}

src/api/routes.ts

@@ -0,0 +1,125 @@
+import { Router, Request, Response } from 'express';
+import { v4 as uuidv4 } from 'uuid';
+import { JobQueue } from '../queue/JobQueue';
+import { ScraperJob } from '../types';
+import { logger } from '../utils/logger';
+
+export function createRouter(jobQueue: JobQueue): Router {
+  const router = Router();
+
+  /**
+   * POST /jobs - Create a new scraping job
+   */
+  router.post('/jobs', (req: Request, res: Response): void => {
+    try {
+      const { url } = req.body;
+
+      if (!url) {
+        res.status(400).json({ error: 'URL is required' });
+        return;
+      }
+
+      // Validate URL format
+      try {
+        new URL(url);
+      } catch (error) {
+        res.status(400).json({ error: 'Invalid URL format' });
+        return;
+      }
+
+      // Create new job
+      const job: ScraperJob = {
+        id: uuidv4(),
+        sitemapUrl: url,
+        status: 'pending',
+        createdAt: new Date(),
+        updatedAt: new Date()
+      };
+
+      jobQueue.addJob(job);
+
+      logger.info(`New job created: ${job.id}`);
+
+      res.status(201).json({
+        job_id: job.id,
+        status: job.status,
+        message: 'Job created successfully'
+      });
+    } catch (error) {
+      logger.error('Error creating job', error);
+      res.status(500).json({ error: 'Internal server error' });
+    }
+  });
+
+  /**
+   * GET /jobs/:id - Get job status and result
+   */
+  router.get('/jobs/:id', (req: Request, res: Response): void => {
+    try {
+      const { id } = req.params;
+      const job = jobQueue.getJob(id);
+
+      if (!job) {
+        res.status(404).json({ error: 'Job not found' });
+        return;
+      }
+
+      const response: any = {
+        job_id: job.id,
+        status: job.status,
+        created_at: job.createdAt,
+        updated_at: job.updatedAt
+      };
+
+      if (job.status === 'completed' && job.result) {
+        response.result = job.result;
+      }
+
+      if (job.status === 'failed' && job.error) {
+        response.error = job.error;
+      }
+
+      res.json(response);
+    } catch (error) {
+      logger.error('Error fetching job', error);
+      res.status(500).json({ error: 'Internal server error' });
+    }
+  });
+
+  /**
+   * GET /jobs - List all jobs
+   */
+  router.get('/jobs', (req: Request, res: Response): void => {
+    try {
+      const jobs = jobQueue.getAllJobs();
+      const stats = jobQueue.getStats();
+
+      res.json({
+        stats,
+        jobs: jobs.map(job => ({
+          job_id: job.id,
+          status: job.status,
+          sitemap_url: job.sitemapUrl,
+          created_at: job.createdAt,
+          updated_at: job.updatedAt
+        }))
+      });
+    } catch (error) {
+      logger.error('Error listing jobs', error);
+      res.status(500).json({ error: 'Internal server error' });
+    }
+  });
+
+  /**
+   * GET /health - Health check endpoint
+   */
+  router.get('/health', (req: Request, res: Response): void => {
+    res.json({
+      status: 'ok',
+      timestamp: new Date().toISOString(),
+      queue_stats: jobQueue.getStats()
+    });
+  });
+
+  return router;
+}

src/api/server.ts

@@ -0,0 +1,44 @@
+import express, { Express } from 'express';
+import { JobQueue } from '../queue/JobQueue';
+import { createRouter } from './routes';
+import { authMiddleware } from './middleware/auth';
+import { logger } from '../utils/logger';
+
+export function createServer(apiKey: string, jobQueue: JobQueue): Express {
+  const app = express();
+
+  // Middleware
+  app.use(express.json());
+
+  // Logging middleware
+  app.use((req, res, next) => {
+    logger.info(`${req.method} ${req.path}`);
+    next();
+  });
+
+  // Health check (no auth required)
+  app.get('/health', (req, res) => {
+    res.json({
+      status: 'ok',
+      timestamp: new Date().toISOString(),
+      queue_stats: jobQueue.getStats()
+    });
+  });
+
+  // Protected routes
+  const router = createRouter(jobQueue);
+  app.use('/api', authMiddleware(apiKey), router);
+
+  // 404 handler
+  app.use((req, res) => {
+    res.status(404).json({ error: 'Not found' });
+  });
+
+  // Error handler
+  app.use((err: any, req: express.Request, res: express.Response, next: express.NextFunction) => {
+    logger.error('Unhandled error', err);
+    res.status(500).json({ error: 'Internal server error' });
+  });
+
+  return app;
+}

src/config/index.ts

@@ -0,0 +1,27 @@
+import { logger } from '../utils/logger';
+
+export interface AppConfig {
+  port: number;
+  apiKey: string;
+  maxConcurrentJobs: number;
+}
+
+export function loadConfig(): AppConfig {
+  const apiKey = process.env.API_KEY;
+
+  if (!apiKey) {
+    logger.error('API_KEY environment variable is not set');
+    throw new Error('API_KEY environment variable is required');
+  }
+
+  const port = parseInt(process.env.PORT || '3000', 10);
+  const maxConcurrentJobs = parseInt(process.env.MAX_CONCURRENT_JOBS || '3', 10);
+
+  logger.info('Configuration loaded', { port, maxConcurrentJobs });
+
+  return {
+    port,
+    apiKey,
+    maxConcurrentJobs
+  };
+}

src/index.ts

@@ -0,0 +1,79 @@
+import { loadConfig } from './config';
+import { createServer } from './api/server';
+import { JobQueue } from './queue/JobQueue';
+import { WebshopScraper } from './scraper/WebshopScraper';
+import { logger } from './utils/logger';
+
+async function main() {
+  try {
+    logger.info('Starting Webshop Scraper application...');
+
+    // Load configuration
+    const config = loadConfig();
+
+    // Initialize job queue
+    const jobQueue = new JobQueue(config.maxConcurrentJobs);
+
+    // Initialize scraper
+    const scraper = new WebshopScraper();
+
+    // Handle job processing
+    jobQueue.on('process-job', async (job) => {
+      logger.info(`Processing job ${job.id}`);
+
+      try {
+        const result = await scraper.scrape(job.sitemapUrl);
+        jobQueue.updateJob(job.id, {
+          status: 'completed',
+          result
+        });
+        logger.info(`Job ${job.id} completed successfully`);
+      } catch (error) {
+        const errorMessage = error instanceof Error ? error.message : 'Unknown error';
+        jobQueue.updateJob(job.id, {
+          status: 'failed',
+          error: errorMessage
+        });
+        logger.error(`Job ${job.id} failed`, error);
+      }
+    });
+
+    // Create and start HTTP server
+    const app = createServer(config.apiKey, jobQueue);
+
+    const server = app.listen(config.port, () => {
+      logger.info(`Server listening on port ${config.port}`);
+      logger.info('API endpoints:');
+      logger.info(`  POST   /api/jobs        - Create new scraping job`);
+      logger.info(`  GET    /api/jobs/:id    - Get job status and result`);
+      logger.info(`  GET    /api/jobs        - List all jobs`);
+      logger.info(`  GET    /health          - Health check`);
+      logger.info('');
+      logger.info('Authentication: Bearer token required for /api/* endpoints');
+    });
+
+    // Graceful shutdown
+    const shutdown = async () => {
+      logger.info('Shutting down gracefully...');
+      server.close(() => {
+        logger.info('Server closed');
+        process.exit(0);
+      });
+
+      // Force shutdown after 10 seconds
+      setTimeout(() => {
+        logger.error('Forced shutdown after timeout');
+        process.exit(1);
+      }, 10000);
+    };
+
+    process.on('SIGTERM', shutdown);
+    process.on('SIGINT', shutdown);
+
+  } catch (error) {
+    logger.error('Failed to start application', error);
+    process.exit(1);
+  }
+}
+
+main();

src/queue/JobQueue.ts

@@ -0,0 +1,117 @@
+import { ScraperJob } from '../types';
+import { logger } from '../utils/logger';
+import { EventEmitter } from 'events';
+
+/**
+ * Simple in-memory job queue with concurrency control
+ * Prevents race conditions and manages system resources
+ */
+export class JobQueue extends EventEmitter {
+  private jobs: Map<string, ScraperJob> = new Map();
+  private processingJobs: Set<string> = new Set();
+  private maxConcurrent: number;
+
+  constructor(maxConcurrent: number = 3) {
+    super();
+    this.maxConcurrent = maxConcurrent;
+    logger.info(`JobQueue initialized with max concurrent jobs: ${maxConcurrent}`);
+  }
+
+  /**
+   * Add a job to the queue
+   */
+  addJob(job: ScraperJob): void {
+    this.jobs.set(job.id, job);
+    logger.info(`Job ${job.id} added to queue`);
+    this.emit('job-added', job);
+    this.processNext();
+  }
+
+  /**
+   * Get a job by ID
+   */
+  getJob(id: string): ScraperJob | undefined {
+    return this.jobs.get(id);
+  }
+
+  /**
+   * Update job status
+   */
+  updateJob(id: string, updates: Partial<ScraperJob>): void {
+    const job = this.jobs.get(id);
+    if (job) {
+      Object.assign(job, updates, { updatedAt: new Date() });
+      this.jobs.set(id, job);
+      logger.info(`Job ${id} updated`, { status: job.status });
+
+      if (job.status === 'completed' || job.status === 'failed') {
+        this.processingJobs.delete(id);
+        this.processNext();
+      }
+    }
+  }
+
+  /**
+   * Mark job as processing
+   */
+  markAsProcessing(id: string): void {
+    this.processingJobs.add(id);
+    this.updateJob(id, { status: 'processing' });
+  }
+
+  /**
+   * Check if we can process more jobs
+   */
+  canProcessMore(): boolean {
+    return this.processingJobs.size < this.maxConcurrent;
+  }
+
+  /**
+   * Get next pending job
+   */
+  getNextPendingJob(): ScraperJob | undefined {
+    for (const job of this.jobs.values()) {
+      if (job.status === 'pending' && !this.processingJobs.has(job.id)) {
+        return job;
+      }
+    }
+    return undefined;
+  }
+
+  /**
+   * Process next job in queue if possible
+   */
+  private processNext(): void {
+    if (!this.canProcessMore()) {
+      logger.debug('Max concurrent jobs reached, waiting...');
+      return;
+    }
+
+    const nextJob = this.getNextPendingJob();
+    if (nextJob) {
+      this.markAsProcessing(nextJob.id);
+      this.emit('process-job', nextJob);
+    }
+  }
+
+  /**
+   * Get all jobs
+   */
+  getAllJobs(): ScraperJob[] {
+    return Array.from(this.jobs.values());
+  }
+
+  /**
+   * Get queue statistics
+   */
+  getStats(): { total: number; pending: number; processing: number; completed: number; failed: number } {
+    const jobs = Array.from(this.jobs.values());
+    return {
+      total: jobs.length,
+      pending: jobs.filter(j => j.status === 'pending').length,
+      processing: jobs.filter(j => j.status === 'processing').length,
+      completed: jobs.filter(j => j.status === 'completed').length,
+      failed: jobs.filter(j => j.status === 'failed').length
+    };
+  }
+}

src/scraper/ContentExtractor.ts

@@ -0,0 +1,113 @@
+import axios from 'axios';
+import * as cheerio from 'cheerio';
+import TurndownService from 'turndown';
+import { logger } from '../utils/logger';
+
+export class ContentExtractor {
+  private turndownService: TurndownService;
+
+  constructor() {
+    this.turndownService = new TurndownService({
+      headingStyle: 'atx',
+      codeBlockStyle: 'fenced'
+    });
+  }
+
+  /**
+   * Fetch and extract main content from a URL
+   */
+  async extractContent(url: string): Promise<string> {
+    try {
+      logger.info(`Extracting content from: ${url}`);
+
+      const response = await axios.get(url, {
+        timeout: 30000,
+        headers: {
+          'User-Agent': 'Mozilla/5.0 (compatible; WebshopScraper/1.0)'
+        }
+      });
+
+      const html = response.data;
+      const $ = cheerio.load(html);
+
+      // Remove unwanted elements
+      $('script').remove();
+      $('style').remove();
+      $('nav').remove();
+      $('header').remove();
+      $('footer').remove();
+      $('.navigation').remove();
+      $('.menu').remove();
+      $('.sidebar').remove();
+      $('.cookie').remove();
+      $('.banner').remove();
+      $('.advertisement').remove();
+      $('.ad').remove();
+
+      // Try to find main content area
+      let mainContent = '';
+
+      // Try common content selectors in order of preference
+      const contentSelectors = [
+        'main',
+        'article',
+        '[role="main"]',
+        '.content',
+        '.main-content',
+        '#content',
+        '#main-content',
+        '.page-content',
+        '.entry-content',
+        'body'
+      ];
+
+      for (const selector of contentSelectors) {
+        const element = $(selector).first();
+        if (element.length > 0) {
+          mainContent = element.html() || '';
+          if (mainContent.trim().length > 100) {
+            break;
+          }
+        }
+      }
+
+      if (!mainContent) {
+        logger.warn(`No main content found for ${url}, using body`);
+        mainContent = $('body').html() || '';
+      }
+
+      // Convert HTML to Markdown
+      const markdown = this.turndownService.turndown(mainContent);
+
+      // Clean up markdown (remove excessive newlines)
+      const cleanedMarkdown = markdown
+        .replace(/\n{3,}/g, '\n\n')
+        .trim();
+
+      logger.info(`Extracted ${cleanedMarkdown.length} characters of content from ${url}`);
+
+      return cleanedMarkdown;
+    } catch (error) {
+      logger.error(`Failed to extract content from ${url}`, error);
+      throw new Error(`Failed to extract content: ${error instanceof Error ? error.message : 'Unknown error'}`);
+    }
+  }
+
+  /**
+   * Extract contact information from content
+   */
+  extractContactInfo(html: string): { emails: string[]; phones: string[] } {
+    const $ = cheerio.load(html);
+    const text = $.text();
+
+    // Email regex
+    const emailRegex = /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/g;
+    const emails = [...new Set(text.match(emailRegex) || [])];
+
+    // Phone regex (international format)
+    const phoneRegex = /[\+]?[(]?[0-9]{1,4}[)]?[-\s\.]?[(]?[0-9]{1,4}[)]?[-\s\.]?[0-9]{1,4}[-\s\.]?[0-9]{1,9}/g;
+    const phones = [...new Set(text.match(phoneRegex) || [])].filter(p => p.length >= 8);
+
+    return { emails, phones };
+  }
+}

src/scraper/SitemapParser.ts

@@ -0,0 +1,118 @@
+import axios from 'axios';
+import { parseString } from 'xml2js';
+import { promisify } from 'util';
+import { SitemapUrl, WebshopType } from '../types';
+import { logger } from '../utils/logger';
+
+const parseXml = promisify(parseString);
+
+export class SitemapParser {
+  /**
+   * Detect webshop type from base URL and get sitemap URL
+   */
+  static getSitemapUrl(baseUrl: string): { sitemapUrl: string; webshopType: WebshopType } {
+    const url = new URL(baseUrl);
+    const hostname = url.hostname;
+
+    // ShopRenter detection
+    if (hostname.includes('shoprenter') || hostname.includes('.sr.hu')) {
+      return {
+        sitemapUrl: `${url.origin}/sitemap.xml`,
+        webshopType: 'shoprenter'
+      };
+    }
+
+    // Shopify detection
+    if (hostname.includes('myshopify.com') || hostname.includes('.shopify.')) {
+      return {
+        sitemapUrl: `${url.origin}/sitemap.xml`,
+        webshopType: 'shopify'
+      };
+    }
+
+    // WooCommerce (default - most common for custom domains)
+    return {
+      sitemapUrl: `${url.origin}/sitemap.xml`,
+      webshopType: 'woocommerce'
+    };
+  }
+
+  /**
+   * Fetch and parse sitemap XML
+   */
+  static async parseSitemap(sitemapUrl: string): Promise<SitemapUrl[]> {
+    try {
+      logger.info(`Fetching sitemap: ${sitemapUrl}`);
+
+      const response = await axios.get(sitemapUrl, {
+        timeout: 30000,
+        headers: {
+          'User-Agent': 'Mozilla/5.0 (compatible; WebshopScraper/1.0)'
+        }
+      });
+
+      const xml = response.data;
+      const parsed: any = await parseXml(xml);
+
+      let urls: SitemapUrl[] = [];
+
+      // Handle sitemap index (contains links to other sitemaps)
+      if (parsed.sitemapindex && parsed.sitemapindex.sitemap) {
+        logger.info('Sitemap index detected, fetching sub-sitemaps');
+        const sitemaps = parsed.sitemapindex.sitemap;
+
+        for (const sitemap of sitemaps) {
+          const subSitemapUrl = sitemap.loc[0];
+          try {
+            const subUrls = await this.parseSitemap(subSitemapUrl);
+            urls = urls.concat(subUrls);
+          } catch (error) {
+            logger.error(`Failed to parse sub-sitemap: ${subSitemapUrl}`, error);
+          }
+        }
+      }
+      // Handle regular sitemap (contains URLs)
+      else if (parsed.urlset && parsed.urlset.url) {
+        urls = parsed.urlset.url.map((urlEntry: any) => ({
+          loc: urlEntry.loc[0],
+          lastmod: urlEntry.lastmod?.[0],
+          changefreq: urlEntry.changefreq?.[0],
+          priority: urlEntry.priority?.[0]
+        }));
+      }
+
+      logger.info(`Parsed ${urls.length} URLs from sitemap`);
+      return urls;
+    } catch (error) {
+      logger.error(`Failed to parse sitemap: ${sitemapUrl}`, error);
+      throw new Error(`Failed to parse sitemap: ${error instanceof Error ? error.message : 'Unknown error'}`);
+    }
+  }
+
+  /**
+   * Filter URLs to find specific page types
+   */
+  static filterUrlsByType(urls: SitemapUrl[]): {
+    shipping: SitemapUrl[];
+    contacts: SitemapUrl[];
+    terms: SitemapUrl[];
+    faq: SitemapUrl[];
+  } {
+    const shippingKeywords = ['shipping', 'delivery', 'szallitas', 'kiszallitas'];
+    const contactKeywords = ['contact', 'kapcsolat', 'elerheto'];
+    const termsKeywords = ['terms', 'conditions', 'aszf', 'feltetel', 'privacy', 'policy'];
+    const faqKeywords = ['faq', 'gyakori', 'kerdes', 'questions'];
+
+    const containsKeyword = (url: string, keywords: string[]): boolean => {
+      const lowerUrl = url.toLowerCase();
+      return keywords.some(keyword => lowerUrl.includes(keyword));
+    };
+
+    return {
+      shipping: urls.filter(u => containsKeyword(u.loc, shippingKeywords)),
+      contacts: urls.filter(u => containsKeyword(u.loc, contactKeywords)),
+      terms: urls.filter(u => containsKeyword(u.loc, termsKeywords)),
+      faq: urls.filter(u => containsKeyword(u.loc, faqKeywords))
+    };
+  }
+}

src/scraper/WebshopScraper.ts

@@ -0,0 +1,85 @@
+import { ScraperResult, PageContent } from '../types';
+import { SitemapParser } from './SitemapParser';
+import { ContentExtractor } from './ContentExtractor';
+import { logger } from '../utils/logger';
+
+export class WebshopScraper {
+  private contentExtractor: ContentExtractor;
+
+  constructor() {
+    this.contentExtractor = new ContentExtractor();
+  }
+
+  /**
+   * Main scraping method
+   */
+  async scrape(baseUrl: string): Promise<ScraperResult> {
+    logger.info(`Starting scrape for: ${baseUrl}`);
+
+    try {
+      // Get sitemap URL based on webshop type
+      const { sitemapUrl, webshopType } = SitemapParser.getSitemapUrl(baseUrl);
+      logger.info(`Detected webshop type: ${webshopType}, sitemap: ${sitemapUrl}`);
+
+      // Parse sitemap
+      const urls = await SitemapParser.parseSitemap(sitemapUrl);
+
+      // Filter URLs by page type
+      const filteredUrls = SitemapParser.filterUrlsByType(urls);
+
+      logger.info('Filtered URLs', {
+        shipping: filteredUrls.shipping.length,
+        contacts: filteredUrls.contacts.length,
+        terms: filteredUrls.terms.length,
+        faq: filteredUrls.faq.length
+      });
+
+      // Extract content from each page type
+      const result: ScraperResult = {
+        initial_sitemap: sitemapUrl,
+        shipping_informations: await this.extractPageContent(filteredUrls.shipping),
+        contacts: await this.extractPageContent(filteredUrls.contacts),
+        terms_of_conditions: await this.extractPageContent(filteredUrls.terms),
+        faq: await this.extractPageContent(filteredUrls.faq)
+      };
+
+      logger.info('Scraping completed successfully');
+      return result;
+    } catch (error) {
+      logger.error('Scraping failed', error);
+      throw error;
+    }
+  }
+
+  /**
+   * Extract content from the best matching URL
+   */
+  private async extractPageContent(urls: any[]): Promise<PageContent | null> {
+    if (urls.length === 0) {
+      return null;
+    }
+
+    // Try to extract from the first URL (usually the most relevant)
+    try {
+      const url = urls[0].loc;
+      const content = await this.contentExtractor.extractContent(url);
+      return { url, content };
+    } catch (error) {
+      logger.error('Failed to extract page content', error);
+
+      // Try next URL if available
+      if (urls.length > 1) {
+        try {
+          const url = urls[1].loc;
+          const content = await this.contentExtractor.extractContent(url);
+          return { url, content };
+        } catch (error2) {
+          logger.error('Failed to extract from fallback URL', error2);
+          return null;
+        }
+      }
+
+      return null;
+    }
+  }
+}

src/types/index.ts

@@ -0,0 +1,31 @@
+export interface ScraperJob {
+  id: string;
+  sitemapUrl: string;
+  status: 'pending' | 'processing' | 'completed' | 'failed';
+  result?: ScraperResult;
+  error?: string;
+  createdAt: Date;
+  updatedAt: Date;
+}
+
+export interface ScraperResult {
+  initial_sitemap: string;
+  shipping_informations: PageContent | null;
+  contacts: PageContent | null;
+  terms_of_conditions: PageContent | null;
+  faq: PageContent | null;
+}
+
+export interface PageContent {
+  url: string;
+  content: string;
+}
+
+export type WebshopType = 'shoprenter' | 'woocommerce' | 'shopify';
+
+export interface SitemapUrl {
+  loc: string;
+  lastmod?: string;
+  changefreq?: string;
+  priority?: string;
+}

src/utils/logger.ts

@@ -0,0 +1,35 @@
+/**
+ * Simple logger that outputs to stdout and stderr
+ * Compatible with systemd/journalctl
+ */
+export class Logger {
+  private prefix: string;
+
+  constructor(prefix: string = 'WebshopScraper') {
+    this.prefix = prefix;
+  }
+
+  private formatMessage(level: string, message: string, ...args: any[]): string {
+    const timestamp = new Date().toISOString();
+    const formattedArgs = args.length > 0 ? ' ' + JSON.stringify(args) : '';
+    return `[${timestamp}] [${level}] [${this.prefix}] ${message}${formattedArgs}`;
+  }
+
+  info(message: string, ...args: any[]): void {
+    console.log(this.formatMessage('INFO', message, ...args));
+  }
+
+  error(message: string, ...args: any[]): void {
+    console.error(this.formatMessage('ERROR', message, ...args));
+  }
+
+  warn(message: string, ...args: any[]): void {
+    console.warn(this.formatMessage('WARN', message, ...args));
+  }
+
+  debug(message: string, ...args: any[]): void {
+    console.log(this.formatMessage('DEBUG', message, ...args));
+  }
+}
+
+export const logger = new Logger();

tsconfig.json

@@ -0,0 +1,20 @@
+{
+  "compilerOptions": {
+    "target": "ES2020",
+    "module": "commonjs",
+    "lib": ["ES2020"],
+    "outDir": "./dist",
+    "rootDir": "./src",
+    "strict": true,
+    "esModuleInterop": true,
+    "skipLibCheck": true,
+    "forceConsistentCasingInFileNames": true,
+    "resolveJsonModule": true,
+    "moduleResolution": "node",
+    "declaration": true,
+    "declarationMap": true,
+    "sourceMap": true
+  },
+  "include": ["src/**/*"],
+  "exclude": ["node_modules", "dist"]
+}