#1 feat: implement the project

Implement this empty project using typescript.

• create shellscripts which install/uninstall this app as systemd service
• the app must logging out to the std out and err which can be handled by systemd / journalctl
• implement a queue system, which prevents high system usage and reace conditions
• the project must know to scapre 3 types of webshop: shoprenter, woocommerce, shopify
• the app must listen on a custom port using http protocoll, so need to expose rest api endpoints to accepp new jobs
• on the rest api, there must be a predefined shared key which must be sent by the client in header Bearer
• the client must send a starting url which is a shop's base url (this app must known these 3 webshops sitemap.xml url )
• the cralwer must give back a job id
• the client must known the job id and the client will query an endpoint to get the status of the job

• the app must find these parts of the webshops: shipping informations, the main contacts of the shop (email, phone), terms of conditions, faq. So when the client send the initial url of the sitemap(.xml), the scraper start crawling the sitemap to find these pages on the site. Then when finished the job, the job details must contains a json structured response if the queue job finished, for example:

  {
  "initial_sitemap": "https://.../sitemap.xml",
  "shipping_informations": {
    "url": "https://.../shipping",
    "content": ""// <-- the content of the site converted into markdown and cleaned up from usefulness elements (menu, navigation etc.. we only need the main content of the page)
  }
  ...
  }
  

If some category missing, then the obejct key must be null:

{
 "shipping_informations": null
}

The preshared key must be set using environment variable. So when the systemd install script runs, ask the user to add the key reading from the input

Take care to the REST API and the queue jobs must run in parallel

@fszontagh I'm starting work on this issue now!

I'll implement the webshop scraper with the following approach:

1. TypeScript project setup with proper dependencies for web scraping and HTTP server
2. REST API server with Bearer token authentication for job submissions and status queries
3. Queue system to manage scraping jobs and prevent race conditions
4. Web scraper supporting ShopRenter, WooCommerce, and Shopify sitemap parsing
5. Content extraction for shipping info, contacts, terms & conditions, and FAQ pages
6. HTML to Markdown conversion for clean content output
7. Shell scripts for systemd service installation/uninstallation with environment variable configuration
8. Proper logging to stdout/stderr for systemd/journalctl integration

I'll start implementing the core components and keep you updated on progress.

@fszontagh I've completed the implementation! 🎉

What's been implemented:

✅ Core Features

• TypeScript project structure with proper configuration
• REST API server with Express.js
• Bearer token authentication for secure API access
• Job queue system with configurable concurrency to prevent race conditions and high system usage
• Multi-webshop support - Automatically detects and scrapes:
  • ShopRenter
  • WooCommerce
    
  • Shopify
• Content extraction for:
  • Shipping information
  • Contact details (email, phone)
  • Terms & Conditions
  • FAQ pages
• HTML to Markdown conversion with cleaned content (removes navigation, menus, etc.)
• Systemd integration with install/uninstall scripts

📁 Project Structure

webshop-scraper/
├── src/
│   ├── api/
│   │   ├── middleware/auth.ts    # Bearer authentication
│   │   ├── routes.ts             # API endpoints
│   │   └── server.ts             # Express server setup
│   ├── config/
│   │   └── index.ts              # Environment configuration
│   ├── queue/
│   │   └── JobQueue.ts           # Job queue with concurrency control
│   ├── scraper/
│   │   ├── ContentExtractor.ts   # HTML to Markdown conversion
│   │   ├── SitemapParser.ts      # Sitemap parsing & webshop detection
│   │   └── WebshopScraper.ts     # Main scraping logic
│   ├── types/
│   │   └── index.ts              # TypeScript type definitions
│   ├── utils/
│   │   └── logger.ts             # Stdout/stderr logging
│   └── index.ts                  # Application entry point
├── scripts/
│   ├── install.sh                # Systemd installation script
│   └── uninstall.sh              # Systemd uninstallation script
├── package.json
├── tsconfig.json
└── README.md                     # Complete documentation

🔌 API Endpoints

Health Check (no auth required):

GET /health

Create Scraping Job (requires Bearer token):

POST /api/jobs
Authorization: Bearer <your-api-key>
Content-Type: application/json

{"url": "https://example-shop.com"}

Get Job Status (requires Bearer token):

GET /api/jobs/:id
Authorization: Bearer <your-api-key>

List All Jobs (requires Bearer token):

GET /api/jobs
Authorization: Bearer <your-api-key>

🚀 Installation

1. As systemd service (recommended):

   sudo ./scripts/install.sh
   

   The script will prompt for:

2. API key (stored as environment variable)

3. Port number (default: 3000)

4. Max concurrent jobs (default: 3)

5. Manual installation:

   npm install
   npm run build
   cp .env.example .env
   # Edit .env with your settings
   npm start
   

📋 Environment Variables

• API_KEY - Pre-shared key for Bearer authentication (required)
• PORT - HTTP server port (default: 3000)
• MAX_CONCURRENT_JOBS - Maximum concurrent scraping jobs (default: 3)

📝 Logging

All logs are written to stdout/stderr and can be viewed via journalctl:

sudo journalctl -u webshop-scraper -f

✨ Key Implementation Details

1. Parallel Processing: The REST API and job queue run in parallel using Node.js event system
2. Race Condition Prevention: Job queue ensures only N jobs run concurrently (configurable)
3. Webshop Detection: Automatically detects webshop type from URL and knows sitemap locations
4. Content Cleaning: Removes navigation, menus, sidebars, ads before Markdown conversion
5. Graceful Shutdown: Handles SIGTERM/SIGINT for clean systemd service management
6. Null Handling: Returns null for missing page types as specified

All changes have been committed and pushed to the repository. The application is ready to use!