# Webshop Scraper

A TypeScript-based web scraper for extracting information from webshops (ShopRenter, WooCommerce, Shopify). The application provides a REST API for submitting scraping jobs and retrieving results.

## Features

- **Multi-webshop Support**: Automatically detects and scrapes ShopRenter, WooCommerce, and Shopify stores
- **REST API**: HTTP API for job submission and status tracking
- **Bearer Authentication**: Secure API access with pre-shared key
- **Job Queue**: Prevents system overload and race conditions with configurable concurrency
- **Content Extraction**: Extracts shipping information, contacts, terms & conditions, and FAQ
- **Markdown Conversion**: Cleans HTML and converts to markdown format
- **Systemd Integration**: Install as a system service with proper logging
- **Logging**: Outputs to stdout/stderr for systemd/journalctl integration

## Requirements

- Node.js (v16 or higher)
- npm
- Linux with systemd (for service installation)

## Installation

### As a systemd service (recommended)

1. Clone the repository:
```bash
git clone <repository-url>
cd webshop-scraper
```

2. Run the installation script:
```bash
sudo ./scripts/install.sh
```

The script will:
- Prompt for the API key
- Ask for port number (default: 3000)
- Ask for max concurrent jobs (default: 3)
- Install dependencies
- Build the application
- Create a systemd service
- Start the service

### Manual installation

1. Install dependencies:
```bash
npm install
```

2. Build the application:
```bash
npm run build
```

3. Create a `.env` file:
```bash
cp .env.example .env
```

4. Edit `.env` and set your configuration:
```
API_KEY=your-secret-key-here
PORT=3000
MAX_CONCURRENT_JOBS=3
```

5. Start the application:
```bash
npm start
```

## Usage

### Starting the service

```bash
sudo systemctl start webshop-scraper
```

### Stopping the service

```bash
sudo systemctl stop webshop-scraper
```

### Viewing logs

```bash
sudo journalctl -u webshop-scraper -f
```

### API Endpoints

All API endpoints (except `/health`) require Bearer authentication.

#### Health Check

```bash
GET /health
```

Example:
```bash
curl http://localhost:3000/health
```

#### Create Scraping Job

```bash
POST /api/jobs
Authorization: Bearer <your-api-key>
Content-Type: application/json

{
  "url": "https://example-shop.com"
}
```

Example:
```bash
curl -X POST http://localhost:3000/api/jobs \
  -H "Authorization: Bearer your-secret-key-here" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-shop.com"}'
```

Response:
```json
{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "pending",
  "message": "Job created successfully"
}
```

#### Get Job Status

```bash
GET /api/jobs/:id
Authorization: Bearer <your-api-key>
```

Example:
```bash
curl http://localhost:3000/api/jobs/a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
  -H "Authorization: Bearer your-secret-key-here"
```

Response (when completed):
```json
{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "created_at": "2024-01-01T12:00:00.000Z",
  "updated_at": "2024-01-01T12:05:00.000Z",
  "result": {
    "initial_sitemap": "https://example-shop.com/sitemap.xml",
    "shipping_informations": {
      "url": "https://example-shop.com/shipping",
      "content": "# Shipping Information\n\n..."
    },
    "contacts": {
      "url": "https://example-shop.com/contact",
      "content": "# Contact Us\n\n..."
    },
    "terms_of_conditions": {
      "url": "https://example-shop.com/terms",
      "content": "# Terms and Conditions\n\n..."
    },
    "faq": {
      "url": "https://example-shop.com/faq",
      "content": "# FAQ\n\n..."
    }
  }
}
```

If a page type is not found, it will be `null`:
```json
{
  "faq": null
}
```

#### List All Jobs

```bash
GET /api/jobs
Authorization: Bearer <your-api-key>
```

Example:
```bash
curl http://localhost:3000/api/jobs \
  -H "Authorization: Bearer your-secret-key-here"
```

## Configuration

Configuration is done via environment variables:

- `API_KEY`: Pre-shared key for Bearer authentication (required)
- `PORT`: HTTP server port (default: 3000)
- `MAX_CONCURRENT_JOBS`: Maximum number of concurrent scraping jobs (default: 3)

## Uninstallation

To remove the service and all files:

```bash
sudo ./scripts/uninstall.sh
```

## Development

### Run in development mode

```bash
npm run dev
```

### Build

```bash
npm run build
```

### Watch mode

```bash
npm run watch
```

## Architecture

- **REST API Server**: Express.js server with Bearer authentication
- **Job Queue**: In-memory queue with concurrency control
- **Sitemap Parser**: Detects webshop type and parses sitemap.xml
- **Content Extractor**: Fetches pages and extracts main content
- **HTML to Markdown**: Converts HTML to clean markdown using Turndown

## Logging

All logs are written to stdout/stderr and can be viewed using journalctl:

```bash
# Follow logs
sudo journalctl -u webshop-scraper -f

# View logs from today
sudo journalctl -u webshop-scraper --since today

# View last 100 lines
sudo journalctl -u webshop-scraper -n 100
```

## License

ISC