docs: complete comprehensive Qdrant documentation

- Add detailed implementation guide (QDRANT_IMPLEMENTATION.md)
- Complete API reference with all endpoints (API_REFERENCE.md)
- Step-by-step setup guide with configuration (SETUP_GUIDE.md)
- Update main README with Qdrant features and MCP tools
- Include troubleshooting, performance tuning, and cost management
- Document collection naming patterns, security considerations
- Provide examples for all API endpoints and MCP tool usage

Documentation covers:
- Architecture overview with all core components
- Database schema extensions and migrations
- Environment configuration and setup steps
- MCP tool signatures and usage examples
- Webhook integration and error handling
- Monitoring, analytics, and debugging procedures
- Security best practices and data protection
- Performance optimization and cost management
- Future enhancement roadmap

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

README.md

@@ -7,10 +7,14 @@ A TypeScript-based web scraper for extracting information from webshops with per
 - **Multi-webshop Support**: ShopRenter, WooCommerce, and Shopify
 - **Persistent Storage**: SQLite database with UUID tracking and analytics
 - **Scheduled Scraping**: Automated scraping based on sitemap frequency rules
-- **Content Change Detection**: MD5 hash-based change tracking for all content
+- **Content Change Detection**: SHA-256 hash-based change tracking for all content
+- **Vector Search Integration**: Qdrant-powered semantic search with OpenRouter embeddings
+- **MCP Tools**: Model Context Protocol integration for LLM applications
 - **REST API**: Full CRUD operations for shops and scraping jobs
+- **Web Interface**: Complete administrative dashboard with real-time monitoring
 - **Bearer Authentication**: Secure API access
 - **Job Queue**: Configurable concurrency control
+- **Webhook Support**: Real-time notifications for scraping events
 - **Systemd Integration**: Install as a system service
 
 ## Quick Start
@@ -31,9 +35,22 @@ npm run build
 
 3. Configure environment:
 ```bash
+# Basic Configuration
 API_KEY=your-secret-key
 PORT=3000
 MAX_CONCURRENT_JOBS=3
+
+# Qdrant Vector Search (Optional)
+QDRANT_API_URL=http://localhost:6333
+QDRANT_API_KEY=your-qdrant-key
+
+# OpenRouter for Embeddings (Optional)
+OPENROUTER_API_KEY=your-openrouter-key
+OPENROUTER_MODEL=openai/text-embedding-3-large
+
+# MCP Integration (Optional)
+MCP_ENABLED=true
+MCP_TRANSPORT=stdio
 ```
 
 4. Start:
@@ -47,9 +64,28 @@ npm start
 sudo ./scripts/install.sh
 ```
 
+## Vector Search & MCP Integration
+
+The system includes optional Qdrant vector search capabilities for semantic search of scraped content:
+
+- **Automatic Embedding**: Content is automatically embedded using OpenRouter's text-embedding-3-large model
+- **Semantic Search**: 4 MCP tools provide semantic search capabilities for LLM applications
+- **Real-time Management**: Web interface for enabling/disabling and monitoring embeddings
+- **Error Tracking**: Comprehensive error logging and debugging capabilities
+
+### MCP Tools Available
+
+- `search_shipping_info(shop_id, query, limit?)` - Search shipping and delivery information
+- `search_contact_info(shop_id, query, limit?)` - Search contact and support information
+- `search_terms_info(shop_id, query, limit?)` - Search terms of service and policies
+- `search_faq_info(shop_id, query, limit?)` - Search frequently asked questions
+
+See the [Qdrant Setup Guide](./docs/SETUP_GUIDE.md) for complete configuration instructions.
+
 ## API Documentation
 
 See [API Documentation](./docs/API.md) for complete endpoint reference.
+See [Qdrant API Reference](./docs/API_REFERENCE.md) for vector search endpoints.
 
 ### Quick Examples
 

docs/API_REFERENCE.md

@@ -0,0 +1,557 @@
+# API Reference - Qdrant & MCP Endpoints
+
+This document provides detailed API reference for the Qdrant vector search and MCP (Model Context Protocol) endpoints.
+
+## Base URL
+
+All endpoints are relative to the API base URL: `http://localhost:3000` (or your configured host/port)
+
+## Authentication
+
+All API endpoints require authentication via the `X-API-Key` header:
+
+```http
+X-API-Key: your-api-key
+```
+
+## Qdrant Management Endpoints
+
+### Get Qdrant Statistics
+
+Get system-wide Qdrant statistics and metrics.
+
+```http
+GET /api/qdrant/stats
+```
+
+**Response:**
+```json
+{
+  "data": {
+    "total_collections": 15,
+    "total_vectors": 1250,
+    "enabled_shops": 8,
+    "recent_embeddings": 45,
+    "embedding_queue_size": 3
+  },
+  "success": true
+}
+```
+
+### Health Check
+
+Check the health status of Qdrant services.
+
+```http
+GET /api/qdrant/health
+```
+
+**Response:**
+```json
+{
+  "data": {
+    "healthy": true,
+    "services": {
+      "qdrant": true,
+      "embedding": true,
+      "database": true
+    },
+    "errors": []
+  },
+  "success": true
+}
+```
+
+### Get Shop Qdrant Information
+
+Get comprehensive Qdrant information for a specific shop.
+
+```http
+GET /api/qdrant/shops/:shopId
+```
+
+**Parameters:**
+- `shopId` (string, required): Shop identifier
+
+**Response:**
+```json
+{
+  "data": {
+    "shop": {
+      "id": "shop-123",
+      "custom_id": "shop123",
+      "url": "https://example-shop.com",
+      "qdrant_enabled": true,
+      "created_at": "2023-01-01T00:00:00Z",
+      "updated_at": "2023-01-01T00:00:00Z"
+    },
+    "embeddings": [
+      {
+        "id": "emb-456",
+        "shop_content_id": "content-789",
+        "collection_name": "shop123-shipping_1",
+        "point_id": "point-uuid",
+        "embedding_status": "completed",
+        "created_at": "2023-01-01T00:00:00Z",
+        "updated_at": "2023-01-01T00:00:00Z"
+      }
+    ],
+    "errors": [
+      {
+        "id": "err-123",
+        "shop_id": "shop-123",
+        "error_type": "embedding",
+        "error_message": "Rate limit exceeded",
+        "metadata": {},
+        "created_at": "2023-01-01T00:00:00Z"
+      }
+    ],
+    "collections": [
+      {
+        "collection_name": "shop123-shipping_1",
+        "vectors_count": 125,
+        "config": {
+          "vector_size": 3072,
+          "distance": "cosine"
+        }
+      }
+    ]
+  },
+  "success": true
+}
+```
+
+### Get Shop Embeddings
+
+Get all embeddings for a specific shop.
+
+```http
+GET /api/qdrant/shops/:shopId/embeddings
+```
+
+**Parameters:**
+- `shopId` (string, required): Shop identifier
+
+**Response:**
+```json
+{
+  "data": {
+    "embeddings": [
+      {
+        "id": "emb-456",
+        "shop_content_id": "content-789",
+        "collection_name": "shop123-shipping_1",
+        "point_id": "point-uuid",
+        "embedding_status": "completed",
+        "created_at": "2023-01-01T00:00:00Z",
+        "updated_at": "2023-01-01T00:00:00Z"
+      }
+    ]
+  },
+  "success": true
+}
+```
+
+### Get Shop Qdrant Errors
+
+Get all Qdrant errors for a specific shop.
+
+```http
+GET /api/qdrant/shops/:shopId/errors
+```
+
+**Parameters:**
+- `shopId` (string, required): Shop identifier
+
+**Response:**
+```json
+{
+  "data": {
+    "errors": [
+      {
+        "id": "err-123",
+        "shop_id": "shop-123",
+        "error_type": "embedding",
+        "error_message": "OpenRouter API rate limit exceeded",
+        "metadata": {
+          "content_id": "content-789",
+          "retry_count": 3
+        },
+        "created_at": "2023-01-01T00:00:00Z"
+      }
+    ]
+  },
+  "success": true
+}
+```
+
+### Clear Shop Qdrant Errors
+
+Clear all Qdrant errors for a specific shop.
+
+```http
+DELETE /api/qdrant/shops/:shopId/errors
+```
+
+**Parameters:**
+- `shopId` (string, required): Shop identifier
+
+**Response:**
+```json
+{
+  "data": {
+    "message": "Qdrant errors cleared successfully"
+  },
+  "success": true
+}
+```
+
+### Enable/Disable Qdrant for Shop
+
+Enable or disable Qdrant functionality for a specific shop.
+
+```http
+PATCH /api/shops/:shopId/qdrant
+```
+
+**Parameters:**
+- `shopId` (string, required): Shop identifier
+
+**Request Body:**
+```json
+{
+  "enabled": true
+}
+```
+
+**Response:**
+```json
+{
+  "data": {
+    "shop_id": "shop-123",
+    "qdrant_enabled": true,
+    "message": "Qdrant settings updated successfully"
+  },
+  "success": true
+}
+```
+
+### Trigger Cleanup
+
+Manually trigger Qdrant cleanup operations.
+
+```http
+POST /api/qdrant/cleanup
+```
+
+**Response:**
+```json
+{
+  "data": {
+    "message": "Cleanup process initiated successfully"
+  },
+  "success": true
+}
+```
+
+### Delete Collection
+
+Delete a specific Qdrant collection.
+
+```http
+DELETE /api/qdrant/collections/:collectionName
+```
+
+**Parameters:**
+- `collectionName` (string, required): Name of the collection to delete
+
+**Response:**
+```json
+{
+  "data": {
+    "message": "Collection deleted successfully"
+  },
+  "success": true
+}
+```
+
+### Schedule Shop Deletion
+
+Schedule a shop for Qdrant data deletion with 24-hour delay.
+
+```http
+POST /api/qdrant/shops/:shopId/schedule-deletion
+```
+
+**Parameters:**
+- `shopId` (string, required): Shop identifier
+
+**Response:**
+```json
+{
+  "data": {
+    "message": "Shop scheduled for Qdrant deletion",
+    "scheduled_at": "2023-01-02T00:00:00Z"
+  },
+  "success": true
+}
+```
+
+## MCP (Model Context Protocol) Endpoints
+
+### Get MCP Statistics
+
+Get system-wide MCP statistics and usage metrics.
+
+```http
+GET /api/mcp/stats
+```
+
+**Response:**
+```json
+{
+  "data": {
+    "total_calls": 1250,
+    "successful_calls": 1180,
+    "failed_calls": 70,
+    "average_response_time": 325,
+    "top_tools": [
+      {
+        "tool_name": "search_shipping_info",
+        "call_count": 450
+      },
+      {
+        "tool_name": "search_faq_info",
+        "call_count": 380
+      }
+    ],
+    "recent_activity": [
+      {
+        "id": "log-123",
+        "shop_id": "shop-123",
+        "tool_name": "search_shipping_info",
+        "query": "international delivery",
+        "response_time_ms": 285,
+        "results_count": 5,
+        "success": true,
+        "error_message": null,
+        "created_at": "2023-01-01T00:00:00Z"
+      }
+    ]
+  },
+  "success": true
+}
+```
+
+### Get Shop MCP Logs
+
+Get MCP usage logs for a specific shop.
+
+```http
+GET /api/mcp/shops/:shopId/logs
+```
+
+**Parameters:**
+- `shopId` (string, required): Shop identifier
+
+**Query Parameters:**
+- `limit` (number, optional): Number of logs to return (default: 50)
+- `success` (boolean, optional): Filter by success status
+- `tool_name` (string, optional): Filter by tool name
+
+**Response:**
+```json
+{
+  "data": {
+    "logs": [
+      {
+        "id": "log-123",
+        "shop_id": "shop-123",
+        "tool_name": "search_shipping_info",
+        "query": "international delivery times",
+        "response_time_ms": 285,
+        "results_count": 5,
+        "success": true,
+        "error_message": null,
+        "created_at": "2023-01-01T00:00:00Z"
+      },
+      {
+        "id": "log-124",
+        "shop_id": "shop-123",
+        "tool_name": "search_contact_info",
+        "query": "customer service phone",
+        "response_time_ms": 412,
+        "results_count": 3,
+        "success": true,
+        "error_message": null,
+        "created_at": "2023-01-01T00:05:00Z"
+      }
+    ]
+  },
+  "success": true
+}
+```
+
+### Clear Shop MCP Logs
+
+Clear all MCP logs for a specific shop.
+
+```http
+DELETE /api/mcp/shops/:shopId/logs
+```
+
+**Parameters:**
+- `shopId` (string, required): Shop identifier
+
+**Response:**
+```json
+{
+  "data": {
+    "message": "MCP logs cleared successfully"
+  },
+  "success": true
+}
+```
+
+### Get MCP Analytics
+
+Get comprehensive MCP analytics and insights.
+
+```http
+GET /api/mcp/analytics
+```
+
+**Response:**
+```json
+{
+  "data": {
+    "total_calls": 1250,
+    "successful_calls": 1180,
+    "failed_calls": 70,
+    "average_response_time": 325,
+    "shops_using_mcp": 8,
+    "top_tools": [
+      {
+        "tool_name": "search_shipping_info",
+        "call_count": 450
+      },
+      {
+        "tool_name": "search_faq_info",
+        "call_count": 380
+      },
+      {
+        "tool_name": "search_contact_info",
+        "call_count": 240
+      },
+      {
+        "tool_name": "search_terms_info",
+        "call_count": 180
+      }
+    ],
+    "recent_activity": [
+      {
+        "id": "log-123",
+        "shop_id": "shop-123",
+        "tool_name": "search_shipping_info",
+        "query": "international delivery",
+        "response_time_ms": 285,
+        "results_count": 5,
+        "success": true,
+        "error_message": null,
+        "created_at": "2023-01-01T00:00:00Z"
+      }
+    ]
+  },
+  "success": true
+}
+```
+
+## Error Responses
+
+All endpoints may return error responses in the following format:
+
+```json
+{
+  "error": "Error message description",
+  "success": false
+}
+```
+
+### Common HTTP Status Codes
+
+- `200 OK`: Request successful
+- `400 Bad Request`: Invalid request parameters
+- `401 Unauthorized`: Missing or invalid API key
+- `404 Not Found`: Resource not found
+- `500 Internal Server Error`: Server error
+
+### Error Types
+
+#### Qdrant Errors
+- `connection`: Qdrant service unavailable
+- `embedding`: OpenRouter API issues
+- `search`: Vector search failures
+- `collection`: Collection management errors
+
+#### MCP Errors
+- `tool_not_found`: Requested tool doesn't exist
+- `invalid_parameters`: Tool called with invalid parameters
+- `search_failed`: Vector search operation failed
+- `no_embeddings`: No embeddings found for shop
+
+## Rate Limiting
+
+API endpoints are subject to rate limiting:
+
+- **General endpoints**: 100 requests per minute per API key
+- **Search endpoints**: 50 requests per minute per API key
+- **Administrative endpoints**: 20 requests per minute per API key
+
+Rate limit headers are included in responses:
+
+```http
+X-RateLimit-Limit: 100
+X-RateLimit-Remaining: 95
+X-RateLimit-Reset: 1609459200
+```
+
+## Examples
+
+### Enable Qdrant for a Shop
+
+```bash
+curl -X PATCH "http://localhost:3000/api/shops/shop-123/qdrant" \
+  -H "X-API-Key: your-api-key" \
+  -H "Content-Type: application/json" \
+  -d '{"enabled": true}'
+```
+
+### Get Shop Qdrant Status
+
+```bash
+curl -X GET "http://localhost:3000/api/qdrant/shops/shop-123" \
+  -H "X-API-Key: your-api-key"
+```
+
+### Clear Shop Errors
+
+```bash
+curl -X DELETE "http://localhost:3000/api/qdrant/shops/shop-123/errors" \
+  -H "X-API-Key: your-api-key"
+```
+
+### Get MCP Analytics
+
+```bash
+curl -X GET "http://localhost:3000/api/mcp/analytics" \
+  -H "X-API-Key: your-api-key"
+```
+
+### Trigger System Cleanup
+
+```bash
+curl -X POST "http://localhost:3000/api/qdrant/cleanup" \
+  -H "X-API-Key: your-api-key"
+```
+
+This API reference provides comprehensive access to all Qdrant and MCP functionality, enabling full management and monitoring of the vector search system.

docs/QDRANT_IMPLEMENTATION.md

@@ -0,0 +1,423 @@
+# Qdrant Vector Search Implementation Guide
+
+This document provides comprehensive information about the Qdrant vector search implementation in the webshop scraper system.
+
+## Overview
+
+The Qdrant integration enables semantic search capabilities for scraped webshop content using text embeddings. The system automatically processes scraped content through OpenRouter's text-embedding-3-large model and stores vectors in Qdrant for efficient similarity search via MCP (Model Context Protocol) tools.
+
+## Architecture
+
+### Core Components
+
+1. **QdrantService** (`src/services/QdrantService.ts`)
+   - Manages Qdrant database connections and operations
+   - Handles collection creation and management
+   - Performs vector search operations
+
+2. **EmbeddingService** (`src/services/EmbeddingService.ts`)
+   - Integrates with OpenRouter API for text embeddings
+   - Uses text-embedding-3-large model (3072 dimensions)
+   - Handles text preprocessing and batching
+
+3. **QdrantEmbeddingWorkflow** (`src/services/QdrantEmbeddingWorkflow.ts`)
+   - Orchestrates the embedding pipeline
+   - Integrates with scraping workflow
+   - Manages error handling and retries
+
+4. **QdrantCleanupService** (`src/services/QdrantCleanupService.ts`)
+   - Handles automated cleanup operations
+   - Manages 24-hour deletion delays
+   - Processes deletion queues
+
+5. **MCP Server** (`src/mcp/McpServer.ts`)
+   - Provides semantic search tools for LLMs
+   - Implements 4 category-specific search tools
+   - Handles analytics and logging
+
+6. **SchedulerService** (`src/services/SchedulerService.ts`)
+   - Background task management
+   - Health monitoring
+   - Automated maintenance
+
+## Configuration
+
+### Environment Variables
+
+```bash
+# Qdrant Configuration
+QDRANT_API_URL=http://localhost:6333
+QDRANT_API_KEY=your-qdrant-api-key  # Optional
+
+# OpenRouter Configuration (for embeddings)
+OPENROUTER_API_KEY=your-openrouter-api-key
+OPENROUTER_MODEL=openai/text-embedding-3-large
+
+# MCP Configuration
+MCP_ENABLED=true
+MCP_TRANSPORT=stdio  # or http
+MCP_PORT=3001        # for HTTP transport
+```
+
+### Database Schema Extensions
+
+The implementation extends the existing SQLite database with new tables:
+
+```sql
+-- Enable Qdrant per shop
+ALTER TABLE shops ADD COLUMN qdrant_enabled BOOLEAN DEFAULT FALSE;
+
+-- Track embeddings
+CREATE TABLE shop_embeddings (
+  id TEXT PRIMARY KEY,
+  shop_content_id TEXT NOT NULL,
+  collection_name TEXT NOT NULL,
+  point_id TEXT NOT NULL,
+  embedding_status TEXT NOT NULL CHECK (embedding_status IN ('pending', 'completed', 'failed')),
+  created_at TEXT NOT NULL,
+  updated_at TEXT NOT NULL,
+  FOREIGN KEY (shop_content_id) REFERENCES shop_contents (id) ON DELETE CASCADE
+);
+
+-- Deletion queue with 24h delays
+CREATE TABLE qdrant_deletion_queue (
+  id TEXT PRIMARY KEY,
+  shop_id TEXT NOT NULL,
+  custom_id TEXT NOT NULL,
+  status TEXT NOT NULL DEFAULT 'pending',
+  scheduled_deletion_at TEXT NOT NULL,
+  created_at TEXT NOT NULL,
+  updated_at TEXT NOT NULL,
+  error_message TEXT,
+  FOREIGN KEY (shop_id) REFERENCES shops (id) ON DELETE CASCADE
+);
+
+-- Error tracking
+CREATE TABLE qdrant_errors (
+  id TEXT PRIMARY KEY,
+  shop_id TEXT NOT NULL,
+  error_type TEXT NOT NULL,
+  error_message TEXT NOT NULL,
+  metadata TEXT,
+  created_at TEXT NOT NULL,
+  FOREIGN KEY (shop_id) REFERENCES shops (id) ON DELETE CASCADE
+);
+
+-- MCP analytics
+CREATE TABLE mcp_logs (
+  id TEXT PRIMARY KEY,
+  shop_id TEXT NOT NULL,
+  tool_name TEXT NOT NULL,
+  query TEXT NOT NULL,
+  response_time_ms INTEGER NOT NULL,
+  results_count INTEGER NOT NULL,
+  success BOOLEAN NOT NULL,
+  error_message TEXT,
+  created_at TEXT NOT NULL,
+  FOREIGN KEY (shop_id) REFERENCES shops (id) ON DELETE CASCADE
+);
+```
+
+## Collection Naming Convention
+
+Collections follow a specific pattern for organization and isolation:
+
+```
+{shop_custom_id}-{category}_{iterator}
+```
+
+**Examples:**
+- `shop123-shipping_1` - First shipping information collection
+- `shop123-contacts_2` - Second contacts collection
+- `shop456-faq_1` - First FAQ collection
+
+**Categories:**
+- `shipping` - Shipping and delivery information
+- `contacts` - Contact information and support
+- `terms` - Terms of service and conditions
+- `faq` - Frequently asked questions
+
+## MCP Tools
+
+The system provides 4 semantic search tools accessible to LLMs:
+
+### Tool Signatures
+
+```typescript
+search_shipping_info(shop_id: string, query: string, limit?: number): Promise<SearchResult[]>
+search_contact_info(shop_id: string, query: string, limit?: number): Promise<SearchResult[]>
+search_terms_info(shop_id: string, query: string, limit?: number): Promise<SearchResult[]>
+search_faq_info(shop_id: string, query: string, limit?: number): Promise<SearchResult[]>
+```
+
+### Usage Examples
+
+```javascript
+// Search for shipping policies
+const results = await search_shipping_info("shop123", "international delivery times", 5);
+
+// Find contact information
+const contacts = await search_contact_info("shop123", "customer service phone", 3);
+
+// Look up return policies
+const terms = await search_terms_info("shop123", "return policy within 30 days", 5);
+
+// Find specific FAQ answers
+const faqs = await search_faq_info("shop123", "how to track my order", 10);
+```
+
+### Response Format
+
+```typescript
+interface SearchResult {
+  id: string;
+  score: number;  // Similarity score (0-1)
+  payload: {
+    shop_id: string;
+    content_id: string;
+    content_type: string;
+    url: string;
+    title?: string;
+    content_hash: string;
+    created_at: string;
+  };
+}
+```
+
+## API Endpoints
+
+### Qdrant Management
+
+- `GET /api/qdrant/stats` - System statistics
+- `GET /api/qdrant/health` - Health check
+- `GET /api/qdrant/shops/:shopId` - Shop Qdrant info
+- `GET /api/qdrant/shops/:shopId/embeddings` - Shop embeddings
+- `GET /api/qdrant/shops/:shopId/errors` - Shop errors
+- `DELETE /api/qdrant/shops/:shopId/errors` - Clear shop errors
+- `PATCH /api/shops/:shopId/qdrant` - Enable/disable Qdrant
+- `POST /api/qdrant/cleanup` - Trigger cleanup
+- `DELETE /api/qdrant/collections/:name` - Delete collection
+- `POST /api/qdrant/shops/:shopId/schedule-deletion` - Schedule deletion
+
+### MCP Analytics
+
+- `GET /api/mcp/stats` - MCP statistics
+- `GET /api/mcp/shops/:shopId/logs` - Shop MCP logs
+- `DELETE /api/mcp/shops/:shopId/logs` - Clear shop logs
+- `GET /api/mcp/analytics` - System analytics
+
+## Workflow Integration
+
+### Automatic Embedding Process
+
+1. **Content Scraping**: WebshopScraper extracts content
+2. **Hash Calculation**: Content hash calculated for change detection
+3. **Qdrant Check**: Verify if shop has Qdrant enabled
+4. **Embedding Generation**: OpenRouter API processes text
+5. **Vector Storage**: Qdrant stores embeddings with metadata
+6. **Database Tracking**: Record embedding status and metadata
+7. **Webhook Notification**: Success/failure notifications sent
+
+### Change Detection
+
+The system tracks content changes using SHA-256 hashes:
+
+```typescript
+function calculateContentHash(content: string): string {
+  return crypto.createHash('sha256').update(content, 'utf8').digest('hex');
+}
+```
+
+Only changed content triggers re-embedding to optimize API usage.
+
+## Frontend Integration
+
+### Shop Detail Page
+
+The shop detail page includes a comprehensive Qdrant section:
+
+- **Toggle Switch**: Enable/disable Qdrant for the shop
+- **Embedding Status**: Real-time counts of completed/pending/failed embeddings
+- **Error Display**: Recent Qdrant errors with clear button
+- **Custom ID Warning**: Alerts when custom_id is required
+
+### Qdrant Management Page
+
+System-wide Qdrant management interface:
+
+- **Health Status**: Service connectivity and status
+- **Statistics**: Collections, vectors, enabled shops counts
+- **Administrative Actions**: Cleanup operations and health checks
+- **Configuration Info**: Environment setup guidance
+
+## Security Considerations
+
+### Custom ID Protection
+
+Once Qdrant is enabled for a shop, the custom_id becomes immutable to prevent collection orphaning:
+
+```typescript
+if (shop.qdrant_enabled && shop.custom_id !== newCustomId) {
+  throw new Error('Cannot modify custom_id when Qdrant is enabled');
+}
+```
+
+### API Key Management
+
+- OpenRouter API key stored securely in environment variables
+- Qdrant API key optional for development environments
+- MCP server uses authenticated endpoints
+
+### Data Isolation
+
+- Collections are isolated by shop custom_id
+- Deletion queues prevent accidental data loss
+- 24-hour deletion delays allow recovery
+
+## Monitoring and Analytics
+
+### Health Monitoring
+
+Continuous health checks monitor:
+
+- Qdrant service connectivity
+- OpenRouter API availability
+- Embedding service status
+- Database connectivity
+
+### MCP Analytics
+
+Comprehensive analytics track:
+
+- Tool usage patterns
+- Response times
+- Success/failure rates
+- Popular search queries
+- Shop usage statistics
+
+### Error Tracking
+
+Centralized error logging captures:
+
+- Embedding failures
+- API connectivity issues
+- Vector storage problems
+- MCP tool errors
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Embedding Failures**
+   - Check OpenRouter API key validity
+   - Verify API rate limits
+   - Review content length restrictions
+
+2. **Qdrant Connection Issues**
+   - Verify QDRANT_API_URL configuration
+   - Check network connectivity
+   - Validate API key if using authentication
+
+3. **Collection Not Found**
+   - Ensure shop has custom_id set
+   - Check collection naming pattern
+   - Verify Qdrant is enabled for shop
+
+4. **MCP Tool Failures**
+   - Check shop_id parameter validity
+   - Verify embeddings exist for shop
+   - Review MCP server logs
+
+### Debugging Commands
+
+```bash
+# Check Qdrant health
+curl http://localhost:6333/collections
+
+# View embedding status
+sqlite3 database.db "SELECT * FROM shop_embeddings WHERE shop_content_id = 'content-id';"
+
+# Check MCP logs
+sqlite3 database.db "SELECT * FROM mcp_logs ORDER BY created_at DESC LIMIT 10;"
+
+# Review error logs
+sqlite3 database.db "SELECT * FROM qdrant_errors ORDER BY created_at DESC;"
+```
+
+## Performance Optimization
+
+### Embedding Efficiency
+
+- Batch processing for multiple content items
+- Content change detection prevents unnecessary re-embedding
+- Configurable delays between API calls
+
+### Vector Search
+
+- Optimized similarity search with configurable limits
+- Cached collection metadata
+- Efficient metadata filtering
+
+### Background Processing
+
+- Automated cleanup scheduling
+- Asynchronous embedding pipeline
+- Health monitoring with alerts
+
+## Migration and Deployment
+
+### Development Setup
+
+1. Install Qdrant locally or use cloud service
+2. Obtain OpenRouter API key
+3. Configure environment variables
+4. Run database migrations
+5. Start MCP server
+
+### Production Deployment
+
+1. Deploy Qdrant cluster with persistence
+2. Configure API keys and authentication
+3. Set up monitoring and alerting
+4. Schedule regular cleanup operations
+5. Monitor embedding costs and usage
+
+### Backup and Recovery
+
+- Regular Qdrant collection backups
+- Database backup includes embedding metadata
+- 24-hour deletion delays enable recovery
+- Collection export/import procedures
+
+## Cost Management
+
+### OpenRouter Usage
+
+- Text preprocessing reduces token count
+- Change detection prevents duplicate embeddings
+- Configurable embedding limits per shop
+
+### Qdrant Storage
+
+- Automatic cleanup of orphaned collections
+- Compression and optimization settings
+- Storage monitoring and alerts
+
+## Future Enhancements
+
+### Planned Features
+
+1. **Multi-language Support**: Language-specific embedding models
+2. **Semantic Clustering**: Content organization by topics
+3. **Advanced Analytics**: Usage patterns and optimization
+4. **Custom Embeddings**: Shop-specific fine-tuning
+5. **Real-time Updates**: Live embedding synchronization
+
+### Integration Opportunities
+
+1. **Search API**: Public search endpoints for shops
+2. **Recommendation Engine**: Product recommendations
+3. **Content Analysis**: Automated content insights
+4. **A/B Testing**: Search relevance optimization
+
+This implementation provides a robust, scalable foundation for semantic search capabilities while maintaining data integrity and providing comprehensive monitoring and management tools.

docs/SETUP_GUIDE.md

@@ -0,0 +1,340 @@
+# Qdrant Vector Search Setup Guide
+
+This guide provides step-by-step instructions to set up and configure the Qdrant vector search integration.
+
+## Prerequisites
+
+- Node.js 18+
+- SQLite database (existing webshop scraper database)
+- OpenRouter API account
+- Qdrant instance (local or cloud)
+
+## Quick Setup
+
+### 1. Install Dependencies
+
+The required dependencies are already included in `package.json`:
+
+```bash
+npm install
+```
+
+**New Dependencies Added:**
+- `@qdrant/js-client-rest` - Qdrant JavaScript client
+- `openai` - OpenRouter API client (OpenAI-compatible)
+- `@modelcontextprotocol/sdk` - MCP server implementation
+
+### 2. Environment Configuration
+
+Create or update your `.env` file with Qdrant and OpenRouter settings:
+
+```bash
+# Existing configuration
+API_KEY=your-existing-api-key
+PORT=3000
+
+# Qdrant Configuration
+QDRANT_API_URL=http://localhost:6333
+QDRANT_API_KEY=                    # Optional for local development
+
+# OpenRouter Configuration (for embeddings)
+OPENROUTER_API_KEY=your-openrouter-api-key
+OPENROUTER_MODEL=openai/text-embedding-3-large
+
+# MCP Configuration
+MCP_ENABLED=true
+MCP_TRANSPORT=stdio                # or 'http'
+MCP_PORT=3001                      # for HTTP transport only
+```
+
+### 3. Qdrant Setup
+
+#### Option A: Local Docker Instance
+
+```bash
+# Pull and run Qdrant
+docker run -p 6333:6333 qdrant/qdrant
+
+# Or with persistence
+docker run -p 6333:6333 -v $(pwd)/qdrant_storage:/qdrant/storage qdrant/qdrant
+```
+
+#### Option B: Qdrant Cloud
+
+1. Sign up at [Qdrant Cloud](https://cloud.qdrant.io/)
+2. Create a cluster
+3. Get your API URL and key
+4. Update environment variables:
+
+```bash
+QDRANT_API_URL=https://your-cluster.qdrant.io
+QDRANT_API_KEY=your-api-key
+```
+
+### 4. OpenRouter API Setup
+
+1. Sign up at [OpenRouter](https://openrouter.ai/)
+2. Create an API key
+3. Add to environment variables:
+
+```bash
+OPENROUTER_API_KEY=sk-or-v1-your-api-key
+```
+
+**Note:** The system uses the `openai/text-embedding-3-large` model which provides 3072-dimensional embeddings optimized for semantic search.
+
+### 5. Database Migrations
+
+The database schema extensions are automatically applied when the application starts. No manual migration is required.
+
+**New Tables Created:**
+- `shop_embeddings` - Embedding tracking
+- `qdrant_deletion_queue` - Deletion management
+- `qdrant_errors` - Error logging
+- `mcp_logs` - MCP analytics
+
+### 6. Start the Application
+
+```bash
+# Development mode
+npm run dev
+
+# Production mode
+npm start
+```
+
+**Services Started:**
+- Main API server (port 3000)
+- MCP server (stdio transport by default)
+- Background scheduler for cleanup and monitoring
+
+## Configuration Options
+
+### Qdrant Settings
+
+```typescript
+// Default configuration in src/config/index.ts
+qdrant: {
+  apiUrl: process.env.QDRANT_API_URL || 'http://localhost:6333',
+  apiKey: process.env.QDRANT_API_KEY,
+  vectorSize: 3072,
+  distance: 'cosine'
+}
+```
+
+### Embedding Settings
+
+```typescript
+// OpenRouter configuration
+openRouter: {
+  apiKey: process.env.OPENROUTER_API_KEY!,
+  model: process.env.OPENROUTER_MODEL || 'openai/text-embedding-3-large',
+  maxTokens: 8192,
+  batchSize: 10
+}
+```
+
+### MCP Settings
+
+```typescript
+// MCP server configuration
+mcp: {
+  enabled: process.env.MCP_ENABLED === 'true',
+  transport: process.env.MCP_TRANSPORT as 'stdio' | 'http',
+  port: parseInt(process.env.MCP_PORT || '3001'),
+  maxResults: 50
+}
+```
+
+## Verification Steps
+
+### 1. Check Service Health
+
+```bash
+# Check main application
+curl http://localhost:3000/health
+
+# Check Qdrant
+curl http://localhost:6333/collections
+```
+
+### 2. Verify API Endpoints
+
+```bash
+# Check Qdrant health via API
+curl -H "X-API-Key: your-api-key" http://localhost:3000/api/qdrant/health
+
+# Get Qdrant statistics
+curl -H "X-API-Key: your-api-key" http://localhost:3000/api/qdrant/stats
+```
+
+### 3. Test MCP Tools
+
+The MCP server provides 4 semantic search tools:
+- `search_shipping_info`
+- `search_contact_info`
+- `search_terms_info`
+- `search_faq_info`
+
+MCP tools are accessed via the stdio transport by LLM applications.
+
+## Usage Workflow
+
+### 1. Enable Qdrant for a Shop
+
+1. Navigate to the shop detail page in the web interface
+2. Ensure the shop has a custom_id set
+3. Toggle the Qdrant switch to "Enabled"
+4. The system will automatically start embedding content during the next scrape
+
+### 2. Monitor Embedding Progress
+
+- View embedding status on the shop detail page
+- Check the Qdrant management page for system-wide statistics
+- Review error logs if embedding fails
+
+### 3. Use Semantic Search
+
+Once embeddings are created, LLM applications can use MCP tools:
+
+```javascript
+// Example: Search for shipping information
+const results = await search_shipping_info("shop123", "international delivery", 5);
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **"Qdrant service unavailable"**
+   - Verify Qdrant is running: `curl http://localhost:6333/collections`
+   - Check QDRANT_API_URL in environment variables
+   - Ensure network connectivity
+
+2. **"OpenRouter API key invalid"**
+   - Verify API key format: `sk-or-v1-...`
+   - Check account balance and rate limits
+   - Test API key: `curl -H "Authorization: Bearer $OPENROUTER_API_KEY" https://openrouter.ai/api/v1/models`
+
+3. **"Custom ID required"**
+   - Set a custom_id for the shop before enabling Qdrant
+   - Custom ID cannot be changed once Qdrant is enabled
+   - Use a descriptive, unique identifier
+
+4. **"Embedding failed"**
+   - Check content length (max ~8K tokens)
+   - Verify OpenRouter API rate limits
+   - Review error logs in the UI or database
+
+### Debug Commands
+
+```bash
+# Check application logs
+docker logs webshop-scraper
+
+# Query embedding status
+sqlite3 database.db "SELECT * FROM shop_embeddings ORDER BY created_at DESC LIMIT 10;"
+
+# Check for errors
+sqlite3 database.db "SELECT * FROM qdrant_errors ORDER BY created_at DESC LIMIT 10;"
+
+# View MCP usage
+sqlite3 database.db "SELECT * FROM mcp_logs ORDER BY created_at DESC LIMIT 10;"
+```
+
+### Log Locations
+
+- Application logs: Console output
+- Database errors: `qdrant_errors` table
+- MCP analytics: `mcp_logs` table
+- Webhook deliveries: `webhook_deliveries` table
+
+## Performance Tuning
+
+### Embedding Optimization
+
+```typescript
+// Adjust in src/services/EmbeddingService.ts
+const config = {
+  batchSize: 5,           // Process N items at once
+  delayMs: 100,          // Delay between API calls
+  maxRetries: 3,         // Retry failed embeddings
+  maxTokens: 8192        // Content truncation limit
+};
+```
+
+### Qdrant Optimization
+
+```typescript
+// Collection settings in src/services/QdrantService.ts
+const collectionConfig = {
+  vectors: {
+    size: 3072,
+    distance: 'Cosine'
+  },
+  optimizers_config: {
+    default_segment_number: 2
+  },
+  replication_factor: 1
+};
+```
+
+### Background Tasks
+
+```typescript
+// Scheduler intervals in src/services/SchedulerService.ts
+const intervals = {
+  cleanup: 24 * 60 * 60 * 1000,      // 24 hours
+  healthCheck: 5 * 60 * 1000,        // 5 minutes
+  deletionQueue: 60 * 60 * 1000,     // 1 hour
+  logCleanup: 7 * 24 * 60 * 60 * 1000 // 7 days
+};
+```
+
+## Security Considerations
+
+### API Keys
+
+- Store API keys in environment variables only
+- Never commit API keys to version control
+- Use different keys for development and production
+- Monitor API usage and costs
+
+### Data Protection
+
+- Qdrant collections are isolated by shop custom_id
+- 24-hour deletion delays prevent accidental data loss
+- Custom_id becomes immutable when Qdrant is enabled
+- Error logs contain no sensitive information
+
+### Network Security
+
+- Use HTTPS in production
+- Configure Qdrant authentication for production
+- Limit API access by IP if required
+- Monitor for unusual usage patterns
+
+## Cost Management
+
+### OpenRouter Costs
+
+- Text embedding costs: ~$0.00002 per 1K tokens
+- Average content size: 2-4K tokens per page
+- Typical cost: $0.04-$0.16 per 1K pages
+- Change detection prevents re-embedding unchanged content
+
+### Qdrant Storage
+
+- Local instance: Free (your storage costs)
+- Qdrant Cloud: Varies by cluster size and usage
+- Regular cleanup removes orphaned collections
+- Monitor storage usage in production
+
+### Monitoring
+
+- Track embedding counts and costs
+- Set up alerts for unusual API usage
+- Monitor Qdrant storage growth
+- Regular cleanup schedules
+
+This setup guide provides everything needed to get the Qdrant vector search integration running successfully. The system is designed to be robust and production-ready while maintaining ease of setup and management.