|
|
@@ -0,0 +1,423 @@
|
|
|
+# Qdrant Vector Search Implementation Guide
|
|
|
+
|
|
|
+This document provides comprehensive information about the Qdrant vector search implementation in the webshop scraper system.
|
|
|
+
|
|
|
+## Overview
|
|
|
+
|
|
|
+The Qdrant integration enables semantic search capabilities for scraped webshop content using text embeddings. The system automatically processes scraped content through OpenRouter's text-embedding-3-large model and stores vectors in Qdrant for efficient similarity search via MCP (Model Context Protocol) tools.
|
|
|
+
|
|
|
+## Architecture
|
|
|
+
|
|
|
+### Core Components
|
|
|
+
|
|
|
+1. **QdrantService** (`src/services/QdrantService.ts`)
|
|
|
+ - Manages Qdrant database connections and operations
|
|
|
+ - Handles collection creation and management
|
|
|
+ - Performs vector search operations
|
|
|
+
|
|
|
+2. **EmbeddingService** (`src/services/EmbeddingService.ts`)
|
|
|
+ - Integrates with OpenRouter API for text embeddings
|
|
|
+ - Uses text-embedding-3-large model (3072 dimensions)
|
|
|
+ - Handles text preprocessing and batching
|
|
|
+
|
|
|
+3. **QdrantEmbeddingWorkflow** (`src/services/QdrantEmbeddingWorkflow.ts`)
|
|
|
+ - Orchestrates the embedding pipeline
|
|
|
+ - Integrates with scraping workflow
|
|
|
+ - Manages error handling and retries
|
|
|
+
|
|
|
+4. **QdrantCleanupService** (`src/services/QdrantCleanupService.ts`)
|
|
|
+ - Handles automated cleanup operations
|
|
|
+ - Manages 24-hour deletion delays
|
|
|
+ - Processes deletion queues
|
|
|
+
|
|
|
+5. **MCP Server** (`src/mcp/McpServer.ts`)
|
|
|
+ - Provides semantic search tools for LLMs
|
|
|
+ - Implements 4 category-specific search tools
|
|
|
+ - Handles analytics and logging
|
|
|
+
|
|
|
+6. **SchedulerService** (`src/services/SchedulerService.ts`)
|
|
|
+ - Background task management
|
|
|
+ - Health monitoring
|
|
|
+ - Automated maintenance
|
|
|
+
|
|
|
+## Configuration
|
|
|
+
|
|
|
+### Environment Variables
|
|
|
+
|
|
|
+```bash
|
|
|
+# Qdrant Configuration
|
|
|
+QDRANT_API_URL=http://localhost:6333
|
|
|
+QDRANT_API_KEY=your-qdrant-api-key # Optional
|
|
|
+
|
|
|
+# OpenRouter Configuration (for embeddings)
|
|
|
+OPENROUTER_API_KEY=your-openrouter-api-key
|
|
|
+OPENROUTER_MODEL=openai/text-embedding-3-large
|
|
|
+
|
|
|
+# MCP Configuration
|
|
|
+MCP_ENABLED=true
|
|
|
+MCP_TRANSPORT=stdio # or http
|
|
|
+MCP_PORT=3001 # for HTTP transport
|
|
|
+```
|
|
|
+
|
|
|
+### Database Schema Extensions
|
|
|
+
|
|
|
+The implementation extends the existing SQLite database with new tables:
|
|
|
+
|
|
|
+```sql
|
|
|
+-- Enable Qdrant per shop
|
|
|
+ALTER TABLE shops ADD COLUMN qdrant_enabled BOOLEAN DEFAULT FALSE;
|
|
|
+
|
|
|
+-- Track embeddings
|
|
|
+CREATE TABLE shop_embeddings (
|
|
|
+ id TEXT PRIMARY KEY,
|
|
|
+ shop_content_id TEXT NOT NULL,
|
|
|
+ collection_name TEXT NOT NULL,
|
|
|
+ point_id TEXT NOT NULL,
|
|
|
+ embedding_status TEXT NOT NULL CHECK (embedding_status IN ('pending', 'completed', 'failed')),
|
|
|
+ created_at TEXT NOT NULL,
|
|
|
+ updated_at TEXT NOT NULL,
|
|
|
+ FOREIGN KEY (shop_content_id) REFERENCES shop_contents (id) ON DELETE CASCADE
|
|
|
+);
|
|
|
+
|
|
|
+-- Deletion queue with 24h delays
|
|
|
+CREATE TABLE qdrant_deletion_queue (
|
|
|
+ id TEXT PRIMARY KEY,
|
|
|
+ shop_id TEXT NOT NULL,
|
|
|
+ custom_id TEXT NOT NULL,
|
|
|
+ status TEXT NOT NULL DEFAULT 'pending',
|
|
|
+ scheduled_deletion_at TEXT NOT NULL,
|
|
|
+ created_at TEXT NOT NULL,
|
|
|
+ updated_at TEXT NOT NULL,
|
|
|
+ error_message TEXT,
|
|
|
+ FOREIGN KEY (shop_id) REFERENCES shops (id) ON DELETE CASCADE
|
|
|
+);
|
|
|
+
|
|
|
+-- Error tracking
|
|
|
+CREATE TABLE qdrant_errors (
|
|
|
+ id TEXT PRIMARY KEY,
|
|
|
+ shop_id TEXT NOT NULL,
|
|
|
+ error_type TEXT NOT NULL,
|
|
|
+ error_message TEXT NOT NULL,
|
|
|
+ metadata TEXT,
|
|
|
+ created_at TEXT NOT NULL,
|
|
|
+ FOREIGN KEY (shop_id) REFERENCES shops (id) ON DELETE CASCADE
|
|
|
+);
|
|
|
+
|
|
|
+-- MCP analytics
|
|
|
+CREATE TABLE mcp_logs (
|
|
|
+ id TEXT PRIMARY KEY,
|
|
|
+ shop_id TEXT NOT NULL,
|
|
|
+ tool_name TEXT NOT NULL,
|
|
|
+ query TEXT NOT NULL,
|
|
|
+ response_time_ms INTEGER NOT NULL,
|
|
|
+ results_count INTEGER NOT NULL,
|
|
|
+ success BOOLEAN NOT NULL,
|
|
|
+ error_message TEXT,
|
|
|
+ created_at TEXT NOT NULL,
|
|
|
+ FOREIGN KEY (shop_id) REFERENCES shops (id) ON DELETE CASCADE
|
|
|
+);
|
|
|
+```
|
|
|
+
|
|
|
+## Collection Naming Convention
|
|
|
+
|
|
|
+Collections follow a specific pattern for organization and isolation:
|
|
|
+
|
|
|
+```
|
|
|
+{shop_custom_id}-{category}_{iterator}
|
|
|
+```
|
|
|
+
|
|
|
+**Examples:**
|
|
|
+- `shop123-shipping_1` - First shipping information collection
|
|
|
+- `shop123-contacts_2` - Second contacts collection
|
|
|
+- `shop456-faq_1` - First FAQ collection
|
|
|
+
|
|
|
+**Categories:**
|
|
|
+- `shipping` - Shipping and delivery information
|
|
|
+- `contacts` - Contact information and support
|
|
|
+- `terms` - Terms of service and conditions
|
|
|
+- `faq` - Frequently asked questions
|
|
|
+
|
|
|
+## MCP Tools
|
|
|
+
|
|
|
+The system provides 4 semantic search tools accessible to LLMs:
|
|
|
+
|
|
|
+### Tool Signatures
|
|
|
+
|
|
|
+```typescript
|
|
|
+search_shipping_info(shop_id: string, query: string, limit?: number): Promise<SearchResult[]>
|
|
|
+search_contact_info(shop_id: string, query: string, limit?: number): Promise<SearchResult[]>
|
|
|
+search_terms_info(shop_id: string, query: string, limit?: number): Promise<SearchResult[]>
|
|
|
+search_faq_info(shop_id: string, query: string, limit?: number): Promise<SearchResult[]>
|
|
|
+```
|
|
|
+
|
|
|
+### Usage Examples
|
|
|
+
|
|
|
+```javascript
|
|
|
+// Search for shipping policies
|
|
|
+const results = await search_shipping_info("shop123", "international delivery times", 5);
|
|
|
+
|
|
|
+// Find contact information
|
|
|
+const contacts = await search_contact_info("shop123", "customer service phone", 3);
|
|
|
+
|
|
|
+// Look up return policies
|
|
|
+const terms = await search_terms_info("shop123", "return policy within 30 days", 5);
|
|
|
+
|
|
|
+// Find specific FAQ answers
|
|
|
+const faqs = await search_faq_info("shop123", "how to track my order", 10);
|
|
|
+```
|
|
|
+
|
|
|
+### Response Format
|
|
|
+
|
|
|
+```typescript
|
|
|
+interface SearchResult {
|
|
|
+ id: string;
|
|
|
+ score: number; // Similarity score (0-1)
|
|
|
+ payload: {
|
|
|
+ shop_id: string;
|
|
|
+ content_id: string;
|
|
|
+ content_type: string;
|
|
|
+ url: string;
|
|
|
+ title?: string;
|
|
|
+ content_hash: string;
|
|
|
+ created_at: string;
|
|
|
+ };
|
|
|
+}
|
|
|
+```
|
|
|
+
|
|
|
+## API Endpoints
|
|
|
+
|
|
|
+### Qdrant Management
|
|
|
+
|
|
|
+- `GET /api/qdrant/stats` - System statistics
|
|
|
+- `GET /api/qdrant/health` - Health check
|
|
|
+- `GET /api/qdrant/shops/:shopId` - Shop Qdrant info
|
|
|
+- `GET /api/qdrant/shops/:shopId/embeddings` - Shop embeddings
|
|
|
+- `GET /api/qdrant/shops/:shopId/errors` - Shop errors
|
|
|
+- `DELETE /api/qdrant/shops/:shopId/errors` - Clear shop errors
|
|
|
+- `PATCH /api/shops/:shopId/qdrant` - Enable/disable Qdrant
|
|
|
+- `POST /api/qdrant/cleanup` - Trigger cleanup
|
|
|
+- `DELETE /api/qdrant/collections/:name` - Delete collection
|
|
|
+- `POST /api/qdrant/shops/:shopId/schedule-deletion` - Schedule deletion
|
|
|
+
|
|
|
+### MCP Analytics
|
|
|
+
|
|
|
+- `GET /api/mcp/stats` - MCP statistics
|
|
|
+- `GET /api/mcp/shops/:shopId/logs` - Shop MCP logs
|
|
|
+- `DELETE /api/mcp/shops/:shopId/logs` - Clear shop logs
|
|
|
+- `GET /api/mcp/analytics` - System analytics
|
|
|
+
|
|
|
+## Workflow Integration
|
|
|
+
|
|
|
+### Automatic Embedding Process
|
|
|
+
|
|
|
+1. **Content Scraping**: WebshopScraper extracts content
|
|
|
+2. **Hash Calculation**: Content hash calculated for change detection
|
|
|
+3. **Qdrant Check**: Verify if shop has Qdrant enabled
|
|
|
+4. **Embedding Generation**: OpenRouter API processes text
|
|
|
+5. **Vector Storage**: Qdrant stores embeddings with metadata
|
|
|
+6. **Database Tracking**: Record embedding status and metadata
|
|
|
+7. **Webhook Notification**: Success/failure notifications sent
|
|
|
+
|
|
|
+### Change Detection
|
|
|
+
|
|
|
+The system tracks content changes using SHA-256 hashes:
|
|
|
+
|
|
|
+```typescript
|
|
|
+function calculateContentHash(content: string): string {
|
|
|
+ return crypto.createHash('sha256').update(content, 'utf8').digest('hex');
|
|
|
+}
|
|
|
+```
|
|
|
+
|
|
|
+Only changed content triggers re-embedding to optimize API usage.
|
|
|
+
|
|
|
+## Frontend Integration
|
|
|
+
|
|
|
+### Shop Detail Page
|
|
|
+
|
|
|
+The shop detail page includes a comprehensive Qdrant section:
|
|
|
+
|
|
|
+- **Toggle Switch**: Enable/disable Qdrant for the shop
|
|
|
+- **Embedding Status**: Real-time counts of completed/pending/failed embeddings
|
|
|
+- **Error Display**: Recent Qdrant errors with clear button
|
|
|
+- **Custom ID Warning**: Alerts when custom_id is required
|
|
|
+
|
|
|
+### Qdrant Management Page
|
|
|
+
|
|
|
+System-wide Qdrant management interface:
|
|
|
+
|
|
|
+- **Health Status**: Service connectivity and status
|
|
|
+- **Statistics**: Collections, vectors, enabled shops counts
|
|
|
+- **Administrative Actions**: Cleanup operations and health checks
|
|
|
+- **Configuration Info**: Environment setup guidance
|
|
|
+
|
|
|
+## Security Considerations
|
|
|
+
|
|
|
+### Custom ID Protection
|
|
|
+
|
|
|
+Once Qdrant is enabled for a shop, the custom_id becomes immutable to prevent collection orphaning:
|
|
|
+
|
|
|
+```typescript
|
|
|
+if (shop.qdrant_enabled && shop.custom_id !== newCustomId) {
|
|
|
+ throw new Error('Cannot modify custom_id when Qdrant is enabled');
|
|
|
+}
|
|
|
+```
|
|
|
+
|
|
|
+### API Key Management
|
|
|
+
|
|
|
+- OpenRouter API key stored securely in environment variables
|
|
|
+- Qdrant API key optional for development environments
|
|
|
+- MCP server uses authenticated endpoints
|
|
|
+
|
|
|
+### Data Isolation
|
|
|
+
|
|
|
+- Collections are isolated by shop custom_id
|
|
|
+- Deletion queues prevent accidental data loss
|
|
|
+- 24-hour deletion delays allow recovery
|
|
|
+
|
|
|
+## Monitoring and Analytics
|
|
|
+
|
|
|
+### Health Monitoring
|
|
|
+
|
|
|
+Continuous health checks monitor:
|
|
|
+
|
|
|
+- Qdrant service connectivity
|
|
|
+- OpenRouter API availability
|
|
|
+- Embedding service status
|
|
|
+- Database connectivity
|
|
|
+
|
|
|
+### MCP Analytics
|
|
|
+
|
|
|
+Comprehensive analytics track:
|
|
|
+
|
|
|
+- Tool usage patterns
|
|
|
+- Response times
|
|
|
+- Success/failure rates
|
|
|
+- Popular search queries
|
|
|
+- Shop usage statistics
|
|
|
+
|
|
|
+### Error Tracking
|
|
|
+
|
|
|
+Centralized error logging captures:
|
|
|
+
|
|
|
+- Embedding failures
|
|
|
+- API connectivity issues
|
|
|
+- Vector storage problems
|
|
|
+- MCP tool errors
|
|
|
+
|
|
|
+## Troubleshooting
|
|
|
+
|
|
|
+### Common Issues
|
|
|
+
|
|
|
+1. **Embedding Failures**
|
|
|
+ - Check OpenRouter API key validity
|
|
|
+ - Verify API rate limits
|
|
|
+ - Review content length restrictions
|
|
|
+
|
|
|
+2. **Qdrant Connection Issues**
|
|
|
+ - Verify QDRANT_API_URL configuration
|
|
|
+ - Check network connectivity
|
|
|
+ - Validate API key if using authentication
|
|
|
+
|
|
|
+3. **Collection Not Found**
|
|
|
+ - Ensure shop has custom_id set
|
|
|
+ - Check collection naming pattern
|
|
|
+ - Verify Qdrant is enabled for shop
|
|
|
+
|
|
|
+4. **MCP Tool Failures**
|
|
|
+ - Check shop_id parameter validity
|
|
|
+ - Verify embeddings exist for shop
|
|
|
+ - Review MCP server logs
|
|
|
+
|
|
|
+### Debugging Commands
|
|
|
+
|
|
|
+```bash
|
|
|
+# Check Qdrant health
|
|
|
+curl http://localhost:6333/collections
|
|
|
+
|
|
|
+# View embedding status
|
|
|
+sqlite3 database.db "SELECT * FROM shop_embeddings WHERE shop_content_id = 'content-id';"
|
|
|
+
|
|
|
+# Check MCP logs
|
|
|
+sqlite3 database.db "SELECT * FROM mcp_logs ORDER BY created_at DESC LIMIT 10;"
|
|
|
+
|
|
|
+# Review error logs
|
|
|
+sqlite3 database.db "SELECT * FROM qdrant_errors ORDER BY created_at DESC;"
|
|
|
+```
|
|
|
+
|
|
|
+## Performance Optimization
|
|
|
+
|
|
|
+### Embedding Efficiency
|
|
|
+
|
|
|
+- Batch processing for multiple content items
|
|
|
+- Content change detection prevents unnecessary re-embedding
|
|
|
+- Configurable delays between API calls
|
|
|
+
|
|
|
+### Vector Search
|
|
|
+
|
|
|
+- Optimized similarity search with configurable limits
|
|
|
+- Cached collection metadata
|
|
|
+- Efficient metadata filtering
|
|
|
+
|
|
|
+### Background Processing
|
|
|
+
|
|
|
+- Automated cleanup scheduling
|
|
|
+- Asynchronous embedding pipeline
|
|
|
+- Health monitoring with alerts
|
|
|
+
|
|
|
+## Migration and Deployment
|
|
|
+
|
|
|
+### Development Setup
|
|
|
+
|
|
|
+1. Install Qdrant locally or use cloud service
|
|
|
+2. Obtain OpenRouter API key
|
|
|
+3. Configure environment variables
|
|
|
+4. Run database migrations
|
|
|
+5. Start MCP server
|
|
|
+
|
|
|
+### Production Deployment
|
|
|
+
|
|
|
+1. Deploy Qdrant cluster with persistence
|
|
|
+2. Configure API keys and authentication
|
|
|
+3. Set up monitoring and alerting
|
|
|
+4. Schedule regular cleanup operations
|
|
|
+5. Monitor embedding costs and usage
|
|
|
+
|
|
|
+### Backup and Recovery
|
|
|
+
|
|
|
+- Regular Qdrant collection backups
|
|
|
+- Database backup includes embedding metadata
|
|
|
+- 24-hour deletion delays enable recovery
|
|
|
+- Collection export/import procedures
|
|
|
+
|
|
|
+## Cost Management
|
|
|
+
|
|
|
+### OpenRouter Usage
|
|
|
+
|
|
|
+- Text preprocessing reduces token count
|
|
|
+- Change detection prevents duplicate embeddings
|
|
|
+- Configurable embedding limits per shop
|
|
|
+
|
|
|
+### Qdrant Storage
|
|
|
+
|
|
|
+- Automatic cleanup of orphaned collections
|
|
|
+- Compression and optimization settings
|
|
|
+- Storage monitoring and alerts
|
|
|
+
|
|
|
+## Future Enhancements
|
|
|
+
|
|
|
+### Planned Features
|
|
|
+
|
|
|
+1. **Multi-language Support**: Language-specific embedding models
|
|
|
+2. **Semantic Clustering**: Content organization by topics
|
|
|
+3. **Advanced Analytics**: Usage patterns and optimization
|
|
|
+4. **Custom Embeddings**: Shop-specific fine-tuning
|
|
|
+5. **Real-time Updates**: Live embedding synchronization
|
|
|
+
|
|
|
+### Integration Opportunities
|
|
|
+
|
|
|
+1. **Search API**: Public search endpoints for shops
|
|
|
+2. **Recommendation Engine**: Product recommendations
|
|
|
+3. **Content Analysis**: Automated content insights
|
|
|
+4. **A/B Testing**: Search relevance optimization
|
|
|
+
|
|
|
+This implementation provides a robust, scalable foundation for semantic search capabilities while maintaining data integrity and providing comprehensive monitoring and management tools.
|
Fszontagh