refactor: replace Qdrant iterator naming with URL hash-based collections

Replace the iterator-based collection naming system with a URL hash-based approach. This ensures that:
- Same URL always maps to the same collection for proper data syncing
- Collections are automatically reused when URLs are rescraped
- When URL arrays change, new collections are created automatically
- No need to track iterator numbers

Changes:
- Update QdrantService.generateCollectionName() to use MD5 URL hash instead of iterator
- Enhance parseCollectionName() to support both old (iterator) and new (hash) formats
- Simplify QdrantEmbeddingWorkflow to use URL-based collection assignment
- Deprecate getNextIterator() method while maintaining backwards compatibility

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

WEB_UI_README.md

@@ -1,195 +0,0 @@
-# Webshop Scraper - Web UI
-
-## 🎉 Successfully Implemented!
-
-A modern, real-time web interface has been integrated into the Webshop Scraper API, allowing users to manage and monitor their webshop scraping operations through an intuitive dashboard.
-
-## ✨ Features
-
-### 🔐 Authentication
-- **API Key Login**: Secure login using your existing API key
-- **Session Management**: Automatic session persistence with local storage
-- **Protected Routes**: All endpoints secured with Bearer token authentication
-
-### 📊 Real-time Dashboard
-- **Live Statistics**: Queue status, job counts, and system health metrics
-- **Auto-refresh**: Statistics update every 5 seconds automatically
-- **Quick Actions**: Fast access to common operations
-
-### 🏪 Shop Management
-- **Shop Overview**: Grid view of all monitored webshops
-- **Add New Shops**: Simple form to add new shops for scraping
-- **Shop Details**: Comprehensive view with analytics, content overview, and history
-- **Delete Operations**: Safe shop removal with confirmation
-
-### 🔄 Job Monitoring
-- **Queue Visualization**: Real-time view of pending, processing, and completed jobs
-- **Job History**: Complete history of all scraping operations
-- **Status Tracking**: Visual indicators for job progress and results
-- **Auto-updates**: Job status refreshes every 3 seconds
-
-### 📱 Modern UI/UX
-- **Responsive Design**: Works on desktop, tablet, and mobile devices
-- **Dark/Light Mode**: Theme toggle with system preference detection
-- **Toast Notifications**: Real-time feedback for all user actions
-- **Loading States**: Visual feedback during async operations
-
-### 🚀 Additional Features
-- **Content Browsing**: View scraped content by category and date
-- **Schedule Management**: Control automated scraping schedules
-- **Webhook Configuration**: Manage webhook endpoints and delivery logs
-- **Custom URL Management**: Add specific URLs per shop category
-
-## 🚀 Getting Started
-
-### 1. Build the Application
-```bash
-# Install dependencies and build everything
-npm run build
-```
-
-### 2. Start the Server
-```bash
-# Set your API key and start the server
-API_KEY=your-secret-api-key npm run start
-```
-
-### 3. Access the Web Interface
-Open your browser and navigate to:
-```
-http://localhost:3000
-```
-
-### 4. Login
-- Enter your API key (the same one used for the REST API)
-- The interface will remember your login session
-
-## 🔧 Development
-
-### Frontend Development
-```bash
-# Start the web UI in development mode (with hot reload)
-npm run dev:web
-
-# Build only the web UI
-npm run build:web
-```
-
-### Backend Development
-```bash
-# Watch TypeScript changes
-npm run watch
-
-# Start backend in development mode
-npm run dev
-```
-
-## 📁 Project Structure
-
-```
-├── src/                    # Backend TypeScript source
-│   ├── api/               # Express server and routes
-│   ├── database/          # SQLite database layer
-│   ├── scraper/          # Web scraping logic
-│   └── ...
-├── web/                   # Frontend source code
-│   ├── src/              # TypeScript/Vite frontend
-│   │   ├── components/   # UI components
-│   │   ├── services/     # API and state management
-│   │   ├── utils/        # Helper functions
-│   │   └── styles/       # Tailwind CSS styles
-│   └── package.json      # Frontend dependencies
-├── dist/                  # Built files
-│   ├── web/              # Built web UI (served by Express)
-│   └── ...               # Compiled backend
-└── package.json          # Main package file
-```
-
-## 🎯 Key Technologies
-
-### Backend Integration
-- **Express Static Serving**: Web UI served directly by the API server
-- **SPA Routing**: Client-side routing with fallback to index.html
-- **CORS-free**: No cross-origin issues since UI and API share the same origin
-
-### Frontend Stack
-- **Vite**: Fast build tool and development server
-- **TypeScript**: Type-safe frontend development
-- **Tailwind CSS**: Utility-first CSS framework
-- **Vanilla JS**: No heavy frameworks, fast and lightweight
-
-### Real-time Features
-- **Polling-based Updates**: Regular API calls for live data
-- **Toast Notifications**: Immediate user feedback
-- **Loading States**: Better UX during async operations
-
-## 🔒 Security
-
-- **API Key Authentication**: Uses the same security model as the REST API
-- **Local Storage**: API key stored securely in browser
-- **HTTPS Ready**: Works with SSL/TLS in production
-- **No Credentials in URL**: API key never exposed in browser history
-
-## 🌐 Production Deployment
-
-The web UI is automatically built and served by the main application. For production:
-
-1. Set a strong API key:
-   ```bash
-   API_KEY=your-production-api-key
-   ```
-
-2. Build the application:
-   ```bash
-   npm run build
-   ```
-
-3. Start the server:
-   ```bash
-   npm run start
-   ```
-
-4. Access the interface at your server's URL:
-   ```
-   https://your-domain.com
-   ```
-
-## 📈 Performance
-
-- **Bundle Size**: ~72KB gzipped JavaScript, ~32KB CSS
-- **Load Time**: Fast initial load with efficient caching
-- **Memory Usage**: Lightweight vanilla JS implementation
-- **Network**: Efficient API polling with minimal data transfer
-
-## 🐛 Troubleshooting
-
-### Web UI not loading
-- Ensure the backend is built: `npm run build`
-- Check the API key is set correctly
-- Verify the server is running on the expected port
-
-### Authentication failing
-- Confirm the API key matches the server configuration
-- Check browser network tab for 401 errors
-- Clear browser cache and localStorage if needed
-
-### Real-time updates not working
-- Check browser network tab for failed API requests
-- Verify the API endpoints are accessible
-- Ensure the API key has proper permissions
-
-## ✅ Implementation Complete
-
-The modern web UI has been successfully integrated with the existing Webshop Scraper API, providing:
-
-✅ **Authentication system** with API key login
-✅ **Real-time dashboard** with live statistics
-✅ **Complete shop management** (CRUD operations)
-✅ **Job queue monitoring** with auto-refresh
-✅ **Content browsing** interface
-✅ **Schedule and webhook management**
-✅ **Responsive design** for all devices
-✅ **Dark/light theme** support
-✅ **Production-ready** build system
-
-The web interface provides a comprehensive, user-friendly way to manage all aspects of the webshop scraping system without needing to use curl commands or API clients.

src/services/QdrantEmbeddingWorkflow.ts

@@ -90,47 +90,30 @@ export class QdrantEmbeddingWorkflow {
 
       logger.info(`Content ${contentId} split into ${chunks.length} chunks (avg: ${chunkStats.avgWordCount} words/chunk)`);
 
-      // Check if this content already has a collection assigned
+      // Generate collection name based on URL hash
+      // This ensures the same URL always maps to the same collection for syncing
+      const collectionName = this.qdrantService.generateCollectionName(shop.custom_id, contentType, url);
+      logger.debug(`Using collection ${collectionName} for content ${contentId} (URL: ${url})`);
+
+      // Check if this content already has embeddings
       const existingEmbedding = existingEmbeddings.find(emb => emb.collection_name);
-      let collectionName: string;
 
-      if (existingEmbedding && existingEmbedding.collection_name) {
+      if (existingEmbedding) {
         // Check if the collection still exists in Qdrant
         const collectionExists = await this.qdrantService.collectionExists(existingEmbedding.collection_name);
 
         if (collectionExists) {
-          // Reuse existing collection for this URL
-          collectionName = existingEmbedding.collection_name;
-          logger.debug(`Reusing existing collection ${collectionName} for content ${contentId}`);
-
           // Delete all existing points in this collection for this content_id
-          // This ensures we don't have duplicate/stale chunks
+          // This ensures we sync fresh data without duplicates
           await this.qdrantService.deletePointsByContentId(collectionName, contentId);
+          logger.debug(`Cleared existing points for content ${contentId} in collection ${collectionName}`);
+        }
 
-          // Delete all existing embedding records for this content
-          // We'll create new ones for each chunk
-          for (const emb of existingEmbeddings) {
-            this.database.deleteShopEmbedding(emb.id);
-          }
-        } else {
-          // Collection was deleted externally, clean up database and get new iterator
-          logger.info(`Collection ${existingEmbedding.collection_name} no longer exists in Qdrant, assigning new collection`);
-
-          // Delete all existing embedding records for this content
-          for (const emb of existingEmbeddings) {
-            this.database.deleteShopEmbedding(emb.id);
-          }
-
-          // Get next iterator for this category (one collection per URL)
-          const iterator = await this.qdrantService.getNextIterator(shop.custom_id, contentType);
-          collectionName = this.qdrantService.generateCollectionName(shop.custom_id, contentType, iterator);
-          logger.info(`Assigned new collection ${collectionName} for content ${contentId}`);
+        // Delete all existing embedding records for this content
+        // We'll create new ones for each chunk
+        for (const emb of existingEmbeddings) {
+          this.database.deleteShopEmbedding(emb.id);
         }
-      } else {
-        // Get next iterator for this category (one collection per URL)
-        const iterator = await this.qdrantService.getNextIterator(shop.custom_id, contentType);
-        collectionName = this.qdrantService.generateCollectionName(shop.custom_id, contentType, iterator);
-        logger.info(`Assigned new collection ${collectionName} for content ${contentId}`);
       }
 
       // Ensure collection exists

src/services/QdrantService.ts

@@ -1,4 +1,5 @@
 import { QdrantClient } from '@qdrant/js-client-rest';
+import { createHash } from 'crypto';
 import { logger } from '../utils/logger';
 import type { AppConfig } from '../config';
 
@@ -54,23 +55,43 @@ export class QdrantService {
   }
 
   /**
-   * Generate collection name using the format: {shop_custom_id}-{category_name}_{iterator}
+   * Generate a short hash from a URL for use in collection names
    */
-  generateCollectionName(shopCustomId: string, category: string, iterator: number): string {
-    return `${shopCustomId}-${category}_${iterator}`;
+  private generateUrlHash(url: string): string {
+    return createHash('md5').update(url).digest('hex').substring(0, 8);
+  }
+
+  /**
+   * Generate collection name using the format: {shop_custom_id}-{category_name}_{url_hash}
+   * The URL hash ensures that the same URL always maps to the same collection,
+   * allowing for overwrites when content is rescraped.
+   */
+  generateCollectionName(shopCustomId: string, category: string, url: string): string {
+    const urlHash = this.generateUrlHash(url);
+    return `${shopCustomId}-${category}_${urlHash}`;
   }
 
   /**
    * Parse collection name to extract shop custom ID and category
+   * Supports both old iterator format and new hash format:
+   * - Old: {shop_custom_id}-{category}_{iterator}
+   * - New: {shop_custom_id}-{category}_{url_hash}
    */
-  parseCollectionName(collectionName: string): { shopCustomId: string; category: string; iterator: number } | null {
-    const match = collectionName.match(/^(.+)-(.+)_(\d+)$/);
+  parseCollectionName(collectionName: string): { shopCustomId: string; category: string; urlHash?: string; iterator?: number } | null {
+    // Try to match the general pattern first
+    const match = collectionName.match(/^(.+)-(.+)_([a-f0-9]+|\d+)$/);
     if (!match) return null;
 
+    const identifier = match[3];
+    const isHash = /^[a-f0-9]{8}$/.test(identifier); // 8 character hex hash
+    const isIterator = /^\d+$/.test(identifier); // numeric iterator
+
+    if (!isHash && !isIterator) return null;
+
     return {
       shopCustomId: match[1],
       category: match[2],
-      iterator: parseInt(match[3], 10)
+      ...(isHash ? { urlHash: identifier } : { iterator: parseInt(identifier, 10) })
     };
   }
 
@@ -366,6 +387,7 @@ export class QdrantService {
 
   /**
    * Get next iterator number for a category
+   * @deprecated This method is deprecated. Collection names now use URL hashes instead of iterators.
    */
   async getNextIterator(shopCustomId: string, category: string): Promise<number> {
     if (!this.enabled) return 0;
@@ -375,11 +397,11 @@ export class QdrantService {
 
       if (collections.length === 0) return 0;
 
-      // Find the highest iterator number
+      // Find the highest iterator number (only for backwards compatibility with old collections)
       let maxIterator = -1;
       for (const collectionName of collections) {
         const parsed = this.parseCollectionName(collectionName);
-        if (parsed && parsed.iterator > maxIterator) {
+        if (parsed && parsed.iterator !== undefined && parsed.iterator > maxIterator) {
           maxIterator = parsed.iterator;
         }
       }