refactor: consolidate Qdrant collections to one per category

Changes:
- Modified QdrantService to use {shop}-{category} naming (one collection per category)
- Updated point ID generation to include URL for uniqueness: SHA256(url:hash:chunk)
- Created QdrantMigrationService for v1 to v2 migration
- Added migration API endpoints:
  - GET /api/shops/:shopId/qdrant/migration-status
  - POST /api/shops/:shopId/qdrant/migrate
  - POST /api/shops/:shopId/qdrant/cleanup-old-collections
- Updated QdrantEmbeddingWorkflow to use new collection structure
- Maintained backward compatibility with legacy format
- Updated documentation with migration guide

Benefits:
- Eliminates URL duplication across categories
- Better organization with consolidated collections
- Improved search performance (single collection vs multi-collection)
- Simpler management and monitoring

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

docs/QDRANT_IMPLEMENTATION.md

@@ -120,16 +120,28 @@ CREATE TABLE mcp_logs (
 
 ## Collection Naming Convention
 
-Collections follow a specific pattern for organization and isolation:
+### Current Structure (v2)
+
+Collections follow a simplified pattern with **one collection per category**:
 
 ```
-{shop_custom_id}-{category}_{iterator}
+{shop_custom_id}-{category}
 ```
 
 **Examples:**
-- `shop123-shipping_1` - First shipping information collection
-- `shop123-contacts_2` - Second contacts collection
-- `shop456-faq_1` - First FAQ collection
+- `shop123-shipping` - All shipping information URLs for shop123
+- `shop123-contacts` - All contact information URLs for shop123
+- `shop456-faq` - All FAQ URLs for shop456
+
+Within each collection, **each URL is stored as unique points** (one per content chunk). Point IDs are deterministic based on:
+```
+SHA256(url + content_hash + chunk_index)
+```
+
+This ensures:
+- **Uniqueness**: Different URLs have different point IDs
+- **Deduplication**: Same URL with same content = same point ID (upsert behavior)
+- **Change tracking**: Same URL with changed content = new point IDs
 
 **Categories:**
 - `shipping` - Shipping and delivery information
@@ -137,6 +149,29 @@ Collections follow a specific pattern for organization and isolation:
 - `terms` - Terms of service and conditions
 - `faq` - Frequently asked questions
 
+### Legacy Structure (v1)
+
+The previous implementation used URL-specific collections:
+
+```
+{shop_custom_id}-{category}_{url_hash}
+```
+
+**Examples:**
+- `shop123-shipping_a1b2c3d4` - Single URL's shipping information
+- `shop123-contacts_e5f6g7h8` - Single URL's contact information
+
+**Migration**: Use the migration API endpoints to convert from legacy to current structure.
+
+### Benefits of v2 Structure
+
+1. **Eliminates URL Duplication**: The same URL appearing in multiple categories (e.g., a help page in both FAQ and Contacts) now gets stored as separate unique points rather than creating duplicate collections
+2. **Better Organization**: One collection per category instead of dozens of URL-specific collections
+3. **Improved Search Performance**: Single collection search is more efficient than aggregating results from multiple collections
+4. **Simpler Management**: Easier to monitor, backup, and maintain consolidated collections
+5. **Reduced Storage**: Less metadata overhead with fewer collections
+6. **Backward Compatible**: System automatically handles both old and new formats during transition
+
 ## MCP Tools
 
 The system provides 4 semantic search tools accessible to LLMs:
@@ -199,6 +234,49 @@ interface SearchResult {
 - `DELETE /api/qdrant/collections/:name` - Delete collection
 - `POST /api/qdrant/shops/:shopId/schedule-deletion` - Schedule deletion
 
+### Migration Management (v1 to v2)
+
+- `GET /api/shops/:shopId/qdrant/migration-status` - Check if shop needs migration
+- `POST /api/shops/:shopId/qdrant/migrate` - Migrate shop to new collection structure
+- `POST /api/shops/:shopId/qdrant/cleanup-old-collections` - Remove old collections after migration
+
+**Migration Workflow:**
+
+1. **Check Status**: `GET /api/shops/:shopId/qdrant/migration-status`
+   ```json
+   {
+     "needsMigration": true,
+     "oldCollections": 15,
+     "newCollections": 0,
+     "categories": [
+       {
+         "category": "faq",
+         "oldCollections": 5,
+         "newCollectionExists": false
+       }
+     ]
+   }
+   ```
+
+2. **Run Migration**: `POST /api/shops/:shopId/qdrant/migrate`
+   - Copies all points from old collections to new consolidated collections
+   - Preserves all metadata and vectors
+   - Creates one collection per category
+
+3. **Verify**: Test that search works with new collections
+
+4. **Cleanup (Dry Run)**: `POST /api/shops/:shopId/qdrant/cleanup-old-collections`
+   ```json
+   { "dryRun": true }
+   ```
+   Shows what would be deleted without actually deleting
+
+5. **Cleanup (Actual)**: `POST /api/shops/:shopId/qdrant/cleanup-old-collections`
+   ```json
+   { "dryRun": false }
+   ```
+   Permanently removes old collection structure
+
 ### MCP Analytics
 
 - `GET /api/mcp/stats` - MCP statistics

src/api/components/QdrantEndpoint.ts

@@ -6,6 +6,7 @@ import { QdrantService } from '../../services/QdrantService';
 import { EmbeddingService } from '../../services/EmbeddingService';
 import { QdrantCleanupService } from '../../services/QdrantCleanupService';
 import { QdrantEmbeddingWorkflow } from '../../services/QdrantEmbeddingWorkflow';
+import { QdrantMigrationService } from '../../services/QdrantMigrationService';
 import { SchedulerService } from '../../services/SchedulerService';
 import type { AppConfig } from '../../config';
 
@@ -587,6 +588,127 @@ export class QdrantEndpoint extends BaseEndpointComponent {
             '200': { description: 'Pending embeddings cleared' }
           }
         }
+      ),
+
+      // Migration Management
+      this.createEndpoint(
+        'GET',
+        '/api/shops/:shopId/qdrant/migration-status',
+        this.getMigrationStatus.bind(this),
+        {
+          summary: 'Get migration status',
+          description: 'Check if shop needs migration from old collection structure to new structure',
+          tags: ['Qdrant'],
+          requiresAuth: true,
+          parameters: [
+            {
+              name: 'shopId',
+              in: 'path',
+              type: 'string',
+              required: true,
+              description: 'Shop ID'
+            }
+          ],
+          responses: {
+            '200': {
+              description: 'Migration status',
+              schema: {
+                type: 'object',
+                properties: {
+                  needsMigration: { type: 'boolean' },
+                  oldCollections: { type: 'number' },
+                  newCollections: { type: 'number' },
+                  categories: { type: 'array' }
+                }
+              }
+            },
+            '404': { description: 'Shop not found' }
+          }
+        }
+      ),
+
+      this.createEndpoint(
+        'POST',
+        '/api/shops/:shopId/qdrant/migrate',
+        this.migrateShop.bind(this),
+        {
+          summary: 'Migrate shop to new collection structure',
+          description: 'Migrate shop from old collection structure (one per URL) to new structure (one per category)',
+          tags: ['Qdrant'],
+          requiresAuth: true,
+          parameters: [
+            {
+              name: 'shopId',
+              in: 'path',
+              type: 'string',
+              required: true,
+              description: 'Shop ID'
+            }
+          ],
+          responses: {
+            '200': {
+              description: 'Migration completed',
+              schema: {
+                type: 'object',
+                properties: {
+                  success: { type: 'boolean' },
+                  collectionsProcessed: { type: 'number' },
+                  pointsMigrated: { type: 'number' },
+                  errors: { type: 'array', items: { type: 'string' } }
+                }
+              }
+            },
+            '404': { description: 'Shop not found' },
+            '503': { description: 'Migration service not available' }
+          }
+        }
+      ),
+
+      this.createEndpoint(
+        'POST',
+        '/api/shops/:shopId/qdrant/cleanup-old-collections',
+        this.cleanupOldCollections.bind(this),
+        {
+          summary: 'Cleanup old collections after migration',
+          description: 'Delete old collection structure after successful migration',
+          tags: ['Qdrant'],
+          requiresAuth: true,
+          parameters: [
+            {
+              name: 'shopId',
+              in: 'path',
+              type: 'string',
+              required: true,
+              description: 'Shop ID'
+            }
+          ],
+          requestBody: {
+            required: false,
+            schema: {
+              type: 'object',
+              properties: {
+                dryRun: {
+                  type: 'boolean',
+                  description: 'If true, only show what would be deleted without actually deleting'
+                }
+              }
+            }
+          },
+          responses: {
+            '200': {
+              description: 'Cleanup completed',
+              schema: {
+                type: 'object',
+                properties: {
+                  deleted: { type: 'array', items: { type: 'string' } },
+                  kept: { type: 'array', items: { type: 'string' } },
+                  errors: { type: 'array', items: { type: 'string' } }
+                }
+              }
+            },
+            '404': { description: 'Shop not found' }
+          }
+        }
       )
     ];
   }
@@ -1197,4 +1319,106 @@ export class QdrantEndpoint extends BaseEndpointComponent {
       res.status(500).json({ error: 'Internal server error' });
     }
   }
+
+  private async getMigrationStatus(req: Request, res: Response): Promise<void> {
+    try {
+      const { shopId } = req.params;
+
+      // Get shop from database
+      const shop = this.db.getShopByAnyId(shopId);
+      if (!shop) {
+        res.status(404).json({ error: 'Shop not found' });
+        return;
+      }
+
+      if (!shop.custom_id) {
+        res.status(400).json({ error: 'Shop must have a custom_id' });
+        return;
+      }
+
+      // Create migration service
+      const migrationService = new QdrantMigrationService(this.config, this.db);
+
+      // Get migration status
+      const status = await migrationService.getMigrationStatus(shop.custom_id);
+
+      res.json(status);
+
+    } catch (error) {
+      logger.error('Failed to get migration status:', error);
+      res.status(500).json({ error: 'Internal server error' });
+    }
+  }
+
+  private async migrateShop(req: Request, res: Response): Promise<void> {
+    try {
+      const { shopId } = req.params;
+
+      // Get shop from database
+      const shop = this.db.getShopByAnyId(shopId);
+      if (!shop) {
+        res.status(404).json({ error: 'Shop not found' });
+        return;
+      }
+
+      if (!shop.custom_id) {
+        res.status(400).json({ error: 'Shop must have a custom_id' });
+        return;
+      }
+
+      // Create migration service
+      const migrationService = new QdrantMigrationService(this.config, this.db);
+
+      // Run migration
+      logger.info(`Starting migration for shop ${shop.custom_id}`);
+      const result = await migrationService.migrateShop(shop.custom_id);
+
+      res.json({
+        ...result,
+        shop_id: shopId,
+        shop_custom_id: shop.custom_id
+      });
+
+    } catch (error) {
+      logger.error('Failed to migrate shop:', error);
+      res.status(500).json({ error: 'Internal server error' });
+    }
+  }
+
+  private async cleanupOldCollections(req: Request, res: Response): Promise<void> {
+    try {
+      const { shopId } = req.params;
+      const { dryRun = true } = req.body || {};
+
+      // Get shop from database
+      const shop = this.db.getShopByAnyId(shopId);
+      if (!shop) {
+        res.status(404).json({ error: 'Shop not found' });
+        return;
+      }
+
+      if (!shop.custom_id) {
+        res.status(400).json({ error: 'Shop must have a custom_id' });
+        return;
+      }
+
+      // Create migration service
+      const migrationService = new QdrantMigrationService(this.config, this.db);
+
+      // Run cleanup
+      logger.info(`Starting cleanup for shop ${shop.custom_id} (dryRun: ${dryRun})`);
+      const result = await migrationService.cleanupOldCollections(shop.custom_id, dryRun);
+
+      res.json({
+        ...result,
+        shop_id: shopId,
+        shop_custom_id: shop.custom_id,
+        dry_run: dryRun
+      });
+
+    } catch (error) {
+      logger.error('Failed to cleanup old collections:', error);
+      res.status(500).json({ error: 'Internal server error' });
+    }
+  }
 }

src/services/QdrantEmbeddingWorkflow.ts

@@ -36,11 +36,14 @@ export class QdrantEmbeddingWorkflow {
   }
 
   /**
-   * Generate deterministic point ID based on content hash and chunk index
-   * This ensures deduplication: same content → same point ID → upsert instead of duplicate
+   * Generate deterministic point ID based on URL, content hash, and chunk index
+   * This ensures uniqueness per URL within a collection and deduplication on re-scrape
+   * Format: SHA256(url + content_hash + chunk_index)
    */
-  private generateDeterministicPointId(contentHash: string, chunkIndex: number): string {
-    const combined = `${contentHash}-chunk-${chunkIndex}`;
+  private generateDeterministicPointId(url: string, contentHash: string, chunkIndex: number): string {
+    // Include URL to ensure uniqueness within the single collection per category
+    // Include content hash to enable deduplication on re-scrape
+    const combined = `${url}:${contentHash}:chunk-${chunkIndex}`;
     return createHash('sha256').update(combined).digest('hex').substring(0, 32);
   }
 
@@ -100,16 +103,29 @@ export class QdrantEmbeddingWorkflow {
 
       logger.info(`Content ${contentId} split into ${chunks.length} chunks (avg: ${chunkStats.avgWordCount} words/chunk)`);
 
-      // Generate collection name based on URL hash
-      // This ensures the same URL always maps to the same collection for syncing
-      const collectionName = this.qdrantService.generateCollectionName(shop.custom_id, contentType, url);
+      // Generate collection name for the category (one collection per shop+category)
+      // All URLs in this category will be stored as points in this collection
+      const collectionName = this.qdrantService.generateCollectionName(shop.custom_id, contentType);
       logger.debug(`Using collection ${collectionName} for content ${contentId} (URL: ${url})`);
 
-      // Note: No need to manually delete old points anymore!
-      // With deterministic point IDs based on content_hash + chunk_index:
-      // - If content unchanged: Same hash → Same point ID → Upsert = no-op or update
-      // - If content changed: New hash → New point ID → New points created
-      // - Old points with different hashes will remain until cleanup
+      // With deterministic point IDs based on URL + content_hash + chunk_index:
+      // - Different URLs: Different point IDs (unique within collection)
+      // - Same URL, unchanged content: Same point ID → Upsert = no-op
+      // - Same URL, changed content: Different point ID → New points created
+      // - Old points with different content hashes should be cleaned up manually
+
+      // Delete old points for this URL when content changes
+      if (contentChanged) {
+        try {
+          await this.qdrantService.deletePointsByContentId(collectionName, contentId);
+          logger.debug(`Deleted old points for content ${contentId} in collection ${collectionName}`);
+        } catch (error: any) {
+          // Collection might not exist yet, which is fine
+          if (error.status !== 404) {
+            logger.warn(`Failed to delete old points for content ${contentId}:`, error);
+          }
+        }
+      }
 
       // Delete old embedding records from database (we'll create new ones)
       if (existingEmbeddings.length > 0) {
@@ -148,9 +164,10 @@ export class QdrantEmbeddingWorkflow {
           }
 
           // Create point with chunk data
-          // Use deterministic point ID based on content hash for deduplication
-          // Same content → same hash → same point ID → upsert overwrites instead of duplicating
-          const pointId = this.generateDeterministicPointId(contentHash, chunk.index);
+          // Use deterministic point ID based on URL + content hash for uniqueness and deduplication
+          // Different URLs → different point IDs (unique within collection)
+          // Same URL + same content → same point ID → upsert overwrites instead of duplicating
+          const pointId = this.generateDeterministicPointId(url, contentHash, chunk.index);
           const point = {
             id: pointId,
             vector: embeddingResult.embedding,

src/services/QdrantMigrationService.ts

@@ -0,0 +1,327 @@
+import { logger } from '../utils/logger';
+import { QdrantService } from './QdrantService';
+import { ShopDatabase } from '../database/Database';
+import type { AppConfig } from '../config';
+
+export interface MigrationResult {
+  success: boolean;
+  collectionsProcessed: number;
+  pointsMigrated: number;
+  errors: string[];
+}
+
+/**
+ * Service to migrate from old collection structure (one per URL) to new structure (one per category)
+ */
+export class QdrantMigrationService {
+  private qdrantService: QdrantService;
+
+  constructor(
+    private config: AppConfig,
+    private database: ShopDatabase
+  ) {
+    this.qdrantService = new QdrantService(config);
+  }
+
+  /**
+   * Migrate a shop's collections from old structure to new structure
+   * Old: {shop_custom_id}-{category}_{url_hash} (many collections per category)
+   * New: {shop_custom_id}-{category} (one collection per category)
+   */
+  async migrateShop(shopCustomId: string): Promise<MigrationResult> {
+    const result: MigrationResult = {
+      success: true,
+      collectionsProcessed: 0,
+      pointsMigrated: 0,
+      errors: []
+    };
+
+    try {
+      logger.info(`Starting migration for shop: ${shopCustomId}`);
+
+      // Get all collections for this shop
+      const collections = await this.qdrantService.getCollectionsForShop(shopCustomId);
+
+      if (collections.length === 0) {
+        logger.info(`No collections found for shop ${shopCustomId}`);
+        return result;
+      }
+
+      // Group collections by category
+      const collectionsByCategory = new Map<string, string[]>();
+
+      for (const collectionName of collections) {
+        const parsed = this.qdrantService.parseCollectionName(collectionName);
+        if (!parsed) {
+          result.errors.push(`Failed to parse collection name: ${collectionName}`);
+          continue;
+        }
+
+        // Skip if already in new format (no hash or iterator)
+        if (!parsed.urlHash && parsed.iterator === undefined) {
+          logger.debug(`Collection ${collectionName} already in new format, skipping`);
+          continue;
+        }
+
+        // Group by category
+        if (!collectionsByCategory.has(parsed.category)) {
+          collectionsByCategory.set(parsed.category, []);
+        }
+        collectionsByCategory.get(parsed.category)!.push(collectionName);
+      }
+
+      // Process each category
+      for (const [category, oldCollections] of collectionsByCategory) {
+        try {
+          const migratedPoints = await this.migrateCategory(
+            shopCustomId,
+            category,
+            oldCollections
+          );
+
+          result.collectionsProcessed += oldCollections.length;
+          result.pointsMigrated += migratedPoints;
+
+          logger.info(`Migrated category ${category}: ${migratedPoints} points from ${oldCollections.length} collections`);
+        } catch (error: any) {
+          const errorMsg = `Failed to migrate category ${category}: ${error.message}`;
+          logger.error(errorMsg, error);
+          result.errors.push(errorMsg);
+          result.success = false;
+        }
+      }
+
+      logger.info(`Migration completed for shop ${shopCustomId}: ${result.pointsMigrated} points from ${result.collectionsProcessed} collections`);
+
+    } catch (error: any) {
+      logger.error(`Migration failed for shop ${shopCustomId}:`, error);
+      result.success = false;
+      result.errors.push(error.message || 'Unknown error');
+    }
+
+    return result;
+  }
+
+  /**
+   * Migrate a single category's collections to the new format
+   */
+  private async migrateCategory(
+    shopCustomId: string,
+    category: string,
+    oldCollections: string[]
+  ): Promise<number> {
+    let pointsMigrated = 0;
+
+    // Generate new collection name
+    const newCollectionName = this.qdrantService.generateCollectionName(shopCustomId, category);
+
+    // Create new collection if it doesn't exist
+    if (!(await this.qdrantService.collectionExists(newCollectionName))) {
+      await this.qdrantService.createCollection(newCollectionName);
+      logger.info(`Created new collection: ${newCollectionName}`);
+    }
+
+    // Copy points from old collections to new collection
+    for (const oldCollectionName of oldCollections) {
+      try {
+        const points = await this.getAllPointsFromCollection(oldCollectionName);
+
+        if (points.length === 0) {
+          logger.debug(`No points found in collection ${oldCollectionName}`);
+          continue;
+        }
+
+        // Upsert points to new collection
+        for (const point of points) {
+          try {
+            await this.qdrantService.upsertPoint(newCollectionName, point);
+            pointsMigrated++;
+          } catch (error: any) {
+            logger.warn(`Failed to migrate point ${point.id} from ${oldCollectionName}:`, error);
+          }
+        }
+
+        logger.info(`Migrated ${points.length} points from ${oldCollectionName} to ${newCollectionName}`);
+
+      } catch (error: any) {
+        logger.error(`Failed to process collection ${oldCollectionName}:`, error);
+        throw error;
+      }
+    }
+
+    return pointsMigrated;
+  }
+
+  /**
+   * Get all points from a collection
+   * Uses scroll API to retrieve all points in batches
+   */
+  private async getAllPointsFromCollection(collectionName: string): Promise<any[]> {
+    const points: any[] = [];
+    const batchSize = 100;
+    let offset: string | number | undefined = undefined;
+
+    try {
+      // Use Qdrant's scroll API to get all points
+      while (true) {
+        const response: any = await (this.qdrantService as any).client.scroll(collectionName, {
+          limit: batchSize,
+          offset,
+          with_payload: true,
+          with_vector: true
+        });
+
+        if (!response.points || response.points.length === 0) {
+          break;
+        }
+
+        for (const point of response.points) {
+          points.push({
+            id: String(point.id),
+            vector: point.vector,
+            payload: point.payload
+          });
+        }
+
+        // Check if there are more points
+        if (!response.next_page_offset) {
+          break;
+        }
+
+        offset = response.next_page_offset;
+      }
+
+      return points;
+
+    } catch (error: any) {
+      if (error.status === 404) {
+        logger.debug(`Collection ${collectionName} not found`);
+        return [];
+      }
+      throw error;
+    }
+  }
+
+  /**
+   * Delete old collections after successful migration
+   * WARNING: This will permanently delete the old collections!
+   */
+  async cleanupOldCollections(shopCustomId: string, dryRun: boolean = true): Promise<{
+    deleted: string[];
+    kept: string[];
+    errors: string[];
+  }> {
+    const result = {
+      deleted: [] as string[],
+      kept: [] as string[],
+      errors: [] as string[]
+    };
+
+    try {
+      const collections = await this.qdrantService.getCollectionsForShop(shopCustomId);
+
+      for (const collectionName of collections) {
+        const parsed = this.qdrantService.parseCollectionName(collectionName);
+
+        if (!parsed) {
+          result.errors.push(`Failed to parse collection name: ${collectionName}`);
+          continue;
+        }
+
+        // Only delete collections in old format (with hash or iterator)
+        if (parsed.urlHash || parsed.iterator !== undefined) {
+          if (dryRun) {
+            logger.info(`[DRY RUN] Would delete collection: ${collectionName}`);
+            result.deleted.push(collectionName);
+          } else {
+            try {
+              await this.qdrantService.deleteCollection(collectionName);
+              logger.info(`Deleted old collection: ${collectionName}`);
+              result.deleted.push(collectionName);
+            } catch (error: any) {
+              const errorMsg = `Failed to delete collection ${collectionName}: ${error.message}`;
+              logger.error(errorMsg);
+              result.errors.push(errorMsg);
+            }
+          }
+        } else {
+          result.kept.push(collectionName);
+        }
+      }
+
+    } catch (error: any) {
+      logger.error(`Failed to cleanup old collections for shop ${shopCustomId}:`, error);
+      result.errors.push(error.message || 'Unknown error');
+    }
+
+    return result;
+  }
+
+  /**
+   * Get migration status for a shop
+   */
+  async getMigrationStatus(shopCustomId: string): Promise<{
+    needsMigration: boolean;
+    oldCollections: number;
+    newCollections: number;
+    categories: {
+      category: string;
+      oldCollections: number;
+      newCollectionExists: boolean;
+    }[];
+  }> {
+    const collections = await this.qdrantService.getCollectionsForShop(shopCustomId);
+
+    const categoriesMap = new Map<string, {
+      oldCollections: number;
+      newCollectionExists: boolean;
+    }>();
+
+    let oldCollectionCount = 0;
+    let newCollectionCount = 0;
+
+    for (const collectionName of collections) {
+      const parsed = this.qdrantService.parseCollectionName(collectionName);
+      if (!parsed) continue;
+
+      const isOldFormat = parsed.urlHash !== undefined || parsed.iterator !== undefined;
+
+      if (isOldFormat) {
+        oldCollectionCount++;
+
+        if (!categoriesMap.has(parsed.category)) {
+          categoriesMap.set(parsed.category, {
+            oldCollections: 0,
+            newCollectionExists: false
+          });
+        }
+
+        const categoryData = categoriesMap.get(parsed.category)!;
+        categoryData.oldCollections++;
+
+      } else {
+        newCollectionCount++;
+
+        if (!categoriesMap.has(parsed.category)) {
+          categoriesMap.set(parsed.category, {
+            oldCollections: 0,
+            newCollectionExists: true
+          });
+        } else {
+          categoriesMap.get(parsed.category)!.newCollectionExists = true;
+        }
+      }
+    }
+
+    return {
+      needsMigration: oldCollectionCount > 0,
+      oldCollections: oldCollectionCount,
+      newCollections: newCollectionCount,
+      categories: Array.from(categoriesMap.entries()).map(([category, data]) => ({
+        category,
+        oldCollections: data.oldCollections,
+        newCollectionExists: data.newCollectionExists
+      }))
+    };
+  }
+}

src/services/QdrantService.ts

@@ -55,42 +55,52 @@ export class QdrantService {
   }
 
   /**
-   * Generate a short hash from a URL for use in collection names
+   * Generate a short hash from a URL for use in collection names (deprecated)
+   * @deprecated URL hashes are no longer used in collection names
    */
   private generateUrlHash(url: string): string {
     return createHash('md5').update(url).digest('hex').substring(0, 8);
   }
 
   /**
-   * Generate collection name using the format: {shop_custom_id}-{category_name}_{url_hash}
-   * The URL hash ensures that the same URL always maps to the same collection,
-   * allowing for overwrites when content is rescraped.
+   * Generate collection name using the format: {shop_custom_id}-{category_name}
+   * One collection per category, with unique points per URL based on content hash
    */
-  generateCollectionName(shopCustomId: string, category: string, url: string): string {
-    const urlHash = this.generateUrlHash(url);
-    return `${shopCustomId}-${category}_${urlHash}`;
+  generateCollectionName(shopCustomId: string, category: string, url?: string): string {
+    // URL parameter kept for backward compatibility but no longer used
+    return `${shopCustomId}-${category}`;
   }
 
   /**
    * Parse collection name to extract shop custom ID and category
-   * Supports both old iterator format and new hash format:
-   * - Old: {shop_custom_id}-{category}_{iterator}
-   * - New: {shop_custom_id}-{category}_{url_hash}
+   * Supports multiple formats:
+   * - Current: {shop_custom_id}-{category}
+   * - Legacy with URL hash: {shop_custom_id}-{category}_{url_hash}
+   * - Legacy with iterator: {shop_custom_id}-{category}_{iterator}
    */
   parseCollectionName(collectionName: string): { shopCustomId: string; category: string; urlHash?: string; iterator?: number } | null {
-    // Try to match the general pattern first
-    const match = collectionName.match(/^(.+)-(.+)_([a-f0-9]+|\d+)$/);
-    if (!match) return null;
+    // Try new format first: {shop_custom_id}-{category}
+    const simpleMatch = collectionName.match(/^(.+)-([^_]+)$/);
+    if (simpleMatch) {
+      return {
+        shopCustomId: simpleMatch[1],
+        category: simpleMatch[2]
+      };
+    }
+
+    // Try legacy format with identifier: {shop_custom_id}-{category}_{identifier}
+    const legacyMatch = collectionName.match(/^(.+)-(.+)_([a-f0-9]+|\d+)$/);
+    if (!legacyMatch) return null;
 
-    const identifier = match[3];
+    const identifier = legacyMatch[3];
     const isHash = /^[a-f0-9]{8}$/.test(identifier); // 8 character hex hash
     const isIterator = /^\d+$/.test(identifier); // numeric iterator
 
     if (!isHash && !isIterator) return null;
 
     return {
-      shopCustomId: match[1],
-      category: match[2],
+      shopCustomId: legacyMatch[1],
+      category: legacyMatch[2],
       ...(isHash ? { urlHash: identifier } : { iterator: parseInt(identifier, 10) })
     };
   }
@@ -269,7 +279,9 @@ export class QdrantService {
   }
 
   /**
-   * Search across multiple collections for a category
+   * Search in a category collection
+   * Searches in the single collection for the shop+category combination.
+   * Falls back to searching in legacy multi-collection format if needed.
    */
   async searchInCategory(
     shopCustomId: string,
@@ -280,23 +292,34 @@ export class QdrantService {
     if (!this.enabled) return [];
 
     try {
-      // Get all collections that match the pattern
-      const collections = await this.getCollectionsForCategory(shopCustomId, category);
+      // Try new single-collection format first
+      const newCollectionName = this.generateCollectionName(shopCustomId, category);
+      const newCollectionExists = await this.collectionExists(newCollectionName);
+
+      if (newCollectionExists) {
+        // Search in the single collection for this category
+        const results = await this.searchInCollection(newCollectionName, queryVector, limit);
+        return results;
+      }
 
-      if (collections.length === 0) {
+      // Fallback: Search in legacy multi-collection format
+      logger.debug(`Using legacy multi-collection search for shop ${shopCustomId} category ${category}`);
+      const legacyCollections = await this.getCollectionsForCategory(shopCustomId, category);
+
+      if (legacyCollections.length === 0) {
         logger.debug(`No collections found for shop ${shopCustomId} category ${category}`);
         return [];
       }
 
-      // Search in all collections and combine results
+      // Search in all legacy collections and combine results
       const allResults: QdrantSearchResult[] = [];
 
-      for (const collectionName of collections) {
+      for (const collectionName of legacyCollections) {
         try {
           const results = await this.searchInCollection(collectionName, queryVector, limit);
           allResults.push(...results);
         } catch (error) {
-          logger.warn(`Failed to search in collection ${collectionName}, skipping:`, error);
+          logger.warn(`Failed to search in legacy collection ${collectionName}, skipping:`, error);
         }
       }