fix: eliminate Qdrant duplicate points using content hash-based IDs

Problem:
- After scraping, duplicate points appeared in Qdrant collections
- Random UUIDs created for each point allowed duplicates
- Manual deletion of old points was unreliable (race conditions, failures)

Root Cause:
- Point IDs were random: uuid.v4() generated new ID each time
- Same content re-scraped → new random ID → duplicate point
- Deletion by content_id filter could fail or miss points

Solution - Deterministic Point IDs:
- Generate point ID from SHA-256(content_hash + chunk_index)
- Same content → same hash → same point ID → automatic deduplication
- Qdrant's upsert naturally overwrites instead of duplicating

Implementation:
1. Added generateDeterministicPointId(contentHash, chunkIndex)
   - Uses SHA-256 hash of content_hash + chunk index
   - Returns first 32 characters as point ID

2. Replaced random UUID with deterministic ID:
   - Before: const pointId = uuidv4()
   - After:  const pointId = this.generateDeterministicPointId(contentHash, chunk.index)

3. Removed manual point deletion:
   - No longer needed with deterministic IDs
   - Reduces API calls to Qdrant
   - Eliminates race condition risks

Benefits:
- Zero duplicate points (automatic deduplication)
- Same content → upsert updates existing point
- Changed content → new hash → new point (old cleaned by cleanup service)
- Reduced Qdrant API calls (no manual deletion)
- More reliable and efficient

How It Works:
- Content unchanged: Same hash → Same ID → Upsert updates timestamp
- Content changed:   New hash → New ID → New point created
- Old points with different hashes cleaned up by QdrantCleanupService

Example:
Content "Shipping: Free over $50" with hash abc123...
- Chunk 0: SHA-256("abc123-chunk-0") → point ID: f4a8e9b2...
- Chunk 1: SHA-256("abc123-chunk-1") → point ID: 7d3c1a5e...
Re-scrape same content → same hashes → same IDs → upsert = no duplicates!

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

src/services/QdrantEmbeddingWorkflow.ts

@@ -1,4 +1,5 @@
 import { v4 as uuidv4 } from 'uuid';
+import { createHash } from 'crypto';
 import { logger } from '../utils/logger';
 import { ShopDatabase, ShopContent } from '../database/Database';
 import { QdrantService } from './QdrantService';
@@ -34,6 +35,15 @@ export class QdrantEmbeddingWorkflow {
     this.webhookManager = new WebhookManager(database);
   }
 
+  /**
+   * Generate deterministic point ID based on content hash and chunk index
+   * This ensures deduplication: same content → same point ID → upsert instead of duplicate
+   */
+  private generateDeterministicPointId(contentHash: string, chunkIndex: number): string {
+    const combined = `${contentHash}-chunk-${chunkIndex}`;
+    return createHash('sha256').update(combined).digest('hex').substring(0, 32);
+  }
+
   /**
    * Process content for embedding after it's saved to database
    */
@@ -95,25 +105,18 @@ export class QdrantEmbeddingWorkflow {
       const collectionName = this.qdrantService.generateCollectionName(shop.custom_id, contentType, url);
       logger.debug(`Using collection ${collectionName} for content ${contentId} (URL: ${url})`);
 
-      // Check if this content already has embeddings
-      const existingEmbedding = existingEmbeddings.find(emb => emb.collection_name);
-
-      if (existingEmbedding) {
-        // Check if the collection still exists in Qdrant
-        const collectionExists = await this.qdrantService.collectionExists(existingEmbedding.collection_name);
-
-        if (collectionExists) {
-          // Delete all existing points in this collection for this content_id
-          // This ensures we sync fresh data without duplicates
-          await this.qdrantService.deletePointsByContentId(collectionName, contentId);
-          logger.debug(`Cleared existing points for content ${contentId} in collection ${collectionName}`);
-        }
+      // Note: No need to manually delete old points anymore!
+      // With deterministic point IDs based on content_hash + chunk_index:
+      // - If content unchanged: Same hash → Same point ID → Upsert = no-op or update
+      // - If content changed: New hash → New point ID → New points created
+      // - Old points with different hashes will remain until cleanup
 
-        // Delete all existing embedding records for this content
-        // We'll create new ones for each chunk
+      // Delete old embedding records from database (we'll create new ones)
+      if (existingEmbeddings.length > 0) {
         for (const emb of existingEmbeddings) {
           this.database.deleteShopEmbedding(emb.id);
         }
+        logger.debug(`Cleared ${existingEmbeddings.length} old embedding records for content ${contentId}`);
       }
 
       // Ensure collection exists
@@ -145,7 +148,9 @@ export class QdrantEmbeddingWorkflow {
           }
 
           // Create point with chunk data
-          const pointId = uuidv4();
+          // Use deterministic point ID based on content hash for deduplication
+          // Same content → same hash → same point ID → upsert overwrites instead of duplicating
+          const pointId = this.generateDeterministicPointId(contentHash, chunk.index);
           const point = {
             id: pointId,
             vector: embeddingResult.embedding,