Acasă Explorează Ajutor
Autentificare
fszontagh
/
webshop-scraper
1
0
Bifurcare 0
Fisiere Probleme 4 Trageți solicitările 0 Wiki
Răsfoiți Sursa

fix: extract real article body, surface dedup skips, include 'other' URLs

Three bugs surfaced on artdent.hu (Yoast + Divi WordPress site) causing
~89 blog posts and 6 misc pages to be missing from the RAG corpus:

1. ContentExtractor was using $(selector).first() and breaking on the
   first match >100 chars. WordPress recent-posts widgets render <article>
   tags too, so .first() grabbed a sidebar card; on Divi sites the real
   post body lives in #main-content (not <article>) and was never reached.
   Now we score every match across all candidate selectors by text length
   and keep the largest. <body> is excluded so it only kicks in as the
   final fallback.

2. saveContent skips pages whose content_hash already exists in another
   content_type. That's intentional to avoid double-embedding, but it
   silently swallowed 88 of 89 broken blog extractions (all collided on
   the same 714-byte widget hash). Bumped that log from debug to warn
   so repeated hash collisions are visible.

3. filterUrlsByType dropped every URL classified as 'other' (homepage,
   /cookie-tajekoztato/, /videotar/, language variants) even though
   ALL_CONTENT_TYPES includes 'other' and the MCP server already exposes
   other_search. Removed the filter so 'other' URLs are scraped, saved,
   and embedded like every other category.
fszontagh 2 luni în urmă
părinte
79ce13e3d8
comite
2cfec66a89
3 a modificat fișierele cu 23 adăugiri și 10 ștergeri
Vizualizare divizată Arată Statisticile Diff
  1. 3 1
      src/database/Database.ts
  2. 19 6
      src/scraper/ContentExtractor.ts
  3. 1 3
      src/scraper/SitemapParser.ts

+ 3 - 1
src/database/Database.ts
Vizualizați fișierul 

@@ -954,7 +954,9 @@ export class SiteDatabase {
 
     `).get(siteId, contentHash, contentType) as { id: string; content_type: string; url: string } | undefined;
 
 
     if (duplicateContent) {
 
-      logger.debug(`Skipping duplicate content for ${contentType} (URL: ${url}) - same content already exists in ${duplicateContent.content_type} (URL: ${duplicateContent.url})`);
 
+      // Warn (not debug): if many URLs collide on the same hash, extraction is probably
 
+      // returning boilerplate instead of the real page body.
 
+      logger.warn(`Skipping duplicate content for ${contentType} (URL: ${url}) - same content already exists in ${duplicateContent.content_type} (URL: ${duplicateContent.url})`);
 
       return { changed: false, contentId: duplicateContent.id, skipped: true };
 
     }

+ 19 - 6
src/scraper/ContentExtractor.ts
Vizualizați fișierul 

@@ -67,14 +67,27 @@ export class ContentExtractor {
 
         'body'
 
       ];
 
 
+      // Pick the element with the most text content across all candidate selectors.
 
+      // Two reasons this is needed instead of .first() with an early break:
 
+      //   1. WordPress/Yoast pages often have several <article> tags (real post +
 
+      //      sidebar widget cards) — .first() grabbed the widget.
 
+      //   2. On Divi/Elementor sites the real article body isn't wrapped in <article>
 
+      //      at all (it lives in #main-content), and the only <article> tags are the
 
+      //      300-char widget cards. Breaking on the first selector that crosses a low
 
+      //      threshold meant we never reached #main-content.
 
+      // <body> is excluded from this search because it's the fallback below — letting it
 
+      // win would pull in any boilerplate not stripped above.
 
+      let bestTextLen = 0;
 
       for (const selector of contentSelectors) {
 
-        const element = $(selector).first();
 
-        if (element.length > 0) {
 
-          mainContent = element.html() || '';
 
-          if (mainContent.trim().length > 100) {
 
-            break;
 
+        if (selector === 'body') continue;
 
+        $(selector).each((_, el) => {
 
+          const $el = $(el);
 
+          const textLen = $el.text().trim().length;
 
+          if (textLen > bestTextLen) {
 
+            bestTextLen = textLen;
 
+            mainContent = $el.html() || '';
 
           }
 
-        }
 
+        });
 
       }
 
 
       if (!mainContent) {

+ 1 - 3
src/scraper/SitemapParser.ts
Vizualizați fișierul 

@@ -371,9 +371,7 @@ export class SitemapParser {
 
 
     for (const u of urls) {
 
       const type = this.classifyUrl(u.loc);
 
-      if (type !== 'other') {
 
-        result[type].push(u);
 
-      }
 
+      result[type].push(u);
 
     }
 
 
     return result as Record<ContentType, SitemapUrl[]>;