docs: add universal scraper implementation plan

20-task plan across 4 phases:
- Phase 1 (Tasks 0-6): shops→sites rename across 45 files + API backward compat
- Phase 2 (Tasks 7-10): site types, platform detection, custom sitemap, modal
- Phase 3 (Tasks 11-14): 13 content categories, link discovery, URL exclusions
- Phase 4 (Tasks 15-19): web UI panels, MCP tools, docs update

docs/superpowers/plans/2026-04-13-universal-scraper.md

@@ -0,0 +1,323 @@
+# Universal Scraper Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox syntax for tracking.
+
+**Goal:** Transform the webshop-only scraper into a universal website scraper with site types, 13 content categories, link discovery, and URL exclusions.
+
+**Architecture:** 4 sequential phases — rename (shops to sites), site types + sitemap, categories + link discovery, web UI. Each phase commits independently. Subagent parallelism within phases 3 and 4.
+
+**Tech Stack:** TypeScript, Express, SQLite (better-sqlite3), Cheerio, Qdrant, Vite + Tailwind (web UI)
+
+**Spec:** docs/superpowers/specs/2026-04-13-universal-scraper-design.md
+
+---
+
+## Pre-flight
+
+### Task 0: Safety tag + build verification
+
+**Files:** none (git only)
+
+- [ ] Create rollback tag: `git tag pre-universal-scraper`
+- [ ] Verify clean TS build: `npx tsc --noEmit` (expect zero errors)
+- [ ] Verify web UI builds: `cd web && npm run build && cd ..`
+
+---
+
+## Phase 1: Rename shops to sites
+
+### Task 1: Rename TypeScript types and interfaces
+
+**Files:**
+- Modify: `src/types/index.ts`
+- Modify: `src/database/Database.ts` (interfaces section only)
+
+- [ ] In src/types/index.ts: `WebshopType` to `SitePlatform`, add `| 'generic'` value, `shopId` to `siteId` in ScraperJob
+- [ ] In src/database/Database.ts interfaces: `Shop` to `Site`, `ShopContent` to `SiteContent`, `ShopAnalytics` to `SiteAnalytics`, `ShopWithAnalytics` to `SiteWithAnalytics`, `ShopDetail` to `SiteDetail`, `ShopEmbedding` to `SiteEmbedding`, `ShopQdrantStatus` to `SiteQdrantStatus`, `ShopCollections` to `SiteCollections`, `ShopMcpAnalytics` to `SiteMcpAnalytics`. Class: `ShopDatabase` to `SiteDatabase`. All `shop_id` fields to `site_id`, `webshop_type` to `site_type`.
+- [ ] Verify types compile (expect downstream errors — those are fixed in later tasks)
+- [ ] Commit: "refactor: rename Shop types to Site types"
+
+---
+
+### Task 2: Database migration — rename tables and columns
+
+**Files:**
+- Modify: `src/database/Database.ts` (DDL, migration, method names, SQL queries)
+
+- [ ] Rename table DDL: `shops` to `sites`, `shop_content` to `site_content`, `shop_analytics` to `site_analytics`, `shop_embeddings` to `site_embeddings`. Rename columns: `shop_id` to `site_id` everywhere, `shop_content_id` to `site_content_id`, `webshop_type` to `site_type`. Rename indexes.
+
+- [ ] Add startup migration method `migrateShopsToSites()` that checks for old `shops` table and renames using ALTER TABLE RENAME and ALTER TABLE RENAME COLUMN (better-sqlite3 bundles SQLite 3.43+ which supports this). This runs before `initializeDatabase()`. Rename all 4 tables and rename columns in all 11 tables that reference `shop_id`.
+
+- [ ] Rename all public methods: `getShopByUrl` to `getSiteByUrl`, `createShop` to `createSite`, `getShopById` to `getSiteById`, `getShopByCustomId` to `getSiteByCustomId`, `getShopByAnyId` to `getSiteByAnyId`, `getAllShopsWithAnalytics` to `getAllSitesWithAnalytics`, `deleteShop` to `deleteSite`, `forceDeleteShop` to `forceDeleteSite`, `getDeletedShops` to `getDeletedSites`, `getAllShopContent` to `getAllSiteContent`, `getShopEmbeddings` to `getSiteEmbeddings`, `getShopQdrantErrors` to `getSiteQdrantErrors`. Rename `shopId` params to `siteId`.
+
+- [ ] Rename all SQL query strings in method bodies: `FROM shops` to `FROM sites`, `INTO shops` to `INTO sites`, all table name and column name references. This is the bulk of the work in this 2270-line file.
+
+- [ ] Verify Database.ts compiles in isolation
+- [ ] Commit: "refactor: rename database tables and methods shops to sites"
+
+---
+
+### Task 3: Rename backend consumers
+
+**Files (all under src/):**
+- Rename: `api/components/ShopsEndpoint.ts` to `api/components/SitesEndpoint.ts`
+- Rename: `scraper/WebshopScraper.ts` to `scraper/WebScraper.ts`
+- Modify: all 30+ backend files (api components, services, scraper, mcp, scheduler, webhooks, index.ts)
+
+- [ ] Rename the two files on disk
+
+- [ ] Systematic find-and-replace across ALL backend files:
+  - Import paths: ShopsEndpoint to SitesEndpoint, WebshopScraper to WebScraper
+  - Classes: ShopDatabase to SiteDatabase, ShopsEndpoint to SitesEndpoint, WebshopScraper to WebScraper
+  - Interfaces: Shop to Site, ShopContent to SiteContent, ShopAnalytics to SiteAnalytics, etc. (careful not to rename Shopify or ShopRenter)
+  - Variables: shopId to siteId, shopCustomId to siteCustomId, webshopType to sitePlatform
+  - Response JSON keys in res.json calls: shop_id to site_id, shops to sites, shop to site, webshop_type to site_type, shop_usage to site_usage
+  - Route paths in createEndpoint calls: /api/shops to /api/sites, /api/qdrant/shops to /api/qdrant/sites, /api/mcp/shops to /api/mcp/sites, :shopId to :siteId
+  - Log messages and string literals: 'Shop ' to 'Site ', 'shop ' to 'site '
+
+- [ ] Update index.ts: WebshopScraper import to WebScraper, ShopDatabase to SiteDatabase
+
+- [ ] Verify full project compiles: `npx tsc --noEmit` (must be zero errors)
+- [ ] Commit: "refactor: rename all backend code shops to sites"
+
+---
+
+### Task 4: API backward compatibility — legacy aliases
+
+**Files:**
+- Create: `src/api/middleware/legacyShopAlias.ts`
+- Modify: `src/api/routes.ts`
+
+- [ ] Create legacyShopAlias.ts: A middleware that monkey-patches res.json to recursively rename keys (site_id to shop_id, site to shop, sites to shops, site_type to webshop_type, site_usage to shop_usage, site_content to shop_content, site_content_id to shop_content_id). About 30 lines total.
+
+- [ ] In routes.ts: After registering all canonical /api/sites routes, loop through all registered endpoints. For any path containing /api/sites or /api/qdrant/sites or /api/mcp/sites, register a second route with the path rewritten to use /shops and :id (instead of :siteId), with the legacyShopAlias middleware prepended.
+
+- [ ] Update all endpoint handlers to accept both param names: `const siteId = req.params.siteId || req.params.id || req.params.shopId`
+
+- [ ] Verify build: `npx tsc --noEmit`
+- [ ] Commit: "feat: add legacy /api/shops backward-compat aliases"
+
+---
+
+### Task 5: Rename web UI
+
+**Files (all under web/src/):**
+- Rename: `components/pages/ShopsPage.ts` to `SitesPage.ts`
+- Rename: `components/pages/ShopDetailPage.ts` to `SiteDetailPage.ts`
+- Modify: Router.ts, ApiService.ts, types/index.ts, Layout.ts, helpers.ts, icons.ts, all page files
+
+- [ ] Rename files on disk
+- [ ] Update Router.ts: imports, routes /shops to /sites
+- [ ] Update ApiService.ts: all method names and API paths from shops to sites
+- [ ] Update web/src/types/index.ts: Shop to Site interfaces, field names
+- [ ] Update Layout.ts: menu labels "Shops" to "Sites", hrefs
+- [ ] Update helpers.ts: getWebshopTypeInfo to getSiteTypeInfo
+- [ ] Update all page files: variable names, UI text, CSS IDs (add-shop-modal to add-site-modal, etc.), ApiService calls
+- [ ] Build web UI: `cd web && npm run build && cd ..`
+- [ ] Full build: `npx tsc --noEmit`
+- [ ] Commit: "refactor: rename web UI shops to sites"
+
+---
+
+### Task 6: Update docs
+
+**Files:** README.md, docs/API.md, docs/SETUP.md
+
+- [ ] Rename all shop references in prose and examples. Add backward compatibility section to API.md. Document both /api/sites (canonical) and /api/shops (legacy) with explanation.
+- [ ] Commit: "docs: update docs for shops to sites rename"
+
+---
+
+## Phase 2: Site types + custom sitemap
+
+### Task 7: Add site_platform and custom_sitemap_url columns
+
+**Files:**
+- Modify: `src/database/Database.ts`
+
+- [ ] Add `site_platform TEXT NOT NULL DEFAULT 'generic'` and `custom_sitemap_url TEXT` to sites table DDL
+- [ ] Add migration: check if site_platform column exists via PRAGMA table_info, if not add columns. Then UPDATE sites SET site_platform = site_type, site_type = 'webshop' WHERE site_type IN ('shoprenter','woocommerce','shopify'). Set remaining non-standard rows to site_type = 'website'.
+- [ ] Update Site interface to include site_platform and custom_sitemap_url
+- [ ] Update createSite method signature to accept site_platform and custom_sitemap_url params
+- [ ] Verify build, commit: "feat: add site_platform and custom_sitemap_url columns"
+
+---
+
+### Task 8: Sitemap auto-detection and platform detection
+
+**Files:**
+- Modify: `src/scraper/SitemapParser.ts`
+
+- [ ] Create `detectPlatform(url)` function: hostname check for shoprenter/shopify, HTML fetch for WooCommerce signatures, default to website/generic. Returns `{ siteType, sitePlatform }`.
+
+- [ ] Create `resolveSitemapUrl(baseUrl, customSitemapUrl, siteType)` function: tries custom URL first, then robots.txt Sitemap: directives, then /sitemap.xml, then /sitemap_index.xml. Returns URL string or null.
+
+- [ ] Update existing getSitemapUrl to use new functions
+- [ ] Verify build, commit: "feat: add sitemap auto-detection and platform detection"
+
+---
+
+### Task 9: Update job creation for site types
+
+**Files:**
+- Modify: `src/api/components/JobsEndpoint.ts`
+- Modify: `src/scraper/WebScraper.ts`
+- Modify: `src/api/components/SitesEndpoint.ts`
+
+- [ ] Update POST /api/jobs handler: accept optional site_type, site_platform, custom_sitemap_url. Auto-detect if not provided. Pass to createSite.
+- [ ] Add PATCH /api/sites/:siteId/settings endpoint to SitesEndpoint: updates site_type, site_platform, custom_sitemap_url, max_crawl_depth, max_pages. Legacy alias registered automatically.
+- [ ] Update WebScraper.scrape() to read custom_sitemap_url from site record and pass to resolveSitemapUrl
+- [ ] Verify build, commit: "feat: site type detection and custom sitemap in job creation"
+
+---
+
+### Task 10: Update Add Site modal
+
+**Files:**
+- Modify: `src/api/components/SitesEndpoint.ts` (add detect endpoint)
+- Modify: `web/src/components/pages/SitesPage.ts`
+- Modify: `web/src/services/ApiService.ts`
+
+- [ ] Add POST /api/sites/detect endpoint: accepts {url}, returns {site_type, site_platform, sitemap_url}
+- [ ] Add detectSite(url) to ApiService.ts
+- [ ] Update modal in SitesPage.ts: add Site Type dropdown (Webshop/Website), Sitemap URL input (pre-fillable), "no sitemap" checkbox. Add blur handler on URL input that calls detect endpoint with spinner.
+- [ ] Build web UI, verify, commit: "feat: add site type selector and sitemap detection to add-site modal"
+
+---
+
+## Phase 3: Expanded categories + link discovery
+
+### Task 11: Expand content categories (parallelizable)
+
+**Files:**
+- Modify: `src/types/index.ts`
+- Modify: `src/scraper/SitemapParser.ts`
+- Modify: `src/database/Database.ts`
+- Modify: `src/api/components/CustomUrlsEndpoint.ts`
+
+- [ ] Add ContentType union with all 13 values to types/index.ts
+- [ ] Add 8 new keyword arrays to SitemapParser.ts filterUrlsByType method (services, about, team, blog, pricing, testimonials, gallery, landing). Add 'other' catch-all for unmatched URLs. Include Hungarian and German keyword variants.
+- [ ] Extract classifyUrl(url) and classifyUrls(urls) methods from filterUrlsByType
+- [ ] Update content_type validation in CustomUrlsEndpoint to accept all 13 values
+- [ ] Update any CHECK constraints in DB DDL
+- [ ] Verify build, commit: "feat: expand content categories from 4 to 13"
+
+---
+
+### Task 12: Link discovery engine (parallelizable)
+
+**Files:**
+- Create: `src/scraper/LinkDiscovery.ts`
+
+- [ ] Create LinkDiscovery class with discoverLinks(baseUrl, maxDepth, maxPages) method. BFS crawl using axios + cheerio. Normalizes URLs (strips fragments, tracking params, collapses trailing slashes). Skips static assets, pagination, wp-admin paths. Respects maxDepth (default 3) and maxPages (default 200) limits. Returns DiscoveredUrl array with url, depth, foundOn fields.
+- [ ] Verify build, commit: "feat: add LinkDiscovery engine for sites without sitemaps"
+
+---
+
+### Task 13: URL exclusion table + API (parallelizable)
+
+**Files:**
+- Modify: `src/database/Database.ts`
+- Create: `src/api/components/UrlExclusionsEndpoint.ts`
+- Modify: `src/api/routes.ts`
+
+- [ ] Add url_exclusions table DDL and migration. Add CRUD methods: createUrlExclusion, getUrlExclusions, deleteUrlExclusion, isUrlExcluded (glob-to-regex matching).
+- [ ] Create UrlExclusionsEndpoint with 3 endpoints: POST (add pattern), GET (list), DELETE (remove). Follow CustomUrlsEndpoint pattern.
+- [ ] Register in routes.ts
+- [ ] Verify build, commit: "feat: add URL exclusion table and API endpoints"
+
+---
+
+### Task 14: Integrate link discovery + exclusions into scrape flow
+
+**Files:**
+- Modify: `src/scraper/WebScraper.ts`
+- Modify: `src/database/Database.ts` (add max_crawl_depth, max_pages columns)
+
+- [ ] Add max_crawl_depth and max_pages columns to sites table (DDL + migration)
+- [ ] Update WebScraper.scrape() with full flow: resolve sitemap, parse sitemap URLs if found, run link discovery as supplement (or primary source if no sitemap), merge URL sets, add custom include URLs, remove excluded URLs via isUrlExcluded, classify all URLs by category, scrape.
+- [ ] Verify build, commit: "feat: integrate link discovery and URL exclusions into scrape flow"
+
+---
+
+## Phase 4: Web UI enhancements
+
+### Task 15: Custom URLs panel (parallelizable)
+
+**Files:**
+- Modify: `web/src/components/pages/SiteDetailPage.ts`
+- Modify: `web/src/services/ApiService.ts`
+
+- [ ] Add ApiService methods: getCustomUrls, addCustomUrl, deleteCustomUrl, toggleCustomUrl, getUrlExclusions, addUrlExclusion, deleteUrlExclusion
+- [ ] Add Custom URLs section to SiteDetailPage: Include URLs table (url, category dropdown with 13 options, enabled toggle, delete) + Add URL inline form. Exclude Patterns table (pattern, delete) + Add Pattern inline form with wildcard hint.
+- [ ] Build web UI, commit: "feat: add custom URLs and exclusion panels to site detail page"
+
+---
+
+### Task 16: Crawl settings panel (parallelizable)
+
+**Files:**
+- Modify: `web/src/components/pages/SiteDetailPage.ts`
+- Modify: `web/src/services/ApiService.ts`
+
+- [ ] Add updateSiteSettings method to ApiService
+- [ ] Add Crawl Settings card to SiteDetailPage: editable sitemap URL with re-detect button, max crawl depth input (1-5), max pages input (1-1000), save button.
+- [ ] Build web UI, commit: "feat: add crawl settings panel to site detail page"
+
+---
+
+### Task 17: Content metadata + sites list updates (parallelizable)
+
+**Files:**
+- Modify: `web/src/components/pages/SiteDetailPage.ts`
+- Modify: `web/src/components/pages/SitesPage.ts`
+- Modify: `web/src/utils/helpers.ts`
+
+- [ ] Update content metadata section in SiteDetailPage to show all 13 categories in responsive grid. Grayed out for 0-count categories.
+- [ ] Update SitesPage: add Type column with badges (Webshop/Website), platform subtitle.
+- [ ] Add getSiteTypeInfo helper function for display names and colors.
+- [ ] Build web UI, commit: "feat: update content metadata display and sites list for 13 categories"
+
+---
+
+### Task 18: MCP tools for new categories
+
+**Files:**
+- Create: 9 new tool files in `src/mcp/tools/` (ServicesSearchTool.ts, AboutSearchTool.ts, TeamSearchTool.ts, BlogSearchTool.ts, PricingSearchTool.ts, TestimonialsSearchTool.ts, GallerySearchTool.ts, LandingSearchTool.ts, OtherSearchTool.ts)
+- Modify: `src/mcp/McpServer.ts`
+- Modify: `src/mcp/tools/ListContentsTool.ts`
+- Modify: `src/api/components/SearchEndpoint.ts`
+
+- [ ] Create 9 search tool files following ShippingSearchTool pattern: extend BaseTool, set category string, provide descriptive getDescription and getInputSchema.
+- [ ] Register all 9 in McpServer.ts initializeTools (services_search, about_search, team_search, blog_search, pricing_search, testimonials_search, gallery_search, landing_search, other_search).
+- [ ] Update ListContentsTool CONTENT_TYPE_TO_TOOL map and contentByType/seenUrls to include all 13 categories.
+- [ ] Update SearchEndpoint CATEGORIES array to include all 13 categories.
+- [ ] Verify build, commit: "feat: add MCP search tools for all 13 content categories"
+
+---
+
+### Task 19: Final docs update + push
+
+**Files:** README.md, docs/API.md, docs/SETUP.md
+
+- [ ] Update README: new title (Web Scraper), feature list (site types, 13 categories, link discovery, URL exclusions), MCP tools list (14 tools), examples
+- [ ] Update API.md: new endpoints (detect, settings, url-exclusions), categories, tools, backward compat section
+- [ ] Update SETUP.md: site types, crawl settings, link discovery
+- [ ] Commit and push: "docs: update all docs for universal scraper features"
+
+---
+
+## Parallelization notes
+
+Sequential dependencies:
+- Tasks 0 through 6 (Phase 1): sequential
+- Tasks 7 through 10 (Phase 2): sequential, depends on Phase 1
+- Tasks 11, 12, 13 (Phase 3): parallelizable with each other
+- Task 14: depends on Tasks 11 + 12 + 13
+- Tasks 15, 16, 17 (Phase 4): parallelizable with each other
+- Task 18: depends on Task 11 (needs category types)
+- Task 19: depends on everything
+
+Best parallelization points:
+- Phase 3: dispatch Tasks 11 + 12 + 13 in parallel, then Task 14
+- Phase 4: dispatch Tasks 15 + 16 + 17 + 18 in parallel, then Task 19