docs: add universal scraper enhancement design spec

4-phase plan to transform the webshop-only scraper into a universal
website scraper:
- Phase 1: Rename shops→sites (DB, TS, API with backward compat aliases, UI)
- Phase 2: Site types (webshop/website) + custom sitemap + auto-detect
- Phase 3: 13 content categories + link discovery engine + URL exclusions
- Phase 4: Web UI enhancements (custom URL panels, crawl settings, modal)

docs/superpowers/specs/2026-04-13-universal-scraper-design.md

@@ -0,0 +1,307 @@
+# Universal Scraper Enhancement — Design Spec
+
+**Date**: 2026-04-13
+**Status**: Approved
+**Goal**: Transform the webshop-only scraper into a universal website scraper that handles any site type, discovers pages without sitemaps, and supports 13 content categories.
+
+## Phases
+
+The work is split into 4 sequential phases. Each phase produces a commit on `main`. A git tag `pre-universal-scraper` is created before phase 1 starts for rollback safety.
+
+---
+
+## Phase 1: Rename `shops` → `sites`
+
+### Database migration
+
+SQLite does not support `ALTER TABLE RENAME COLUMN` in older versions, so migration creates new tables, copies data, drops old tables.
+
+| Old table | New table |
+|---|---|
+| `shops` | `sites` |
+| `shop_content` | `site_content` |
+| `shop_analytics` | `site_analytics` |
+| `shop_embeddings` | `site_embeddings` |
+
+Column renames inside tables:
+
+| Old column | New column | Tables affected |
+|---|---|---|
+| `shop_id` | `site_id` | `custom_urls`, `webhooks`, `webhook_deliveries`, `scheduled_jobs`, `scrape_history`, `qdrant_deletion_queue`, `qdrant_errors`, `mcp_logs`, `site_content`, `site_analytics`, `site_embeddings` |
+| `shop_content_id` | `site_content_id` | `site_embeddings` |
+| `webshop_type` | `site_type` | `sites` |
+
+All foreign keys, indexes, and unique constraints are recreated on the new tables.
+
+### TypeScript renames
+
+| Old | New |
+|---|---|
+| `Shop`, `ShopDetail`, `ShopWithAnalytics` | `Site`, `SiteDetail`, `SiteWithAnalytics` |
+| `ShopContent`, `ShopAnalytics`, `ShopEmbedding` | `SiteContent`, `SiteAnalytics`, `SiteEmbedding` |
+| `ShopDatabase` | `SiteDatabase` |
+| `ShopsEndpoint` | `SitesEndpoint` |
+| `ShopDetailPage`, `ShopsPage` | `SiteDetailPage`, `SitesPage` |
+| All `shopId` params | `siteId` |
+| `WebshopScraper` | `WebScraper` |
+
+### API routes — dual registration
+
+Every current `/api/shops/*` endpoint is registered twice:
+
+1. **Canonical**: `/api/sites/*` — handlers return responses with `site_*` keys.
+2. **Legacy alias**: `/api/shops/*` — same handlers, with a response-rewriting middleware that renames JSON keys before sending.
+
+Legacy key mapping (applied recursively to the JSON response):
+
+| Canonical key | Legacy key |
+|---|---|
+| `site_id` | `shop_id` |
+| `site` | `shop` |
+| `sites` | `shops` |
+| `site_type` | `webshop_type` |
+| `site_usage` | `shop_usage` |
+| `site_content` | `shop_content` |
+| `site_content_id` | `shop_content_id` |
+
+Message strings (the `message` field value) are NOT rewritten — they say "Site" on both route sets.
+
+The middleware is a single `rewriteKeysForLegacy(obj)` function (~20 lines) that recursively walks the response JSON and applies the key map. It monkey-patches `res.json()` on legacy routes.
+
+Qdrant-related routes follow the same pattern:
+- `/api/qdrant/shops/:shopId/*` → legacy alias
+- `/api/qdrant/sites/:siteId/*` → canonical
+
+MCP routes:
+- `/api/mcp/shops/:shopId/*` → legacy alias
+- `/api/mcp/sites/:siteId/*` → canonical
+
+Search endpoint:
+- `/api/shops/:id/search` → legacy alias (returns `shop_id` in response)
+- `/api/sites/:id/search` → canonical (returns `site_id` in response)
+
+### Web UI
+
+- Router paths: `/shops` → `/sites`, `/shops/:id` → `/sites/:id`
+- Menu label: "Shops" → "Sites"
+- Page class names: `ShopsPage` → `SitesPage`, `ShopDetailPage` → `SiteDetailPage`
+- All visible text: "shop" → "site" throughout templates
+- Helper function: `getWebshopTypeInfo()` → `getSiteTypeInfo()`
+
+### MCP tools
+
+The `shop_id` parameter name in MCP tool input schemas gets `site_id` added as an accepted alias. Both work — the handler resolves via `db.getSiteByAnyId()`. Existing MCP clients using `shop_id` are unaffected.
+
+---
+
+## Phase 2: Site types + custom sitemap
+
+### New columns on `sites` table
+
+| Column | Type | Default | Purpose |
+|---|---|---|---|
+| `site_platform` | TEXT NOT NULL | `'generic'` | Platform subtype: `shopify`, `woocommerce`, `shoprenter`, `generic` |
+| `custom_sitemap_url` | TEXT | NULL | User-provided sitemap URL override |
+
+The existing `site_type` column (renamed from `webshop_type` in phase 1) stores: `webshop` or `website`.
+
+Existing rows: platform detection values (`shopify`, `woocommerce`, `shoprenter`) move from `site_type` to `site_platform`. The `site_type` for those rows becomes `webshop`. New generic sites get `site_type = 'website'`, `site_platform = 'generic'`.
+
+### Sitemap resolution order
+
+In `SitemapParser`, the new `resolveSitemapUrl()` method:
+
+1. If `custom_sitemap_url` is set on the site → use it directly, no detection.
+2. If `site_type = 'webshop'` → try platform-specific default (currently `/sitemap.xml` for all platforms).
+3. Auto-detect sequence:
+   a. Fetch `/robots.txt`, parse `Sitemap:` directives.
+   b. Try `/sitemap.xml`.
+   c. Try `/sitemap_index.xml`.
+4. Return `null` if nothing found (phase 3 link discovery handles this).
+
+### API changes
+
+- `POST /api/jobs` (create job): new optional fields `site_type` (default: auto-detect), `site_platform`, `custom_sitemap_url`. The job handler creates the site record with these values. The legacy `POST /api/jobs` still works — new fields are optional.
+- `PATCH /api/sites/:id/settings`: new endpoint to update `site_type`, `site_platform`, `custom_sitemap_url`, `max_crawl_depth`, `max_pages` after creation. Legacy alias: `PATCH /api/shops/:id/settings`.
+- `GET /api/sites/:id` response gains: `site_type`, `site_platform`, `custom_sitemap_url`.
+
+### Platform auto-detection
+
+The existing hostname-based detection in `SitemapParser.getSitemapUrl()` is extracted into a dedicated `detectPlatform(url)` function:
+
+- Hostname contains `shoprenter` or `.sr.hu` → `{ site_type: 'webshop', site_platform: 'shoprenter' }`
+- Hostname contains `myshopify.com` or `.shopify.` → `{ site_type: 'webshop', site_platform: 'shopify' }`
+- HTML contains WooCommerce signatures (meta generator, `wc-` classes) → `{ site_type: 'webshop', site_platform: 'woocommerce' }`
+- Otherwise → `{ site_type: 'website', site_platform: 'generic' }`
+
+This runs during job creation and pre-fills the site record. The user can override via the modal or the settings endpoint.
+
+### "Add site" modal
+
+Updated modal fields:
+
+1. **Site URL** (required) — text input
+2. **Site Type** (required) — dropdown: "Webshop" / "Website (generic)". Pre-selected by auto-detect on URL blur.
+3. **Sitemap URL** (optional) — text input, pre-filled by auto-detect. Clearable. Checkbox: "No sitemap — discover pages by crawling".
+4. **Custom ID** (optional) — UUID text input, same as today.
+
+Auto-detect fires on URL input blur: lightweight GET to the entered URL to check platform + sitemap. Shows spinner and status text ("Sitemap found at /sitemap.xml" or "No sitemap detected").
+
+---
+
+## Phase 3: Expanded categories + link discovery
+
+### Content categories
+
+13 categories total, all available to all site types:
+
+| Category | URL keyword examples |
+|---|---|
+| `shipping` | shipping, delivery, szallitas, postage, courier, fulfillment, szallitasi |
+| `contacts` | contact, kapcsolat, impresszum, email, phone, address, elerhetoseg |
+| `terms` | terms, privacy, aszf, gdpr, legal, cookie, adatvedelmi, refund |
+| `faq` | faq, gyik, help, support, knowledge, kerdesek, gyakori |
+| `services` | services, szolgaltatas, treatments, kezeles, offerings, megoldasok, solutions |
+| `about` | about, rolunk, history, our-story, tortenet, ars-poetica |
+| `team` | team, csapat, munkatars, our-team, doctors, staff, orvos, specialist |
+| `blog` | blog, articles, hirek, cikkek, posts, news, magazin |
+| `pricing` | pricing, prices, araink, arajanlat, plans, packages, fees, dijszabas |
+| `testimonials` | testimonials, reviews, velemenyek, mondtak, kamera-elott, feedback, ertekelesek |
+| `gallery` | gallery, kepek, photos, pictures, portfolio, esetbemutatas, case-stud, album |
+| `landing` | akcio, promo, campaign, landing, offer, ajanlat, special, deal |
+| `other` | *(catch-all: any URL not matching the above)* |
+
+The `content_type` CHECK constraint in the DB is updated to allow all 13 values.
+
+The keyword matching in `SitemapParser.filterUrlsByType()` is extended with 8 new arrays. The matching logic stays the same: lowercase URL path substring match, first match wins, unmatched URLs get `other`.
+
+### Link discovery engine
+
+New file: `src/scraper/LinkDiscovery.ts`
+
+```typescript
+class LinkDiscovery {
+  async discoverLinks(
+    baseUrl: string,
+    maxDepth: number,    // default 3
+    maxPages: number     // default 200
+  ): Promise<DiscoveredUrl[]>
+}
+
+interface DiscoveredUrl {
+  url: string;
+  depth: number;        // how many clicks from homepage
+  foundOn: string;      // parent page URL
+}
+```
+
+Algorithm:
+1. Fetch homepage, extract all internal `<a href>` links (same domain).
+2. Normalize URLs: strip fragments, collapse trailing slashes, remove common tracking params (`utm_*`, `fbclid`), lowercase path.
+3. BFS queue: process each discovered URL up to `maxDepth` levels.
+4. Skip non-content URLs: static assets (`.js`, `.css`, `.png`, `.jpg`, `.gif`, `.pdf`, `.svg`, `.woff`), pagination (`?page=`, `/page/N`), WordPress internals (`/wp-admin/`, `/wp-json/`, `/wp-content/`).
+5. Stop when `maxPages` unique URLs are collected or the queue is empty.
+6. Return deduplicated list with depth metadata.
+
+### Integration in scrape flow
+
+Updated flow in `WebScraper.scrape()`:
+
+```
+1. Resolve sitemap URL (phase 2 logic)
+2. If sitemap found:
+   a. Parse sitemap → sitemap URLs
+   b. Run link discovery → discovered URLs (supplement)
+   c. Merge: sitemap URLs ∪ discovered URLs
+3. If no sitemap:
+   a. Run link discovery → discovered URLs (primary source)
+4. Add custom include URLs (from custom_urls table, enabled=true)
+5. Remove excluded URLs (from url_exclusions table, pattern match)
+6. Classify all remaining URLs by content category keywords
+7. Scrape each URL
+```
+
+### URL exclusion table
+
+```sql
+CREATE TABLE url_exclusions (
+  id TEXT PRIMARY KEY,
+  site_id TEXT NOT NULL,
+  pattern TEXT NOT NULL,
+  created_at TEXT NOT NULL,
+  FOREIGN KEY (site_id) REFERENCES sites(id) ON DELETE CASCADE,
+  UNIQUE(site_id, pattern)
+);
+```
+
+Patterns are matched against the full URL path. Supported formats:
+- Exact URL: `https://example.com/admin/login`
+- Glob: `*/admin/*`, `*/tag/*`, `*/page/*`, `*.pdf`
+
+Matching function: simple glob-to-regex conversion (replace `*` with `.*`).
+
+### API endpoints for exclusions
+
+- `POST /api/sites/:id/url-exclusions` — add pattern
+- `GET /api/sites/:id/url-exclusions` — list patterns
+- `DELETE /api/sites/:id/url-exclusions/:exclusionId` — remove pattern
+
+### New columns on `sites` table
+
+| Column | Type | Default |
+|---|---|---|
+| `max_crawl_depth` | INTEGER | 3 |
+| `max_pages` | INTEGER | 200 |
+
+---
+
+## Phase 4: Web UI enhancements
+
+### Custom URLs panel (on site detail page)
+
+New section between content metadata and scrape history. Two sub-panels:
+
+**Include URLs**: Lists entries from `custom_urls` table. Each row shows URL (truncated), category dropdown, enabled toggle, delete button. "+ Add URL" button opens inline form with URL input + category dropdown.
+
+**Exclude Patterns**: Lists entries from `url_exclusions` table. Each row shows pattern + delete button. "+ Add Pattern" opens inline form with pattern input and hint text ("Use * as wildcard, e.g. */admin/*").
+
+### Crawl settings panel (on site detail page)
+
+Shows current sitemap URL (with edit button and re-detect button), max crawl depth input (1-5), max pages input (1-1000). Changes saved via `PATCH /api/sites/:id/settings`.
+
+### Content metadata display
+
+Updated to show all 13 categories with page counts. Categories with 0 pages are shown grayed out. Current 4-category layout expands to a responsive grid.
+
+### Sites list page
+
+- Column "Type" shows site_type badge ("Webshop" / "Website")
+- Platform shown as subtitle when applicable (e.g. "Webshop · Shopify")
+- "Add New" button opens the reworked modal from phase 2
+
+---
+
+## Subagent parallelization strategy
+
+- **Phase 1**: Mostly sequential (rename is one big coordinated change). Can parallelize: DB migration + TS type renames vs. web UI renames.
+- **Phase 2**: Sequential (depends on phase 1 types). Auto-detect logic + modal can be parallel.
+- **Phase 3**: High parallelism potential:
+  - Agent A: Link discovery engine (`LinkDiscovery.ts`)
+  - Agent B: Content category keyword expansion (`SitemapParser.ts`)
+  - Agent C: URL exclusion table + API endpoints
+  - Agent D: Scrape flow integration
+- **Phase 4**: High parallelism:
+  - Agent A: Custom URLs panel
+  - Agent B: Crawl settings panel
+  - Agent C: Content metadata display + sites list updates
+
+---
+
+## Out of scope
+
+- Rate limiting (doesn't exist today, not adding it)
+- Multi-language detection or language-specific crawl strategies
+- Authentication-protected page scraping
+- JavaScript-rendered SPA crawling (Cheerio is HTML-only)
+- Custom category creation via UI (categories are code-defined)
+- RSS/Atom feed discovery as a sitemap alternative