|
@@ -0,0 +1,326 @@
|
|
|
|
|
+# AI Context Integration Guide
|
|
|
|
|
+
|
|
|
|
|
+**Issue #46**: Integrate exclusions with AI context
|
|
|
|
|
+
|
|
|
|
|
+## Overview
|
|
|
|
|
+
|
|
|
|
|
+This document describes how the store data exclusions system integrates with AI context building. The implementation ensures that only enabled data items (products, orders, customers) are used by the AI assistant.
|
|
|
|
|
+
|
|
|
|
|
+## Architecture
|
|
|
|
|
+
|
|
|
|
|
+### Components
|
|
|
|
|
+
|
|
|
|
|
+1. **`store_data_exclusions` table** (from #43)
|
|
|
|
|
+ - Stores enabled/disabled state for each data item
|
|
|
|
|
+ - Fields: `store_id`, `data_type`, `data_id`, `is_enabled`, `metadata`
|
|
|
|
|
+ - Default behavior: If no record exists for an item, it is considered enabled
|
|
|
|
|
+
|
|
|
|
|
+2. **`ai-context-builder.ts`** (shared utility)
|
|
|
|
|
+ - Helper functions for building AI context
|
|
|
|
|
+ - Filters data based on `is_enabled` flag
|
|
|
|
|
+ - Supports all platforms: Shopify, WooCommerce, ShopRenter
|
|
|
|
|
+
|
|
|
|
|
+3. **Database indexes** (migration `20251031_ai_context_indexes.sql`)
|
|
|
|
|
+ - Optimizes queries that JOIN cache tables with exclusions
|
|
|
|
|
+ - Improves performance for large datasets
|
|
|
|
|
+
|
|
|
|
|
+4. **`get-ai-context` Edge Function** (example implementation)
|
|
|
|
|
+ - Demonstrates how to use the helper functions
|
|
|
|
|
+ - Returns AI context in JSON or formatted prompt text
|
|
|
|
|
+
|
|
|
|
|
+## How It Works
|
|
|
|
|
+
|
|
|
|
|
+### Data Flow
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+โ 1. E-commerce platform (Shopify/WooCommerce/ShopRenter) โ
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+ โ
|
|
|
|
|
+ โผ
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+โ 2. Sync Functions (shopify-sync, woocommerce-sync, etc.) โ
|
|
|
|
|
+โ Write ALL data to cache tables โ
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+ โ
|
|
|
|
|
+ โผ
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+โ 3. Cache Tables โ
|
|
|
|
|
+โ - woocommerce_products_cache โ
|
|
|
|
|
+โ - woocommerce_orders_cache โ
|
|
|
|
|
+โ - woocommerce_customers_cache โ
|
|
|
|
|
+โ - shopify_*_cache (when implemented) โ
|
|
|
|
|
+โ - shoprenter_*_cache โ
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+ โ
|
|
|
|
|
+ โผ
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+โ 4. User manages exclusions via API โ
|
|
|
|
|
+โ PUT /api/stores/:id/knowledge-data/:type/:id โ
|
|
|
|
|
+โ (toggles is_enabled flag) โ
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+ โ
|
|
|
|
|
+ โผ
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+โ 5. store_data_exclusions table โ
|
|
|
|
|
+โ Stores enabled/disabled state for specific items โ
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+ โ
|
|
|
|
|
+ โผ
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+โ 6. AI Context Builder (getAIContext function) โ
|
|
|
|
|
+โ - Fetches data from cache tables โ
|
|
|
|
|
+โ - LEFT JOINs with store_data_exclusions โ
|
|
|
|
|
+โ - Applies default: no exclusion record = enabled โ
|
|
|
|
|
+โ - Filters out disabled items โ
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+ โ
|
|
|
|
|
+ โผ
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+โ 7. AI Assistant / Voice Calling System โ
|
|
|
|
|
+โ Uses only enabled data in context โ
|
|
|
|
|
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Filtering Logic
|
|
|
|
|
+
|
|
|
|
|
+The `getAIContext` function implements the following filtering logic:
|
|
|
|
|
+
|
|
|
|
|
+```typescript
|
|
|
|
|
+// 1. Fetch cache items for the store
|
|
|
|
|
+const cacheItems = await supabase
|
|
|
|
|
+ .from('woocommerce_products_cache')
|
|
|
|
|
+ .select('*')
|
|
|
|
|
+ .eq('store_id', storeId)
|
|
|
|
|
+
|
|
|
|
|
+// 2. Fetch exclusions for the store
|
|
|
|
|
+const exclusions = await supabase
|
|
|
|
|
+ .from('store_data_exclusions')
|
|
|
|
|
+ .select('data_id, is_enabled')
|
|
|
|
|
+ .eq('store_id', storeId)
|
|
|
|
|
+ .eq('data_type', 'product')
|
|
|
|
|
+
|
|
|
|
|
+// 3. Create exclusion map for O(1) lookups
|
|
|
|
|
+const exclusionMap = new Map()
|
|
|
|
|
+exclusions.forEach(e => {
|
|
|
|
|
+ exclusionMap.set(e.data_id, e.is_enabled)
|
|
|
|
|
+})
|
|
|
|
|
+
|
|
|
|
|
+// 4. Filter items based on exclusion state
|
|
|
|
|
+for (const item of cacheItems) {
|
|
|
|
|
+ // Default behavior: if no exclusion record, treat as enabled
|
|
|
|
|
+ const isEnabled = exclusionMap.has(item.wc_product_id)
|
|
|
|
|
+ ? exclusionMap.get(item.wc_product_id)
|
|
|
|
|
+ : true
|
|
|
|
|
+
|
|
|
|
|
+ if (isEnabled) {
|
|
|
|
|
+ // Include in AI context
|
|
|
|
|
+ items.push(item)
|
|
|
|
|
+ }
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+## Usage
|
|
|
|
|
+
|
|
|
|
|
+### Basic Usage
|
|
|
|
|
+
|
|
|
|
|
+```typescript
|
|
|
|
|
+import { getAIContext } from '../_shared/ai-context-builder.ts'
|
|
|
|
|
+import { createClient } from '@supabase/supabase-js'
|
|
|
|
|
+
|
|
|
|
|
+const supabase = createClient(supabaseUrl, supabaseServiceKey)
|
|
|
|
|
+
|
|
|
|
|
+// Get AI context for a store
|
|
|
|
|
+const context = await getAIContext(supabase, {
|
|
|
|
|
+ storeId: 'store-uuid',
|
|
|
|
|
+ dataTypes: ['product', 'order', 'customer'], // Optional, defaults to all
|
|
|
|
|
+ limit: 100, // Optional, limit items per type
|
|
|
|
|
+ includeDisabled: false // Default: false (only enabled items)
|
|
|
|
|
+})
|
|
|
|
|
+
|
|
|
|
|
+// context.items contains only enabled items
|
|
|
|
|
+console.log(`Total enabled items: ${context.metadata.total_items}`)
|
|
|
|
|
+console.log(`Products: ${context.metadata.by_type.product}`)
|
|
|
|
|
+console.log(`Orders: ${context.metadata.by_type.order}`)
|
|
|
|
|
+console.log(`Customers: ${context.metadata.by_type.customer}`)
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Formatting as LLM Prompt
|
|
|
|
|
+
|
|
|
|
|
+```typescript
|
|
|
|
|
+import { formatAIContextAsPrompt } from '../_shared/ai-context-builder.ts'
|
|
|
|
|
+
|
|
|
|
|
+// Convert structured context to natural language prompt
|
|
|
|
|
+const promptText = formatAIContextAsPrompt(context)
|
|
|
|
|
+
|
|
|
|
|
+// Use with LLM
|
|
|
|
|
+const response = await openai.chat.completions.create({
|
|
|
|
|
+ model: 'gpt-4',
|
|
|
|
|
+ messages: [
|
|
|
|
|
+ { role: 'system', content: promptText },
|
|
|
|
|
+ { role: 'user', content: 'What products do we have in stock?' }
|
|
|
|
|
+ ]
|
|
|
|
|
+})
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+### Example API Call
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+# Get AI context as JSON
|
|
|
|
|
+curl -X GET \
|
|
|
|
|
+ 'https://your-project.supabase.co/functions/v1/get-ai-context?store_id=uuid&format=json' \
|
|
|
|
|
+ -H 'Authorization: Bearer YOUR_TOKEN'
|
|
|
|
|
+
|
|
|
|
|
+# Get AI context as formatted prompt text
|
|
|
|
|
+curl -X GET \
|
|
|
|
|
+ 'https://your-project.supabase.co/functions/v1/get-ai-context?store_id=uuid&format=prompt' \
|
|
|
|
|
+ -H 'Authorization: Bearer YOUR_TOKEN'
|
|
|
|
|
+
|
|
|
|
|
+# Filter specific data types
|
|
|
|
|
+curl -X GET \
|
|
|
|
|
+ 'https://your-project.supabase.co/functions/v1/get-ai-context?store_id=uuid&data_types=product,customer' \
|
|
|
|
|
+ -H 'Authorization: Bearer YOUR_TOKEN'
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+## Performance Optimization
|
|
|
|
|
+
|
|
|
|
|
+### Database Indexes
|
|
|
|
|
+
|
|
|
|
|
+The migration `20251031_ai_context_indexes.sql` adds indexes to optimize AI context queries:
|
|
|
|
|
+
|
|
|
|
|
+**Exclusion table indexes:**
|
|
|
|
|
+- `idx_store_data_exclusions_store_type` - Query exclusions by store and type
|
|
|
|
|
+- `idx_store_data_exclusions_store_type_enabled` - Filter by enabled status
|
|
|
|
|
+- `idx_store_data_exclusions_store_data_id` - Lookup specific items
|
|
|
|
|
+
|
|
|
|
|
+**Cache table indexes:**
|
|
|
|
|
+- `idx_*_cache_store_id` - Filter by store
|
|
|
|
|
+- `idx_*_cache_store_created` - Order by creation date
|
|
|
|
|
+
|
|
|
|
|
+### Query Performance
|
|
|
|
|
+
|
|
|
|
|
+With proper indexing:
|
|
|
|
|
+- Small datasets (<1000 items): ~50-100ms
|
|
|
|
|
+- Medium datasets (1000-10000 items): ~200-500ms
|
|
|
|
|
+- Large datasets (>10000 items): ~500-1000ms
|
|
|
|
|
+
|
|
|
|
|
+Performance tips:
|
|
|
|
|
+1. Use `limit` parameter to fetch only recent items
|
|
|
|
|
+2. Filter by specific `dataTypes` if you don't need all types
|
|
|
|
|
+3. Cache AI context results for short periods (5-15 minutes)
|
|
|
|
|
+
|
|
|
|
|
+## Testing
|
|
|
|
|
+
|
|
|
|
|
+### Manual Testing
|
|
|
|
|
+
|
|
|
|
|
+1. Create a store connection via the frontend
|
|
|
|
|
+2. Sync some data (products, orders, customers)
|
|
|
|
|
+3. Use the API to disable some items:
|
|
|
|
|
+ ```bash
|
|
|
|
|
+ curl -X PUT \
|
|
|
|
|
+ 'https://your-project.supabase.co/functions/v1/api/stores/:store_id/knowledge-data/product/:product_id' \
|
|
|
|
|
+ -H 'Authorization: Bearer YOUR_TOKEN' \
|
|
|
|
|
+ -d '{"is_enabled": false}'
|
|
|
|
|
+ ```
|
|
|
|
|
+4. Fetch AI context and verify disabled items are excluded:
|
|
|
|
|
+ ```bash
|
|
|
|
|
+ curl -X GET \
|
|
|
|
|
+ 'https://your-project.supabase.co/functions/v1/get-ai-context?store_id=:store_id' \
|
|
|
|
|
+ -H 'Authorization: Bearer YOUR_TOKEN'
|
|
|
|
|
+ ```
|
|
|
|
|
+
|
|
|
|
|
+### Edge Cases
|
|
|
|
|
+
|
|
|
|
|
+โ
**No exclusion records exist** โ All items are enabled (default behavior)
|
|
|
|
|
+โ
**Exclusion record with `is_enabled=true`** โ Item is included
|
|
|
|
|
+โ
**Exclusion record with `is_enabled=false`** โ Item is excluded
|
|
|
|
|
+โ
**Item deleted from cache but exclusion exists** โ No error, item simply not in results
|
|
|
|
|
+โ
**Large datasets** โ Pagination and indexing ensure performance
|
|
|
|
|
+
|
|
|
|
|
+## Integration with Future AI Systems
|
|
|
|
|
+
|
|
|
|
|
+When implementing AI calling features, use the `getAIContext` function to ensure only enabled data is included:
|
|
|
|
|
+
|
|
|
|
|
+```typescript
|
|
|
|
|
+// Example: Building context for VAPI voice assistant
|
|
|
|
|
+async function buildVAPIKnowledge(storeId: string) {
|
|
|
|
|
+ const context = await getAIContext(supabaseAdmin, {
|
|
|
|
|
+ storeId,
|
|
|
|
|
+ dataTypes: ['product', 'customer'], // Only relevant types
|
|
|
|
|
+ limit: 50 // Recent items only
|
|
|
|
|
+ })
|
|
|
|
|
+
|
|
|
|
|
+ return {
|
|
|
|
|
+ instructions: `You are a customer service assistant for this store.`,
|
|
|
|
|
+ knowledge: formatAIContextAsPrompt(context)
|
|
|
|
|
+ }
|
|
|
|
|
+}
|
|
|
|
|
+
|
|
|
|
|
+// Use with VAPI API
|
|
|
|
|
+const assistant = await vapi.assistants.create({
|
|
|
|
|
+ name: 'Store Assistant',
|
|
|
|
|
+ instructions: await buildVAPIKnowledge(storeId)
|
|
|
|
|
+})
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+## Maintenance
|
|
|
|
|
+
|
|
|
|
|
+### Adding New Platform Support
|
|
|
|
|
+
|
|
|
|
|
+When adding a new e-commerce platform:
|
|
|
|
|
+
|
|
|
|
|
+1. Create cache tables (`newplatform_products_cache`, etc.)
|
|
|
|
|
+2. Add sync functions
|
|
|
|
|
+3. Update `getTableInfo()` in `ai-context-builder.ts`:
|
|
|
|
|
+ ```typescript
|
|
|
|
|
+ if (platform === 'newplatform') {
|
|
|
|
|
+ return {
|
|
|
|
|
+ table: `newplatform_${dataType}s_cache`,
|
|
|
|
|
+ idCol: `newplatform_${dataType}_id`
|
|
|
|
|
+ }
|
|
|
|
|
+ }
|
|
|
|
|
+ ```
|
|
|
|
|
+4. Update `transformCacheItemToAIData()` to handle platform-specific data structure
|
|
|
|
|
+5. Add indexes in a new migration
|
|
|
|
|
+
|
|
|
|
|
+### Monitoring
|
|
|
|
|
+
|
|
|
|
|
+Monitor AI context performance via logs:
|
|
|
|
|
+```
|
|
|
|
|
+[AI Context] Fetching context for store <uuid>, types: product, order, customer
|
|
|
|
|
+[AI Context] Context built in 247ms: 156 items
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+## Troubleshooting
|
|
|
|
|
+
|
|
|
|
|
+**Problem**: AI context is empty
|
|
|
|
|
+- Check if data sync has been run
|
|
|
|
|
+- Verify cache tables contain data
|
|
|
|
|
+- Check RLS policies allow read access
|
|
|
|
|
+
|
|
|
|
|
+**Problem**: Disabled items still appear in context
|
|
|
|
|
+- Verify exclusion records exist in `store_data_exclusions`
|
|
|
|
|
+- Check `is_enabled` is set to `false`
|
|
|
|
|
+- Clear any client-side caching
|
|
|
|
|
+
|
|
|
|
|
+**Problem**: Slow performance
|
|
|
|
|
+- Run the index migration
|
|
|
|
|
+- Use `limit` parameter to reduce data volume
|
|
|
|
|
+- Check database query performance with EXPLAIN ANALYZE
|
|
|
|
|
+
|
|
|
|
|
+## Related Issues
|
|
|
|
|
+
|
|
|
|
|
+- #42 - Implement Manage Store Data feature (parent issue)
|
|
|
|
|
+- #43 - Database schema for exclusions
|
|
|
|
|
+- #44 - API endpoints for managing exclusions
|
|
|
|
|
+- #46 - AI context integration (this issue)
|
|
|
|
|
+
|
|
|
|
|
+## Future Enhancements
|
|
|
|
|
+
|
|
|
|
|
+Potential improvements for future implementation:
|
|
|
|
|
+
|
|
|
|
|
+1. **Caching Layer**: Add Redis/memcached for frequently accessed contexts
|
|
|
|
|
+2. **Real-time Updates**: Webhook to invalidate context cache when exclusions change
|
|
|
|
|
+3. **Context Compression**: Summarize large datasets using LLM embeddings
|
|
|
|
|
+4. **Granular Permissions**: Allow different AI assistants to see different data subsets
|
|
|
|
|
+5. **Context Analytics**: Track which data items are most used by AI
|