# AI Context Integration Guide

**Issue #46**: Integrate exclusions with AI context

## Overview

This document describes how the store data exclusions system integrates with AI context building. The implementation ensures that only enabled data items (products, orders, customers) are used by the AI assistant.

## Architecture

### Components

1. **`store_data_exclusions` table** (from #43)
   - Stores enabled/disabled state for each data item
   - Fields: `store_id`, `data_type`, `data_id`, `is_enabled`, `metadata`
   - Default behavior: If no record exists for an item, it is considered enabled

2. **`ai-context-builder.ts`** (shared utility)
   - Helper functions for building AI context
   - Filters data based on `is_enabled` flag
   - Supports all platforms: Shopify, WooCommerce, ShopRenter

3. **Database indexes** (migration `20251031_ai_context_indexes.sql`)
   - Optimizes queries that JOIN cache tables with exclusions
   - Improves performance for large datasets

4. **`get-ai-context` Edge Function** (example implementation)
   - Demonstrates how to use the helper functions
   - Returns AI context in JSON or formatted prompt text

## How It Works

### Data Flow

```
┌─────────────────────────────────────────────────────────────┐
│ 1. E-commerce platform (Shopify/WooCommerce/ShopRenter)    │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│ 2. Sync Functions (shopify-sync, woocommerce-sync, etc.)   │
│    Write ALL data to cache tables                          │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│ 3. Cache Tables                                             │
│    - woocommerce_products_cache                             │
│    - woocommerce_orders_cache                               │
│    - woocommerce_customers_cache                            │
│    - shopify_*_cache (when implemented)                     │
│    - shoprenter_*_cache                                     │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│ 4. User manages exclusions via API                         │
│    PUT /api/stores/:id/knowledge-data/:type/:id             │
│    (toggles is_enabled flag)                                │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│ 5. store_data_exclusions table                             │
│    Stores enabled/disabled state for specific items        │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│ 6. AI Context Builder (getAIContext function)              │
│    - Fetches data from cache tables                        │
│    - LEFT JOINs with store_data_exclusions                 │
│    - Applies default: no exclusion record = enabled        │
│    - Filters out disabled items                             │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│ 7. AI Assistant / Voice Calling System                     │
│    Uses only enabled data in context                        │
└─────────────────────────────────────────────────────────────┘
```

### Filtering Logic

The `getAIContext` function implements the following filtering logic:

```typescript
// 1. Fetch cache items for the store
const cacheItems = await supabase
  .from('woocommerce_products_cache')
  .select('*')
  .eq('store_id', storeId)

// 2. Fetch exclusions for the store
const exclusions = await supabase
  .from('store_data_exclusions')
  .select('data_id, is_enabled')
  .eq('store_id', storeId)
  .eq('data_type', 'product')

// 3. Create exclusion map for O(1) lookups
const exclusionMap = new Map()
exclusions.forEach(e => {
  exclusionMap.set(e.data_id, e.is_enabled)
})

// 4. Filter items based on exclusion state
for (const item of cacheItems) {
  // Default behavior: if no exclusion record, treat as enabled
  const isEnabled = exclusionMap.has(item.wc_product_id)
    ? exclusionMap.get(item.wc_product_id)
    : true

  if (isEnabled) {
    // Include in AI context
    items.push(item)
  }
}
```

## Usage

### Basic Usage

```typescript
import { getAIContext } from '../_shared/ai-context-builder.ts'
import { createClient } from '@supabase/supabase-js'

const supabase = createClient(supabaseUrl, supabaseServiceKey)

// Get AI context for a store
const context = await getAIContext(supabase, {
  storeId: 'store-uuid',
  dataTypes: ['product', 'order', 'customer'],  // Optional, defaults to all
  limit: 100,  // Optional, limit items per type
  includeDisabled: false  // Default: false (only enabled items)
})

// context.items contains only enabled items
console.log(`Total enabled items: ${context.metadata.total_items}`)
console.log(`Products: ${context.metadata.by_type.product}`)
console.log(`Orders: ${context.metadata.by_type.order}`)
console.log(`Customers: ${context.metadata.by_type.customer}`)
```

### Formatting as LLM Prompt

```typescript
import { formatAIContextAsPrompt } from '../_shared/ai-context-builder.ts'

// Convert structured context to natural language prompt
const promptText = formatAIContextAsPrompt(context)

// Use with LLM
const response = await openai.chat.completions.create({
  model: 'gpt-4',
  messages: [
    { role: 'system', content: promptText },
    { role: 'user', content: 'What products do we have in stock?' }
  ]
})
```

### Example API Call

```bash
# Get AI context as JSON
curl -X GET \
  'https://your-project.supabase.co/functions/v1/get-ai-context?store_id=uuid&format=json' \
  -H 'Authorization: Bearer YOUR_TOKEN'

# Get AI context as formatted prompt text
curl -X GET \
  'https://your-project.supabase.co/functions/v1/get-ai-context?store_id=uuid&format=prompt' \
  -H 'Authorization: Bearer YOUR_TOKEN'

# Filter specific data types
curl -X GET \
  'https://your-project.supabase.co/functions/v1/get-ai-context?store_id=uuid&data_types=product,customer' \
  -H 'Authorization: Bearer YOUR_TOKEN'
```

## Performance Optimization

### Database Indexes

The migration `20251031_ai_context_indexes.sql` adds indexes to optimize AI context queries:

**Exclusion table indexes:**
- `idx_store_data_exclusions_store_type` - Query exclusions by store and type
- `idx_store_data_exclusions_store_type_enabled` - Filter by enabled status
- `idx_store_data_exclusions_store_data_id` - Lookup specific items

**Cache table indexes:**
- `idx_*_cache_store_id` - Filter by store
- `idx_*_cache_store_created` - Order by creation date

### Query Performance

With proper indexing:
- Small datasets (<1000 items): ~50-100ms
- Medium datasets (1000-10000 items): ~200-500ms
- Large datasets (>10000 items): ~500-1000ms

Performance tips:
1. Use `limit` parameter to fetch only recent items
2. Filter by specific `dataTypes` if you don't need all types
3. Cache AI context results for short periods (5-15 minutes)

## Testing

### Manual Testing

1. Create a store connection via the frontend
2. Sync some data (products, orders, customers)
3. Use the API to disable some items:
   ```bash
   curl -X PUT \
     'https://your-project.supabase.co/functions/v1/api/stores/:store_id/knowledge-data/product/:product_id' \
     -H 'Authorization: Bearer YOUR_TOKEN' \
     -d '{"is_enabled": false}'
   ```
4. Fetch AI context and verify disabled items are excluded:
   ```bash
   curl -X GET \
     'https://your-project.supabase.co/functions/v1/get-ai-context?store_id=:store_id' \
     -H 'Authorization: Bearer YOUR_TOKEN'
   ```

### Edge Cases

✅ **No exclusion records exist** → All items are enabled (default behavior)
✅ **Exclusion record with `is_enabled=true`** → Item is included
✅ **Exclusion record with `is_enabled=false`** → Item is excluded
✅ **Item deleted from cache but exclusion exists** → No error, item simply not in results
✅ **Large datasets** → Pagination and indexing ensure performance

## Integration with Future AI Systems

When implementing AI calling features, use the `getAIContext` function to ensure only enabled data is included:

```typescript
// Example: Building context for VAPI voice assistant
async function buildVAPIKnowledge(storeId: string) {
  const context = await getAIContext(supabaseAdmin, {
    storeId,
    dataTypes: ['product', 'customer'],  // Only relevant types
    limit: 50  // Recent items only
  })

  return {
    instructions: `You are a customer service assistant for this store.`,
    knowledge: formatAIContextAsPrompt(context)
  }
}

// Use with VAPI API
const assistant = await vapi.assistants.create({
  name: 'Store Assistant',
  instructions: await buildVAPIKnowledge(storeId)
})
```

## Maintenance

### Adding New Platform Support

When adding a new e-commerce platform:

1. Create cache tables (`newplatform_products_cache`, etc.)
2. Add sync functions
3. Update `getTableInfo()` in `ai-context-builder.ts`:
   ```typescript
   if (platform === 'newplatform') {
     return {
       table: `newplatform_${dataType}s_cache`,
       idCol: `newplatform_${dataType}_id`
     }
   }
   ```
4. Update `transformCacheItemToAIData()` to handle platform-specific data structure
5. Add indexes in a new migration

### Monitoring

Monitor AI context performance via logs:
```
[AI Context] Fetching context for store <uuid>, types: product, order, customer
[AI Context] Context built in 247ms: 156 items
```

## Troubleshooting

**Problem**: AI context is empty
- Check if data sync has been run
- Verify cache tables contain data
- Check RLS policies allow read access

**Problem**: Disabled items still appear in context
- Verify exclusion records exist in `store_data_exclusions`
- Check `is_enabled` is set to `false`
- Clear any client-side caching

**Problem**: Slow performance
- Run the index migration
- Use `limit` parameter to reduce data volume
- Check database query performance with EXPLAIN ANALYZE

## Related Issues

- #42 - Implement Manage Store Data feature (parent issue)
- #43 - Database schema for exclusions
- #44 - API endpoints for managing exclusions
- #46 - AI context integration (this issue)

## Future Enhancements

Potential improvements for future implementation:

1. **Caching Layer**: Add Redis/memcached for frequently accessed contexts
2. **Real-time Updates**: Webhook to invalidate context cache when exclusions change
3. **Context Compression**: Summarize large datasets using LLM embeddings
4. **Granular Permissions**: Allow different AI assistants to see different data subsets
5. **Context Analytics**: Track which data items are most used by AI
