# Phase 10 — LLM Integration & Smart Features **Goal**: Wire up the `ILlmProvider` interface (defined in Phase 5) to real LLM backends. Enable all the smart features that have had placeholder endpoints throughout Phases 5-9. Add new natural language interaction capabilities for both medicine and food domains. **Depends on**: All previous phases (feature endpoints already exist) --- ## Deliverables 1. Concrete `ILlmProvider` implementations (OpenAI, Anthropic, Ollama) 2. Config-driven provider selection 3. LLM request/response logging and cost tracking 4. Rate limiting and budget controls 5. Smart features fully wired: - Product recognition (photo → nutrition) - Receipt parsing (photo → store + items + prices) - Recipe import (text/URL → structured recipe) - Natural language pantry entry - LLM-powered meal plan suggestions 6. Prompt templates and versioning --- ## Architecture ### Provider Selection ``` Environment variable: LLM_PROVIDER_TYPE=openai|anthropic|ollama|noop LlmModule registers the provider dynamically: @Module({}) export class LlmModule { static forRoot(): DynamicModule { return { providers: [{ provide: LLM_PROVIDER, useFactory: (config: ConfigService) => { switch (config.get('LLM_PROVIDER_TYPE')) { case 'openai': return new OpenAiProvider(config); case 'anthropic': return new AnthropicProvider(config); case 'ollama': return new OllamaProvider(config); default: return new NoOpLlmProvider(); } }, inject: [ConfigService], }], exports: [LLM_PROVIDER], }; } } ``` ### Provider Implementations Each provider implements `ILlmProvider` and handles: - API authentication (keys from env) - Model selection (configurable per provider) - Request/response mapping to/from vendor format - Error handling and retries (exponential backoff) - Timeout management ```typescript // packages/api/src/modules/llm/providers/ ├── noop.provider.ts # Returns null for everything (already exists from Phase 5) ├── openai.provider.ts # GPT-4o / GPT-4o-mini ├── anthropic.provider.ts # Claude 3.5 Sonnet / Haiku └── ollama.provider.ts # Local models (Llama 3, Mistral, etc.) ``` ### Environment Configuration ```env # Provider selection LLM_PROVIDER_TYPE=openai # OpenAI OPENAI_API_KEY=sk-... OPENAI_MODEL=gpt-4o OPENAI_VISION_MODEL=gpt-4o # For image inputs # Anthropic ANTHROPIC_API_KEY=sk-ant-... ANTHROPIC_MODEL=claude-sonnet-4-20250514 # Ollama OLLAMA_BASE_URL=http://ollama:11434 OLLAMA_MODEL=llama3 OLLAMA_VISION_MODEL=llava # Budget LLM_MONTHLY_BUDGET_USD=20.00 LLM_RATE_LIMIT_RPM=30 # Requests per minute per household ``` --- ## Tasks ### 10.1 — OpenAI Provider ```typescript class OpenAiProvider implements ILlmProvider { // Uses OpenAI Node.js SDK // Text endpoints: chat completions with JSON mode // Vision endpoints: chat completions with image_url or base64 content // Structured output: use function calling or response_format: json_schema } ``` - `extractNutrition`: Send product text/image → prompt asks for structured nutrition JSON - `parseRecipe`: Send recipe text → prompt extracts name, servings, ingredients[], steps[] - `parseRecipeFromUrl`: Fetch URL content first, then parse as text - `parseReceipt`: Send receipt image → prompt extracts store, date, line items with prices - `suggestMealPlan`: Send pantry summary + targets + preferences → get 7-day plan - `parseNaturalLanguage`: Send user text → extract intent + entities (add product, log purchase, etc.) ### 10.2 — Anthropic Provider ```typescript class AnthropicProvider implements ILlmProvider { // Uses Anthropic SDK // Similar structure to OpenAI but with Claude-specific API format // Vision: send image as base64 in messages // Structured output: use tool_use for JSON extraction } ``` ### 10.3 — Ollama Provider ```typescript class OllamaProvider implements ILlmProvider { // Uses Ollama REST API (http://host:11434/api/generate or /api/chat) // Text: standard chat endpoint // Vision: requires multimodal model (llava, bakllava) // Note: local models may be less accurate — adjust prompts for simpler output // No cost tracking needed (self-hosted) } ``` ### 10.4 — Prompt Templates Create versioned prompt templates in `packages/api/src/modules/llm/prompts/`: ```typescript // prompts/extract-nutrition.ts export const EXTRACT_NUTRITION_PROMPT = { version: '1.0', system: `You are a nutrition data extraction assistant. Given a food product description or image, extract nutritional information. Return ONLY valid JSON matching the schema below. If you cannot determine a value, use null. Be conservative with estimates.`, schema: { name: 'string', brand: 'string | null', servingSize: 'number', servingUnit: 'g | ml | oz | piece', nutrition: { calories: 'number', protein: 'number (grams)', carbs: 'number (grams)', fat: 'number (grams)', fiber: 'number | null', sugar: 'number | null', sodium: 'number | null (mg)', }, }, }; // prompts/parse-recipe.ts // prompts/parse-receipt.ts // prompts/suggest-meal-plan.ts // prompts/parse-natural-language.ts ``` - Each prompt has a `version` for tracking which prompt produced which results - Prompts are provider-agnostic (providers may wrap them differently) ### 10.5 — LLM Logging & Cost Tracking ```typescript // Schema export interface LlmLog { id: string; householdId: string; userId: string; provider: string; // 'openai' | 'anthropic' | 'ollama' model: string; feature: string; // 'extract_nutrition' | 'parse_recipe' | etc. promptVersion: string; inputTokens: number; outputTokens: number; totalTokens: number; costUsd: number; // Computed from token count + model pricing latencyMs: number; success: boolean; errorMessage?: string; createdAt: Date; } ``` - All provider calls are wrapped in a logging decorator/interceptor - Cost computed from known per-model pricing (configurable) - Monthly cost aggregation endpoint for household admins ### 10.6 — Rate Limiting & Budget Controls ```typescript class LlmBudgetGuard { /** * Before each LLM call: * 1. Check per-household rate limit (requests per minute) * 2. Check monthly budget: sum costUsd for current month vs LLM_MONTHLY_BUDGET_USD * 3. If exceeded, throw BudgetExceededException (HTTP 429) */ async canProceed(householdId: string): Promise; } ``` - Rate limiting: Redis-backed or in-memory (household-level RPM) - Budget: MongoDB aggregation on `LlmLog` collection ### 10.7 — Wire Up All Feature Endpoints Each of these endpoints already exists as a placeholder from earlier phases. Now they get real LLM calls: | Feature | Endpoint (existing) | Phase | LLM Method | | --------------------- | ----------------------------------- | ----- | ---------------------- | | Product recognition | `POST /products/smart-add` | 5 | `extractNutrition()` | | Recipe import (text) | `POST /recipes/import-text` | 6 | `parseRecipe()` | | Recipe import (URL) | `POST /recipes/import-url` | 6 | `parseRecipeFromUrl()` | | Receipt parsing | `POST /prices/parse-receipt` | 9 | `parseReceipt()` | | Meal plan suggestions | `POST /meal-plans/suggest-with-llm` | 8 | `suggestMealPlan()` | ### 10.8 — Natural Language Input (New Feature) New universal endpoint: ``` POST /api/v1/nlp/parse Body: { text: string } Response: { intent: string, action: StructuredAction, confidence: number } ``` Supported intents: - `add_pantry_item`: "I bought 2 lbs of chicken at Costco for $12" → create pantry item + price record - `add_product`: "Add whole milk, 240ml serving, 150 cal, 8g protein, 12g carbs, 8g fat" → create product - `check_expiry`: "What's expiring this week?" → redirect to pantry query - `find_recipe`: "What can I make with chicken and rice?" → trigger suggestion engine - `add_to_list`: "Add eggs and butter to my shopping list" → add items to active list Each recognized intent maps to an existing API operation, executed automatically or returned as a confirmation prompt. ### 10.9 — Web UI: LLM Settings & Features - `/settings/llm` page (admin only): - Provider selection display - Monthly cost usage bar - Rate limit configuration - LLM log viewer (recent calls, success/failure, latency, cost) - Enhance existing UI with LLM-powered features: - Product add modal: "Smart Add" tab with camera/text → LLM pre-fill - Recipe page: "Import from text" and "Import from URL" now functional - Shopping list: "Scan receipt" button with camera - Dashboard: natural language input bar ("What should I cook tonight?") ### 10.10 — Docker: Ollama Service (Optional) If user wants local LLM, add to Docker Compose: ```yaml ollama: image: ollama/ollama ports: ['11434:11434'] volumes: [ollama-models:/root/.ollama] deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] profiles: [llm-local] ``` --- ## Acceptance Criteria - [ ] Can switch LLM provider via environment variable - [ ] OpenAI provider successfully extracts nutrition from product photo - [ ] Recipe import from text returns a valid structured recipe - [ ] Receipt parsing extracts store, items, and prices from receipt image - [ ] All LLM calls are logged with token counts and cost - [ ] Rate limiting prevents exceeding configured RPM - [ ] Monthly budget guard blocks calls when budget is exceeded - [ ] Natural language input correctly identifies intents and executes actions - [ ] NoOp provider still works gracefully when no LLM is configured - [ ] LLM settings page shows usage and cost statistics --- ## Estimated Effort Large. Multiple provider implementations, prompt engineering, testing across different models, cost tracking infrastructure, and NLP intent parsing are all significant. --- ## Notes - Prompt engineering is iterative — expect to refine prompts based on real-world testing - Different providers/models will have varying accuracy — consider model-specific prompt tuning - Local models (Ollama) will be less accurate but free — document quality trade-offs - Consider caching LLM results for identical inputs (e.g., same barcode photo → same product)