315 lines
10 KiB
Markdown
315 lines
10 KiB
Markdown
# Phase 10 — LLM Integration & Smart Features
|
|
|
|
**Goal**: Wire up the `ILlmProvider` interface (defined in Phase 5) to real LLM backends. Enable all the smart features that have had placeholder endpoints throughout Phases 5-9. Add new natural language interaction capabilities for both medicine and food domains.
|
|
|
|
**Depends on**: All previous phases (feature endpoints already exist)
|
|
|
|
---
|
|
|
|
## Deliverables
|
|
|
|
1. Concrete `ILlmProvider` implementations (OpenAI, Anthropic, Ollama)
|
|
2. Config-driven provider selection
|
|
3. LLM request/response logging and cost tracking
|
|
4. Rate limiting and budget controls
|
|
5. Smart features fully wired:
|
|
- Product recognition (photo → nutrition)
|
|
- Receipt parsing (photo → store + items + prices)
|
|
- Recipe import (text/URL → structured recipe)
|
|
- Natural language pantry entry
|
|
- LLM-powered meal plan suggestions
|
|
6. Prompt templates and versioning
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
### Provider Selection
|
|
|
|
```
|
|
Environment variable: LLM_PROVIDER_TYPE=openai|anthropic|ollama|noop
|
|
|
|
LlmModule registers the provider dynamically:
|
|
|
|
@Module({})
|
|
export class LlmModule {
|
|
static forRoot(): DynamicModule {
|
|
return {
|
|
providers: [{
|
|
provide: LLM_PROVIDER,
|
|
useFactory: (config: ConfigService) => {
|
|
switch (config.get('LLM_PROVIDER_TYPE')) {
|
|
case 'openai': return new OpenAiProvider(config);
|
|
case 'anthropic': return new AnthropicProvider(config);
|
|
case 'ollama': return new OllamaProvider(config);
|
|
default: return new NoOpLlmProvider();
|
|
}
|
|
},
|
|
inject: [ConfigService],
|
|
}],
|
|
exports: [LLM_PROVIDER],
|
|
};
|
|
}
|
|
}
|
|
```
|
|
|
|
### Provider Implementations
|
|
|
|
Each provider implements `ILlmProvider` and handles:
|
|
|
|
- API authentication (keys from env)
|
|
- Model selection (configurable per provider)
|
|
- Request/response mapping to/from vendor format
|
|
- Error handling and retries (exponential backoff)
|
|
- Timeout management
|
|
|
|
```typescript
|
|
// packages/api/src/modules/llm/providers/
|
|
├── noop.provider.ts # Returns null for everything (already exists from Phase 5)
|
|
├── openai.provider.ts # GPT-4o / GPT-4o-mini
|
|
├── anthropic.provider.ts # Claude 3.5 Sonnet / Haiku
|
|
└── ollama.provider.ts # Local models (Llama 3, Mistral, etc.)
|
|
```
|
|
|
|
### Environment Configuration
|
|
|
|
```env
|
|
# Provider selection
|
|
LLM_PROVIDER_TYPE=openai
|
|
|
|
# OpenAI
|
|
OPENAI_API_KEY=sk-...
|
|
OPENAI_MODEL=gpt-4o
|
|
OPENAI_VISION_MODEL=gpt-4o # For image inputs
|
|
|
|
# Anthropic
|
|
ANTHROPIC_API_KEY=sk-ant-...
|
|
ANTHROPIC_MODEL=claude-sonnet-4-20250514
|
|
|
|
# Ollama
|
|
OLLAMA_BASE_URL=http://ollama:11434
|
|
OLLAMA_MODEL=llama3
|
|
OLLAMA_VISION_MODEL=llava
|
|
|
|
# Budget
|
|
LLM_MONTHLY_BUDGET_USD=20.00
|
|
LLM_RATE_LIMIT_RPM=30 # Requests per minute per household
|
|
```
|
|
|
|
---
|
|
|
|
## Tasks
|
|
|
|
### 10.1 — OpenAI Provider
|
|
|
|
```typescript
|
|
class OpenAiProvider implements ILlmProvider {
|
|
// Uses OpenAI Node.js SDK
|
|
// Text endpoints: chat completions with JSON mode
|
|
// Vision endpoints: chat completions with image_url or base64 content
|
|
// Structured output: use function calling or response_format: json_schema
|
|
}
|
|
```
|
|
|
|
- `extractNutrition`: Send product text/image → prompt asks for structured nutrition JSON
|
|
- `parseRecipe`: Send recipe text → prompt extracts name, servings, ingredients[], steps[]
|
|
- `parseRecipeFromUrl`: Fetch URL content first, then parse as text
|
|
- `parseReceipt`: Send receipt image → prompt extracts store, date, line items with prices
|
|
- `suggestMealPlan`: Send pantry summary + targets + preferences → get 7-day plan
|
|
- `parseNaturalLanguage`: Send user text → extract intent + entities (add product, log purchase, etc.)
|
|
|
|
### 10.2 — Anthropic Provider
|
|
|
|
```typescript
|
|
class AnthropicProvider implements ILlmProvider {
|
|
// Uses Anthropic SDK
|
|
// Similar structure to OpenAI but with Claude-specific API format
|
|
// Vision: send image as base64 in messages
|
|
// Structured output: use tool_use for JSON extraction
|
|
}
|
|
```
|
|
|
|
### 10.3 — Ollama Provider
|
|
|
|
```typescript
|
|
class OllamaProvider implements ILlmProvider {
|
|
// Uses Ollama REST API (http://host:11434/api/generate or /api/chat)
|
|
// Text: standard chat endpoint
|
|
// Vision: requires multimodal model (llava, bakllava)
|
|
// Note: local models may be less accurate — adjust prompts for simpler output
|
|
// No cost tracking needed (self-hosted)
|
|
}
|
|
```
|
|
|
|
### 10.4 — Prompt Templates
|
|
|
|
Create versioned prompt templates in `packages/api/src/modules/llm/prompts/`:
|
|
|
|
```typescript
|
|
// prompts/extract-nutrition.ts
|
|
export const EXTRACT_NUTRITION_PROMPT = {
|
|
version: '1.0',
|
|
system: `You are a nutrition data extraction assistant. Given a food product description or image, extract nutritional information. Return ONLY valid JSON matching the schema below. If you cannot determine a value, use null. Be conservative with estimates.`,
|
|
schema: {
|
|
name: 'string',
|
|
brand: 'string | null',
|
|
servingSize: 'number',
|
|
servingUnit: 'g | ml | oz | piece',
|
|
nutrition: {
|
|
calories: 'number',
|
|
protein: 'number (grams)',
|
|
carbs: 'number (grams)',
|
|
fat: 'number (grams)',
|
|
fiber: 'number | null',
|
|
sugar: 'number | null',
|
|
sodium: 'number | null (mg)',
|
|
},
|
|
},
|
|
};
|
|
|
|
// prompts/parse-recipe.ts
|
|
// prompts/parse-receipt.ts
|
|
// prompts/suggest-meal-plan.ts
|
|
// prompts/parse-natural-language.ts
|
|
```
|
|
|
|
- Each prompt has a `version` for tracking which prompt produced which results
|
|
- Prompts are provider-agnostic (providers may wrap them differently)
|
|
|
|
### 10.5 — LLM Logging & Cost Tracking
|
|
|
|
```typescript
|
|
// Schema
|
|
export interface LlmLog {
|
|
id: string;
|
|
householdId: string;
|
|
userId: string;
|
|
provider: string; // 'openai' | 'anthropic' | 'ollama'
|
|
model: string;
|
|
feature: string; // 'extract_nutrition' | 'parse_recipe' | etc.
|
|
promptVersion: string;
|
|
inputTokens: number;
|
|
outputTokens: number;
|
|
totalTokens: number;
|
|
costUsd: number; // Computed from token count + model pricing
|
|
latencyMs: number;
|
|
success: boolean;
|
|
errorMessage?: string;
|
|
createdAt: Date;
|
|
}
|
|
```
|
|
|
|
- All provider calls are wrapped in a logging decorator/interceptor
|
|
- Cost computed from known per-model pricing (configurable)
|
|
- Monthly cost aggregation endpoint for household admins
|
|
|
|
### 10.6 — Rate Limiting & Budget Controls
|
|
|
|
```typescript
|
|
class LlmBudgetGuard {
|
|
/**
|
|
* Before each LLM call:
|
|
* 1. Check per-household rate limit (requests per minute)
|
|
* 2. Check monthly budget: sum costUsd for current month vs LLM_MONTHLY_BUDGET_USD
|
|
* 3. If exceeded, throw BudgetExceededException (HTTP 429)
|
|
*/
|
|
async canProceed(householdId: string): Promise<boolean>;
|
|
}
|
|
```
|
|
|
|
- Rate limiting: Redis-backed or in-memory (household-level RPM)
|
|
- Budget: MongoDB aggregation on `LlmLog` collection
|
|
|
|
### 10.7 — Wire Up All Feature Endpoints
|
|
|
|
Each of these endpoints already exists as a placeholder from earlier phases. Now they get real LLM calls:
|
|
|
|
| Feature | Endpoint (existing) | Phase | LLM Method |
|
|
| --------------------- | ----------------------------------- | ----- | ---------------------- |
|
|
| Product recognition | `POST /products/smart-add` | 5 | `extractNutrition()` |
|
|
| Recipe import (text) | `POST /recipes/import-text` | 6 | `parseRecipe()` |
|
|
| Recipe import (URL) | `POST /recipes/import-url` | 6 | `parseRecipeFromUrl()` |
|
|
| Receipt parsing | `POST /prices/parse-receipt` | 9 | `parseReceipt()` |
|
|
| Meal plan suggestions | `POST /meal-plans/suggest-with-llm` | 8 | `suggestMealPlan()` |
|
|
|
|
### 10.8 — Natural Language Input (New Feature)
|
|
|
|
New universal endpoint:
|
|
|
|
```
|
|
POST /api/v1/nlp/parse
|
|
Body: { text: string }
|
|
Response: { intent: string, action: StructuredAction, confidence: number }
|
|
```
|
|
|
|
Supported intents:
|
|
|
|
- `add_pantry_item`: "I bought 2 lbs of chicken at Costco for $12" → create pantry item + price record
|
|
- `add_product`: "Add whole milk, 240ml serving, 150 cal, 8g protein, 12g carbs, 8g fat" → create product
|
|
- `check_expiry`: "What's expiring this week?" → redirect to pantry query
|
|
- `find_recipe`: "What can I make with chicken and rice?" → trigger suggestion engine
|
|
- `add_to_list`: "Add eggs and butter to my shopping list" → add items to active list
|
|
|
|
Each recognized intent maps to an existing API operation, executed automatically or returned as a confirmation prompt.
|
|
|
|
### 10.9 — Web UI: LLM Settings & Features
|
|
|
|
- `/settings/llm` page (admin only):
|
|
- Provider selection display
|
|
- Monthly cost usage bar
|
|
- Rate limit configuration
|
|
- LLM log viewer (recent calls, success/failure, latency, cost)
|
|
- Enhance existing UI with LLM-powered features:
|
|
- Product add modal: "Smart Add" tab with camera/text → LLM pre-fill
|
|
- Recipe page: "Import from text" and "Import from URL" now functional
|
|
- Shopping list: "Scan receipt" button with camera
|
|
- Dashboard: natural language input bar ("What should I cook tonight?")
|
|
|
|
### 10.10 — Docker: Ollama Service (Optional)
|
|
|
|
If user wants local LLM, add to Docker Compose:
|
|
|
|
```yaml
|
|
ollama:
|
|
image: ollama/ollama
|
|
ports: ['11434:11434']
|
|
volumes: [ollama-models:/root/.ollama]
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
count: all
|
|
capabilities: [gpu]
|
|
profiles: [llm-local]
|
|
```
|
|
|
|
---
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] Can switch LLM provider via environment variable
|
|
- [ ] OpenAI provider successfully extracts nutrition from product photo
|
|
- [ ] Recipe import from text returns a valid structured recipe
|
|
- [ ] Receipt parsing extracts store, items, and prices from receipt image
|
|
- [ ] All LLM calls are logged with token counts and cost
|
|
- [ ] Rate limiting prevents exceeding configured RPM
|
|
- [ ] Monthly budget guard blocks calls when budget is exceeded
|
|
- [ ] Natural language input correctly identifies intents and executes actions
|
|
- [ ] NoOp provider still works gracefully when no LLM is configured
|
|
- [ ] LLM settings page shows usage and cost statistics
|
|
|
|
---
|
|
|
|
## Estimated Effort
|
|
|
|
Large. Multiple provider implementations, prompt engineering, testing across different models, cost tracking infrastructure, and NLP intent parsing are all significant.
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
- Prompt engineering is iterative — expect to refine prompts based on real-world testing
|
|
- Different providers/models will have varying accuracy — consider model-specific prompt tuning
|
|
- Local models (Ollama) will be less accurate but free — document quality trade-offs
|
|
- Consider caching LLM results for identical inputs (e.g., same barcode photo → same product)
|