MeshiTrack/docs/phases/phase-10-llm.md

# Phase 10 — LLM Integration & Smart Features

**Goal**: Wire up the `ILlmProvider` interface (defined in Phase 5) to real LLM backends. Enable all the smart features that have had placeholder endpoints throughout Phases 5-9. Add new natural language interaction capabilities for both medicine and food domains.

**Depends on**: All previous phases (feature endpoints already exist)

---

## Deliverables

1. Concrete `ILlmProvider` implementations (OpenAI, Anthropic, Ollama)
2. Config-driven provider selection
3. LLM request/response logging and cost tracking
4. Rate limiting and budget controls
5. Smart features fully wired:
   - Product recognition (photo → nutrition)
   - Receipt parsing (photo → store + items + prices)
   - Recipe import (text/URL → structured recipe)
   - Natural language pantry entry
   - LLM-powered meal plan suggestions
6. Prompt templates and versioning

---

## Architecture

### Provider Selection

```
Environment variable: LLM_PROVIDER_TYPE=openai|anthropic|ollama|noop

LlmModule registers the provider dynamically:

@Module({})
export class LlmModule {
  static forRoot(): DynamicModule {
    return {
      providers: [{
        provide: LLM_PROVIDER,
        useFactory: (config: ConfigService) => {
          switch (config.get('LLM_PROVIDER_TYPE')) {
            case 'openai':    return new OpenAiProvider(config);
            case 'anthropic': return new AnthropicProvider(config);
            case 'ollama':    return new OllamaProvider(config);
            default:          return new NoOpLlmProvider();
          }
        },
        inject: [ConfigService],
      }],
      exports: [LLM_PROVIDER],
    };
  }
}
```

### Provider Implementations

Each provider implements `ILlmProvider` and handles:

- API authentication (keys from env)
- Model selection (configurable per provider)
- Request/response mapping to/from vendor format
- Error handling and retries (exponential backoff)
- Timeout management

```typescript
// packages/api/src/modules/llm/providers/
├── noop.provider.ts          # Returns null for everything (already exists from Phase 5)
├── openai.provider.ts        # GPT-4o / GPT-4o-mini
├── anthropic.provider.ts     # Claude 3.5 Sonnet / Haiku
└── ollama.provider.ts        # Local models (Llama 3, Mistral, etc.)
```

### Environment Configuration

```env
# Provider selection
LLM_PROVIDER_TYPE=openai

# OpenAI
OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4o
OPENAI_VISION_MODEL=gpt-4o       # For image inputs

# Anthropic
ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_MODEL=claude-sonnet-4-20250514

# Ollama
OLLAMA_BASE_URL=http://ollama:11434
OLLAMA_MODEL=llama3
OLLAMA_VISION_MODEL=llava

# Budget
LLM_MONTHLY_BUDGET_USD=20.00
LLM_RATE_LIMIT_RPM=30             # Requests per minute per household
```

---

## Tasks

### 10.1 — OpenAI Provider

```typescript
class OpenAiProvider implements ILlmProvider {
  // Uses OpenAI Node.js SDK
  // Text endpoints: chat completions with JSON mode
  // Vision endpoints: chat completions with image_url or base64 content
  // Structured output: use function calling or response_format: json_schema
}
```

- `extractNutrition`: Send product text/image → prompt asks for structured nutrition JSON
- `parseRecipe`: Send recipe text → prompt extracts name, servings, ingredients[], steps[]
- `parseRecipeFromUrl`: Fetch URL content first, then parse as text
- `parseReceipt`: Send receipt image → prompt extracts store, date, line items with prices
- `suggestMealPlan`: Send pantry summary + targets + preferences → get 7-day plan
- `parseNaturalLanguage`: Send user text → extract intent + entities (add product, log purchase, etc.)

### 10.2 — Anthropic Provider

```typescript
class AnthropicProvider implements ILlmProvider {
  // Uses Anthropic SDK
  // Similar structure to OpenAI but with Claude-specific API format
  // Vision: send image as base64 in messages
  // Structured output: use tool_use for JSON extraction
}
```

### 10.3 — Ollama Provider

```typescript
class OllamaProvider implements ILlmProvider {
  // Uses Ollama REST API (http://host:11434/api/generate or /api/chat)
  // Text: standard chat endpoint
  // Vision: requires multimodal model (llava, bakllava)
  // Note: local models may be less accurate — adjust prompts for simpler output
  // No cost tracking needed (self-hosted)
}
```

### 10.4 — Prompt Templates

Create versioned prompt templates in `packages/api/src/modules/llm/prompts/`:

```typescript
// prompts/extract-nutrition.ts
export const EXTRACT_NUTRITION_PROMPT = {
  version: '1.0',
  system: `You are a nutrition data extraction assistant. Given a food product description or image, extract nutritional information. Return ONLY valid JSON matching the schema below. If you cannot determine a value, use null. Be conservative with estimates.`,
  schema: {
    name: 'string',
    brand: 'string | null',
    servingSize: 'number',
    servingUnit: 'g | ml | oz | piece',
    nutrition: {
      calories: 'number',
      protein: 'number (grams)',
      carbs: 'number (grams)',
      fat: 'number (grams)',
      fiber: 'number | null',
      sugar: 'number | null',
      sodium: 'number | null (mg)',
    },
  },
};

// prompts/parse-recipe.ts
// prompts/parse-receipt.ts
// prompts/suggest-meal-plan.ts
// prompts/parse-natural-language.ts
```

- Each prompt has a `version` for tracking which prompt produced which results
- Prompts are provider-agnostic (providers may wrap them differently)

### 10.5 — LLM Logging & Cost Tracking

```typescript
// Schema
export interface LlmLog {
  id: string;
  householdId: string;
  userId: string;
  provider: string; // 'openai' | 'anthropic' | 'ollama'
  model: string;
  feature: string; // 'extract_nutrition' | 'parse_recipe' | etc.
  promptVersion: string;
  inputTokens: number;
  outputTokens: number;
  totalTokens: number;
  costUsd: number; // Computed from token count + model pricing
  latencyMs: number;
  success: boolean;
  errorMessage?: string;
  createdAt: Date;
}
```

- All provider calls are wrapped in a logging decorator/interceptor
- Cost computed from known per-model pricing (configurable)
- Monthly cost aggregation endpoint for household admins

### 10.6 — Rate Limiting & Budget Controls

```typescript
class LlmBudgetGuard {
  /**
   * Before each LLM call:
   * 1. Check per-household rate limit (requests per minute)
   * 2. Check monthly budget: sum costUsd for current month vs LLM_MONTHLY_BUDGET_USD
   * 3. If exceeded, throw BudgetExceededException (HTTP 429)
   */
  async canProceed(householdId: string): Promise<boolean>;
}
```

- Rate limiting: Redis-backed or in-memory (household-level RPM)
- Budget: MongoDB aggregation on `LlmLog` collection

### 10.7 — Wire Up All Feature Endpoints

Each of these endpoints already exists as a placeholder from earlier phases. Now they get real LLM calls:

| Feature               | Endpoint (existing)                 | Phase | LLM Method             |
| --------------------- | ----------------------------------- | ----- | ---------------------- |
| Product recognition   | `POST /products/smart-add`          | 5     | `extractNutrition()`   |
| Recipe import (text)  | `POST /recipes/import-text`         | 6     | `parseRecipe()`        |
| Recipe import (URL)   | `POST /recipes/import-url`          | 6     | `parseRecipeFromUrl()` |
| Receipt parsing       | `POST /prices/parse-receipt`        | 9     | `parseReceipt()`       |
| Meal plan suggestions | `POST /meal-plans/suggest-with-llm` | 8     | `suggestMealPlan()`    |

### 10.8 — Natural Language Input (New Feature)

New universal endpoint:

```
POST /api/v1/nlp/parse
Body: { text: string }
Response: { intent: string, action: StructuredAction, confidence: number }
```

Supported intents:

- `add_pantry_item`: "I bought 2 lbs of chicken at Costco for $12" → create pantry item + price record
- `add_product`: "Add whole milk, 240ml serving, 150 cal, 8g protein, 12g carbs, 8g fat" → create product
- `check_expiry`: "What's expiring this week?" → redirect to pantry query
- `find_recipe`: "What can I make with chicken and rice?" → trigger suggestion engine
- `add_to_list`: "Add eggs and butter to my shopping list" → add items to active list

Each recognized intent maps to an existing API operation, executed automatically or returned as a confirmation prompt.

### 10.9 — Web UI: LLM Settings & Features

- `/settings/llm` page (admin only):
  - Provider selection display
  - Monthly cost usage bar
  - Rate limit configuration
  - LLM log viewer (recent calls, success/failure, latency, cost)
- Enhance existing UI with LLM-powered features:
  - Product add modal: "Smart Add" tab with camera/text → LLM pre-fill
  - Recipe page: "Import from text" and "Import from URL" now functional
  - Shopping list: "Scan receipt" button with camera
  - Dashboard: natural language input bar ("What should I cook tonight?")

### 10.10 — Docker: Ollama Service (Optional)

If user wants local LLM, add to Docker Compose:

```yaml
ollama:
  image: ollama/ollama
  ports: ['11434:11434']
  volumes: [ollama-models:/root/.ollama]
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: all
            capabilities: [gpu]
  profiles: [llm-local]
```

---

## Acceptance Criteria

- [ ] Can switch LLM provider via environment variable
- [ ] OpenAI provider successfully extracts nutrition from product photo
- [ ] Recipe import from text returns a valid structured recipe
- [ ] Receipt parsing extracts store, items, and prices from receipt image
- [ ] All LLM calls are logged with token counts and cost
- [ ] Rate limiting prevents exceeding configured RPM
- [ ] Monthly budget guard blocks calls when budget is exceeded
- [ ] Natural language input correctly identifies intents and executes actions
- [ ] NoOp provider still works gracefully when no LLM is configured
- [ ] LLM settings page shows usage and cost statistics

---

## Estimated Effort

Large. Multiple provider implementations, prompt engineering, testing across different models, cost tracking infrastructure, and NLP intent parsing are all significant.

---

## Notes

- Prompt engineering is iterative — expect to refine prompts based on real-world testing
- Different providers/models will have varying accuracy — consider model-specific prompt tuning
- Local models (Ollama) will be less accurate but free — document quality trade-offs
- Consider caching LLM results for identical inputs (e.g., same barcode photo → same product)