Setup initial project

2026-03-27 14:50:34 +09:00 · 2026-03-27 14:50:34 +09:00 · db79af06f7
commit db79af06f7
119 changed files with 20761 additions and 0 deletions
--- a/docs/phases/phase-10-llm.md
+++ b/docs/phases/phase-10-llm.md
@ -0,0 +1,315 @@
+# Phase 10 — LLM Integration & Smart Features
+
+**Goal**: Wire up the `ILlmProvider` interface (defined in Phase 5) to real LLM backends. Enable all the smart features that have had placeholder endpoints throughout Phases 5-9. Add new natural language interaction capabilities for both medicine and food domains.
+
+**Depends on**: All previous phases (feature endpoints already exist)
+
+---
+
+## Deliverables
+
+1. Concrete `ILlmProvider` implementations (OpenAI, Anthropic, Ollama)
+2. Config-driven provider selection
+3. LLM request/response logging and cost tracking
+4. Rate limiting and budget controls
+5. Smart features fully wired:
+   - Product recognition (photo → nutrition)
+   - Receipt parsing (photo → store + items + prices)
+   - Recipe import (text/URL → structured recipe)
+   - Natural language pantry entry
+   - LLM-powered meal plan suggestions
+6. Prompt templates and versioning
+
+---
+
+## Architecture
+
+### Provider Selection
+
+```
+Environment variable: LLM_PROVIDER_TYPE=openai|anthropic|ollama|noop
+
+LlmModule registers the provider dynamically:
+
+@Module({})
+export class LlmModule {
+  static forRoot(): DynamicModule {
+    return {
+      providers: [{
+        provide: LLM_PROVIDER,
+        useFactory: (config: ConfigService) => {
+          switch (config.get('LLM_PROVIDER_TYPE')) {
+            case 'openai':    return new OpenAiProvider(config);
+            case 'anthropic': return new AnthropicProvider(config);
+            case 'ollama':    return new OllamaProvider(config);
+            default:          return new NoOpLlmProvider();
+          }
+        },
+        inject: [ConfigService],
+      }],
+      exports: [LLM_PROVIDER],
+    };
+  }
+}
+```
+
+### Provider Implementations
+
+Each provider implements `ILlmProvider` and handles:
+
+- API authentication (keys from env)
+- Model selection (configurable per provider)
+- Request/response mapping to/from vendor format
+- Error handling and retries (exponential backoff)
+- Timeout management
+
+```typescript
+// packages/api/src/modules/llm/providers/
+├── noop.provider.ts          # Returns null for everything (already exists from Phase 5)
+├── openai.provider.ts        # GPT-4o / GPT-4o-mini
+├── anthropic.provider.ts     # Claude 3.5 Sonnet / Haiku
+└── ollama.provider.ts        # Local models (Llama 3, Mistral, etc.)
+```
+
+### Environment Configuration
+
+```env
+# Provider selection
+LLM_PROVIDER_TYPE=openai
+
+# OpenAI
+OPENAI_API_KEY=sk-...
+OPENAI_MODEL=gpt-4o
+OPENAI_VISION_MODEL=gpt-4o       # For image inputs
+
+# Anthropic
+ANTHROPIC_API_KEY=sk-ant-...
+ANTHROPIC_MODEL=claude-sonnet-4-20250514
+
+# Ollama
+OLLAMA_BASE_URL=http://ollama:11434
+OLLAMA_MODEL=llama3
+OLLAMA_VISION_MODEL=llava
+
+# Budget
+LLM_MONTHLY_BUDGET_USD=20.00
+LLM_RATE_LIMIT_RPM=30             # Requests per minute per household
+```
+
+---
+
+## Tasks
+
+### 10.1 — OpenAI Provider
+
+```typescript
+class OpenAiProvider implements ILlmProvider {
+  // Uses OpenAI Node.js SDK
+  // Text endpoints: chat completions with JSON mode
+  // Vision endpoints: chat completions with image_url or base64 content
+  // Structured output: use function calling or response_format: json_schema
+}
+```
+
+- `extractNutrition`: Send product text/image → prompt asks for structured nutrition JSON
+- `parseRecipe`: Send recipe text → prompt extracts name, servings, ingredients[], steps[]
+- `parseRecipeFromUrl`: Fetch URL content first, then parse as text
+- `parseReceipt`: Send receipt image → prompt extracts store, date, line items with prices
+- `suggestMealPlan`: Send pantry summary + targets + preferences → get 7-day plan
+- `parseNaturalLanguage`: Send user text → extract intent + entities (add product, log purchase, etc.)
+
+### 10.2 — Anthropic Provider
+
+```typescript
+class AnthropicProvider implements ILlmProvider {
+  // Uses Anthropic SDK
+  // Similar structure to OpenAI but with Claude-specific API format
+  // Vision: send image as base64 in messages
+  // Structured output: use tool_use for JSON extraction
+}
+```
+
+### 10.3 — Ollama Provider
+
+```typescript
+class OllamaProvider implements ILlmProvider {
+  // Uses Ollama REST API (http://host:11434/api/generate or /api/chat)
+  // Text: standard chat endpoint
+  // Vision: requires multimodal model (llava, bakllava)
+  // Note: local models may be less accurate — adjust prompts for simpler output
+  // No cost tracking needed (self-hosted)
+}
+```
+
+### 10.4 — Prompt Templates
+
+Create versioned prompt templates in `packages/api/src/modules/llm/prompts/`:
+
+```typescript
+// prompts/extract-nutrition.ts
+export const EXTRACT_NUTRITION_PROMPT = {
+  version: '1.0',
+  system: `You are a nutrition data extraction assistant. Given a food product description or image, extract nutritional information. Return ONLY valid JSON matching the schema below. If you cannot determine a value, use null. Be conservative with estimates.`,
+  schema: {
+    name: 'string',
+    brand: 'string | null',
+    servingSize: 'number',
+    servingUnit: 'g | ml | oz | piece',
+    nutrition: {
+      calories: 'number',
+      protein: 'number (grams)',
+      carbs: 'number (grams)',
+      fat: 'number (grams)',
+      fiber: 'number | null',
+      sugar: 'number | null',
+      sodium: 'number | null (mg)',
+    },
+  },
+};
+
+// prompts/parse-recipe.ts
+// prompts/parse-receipt.ts
+// prompts/suggest-meal-plan.ts
+// prompts/parse-natural-language.ts
+```
+
+- Each prompt has a `version` for tracking which prompt produced which results
+- Prompts are provider-agnostic (providers may wrap them differently)
+
+### 10.5 — LLM Logging & Cost Tracking
+
+```typescript
+// Schema
+export interface LlmLog {
+  id: string;
+  householdId: string;
+  userId: string;
+  provider: string; // 'openai' | 'anthropic' | 'ollama'
+  model: string;
+  feature: string; // 'extract_nutrition' | 'parse_recipe' | etc.
+  promptVersion: string;
+  inputTokens: number;
+  outputTokens: number;
+  totalTokens: number;
+  costUsd: number; // Computed from token count + model pricing
+  latencyMs: number;
+  success: boolean;
+  errorMessage?: string;
+  createdAt: Date;
+}
+```
+
+- All provider calls are wrapped in a logging decorator/interceptor
+- Cost computed from known per-model pricing (configurable)
+- Monthly cost aggregation endpoint for household admins
+
+### 10.6 — Rate Limiting & Budget Controls
+
+```typescript
+class LlmBudgetGuard {
+  /**
+   * Before each LLM call:
+   * 1. Check per-household rate limit (requests per minute)
+   * 2. Check monthly budget: sum costUsd for current month vs LLM_MONTHLY_BUDGET_USD
+   * 3. If exceeded, throw BudgetExceededException (HTTP 429)
+   */
+  async canProceed(householdId: string): Promise<boolean>;
+}
+```
+
+- Rate limiting: Redis-backed or in-memory (household-level RPM)
+- Budget: MongoDB aggregation on `LlmLog` collection
+
+### 10.7 — Wire Up All Feature Endpoints
+
+Each of these endpoints already exists as a placeholder from earlier phases. Now they get real LLM calls:
+
+| Feature               | Endpoint (existing)                 | Phase | LLM Method             |
+| --------------------- | ----------------------------------- | ----- | ---------------------- |
+| Product recognition   | `POST /products/smart-add`          | 5     | `extractNutrition()`   |
+| Recipe import (text)  | `POST /recipes/import-text`         | 6     | `parseRecipe()`        |
+| Recipe import (URL)   | `POST /recipes/import-url`          | 6     | `parseRecipeFromUrl()` |
+| Receipt parsing       | `POST /prices/parse-receipt`        | 9     | `parseReceipt()`       |
+| Meal plan suggestions | `POST /meal-plans/suggest-with-llm` | 8     | `suggestMealPlan()`    |
+
+### 10.8 — Natural Language Input (New Feature)
+
+New universal endpoint:
+
+```
+POST /api/v1/nlp/parse
+Body: { text: string }
+Response: { intent: string, action: StructuredAction, confidence: number }
+```
+
+Supported intents:
+
+- `add_pantry_item`: "I bought 2 lbs of chicken at Costco for $12" → create pantry item + price record
+- `add_product`: "Add whole milk, 240ml serving, 150 cal, 8g protein, 12g carbs, 8g fat" → create product
+- `check_expiry`: "What's expiring this week?" → redirect to pantry query
+- `find_recipe`: "What can I make with chicken and rice?" → trigger suggestion engine
+- `add_to_list`: "Add eggs and butter to my shopping list" → add items to active list
+
+Each recognized intent maps to an existing API operation, executed automatically or returned as a confirmation prompt.
+
+### 10.9 — Web UI: LLM Settings & Features
+
+- `/settings/llm` page (admin only):
+  - Provider selection display
+  - Monthly cost usage bar
+  - Rate limit configuration
+  - LLM log viewer (recent calls, success/failure, latency, cost)
+- Enhance existing UI with LLM-powered features:
+  - Product add modal: "Smart Add" tab with camera/text → LLM pre-fill
+  - Recipe page: "Import from text" and "Import from URL" now functional
+  - Shopping list: "Scan receipt" button with camera
+  - Dashboard: natural language input bar ("What should I cook tonight?")
+
+### 10.10 — Docker: Ollama Service (Optional)
+
+If user wants local LLM, add to Docker Compose:
+
+```yaml
+ollama:
+  image: ollama/ollama
+  ports: ['11434:11434']
+  volumes: [ollama-models:/root/.ollama]
+  deploy:
+    resources:
+      reservations:
+        devices:
+          - driver: nvidia
+            count: all
+            capabilities: [gpu]
+  profiles: [llm-local]
+```
+
+---
+
+## Acceptance Criteria
+
+- [ ] Can switch LLM provider via environment variable
+- [ ] OpenAI provider successfully extracts nutrition from product photo
+- [ ] Recipe import from text returns a valid structured recipe
+- [ ] Receipt parsing extracts store, items, and prices from receipt image
+- [ ] All LLM calls are logged with token counts and cost
+- [ ] Rate limiting prevents exceeding configured RPM
+- [ ] Monthly budget guard blocks calls when budget is exceeded
+- [ ] Natural language input correctly identifies intents and executes actions
+- [ ] NoOp provider still works gracefully when no LLM is configured
+- [ ] LLM settings page shows usage and cost statistics
+
+---
+
+## Estimated Effort
+
+Large. Multiple provider implementations, prompt engineering, testing across different models, cost tracking infrastructure, and NLP intent parsing are all significant.
+
+---
+
+## Notes
+
+- Prompt engineering is iterative — expect to refine prompts based on real-world testing
+- Different providers/models will have varying accuracy — consider model-specific prompt tuning
+- Local models (Ollama) will be less accurate but free — document quality trade-offs
+- Consider caching LLM results for identical inputs (e.g., same barcode photo → same product)