Aerilyn Dziedzelė db79af06f7 Setup initial project

2026-03-27 14:50:34 +09:00

10 KiB

Raw Blame History

Phase 10 — LLM Integration & Smart Features

Goal: Wire up the ILlmProvider interface (defined in Phase 5) to real LLM backends. Enable all the smart features that have had placeholder endpoints throughout Phases 5-9. Add new natural language interaction capabilities for both medicine and food domains.

Depends on: All previous phases (feature endpoints already exist)

Deliverables

Concrete ILlmProvider implementations (OpenAI, Anthropic, Ollama)
Config-driven provider selection
LLM request/response logging and cost tracking
Rate limiting and budget controls
Smart features fully wired:
- Product recognition (photo → nutrition)
- Receipt parsing (photo → store + items + prices)
- Recipe import (text/URL → structured recipe)
- Natural language pantry entry
- LLM-powered meal plan suggestions
Prompt templates and versioning

Architecture

Provider Selection

Environment variable: LLM_PROVIDER_TYPE=openai|anthropic|ollama|noop

LlmModule registers the provider dynamically:

@Module({})
export class LlmModule {
  static forRoot(): DynamicModule {
    return {
      providers: [{
        provide: LLM_PROVIDER,
        useFactory: (config: ConfigService) => {
          switch (config.get('LLM_PROVIDER_TYPE')) {
            case 'openai':    return new OpenAiProvider(config);
            case 'anthropic': return new AnthropicProvider(config);
            case 'ollama':    return new OllamaProvider(config);
            default:          return new NoOpLlmProvider();
          }
        },
        inject: [ConfigService],
      }],
      exports: [LLM_PROVIDER],
    };
  }
}

Provider Implementations

Each provider implements ILlmProvider and handles:

API authentication (keys from env)
Model selection (configurable per provider)
Request/response mapping to/from vendor format
Error handling and retries (exponential backoff)
Timeout management

// packages/api/src/modules/llm/providers/
├── noop.provider.ts          # Returns null for everything (already exists from Phase 5)
├── openai.provider.ts        # GPT-4o / GPT-4o-mini
├── anthropic.provider.ts     # Claude 3.5 Sonnet / Haiku
└── ollama.provider.ts        # Local models (Llama 3, Mistral, etc.)

Environment Configuration

# Provider selection
LLM_PROVIDER_TYPE=openai

# OpenAI
OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4o
OPENAI_VISION_MODEL=gpt-4o       # For image inputs

# Anthropic
ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_MODEL=claude-sonnet-4-20250514

# Ollama
OLLAMA_BASE_URL=http://ollama:11434
OLLAMA_MODEL=llama3
OLLAMA_VISION_MODEL=llava

# Budget
LLM_MONTHLY_BUDGET_USD=20.00
LLM_RATE_LIMIT_RPM=30             # Requests per minute per household

Tasks

10.1 — OpenAI Provider

class OpenAiProvider implements ILlmProvider {
  // Uses OpenAI Node.js SDK
  // Text endpoints: chat completions with JSON mode
  // Vision endpoints: chat completions with image_url or base64 content
  // Structured output: use function calling or response_format: json_schema
}

extractNutrition: Send product text/image → prompt asks for structured nutrition JSON
parseRecipe: Send recipe text → prompt extracts name, servings, ingredients[], steps[]
parseRecipeFromUrl: Fetch URL content first, then parse as text
parseReceipt: Send receipt image → prompt extracts store, date, line items with prices
suggestMealPlan: Send pantry summary + targets + preferences → get 7-day plan
parseNaturalLanguage: Send user text → extract intent + entities (add product, log purchase, etc.)

10.2 — Anthropic Provider

class AnthropicProvider implements ILlmProvider {
  // Uses Anthropic SDK
  // Similar structure to OpenAI but with Claude-specific API format
  // Vision: send image as base64 in messages
  // Structured output: use tool_use for JSON extraction
}

10.3 — Ollama Provider

class OllamaProvider implements ILlmProvider {
  // Uses Ollama REST API (http://host:11434/api/generate or /api/chat)
  // Text: standard chat endpoint
  // Vision: requires multimodal model (llava, bakllava)
  // Note: local models may be less accurate — adjust prompts for simpler output
  // No cost tracking needed (self-hosted)
}

10.4 — Prompt Templates

Create versioned prompt templates in packages/api/src/modules/llm/prompts/:

// prompts/extract-nutrition.ts
export const EXTRACT_NUTRITION_PROMPT = {
  version: '1.0',
  system: `You are a nutrition data extraction assistant. Given a food product description or image, extract nutritional information. Return ONLY valid JSON matching the schema below. If you cannot determine a value, use null. Be conservative with estimates.`,
  schema: {
    name: 'string',
    brand: 'string | null',
    servingSize: 'number',
    servingUnit: 'g | ml | oz | piece',
    nutrition: {
      calories: 'number',
      protein: 'number (grams)',
      carbs: 'number (grams)',
      fat: 'number (grams)',
      fiber: 'number | null',
      sugar: 'number | null',
      sodium: 'number | null (mg)',
    },
  },
};

// prompts/parse-recipe.ts
// prompts/parse-receipt.ts
// prompts/suggest-meal-plan.ts
// prompts/parse-natural-language.ts

Each prompt has a version for tracking which prompt produced which results
Prompts are provider-agnostic (providers may wrap them differently)

10.5 — LLM Logging & Cost Tracking

// Schema
export interface LlmLog {
  id: string;
  householdId: string;
  userId: string;
  provider: string; // 'openai' | 'anthropic' | 'ollama'
  model: string;
  feature: string; // 'extract_nutrition' | 'parse_recipe' | etc.
  promptVersion: string;
  inputTokens: number;
  outputTokens: number;
  totalTokens: number;
  costUsd: number; // Computed from token count + model pricing
  latencyMs: number;
  success: boolean;
  errorMessage?: string;
  createdAt: Date;
}

All provider calls are wrapped in a logging decorator/interceptor
Cost computed from known per-model pricing (configurable)
Monthly cost aggregation endpoint for household admins

10.6 — Rate Limiting & Budget Controls

class LlmBudgetGuard {
  /**
   * Before each LLM call:
   * 1. Check per-household rate limit (requests per minute)
   * 2. Check monthly budget: sum costUsd for current month vs LLM_MONTHLY_BUDGET_USD
   * 3. If exceeded, throw BudgetExceededException (HTTP 429)
   */
  async canProceed(householdId: string): Promise<boolean>;
}

Rate limiting: Redis-backed or in-memory (household-level RPM)
Budget: MongoDB aggregation on LlmLog collection

10.7 — Wire Up All Feature Endpoints

Each of these endpoints already exists as a placeholder from earlier phases. Now they get real LLM calls:

Feature	Endpoint (existing)	Phase	LLM Method
Product recognition	`POST /products/smart-add`	5	`extractNutrition()`
Recipe import (text)	`POST /recipes/import-text`	6	`parseRecipe()`
Recipe import (URL)	`POST /recipes/import-url`	6	`parseRecipeFromUrl()`
Receipt parsing	`POST /prices/parse-receipt`	9	`parseReceipt()`
Meal plan suggestions	`POST /meal-plans/suggest-with-llm`	8	`suggestMealPlan()`

10.8 — Natural Language Input (New Feature)

New universal endpoint:

POST /api/v1/nlp/parse
Body: { text: string }
Response: { intent: string, action: StructuredAction, confidence: number }

Supported intents:

add_pantry_item: "I bought 2 lbs of chicken at Costco for $12" → create pantry item + price record
add_product: "Add whole milk, 240ml serving, 150 cal, 8g protein, 12g carbs, 8g fat" → create product
check_expiry: "What's expiring this week?" → redirect to pantry query
find_recipe: "What can I make with chicken and rice?" → trigger suggestion engine
add_to_list: "Add eggs and butter to my shopping list" → add items to active list

Each recognized intent maps to an existing API operation, executed automatically or returned as a confirmation prompt.

10.9 — Web UI: LLM Settings & Features

/settings/llm page (admin only):
- Provider selection display
- Monthly cost usage bar
- Rate limit configuration
- LLM log viewer (recent calls, success/failure, latency, cost)
Enhance existing UI with LLM-powered features:
- Product add modal: "Smart Add" tab with camera/text → LLM pre-fill
- Recipe page: "Import from text" and "Import from URL" now functional
- Shopping list: "Scan receipt" button with camera
- Dashboard: natural language input bar ("What should I cook tonight?")

10.10 — Docker: Ollama Service (Optional)

If user wants local LLM, add to Docker Compose:

ollama:
  image: ollama/ollama
  ports: ['11434:11434']
  volumes: [ollama-models:/root/.ollama]
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: all
            capabilities: [gpu]
  profiles: [llm-local]

Acceptance Criteria

Can switch LLM provider via environment variable
OpenAI provider successfully extracts nutrition from product photo
Recipe import from text returns a valid structured recipe
Receipt parsing extracts store, items, and prices from receipt image
All LLM calls are logged with token counts and cost
Rate limiting prevents exceeding configured RPM
Monthly budget guard blocks calls when budget is exceeded
Natural language input correctly identifies intents and executes actions
NoOp provider still works gracefully when no LLM is configured
LLM settings page shows usage and cost statistics

Estimated Effort

Large. Multiple provider implementations, prompt engineering, testing across different models, cost tracking infrastructure, and NLP intent parsing are all significant.

Notes

Prompt engineering is iterative — expect to refine prompts based on real-world testing
Different providers/models will have varying accuracy — consider model-specific prompt tuning
Local models (Ollama) will be less accurate but free — document quality trade-offs
Consider caching LLM results for identical inputs (e.g., same barcode photo → same product)

10 KiB Raw Blame History