axiom-foundation-models-diag

Foundation Models Diagnostics

Overview

Foundation Models issues manifest as context window exceeded errors, guardrail violations, slow generation, availability failures, and unexpected output. Core principle 80% of Foundation Models problems stem from misunderstanding model capabilities (3B parameter device-scale model, not world knowledge), context limits (4096 tokens), or availability requirements—not framework bugs.

Red Flags — Suspect Foundation Models Issue

If you see ANY of these, suspect a Foundation Models misunderstanding, not framework breakage:

• Generation takes >5 seconds
• Error: exceededContextWindowSize
• Error: guardrailViolation
• Error: unsupportedLanguageOrLocale
• Model gives hallucinated/wrong output
• UI freezes during generation
• Feature works in simulator but not on device
• ❌ FORBIDDEN "Foundation Models is broken, we need a different AI"
  • Foundation Models powers Apple Intelligence across millions of devices
  • Wrong output = wrong use case (world knowledge vs summarization)
  • Do not rationalize away the issue—diagnose it

Critical distinction Foundation Models is a device-scale model (3B parameters) optimized for summarization, extraction, classification—NOT world knowledge or complex reasoning. Using it for the wrong task guarantees poor results.

Mandatory First Steps

ALWAYS run these FIRST (before changing code):

// 1. Check availability
let availability = SystemLanguageModel.default.availability

switch availability {
case .available:
    print("✅ Available")
case .unavailable(let reason):
    print("❌ Unavailable: \(reason)")
    // Possible reasons:
    // - Device not Apple Intelligence-capable
    // - Region not supported
    // - User not opted in
}

// Record: "Available? Yes/no, reason if not"

// 2. Check supported languages
let supported = SystemLanguageModel.default.supportedLanguages
print("Supported languages: \(supported)")
print("Current locale: \(Locale.current.language)")

if !supported.contains(Locale.current.language) {
    print("⚠️ Current language not supported!")
}

// Record: "Language supported? Yes/no"

// 3. Check context usage
let session = LanguageModelSession()
// After some interactions:
print("Transcript entries: \(session.transcript.entries.count)")

// Rough estimation (not exact):
let transcriptText = session.transcript.entries
    .map { $0.content }
    .joined()
print("Approximate chars: \(transcriptText.count)")
print("Rough token estimate: \(transcriptText.count / 3)")
// 4096 token limit ≈ 12,000 characters

// Record: "Approaching context limit? Yes/no"

// 4. Profile with Instruments
// Run with Foundation Models Instrument template
// Check:
// - Initial model load time
// - Token counts (input/output)
// - Generation time per request
// - Areas for optimization

// Record: "Latency profile: [numbers from Instruments]"

// 5. Inspect transcript for debugging
print("Full transcript:")
for entry in session.transcript.entries {
    print("Entry: \(entry.content.prefix(100))...")
}

// Record: "Any unusual entries? Repeated content?"