project-careti: Skill for Claude Code

API Provider Testing Skill

Overview

Guidelines for testing LLM API providers when adding or modifying support in Caret.

Test Categories

1. Basic Conversation Test

• Simple prompt → response verification
• Streaming chunk reception
• Token usage reporting

Example:

const response = await handler.createMessage("System prompt", [
  { role: 'user', content: 'Hello' }
])
// Verify: text chunks + usage chunk

2. Tool Calling Test

Core verification for agentic capabilities.

Test Flow:

1. Send message with tool definitions
2. Verify tool_calls chunk received
3. Verify function name and arguments
4. Send tool result back
5. Verify final text response

Provider-Specific Formats:

3. Timeout Test

• Use AbortController with configurable timeout
• Default: 60 seconds recommended
• Verify timeout error is thrown correctly

4. Error Handling Test

• Invalid API key → 401/403 error
• Rate limiting → 429 error with retry
• Empty response → specific error message

Test Script Template

Location: scripts/test-{provider}-api.js

// Load .env
const fs = require('fs')
const path = require('path')
const envPath = path.resolve(__dirname, '../.env')
if (fs.existsSync(envPath)) {
  fs.readFileSync(envPath, 'utf-8').split('\n').forEach((line) => {
    const match = line.match(/^([^=]+)=(.*)$/)
    if (match && !process.env[match[1]]) {
      process.env[match[1]] = match[2]
    }
  })
}

// Test structure
async function testBasicConversation() { ... }
async function testToolCalling() { ... }
async function testToolResultFlow() { ... }
async function testTimeout() { ... }

async function main() {
  const results = {
    basic: await testBasicConversation(),
    toolCall: await testToolCalling(),
    toolFlow: await testToolResultFlow(),
  }
  console.log('Results:', results)
}