llm-competition-pipeline

LLM Competition Pipeline

Problem

LLM competitions (Industrial Automation, ARC-AGI-3, ai-agent-security) have unique constraints vs traditional ML:

• Open-source models only (Qwen, DeepSeek, LLaMA — no GPT/Claude)
• No internet at inference (all weights/data must be bundled)
• GPU time limited (30h/week shared, 9-12h per notebook run)
• Prompt engineering is 80% of the work (not model architecture)
• Closed-book setting (no RAG, no tools, no retrieval)

The Pipeline (6 Stages)

Stage 1: Base Model Selection

Choose based on competition rules + GPU memory:

| Model | Params | VRAM (FP8) | Speed | Use Case |
|-------|--------|-------------|-------|----------|
| Qwen 2.5-7B | 7B | ~8GB | Fast | MCQ, classification |
| Qwen 3.6-27B | 27B | ~28GB | Medium | Reasoning, ARC-AGI-3 |
| DeepSeek-R1 | 7B/14B | ~8-16GB | Fast | Math, logic |
| LLaMA-3.1-8B | 8B | ~8GB | Fast | General purpose |

Decision factors:
  □ Competition rule: which model families are allowed?
  □ VRAM budget: Kaggle T4 = 16GB, P100 = 16GB, dual T4 = 30GB
  □ Inference time: 7B ~ 1-2s/query, 27B ~ 5-10s/query
  □ Task type: reasoning → larger, classification → smaller

Stage 2: Prompt Engineering

Start simple, then add complexity:

Level 0 — Zero-shot:
  "Answer the following question: {question}\nAnswer:"

Level 1 — Instruction:
  "You are an expert in {domain}. Read the passage and answer.
   Passage: {passage}
   Question: {question}
   Provide only the letter of the correct answer."

Level 2 — Few-shot (3-5 examples):
  "Here are examples of correct answers:
   Example 1: {input} → {output}
   Example 2: {input} → {output}
   Example 3: {input} → {output}
   Now answer: {question}"

Level 3 — Chain-of-thought:
  "Think step by step, then give the final answer."

Level 4 — Dynamic few-shot (retrieve relevant examples):
  For each test question, retrieve the most similar training examples
  and include them as few-shot context.

Key rule: Each level adds latency. Level 0 is fastest. Only escalate if OOF accuracy improves by >2%.