cjk-structured-llm-output-truncates-json-needs-2x-tokens

Long CJK structured output truncates JSON — budget ~2× tokens, and salvage the rest

Problem

You ask an OpenAI-compatible LLM for a long JSON report. In English it returns valid JSON. In Chinese (or Japanese/Korean) the identical request returns a string that starts with valid JSON but is cut off mid-object → json.loads fails, your regex \{.*\} grabs an unbalanced blob, and you get "no parseable JSON".

Why

• CJK is token-dense. One Han character is often 1–2 BPE tokens; English averages ~0.25 tokens/char. The same report is ~1.5–2× more output tokens in Chinese. A max_tokens that comfortably fit the English report truncates the Chinese one.
• Per-model output caps differ and some are low. Requesting max_tokens above a model's hard cap is either rejected (400 ... Range of max_tokens should be [1, 8192]) or silently clamped, then the long CJK report truncates. Observed: qwen-max hard-caps at 8192 (can't emit a full Chinese report); glm-4.6 / minimax ≈16K; qwen-plus / qwen3-235b / glm-5.2 / kimi / deepseek handle 24–32K. Reasoning models also spend tokens on reasoning_content before the JSON, compounding it.

Fix

1. Budget ~2× the English max_tokens for CJK (e.g. 16000–32000 for a full report), per model, up to its cap.
2. Probe the cap if a 400 appears — binary-search max_tokens (a tiny call that 400s vs 200s reveals the ceiling). For reasoning models, leave headroom for reasoning_content.
3. Salvage truncated JSON instead of failing — close the object after the last complete top-level section. Walk the string tracking brace/bracket depth and string/escape state; record the index each time depth returns to 1 (a top-level value just closed); cut there and append }:

   def salvage(c):
       s = c[c.find("{"):]; depth=0; instr=False; esc=False; cut=None
       for i,ch in enumerate(s):
           if esc: esc=False; continue
           if instr:
               if ch=="\\": esc=True
               elif ch=='"': instr=False
               continue
           if ch=='"': instr=True
           elif ch in "{[": depth+=1
           elif ch in "}]":
               depth-=1
               if depth==1: cut=i+1
       if cut:
           try:
               d=json.loads(s[:cut].rstrip().rstrip(",")+"}")
               if isinstance(d,dict) and len(d)>=MIN_SECTIONS: return d
           except Exception: pass
       return None
   

   Truncation then degrades gracefully — the report keeps its complete sections, and the missing tail shows up as lower "comprehensiveness" in scoring rather than a hard failure.
4. Also strip reasoning prefixes before parsing: some models inline chain-of-thought as <think>…</think> before the JSON (MiniMax-M3) — re.sub(r"(?is)^.*?</think>","",c) first.