csv-over-json-tables

Tabular Serialization Protocol (CSV / TSV over JSON Arrays)

Overview

When an LLM extracts or analyzes tabular datasets (e.g., customer transaction lists, database query results, or server metric logs), defaulting to an array of JSON objects forces the model to repeat every column key on every single row ("first_name": "...", "transaction_amount": "...", "status": "...").

In a 100-row dataset, repeating JSON keys burns thousands of redundant tokens purely on structural quotation marks, colons, and braces.

The Tabular Serialization Protocol replaces JSON object arrays with Comma-Separated Values (CSV) or Tab-Separated Values (TSV) - defining the header keys once at row 1 and streaming raw comma-delimited data rows, cutting token usage by 65% to 75%.

---

JSON Object Array vs. CSV Data Stream

┌─────────────────────────────────────────────────────────────┐
│                 Tabular Payload Comparison                  │
│                                                             │
│  JSON Object Array (145 Tokens for 2 Rows):                 │
│  [                                                          │
│    {"user_id": 101, "email": "[email protected]", "tier": "pro"}│
│    {"user_id": 102, "email": "[email protected]", "tier": "free"} │
│  ]                                                          │
│  ↳ Keys `user_id`, `email`, `tier` repeated on every row!   │
│                                                             │
│  CSV Data Stream (34 Tokens - 76.5% Reduction!):            │
│  user_id,email,tier                                         │
│  101,[email protected],pro                                      │
│  102,[email protected],free                                       │
│  ↳ Keys defined ONCE. Zero quotation/brace syntax waste.    │
└─────────────────────────────────────────────────────────────┘

---

The Master CSV Extraction Prompt Template

When querying an LLM to extract or output structured rows:

Extract the customer records from the text below:
<source_data>
[PASTE UNSTRUCTURED TEXT]
</source_data>

Output Constraints:
- Format as raw **CSV (Comma-Separated Values)**.
- Line 1 MUST be the exact header row: `user_id,name,email,plan_tier,monthly_spend`
- Do NOT output JSON.
- Output ONLY the CSV block; zero introductory or concluding commentary.