thai-text-processing

Thai Text Processing (Segmentation, Normalization, Collation, Romanization)

Overview

Thai script has no spaces between words within a sentence — "ฉันรักการเขียนโค้ด".split(" ") returns the entire string as one token, not a word list, because there is nothing to split on. A model without this skill reaches for whitespace-based tokenization out of habit (it works for English) and silently produces garbage for Thai. This skill flags that failure mode and points to the correct tools — it does not reimplement a Thai tokenizer or transliterator, since both need a maintained dictionary/model to be accurate, not hand-written rules.

When to use

• ตัดคำภาษาไทยเป็นคำๆ (word segmentation/tokenization) สำหรับ search, NLP, หรือ text analytics
• Normalizing Thai text before comparison, search, deduplication, or storage (Unicode NFC)
• Sorting Thai strings correctly (Thai collation order differs from raw Unicode codepoint order)
• Romanizing Thai text to Latin script (e.g. for a passport-style name, a street sign, or an international form) using the RTGS (Royal Thai General System of Transcription) convention

When NOT to use

• Plain reading/writing/translating Thai text with no segmentation, sorting, or comparison step involved
• The romanization only needs to be "readable," not standards-compliant (e.g. casual transliteration in a chat) — RTGS precision isn't necessary there

Core knowledge

Word segmentation: Thai does not use spaces between words (spaces mark clause/sentence boundaries, not word boundaries). Segmentation requires a dictionary- or model-based tokenizer — PyThaiNLP (pip install pythainlp, pythainlp.tokenize.word_tokenize) is the standard open-source option. This skill does not vendor or reimplement a tokenizer; a hand-rolled rule-based splitter would silently produce wrong word boundaries on real text.

Unicode normalization (NFC): Thai vowels and tone marks are combining Unicode characters. The same visible text can be encoded as a single precomposed codepoint or as a base character plus a combining mark — two strings that look identical can compare unequal, fail a lookup, or break deduplication if they aren't normalized to the same form first. Normalize to NFC (unicodedata.normalize("NFC", text), pure Python stdlib, no dependency needed) before comparing, searching, hashing, or storing user-submitted Thai text.