tree-sitter-parse-testing

Tree-Sitter Parse Testing

Overview

Use tree-sitter parse -d to debug parsing issues. The debug output shows the state machine transitions, helping identify where parsing fails.

Parsing a Code Snippet

Run from the parser directory (fsharp/ or fsharp_signature/):

echo 'let x = 1' | npx tree-sitter parse -d

Interpreting Output

Successful Parse

(file [0, 0] - [1, 0]
  (declaration_expression [0, 0] - [0, 9]
    ...))

• Node format: (nodename [start_row, start_col] - [end_row, end_col])
• Children indented inside parent nodes

Parse Errors

(ERROR [0, 0] - [0, 7]
  ...)

• ERROR nodes indicate parsing failure at that location

Missing Tokens

(MISSING identifier [0, 8] - [0, 8])

• MISSING indicates expected token that wasn't found

Debug Output Analysis

Key patterns to look for:

Pattern	Meaning
detect_error	Parser encountered unexpected token
recover_with_missing	Parser inserted placeholder to recover
recover_with_error	Parser skipped tokens to recover
shift state:N	Parser shifted to state N
reduce sym:X	Parser reduced using rule X
accept	Parse completed successfully

Debugging Workflow

1. Run parse without -d to see the tree
2. If tree looks wrong (ERROR nodes, unexpected structure):
3. Run with -d to see the debug output
4. Look for detect_error near the problem location
5. Check which state the parser was in when error occurred
6. Compare expected tokens at that state against actual input

Infinite Loops

If the debug output repeatedly emits the same token or state indefinitely, the parser has entered an infinite loop. This typically happens when:

• A grammar rule injects tokens without consuming any input
• The scanner (src/scanner.c) is returning tokens that don't advance the parse position

** Symptoms:**

• Debug output shows the same token being processed repeatedly
• The parser never reaches accept or an error state
• Command appears to hang