04-discourse-parser

Agent 4: Discourse Parser

Role

Constructs a hierarchical RST (Rhetorical Structure Theory) discourse tree for each paper section. Each tree captures how paragraphs relate to each other (elaboration, contrast, evidence, etc.), enabling discourse-guided slide grouping downstream.

Position in Pipeline

• Phase: 2 (Narrative Outline Generation)
• Depends on: A1 output (workspace/markdown.md)
• Produces for: A5 (Slide Planner)
• LLM Required: Yes
• Execution: Per-section (loop over all sections in the paper)

Inputs

Input	Source	Format
Paper markdown	workspace/markdown.md	Plain text

Preprocessing Steps (before LLM calls)

Step 1: Split Markdown into Sections

1. Split at ## headings (level-2 headers)
2. Group subsections (e.g., ### 3.1, ### 3.2) under their parent section
3. Remove the "References" section and everything below it
4. Create a section key for each: slugify the heading (e.g., "Related Work" → related_work)

Step 2: Clean Text for RST Parsing

For each section's text:

• Remove code blocks (...)
• Remove image references (![...](...)), links ([text](url))
• Remove bold/italic markers (**, *, __, _)
• Strip inline LaTeX
• Normalize whitespace

Step 3: Split into Paragraphs

For each section:

1. Split cleaned text at double-newline boundaries
2. Skip paragraphs shorter than 20 characters
3. Name each paragraph: {section_key}_{index} (e.g., introduction_0, introduction_1)
4. Save to workspace/rst/{section_key}/paragraphs.json

Paragraph JSON format:

{
  "introduction_0": "We present ArcDeck, a narrative-driven system...",
  "introduction_1": "Prior work on paper-to-slides conversion...",
  "introduction_2": "Our key insight is that rhetorical structure..."
}

Step 4: Skip sections with fewer than 2 paragraphs

RST requires at least 2 EDUs (paragraphs) to build a tree. For single-paragraph sections, create a trivial tree with just that paragraph as root.