rayomd: Instructions file for Codex

RayoMD Agent Guide

Read this file before changing the project. It records the product goal, architecture, release hygiene, and benchmark guardrails that are easy to lose between sessions.

Product Goal

RayoMD is a tiny native Markdown-to-PDF converter. The default value is:

• very small packaged binaries
• fast startup and conversion
• low memory use for normal documents
• no browser, LaTeX, or Pandoc dependency
• a compact cross-platform CLI; the desktop app lives in the separate rayomd-gui repository, which embeds the engine through rayomd::engine

Treat the native renderer as a fast Markdown subset, not a full Pandoc clone.

Release Readiness Priorities

• Keep the repository root clean: source entry points, README.md, LICENSE, CMakeLists.txt, CMakePresets.json, .editorconfig, AGENTS.md, CONTRIBUTING.md, VERSION, the main smoke document and skills/ (the Agent Skill) belong there.
• Keep source-coupled fixtures, specifications, and decision documents under docs/; keep longer user guides, dated reports, benchmark results, and research archives in the GitHub wiki. Release benchmark records are not kept in the repository.
• Keep generated build trees, generated PDFs, benchmark corpora, local binaries, caches, and one-off scratch files out of source.
• Prefer reproducible helper scripts under scripts/ over ad hoc command blobs.
• Keep README claims accurate, conservative, and easy to reproduce.
• Do not advertise native mode as full CommonMark, Pandoc, LaTeX math, or syntax-highlight compatible unless those features are actually implemented.

Current Feature Surface

Native PDF mode supports the core document features listed in README.md, including headings, paragraphs, lists (with GitHub task lists; bullets are drawn as paths by depth, AppendBulletMarker, except in quotes), block quotes and GitHub alerts, fenced code blocks, pipe tables (whose header row repeats on every page), rule lines, natively typeset math for a TeX subset, inline emphasis cleanup, clickable Markdown links (also bare URLs, email addresses and shortcut references), bookmarks and #anchor links from headings, HTML comments, <br> and character references, document metadata from the front matter (/Info title, author, subject and keywords, the catalog's /Lang; FrontMatterMetadata reads the YAML subset), opt-in page numbers, page sizes (A4 by default, presets, landscape and custom sizes; the renderers' pageW/pageH, never constants; one long page, --page-size=WIDTHxauto, PdfPageSize::autoHeight: ResolvePageSize gives the nominal height, width × √2, and what sizes content to a page (image, diagram and formula fits, MaxMathHeight, title pages, the cover) reads it as fitH; a page lays out on fullH, kMaxPageSide, from its top pageH, kLongPageTop (9,900), down to −4,500, so every coordinate stays under 10,000 and takes WriteFixed2's fast branch; what tests for room on a fresh page reads fullH, and the bottom limit textBottom starts a margin above the page's bottom (margin + (pageH - fullH), so margin bit for bit on fixed sizes, where fitH and fullH are pageH). NewPage notes each finished long page's box bottom (cold EndLongPage, a margin below the cursor; PageBottoms is nullptr otherwise): its MediaBox, page number and footer band start there), standalone local images (PNG, JPEG, and SVG drawn as vectors), and HTTP/HTTPS images on Windows or curl-enabled Linux builds with fallback text. Inline HTML (WriteHtmlTag in inline_markdown.cpp, on html.h's ScanTag): emphasis, code and link tags set counters in RunWriter::html, which only then (html.active) takes the out-of-line FlushHtml/CloseHtml; <u>/<ins> and <mark> are InlineRun::decor, which both renderers paint around a run's text (PaintDecor; the standard renderer's style bits 32 and 64); <sup>/<sub> are InlineMath::Sup/Sub runs that InlineRuns::scripts sends to the general paths, which box them in the MathPool like footnote numbers (PaintScript), and that table cells lay out as text. Elements dropped with their content and scripts skip to their close tag, so each byte is read once. HTML blocks give Block::align (2 centre, 3 right take the cold RenderAlignedParagraph/RenderAlignedHeading, each line moved by what it leaves; 0 and 1 keep the unchanged renderers), image sizes (ImageSource::widthHint/heightHint, applied before the fit to the page by ApplySizeHints) and rows (ImageSource::row, RenderImageRow: 6 points apart, one bottom per line, the alt texts joined by " · " when one image cannot be shown), and BlockType::Anchor, a #id target where the next block starts (MarkAnchor, the renderers' ids). Of style and class only page breaks are read (PageBreaks in html_blocks.cpp: page-break-*/break-* that force one, the classes page-break, pagebreak, new-page), as BlockType::PageBreak blocks before an element or at its close (HtmlState::Open::breakAfter). PdfOptions::pageBreakBeforeLevel (--page-break-before=hN) inserts PageBreak blocks before the top-level headings of those levels and their anchors (BreakBeforeHeadings, contents.cpp, before PlaceContents); renderers break only a page with something on it. Footnotes as on GitHub ([^label] and [^label]: definitions, also in quotes and list items): numbered by first reference in reading order, drawn as raised link numbers (one MathPool box each, so they take the cold formula paths and the fast paragraph path stays as it was), and set after the text at 0.85 of the body size, each note's number linking back. Only ParseMarkdown(markdown, &footnotes) reads them; footnotes.cpp holds their numbering, kept out of the parser's translation unit, where it moved GCC's inlining of the line classifier. A table with a reference shows its cells as plain text, as one with a formula does. Without definitions the PDF bytes stay as they are. --footnotes=page (PdfOptions::footnotes, FootnotePlacement::Page) sets each note at the foot of the page of its first reference. The steps are one copy for both renderers in page_notes.cpp (PageNoteCore, at -Os, beside the account PageNotes), which sees a renderer through a NoteHost (references to the fields both have, one NoteCall dispatch, PageNoteLayout<R>::Call, for what each does its own way); tiny_pdf.cpp keeps only that dispatch, the line and row hooks of the cold paths and SetNotesAside: every note is laid out once before the text (Measure, in footMode: no page breaks, the content, links, marks, images and glyph counts it made taken back) for its height and the notes it refers to; the cold paths that paint references (RenderMathLines, RenderMathTextLines with EnsureNotes, RenderMathTable rows) keep room below the line for its notes not placed yet (and the 18-point rule), which then wait at the foot (textBottom, what Ensure and the tables' bottom tests compare with: margin bit for bit outside this mode on fixed sizes, a margin above a long page's bottom; nested blocks move both with Indent/RestoreMargin, so Ensure keeps its size); NewPage (through the PageNotes::pageEnds pointer: a direct call put it in a cycle with the paint paths and moved GCC's inlining of MarkHeading), the end of a document and of each book file paint them (Paint). The measuring layout writes into the stash (PageNotes::content, kept by the thread from export to export, and FootStash's links, headings and anchors, swapped for the renderer's while it runs; one PageNotes::Info a note holds its height, state and where its layout ended). A note whose blocks after its first paragraph refer to no note (MayCut) is measured with its bottom at PageNoteBase::cut, half a page below where it is too tall for a page: once a page would end there below it, Break sets stop, at which NoteHost::cancel (the build's g_cancel) points meanwhile, so BuildCancelled ends its blocks and a note of many pages, which the end section lays out again, is not laid out whole twice; the half page is more than the room any layout keeps without filling it, so nothing changes in a note that fits. Paint replays a note from the stash (ReplayNote) under one cm translation, its links, marks and references moved by as much, instead of laying it out again; only under PDF/UA and for a note whose layout changed the images is it laid out a second time (PageNotes::Replayed), and only then is the renderer set aside (FootScope: its shared buffers traded with FootBuffers by SwapFootBuffers, so a paragraph or table row broken by the page goes on; the paragraph and table runs trade places with the thread's KeptBuffer slots, which the renderer emptied when it started, as a second destructor of those types moved GCC's inlining of the renderers' destructors). A note only notes refer to (Footnotes::referenced) goes with the first that does, at most 8 deep; a cluster too tall for a page with its line stays in the end section, which then holds only such notes. Under PDF/UA a foot note is a Note of the Document opened over the open elements (StructTree::BeginUnder/EndUnder, which keep the flow's label, table and artifact state), painted before PageBreak, so its MCIDs are its page's. End placement keeps the code path and the bytes it had. Fenced code whose info string names a language GitHub knows (29 lexers: shell and console, PowerShell, batch, JavaScript and TypeScript, Python, C, C++, C#, Java, Kotlin, Go, Rust, Swift, PHP, Ruby, SQL, HTML and XML, CSS and SCSS, JSON, YAML, TOML, INI, Dockerfile, diff) is drawn in the colours of GitHub's light theme; --no-highlight (PdfOptions::highlightCode) and blocks in no known language keep the one-colour path and its bytes. The lexers in highlight.cpp read every byte once in one forward pass; every rule is bounded, so time stays linear for any input. Keep it that way: a new rule that scans ahead must stop at the end of its line or mark what it scanned, never rescan. A ```mermaid fence (Block::codeLanguage == kMermaidCode, set by the parser's FenceLanguage) is drawn as its diagram: RenderCode asks RenderDiagram first, which takes it through ImageRegistry::AddDiagram (Mermaid::ToSvg with the renderer's SvgHost, a theme's accent colour, then ConvertSvgImage) and draws the form centred as an image, a Figure under PDF/UA with the source as its alt text. With --no-diagrams (PdfOptions::diagrams), or for a type, syntax or size the module refuses, the block takes the one-colour code path with the bytes of a fence without a language; documents without such fences are unchanged. A drawn diagram is kept for the process (diagram_cache.cpp, at most 16 MiB) by its renderer's fonts (their TtfFont::loadId), palette and source, with the text runs its form shows: a later build, such as the GUI's preview on every edit, takes the form and replays the runs into its own glyph sets (CidText), so the PDF is the one an uncached build makes (Mermaid::Layouts counts the layouts for the tests). Its containers live in that unit: inside tiny_pdf.cpp they changed what GCC inlined into PdfObjects::BuildInto, moving WrapAsciiInline 144 bytes and essay.md 2 to 4 % slower at equal instructions. A table of contents comes at the first paragraph [TOC] or [[_TOC_]] (later ones draw nothing), or with --toc (PdfOptions::toc) first or after a title heading; it lists the top-level headings to --toc-depth levels, without a document's only level-1 heading when that comes first and without headings of footnote references alone. The renderers lay out its entries in the flow and link them to the headings; the builder adds the page numbers and the dot leaders afterwards as a content stream of each table page's own, beside a theme's header and footer, so one layout pass suffices. A TOC entry finds its heading by the address of the heading's text, which HeadingMark::text keeps. --number-headings[=N] (PdfOptions::numberHeadings) numbers the headings of the top level as the table would list them (NumberHeadings, NumberBookHeadings in contents.cpp: a side table of numbers by the heading's text address, never written into Block::text); the renderers draw a numbered heading through RenderAlignedHeading with the number before the text, its mark keeping the text alone so anchors and titles do not change, the table's entries carry the number (NumberContents) and the bookmarks a numbered copy of the marks (NumberedHeadings). Obsidian's Markdown (--dialect=obsidian, PdfOptions::dialect; dialect_rewrite.cpp, at -O2): BuildPdf and BuildBookPdf keep the dialect for their thread (DialectScope), and ParseMarkdown then turns Obsidian's syntax into RayoMD's before it parses (ObsidianToCommonMark: one pass, every line kept where it was for the source map, fences, code spans, formulas and the front matter left alone; each opener's search is cached per line, so text of unclosed openers stays linear). The pass changes only lines with "[[", "==" or "%%" (or in a comment): it jumps between those pairs (FindSyntaxPair, SSE2) and copies the text between them in runs, reading the other lines only for the fences and formulas they open; within a line it skips to the next byte that can start syntax (NextSyntaxByte). A document without a pair is parsed as it is. Keep it so: a preview builds the note on every edit, and a byte-at-a-time pass at -Os made README.md 32 % slower to build in this dialect, now 2 %. Highlights become <mark>, comments go, wiki links become links to note.md#heading and so pass Html::SafeUrl as any Markdown link, embeds Markdown images or a sized <img>. Callouts are the parser's (TakeCallout): Block::alert 1 to 7 picks the bar colour, Block::text holds the title, which the renderers draw (RenderCallout); GitHub alerts keep an empty text. In a book BookTargets::FileByName resolves a link by the note's name. The GitHub dialect never runs any of it: its bytes stay as they are. The pass also leaves out Obsidian's block ids ( ^id ending a line, ^id alone): FindSyntaxPair takes a ^ after a blank or a line start as a pair, so a note without one keeps the fast path. Note embeds (embeds.cpp, vault.cpp, both -Os): under the dialect, with a sourcePath and without PdfOptions::embedsAsLinks (--embeds=link), a line ![[note]] from the first column at the top level prints the note (#heading its section, #^id its block). BuildPdf tests MayEmbedNotes (the dialect, then one find("![[")) and only then BuildWithEmbeds (cold): ResolveEmbeds reads the notes through a Vault (root PdfOptions::vaultRoot, else the note's folder, canonical; relative to the note, to the root, then by name in an index built once, at most 20,000 entries and 32 levels; on Windows no reparse point below the root, DirectPathHasLink), at most 1,000 embeds, 8 deep, 32 MiB, cycles by path and file identity, each entry's footnote labels suffixed ~eN; the fonts are chosen on all of the text (EmbedSet::all, also for GuessFallbackFont). Paths are UTF-8 strings throughout (normalized in place, Normalize, only when a name has a "." or empty component); on Linux a candidate is one openat2 beneath the root's descriptor (opened on the first note not in the cache), and a note read costs that, an fstat, one read into the string kept and a close; each folder and name is looked up once a build, and the name index is one readdir walk (Walk) into a single string buffer, ordered by an 8-byte key of each name. What a vault read is kept for the process (Kept in vault.cpp: notes as shared_ptr<const std::string>, at most kMaxNoteCacheBytes (64 MiB) and 4,096 notes, least recently used out first; the name indexes of the last four roots; resolved roots), so a preview, --serve or a batch builds the same note again for one stat a note (one GetFileAttributesExW on Windows): size, last write and status change to the nanosecond, device and inode must be as they were, a note written within SetNoteSettleTime (2 s) of its read is never kept, and only what Check accepted (beneath the root, through no link) enters; an index is kept with the identity of every folder it read and every link's target, each checked by one stat. Keep new caching of files on this rule: a file that changed, moved or became a link out of the vault must be read again on the next build (CheckNoteCache). The parse does not change for any other document: ParseObsidian takes the rewrite from RewriteDocument, which in a build in an EmbedScope first turns the document's embed lines into <!----> (a comment ends the block before it and gives none; the rewrite leaves it, so it turns no embed into a link), and ParseWithFootnotes (footnotes.cpp) hands that build to ParseWithEmbeds, which splices each entry's blocks after the parse at the first block past its line, stamped with that line (SpliceEmbeds). A test for embeds in ParseObsidian (+0.35 % instructions on notes.md) or ParseMarkdownImpl, or a field in ImageSource, moved what GCC inlined into ParseMarkdownImpl: images of an entry take its folder from a side table by address (EmbedImageBase), which LocalImagePolicy resolves inside the root; with a vault, local images may lie anywhere in it. Books of linked notes (--follow-links[=N], RayoMd::Book::ReadLinkedBook in book.cpp): breadth first through the same Vault, at most 1,000 notes and kMaxBookBytes; PdfOptions::backlinks makes BuildBookPdf append "Linked from:" paragraphs (AddBacklinks), whose links resolve as any book link (BookFileOfLink, shared with BookTargets). PdfOptions::hideTags (--hide-tags, Obsidian's dialect only; DialectScope keeps it for the thread, HiddenTags()) drops Obsidian's #tags before that pass (HideTags, TagHider in dialect_rewrite.cpp: a note with no possible tag, a '#' at a line's start or after a blank with a tag's bytes after it (NextTag), is read once by memchr and left as it is; else lines read up to the last possible tag, a line of text from the first column without further reading, only those with a possible tag or "%%" scanned, fences, display formulas, indented code, code spans, formulas, comments, wiki links and the front matter left alone; a tag goes with one space next to it and a line keeps its place; the text without tags is written into RewriteObsidian's out, which the pair-gated pass then reads). Without the flag RewriteObsidian only asks HiddenTags(), and the pair-gated pass is as it was. Print output (--black-text, PdfOptions::blackText): after layout and before CompressPageContents, both builders hand the page content, a theme's bands and cover and the table of contents' overlays to Internal::PrintColors (print_colors.cpp), as AddPageNumber its stream; every rg/RG/g/G becomes the grey of its luminance (black below 0.5) over the same bytes, padded with spaces, so no offset or length moves. Diagrams take Mermaid::kPrintPalette (the cache keys on the palette) and their form's colours the same pass; images keep theirs. A new paint needs nothing: its colour operator is rewritten with the rest. Without the flag nothing runs and the bytes stay as they are. Books (--book, BuildBookPdf) make one PDF of the files a SUMMARY.md lists (mdBook, GitBook), of a folder, or of the files given. Each file is parsed on its own (its link definitions, footnotes and heading anchors are its own) and begins on a new page, so the page of a link, a heading mark or a note mark tells its file: BookTargets resolves #fragment within the file and b.md#fragment across files with no file field on the renderers' structures. Single documents pass book == nullptr through the builders and must keep their bytes; book-only work stays in RAYOMD_COLD code and book.cpp. The files a SUMMARY.md lists or a book folder holds must lie in that folder once links and .. are resolved (ChapterReader's root; GetFinalPathNameByHandleW on Windows, whose standard library resolves no link): a book from elsewhere must not put the reader's files into its PDF. Files named on the command line are the user's choice and are not confined. Opt-in compression (--compress, PdfOptions::compress) writes page content, the font program and both CMaps as FlateDecode where that makes the file smaller, with RayoMD's own DEFLATE encoder; never the XMP metadata or an embedded source, which the reversible profile reads uncompressed. Without it the PDF bytes stay as they are. Company themes (--theme=FILE, PdfOptions::theme): a TrueType or OpenType font for all text (no fallback font then), with optional bold, italic and bold italic faces (/F2, /F4, /F5; the Unicode renderer's faces, ChooseFace): with any of them multiFace is set, paragraphs take the general WrapStyled path, and every width and paint of styled text goes through SpanFont/PaintSpanText, so measure and paint use the same face; without them every one of those calls is font/PaintText as before. Heading, link and accent colours (the renderers' headingColor/linkColor/ruleColor/plainQuoteBar, from ThemePalette, never literals), header and footer fields with placeholders and a logo (one overlay stream per page, like the page numbers, whose glyphs join the subset before it is cut), and a cover page before the first page. Without a theme the PDF bytes stay as they are. PDF/A-3b (--pdfa, PdfOptions::pdfa) takes the Unicode renderer for all text, as every font must be embedded: page numbers become a footer band in the document's font, formulas are typeset in the embedded formula fonts as in any export, and without a TrueType or OpenType font the export fails (PdfaFontUnavailable) instead of falling back to the standard fonts. It adds an sRGB output intent, XMP metadata that tells what the Info dictionary does (PdfaText makes both hold the same text), a trailer /ID and /F 4 on links, and stays PDF 1.7 with an embedded source. Check changes with veraPDF (verapdf -f 3b). Without it the PDF bytes stay as they are. PdfOptions::pdfua (--pdfua) writes tagged PDF/UA-1: the fonts as with pdfa, formulas as their TeX source (MathPool::Disable: they have no structure tags yet), and a structure tree (src/core/tagged.h) that the Unicode renderer fills as it paints. Every paint of that renderer goes through it: PaintText and DrawImage mark content of the open element, DrawRect, DrawStrokeRect and rules mark artifacts, and every block opens its element. A new paint call in the Unicode renderer must do the same, or veraPDF fails with "content neither marked as Artifact nor tagged" (verapdf -f ua1), and Debug builds assert. Untagged documents keep their bytes: each hook tests tags first, and the tagged-only work is in RAYOMD_COLD functions and tagged.cpp. PdfOptions::sourceMap (rayomd-gui keeps its editor and preview together with it) gets each top-level block's source lines and where it is laid out (SourceBlock: pages counted with a theme's cover, y down from the page's top edge). The parser stamps the lines at depth 0 only (Block::sourceLine/sourceLines, in the padding after align; StampSourceLines), and RenderBlocks notes each stamped block after drawing it (NoteSourceBlock, cold); inserted blocks (the --toc table, a book's titles), nested ones and footnotes have no entry, and books never fill it. The hooks only read layout state: the PDF bytes are the same with or without a map (CheckSourceMap). TinyPdf::BuildSession (rayomd-gui's preview on every edit; src/core/build_session.h and .cpp, design and numbers in docs/development/incremental_builds.md) gives the bytes, result and source map BuildPdf gives and takes the layout of the top-level blocks an edit left alone from its last build: a build of a session is BuildPdf with the session set for the thread (CurrentLayoutSession); the builders lay its blocks out one at a time (SessionAccess::Render, RenderBlockRange, reached through SessionCalls; compiled only with RAYOMD_BUILD_SESSION, which the engine library and the core tests set and the command lines do not, so they carry none of it), note the renderer's state before each (LayoutMark) and replay a run of blocks equal to the last build's from the same state (page content, pages, links, headings, anchors, contents lines, source map entries, CIDs and formula glyphs). A new piece of renderer state that a later block reads goes into LayoutMark and SessionAccess (compared where a run starts, set where it ends), a new field of Block into SameBlock, a new option into SameOptions (static_asserts on their sizes catch it); anything that a block's page content depends on beyond its block and that state (as footnote numbers, image names, PDF/UA's tags) must make the session lay those blocks out, or the whole document. Ordinary builds pay one test of the thread's session per build. Core check 137 (tests/session_tests.inc) compares random edits built both ways. A session's parse goes on from its last build's (ParseInSession, src/core/session_parse.h): the builders parse into the session's blocks (BuildBlocks), and the session's copy of the level parse notes its block loop's state at every pass (DocumentPass). The level parse's body is src/core/markdown_parser_level.inc: ParseMarkdownImpl includes it without hooks (RAYOMD_SESSION_LEVEL 0), and src/core/session_parse.cpp, which compiles markdown_parser.cpp a second time in the engine library only (its external functions renamed), as ParseSessionDocument with them; hooks in the plain copy, or both copies in one unit, cost BuildPdf 0.4 to 2 % (round 5 in performance.md). A new external function of markdown_parser.cpp must be renamed there too. New state that the loop carries from one pass to the next (beyond the line, the open HTML elements and the ParseBudget) must go into ParsePass and be compared there, and anything a later pass reads before its own line, or a transformation of the blocks after the parse, must make the session parse whole. The session's copy classifies only the lines an edit changed (ClassifyInSession): the others are the last parse's, shifted (ParseLines in the record), and ResolveMathOpeners and the definitions' pass run again between blank lines with no fence open. A new pass over the lines before the block loop, or a line class that depends on lines past a blank line, must go there too (or make it classify whole). The tape keeps the page content in pieces of buffers its tapes share (LayoutTape::pieces, without the page gaps); nothing outside build_session.cpp reads it. A session builds from its own copy of the text (SessionText, KeptText in build_session.cpp), changed in place where the next text differs, with its WinAnsi text: the parse record views it, the font choice reads counts kept with it (bytes from 0x80, CharacterReferenceNeedAt, "]:"). A new test of ChooseFonts' over the text, or a new rule of TranscodeToWinAnsi that reads past a character, must go into SessionText::Fonts too. A session's assembly takes the last build's bytes of link annotations and outline entries by what made them (AssemblyTape): a new input of their bytes goes into its keys (SessionAnnotations::Add, TakeOutlineEntry), and a page dictionary's text before /Contents must stay the same on every page of fixed size (NotePagePrefix). The in-place replay trusts only bytes no laid-out block wrote near: keep it before the first one. Session-only code in tiny_pdf.cpp stays behind RAYOMD_BUILD_SESSION, with the command line's code unchanged (cmp of rayomd against master's build). Native exports can opt into the rayomd-source/1 reversible PDF profile. Embedding is disabled by default because it exposes the complete source, including content not visible on rendered pages. Recovery is byte-exact and format-specific; it must never silently fall back to heuristic PDF conversion. Keep ordinary non-reversible exports on their unchanged PDF 1.7 fast path. The profile limits PDFs to 256 MiB and source to 10 MiB. The source cap is based on the measured maximum exact-recovery case; reject larger inputs before rendering to avoid excessive output and peak memory.