fern-docs-generation

Fern Documentation Generation System - Internal Guide

Overview

The custom internal Fern documentation generator automatically creates MDX documentation for Airweave connectors by introspecting Python source code using AST (Abstract Syntax Tree) parsing.

Architecture Flow

1. Discovery Phase

Entry: update_connector_docs/__main__.py:main()

• Scans backend/airweave/platform/sources/ directory
• Identifies all .py files as potential connectors
• Example: asana.py, slack.py, google_drive.py

2. Parsing Phase

Source Parser (source_parser.py)

Extracts metadata from @source decorator:

@source(
    name="Asana",                    # Display name
    short_name="asana",              # File/URL identifier
    auth_methods=[...],              # Auth methods list
    oauth_type=OAuthType.WITH_REFRESH,  # OAuth type enum
    auth_config_class="AsanaConfig", # String reference to config
    config_class="AsanaConfig",      # Source-specific config
    labels=["Project Management"]    # Categories
)

AST Extraction Process:

1. Parse decorator arguments positionally and by keyword
2. Fall back to class attributes (_auth_type, _config_class)
3. Last resort: regex pattern matching in raw source

Entity Parser (entity_parser.py)

• Scans platform/entities/{connector_name}.py
• Extracts entity classes and their field definitions
• Builds entity hierarchy for documentation

Auth/Config Parsers

• auth_parser.py: Maps auth config class names to their field definitions
• config_parser.py: Maps source config classes to their field requirements

3. Generation Phase

MDX Generator (mdx_generator.py)

Creates structured MDX with:

• Header: Icon + connector name
• Configuration: Source docstring description
• Authentication:
  • OAuth flows (managed vs BYOC)
  • Direct auth field requirements
• Source Config: Additional configuration fields
• Entity Schema: Data structure documentation