format-implementation

Format Implementation Guide

Adding New Formats

Step-by-Step Process

1. Create format file: iterable/datatypes/<format>.py
2. Implement BaseIterable: Inherit from BaseIterable in iterable/base.py
3. Required methods: read(), write(), read_bulk(), write_bulk(), etc.
4. Add detection: Update iterable/helpers/detect.py
5. Create tests: tests/test_<format>.py
6. Update dependencies: Add optional dependency to pyproject.toml if needed
7. Update documentation: Add format to docs

Implementation Pattern

from iterable.base import BaseIterable

class NewFormatIterable(BaseIterable):
    def __init__(self, source, mode='r', **kwargs):
        super().__init__(source, mode, **kwargs)
        # Initialize format-specific resources
    
    def read(self):
        # Return iterator of dict objects
        pass
    
    def write(self, data):
        # Write dict objects to file
        pass
    
    def read_bulk(self, size=1000):
        # Bulk read for performance
        pass
    
    def write_bulk(self, data):
        # Bulk write for performance
        pass

Format Detection

Update iterable/helpers/detect.py:

1. Add file extension detection
2. Add magic number detection (for binary formats)
3. Add content-based heuristics if needed
4. Update detect_file_type() function

Example:

def detect_file_type(filename, content=None):
    # Check extension
    if filename.endswith('.newformat'):
        return 'newformat'
    
    # Check magic numbers
    if content and content.startswith(b'MAGIC'):
        return 'newformat'
    
    # ... existing detection logic

Adding New Codecs

Step-by-Step Process

1. Create codec file: iterable/codecs/<codec>codec.py
2. Implement codec class: read(), write(), close() methods
3. Add detection: Update iterable/helpers/detect.py
4. Add compression detection: Update format detection logic
5. Create tests: Add to relevant test file or create new one
6. Update dependencies: Add optional dependency to pyproject.toml