news-llama: Instructions file for Claude Code

News Llama

Project Description

An AI-powered news curation engine that aggregates content from RSS, Reddit, and web search (DuckDuckGo), then summarizes the most relevant articles using a local LLM via open-agent-sdk. Runs in CLI batch mode or as a persistent web application with multi-user support and automatic daily newsletter generation. (Hacker News aggregator exists but is disabled due to empty content extraction. Twitter/X aggregator exists but is a placeholder — _collect_from_user() returns hardcoded fake tweets; tweepy is not called.)

Repository Structure

news-llama/
├── main.py                    # CLI entry point (NewsLlama class, async orchestration)
├── setup.py                   # Automated setup script
├── take_screenshots.py        # Screenshot helper for macOS app
├── requirements.txt           # 34 Python dependencies
├── alembic.ini                # Database migration configuration
├── pytest.ini                 # Test configuration (80%+ coverage target)
├── CONTRIBUTING.md            # Contribution guidelines
├── CHANGELOG.md               # Project changelog
├── LICENSE
├── src/
│   ├── aggregators/           # Source-specific aggregators
│   │   ├── base.py            # BaseAggregator abstract class
│   │   ├── rss_aggregator.py
│   │   ├── twitter_aggregator.py    # Unimplemented placeholder: returns hardcoded fake tweets, tweepy not called
│   │   ├── reddit_aggregator.py # asyncpraw with 24h smart time filtering
│   │   ├── hackernews_aggregator.py  # Disabled: empty content
│   │   └── dynamic_aggregator.py    # AI-discovered sources (Twitter delegate also disabled: delegates to placeholder)
│   ├── processors/            # Content processing
│   │   ├── content_processor.py    # Cleaning, filtering, categorization, scoring
│   │   ├── duplicate_detector.py   # Cosine similarity dedup (threshold 0.8)
│   │   └── source_discovery.py     # LLM-powered five-tier source discovery
│   ├── summarizers/           # LLM summarization via open-agent-sdk
│   │   └── llm_summarizer.py       # Batch summarization, streaming JSON-first
│   ├── generators/            # Output generation
│   │   ├── html_generator.py  # Responsive HTML with Jinja2 templates
│   │   ├── json_generator.py
│   │   └── rss_generator.py
│   ├── utils/                 # Configuration, models, utilities
│   │   ├── config.py          # Pydantic-based configuration (LLM, social, processing)
│   │   ├── models.py          # Article, SummarizedArticle dataclasses
│   │   ├── logger.py          # Logging setup (loguru)
│   │   ├── scheduler.py       # CLI mode scheduler
│   │   ├── constants.py       # Predefined interests and source patterns
│   │   ├── llm_prompts.py     # LLMPrompts class: cache-optimized prompt factory separating static system prompts from dynamic user prompts; token estimate constants (ARTICLE_SUMMARY_SYSTEM_TOKENS=250, etc.); static methods for article summary, subreddit discovery, and multi-source discovery prompts. Note: CLI-mode summarizer and source discovery reverted to inline prompts due to ROCm/AMD GPU memory issues; only web layer's direct_search_service uses this class currently
│   │   ├── image_cache.py     # Image caching
│   │   └── security.py        # Security utilities
│   └── web/                   # FastAPI web application
│       ├── app.py             # FastAPI app, route registration, lifespan management
│       ├── models.py          # SQLAlchemy ORM (8 tables: users, newsletters,
│       │                      #   user_interests, tier1_sources, source_blacklist,
│       │                      #   discovered_sources, source_health, source_contributions)
│       ├── schemas.py         # Pydantic request/response schemas
│       ├── database.py        # SQLite WAL mode + connection pooling + Alembic
│       ├── config.py          # Web app configuration
│       ├── dependencies.py    # FastAPI dependency injection (get_db, get_current_user)
│       ├── error_handlers.py  # Global error handling (no stack traces exposed)
│       ├── rate_limiter.py    # Sliding window rate limiter (10 req/min default)
│       ├── file_cache.py      # LRU cache for newsletter HTML (100 files, ~10MB cap)
│       ├── static/            # Static assets (CSS, JS, favicon, logo)
│       │   ├── styles.css
│       │   ├── avatar-manager.js
│       │   ├── interest-manager.js
│       │   ├── form-accessibility.js
│       │   ├── favicon.ico
│       │   └── logo.png
│       ├── templates/         # Jinja2 HTML templates
│       │   ├── base.html
│       │   ├── profile_select.html
│       │   ├── profile_create.html
│       │   ├── profile_settings.html
│       │   ├── calendar.html
│       │   └── metrics.html
│       ├── api/               # RESTful JSON API (v1)
│       │   ├── schemas.py     # Pydantic schemas for API v1 responses
│       │   └── v1/            # v1 routes: users.py, interests.py, newsletters.py
│       └── services/          # 15 service modules: core (user, interest, newsletter,
│                              #   generation, scheduler) + discovery (autonomous_discovery,
│                              #   direct_search, list_mining, discovery_metrics) +
│                              #   AI (tier1, llama_wrapper, llama_wrapper_tier1) +
│                              #   support (blacklist, health_check, quality_scoring)
├── NewsLlama/                 # Native macOS SwiftUI app (XcodeGen project)
├── tests/
│   ├── test_*.py              # Root-level CLI tests (5 files: models, content_processor,
│   │                          #   duplicate_detector, security, integration)
│   ├── unit/                  # Additional CLI/batch mode tests (4 files: llm_prompts,
│   │                          #   llm_summarizer_caching, main_tier1_integration, models)
│   ├── web/
│       ├── __init__.py
│   ├── web/unit/              # Web application tests (26 files + api/ subdir, 416 test functions; 528 total across all test directories)
│       ├── __init__.py
│       ├── conftest.py        # Shared fixtures (in-memory SQLite for isolation)
│       ├── api/               # API v1 endpoint tests (4 files)
│       │   ├── __init__.py
│       │   ├── test_api_users.py
│       │   ├── test_api_interests.py
│       │   └── test_api_newsletters.py
│       ├── test_user_service.py
│       ├── test_interest_service.py
│       ├── test_newsletter_service.py
│       ├── test_generation_service.py  # Newsletter generation orchestration
│       ├── test_scheduler_service.py
│       ├── test_llama_wrapper.py
│       ├── test_llama_wrapper_tier1.py
│       ├── test_autonomous_discovery_service.py
│       ├── test_direct_search_service.py
│       ├── test_list_mining_service.py
│       ├── test_discovery_metrics.py
│       ├── test_tier1_service.py
│       ├── test_blacklist_service.py
│       ├── test_quality_scoring.py
│       ├── test_health_check_service.py
│       ├── test_routes_profile.py      # Profile creation/selection/deletion routes
│       ├── test_routes_calendar.py     # Calendar view routes
│       ├── test_routes_settings.py     # Profile settings routes
│       ├── test_routes_newsletter.py   # Newsletter view/generate/retry routes
│       ├── test_routes_health.py       # Health check routes
│       ├── test_error_handlers.py      # Error handling (user-friendly messages)
│       ├── test_performance.py         # Indexes, rate limiting, LRU caching
│       ├── test_ui_states.py           # Empty/loading/error UI states
│       └── test_source_discovery_models.py
│   └── web/integration/       # Integration tests placeholder (contains only __init__.py; future use)
├── docs/                      # Architecture, deployment, user guide
├── config/                    # Configuration templates (config.example.yaml)
├── assets/                    # Static assets (logo.png)
├── screenshots/               # Demo screenshots and GIFs
├── db/                        # Alembic migration files
└── .env.example               # Environment variables template