LogLead CLAUDE.md

CLAUDE.md

This file provides guidance to coding agents when working with code in this repository.

Project overview

LogLead (Log Loader, Enhancer, Anomaly Detector) is a Python library for benchmarking log anomaly detection algorithms and log representations. It provides custom loaders for ~10 public log datasets, ~11 log representation "enhancers" (parsers, tokenizers, n-grams, embeddings), and ~11 anomaly detection classifiers, so a given dataset/representation/classifier combination can be swapped independently. It is also used as a backend library by the sibling projects LogDelta and VisualLogAnalyzer, so changes to public APIs in loglead/ can affect those consumers.

Data is represented as Polars DataFrames throughout (not Pandas), chosen for speed.

Environment setup

• Python 3.9–3.12 (.python-version pins 3.11 for local dev). Dependency/venv management is via uv; uv run <script> syncs the environment from pyproject.toml/uv.lock automatically, installing loglead itself editable into .venv — there is no separate install step, and no need to fiddle with sys.path to make import loglead work.
• A .env file is not required for the normal uv workflow (there isn't one checked in, and none is needed for the smoke demos or tests/main.py). It only matters if you're pointing scripts at your own full-size dataset copies on disk:
  • LOG_DATA_PATH is read (via python-dotenv) by the "bring your own full dataset" scripts — demo/RawLoader_*, demo/parser_benchmark/*, demo/saner_2024_paper/*, demo/unsupervised_models.py. The quick demos (demo/HDFS_samples.py, demo/TB_samples.py) use bundled sample parquet files instead and never touch it. The downloader also doesn't use it — downloader/download_data.py reads root_folder from the YAML config (downloader/datasets.yml or one of tests/datasets_*.yml) instead.
  • See .env.sample for the format if you do need LOG_DATA_PATH.
• There are two independent, unlinked ways to point tooling at a data directory on disk — nothing in the code cross-references them, so keeping them in sync (e.g. both pointing at ~/Datasets) is on you:
  • LOG_DATA_PATH in .env — used only by the demo scripts listed above.
  • root_folder in a dataset YAML config — used only by downloader/download_data.py, optionally overridden by its --location CLI flag. A test config's local_copy_folder (see Common commands below) links it back to downloader/datasets.yml's root_folder, but that's opt-in per config — it does not make root_folder itself a shared setting.
• scikit-learn needs gcc/g++ to build. The pip-installed package does not pull in tensorflow, so BertEmbeddings (loglead/parsers/bert/) must have TF installed manually to work.