vaex

Vaex

Overview

Vaex is a high-performance Python library designed for lazy, out-of-core DataFrames to process and visualize tabular datasets that are too large to fit into RAM. Vaex can process over a billion rows per second, enabling interactive data exploration and analysis on datasets with billions of rows.

When to Use This Skill

Use Vaex when:

• Processing tabular datasets larger than available RAM (gigabytes to terabytes)
• Performing fast statistical aggregations on massive datasets
• Creating visualizations and heatmaps of large datasets
• Building machine learning pipelines on big data
• Converting between data formats (CSV, HDF5, Arrow, Parquet)
• Needing lazy evaluation and virtual columns to avoid memory overhead
• Working with astronomical data, financial time series, or other large-scale scientific datasets

Core Capabilities

Vaex provides six primary capability areas, each documented in detail in the references directory:

1. DataFrames and Data Loading

Load and create Vaex DataFrames from various sources including files (HDF5, CSV, Arrow, Parquet), pandas DataFrames, NumPy arrays, and dictionaries. Reference references/core_dataframes.md for:

• Opening large files efficiently
• Converting from pandas/NumPy/Arrow
• Working with example datasets
• Understanding DataFrame structure

2. Data Processing and Manipulation

Perform filtering, create virtual columns, use expressions, and aggregate data without loading everything into memory. Reference references/data_processing.md for:

• Filtering and selections
• Virtual columns and expressions
• Groupby operations and aggregations
• String operations and datetime handling
• Working with missing data

3. Performance and Optimization

Leverage Vaex's lazy evaluation, caching strategies, and memory-efficient operations. Reference references/performance.md for:

• Understanding lazy evaluation
• Using delay=True for batching operations
• Materializing columns when needed
• Caching strategies
• Asynchronous operations