arrow-columnar

sparq-arrow — Arrow, Parquet, IPC, and CSV interop

Use sparq-arrow when a Rust application needs a faithful columnar representation of a sparq_engine::QueryResult. The crate maps every SELECT variable to one nullable Arrow Struct column with five nullable UTF-8 children: kind, value, datatype, language, and direction.

Choose a feature

• Enable arrow for to_record_batch, from_record_batch, term_schema, and term_struct_type.
• Enable parquet for to_parquet_bytes, from_parquet_bytes, parquet_variables_from_bytes, and parquet_row_count_from_bytes; it implies arrow.
• Enable ipc for to_ipc_bytes, from_ipc_bytes, and ipc_variables_from_bytes; it implies arrow.
• Enable csv for to_csv_bytes, from_csv_bytes, and csv_variables_from_bytes; it implies arrow and adds no further dependencies (the RFC 4180 serializer and parser are hand-rolled over std).
• Leave all features disabled to retain only the dependency-free field-name constants.

All features are default-OFF. The crate is a leaf capability crate, so no Arrow container dependency enters sparq-core, sparq-engine, or the WebAssembly bundle.

Use a RecordBatch

use sparq_arrow::{from_record_batch, to_record_batch};

let batch = to_record_batch(&result)?;
let restored = from_record_batch(&batch)?;
assert_eq!(restored.vars, result.vars);
assert_eq!(restored.rows, result.rows);
# Ok::<(), Box<dyn std::error::Error>>(())

Add the feature with cargo add sparq-arrow --features arrow.

Use Parquet bytes

use sparq_arrow::{
    from_parquet_bytes, parquet_row_count_from_bytes, parquet_variables_from_bytes,
    to_parquet_bytes,
};

let bytes: Vec<u8> = to_parquet_bytes(&result)?;
let variables = parquet_variables_from_bytes(&bytes)?;
let row_count = parquet_row_count_from_bytes(&bytes)?;
let restored = from_parquet_bytes(&bytes)?;
assert_eq!(variables, result.vars);
assert_eq!(row_count, result.rows.len());
assert_eq!(restored.vars, result.vars);
assert_eq!(restored.rows, result.rows);
# Ok::<(), Box<dyn std::error::Error>>(())