s3dlio: Instructions file for GitHub Copilot

s3dlio AI Coding Agent Instructions

---

🚨 Prime Directive

Prime Directive: If told to investigate something, I will ONLY investigate, and then report and possibly document my findings. I will NEVER assume to proceed to making ANY code changes, unless SPECIFICALLY asked to do so.

Secondary Directive: Whenever given a task, first check that I am not violating my Prime Directive before proceeding.

---

Project Overview

s3dlio is a high-performance, multi-protocol storage library built in Rust with Python bindings, designed for AI/ML workloads. It provides universal copy operations across S3, Azure, local file systems, and DirectIO with near line-speed performance.

Current Version: v0.9.5 (October 2025)

Performance Targets

• Read (GET): Minimum 5 GB/s (50 Gb/s) sustained, target higher
• Write (PUT): Minimum 2.5 GB/s (25 Gb/s) sustained, target higher
• Infrastructure: Tested against Vast storage systems with bonded 100 Gb ports
• S3 Compatibility: MinIO, Vast, AWS S3, and other S3-compatible storage systems

Core Architecture

Dual-Backend System

The project uses mutually exclusive backend selection at compile-time:

• native-backends feature: AWS SDK + Azure SDK (DEFAULT - recommended for production)
• arrow-backend feature: Apache Arrow object_store implementation (experimental, optional)

# Default build (uses native-backends)
cargo build --release

# Explicit native-backends (RECOMMENDED)
cargo build --no-default-features --features native-backends

# Experimental arrow backend (NOT RECOMMENDED for production)
cargo build --no-default-features --features arrow-backend

Critical: These features are mutually exclusive by design (compile_error! in src/lib.rs).

Backend Status:

• native-backends: Default, proven performance (5+ GB/s reads, 2.5+ GB/s writes), production-ready
• arrow-backend: Experimental only, no proven performance benefit, kept for comparison testing

Build Quality Standards