data-distributed-compute

Data Distributed Compute

Purpose

Design and tune distributed compute systems for large-scale data processing. Select the right framework (Spark, Dask, Ray, MapReduce), configure YARN/K8s resource management, optimize shuffle and data locality, and tune executors for throughput.

Agent Protocol

Trigger

Exact user phrases: "Hadoop MapReduce", "Spark", "Dask", "Ray", "YARN", "cluster computing", "resource manager", "shuffle", "data locality", "executor", "worker", "task scheduling", "distributed compute", "cluster mode", "dynamic allocation", "speculative execution".

Input Context

Before activating, verify:

• Compute framework preference (Spark, Dask, Ray, MapReduce)
• Data size and shape (TB per run, row counts, join complexity)
• Cluster size and resource per node (cores, memory, network)
• Workload type (batch ETL, ML training, real-time inference, iterative algorithms)
• Storage backend (HDFS, S3, local SSD)
• Scheduling layer (YARN, K8s, standalone)

Output Artifact

Distributed compute architecture with framework selection, cluster configuration, and tuning parameters.

Response Format

Compute Framework: {Spark | Dask | Ray | MapReduce}
Cluster Mode: {YARN | K8s | Standalone | Slurm}
Execution Model: {driver-executor | scheduler-worker | GCS}
Resource: {N executors x M cores x G memory}
Shuffle: {sort-based | hash-based | external}
Locality: {PROCESS_LOCAL | NODE_LOCAL | RACK_LOCAL | ANY}

# spark-submit or Ray cluster config
# Tuning parameters

No preamble. No postamble. No explanations. No filler/hedging/transitions. Compress output — why use many token when few do trick.

Completion Criteria

• Framework selected with trade-off analysis
• Cluster resource config calculated (executors, cores, memory, overhead)
• Shuffle strategy defined with spill/tune settings
• Data locality configuration set
• Speculative execution policy defined
• Dynamic allocation or static partitioning configured