processing-s3-uploads-with-step-functions

Step Functions Workflow: Route S3 Uploads to Lambda or Fargate

Overview

This skill deploys an event-driven workflow using AWS CLI. When a file is uploaded to an S3 bucket, EventBridge triggers a Step Functions state machine. The state machine checks the file size and routes processing to either a Lambda function (files ≤ 6 MB) or a Fargate task (files > 6 MB).

The architecture includes:

• An S3 bucket with EventBridge notifications enabled
• An EventBridge rule that triggers Step Functions on S3 object creation
• A Step Functions state machine with a Choice state for routing
• A Lambda function for processing small files
• An ECS Fargate task for processing large files
• A VPC with two subnets, internet gateway, and security group
• An ECR repository for the Fargate container image
• Scoped IAM roles for Lambda, Step Functions, and ECS tasks

Use this skill when:

• You need to process S3 uploads with different compute based on file size
• You want a serverless workflow that can handle both small and large files
• You need Step Functions orchestration with Lambda and Fargate

Do not use this skill when:

• All files are small enough for Lambda (use S3 → Lambda directly)
• You need real-time streaming (use Kinesis)
• You don't need file-size-based routing

Prerequisites

1. AWS CLI v2 — Installed and configured. Verify with aws sts get-caller-identity.
2. Python 3.12 — For the Lambda function runtime.
3. Docker — For building and pushing the Fargate container image.

Parameters

• bucket_name (required): Name for the S3 bucket (globally unique, lowercase, 3-63 characters)
• region (required): AWS region for all resources
• ecr_repo_name (required): Name for the ECR repository
• state_machine_name (required): Name for the Step Functions state machine
• kms_key_arn (optional): ARN of a KMS key for CloudWatch Logs encryption. If not provided, create one with aws kms create-key --description "Key for CloudWatch Logs encryption" --region {region}