kinesis-stream

aws.kinesis.stream (Kinesis Data Stream)

Managed, ordered, replayable event stream — the deployable streaming primitive. Records are retained (default 24h, up to 365 days) and each consumer reads independently at its own offset, so the same events can feed many pipelines and be re-read after a bug fix. Config knobs: streamMode, retentionHours.

When to use

• Event sourcing, activity/clickstream firehoses, fan-in from many producers to many independent consumers, or any "durable log I can replay" need. Pairs with Lambda (event-source mapping) or ECS consumers.

vs other messaging primitives

• vs aws.sqs.queue: SQS is a work queue — one consumer deletes each message, no replay. Kinesis is a log — many consumers, ordered per shard, replayable. Use SQS for task distribution; Kinesis when you need ordering + replay + multiple independent readers.
• vs aws.sns.topic: SNS fans out a message to subscribers at publish time with no retention. Kinesis retains, so a consumer added later (or recovering) can read history.
• vs aws.msk.cluster (Kafka, design-only): Kinesis covers most streaming needs and deploys today. Reach for MSK only when the app specifically needs Kafka API/client compatibility or Kafka-only semantics — and note that MSK isn't deployable yet. Prefer Kinesis over design-only MSK.

Config

• streamMode ON_DEMAND (default — no capacity planning, scales automatically) or PROVISIONED; retentionHours (raise it when consumers need a longer replay window). Encrypted at rest by default (KMS).

In our catalog

• v1 limitation: PROVISIONED mode ships a single shard (~1MB/s in, 2MB/s out); raise it in the template for higher throughput.
• Edge kinds: iam producer/consumer → kinesis (compute gets kinesis:PutRecord* / kinesis:GetRecords + {PREFIX}_STREAM_NAME env). Lambda consumers also use an event-source mapping (wire queue→lambda-style intent in agentNotes).
• agentNotes: pick a good partition key (even distribution, preserves per-key ordering); consumers must checkpoint their position and handle reprocessing idempotently.