All cheat sheets
Databasescheat sheet

MSK Serverless

MSK Serverless is Apache Kafka billed on throughput (MB-hour) and storage (GB-month) instead of 24/7 brokers — often ~97% cheaper than provisioned at low/variable volume. Same Kafka APIs; the catch is fixed per-partition throughput caps.

Last reviewed: July 14, 2026

Prefer step by step? Take the guided MSK Serverless walkthrough — free, anonymous, interactive.Have an AI assistant? Jump to the copy-paste audit prompt at the bottom of this sheet.

TL;DR: MSK Serverless (2022) is the first true pay-per-use Kafka in AWS: no brokers to size, no 24/7 idle capacity — you're billed on throughput (MB-hour) and storage (GB-month). It's still Apache Kafka (same APIs, clients, consumer groups), so apps need no code changes. At low or variable volume it's routinely ~97% cheaper than running provisioned brokers; at sustained terabyte-scale the math reverses and provisioned + reserved wins.

The numbers

  • Two line items: ingress ~$0.10/GB, egress ~$0.05/GB, storage ~$0.10/GB-month — no hourly broker fee.
  • Worked example — medium SaaS logs, 50 GB in / 100 GB out / 80 GB retained: serverless = $5 + $5 + $8 = ~$18/mo vs 3× kafka.m5.large provisioned ≈ ~$550/mo running 24/7 regardless of volume — ~97% cheaper.
  • The first cap you hit: per-partition ~5 MB/s ingress, ~10 MB/s egress — serverless scales the cluster, not a single partition; the fix is more partitions.
  • Field examples: a 4-person analytics startup ran $30–50/mo vs $500+ provisioned; an enterprise went hybrid (serverless for dev/test + low-volume topics, provisioned for mission-critical high-throughput streams) and got ~10 ops-hours/week back.

Do this

  1. Start serverless by default — create the cluster, pick VPC/subnets, get a bootstrap endpoint, point producers/consumers at it. Most teams overestimate their throughput.
  2. Spread load across partitions — if a logical topic needs >5 MB/s ingress, add partitions rather than reaching for a bigger broker (there isn't one).
  3. Use IAM auth — natively supported and far easier than rotating SASL creds or SSL certs by hand.
  4. Set retention to your real replay/audit window — indefinite retention quietly inflates the storage line.
  5. Watch Cost Explorer as volume grows — serverless cost scales linearly with throughput; there are no "free broker hours" to absorb growth, so know where the cross-over to provisioned sits.

Gotchas

  • Per-partition throttling, not auto-scaling magic — push 15 MB/s into one partition and you get throttled; serverless never splits partitions for you.
  • Cold-start latency — a cluster idle for a while can see higher latency on the first requests of a burst; fine for logs/analytics, a risk for 24/7 single-digit-ms p99s.
  • Locked-down broker configs — serverless hides many broker-level knobs; custom tuning and very high partition counts may not fit.
  • Linear cost, no free tier of capacity — cheap at low/moderate volume, but at terabyte-scale 24/7 the flat provisioned + reserved model wins.

Skip this if

  • You stream sustained massive throughput (terabytes/day, 24/7) — provisioned MSK with reserved capacity is cheaper; run the numbers.
  • You need ultra-low-latency p99s or custom broker configs — fine-tuned provisioned brokers win. For the always-on side, run provisioned MSK; and right-size the Lambda consumers processing your Kafka events.

Run this audit with your AI assistant

Paste this into Claude, ChatGPT, or any agent that can run the AWS CLI with read-only credentials. It audits your account for exactly the waste this sheet describes — and changes nothing.

You are auditing an AWS account's Amazon MSK usage to decide between
provisioned and serverless Kafka. Use the AWS CLI with READ-ONLY
credentials. Do not create, modify, or delete anything — report findings
and recommended (unapplied) fixes only.

1. Inventory: aws kafka list-clusters-v2 — capture cluster type
   (PROVISIONED vs SERVERLESS), broker instance type/count, and storage
   for provisioned clusters.
2. Throughput shape: pull CloudWatch AWS/Kafka BytesInPerSec /
   BytesOutPerSec hourly over 30 days. Classify: variable/spiky/low
   (serverless wins), or sustained high (terabytes/day — provisioned +
   reserved wins).
3. Cost compare: provisioned = broker $/hr × count × 730 + EBS. Serverless
   = ingress GB × $0.10 + egress GB × $0.05 + retained GB × $0.10. Show
   delta per cluster.
4. Partition check: flag topics likely to exceed ~5 MB/s ingress or
   ~10 MB/s egress per partition (serverless throttles per-partition;
   fix is more partitions, not bigger brokers).

Report a table: cluster | type | avg throughput | est. provisioned $/mo |
est. serverless $/mo | recommendation | notes (partition limits, retention).
Change nothing.
Works with any assistant that can run shell commands.

Want the guided version?

The MSK Serverless walkthrough covers this topic interactively — it asks about your setup, branches to what’s relevant, and quizzes you on the tricky parts. Free and anonymous.

Start the walkthrough

More Databases cheat sheets

Amazon RDS Proxy

Connection count, not CPU, is often what forces an RDS upsize. RDS Proxy pools thousands of client connections (especially from Lambda) onto ~10–30 real ones, so a small instance stays small — the proxy fee typically covers 10–25% of the avoided upsize.

Athena Partition Projection

Instead of asking S3 which partitions exist before every query, Partition Projection lets Athena calculate the paths from a pattern you declare — near-instant query startup on tables with thousands of partitions, zero LIST metadata calls.

Athena Query Result Reuse

Athena caches identical query results in S3, so a repeated dashboard refresh or scheduled report returns instantly with zero bytes scanned. A workgroup checkbox that routinely cuts thousands off a scan-heavy bill.

Aurora I/O-Optimized

Aurora bills a sneaky third bucket — I/O operations at ~$0.20/million — that can dwarf compute and storage on I/O-heavy workloads. I/O-Optimized zeroes it out for a ~30–40% higher instance rate. A break-even calc, not a default.