All cheat sheets
Serverless & AIcheat sheet

SageMaker Serverless Inference

Stop renting a 24/7 endpoint for a model invoked sporadically — serverless inference bills only compute-milliseconds and memory-GB actually used, turning $100+/month idle endpoints into single-digit bills. The trade: 5–15 s cold starts, 6 GB/60 s/CPU-only limits.

Last reviewed: July 11, 2026

Prefer step by step? Take the guided SageMaker Serverless Inference walkthrough — free, anonymous, interactive.Have an AI assistant? Jump to the copy-paste audit prompt at the bottom of this sheet.

TL;DR: A traditional real-time endpoint is a rented server billing 24/7 whether it serves one request a day or a million. Serverless Inference provisions compute per request and bills compute-milliseconds × memory-GB — for sporadic or business-hours traffic, that's routinely a 75–95% cut. The costs: cold starts of 5–15 s after idle gaps, and hard limits (6 GB memory, 60 s per request, CPU only, ~200 default concurrency).

The numbers

  • 100k requests/month × 200 ms × 4 GB ≈ single-digit dollars, vs ~$100/month minimum for an always-on ml.m5.large-class endpoint
  • Break-even rule: an endpoint busy under ~30% of the time is cheaper serverless; above that, a right-sized real-time endpoint (plus Savings Plan) wins
  • Field example: a support-ticket classifier used 40 of 168 weekly hours cut ~75% by going serverless — the morning's first-ticket cold start was invisible to agents

Do this

  1. Find the idle renters: InvocationsPerInstance over 30 days per endpoint. Under ~30% duty cycle → convert; zero invocations → delete (the best optimization of all).

  2. Check the hard limits before converting: model ≤ 6 GB memory, inference ≤ 60 s, no GPU requirement. Any violation → stay real-time or use Asynchronous Inference (24-hour timeout) for long-running models.

  3. Convert via endpoint config: replace the instance spec with a ServerlessConfig (memory 1–6 GB, max concurrency). Test a couple of memory sizes — more memory = more CPU = faster per-ms, so the cheapest setting isn't always the smallest.

  4. Tame cold starts where they matter: a CloudWatch-scheduled keep-alive ping every few minutes costs pennies and eliminates them for business-hours tools; provisioned concurrency is the heavier, pricier hammer for true hot paths.

  5. Set the guardrails: max concurrency bounds a runaway retry loop's blast radius; a billing alarm on daily SageMaker spend catches the rest. Five minutes, cheap insurance.

  6. Default new/unknown-traffic models to serverless — migrating out later is easy; months of overpaying on an idle real-time endpoint is just regret.

Gotchas

  • Cold starts are the tax: 5–15 s on first request after a quiet period. Fine for dashboards, async pipelines, internal tools; wrong for user-facing hot paths.
  • The limits are hard: >6 GB models don't fit; >60 s requests fail (medical-imaging-style long inference wants async endpoints); no GPU at all.
  • High steady volume inverts the math — always-warm means no idle savings, and per-ms pricing exceeds amortized instance pricing at scale.
  • The mature architecture is a mix: serverless for prototypes/internal/sporadic, real-time + Savings Plans for high-volume production, Multi-Model Endpoints for per-tenant fleets, async for big/slow models.

Skip this if

  • Traffic is consistently high with strict latency SLOs — dedicated endpoint + Savings Plan is the recipe.
  • The model needs GPU or exceeds the size/timeout envelope.
  • You're doing large offline scoring — that's Batch Transform, not an endpoint at all.

Run this audit with your AI assistant

Paste this into Claude, ChatGPT, or any agent that can run the AWS CLI with read-only credentials. It audits your account for exactly the waste this sheet describes — and changes nothing.

You are auditing SageMaker inference endpoints for serverless
conversion. Use the AWS CLI with READ-ONLY credentials. Do not create,
modify, or delete anything.

1. Inventory: aws sagemaker list-endpoints + describe-endpoint /
   describe-endpoint-config — instance types, counts, serverless
   configs already in use.
2. Utilization per real-time endpoint (CloudWatch, 30 days):
   Invocations, InvocationsPerInstance, ModelLatency, CPU/Memory
   utilization. Compute a duty-cycle estimate: (invocations ×
   avg latency) / (instances × seconds in period).
3. Conversion verdicts:
   - Duty cycle < ~30% AND model fits limits (≤6 GB memory, ≤60 s
     inference, CPU-only) → serverless candidate; estimate new cost
     from invocations × duration × memory-GB rates vs current
     instance-hours.
   - Zero-invocation endpoints (30 days) → flag for DELETION, the
     better-than-serverless outcome.
   - High-utilization endpoints → keep real-time; note Savings Plan
     instead.
   - GPU instance types → ineligible; note asynchronous inference for
     long-running models.
4. Include guardrail reminders per conversion: max concurrency cap,
   CloudWatch billing alarm, and cold-start tolerance check with the
   endpoint's consumers.

Report: endpoint table (name | type | duty cycle | $/mo now | verdict |
est. $/mo after), the delete list, and conversion order. Change
nothing.
Works with any assistant that can run shell commands.

Want the guided version?

The SageMaker Serverless Inference walkthrough covers this topic interactively — it asks about your setup, branches to what’s relevant, and quizzes you on the tricky parts. Free and anonymous.

Start the walkthrough

More Serverless & AI cheat sheets

API Gateway HTTP APIs

HTTP APIs cost ~$1.00 per million requests versus REST's ~$3.50 — a 71% cut with lower latency and native JWT auth. Most REST APIs use none of the features that premium pays for; audit and migrate the eligible 80%.

App Runner Automatic Pausing

App Runner Automatic Pausing scales your service to zero when idle — you stop paying for provisioned compute and trade it for a 2–5s cold start on the first request. Business-hours apps save 60–70%, sporadic ones 80–95%.

AWS Lambda Power Tuning

Lambda bills GB-seconds, so memory and speed trade off — and for CPU-bound code, more memory is often cheaper. Power Tuning tests every memory setting against your real function in ~5 minutes and typically finds 20–50% savings.

GameLift Spot Fleets

Run multiplayer game servers on Spot capacity with an On-Demand buffer and let GameLift FleetIQ handle the 2-minute interruption choreography — typically 70–85% cheaper than pure On-Demand, invisible to players.