All cheat sheets
Serverless & AIcheat sheet

SageMaker Multi-Model Endpoints

Host dozens to hundreds of models on one endpoint: hot models stay in RAM, cold ones live in S3 and load on demand. For per-customer or per-category model fleets with skewed traffic, it's an 85–95% hosting cut — and the only architecture that scales economically.

Last reviewed: July 11, 2026

Prefer step by step? Take the guided SageMaker Multi-Model Endpoints walkthrough — free, anonymous, interactive.Have an AI assistant? Jump to the copy-paste audit prompt at the bottom of this sheet.

TL;DR: Fifty barely-used models on fifty endpoints is a $4,000+/month bill for idle RAM. A Multi-Model Endpoint is an LRU cache for models: all fifty live in one S3 prefix, requests name their model (TargetModel), hot ones stay loaded, cold ones load from S3 in 5–20 s when called. Two shared instances replace the fleet — typically 85–95% off — and adding model #300 is just an S3 upload.

The numbers

  • 50 models × ml.m5.large endpoints ≈ $4,140/month; the same 50 on one MME with 2× ml.m5.xlarge ≈ $331/month (−92%)
  • Field examples: a 300-customer SaaS (≈50 active at any moment) went from $15k+/month, growing linearly per signup, to $500–800/month flat; a 40-category recommender consolidated to 3–4 instances at −85–90% with the top 5 categories permanently warm
  • Memory is the sizing knob: ~14 × 500 MB models fit in an 8 GB instance, ~120 in a 64 GB one — size for the working set, not the catalog

Do this

  1. Group models by container: every model on an MME must share the same inference container (one framework per MME — no PyTorch+TensorFlow mixing). Groups of 5+ same-framework endpoints are your candidates.

  2. Deploy one endpoint over an S3 prefix, invoke with the model name:

    predictor.predict(data, target_model="customer-1234.tar.gz")

    Adding/removing a model is an S3 upload/delete — zero infrastructure change.

  3. Size RAM to the working set and watch ModelCacheHit — 90%+ means sized right; sliding lower means more memory (the fix is RAM, not CPU).

  4. Pre-warm the VIPs: a scheduled Lambda pinging top-customer/hot-category models every few minutes keeps them off the cold path for pennies.

  5. Stack a Savings Plan on the consolidated instances — a small, steady fleet is exactly what's safe to commit on; combined with consolidation that's routinely 90–95% off the naive baseline.

  6. Auto-scale instance count on InvocationsPerInstance like any endpoint — capacity scales elastically while the model catalog just sits in S3.

Gotchas

  • Cold loads are 5–20 s for a model not in RAM. Invisible for internal tools and async flows; unacceptable for a hot user path — keep those models on dedicated endpoints and put the long tail on the MME.
  • One container per MME — mixed frameworks means one MME per framework.
  • GPU support is weak — multi-GB GPU models belong on dedicated endpoints.
  • ModelLoadingWaitTime spiking = requests queuing behind loads = undersized instance.
  • A/B testing is a hidden bonus fit: all variants on one endpoint, routed by TargetModel, promoted by config change.

Skip this if

  • You run 1–3 models — the operational surface isn't worth it; dedicated endpoints (or serverless for the sporadic ones) are simpler.
  • One model dominates all traffic — that's an auto-scaled single endpoint, not a cache problem.
  • Every request needs strict uniform latency — the LRU trade is the wrong trade.

Run this audit with your AI assistant

Paste this into Claude, ChatGPT, or any agent that can run the AWS CLI with read-only credentials. It audits your account for exactly the waste this sheet describes — and changes nothing.

You are auditing SageMaker endpoints for Multi-Model Endpoint (MME)
consolidation. Use the AWS CLI with READ-ONLY credentials. Do not
create, modify, or delete anything.

1. Inventory: aws sagemaker list-endpoints + describe-endpoint-config
   + describe-model per endpoint — instance types/counts, container
   images, and whether any MMEs already exist (Mode: MultiModel).
2. Consolidation candidates: groups of ≥5 single-model endpoints that
   share the SAME inference container/framework (compare container
   image URIs). Per group: sum current instance cost vs a modeled MME
   on 2–4 right-sized instances.
3. Traffic skew check (CloudWatch Invocations per endpoint): heavy
   skew (a few hot, many idle) strengthens the case; uniformly hot
   endpoints with strict latency stay dedicated.
4. Fit checks per group: model sizes vs candidate instance RAM
   (working set should fit with room; note ~500 MB models per 8/16/
   64 GB tiers), GPU models → poor MME fit, latency-critical
   endpoints → flag cold-load risk (5–20 s per S3 load).
5. For existing MMEs: CloudWatch ModelCacheHit (target 90%+),
   ModelLoadingWaitTime — low hit rate = undersized RAM finding.

Report: candidate groups table (endpoints | shared container | $/mo
now | modeled MME $/mo | caveats), existing-MME health, and the
reminder that a Savings Plan layered on the consolidated instances
stacks further. Change nothing.
Works with any assistant that can run shell commands.

Want the guided version?

The SageMaker Multi-Model Endpoints walkthrough covers this topic interactively — it asks about your setup, branches to what’s relevant, and quizzes you on the tricky parts. Free and anonymous.

Start the walkthrough

More Serverless & AI cheat sheets

API Gateway HTTP APIs

HTTP APIs cost ~$1.00 per million requests versus REST's ~$3.50 — a 71% cut with lower latency and native JWT auth. Most REST APIs use none of the features that premium pays for; audit and migrate the eligible 80%.

App Runner Automatic Pausing

App Runner Automatic Pausing scales your service to zero when idle — you stop paying for provisioned compute and trade it for a 2–5s cold start on the first request. Business-hours apps save 60–70%, sporadic ones 80–95%.

AWS Lambda Power Tuning

Lambda bills GB-seconds, so memory and speed trade off — and for CPU-bound code, more memory is often cheaper. Power Tuning tests every memory setting against your real function in ~5 minutes and typically finds 20–50% savings.

GameLift Spot Fleets

Run multiplayer game servers on Spot capacity with an On-Demand buffer and let GameLift FleetIQ handle the 2-minute interruption choreography — typically 70–85% cheaper than pure On-Demand, invisible to players.