Compute Optimizer GPU Recommendations

TL;DR: GPU instances are among the most expensive resources in AWS — a single one can cost 10–20× a comparable CPU box — and most teams utilize only 20–30% of what they rent, over-provisioning out of fear of performance issues. Compute Optimizer added workload-aware GPU rightsizing in 2023 (still unknown to many teams): it watches GPU compute and GPU memory separately and recommends the right P, G, or Inf instance, with projected savings. Because the base cost is so high, even small right-sizing produces massive dollar wins.

The numbers

• Most teams run at 20–30% GPU utilization — the fear-driven over-provisioning this check targets.
• Needs ≥30 hours of GPU metrics; analyzes GPU compute, GPU memory, and network separately (some workloads barely touch cores but max memory, or vice versa).
• Inference on training-grade GPUs is the classic waste — purpose-built Inferentia or G5g often cost ~70% less.
• Field examples: a nightly training job on p3.16xlarge ($24/hr) at 31% GPU util moved to p3.8xlarge + parallel jobs = $17,280 → $8,640/mo (>$100K/yr); a spiky render pipeline on G5.12xlarge went to a G5.4xlarge baseline + on-demand scale-up = $3,600 → $1,200/mo (67%).

Do this

1. Enable Compute Optimizer and ensure GPU metrics flow (CloudWatch Agent / DCGM) from day one on GPU instances — no metrics, no recommendations.
2. Read GPU compute and GPU memory separately — a recommendation may say "cores at 15% but memory at 92% → memory-optimized target," which is more insightful than a blanket downsize.
3. Right-size inference off training-grade GPUs — test Inferentia (inf1/inf2) or G5g for inference; they're built for it at a fraction of the cost.
4. Handle spiky render/training with a small baseline + Auto Scaling rather than a permanent large instance — scale up on demand, terminate after.
5. Validate framework/driver compatibility before switching families — P→G or x86→Inferentia can hit compatibility issues if your ML framework doesn't support the new architecture.

Gotchas

• Not always apples-to-apples — a cheaper family may not run your model cleanly; test before committing production.
• Recommendations reflect recent usage — checking the day after a big training run skews the data; let it observe a representative period.
• Very short-lived instances (a GPU spun up 20 min/week) don't generate enough metrics.
• It's conservative — a plus for GPUs (it won't push risky downsizes), but it means some savings need your own analysis.

Skip this if

• You genuinely need bleeding-edge GPUs for research where cost is secondary, or your GPU usage swings wildly week-to-week so recommendations stay fuzzy.
• The workload is SageMaker inference — SageMaker Inference Recommender benchmarks instances by cost-per-inference directly. For spiky patterns, pair with Auto Scaling Groups and Spot Instances; this is the GPU-specific lens of Compute Optimizer.