Back to Blog
AWSCloudWatchmonitoringreduce_costs

The CloudWatch Logs bill that ate a startup

Ingestion is what you pay for, not storage — how one debug line in a hot path becomes a five-figure monthly bill, and the double-billed data-event trap.

JohannaDecember 9, 20256 min read

The first time a CloudWatch Logs bill surprises you, the instinct is to go delete old log groups to reclaim storage. That instinct is aimed at the wrong number. Storage is cheap and almost never the problem. The money is in ingestion, and ingestion is charged the moment a log line is written, forever, regardless of whether anyone ever reads it. Once you internalize that, a whole category of runaway bills becomes predictable — and preventable.

The three-part cost model

CloudWatch Logs bills on three axes, and their magnitudes are wildly different. Using us-east-1 as the reference (prices vary by region):

  • Ingestion: roughly $0.50 per GB for the Standard log class. This is the line that dominates almost every surprising bill.
  • Storage: roughly $0.03 per GB per month, on already-compressed data. To match one day of heavy ingestion in storage cost, you would have to retain that data for over a year.
  • Query (Logs Insights): roughly $0.005 per GB scanned. Death by a thousand fields @message | filter cuts is possible on huge log groups, but it is usually a distant third.

The ratio is the whole story. Ingesting a gigabyte costs about the same as storing it for sixteen months. So the question that controls your bill is never "how long do I keep logs" first — it is "how many bytes am I writing in the first place."

How one log line becomes five figures

Do the arithmetic on a single careless statement. Suppose a checkout service handles 5,000 requests per second at peak and averages a few thousand off-peak, and someone ships logger.debug("event=" + JSON.stringify(event)) in the hot path, with the log level left at DEBUG in production. Say each line lands around 1 KB after the serialized event.

At a sustained 3,000 requests per second, that is 3 MB per second, which is about 259 GB per day, which is about 7.8 TB per month. At $0.50 per GB that single log statement costs roughly $3,900 a month. Bump the volume or the payload size and you are past five figures from one line of code that a reviewer waved through because "it's just a debug log." Nobody notices for weeks, because logs don't page you and the bill arrives in arrears.

High-cardinality metrics, the quieter version

The same failure mode hides in metrics. Custom metrics via PutMetricData cost about $0.30 per metric per month, and — this is the trap — each unique combination of dimension values is a separate metric. Put a customer_id or request_id dimension on a metric and you have not created one metric; you have created one per customer or per request. A service with 50,000 users and a per-user latency metric just provisioned 50,000 metrics at $0.30 each: $15,000 a month to measure something you will never query per-user anyway. The PutMetricData API calls are billed on top of that.

The discipline is to keep dimensions low-cardinality — service, region, environment, status class — and push anything high-cardinality into the log stream instead, where you can aggregate it at query time. The Embedded Metric Format (EMF) is the sanctioned way to emit metrics through logs, but even that respects cardinality: high-cardinality dimensions in EMF still materialize as billable metrics.

The controls that actually move the number

  • Set retention on every log group. The default retention is "Never expire," which means every log group you have ever created is billing storage forever. It is not the biggest lever, but leaving it unset is pure waste. aws logs put-retention-policy with a sane value — 30, 60, 90 days — should be part of every log group's creation, ideally enforced in your IaC.
  • Use the Infrequent Access log class for logs you rarely read. The IA class ingests at roughly half the Standard price in exchange for a reduced feature set — no Live Tail, no metric filters, no subscription filters, a subset of Insights. For audit trails and compliance logs you keep but almost never touch, that is a free 50% cut on the dominant cost axis.
  • Control log level at the platform, not just the code. Lambda's advanced logging controls let you set application log level and emit structured JSON without a redeploy, so you can dial a chatty function down to WARN in production without shipping. Sample verbose logs rather than emitting every one.

The vended-logs volume trap

"Vended logs" are the logs AWS services emit on your behalf — VPC Flow Logs, WAF logs, Route 53 Resolver query logs. They have a distinct, tiered ingestion price, but the danger is volume, not unit price. VPC Flow Logs on a busy ENI record every accepted and rejected flow; WAF logs on a high-traffic distribution record every single request. Turn these on across a fleet "for visibility" and you can add terabytes a month without writing a line of application code.

The fix is usually the destination. If you need to query these logs interactively, CloudWatch is fine — but if you mostly need them for occasional forensics or compliance, send them to S3 instead, where per-GB storage is far cheaper than CloudWatch ingestion, and query with Athena when you actually need to.

The CloudTrail data-event gotcha, and the double bill

CloudTrail management events (API calls that change configuration) are essentially free for the first copy. Data events — S3 object-level GetObject/PutObject, DynamoDB item-level operations, Lambda invocations — are not. They bill around $0.10 per 100,000 events, and they are generated by data-plane traffic, which on a busy resource is enormous. Enable S3 data events on a bucket that serves millions of object reads, or DynamoDB data events on a hot table, and you have attached a meter to your application's core loop.

Then comes the part that catches even careful teams: the same activity can be billed twice. If you configure a trail to record data events and you also route those events into CloudWatch Logs, you pay the CloudTrail data-event charge and the CloudWatch Logs ingestion charge for the identical stream. I have seen a DynamoDB-heavy service where the Streams pollers generated data events that landed in both sinks, and the workload was paying for its own reads twice over. The activity was legitimate; the double accounting was not.

An incident, and the fix

The pattern plays out the same way every time. The bill jumps — say from a comfortable few thousand a month to fifteen thousand — and nothing in the deploy history obviously explains it. You open Cost Explorer, group by usage type, and the spike is sitting in a CloudWatch DataProcessing-Bytes line. You go to the CloudWatch Logs console, sort log groups by incoming bytes, and one group — /aws/lambda/checkout-processor — is ingesting hundreds of gigabytes a day. Alongside it, a CloudTrail with S3 data events enabled on the asset bucket is quietly adding its own tax.

The remediation is unglamorous and fast:

  1. Raise the log level in production, kill the hot-path debug line, and add sampling to anything verbose that must stay.
  2. Put a retention policy on every log group and move audit-grade groups to the Infrequent Access class.
  3. Scope CloudTrail data-event selectors to the specific prefixes and tables you truly need to audit, exclude the hot ones, and make sure you are not sinking the same events into two billed destinations.
  4. Move vended logs you don't query interactively to S3.

None of this requires new tooling. It requires knowing that ingestion is the meter, that "never expire" is the default, and that a single log statement in a hot path is a financial decision. Treat log volume as a first-class part of a code review and the startup-eating bill simply never arrives.

Back to all posts