CloudWatch: metrics, logs, alarms, and the bill

Metrics: namespaces, dimensions, and the resolution you pay for

Before you publish your first custom metric, ask yourself: what will this cost at 10× today’s traffic, and am I alarming on the right number? The answers live in three decisions — namespace/dimensions, resolution, and which statistic you read.

A CloudWatch metric is a time-ordered set of data points — a single variable over time, like one EC2 instance’s CPU. Every metric is uniquely identified by a namespace (a container, e.g. AWS/EC2 or your own MyApp/Checkout), a metric name, and zero or more dimensions — name/value pairs like InstanceId=i-0abc. The subtlety that bites people: each unique combination of dimensions is a separate metric. Add a customerId dimension with 50,000 values and you’ve just created 50,000 metrics, each billed independently. That is the high-cardinality trap, and it is the fastest way to turn a metrics bill into a finance ticket.

Resolution is the other dial. A metric is either standard resolution (1-minute granularity) or high resolution (down to 1 second). AWS service metrics are standard by default; for EC2 you can opt into detailed monitoring for 1-minute data instead of the basic 5-minute data. For your own metrics you call PutMetricData and decide. Resolution costs: every PutMetricData call is charged, so publishing a high-resolution metric every second is dramatically more requests — and more money — than publishing once a minute. CloudWatch also ages metric data out automatically: 1-second data lives about 3 hours, 1-minute data about 15 days, 5-minute data about 63 days, and 1-hour rollups about 455 days (15 months), after which the metric expires if no new data arrives.

# Publish a standard-resolution custom metric (1-minute granularity, the default)
aws cloudwatch put-metric-data \
  --namespace "MyApp/Checkout" \
  --metric-name CheckoutLatencyMs \
  --dimensions Service=checkout,Env=prod \
  --value 412 --unit Milliseconds

# High resolution: --storage-resolution 1 stores at 1-second granularity.
# Every call is billed, so 1-sec publishing means ~60x the PutMetricData requests of 1-min.
aws cloudwatch put-metric-data \
  --namespace "MyApp/Checkout" \
  --metric-name CheckoutLatencyMs \
  --storage-resolution 1 \
  --value 412 --unit Milliseconds

When you read a metric you choose a statistic — Average, Sum, Maximum, or a percentile like p99. This choice is not cosmetic. The average hides the tail: if 99 percent of requests are fast and 1 percent time out, the average barely moves while real users are suffering. For any latency SLO you alarm on p99 (or p95), never the average. (Note: percentiles need the raw data points; if you publish a pre-aggregated statistic set, p99 generally isn’t recoverable.) Prices below are illustrative and region-dependent — in US East (N. Virginia) a custom metric runs about $0.30/metric/month for the first 10,000; always confirm on the CloudWatch pricing page.

Logs: groups, streams, and the retention trap

CloudWatch Logs organizes data as log streams (one sequence of events per source — e.g. one container) grouped into log groups (everything sharing the same retention, access, and monitoring settings). Retention is set per log group, and here is the trap stated plainly in the docs: by default, log data is stored indefinitely — the console shows it as Never Expire. Ingest is billed once per GB (about $0.50/GB in us-east-1, illustrative); storage is billed every month (about $0.03/GB/month). Never-expire plus high-volume DEBUG logs means the storage line grows without bound. Set retention deliberately — 7, 14, 30, or 90 days for most app logs — and drop noisy debug logging in production.

To turn logs into signal you have two tools. Logs Insights is ad-hoc query for investigation: write a query, scan a time range, get results. A metric filter is the durable version: it matches a pattern in incoming log events and emits a CloudWatch metric you can graph and alarm on — for example, counting ERROR lines so a spike pages someone.

# Set retention deliberately — this one command is the whole fix for the never-expire trap
aws logs put-retention-policy --log-group-name /myapp/checkout --retention-in-days 14

-- Logs Insights: top 20 slowest checkout requests in the last hour, for ad-hoc triage
fields @timestamp, requestId, durationMs
| filter service = "checkout" and durationMs > 1000
| sort durationMs desc
| limit 20

// Metric filter: turn every ERROR line into a count metric you can alarm on
{
  "filterName": "checkout-errors",
  "filterPattern": "ERROR",
  "logGroupName": "/myapp/checkout",
  "metricTransformations": [
    { "metricNamespace": "MyApp/Checkout", "metricName": "ErrorCount", "metricValue": "1", "defaultValue": 0 }
  ]
}

Alarms: states, evaluation windows, and what to watch

An alarm watches one metric (or a metric-math expression) and compares it to a threshold over a period, across a number of evaluation periods. It sits in one of three states: OK, ALARM, or INSUFFICIENT_DATA (not enough data points to decide — common right after creation, or when a metric stops reporting). Alarms only fire actions on a sustained state change, not merely for being in a state, which is why you tune “N of M datapoints breaching” to balance sensitivity against flapping. Actions go to an SNS topic (paging, ChatOps) or to an Auto Scaling policy.

Two refinements matter at senior level. Composite alarms combine other alarms with a boolean rule expression — for example (ALARM("CPUHigh") OR ALARM("DiskHigh")) AND OK("Deploying") — so you page once on a real condition instead of getting ten correlated pages from one root cause. Anomaly detection trains a band of expected values from history and alarms when the metric leaves the band, which beats a static threshold for metrics with daily or weekly seasonality. And the recurring mistake: alarming on CPU when your users feel latency and error rate. Alarm on p99 latency and on the error-count metric your metric filter produces — those are what an SLO is made of.