AWS / Guide 9

Monitoring and alerts

CloudWatch building blocks, Logs Insights queries worth keeping, and alarms people act on.

Monitoring in AWS is mostly Amazon CloudWatch, with EventBridge for events and SNS for notifications. The goal isn't more dashboards; it's a small number of alerts that people trust and act on, and logs that are there when you need them.

The building blocks

Set up the foundation

• One SNS topic per audience and severity, for example notes-test-alerts for the operations team and notes-test-critical for on-call. Alarms point at topics, never at individual email addresses.
• The CloudWatch agent on every instance, installed and configured through Systems Manager (Guide 7), for memory, disk space and the logs you troubleshoot with.
• Set retention on every log group. The default is never expire, which is how log storage becomes a noticeable line on the bill. 30 days for application logs is a common starting point, with anything you need longer exported to S3.
• Cross-account observability: link workload accounts to a central monitoring account, so one set of dashboards and queries covers everything.

# Log groups with no retention set, largest first
Get-CWLLogGroup | Where-Object { -not $_.RetentionInDays } | Sort-Object StoredBytes -Descending |
    Select-Object LogGroupName, @{n='StoredGB';e={ [math]::Round($_.StoredBytes / 1GB, 2) }}

# Set 30 days on one of them
Write-CWLRetentionPolicy -LogGroupName '/notes-test/app' -RetentionInDays 30

Logs Insights queries worth keeping

Save these as saved queries in CloudWatch Logs Insights so the whole team has them.

Errors in an application log, per 5 minutes:

fields @timestamp, @message
| filter @message like /(?i)(error|exception|fatal)/
| stats count(*) as errors by bin(5m)
| sort errors desc

Rejected connections in VPC flow logs, by source and port (who is knocking on closed doors):

fields srcAddr, dstPort
| filter action = "REJECT"
| stats count(*) as attempts by srcAddr, dstPort
| sort attempts desc
| limit 25

Console sign-ins without MFA, from CloudTrail in CloudWatch Logs:

fields @timestamp, userIdentity.arn, sourceIPAddress
| filter eventName = "ConsoleLogin" and additionalEventData.MFAUsed != "Yes"
| sort @timestamp desc

Who changed security groups today:

fields @timestamp, eventName, userIdentity.arn, requestParameters.groupId
| filter eventSource = "ec2.amazonaws.com" and eventName like /SecurityGroup(Ingress|Egress)/
| sort @timestamp desc

Slowest Lambda invocations:

filter @type = "REPORT"
| stats max(@duration) as maxMs, avg(@duration) as avgMs, count(*) as calls by bin(1h)

Alarms people act on

An alert that fires every day gets ignored, and then the real one does too. A few rules:

• Alert on symptoms users feel, not every metric: load balancer 5xx errors and response time, failed health checks, queue age, not CPU at 80% on one instance of ten.
• Require several data points. "3 out of 5 minutes over threshold" stops single spikes from waking people up.
• Use composite alarms to page only when related alarms fire together (high latency and errors), and anomaly detection for metrics with a daily pattern.
• Every alarm has an owner and a runbook link in its description.

# Alarm when an instance fails its status checks, and recover it automatically
Write-CWMetricAlarm -AlarmName 'ec2-notes-test-01-status-check' -Namespace 'AWS/EC2' -MetricName 'StatusCheckFailed_System' `
    -Dimension (New-Object Amazon.CloudWatch.Model.Dimension -Property @{ Name = 'InstanceId'; Value = 'i-0123456789abcdef0' }) -Statistic Maximum -Period 60 `
    -EvaluationPeriod 2 -DatapointsToAlarm 2 -Threshold 1 -ComparisonOperator GreaterThanOrEqualToThreshold `
    -AlarmAction 'arn:aws:automate:us-east-1:ec2:recover', 'arn:aws:sns:us-east-1:111122223333:notes-test-alerts'

# Alarms currently firing
Get-CWAlarm -StateValue ALARM | Select-Object AlarmName, StateUpdatedTimestamp, StateReason

For events rather than metrics, use EventBridge rules to the same SNS topics: GuardDuty findings of severity 7 and above, Security Hub critical findings, AWS Backup job failures, AWS Health events for your accounts, and root user sign-ins.

Keep an eye on monitoring costs

• Log ingestion is the big one. Check which log groups ingest the most (the IncomingBytes metric per log group) and cut debug logging in production.
• Use the Infrequent Access log class for logs you keep for audits but rarely query: cheaper ingestion, fewer features.
• Send VPC flow logs and CloudTrail to S3 for long-term storage and query them with Athena; send only what you actively troubleshoot to CloudWatch Logs.
• Custom metrics and detailed monitoring are charged per metric. Collect what you alarm or graph on, not everything the agent can produce.

Examples to try

From aws.ravindran.in: CloudWatch and CloudWatch Logs, CloudTrail to CloudWatch, SNS and S3 event notifications.