GCP / Guide 9
Monitoring and alerts
Cloud Monitoring and Logging building blocks, log queries worth keeping, and alerts people act on.
Monitoring in Google Cloud is Cloud Monitoring for metrics and alerts and Cloud Logging for logs, together called Google Cloud Observability. The goal isn't more dashboards; it's a small number of alerts that people trust and act on, and logs that are there when you need them.
The building blocks
| Piece | What it does |
|---|---|
| Metrics | Numbers over time. Google Cloud services publish them for free; the Ops Agent adds memory, disk and application metrics |
| Metrics scope | Which projects one project's Monitoring can see. Put production projects in the scope of a central monitoring project |
| Log buckets and sinks | Where logs are stored and for how long; sinks route logs to buckets, BigQuery, Cloud Storage or Pub/Sub |
| Alerting policies | Conditions on metrics, logs (log-based alerts) or PromQL, with notification channels |
| Notification channels | Email, SMS, Google Chat, Slack, PagerDuty, Pub/Sub or a webhook for your ticketing system |
| Uptime checks | HTTP, HTTPS or TCP checks from locations around the world |
| Personalized Service Health | Google's own incidents that affect your projects |
Set up the foundation
- A central monitoring project whose metrics scope includes every production project, so one set of dashboards and alerts covers everything.
- Notification channels per audience, for example an email group for the operations team and PagerDuty for on-call. Alerts point at channels, never at individual people's addresses.
- The Ops Agent on every VM, installed by policy (Guide 7).
- Know your log retention. Each project has a
_Requiredbucket (audit logs, kept 400 days, free and unchangeable) and a_Defaultbucket (everything else, 30 days by default). Change_Defaultretention if you need more, or route important logs to a custom bucket with its own retention. - Upgrade log buckets to Log Analytics to query logs with SQL and join them with other data in BigQuery.
# Retention of the log buckets in a project
gcloud logging buckets list --project=prj-notes-test --format="table(name.basename(), location, retentionDays, analyticsEnabled)"
# Keep _Default logs for 60 days, and turn on Log Analytics
gcloud logging buckets update _Default --location=global --retention-days=60 --enable-analytics --project=prj-notes-test
Log queries worth keeping
Save these as saved queries in Logs Explorer so the whole team has them. They use the Logging query language.
Errors from VMs (set the time range with the picker):
resource.type="gce_instance"
severity>=ERROR
Who changed IAM policies:
logName:"cloudaudit.googleapis.com%2Factivity"
protoPayload.methodName="SetIamPolicy"
Firewall rule changes:
logName:"cloudaudit.googleapis.com%2Factivity"
resource.type="gce_firewall_rule"
protoPayload.methodName=~"(insert|patch|delete)$"
Connections denied by firewall rules (for rules with logging on):
logName:"compute.googleapis.com%2Ffirewall"
jsonPayload.disposition="DENIED"
Service account key creation (should be rare or never):
protoPayload.methodName="google.iam.admin.v1.CreateServiceAccountKey"
In a Log Analytics bucket, the same data can be summarised with SQL, for example the top callers of a sensitive API over the last week.
Alerts people act on
An alert that fires every day gets ignored, and then the real one does too. A few rules:
- Alert on symptoms users feel, not every metric: load balancer 5xx responses and latency, failed uptime checks, backlog in queues, not CPU at 80% on one VM of ten.
- Use a duration. "Above threshold for 5 minutes" stops single spikes from waking people up.
- Every alerting policy has documentation (a runbook link and an owner) in its documentation field, which appears in the notification.
- Use log-based alerts for events rather than metrics: IAM changes on production projects, service account key creation, super admin sign-ins.
| Alert | Metric | Typical threshold |
|---|---|---|
| Disk nearly full | agent.googleapis.com/disk/percent_used (Ops Agent) | > 85% for 15 minutes |
| Memory pressure | agent.googleapis.com/memory/percent_used (Ops Agent) | > 90% for 15 minutes |
| Website errors | loadbalancing.googleapis.com/https/request_count filtered to 5xx | Above normal for 5 minutes |
| Site down | Uptime check | Failing from 2 or more regions |
| Cloud SQL storage | cloudsql.googleapis.com/database/disk/utilization | > 85% |
| VM stopped unexpectedly | Log-based: compute.instances.stop or guest shutdown on production VMs | Any, outside maintenance windows |
# Notification channels and alerting policies in the monitoring project
gcloud beta monitoring channels list --project=prj-monitoring --format="table(displayName, type, enabled)"
gcloud alpha monitoring policies list --project=prj-monitoring --format="table(displayName, enabled, conditions[0].displayName)"
# Create an alerting policy from a JSON or YAML definition kept in source control
gcloud alpha monitoring policies create --policy-from-file=.\alert-disk-85.yaml --project=prj-monitoring
Keep an eye on logging costs
- Ingestion is what costs. Check which logs are biggest (Logs Storage page, or the
logging.googleapis.com/billing/bytes_ingestedmetric) and cut debug logging in production. - Exclusion filters on the
_Defaultsink drop logs you never read (load balancer logs for health checks, verbose VPC Flow Logs in test). Excluded logs aren't charged. - Sample VPC Flow Logs (for example 10%) and aggregate them over longer intervals except where you need every flow.
- Data Access audit logs are charged: turn them on where they matter, not everywhere (Guide 6).
- Route long-term logs to Cloud Storage (cheap archive classes) or BigQuery rather than keeping everything in log buckets for years.