Azure / Guide 9

Monitoring and alerts

Azure Monitor building blocks, KQL queries worth keeping, and alerts people act on.

Good monitoring tells the right people about real problems quickly, and stays quiet the rest of the time. This guide covers the building blocks of Azure Monitor, a few KQL queries worth keeping, and how to set up alerts people actually act on.

The building blocks

Set up the foundation

1. One central Log Analytics workspace per environment (for example log-notes-test), in the management subscription. Fewer workspaces make queries and access simpler.
2. Send the activity log of every subscription to it (Activity log > Export activity logs > diagnostic setting). The activity log is kept only 90 days otherwise.
3. Enable VM insights on VMs; it deploys the Azure Monitor Agent and a data collection rule.
4. Send resource diagnostic logs (Key Vault, firewalls, App Services, SQL) with an Azure Policy "configure diagnostic settings" assignment, so new resources are covered automatically.

KQL queries worth keeping

VMs that stopped reporting:

Heartbeat
| summarize LastHeartbeat = max(TimeGenerated) by Computer
| where LastHeartbeat < ago(15m)
| order by LastHeartbeat asc

CPU over the last day (VM insights):

InsightsMetrics
| where TimeGenerated > ago(1d)
| where Namespace == "Processor" and Name == "UtilizationPercentage"
| summarize AvgCPU = avg(Val), MaxCPU = max(Val) by Computer
| order by AvgCPU desc

Disks running out of space (VM insights):

InsightsMetrics
| where TimeGenerated > ago(30m)
| where Namespace == "LogicalDisk" and Name == "FreeSpacePercentage"
| extend Disk = tostring(parse_json(Tags)["vm.azm.ms/mountId"])
| summarize FreePercent = min(Val) by Computer, Disk
| where FreePercent < 15

Who deleted what (activity log):

AzureActivity
| where TimeGenerated > ago(7d)
| where OperationNameValue endswith "/DELETE" and ActivityStatusValue in~ ("Success", "Succeeded")
| project TimeGenerated, Caller, OperationNameValue, ResourceGroup, _ResourceId
| order by TimeGenerated desc

Run them in the workspace's Logs page, or from PowerShell:

$ws = Get-AzOperationalInsightsWorkspace -ResourceGroupName rg-notes-test -Name log-notes-test
(Invoke-AzOperationalInsightsQuery -WorkspaceId $ws.CustomerId -Query 'Heartbeat | summarize max(TimeGenerated) by Computer').Results

Alerts people act on

• Alert on symptoms users feel, not on every counter: service unavailable, errors rising, response time up, disk nearly full, backup failed.
• Every alert needs an owner and an action. If nobody would do anything when it fires, delete it.
• Use severities consistently: Sev 0–1 wakes someone up; Sev 2–3 creates a ticket; Sev 4 is informational.
• Suppress during maintenance with alert processing rules (for example during the Saturday patch window), rather than disabling alerts and forgetting to turn them back on.
• Always create Service Health alerts for the services and regions you use. They're free and tell you when the problem is Azure's, not yours.
• Use metric alerts where you can. They're faster and cheaper than log search alerts.

Keep an eye on logging costs

Log Analytics is charged by data ingested. It can quietly become one of the bigger monitoring costs:

• Check what's sending the most data: Usage | where TimeGenerated > ago(30d) | summarize GB = sum(Quantity) / 1000 by DataType | order by GB desc.
• Collect only the performance counters and event logs you use; data collection rules make this easy.
• Use a commitment tier once ingestion is steady and large enough, and set retention per table to what you actually need.
• A daily cap stops runaway costs, but when it's hit you lose data until the next day, so use it as a safety net with an alert, not as a budget.