GCP / Guide 8

Backup and recovery

Snapshot schedules, Backup and DR vaults, Cloud SQL and Cloud Storage protection, restores and disaster recovery.

Google Cloud keeps your data durable, but it doesn't keep it safe from you: a deleted disk, a script run against the wrong project, or ransomware using stolen admin credentials. That's what backups are for. This guide covers snapshots, the Backup and DR Service with backup vaults, Cloud SQL and Cloud Storage protection, restores, and disaster recovery.

Backup versus disaster recovery

You need both for important systems. Replication alone faithfully copies a deletion to the other side.

Find what is not protected

Get-GcpBackupCoverage.ps1 checks every VM in each project for a Backup and DR backup plan or a snapshot schedule on all of its disks, and every Cloud SQL instance for automated backups and point-in-time recovery:

.\Get-GcpBackupCoverage.ps1 -ProjectId prj-notes-test -UnprotectedOnly

Snapshots and snapshot schedules

A snapshot is an incremental, crash-consistent copy of a persistent disk or Hyperdisk, stored in multi-regional or regional Cloud Storage. A snapshot schedule (a resource policy) takes them automatically and deletes them after the retention you set:

# Daily at 03:00 UTC, kept 14 days, stored in the US multi-region
gcloud compute resource-policies create snapshot-schedule sched-daily-14d --region=us-east1 `
    --daily-schedule --start-time=03:00 --max-retention-days=14 --storage-location=us `
    --on-source-disk-delete=keep-auto-snapshots

# Attach it to a VM's disk
gcloud compute disks add-resource-policies vm-notes-test-01 --zone=us-east1-b --resource-policies=sched-daily-14d

Snapshots are simple and cheap, but anyone with enough rights in the project can delete them. For production, use backup vaults (below). Enable guest flush (application-consistent snapshots with VSS on Windows, or pre/post scripts on Linux) for databases running on VMs.

Backup and DR Service

Backup and DR Service is Google Cloud's managed backup service for Compute Engine VMs and disks, Cloud SQL, AlloyDB, Filestore, and databases running on VMs (SQL Server, Oracle, SAP HANA and more). The pieces:

• Backup vault: Google-managed storage for backups, in a separate backup project if you like. It enforces a minimum retention: backups can't be deleted before it ends, by anyone, which is what protects them from ransomware and malicious admins. Retention can be locked so it can't be shortened.
• Backup plan: the rules (hourly to yearly backups, backup window, retention) and the vault they go to.
• Backup plan association: links a VM, disk or database to a plan.

# Vaults and plans in a project
gcloud backup-dr backup-vaults list --project=prj-notes-backup --location=us-east1
gcloud backup-dr backup-plans list --project=prj-notes-backup --location=us-east1

# What is protected, and the state of its last backup
gcloud backup-dr backup-plan-associations list --project=prj-notes-test --format=json | ConvertFrom-Json |
    Select-Object resource, resourceType, @{n='LastBackup';e={ $_.rulesConfigInfo[0].lastBackupState }}

Good practice: vaults in a dedicated backup project that workload teams have no roles in, enforced retention that matches your policy (for example 30 days for daily backups), and a separate vault in another region for copies of critical systems.

Cloud SQL and Cloud Storage

• Cloud SQL: turn on automated daily backups and point-in-time recovery for every production instance; keep 7 to 35 days. Retain backups after instance deletion (the retained backups setting) so deleting the database doesn't delete its backups too. Add Backup and DR for longer, vault-protected retention.
• Cloud Storage soft delete keeps deleted objects recoverable for 7 days by default on new buckets (configurable up to 90). Keep it on for important data.
• Object versioning keeps previous versions when objects are overwritten; pair it with a lifecycle rule that deletes old versions after a while.
• Bucket Lock (a locked retention policy) makes objects immutable for a set period: the Cloud Storage answer to compliance storage.
• Dual-region or multi-region buckets keep data readable if a region is lost.

# Cloud SQL instances without backups or point-in-time recovery
gcloud sql instances list --project=prj-notes-test --format=json | ConvertFrom-Json | ForEach-Object {
    [pscustomobject]@{
        Instance = $_.name
        Backups  = [bool]$_.settings.backupConfiguration.enabled
        PITR     = [bool]($_.settings.backupConfiguration.pointInTimeRecoveryEnabled -or $_.settings.backupConfiguration.binaryLogEnabled)
    }
}

Watch for failed backups

A backup nobody checks isn't a backup. Backup and DR writes job logs to Cloud Logging and exposes metrics in Cloud Monitoring: create an alerting policy on failed backup jobs, and a weekly report of resources whose last successful backup is more than a day old. For snapshot schedules, alert on errors from compute.googleapis.com with the method CreateSnapshot in the audit logs.

Restoring

• VM from a snapshot: create a disk from the snapshot, then a VM from the disk (or attach it to an existing VM to copy files). Restore into an isolated network first if you're recovering from an incident.
• VM from Backup and DR: restore to a new VM in the same or another project and zone, or restore individual disks.
• Cloud SQL: restore a backup to the same or a new instance, or clone to a point in time (a new instance at any second in the recovery window).
• Record what isn't in the backup: network tags, service account, firewall rules and IP addresses. Keep them in Terraform, or in the runbook.

Test restores on a schedule

Restore one critical system a month into a test project, check that the application starts and the data is there, time it, and delete it. Script it (the gcloud commands above, plus a check) so it's cheap to repeat. Keep the results: they're the evidence auditors ask for, and the real recovery time you can promise.

Disaster recovery

Zonal failures are the more common event: regional MIGs, regional disks and Cloud SQL high availability (a standby in another zone) handle them automatically. Run a regional failover drill at least twice a year; the first one always finds something.