What Proxmox taught me about failure¶

A homelab is a safe place to discover how many assumptions live between a working VM and a recoverable service.

The VM was not the service¶

The first lesson was simple: a running guest does not mean the workload is healthy. Storage, networking, DNS, time synchronization, and application state all have their own failure modes.

The Proxmox summary can show a green guest while the application inside it is unable to answer requests. A process may be running but waiting on a database; the database may be healthy but unreachable because a bridge or firewall rule changed. Infrastructure status is useful, but it is only one layer of service health.

That changes what should be monitored. Host CPU and storage matter, but so do DNS resolution, certificate expiry, application response time, queue depth, and the ability to complete a real request. The best health check is close to what a user actually needs the service to do.

A snapshot was not a backup¶

Snapshots are excellent for short-term rollback before a risky change. They are fast and convenient because they remain connected to the original virtual disk and storage system. That same relationship is their limitation: if the storage is lost or corrupted, its snapshots may disappear with it.

A backup must have an independent failure boundary. For a homelab, that might mean storing scheduled backups on another machine and copying important ones to an additional location. The exact tooling matters less than being able to answer these questions:

• Is the backup separate from the host and its primary storage?
• How many historical versions are retained?
• Are failures reported somewhere I will notice?
• Is the data encrypted where appropriate?
• Has a complete restore been tested recently?

A successful backup job proves that files were written. Only a successful restore proves that they are useful.

Configuration is part of recovery¶

Guest disks are not the entire platform. Recovery may also require the VM configuration, network layout, storage definitions, firewall policy, DNS records, and the secrets needed by the application.

It helps to keep a small inventory outside the cluster. For each important guest, record its purpose, owner, addresses, storage, dependencies, backup policy, and the order in which services should return. Exportable configuration and infrastructure-as-code are valuable here, but even a reviewed text file is better than relying on memory.

Boot order deserves attention as well. Starting an application before its DNS, database, or network storage is ready can produce a confusing second incident during recovery. Dependencies should tolerate retries, and the recovery plan should state which foundational services come first.

Break it on purpose¶

Controlled failure turns vague confidence into evidence:

1. Stop a guest unexpectedly.
2. Restore it somewhere else.
3. Verify the application, not only the process.
4. Record what required human memory.

The phrase “somewhere else” is important. Restoring onto the same host tests a backup file; restoring onto different storage or a replacement node tests more of the recovery path. Use an isolated network when a restored guest could conflict with production addresses or begin processing real jobs.

A useful drill measures time and captures friction. How long did it take to find the backup, recreate networking, obtain credentials, and validate the application? Which step depended on a command from shell history? The answers turn a generic disaster-recovery plan into a concrete list of improvements.

High availability is not recovery¶

High availability can restart a guest on another node after a host failure, but it does not correct every failure. It cannot make corrupted application data healthy, undo an accidental deletion, or guarantee that a broken configuration will work after restart. Shared storage and cluster membership also introduce their own dependencies.

This is why availability and recoverability should be designed separately. HA reduces interruption for a set of infrastructure failures. Backups, tested restores, and documented rebuild procedures address a different set. A system can have either one without the other.

For small environments, complexity has a cost. A simple single-node setup with reliable off-host backups may be easier to recover than a cluster whose quorum, storage, and network behavior are not well understood. Add clustering when its failure modes are acceptable and there is a clear reason for the additional moving parts.

Define success before the incident¶

Recovery is complete when the service works, not when the VM boots. Validation should include the operations that matter: authenticate, read and write data, resolve names, reach dependencies, and confirm that scheduled jobs run. Check logs after the test as well; a service can appear functional while repeatedly failing background work.

Two simple targets make planning less vague:

• Recovery point objective: the maximum acceptable amount of data loss.
• Recovery time objective: the maximum acceptable time to restore service.

These targets do not need to be enterprise-sized promises. Even approximate values force useful decisions about backup intervals, retention, spare hardware, and how much of the rebuild should be automated.

The useful lesson¶

Proxmox makes it easy to create and operate virtual machines, but the important lesson sits above the hypervisor: virtualizing a server does not automatically make its service recoverable.

Confidence should come from evidence. Monitor the application, keep backups outside the primary failure boundary, preserve the configuration needed to rebuild, and practice restoration before it becomes urgent. A homelab is the ideal place to learn this because failure can be scheduled—and every controlled failure can leave the real system a little less surprising.