Proxmox LXC Containers Fail to Start After Reboot

After a routine reboot of my Proxmox host (along with the network switch and router), I found that all of my VMs started fine, but none of my LXC containers would come up. Trying to start a container from the UI or CLI resulted in the following message:

problem with monitor socket, but continuing anyway: got timeout

TASK OK

After trying to start an lxc and then running journalctl -xe to get the immediate logs one of the log entries was.

(xc-start)[3883]: [email protected]: Failed at step STDERR spawning /usr/bin/lxc-start: No such file or directory

However /usr/bin/lxc-start did exist and was owned by the correct package:

ls -l /usr/bin/lxc-start
-rwxr-xr-x 1 root root 52904 Sep  2 09:08 /usr/bin/lxc-start

dpkg -S /usr/bin/lxc-start
lxc-pve: /usr/bin/lxc-start

Another log entry from journalctl was:

could not populate metric data cache: mkstemp "/run/pve/metrics.tmp_XXXXXX" failed: ENOENT

It turned out that /run/pve didn’t exist.

ls -ld /run/pve
ls: cannot access '/run/pve': No such file or directory

This normally gets recreated on boot but some issue with the shutdown procedure must have caused issues with this. Happily the solution was simply to create the directory and then restart the service deamons.

mkdir -p /run/pve
chown root:root /run/pve
chmod 755 /run/pve

systemctl restart pvedaemon
systemctl restart pvestatd
systemctl restart lxc