Proxmox Full Disk Encryption with SSH Remote Unlock

February 23, 2021Proxmox, Security, Sysadmin Standard

Read time 3 minutes

Although ZFS also offers encryption there is no support for full disk encryption yet so we will use LUKS instead.

Requirements: ZFS mirror (raid1) install

In order to encrypt a running system rpool must be a ZFS mirror.

To verify you are running a ZFS mirror execute the following command.

zpool status rpool | grep mirror >/dev/null 2>&1 && echo "zfs mirror - continue with the encryption" || echo "not supported, reinstall as zfs (raid1) mirror"

Note: If you don’t want to use ZFS you can also install Debian with LUKS and install Proxmox on top.

LUKS Encryption

Install cryptsetup and run a benchmark to select the best encryption algorithm.

apt install cryptsetup cryptsetup-initramfs

cryptsetup benchmark

I’m going with aes-xts 512b for mixed write and read speed.

aes-cbc 256b is better suited for read heavy workloads on nvme disks.

In order to start list the drives in your rpool via “zpool status”.

zpool status rpool

I will use /dev/sda3 as example for the first disk and /dev/sdb3 for the second disk.

You will have to use the correct disk printed by zpool which should be in the format /dev/disk/by-id/XXXXXXXX.

Write down both disk names and detach the first disk.

zpool detach rpool /dev/sda3

After that encrypt the drive with LUKS. I’m also hardening brute force attempts by increasing the iteration time to 10 seconds.

Make sure to use a good passphrase.

cryptsetup luksFormat -v -c aes-xts-plain64 -s 512 -h sha512 -i 10000 -y /dev/sda3

Unlock the disk and attach it back to rpool.

cryptsetup luksOpen /dev/sda3 cryptroot1
zpool attach rpool /dev/sdb3 cryptroot1

ZFS will start a resilver which will encrypt all data on cryptroot1.

watch zpool status rpool

Once resilver completes repeat with the second disk.

zpool detach rpool /dev/sdb3

Instead of creating a new LUKS header for the second disk we can simply clone it from the first disk and change the UUID.

dd if=/dev/sda3 of=/dev/sdb3 bs=2M count=1
UUID=$(cat /proc/sys/kernel/random/uuid)
cryptsetup luksUUID --uuid $UUID /dev/sdb3
cryptsetup luksOpen /dev/sdb3 cryptroot2
zpool attach rpool cryptroot1 cryptroot2

Wait again for the resilver to finish.

watch zpool status rpool

Now all that’s left is to create the crypttab mapping.

Print the UUID for both disks and add them to /etc/crypttab.

blkid | grep crypto

cat << 'EOF' > /etc/crypttab
cryptroot1  UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX   cryptroot   luks,initramfs,keyscript=decrypt_keyctl
cryptroot2  UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX   cryptroot   luks,initramfs,keyscript=decrypt_keyctl
EOF

Update initramfs to apply the changes.

update-initramfs -u

Note: You can ignore the following message

cryptsetup: ERROR: Couldn't resolve device rpool/ROOT/pve-1
cryptsetup: WARNING: Couldn't determine root device

If you reboot now you will have to enter the passphrase via keyboard.

SSH Remote Unlock

Copy your RSA key to the server for authentication.

If you use Linux you can simply use ssh-copy-id.

ssh-copy-id -i /root/.ssh/id_rsa root@192.168.1.10

The key should be listed in /root/.ssh/authorized_keys now.

Install Dropbear.

apt install dropbear-initramfs

Configure static IP for initramfs.

Make sure to adjust the network interface “enp2s0” and the IP “192.168.1.10” as well as gateway and netmask if needed.

cat << 'EOF' >> /etc/initramfs-tools/initramfs.conf
IP=192.168.1.10::192.168.1.1:255.255.255.0::enp2s0:off
EOF

Copy the RSA key to initramfs.

cp /root/.ssh/authorized_keys /etc/dropbear/initramfs/authorized_keys
chmod 600 /etc/dropbear/initramfs/authorized_keys

Apply changes and reboot.

update-initramfs -u
reboot

Note: You can ignore the following messages

cryptsetup: ERROR: Couldn't resolve device rpool/ROOT/pve-1
cryptsetup: WARNING: Couldn't determine root device
cryptsetup: ERROR: cryptroot1: Source mismatch
cryptsetup: ERROR: cryptroot2: Source mismatch

To decrypt the disks connect via SSH and execute “cryptroot-unlock”

cryptroot-unlock

Enter the passphrase and the server should continue to boot.

If you increased the iteration time you might get a timeout error message but the server will boot shortly after as long as the passphrase is correct.

Securely Delete Leftovers

There will still be unencrypted leftovers on the disks.

In order to securely delete any leftovers fill the disks with random data.

apt install pv
cat /dev/urandom | pv > /tmp/fill.tmp || rm /tmp/fill.tmp

SSD TRIM

If your rpool is installed on SSD’s you can enable TRIM for better performance.

First make sure TRIM is supported. “Disc-Gran” should not be zero.

lsblk --discard | grep sda3

Add “discard” to /etc/crypttab options.

cryptroot1  UUID=XXXX   cryptroot   luks,discard,...
cryptroot2  UUID=XXXX   cryptroot   luks,discard,...

Apply change and reboot.

update-initramfs -u
reboot

dmsetup should print “allow_discards” if it is enabled.

dmsetup table | grep allow_discards

Setup a cron job to trim rpool once a week.

cat << 'EOF' > /etc/cron.d/zfs-trim
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# trim rpool every sunday at 4am
0 4 * * 7 root /usr/sbin/zpool trim rpool
EOF

chmod +x /etc/cron.d/zfs-trim

You can verify the trim status with.

watch zpool status -t

Header Backup

It’s recommended to create a LUKS header dump for disaster recovery.

With the header the whole disk can be decrypted without the passphrase.

Make sure to store it somewhere safe.

cryptsetup luksHeaderBackup /dev/sda3 --header-backup-file /mnt/usbstick/headerbackup

Comments

Lerner Forever November 5, 2022 at 04:40 - Reply

Hi, thank you very much for the professional, yet easy to follow article.
As I understood, it is supposed to have a running Proxmox server and this instruction applies full disk encryption on a raid 1 ZFS pool.
Is there any difference if I want to install the Proxmox from scratch? Should I always first install the Proxmox as usual and then follow this instruction?

Regards

Constantin December 10, 2022 at 17:34 - Reply

Sorry for the late reply.

Yes just install Proxmox and then continue with the encryption.

During Proxmox install select ZFS raid1 and select both disks.

Timor August 25, 2023 at 18:51 - Reply

Thanks for the How to.

Followed
LUKS Encryption & SSH Remote Unlock
on a PVE 8.0-2 installation with a ZFS Raid1 setup, everything went well. But after rebooting I end up in Busybox with the error

Failed to import pool ´rpool´: no such pool or dataset

Now I don’t know how to proceed

What I tried:
– Busybox has no cryptsetup
– Also waiting and manually trying to import the rpool fails

Constantin October 5, 2023 at 23:07 - Reply

With Debian 12 some things changed.

It should work if you install dropbear-initramfs instead of dropbear.

John January 7, 2024 at 17:41 - Reply

1. I tried to reboot after “If you reboot now you will have to enter the passphrase via keyboard.”, i.e. before setting up dropbear. I had a monitor and keyboard attached and wanted to see if that works.

However, it failed to boot, and dropped me into the initramfs shell with the message:
“cannot import ‘rpool’: no such pool or dataset”.

Turns out “apt install cryptsetup” does NOT install the cryptsetup-initramfs package. Thus “update-initramfs -u” did not pick up the crypttab. (Since this is a test system, I wiped and started again.)

Note that “apt install dropbear-initramfs” will pull in “cryptsetup-initramfs”.
Still, maybe add an “apt install cryptsetup cryptsetup-initramfs” to the top of this post, it won’t hurt.

2. Here is more info on dropbear and how the paths changed in Debian 12:
https://www.cyberciti.biz/security/how-to-unlock-luks-using-dropbear-ssh-keys-remotely-in-linux/

3. With dropbear-initramfs I managed to get unlock over SSH working.
But I also want direct physical access. However, attaching monitor+keyboard doesn’t give me a way to unlock the LUKS vault. It just shows:
“EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Measured initrd data into PCR 9”
It would be nice to have a shell there.

As a workaround, I connected my laptop directly to the server via an ethernet cable, told my laptop via ifconfig the correct IP and subnet mask, and SSHed in over that to unlock the vault.

Constantin January 12, 2024 at 21:42 - Reply

Thanks for your comment, with Proxmox 8 (Debian 12) dropbear-initramfs must be explicitly installed, old systems that upgrade are not affected. I haven’t updated this blog for quite some time, all changes are only in my IaC collection.

edit: Blog post updated and tested working with Proxmox 8.1

hames March 14, 2024 at 21:50 - Reply

hey,
I just wanted to say thank you for this great and detailed how-to. You made my day!
One little correction that I ran into: authorized_keys needs to be copied to: /etc/dropbear/initramfs/authorized_keys

xmif January 14, 2025 at 02:17 - Reply

I’m endeavoring to setup a fresh Proxmox 8 server (mini pc) with 2 ssds in a mirror. Is this guide suitable if there is no other boot drive in the system? I would also like to use TPM to unlock automatically on boot. Perhaps Clevis can help with that.

Constantin January 14, 2025 at 10:09 - Reply

Yes this is exactly what this is for, depending on your needs you can always move the VM’s to another storage (ssd/hdd/network attached) later if needed. You will have to check how to unlock Luks2 automatically via TPM and personally I would recommend against it, a mini pc is usually stolen/taken as one, nobody will remove the drives from it, so depending on the use case the encryption is worthless in this scenario.

Menu