Proxmox Full Disk Encryption with SSH Remote Unlock

Read time 3 minutes

Although ZFS also offers encryption there is no support for full disk encryption yet so we will use LUKS instead.

Requirements: ZFS mirror (raid1) install

In order to encrypt a running system rpool must be a ZFS mirror.

To verify you are running a ZFS mirror execute the following command.

zpool status rpool | grep mirror >/dev/null 2>&1 && echo "zfs mirror - continue with the encryption" || echo "not supported, reinstall as zfs (raid1) mirror"

Note: If you don’t want to use ZFS you can also install Debian with LUKS and install Proxmox on top.

LUKS Encryption

Install cryptsetup and run a benchmark to select the best encryption algorithm.

apt install cryptsetup cryptsetup-initramfs

cryptsetup benchmark

I’m going with aes-xts 512b for mixed write and read speed.

aes-cbc 256b is better suited for read heavy workloads on nvme disks.

In order to start list the drives in your rpool via “zpool status”.

zpool status rpool

I will use /dev/sda3 as example for the first disk and /dev/sdb3 for the second disk.

You will have to use the correct disk printed by zpool which should be in the format /dev/disk/by-id/XXXXXXXX.

Write down both disk names and detach the first disk.

zpool detach rpool /dev/sda3

After that encrypt the drive with LUKS. I’m also hardening brute force attempts by increasing the iteration time to 10 seconds.

Make sure to use a good passphrase.

cryptsetup luksFormat -v -c aes-xts-plain64 -s 512 -h sha512 -i 10000 -y /dev/sda3

Unlock the disk and attach it back to rpool.

cryptsetup luksOpen /dev/sda3 cryptroot1
zpool attach rpool /dev/sdb3 cryptroot1

ZFS will start a resilver which will encrypt all data on cryptroot1.

watch zpool status rpool

Once resilver completes repeat with the second disk.

zpool detach rpool /dev/sdb3

Instead of creating a new LUKS header for the second disk we can simply clone it from the first disk and change the UUID.

dd if=/dev/sda3 of=/dev/sdb3 bs=2M count=1
UUID=$(cat /proc/sys/kernel/random/uuid)
cryptsetup luksUUID --uuid $UUID /dev/sdb3
cryptsetup luksOpen /dev/sdb3 cryptroot2
zpool attach rpool cryptroot1 cryptroot2

Wait again for the resilver to finish.

watch zpool status rpool

Now all that’s left is to create the crypttab mapping.

Print the UUID for both disks and add them to /etc/crypttab.

blkid | grep crypto

cat << 'EOF' > /etc/crypttab
cryptroot1  UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX   cryptroot   luks,initramfs,keyscript=decrypt_keyctl
cryptroot2  UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX   cryptroot   luks,initramfs,keyscript=decrypt_keyctl
EOF

Update initramfs to apply the changes.

update-initramfs -u

Note: You can ignore the following message

cryptsetup: ERROR: Couldn't resolve device rpool/ROOT/pve-1
cryptsetup: WARNING: Couldn't determine root device

If you reboot now you will have to enter the passphrase via keyboard.

SSH Remote Unlock

Copy your RSA key to the server for authentication.

If you use Linux you can simply use ssh-copy-id.

ssh-copy-id -i /root/.ssh/id_rsa root@192.168.1.10

The key should be listed in /root/.ssh/authorized_keys now.

Install Dropbear.

apt install dropbear-initramfs

Configure static IP for initramfs.

Make sure to adjust the network interface “enp2s0” and the IP “192.168.1.10” as well as gateway and netmask if needed.

cat << 'EOF' >> /etc/initramfs-tools/initramfs.conf
IP=192.168.1.10::192.168.1.1:255.255.255.0::enp2s0:off
EOF

Copy the RSA key to initramfs.

cp /root/.ssh/authorized_keys /etc/dropbear/initramfs/authorized_keys
chmod 600 /etc/dropbear/initramfs/authorized_keys

Apply changes and reboot.

update-initramfs -u
reboot

Note: You can ignore the following messages

cryptsetup: ERROR: Couldn't resolve device rpool/ROOT/pve-1
cryptsetup: WARNING: Couldn't determine root device
cryptsetup: ERROR: cryptroot1: Source mismatch
cryptsetup: ERROR: cryptroot2: Source mismatch

To decrypt the disks connect via SSH and execute “cryptroot-unlock”

cryptroot-unlock

Enter the passphrase and the server should continue to boot.

If you increased the iteration time you might get a timeout error message but the server will boot shortly after as long as the passphrase is correct.

Securely Delete Leftovers

There will still be unencrypted leftovers on the disks.

In order to securely delete any leftovers fill the disks with random data.

apt install pv
cat /dev/urandom | pv > /tmp/fill.tmp || rm /tmp/fill.tmp

SSD TRIM

If your rpool is installed on SSD’s you can enable TRIM for better performance.

First make sure TRIM is supported. “Disc-Gran” should not be zero.

lsblk --discard | grep sda3

Add “discard” to /etc/crypttab options.

cryptroot1  UUID=XXXX   cryptroot   luks,discard,...
cryptroot2  UUID=XXXX   cryptroot   luks,discard,...

Apply change and reboot.

update-initramfs -u
reboot

dmsetup should print “allow_discards” if it is enabled.

dmsetup table | grep allow_discards

Setup a cron job to trim rpool once a week.

cat << 'EOF' > /etc/cron.d/zfs-trim
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# trim rpool every sunday at 4am
0 4 * * 7 root /usr/sbin/zpool trim rpool
EOF

chmod +x /etc/cron.d/zfs-trim

You can verify the trim status with.

watch zpool status -t

Header Backup

It’s recommended to create a LUKS header dump for disaster recovery.

With the header the whole disk can be decrypted without the passphrase.

Make sure to store it somewhere safe.

cryptsetup luksHeaderBackup /dev/sda3 --header-backup-file /mnt/usbstick/headerbackup