Proxmox Full Disk Encryption with SSH Remote Unlock

February 23, 2021Proxmox, Security, Sysadmin Standard
Read time 5 minutes

Although ZFS also offers encryption there is no support for full disk encryption yet so we will use LUKS instead. This is advanced and I assume you know your ways around Linux!

Note: verified to still work with Proxmox VE 9.0 (upgrades and fresh install)

Requirements: ZFS mirror (raid1) install

In order to encrypt a running system rpool must be a ZFS mirror.

To verify you are running a ZFS mirror execute the following command.

zpool status rpool | grep mirror >/dev/null 2>&1 && echo "zfs mirror - continue with the encryption" || echo "not supported, reinstall as zfs (raid1) mirror"

Note: If you don’t want to use ZFS you can also install Debian with LUKS and install Proxmox on top.

LUKS Encryption

Install cryptsetup and run a benchmark to select the best encryption algorithm.

apt install cryptsetup cryptsetup-initramfs

cryptsetup benchmark

I’m going with aes-xts 512b for mixed write and read speed.

aes-cbc 256b could be better suited for read heavy workloads on nvme disks.

In order to start list the disks in your rpool via “zpool status”.

zpool status rpool

I will use /dev/sda3 as example for the first disk and /dev/sdb3 for the second disk.

You will have to use the correct disk printed by zpool which should be in the format /dev/disk/by-id/XXXXXXXX.

Write down both disk names and detach the first disk.

zpool detach rpool /dev/sda3

After that encrypt the disk with LUKS. I’m also hardening brute force attempts by increasing the iteration time to 10 seconds.

Make sure to use a good passphrase.

cryptsetup luksFormat -v -c aes-xts-plain64 -s 512 -h sha512 -i 10000 -y /dev/sda3

Unlock the disk and attach it back to rpool.

cryptsetup luksOpen /dev/sda3 cryptroot1
zpool attach rpool /dev/sdb3 cryptroot1

ZFS will start a resilver which will encrypt all data on cryptroot1.

watch zpool status rpool

Once resilver completes repeat with the second disk.

zpool detach rpool /dev/sdb3

Instead of creating a new LUKS header for the second disk we can simply clone it from the first disk and change the UUID.

dd if=/dev/sda3 of=/dev/sdb3 bs=2M count=1
UUID=$(cat /proc/sys/kernel/random/uuid)
cryptsetup luksUUID --uuid $UUID /dev/sdb3
cryptsetup luksOpen /dev/sdb3 cryptroot2

zpool attach rpool cryptroot1 cryptroot2

Wait again for the resilver to finish.

watch zpool status rpool

Now all that’s left is to create the crypttab mapping.

Print the UUID for both disks and add them to /etc/crypttab.

blkid | grep crypto
cat << 'EOF' > /etc/crypttab
cryptroot1  UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX   cryptroot   luks,initramfs,keyscript=decrypt_keyctl
cryptroot2  UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX   cryptroot   luks,initramfs,keyscript=decrypt_keyctl
EOF

Update initramfs to apply the changes.

update-initramfs -u

Note: You can ignore the following message

cryptsetup: ERROR: Couldn't resolve device rpool/ROOT/pve-1
cryptsetup: WARNING: Couldn't determine root device

If you reboot now you will have to enter the passphrase via keyboard.

SSH Remote Unlock

Copy your RSA key to the server for authentication.

If you use Linux you can simply use ssh-copy-id.

ssh-copy-id -i /root/.ssh/id_rsa root@192.168.1.10

The key should be listed in /root/.ssh/authorized_keys now.

Install Dropbear.

apt install dropbear-initramfs

Configure static IP for initramfs.

Make sure to adjust the network interface “enp2s0” and the IP “192.168.1.10” as well as gateway and netmask if needed. You can figure out your network interfaces with “ip a”, note: it’s not vmbr0!

cat << 'EOF' >> /etc/initramfs-tools/initramfs.conf
IP=192.168.1.10::192.168.1.1:255.255.255.0::enp2s0:off
EOF

Copy the RSA key to initramfs.

cp /root/.ssh/authorized_keys /etc/dropbear/initramfs/authorized_keys
chmod 600 /etc/dropbear/initramfs/authorized_keys

Apply changes and reboot.

update-initramfs -u
reboot

Note: You can ignore the following messages

cryptsetup: ERROR: Couldn't resolve device rpool/ROOT/pve-1
cryptsetup: WARNING: Couldn't determine root device
cryptsetup: ERROR: cryptroot1: Source mismatch
cryptsetup: ERROR: cryptroot2: Source mismatch

To decrypt the disks connect via SSH and execute “cryptroot-unlock”

cryptroot-unlock

Enter the passphrase and the server should continue to boot.

If you increased the iteration time you might get a timeout error message but the server will boot shortly after as long as the passphrase is correct.

Securely Delete Leftovers

There will still be unencrypted leftovers on the disks.

In order to securely delete any leftovers fill the disks with random data.

apt install pv
cat /dev/urandom | pv > /tmp/fill.tmp || rm /tmp/fill.tmp

SSD TRIM

If your rpool is installed on SSD’s you can enable TRIM for better performance.

First make sure TRIM is supported. “Disc-Gran” should not be zero.

lsblk --discard | grep sda3

Add “discard” to /etc/crypttab options.

cryptroot1  UUID=XXXX   cryptroot   luks,discard,...
cryptroot2  UUID=XXXX   cryptroot   luks,discard,...

Apply change and reboot.

update-initramfs -u
reboot

dmsetup should print “allow_discards” if it is enabled.

dmsetup table | grep allow_discards

Setup a cron job to trim rpool once a week.

cat << 'EOF' > /etc/cron.d/zfs-trim
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# trim rpool every sunday at 4am
0 4 * * 7 root /usr/sbin/zpool trim rpool
EOF

chmod +x /etc/cron.d/zfs-trim

You can verify the trim status with.

watch zpool status -t

Header Backup

It’s recommended to create a LUKS header dump for disaster recovery.

With the header the whole disk can be decrypted without the passphrase!

Make sure to store it somewhere safe e.g. give it to a friend.

cryptsetup luksHeaderBackup /dev/sda3 --header-backup-file /mnt/usbstick/headerbackup

Disk Replacement

If you have a disk failing, replace it with a new one. The new disk must be bigger or exactly the same size as the old one. Note that different disks can have different byte sizes, so replacing a 4TB disk with a 4TB disk could still fail if it’s some bytes short, it’s best to replace the failing disk with an identical one.

You can check the status with “zpool status”, in this example a failing disk has already been replaced and the root pool is currently degraded e.g. booted from the single remaining disk

zpool status
        NAME                     STATE     READ WRITE CKSUM
        rpool                    DEGRADED     0     0     0
          mirror-0               DEGRADED     0     0     0
            cryptroot1           ONLINE       0     0     0
            1046765708608951588  UNAVAIL      0     0     0  was /dev/mapper/cryptroot2

note: if your pool becomes degraded due to read/write/cksum errors randomly you can clear them using “zpool clear rpool” and do a scrub using “zpool scrub rpool” to autoheal, if the problem does not go away replace the disk. The problem could also be related to a bad HBA, Motherboard fault or cabling issue.

The newly added disk is /dev/sdb and is not partitioned yet. lsblk shows that it has no partitions.

lsblk
NAME           MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda              8:0    0  256G  0 disk  
├─sda1           8:1    0 1007K  0 part  
├─sda2           8:2    0    1G  0 part  
└─sda3           8:3    0  255G  0 part  
  └─cryptroot1 252:0    0  255G  0 crypt 
sdb              8:16   0  256G  0 disk

We need to setup /dev/sdb so that it’s bootable and then encrypt it and add it back to the zfs root pool.

If you had something on the disk clean it first, this will remove any existing partitions.

wipefs -a /dev/sdb

Now we copy the partition layout from the working drive /dev/sda on to it, randomize the uuids and add it to the proxmox esp boot sync tool.

sgdisk /dev/sda -R /dev/sdb
sgdisk -G /dev/sdb

proxmox-boot-tool format /dev/sdb2
proxmox-boot-tool init /dev/sdb2
proxmox-boot-tool refresh

proxmox-boot-tool will warn you about the removed faulty disk, e.g.

WARN: /dev/disk/by-uuid/B9A7-C968 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping

remove the line B9A7-C968 from /etc/kernel/proxmox-boot-uuids

If you get the message

/dev/disk/by-uuid/E3A6-F728 contains no grub directory - skipping

you have a grub install and not systemd-boot, in that case you have to run this instead:

proxmox-boot-tool format /dev/sdb2
proxmox-boot-tool init /dev/sdb2 grub
proxmox-boot-tool refresh

You should not see any errors / warnings anymore, the boot loader is successfully setup.

Now it’s time to add the disk back to rpool and update crypttab, essentially we repeat the steps in the original setup from above but this time we replace the faulty disk instead of attaching it.

dd if=/dev/sda3 of=/dev/sdb3 bs=2M count=1
UUID=$(cat /proc/sys/kernel/random/uuid)
cryptsetup luksUUID --uuid $UUID /dev/sdb3
cryptsetup luksOpen /dev/sdb3 cryptroot2

zpool replace rpool 1046765708608951588 cryptroot2

Wait for the resilver to finish, since there is a lot of data this time it can take a couple of hours!

watch zpool status rpool

Now all that’s left is to update the UUID in crypttab, replace the old UUID for cryproot2 with the new one.

blkid | grep crypto
cat << 'EOF' > /etc/crypttab
cryptroot1  UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX   cryptroot   luks,initramfs,keyscript=decrypt_keyctl
cryptroot2  UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX   cryptroot   luks,initramfs,keyscript=decrypt_keyctl
EOF

Update initramfs to apply the changes.

update-initramfs -u

Note: You can ignore the following message

cryptsetup: ERROR: Couldn't resolve device rpool/ROOT/pve-1
cryptsetup: WARNING: Couldn't determine root device

That’s it, you can reboot now optionally to test if everything worked, removing any disk should still let the system boot.

Comments

Lerner Forever

Hi, thank you very much for the professional, yet easy to follow article.
As I understood, it is supposed to have a running Proxmox server and this instruction applies full disk encryption on a raid 1 ZFS pool.
Is there any difference if I want to install the Proxmox from scratch? Should I always first install the Proxmox as usual and then follow this instruction?

Regards

    Constantin

    Sorry for the late reply.

    Yes just install Proxmox and then continue with the encryption.

    During Proxmox install select ZFS raid1 and select both disks.

Timor

Thanks for the How to.

Followed
LUKS Encryption & SSH Remote Unlock
on a PVE 8.0-2 installation with a ZFS Raid1 setup, everything went well. But after rebooting I end up in Busybox with the error

Failed to import pool ´rpool´: no such pool or dataset

Now I don’t know how to proceed

What I tried:
– Busybox has no cryptsetup
– Also waiting and manually trying to import the rpool fails

    Constantin

    With Debian 12 some things changed.

    It should work if you install dropbear-initramfs instead of dropbear.

John

1. I tried to reboot after “If you reboot now you will have to enter the passphrase via keyboard.”, i.e. before setting up dropbear. I had a monitor and keyboard attached and wanted to see if that works.

However, it failed to boot, and dropped me into the initramfs shell with the message:
“cannot import ‘rpool’: no such pool or dataset”.

Turns out “apt install cryptsetup” does NOT install the cryptsetup-initramfs package. Thus “update-initramfs -u” did not pick up the crypttab. (Since this is a test system, I wiped and started again.)

Note that “apt install dropbear-initramfs” will pull in “cryptsetup-initramfs”.
Still, maybe add an “apt install cryptsetup cryptsetup-initramfs” to the top of this post, it won’t hurt.

2. Here is more info on dropbear and how the paths changed in Debian 12:
https://www.cyberciti.biz/security/how-to-unlock-luks-using-dropbear-ssh-keys-remotely-in-linux/

3. With dropbear-initramfs I managed to get unlock over SSH working.
But I also want direct physical access. However, attaching monitor+keyboard doesn’t give me a way to unlock the LUKS vault. It just shows:
“EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Measured initrd data into PCR 9”
It would be nice to have a shell there.

As a workaround, I connected my laptop directly to the server via an ethernet cable, told my laptop via ifconfig the correct IP and subnet mask, and SSHed in over that to unlock the vault.

    Constantin

    Thanks for your comment, with Proxmox 8 (Debian 12) dropbear-initramfs must be explicitly installed, old systems that upgrade are not affected. I haven’t updated this blog for quite some time, all changes are only in my IaC collection.

    edit: Blog post updated and tested working with Proxmox 8.1

hames

hey,
I just wanted to say thank you for this great and detailed how-to. You made my day!
One little correction that I ran into: authorized_keys needs to be copied to: /etc/dropbear/initramfs/authorized_keys

xmif

I’m endeavoring to setup a fresh Proxmox 8 server (mini pc) with 2 ssds in a mirror. Is this guide suitable if there is no other boot drive in the system? I would also like to use TPM to unlock automatically on boot. Perhaps Clevis can help with that.

    Constantin

    Yes this is exactly what this is for, depending on your needs you can always move the VM’s to another storage (ssd/hdd/network attached) later if needed. You will have to check how to unlock Luks2 automatically via TPM and personally I would recommend against it, a mini pc is usually stolen/taken as one, nobody will remove the drives from it, so depending on the use case the encryption is worthless in this scenario.

xib

Hey, first of all thanks for the great article – worked like a charm.
I”m in the Situation that both NVME (zpool mirror) are wearout >91% (spare both at a 100%) and I’m asking myself how I can switch one?
I don’t think it will be as simple as
– Shutdown
– Replace one NVME
– Reboot
– Add to Pool / Resilver
right?

Thanks!

    Constantin

    Sup Xib,

    yes do like you wrote but for the last step “Add to Pool / Resilver” you have to do the following:

    You first have to encrypt the new drive with Luks e.g. follow the second disk part of “LUKS Encryption” right after “Once resilver completes repeat with the second disk.” above. Best to read the whole part and prepare the commands with the right disks.

    ‘Zpool status’ should show you one drive unavailable after reboot, that’s the old disk you removed. Encrypt the new drive by copying the header from the existing disk, update /etc/crypttab, update initram, luksopen new drive and do a ‘zfs replace old_cryptroot new_cryptroot’ (or you can also zfs dettach followed by zfs attach) to replace the old disk with the new one. Then wait for the resilver to finish.

    In case ‘zpool status’ only lists the disk by uuid, you can just ‘ls -lah /dev/xxx’ it to check where the symlink goes to, ‘lsblk’ shows you what disk has which crypt partition.

    It’s important to not mix up the disks / crypt partitions and double check them, but in your case you still have the other disk in case something goes wrong.

    Hope this helps. FYI wearout is just an indicator, nvmes usually exceed the tbw by 2-3 times and then usually just go read only.

xib

In my case I’ve added two of the same NVME type and on the same date so I think it’s best to test this because one really fails. Just ordered a new one to test it.
Thanks a lot for your swift reply!

Best Regards

    Constantin

    Actually I forgot about making the drive bootable using ‘proxmox-boot-tool’, I will update the post with a full section for disk replacement.

Write a Reply or Comment

Your email address will not be published. Required fields are marked *

Recent Posts

Categories