Laptop keyboard not working at passphrase prompt after cold boot

[jmai444]

I've been using zfsbootmenu on this Thinkpad p14s (AMD Gen 3) for around 10 months now. It's been working without issue, but recently I have to cold boot using an external keyboard so that I can enter my passphrase to boot into the guest OS.

The built in keyboard works on the zfsbootmenu menu, but not on the following screen that requests my passphrase to boot into the OS if I have cold booted.

It works fine from a reboot.

It could be a hardware or firmware issue I guess, any thoughts on how to diagnose? I've seen a similar issue reported for cryptsetup passphrases. Does zfsbootmenu use it behind the scenes to unlock the encryption key? The really odd thing is that it just stopped working. No update or config change to zfsbootmenu itself since the first install 10 months ago.

I followed these instructions at install with the following choices: single NVMe disk, encrypted, prebuilt, direct.
https://docs.zfsbootmenu.org/en/v2.3.x/guides/void-linux/uefi.html

Thanks in advance.

[ahesford]

Sounds like a kernel change, if your version of ZFSBootMenu is unchanged and this all used to work. Does the keyboard appear as a USB device?

[jmai444]

Thanks for the reply.

The guest OS kernels reside in /boot. Is this directory part of the encrypted file system? I thought it was, and therefore updates to them could not be the source of the issue since no passphrase could be provided.

There is also a void directory under /boot/efi and it contains the following:

[jeff@p14s-void ~]$ ls -al /boot/efi/EFI/void
total 72
drwxr-xr-x 3 root root  4096 Jan  2 20:43 .
drwxr-xr-x 4 root root  4096 Jan  2 20:43 ..
drwxr-xr-x 2 root root  4096 Jan  2 20:43 fw
-rwxr-xr-x 1 root root 59670 May  4  2022 fwupdx64.efi

[jeff@p14s-void ~]$ ls -al /boot/efi/EFI/void/fw
total 34916
drwxr-xr-x 2 root root     4096 Jan  2 20:43 .
drwxr-xr-x 3 root root     4096 Jan  2 20:43 ..
-rwxr-xr-x 1 root root  1182160 Jan  2 20:43 fwupd-6be552b4-eda3-46a7-ab1b-d8ba1ff451bd.cap
-rwxr-xr-x 1 root root 34561776 Jan  2 20:43 fwupd-e40809b1-7d9f-4dc4-7c0f-8ace63b43687.cap

Looks like there were updates as early as Jan, though I would have installed them after that. I definitely applied some fw updates because this AMD Thinkpad hasn't been a stable experience so far. Maybe a firmware update has caused this?

Does the keyboard appear as a USB device?

Nothing here checks out as a keyboard so I guess no?

[jeff@p14s-void ~]$ lsusb 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. Hub
Bus 001 Device 003: ID 10ab:9309 USI Co., Ltd 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 003: ID 1a7c:0195 Evoluent VerticalMouse C Wireless
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 002: ID 5986:2142 Acer, Inc Integrated RGB Camera
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 009 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 010 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

[classabbyamp]

The built in keyboard works on the zfsbootmenu menu, but not on the following screen that requests my passphrase to boot into the OS if I have cold booted.

that you have to enter the passphrase after zfsbootmenu suggests that your key is not in the boot environment's initramfs. does your dracut conf have install_items+=" /etc/zfs/zroot.key " as is specified here for encrypted install?

[jmai444]

does your dracut conf have install_items+=" /etc/zfs/zroot.key " as is specified here for encrypted install?

Yes:

[jeff@p14s-void ~]$ cat /etc/dracut.conf.d/zol.conf 
nofsck="yes"
add_dracutmodules+=" zfs "
omit_dracutmodules+=" btrfs "
install_items+=" /etc/zfs/zroot.key "

It also works fine on reboot. I think a config issue like you describe would either work or never work.

Edit:

that you have to enter the passphrase after zfsbootmenu suggests...

That's an imprecision on my part. I don't know that the failure is after zfsbootmenu. All I can say is that it's after I choose the OS from the initial zfsbootmenu menu.

[jmai444]

When I can successfully enter the passphrase, after I press enter I see the console output from this script: contrib/xhci-teardown.sh.

This indicates that when the issue occurs it is prior to zfsbootmenu launching the OS boot environment.

[samy-mahmoudi]

Hi all,

I am experiencing a similar issue in Arch Linux, although before I am shown my boot environments (zbm.show).

I have been building EFI images against linux-lts and using them successfully to boot my laptop for months:
Dracut-based ZFSBootMenu ---> Dracut-based Arch Linux RAM disk

Recently, I noticed a package upgrade of systemd and read the release notes quickly in relation to booting my environment (via a Dracut-based Arch Linux RAM disk) after ZFSBootMenu has started. The only potential problem that I could identify was related to Dracut but Arch had already upgraded the Dracut package so that I could not actually be affected.

After building everything (kernel, BE RAM disk and EFI image) as usual and rebooting, my screen apparently got locked at the prompt for my passphrase, i.e. before ZFSBootMenu shows me the menu (zbm.show) with the boot environments of my pool.

I dove into some of the ZFSBootMenu scripts and noticed that before prompting the passphrase, ZFSBootMenu tries to load the passphrase from the relevant location of its filesystem. So I decided to include my passphrase into my EFI image for debugging purposes, given that I could not get much of the debugging options at any level: kernel, RAM disk, ZFSBootMenu loglevel and scripts. Result: the encryption key apparently got loaded properly, I could see the ZFSBootMenu with my boot environments but the menu/keyboard was not responding.

I plugged an external USB keyboard and retried the same experiment:

• to check an other keyboard (resp. keyboard interface) and
• to check the CAPS LOCK LED after the loss of control (I do not have any such LED on the internal keyboard and I often relate crashes to such a LED).

Same result: menu not responding to any keystroke, plus the external keyboard with the CAPS LOCK LED not blinking upon my keystrokes.

Therefore my experience of this issue is less in the prompt for passphrase than in the keyboard input.

Note: I can boot my environment with an EFI image of ZFSBootMenu that I built the same way in the past and kept as a backup.

[samy-mahmoudi]

I have troubleshot the issue I experience and created issue #648.

Not sure how my issue relates to the one of @jmai444 , if any relation at all.

[samy-mahmoudi]

Given:

• https://voidlinux.org/packages/?arch=x86_64&q=systemd and
• void-linux/void-packages@016e886,

I would be inclined to assert that Void Linux has fully moved systemd to version 256.

@jmai444 Could you confirm you experience your issue with systemd version 256? E.g., by providing us with the output of systemctl --version | grep '^systemd'.

[classabbyamp]

void doesn't have systemd...

[samy-mahmoudi]

Thanks!

Since Void Linux does not use systemd and @jmai444 uses prebuilt EFI bundles, I am becoming less and less confident our issues are related.

[jmai444]

The issue I have persists.

It happens after a cold boot in most cases.

I don't think it's necessarily an issue with ZFSBootMenu, but I am stumped on how to diagnose what is happening as there is no log to look at after the issue occurs. The void linux logs are completely clean.

[jmai444]

Two things I have tried are:

• resetting the BIOS to default (took me a bit of time to understand I had to re-whitelist the ZFSBootMenu images using efibootmgr before I could boot again after I did that)
• updated to the latest Lenovo firmware for this machine

Neither have had any effect.

[jmai444]

I haven't tried reinstalling the ZFSBootMenu prebuilt images.

Is the following all that is needed? I would back up the images currently in use first. Any caveats to doing this? I guess it would also upgrade ZFSBootMenu.

curl -o /boot/efi/EFI/ZBM/VMLINUZ.EFI -L https://get.zfsbootmenu.org/efi
cp /boot/efi/EFI/ZBM/VMLINUZ.EFI /boot/efi/EFI/ZBM/VMLINUZ-BACKUP.EFI

[classabbyamp]

if you don't already have efibootmgr entries for those, you'll need to add those

[jmai444]

Just following up that reinstall/upgrade of ZFSBootMenu images changed nothing. The issue persists.