Tagged to request a meeting.

cc: @jwrdegoede

From Chris in the fedora-btrfs tracker:
https://pagure.io/fedora-btrfs/project/issue/7#comment-696274

Background

Fedora has a Hidden GRUB menu feature. It depends on the ability of GRUB to modify the values of boot_success and boot_indeterminate located in the grubenv file, from within the GRUB preboot environment.

Some configurations (e.g. BIOS x86_64) place grubenv in /boot/grub2, which would be on Btrfs if we proceed with /boot on Btrfs. UEFI configurations can be ignored, since grubenv is on the FAT formatted EFI System partition.

Problem

GRUB file system drivers are read-only. GRUB writes to grubenv directly modifying those blocks.
Writing to a grubenv file located on Btrfs from GRUB would be indistinguishable from corruption, and therefore the GRUB btrfs driver currently refusessave_env command when grubenv is on Btrfs (or ZFS, LUKS, or md RAID and maybe LVM.)

Workarounds

I don't know whether it's possible or appropriate for GRUB to dynamically insert a boot parameter as a way of sending a message to the booted environment, or some other way of sending messages from preboot to boot.

Is there a reason you want to use btrfs for /boot on BIOS? Would it be OK to put btrfs on /boot only when using UEFI?

As @catanzaro already said, I believe that the best way to deal with this is to put /boot on brtfs for EFI installs and leave it on a separate normal partition for classic BIOS boot.

I believe that classic BIOS boot will be with us for a long time yet (and I'm in favor of that because the alternative is dropping support for perfectly fine hardware), so this is not ideal, but I do believe it is the best we can do;

And TBH even though I do believe that classic BIOS boot will and should stick a round for a long time yet, I don't think it is necessarily bad to have some sub-optimal settings there in some cases, we very much do want people to move to EFI on all systems where it is supported.

Note this also puts classic and EFI boot on more or less equal footing. Classic boot will need a separate /boot partition and EFI boot needs the ESP. So both still need 1 partition which is not managed by BRTFS.

The problem here is that it's still not on equal footing, since we install RPM content into /boot in both scenarios and don't have a way to deal with if both can't be managed by a snapshotting/system recovery scheme.

So for A/B booting and the ESP we have been considering that we should stop directly installing things like grubx64.efi into the ESP and instead have them under say /usr/lib/grub2-efi and use a script which installs them. This script could then also install a new version as "B" version (with a separate efibootmgr entry) and swap the "B" version to become the "A" version after a successful boot. Note there are plans for this, but no more then that,

One option might be to manage /boot similarly and stop installing RPMs there directly, OTOH when restoring a snapshot because of issues, you likely also want to go back to the old kernel...

An alternative would be to disable the hidden-boot-menu feature on classic BIOS boot installs, assuming the saveenv alls in grub2.cfg will fail silently on BTRFs then just not having menu_auto_hide set in the grubenv (which is currently set there on fresh installs by anaconda) is enough to make it not matter that grub cannot clear the boot_success boot_indeterminate variables.

But note that the hidden-grub-menu is far from the only grub feature relying on save_env. Looking for save_env in grub2.cfg I see the following occurrences:

1. save_env next_entry, save_env saved_entry, save_env prev_saved_entry, these have to do with grub's ability to single shot boot a boot-menu entry other then the default once

2. save_env boot_counter, this is used by /etc/grub.d/08_fallback_counting which AFAIK is used by the CoreOS A/B booting stuff

3. save_env menu_show_once this is used by e.g. "systemctl reboot --boot-loader-menu=TIME" which is used to force the menu to show for X seconds even when it is hidden (either always hidden or hidden by the grub-hidden-bootmenu feature)
   This could be worked around by having userspace clear it on the next boot instrad of relying on grub clearing it itself.

4. save_env boot_success boot_indeterminate, as discussed here.

Now 1. is usually not used in Fedora, 4, can be disabled for classic BIOS boot, and 3. could be worked around as mentioned. Which leaves 2. which AFAIK is probably a real problem .

So I guess we could put /boot on BRTFS even with classic BIOS boot, if we disable the grub-hidden-menu feature. The downside of this is ugly-fying the boot for everyone using classic BIOS boot.

Cc @javierm Javier, this is probably of interest to you too, see the above discussion.

Actually since you are (more or less) the Fedora grub2 maintainer I guess your opinion on this matters, so please let us know what you think.

So for A/B booting and the ESP we have been considering that we should stop directly installing things like grubx64.efi into the ESP and instead have them under say /usr/lib/grub2-efi and use a script which installs them. This script could then also install a new version as "B" version (with a separate efibootmgr entry) and swap the "B" version to become the "A" version after a successful boot. Note there are plans for this, but no more then that,

This is what SUSE does already for all of the content installed into /boot and /boot/efi today. I would be okay with this approach because then it also gives us a path to being able to trigger updates of the content installed in those locations, too.

@jwrdegoede, @javierm, would you be available to attend the Workstation WG meeting on January 5 at 14:30 UTC?

@catanzaro yes I can attend then. But I believe it will be more important that @javierm will be present then me. If any boot-related changes are necessary then @javierm will need to be involved.

Also I really do not have the time to work on this at all. Talking about time AFAIK @javierm is also really busy with some (non-public) stuff atm and that will likely go on for a while. So I think that depending on what we come up with as a solution, actually implementing it might need to be delayed to F35. Note I'm not speaking for @javierm here, just setting expectations.

@catanzaro I'll be on vacations on Jan 5, I could try to attend the meeting but I can't promise to make it.

As @jwrdegoede mentioned I'm quite busy right now (and so is pjones) and any bootloader work related to this would likely have to wait for F35.

For F34 we were discussing to align the GRUB configuration layout with what other distros do. That is, to have just a minimal config file in the ESP that loads the one from /boot/grub2. If that's the case, the grubenv file will also be in /boot even for EFI.

The fact that Fedora has different paths for the GRUB configuration file depending on whether legacy BIOS or EFI is used is a constant source of confusion and frustration for users as far as I can tell. Same for the grubenv being a symbolic link for EFI, this also is very error prone.

The current approach also has practical implications, since it means that an installation is either for legacy BIOS or EFI. By having consistency and always using the same path for the GRUB config file, the same installation could be booted using either legacy BIOS or EFI.

But this will make using btrfs for /boot much more complicated, since we won't be able to rely on the grubenv being in the ESP anymore.

I still need to write a Change Proposal for this but wanted to know your opinions about it.

Probably worth looping in the CoreOS/bootupd folks into this discussion.

Probably worth looping in the CoreOS/bootupd folks into this discussion.

Indeed. I meant to comment the same but then forgot.

@catanzaro I'll be on vacations on Jan 5, I could try to attend the meeting but I can't promise to make it.

Let's plan on January 12, then, if that hopefully works well for everyone. (Of course, even better if we can work things out in this issue before the meeting date.)

Short version: Fix or find an alternative for grubenv. Standardize on Btrfs for /boot/ across all of Fedora, fallback to FAT where necessary. And decouple RPM from /boot/ and /boot/efi/.

Long version:

grubenv is in an unreliable location. This was known 7 years ago. When located on btrfs, zfs, luks, mdraid, and maybe LVM, GRUB will refuse to write to grubenv. 2 of 3 file system upstreams, ext4 and XFS, consider it inappropriate for non-kernel fs code doing writes to files on the file system - even though, for now, it does work for them.

Following a crash during write operations, ext4 and XFS depend on journal replay to make the file system consistent, which GRUB doesn't do. Having discussed it with upstreams, they're leary of using complex journaled file systems for /boot/ at all. They don't want to support journal replay in the boot loader. Nor support an atomic freeze/thaw for updates of /boot/. Therefore, I suggest deprecating them in favor of either FAT or Btrfs. Btrfs raid1 is self-healing, supported by GRUB for ~10 years including degraded operation, and simpler to understand, troubleshoot, and recover from device failure, than either mdadm or LVM raid.

Decoupling /boot/ and /boot/efi/ from RPM packages sounds good. I don't know if populating these volumes should be the job of bootupd, but it makes sense. A "source of truth" could be in /usr, used for syncing, repairing, updating both /boot/ and /boot/efi/. That'd be consistent with rpm-ostree as well as traditional rpm systems.

Decoupling /boot/ and /boot/efi/ from RPM packages sounds good. I don't know if populating these volumes should be the job of bootupd, but it makes sense. A "source of truth" could be in /usr, used for syncing, repairing, updating both /boot/ and /boot/efi/. That'd be consistent with rpm-ostree as well as traditional rpm systems.

Yes, there are BZ that were filed a long time ago about this for both shim and grub2. And I agree that bootupd is the correct component to do it.

We should also make bootupd to update GRUB for non-EFI (x86 with legacy BIOS and ppc64le OF) in their embedding area, that is the gap between the MBR and the first partition for x86 and the PReP partition for ppc64le.

For the latter, we not only will have to update it to roll out any fixes / new features but also because ppc64le will gain Secure Boot support by appending signatures to the GRUB core.img.

This will also allow to support things like A/B update mechanisms as @jwrdegoede mentioned.

Adding @walters for bootupd.

Let's plan on January 12, then, if that hopefully works well for everyone. (Of course, even better if we can work things out in this issue before the meeting date.)

Reminder: we'll discuss at our regular BlueJeans meeting at 14:30 UTC on January 12.

• Downstream change proposal
• Action: Javier to share a link to the upstream discussion when it happens.
• Agreed: We will wait for the grub proposal to be agreed upstream before proceeding.

Well the change proposal was accepted and implemented.

@salimma since you reported this issue, could you provide a status update here please?

Does the Workstation WG really still need to track this? I'm not really sure why we care about the filesystem that we use for /boot.

I don't think we actually need to track this, so I'll close this issue.