Does anybody here know why the team behind GNOME OS choose systemd-sysupdate over bootc? I don’t understand why, because bootc seems much more applicable, because this way you can use all the container tooling for testing the GNOME OS image. As far as I know systemd-systupdate is ‘only’ an A/B system.
I find sysupdate’s approach preferable for a few reasons (in no particular order):
- It’s already deeply integrated into systemd
- We don’t have a package manager, so any benefit of being able to generate custom OCI images with custom package overlays and deploying them via bootc is completely lost on us. We simply can’t do that anyways, even if we use bootc. So really it’s unclear what the benefit of OCI would be for us. If we were to use bootc, we’d be using it as a basic A/B system anyways.
- It enables UKIs, secure boot, and TPM-backed encryption of data partitions. Bootc has also started offering this, but at the time the decision was made this was simply not the case. Anyways, bootc’s approach also has some security gaps that sysupdate doesn’t: bootc requires the OS to mount an untrusted filesystem image before it can continue loading the verified OS contents off of that image. Linux’s filesystem developers have been very clear: filesystem drivers are not designed to be robust against a malicious filesystem image. So an attacker could theoretically circumvent bootc’s code signing by maliciously crafting the filesystem itself to attack the kernel and disable the checksumming. sysupdate doesn’t need to mount any untrusted filesystems to boot; everything is always trusted, and so no such attack is even theoretically possible.
- bootc is designed to share the filesystem with your actual user data. This has reprocussions:
- If you want to encrypt your data, you have to encrypt the OS too. If the OS is truly immutable, like GNOME OS is, the only thing you get by encrypting the OS is worse performance. Encrypting the OS is also bootc’s suggested workaround for the above mentioned secure-boot gap
- If you have too many files in your home directory and fill up enough of the disk, there won’t be enough space left over for bootc to install any more system updates. This is an issue that plagued ostree too; I’ve definitely run into situations where I need to go delete a bunch of stuff to free up space to download a system update. sysupdate pre-allocates all the disk space that it would ever need, and it always downloads updates directly into their final destination, so it never actually needs any additional disk space to apply an update. So updates can never fail due to lack of disk space with sysupdate, which is very very important IMO.
- It makes it harder to implement a factory-reset feature, and even when implemented it’ll be less secure. With sysupdate, you can do a secure wipe of your system just by by discarding the encryption keys and marking the root partition as deleted. At that point, there’s no chance of recovering any data. On the next boot, the OS detects that it doesn’t have a root partition, so fresh encryption keys are generated and a new root partition is created. With bootc, since the filesystem is shared, the best you can do is just delete the data folders within the running filesystem. This just marks the space as available, but doesn’t actually delete any data. This means that the data is still there and is easily recoverable by various tools. The only way to securely wipe a bootc system is to erase the encryption keys, like you would on sysupdate. But this erases the OS too, which means that you’ll need to then re-install the OS from scratch from external media.
- sysupdate is generally the less complicated solution, and our needs aren’t that complicated for GNOME OS. It seemed like the better fit for us.
To be clear, I don’t have anything against bootc. For hybrid-immutable distributions like Fedora Silverblue, it’s probably better suited, and the limitations of sysupdate might get in the way in those usecases. Also, the OCI tooling combined with a package manager does make deployment of custom images super convenient for an enthusiast Linux user, and generally sysupdate images are harder to generate. Our usecase just happens to be better suited by sysupdate.
5 Likes
Helllo @adrianvovk,
Thanks for your response!
You write ‘we’ all the time, so I guess you are one of the maintainers of GNOME OS?
I’m wondering: is a combination of bootable containers also possible with systemd-sysupdate? Or is it only meant to work in combination with ostree?
Do you happen to know if there is a document that describes the vision of how all these technologies (systemd-sysupdate, bootc, ostree, dnf5, bootupd, etc) should work together? Something like a ‘master plan’ for future Linux distributions (Fedora/CentOS Stream/RHEL)?
1 Like
You write ‘we’ all the time, so I guess you are one of the maintainers of GNOME OS?
Nowadays I’m one of the people involved, yes. I wasn’t the one making the decision for GNOME OS, but I was making the exact same decision for carbonOS (my own distro) for the same reasons.
I’m wondering: is a combination of bootable containers also possible with systemd-sysupdate? Or is it only meant to work in combination with ostree?
I don’t fully understand the question.
If you’re just asking whether or not you can use sysupdate and bootc at the same time on the same system, the answer is no.
Do you happen to know if there is a document that describes the vision of how all these technologies (systemd-sysupdate, bootc, ostree, dnf5, bootupd, etc) should work together? Something like a ‘master plan’ for future Linux distributions (Fedora/CentOS Stream/RHEL)?
I don’t know of one. That’s a question for RedHat people. But I can probably explain the vision as I understand it for future RHEL-adjacent Linux distributions:
- sysupdate: Will not be used
- bootc: The software that manages making an OCI container bootable. It has the code that manages downloading and unpacking the OCI image, copying out the kernel for the bootloader, and finding and mounting the right container image at boot time.
- ostree: A content-addressed-storage for binaries. Historically, this also took on a role very similar to bootc, just instead of OCI it used its own format. As far as I understand it, bootc’s OCI images still contain an ostree file structure inside, but it seems like RH wants to move away from that at some point?
- rpm-ostree: Fedora’s existing custom package manager for atomic systems, built on ostree. Basically it lets you install packages into the ostree CAS, and then makes that all bootable. I’m guessing this will go away with bootc, though I don’t know. Essentially, it’s the package manager you use instead of DNF on Fedora Silverblue, Fedora CoreOS, etc
- dnf5: A new faster package manager for non-atomic Fedora. So you wouldn’t have dnf5 alongside with bootc, just like you don’t have dnf alongside rpm-ostree. This will continue to be used on Fedora Workstation, non-atomic editions of RHEL and CentOS, etc
- bootupd: A tool for updating non-kernel stuff on the EFI system partition. Specifically grub and the secure-boot shim. Currently, with rpm-ostree, there’s no way to update grub. bootupd was supposed to fix that, but it has taken a long time to roll out
Again, this is just my understanding through talking with RedHat folks. I’m not involved with Fedora or similar projects
2 Likes
Yes, this was exactly what I meant!
I thought I read a few times that they want to integrate ostree (or parts of it) into DNF5, but I don’t know if that’s true. Would be great if someone could confirm this.
Bootc needs some kind of way to layer new packages, but I have no clue how they want to that. Something like a local container file would be interesting.
Sure but that argument only holds as long as the operating system never mounts any native Linux filesystem which could be corrupted (ok, or you do some “metadata only fsck” before mounting them perhaps).
In the case GNOME OS, / today is btrfs (right?) which quickly gets mounted and if it’s corrupted that’s not any different than the /usr partition being corrupted in practice.
Yeah that’s true. Basically any filesystem that’s auto-mounted should be either encrypted, or unencrypted and protected with dm-integrity. And that’s only if you believe that encryption is sufficient “poor man’s integrity”, because otherwise you’re going to want to do a whole AEAD stack of LUKS+dm-integrity. As far as I know though, in practice “poor man’s integrity” seems to work fine and the whole industry relies on it in practice (for instance, blk-crypto engines can only do encryption and not integrity, and all Android phones use blk-crypto nowadays)
GNOME OS encrypts the / partition, and btrfs’s checksumming really strengthens the “poor man’s integrity” for us. Eventually, once btrfs supports it, we can probably turn on btrfs’s HMAC checksumming to give ourselves proper AEAD without so much of a performance cost.
So yes, both schemes need to encrypt partitions to cover the filesystem integrity hole. The difference is that bootc must encrypt the operating system to cover the hole, and sysupdate doesn’t need to encrypt the operating system.
1 Like
When discussing some of this one thing that needs to be really clear is precisely what threat models we’re talking about.
You seem to be talking about the Evil maid, which is valid. But a different though related important threat model is the “anti-persistence” one…i.e. assuming something breaks out of a container and gains root, can it persist across reboot? That’s significantly more important for e.g. datacenter or even cloud cases as the “evil maid” is rather harder to perform there. And in that model, encryption/dm-integrity doesn’t help at all.
bootc requires the OS to mount an untrusted filesystem image before it can continue loading the verified OS contents off of that image.
I wouldn’t say “requires” so much as “encourages or defaults-to” to be clear, but yes.
If the OS is truly immutable, like GNOME OS is, the only thing you get by encrypting the OS is worse performance.
I very much disagree that GNOME OS is “truly immutable”, but we covered that in mastodon. Can you stop using that term please? I think it’s just misleading.
the only thing you get by encrypting the OS is worse performance.
Except that many use cases for custom derived operating systems actually may end up having some private/secret data that ends up as part of “the OS” and they really want to encrypt it.
So in the end honestly I don’t think there’s a practical difference: On the bootc side you can equally well apply the same encryption/dm-integrity to the single root filsystem that has both the OS and user data.
@adrianvovk and @walters: I think your understanding of everything is much higher than mine, so I won’t go in your discussion, but I still have some remarks and questions:
@adrianvovk:
You wrote earlier that GNOME OS doesn’t have a package manager and won’t use one, but that doesn’t mean you can’t use bootc. In the future it probably will be possible to use bootc with and without overlaying packages.
@walters: do I say this correct? Will it be possible to overlay packages with bootc in the future (as with rpm-ostree)? And if so: how? By editing a local container file?
@adrianvovk:
The advantages of using bootc for GNOME OS would be that you can use all the tooling for checking the image/container?
I watched this video on Youtube. So they are working on secure boot!
@walters: is there a document that describes the big vision of how all the technologies (bootc, ostree, dnf5, bootupd, etc) should work together and what the path forward is? Is there some kind of master plan? Am I correct and a part of ostree will be integrated into dnf5?
You seem to be talking about the Evil maid , which is valid. But a different though related important threat model is the “anti-persistence” one…i.e. assuming something breaks out of a container and gains root, can it persist across reboot? That’s significantly more important for e.g. datacenter or even cloud cases as the “evil maid” is rather harder to perform there. And in that model, encryption/dm-integrity doesn’t help at all.
I agree. Fixing vectors for persistence is important.
The Evil Maid defenses close up access to the kernel, and prevent persistence by attacking the OS itself. But, it’s possible to attack /etc, ~/.bashrc, and so on.
I think the long-term goal is to mitigate these things too, as I mentioned on Mastodon. Only run signed code by default, don’t allow changes to /etc unless they’re signed, and so on.
I very much disagree that GNOME OS is “truly immutable”, but we covered that in mastodon. Can you stop using that term please? I think it’s just misleading.
The distinction between an immutable distro that has no opt-out or layering mechanism and an immutable distro that does still let you modify the OS with packages is a big one, in my opinion. I’ve been calling these paradigms completely/fully immutable, and hybrid-immutable. Maybe “strictly immutable” is better terminology? I’m happy to take other suggestions if you have any!
Except that many use cases for custom derived operating systems actually may end up having some private/secret data that ends up as part of “the OS” and they really want to encrypt it.
Sure, yes this is true. This also isn’t the case for GNOME OS. It’s also possible to do this with sysupdate: encrypt the code server-side, decrypt on boot. This lets it be encrypted in transit even. You can actually have distinct code-encryption keys and data-encryption keys with sysupdate too. Embedded systems have been doing this kind of thing with RAUC and A/B partitions for a long time.
1 Like
You wrote earlier that GNOME OS doesn’t have a package manager and won’t use one, but that doesn’t mean you can’t use bootc. In the future it probably will be possible to use bootc with and without overlaying packages.
Yes we can use bootc, but we don’t need to use bootc. We take no benefit from package layering because we don’t have packages to layer.
It’s already possible to use bootc without layering any packages. Just in my opinion, package layering is bootc’s biggest advantage, which outweighs the disadvantages. GNOME OS doesn’t want/need package layering, so the (relatively minor!) disadvantages are a bigger factor for us. Thus, we decided that sysupdate was a better fit for us.
The advantages of using bootc for GNOME OS would be that you can use all the tooling for checking the image/container?
What do you mean by checking?
I watched this video on Youtube. So they are working on secure boot!
Still need to get around to watching the bootc/composefs ASG talks. It’s on my TODO list. But yes, I’m aware that they’re working on secure boot. We’ve talked about this with Colin (and others) a lot at the conference. I mentioned it in my original reply:
It enables UKIs, secure boot, and TPM-backed encryption of data partitions. Bootc has also started offering this, but at the time the decision was made this was simply not the case
1 Like
bootc and rpm-ostree will exist side by side for the time being, although rpm-ostree will be phased out.
The plan is for dnf to also work on the atomic desktops so people don’t have to learn another tool (rpm-ostree).
bootupd is now in Fedora 41 Atomic, though it never runs by default.
1 Like
I thought I read somewhere (or it was in one of the videos) that the benefit of using containers is that you can now automate all kind of security and integrity checks. Sorry, I can’t specify that further. Maybe someone else can comment on this?
So it’s safe to say, that parts of rpm-ostree are integrated into dnf5?
Is it already clear which parts/commands and which not? Or is that too early to tell? Is there already documentation about this or not?
I think you want to be taking these questions to https://discussion.fedoraproject.org/ and not here
2 Likes
Of course I don’t mind asking the questions there, but to me that also feels a little bit redundant, because the questions came up here. If you could answer them here I would appreciate that. If you still feel they belong in the forum of Fedora, I will ask the questions there.
Although a change was proposed and accepted for dnf5 to be added to Fedora 41, the current status is listed as “unknown”. And on my Fedora 41 system, dnf5 isn’t installed.
So right now none of the dnf5 transition has happened, at least what has been shipped to users. There’s very likely some experimental builds being tested by develoeprs.
1 Like
I found the original text:
https://docs.fedoraproject.org/en-US/bootc/getting-started/#_simplified_security
My Fedora 41 beta has dnf5 I think, but might have been pulled then?
Oops, I was a bit vague there. I meant to say Fedora Silverblue 41. Fedora Workstation 41 does have dnf5.
1 Like