Towards a better way to hack and test your system components

Desktop
1

Introduction

Hey everyone,

I recently joined Codethink and I am contributing to the improvements being made to GNOME OS, through the Sovereign Tech Fund (STF), to make it easier and more useful in day-to-day development and testing tasks, among other things.

For those who haven’t heard of GNOME OS, it’s an immutable Linux distribution that ships the latest in-development GNOME desktop, core applications, and stack, aimed at developers and testers.

An overview of what we’re proposing here is the following:

  1. A story that would allow developers to hack and test system components in a more efficient and safer way.
  2. An utility that would make that story compelling.

Ultimately, we want these improvements to be useful to all other immutable and non-immutable OSes out there.

Rationale

In case you missed it, Jordan Petridis previously shared an insightful blog post about developing software on immutable OSes. A rough summary would be the following: on the applications development front, we’re in a pretty good shape thanks to Flatpak. Unfortunately, not so much on the system components development front.

Luckily, as Jordan’s demonstrates, there’s an alternative thanks to systemd-sysext which allow us to extend the host /usr with system extensions images, while keeping all the benefits of an immutable OS intact.

Therefore, we’re currently working on this proposal around systemd-sysext in hopes that it would bring the system components development experience closer to the applications.

The proposal looks roughly like this:

  1. Spin an instance of GNOME OS and pull all the core development tools (if you haven’t already).
  2. Hack system components and build a sysext image out of those.
  3. Deploy the sysext image and tests these components (alternatively, deploy these to a GNOME OS virtual machine).
  4. Share that sysext image with others to try it out (if you have a need for that).
  5. Remove the sysext image, forget these changes ever happened and smile at the that fact you didn’t have to build an entire OS.

Given the success of Flatpak, this proposal draws some conceptual parallels to it, where GNOME OS acts as the Platform and Sdk runtimes, while a sysext image acts as the application bundle.

Sample scenarios

This proposal is primarily aimed at system components, e.g., mutter, GNOME Shell, etc.

  • As a mutter developer, I want to develop and test my changes on cutting-edge components like a newer version of wayland.
  • As a GNOME Shell developer, I want to develop and test my changes on cutting-edge components like a newer version of systemd.

All the above while working on an immutable OS like GNOME OS, and without the risk of breaking it permanently.

State of the art

The recommended way in which these system components are developed and tested is through the use of Toolbx, see the following examples:

Unfortunately, the Toolbx approach alone is not sufficient for those scenarios described above. At least not in a way that isn’t painful. Thus, there’s a need for alternatives.

Pieces of a proposal

The proposal being presented here is built on top of previous improvements to GNOME OS e.g., switching to systemd-sysupdate and enabling the development tree as a sysext, other systemd features like systemd-repart and systemd-importctl, BuildStream and other common tools for building software for GNOME.

Putting these pieces together

To understand how these pieces fit together, let’s go through different aspects of this proposal.

What would be the development environment?

The primary option in this proposal is to use the regular version of GNOME OS, plus the development tree sysext, which comes with all the development dependencies to build any core GNOME project.

The fact that the development tree sysext comes with Toolbx provides extra flexibility, and is what many of us using immutable OSes have already been doing for the last few years.

Talking about containers, there’s also the core GNOME OS OCI which would be useful in scenarios where building the sysext images occurs elsewhere, like in continuous integration pipelines.

How do we build things?

Having the development tree sysext ensures that we’re provided with all core dependencies and tools to build any GNOME project. Of course, that doesn’t account for anything else. But that’s another reason for having Toolbx around.

In any case, the developer can use BuildStream and checkout the artifacts, or clone a particular core GNOME repository and build it manually with meson plus a custom DESTDIR.

Besides the artifacts we build e.g, mutter artifacts, what we would be ultimately building are sysext images that can extend the host. As noted by Jordan, having a single image file would make things easier to manage. Specially in near-future scenarios related to continuous integration (keeping that parallel to Flatpak in mind, imagine flatpak-github-actions or Flathub’s build bot).

Building these images could be done with an extended version of Jordan’s snippet, a proof of concept for gnome-build-meta (which uses systemd-repart to build the images), or even a combination of both.

Ultimately, what we want is some form of:

$ cli-tool build-sysext <name> </path/to/input/artifact/> </path/to/output/images/>

An example using BuildStream would look like the following:

$ bst build sdk/gtk.bst
$ bst artifact checkout --deps=none --directory sdk/gtk/
$ cli-tool build-sysext gtk ./sdk/gtk/ .
$ ls gtk.sysext.raw

How do we run what we build?

This one is simple. Once the host has been extended with the sysext images, the developer can access these artifacts in the exact same way they would access any other component of the host system. In fact, there’s no distinction at that point.

How do we manage these sysext images?

Once a sysext image is created, it does not take much to import it. See the following example:

$ importctl import-raw --class=sysext gtk.sysext.raw gtk
$ systemd-sysext refresh

Similarly, to remove it:

$ rm /var/lib/extensions/gtk.raw
$ systemd-sysext refresh

Although these steps might not seem like much, it would quickly become tedious. Specially when considering other details like pulling images from remote locations and handling integration steps like recompiling schemas, etc.

Therefore, similarly to the image building step, we could consider other ways to automate these details behind something in the form of:

$ cli-tool add-sysext gtk.sysext.raw
$ cli-tool remove-sysext gtk

A developer story

Putting together all what was mentioned above, a more detailed proposal looks like this:

  1. A developer installs the sysupdate version of GNOME OS.
  2. Turns that installation in to a development environment by using systemd-sysupdate to import the development tree sysext.
  3. Having all the development tools available, the developer would build their projects in a way that finds most convenient.
  4. Once the artifacts are available, the developer would make use of $ cli-tool build-sysext to obtain a sysext image.
  5. Then, the developer would run $ cli-tool add-sysext to test these artifacts from host itself.
  6. Lastly, the developer removes the sysext image by running $ cli-tool remove-sysext.

An early proof of concept for this proposal is being reviewed in gnome-build-meta MR #2886.

What else could be done?

Given that the $ cli-tool mentioned here only covers steps related to the creation and management of these sysext images, I wonder if more is needed to make that story compelling.

E.g., should we cover the artifacts building steps behind something in the form of:

$ cli-tool build-artifact /path/to/output/artifact/ bst ...
$ cli-tool build-artifact /path/to/output/artifact/ meson ...

Something completely different perhaps?

Your feedback is needed

I have been reaching out to some you for different types of feedback related to this, and I am also sharing this proposal here in hopes that it would reach to even more of you.

Your thoughts and ideas would help greatly to set this project on the right track.

You can also follow curent discussions and progress on gnome-build-meta, issues #819 and #825.

6 Likes
2

For the past couple weeks, I have been putting together a new project called sysext-utils with these ideas. Your feedback is welcome :slight_smile:

3

Hi!

It’s great to see some work happening in this, our current story for system components development is far from perfect, and we really need a better story here!

So here comes some questions from someone who spends a fair amount of time doing system component development.

Anyhow, question related to my particular setup:

  • I for the most part use a single laptop for everything. That means developing, testing, debugging, attending meetings, instant messaging, E-mail, listening to music, and so on.
  • I often work on problems that require direct hardware access, meaning virtual machines is not an option.

In what way is the proposal useful for a setup like this, or does it require the “host” (be it in a virtual machine or on bare metal) running GNOME OS?

For example, on my developer machine, the host OS cannot (for me personally) be GNOME OS, it’d have to be Fedora (Workstation or Silverblue); and the root of the host cannot be temporarily be “replaced” with something else via systemd-sysext, as that’d mean I’d have to close any “real” session just to test something, which is quite disruptive, to say the least.

4

Hey Jonas,

In what way is the proposal useful for a setup like this, or does it require the “host” (be it in a virtual machine or on bare metal) running GNOME OS?

In principle, the idea of using these extensions is not GNOME OS specific.

In practice, the reasons why this proposal is considered GNOME OS specific is due to the scope that I am currently aiming for in terms of a) the availability of tools required to build and manage these extensions and more importantly b) the steps required to properly integrate components into the host. As both of these things are quite specific to GNOME OS and the components I am targeting at the moment.

But, assuming your system does have the required tools and that it is possible to extend the tooling with the integration steps needed for your specific component. Then, this could be used in any other OS. I just haven’t got that far.

as that’d mean I’d have to close any “real” session just to test something, which is quite disruptive, to say the least.

That’s a good point. If we are strictly speaking about the “testing on my actual host” scenario, being completely nondisruptive it’s not what I was aiming for here. But, assuming this can be extended beyond the scope I described above, it would still allow you a) to test your components directly on your host (with direct access to HW, even on immutable OSes), with b) a better recovery story when things becomes disruptive (could be just a reboot away to safety) and c) in a way that is less painful (compared the alternatives I mentioned in the proposal).

If you have any suggestion, I am open to hear it !

5

So, please correct me if I’m wrong, a very short summary is, it’s command line utilities that helps to create and use an overlay (systemd-sysext) containing needed pieces of our platform that one can “soft” boot into.

If we are strictly speaking about the “testing on my actual host” scenario, being completely nondisruptive it’s not what I was aiming for here.

I see. A point to be made is that hacking on e.g. gdm, which is very difficult to do with without manipulating the actual host, could prove a perfect target for the work here, as it can’t really be done via containers. But when major disruptions (like needing to log out) is not an option, I fail to see any other option than containers (ignoring jhbuild).

Is any of the work here potentially applicable to containers? Can one use systemd-sysext in a container that itself runs systemd?

6

it’s command line utilities that helps to create and use an overlay (systemd-sysext) containing needed pieces of our platform that one can “soft” boot into.

Pretty much and, beyond the local development scenario, we are looking to use this in CI settings, e.g., someone sends a MR to system module X, the MR generates and publishes an extension that someone (or something) can test directly in GNOME OS.

Check the project README for more details :slight_smile:

I see. A point to be made is that hacking on e.g. gdm, which is very difficult to do with without manipulating the actual host, could prove a perfect target for the work here

Precisely !

Is any of the work here potentially applicable to containers? Can one use systemd-sysext in a container that itself runs systemd?

I haven’t tried this yet, but I think the approach would be to use systemd-nspawn, check this article from Lennart where this idea is explored.

7

I’d recommend using a VM at that point - a specific aim here is to make “build a sysext, boot a GNOME OS system with the changes” into something you can do as part of the edit/compile/debug cycle (i.e. something that takes <1 minute to do). Is that a viable option?

8

If systemd-nspawn and systemd-vmspawn start supporting sysext, that would be pretty much perfect. With them you can start the entire OS including systemd in it to test a proper boot and in theory the former might even work on a tty with drm/kms.

9

I am not sure nspawn will ever be suitable for testing changes to GDM, gnome-shell or gnome-session - I’ll be happy to be proved wrong :slight_smile:

The new systemd-vmspawn looks like an excellent way to test changes in a VM – it’s new in systemd 255 so apparently available in Fedora 40 – I’m upgrading my system now so I can play with it and see how well it works for this use case :slight_smile:

(Fun fact, systemd-vmspawn was developed in a separate collaboration between Codethink and the STF working on systemd’s integration tests)

10

I am tracking information regarding the use of nspawn and vmspawn in a sysext-utils issue. Check it out!

11

I’d recommend using a VM at that point - a specific aim here is to make “build a sysext, boot a GNOME OS system with the changes” into something you can do as part of the edit/compile/debug cycle (i.e. something that takes <1 minute to do). Is that a viable option?

It depends, VM may be perfectly valid for UI/logic changes, less so if you are dealing with issues close to hardware, or things like multimonitor. Testing changes to the greeter was never convenient, so a slightly longer change/test cycle is acceptable as long as it remains possible.

12

Thanks Martín for looking into this convoluted topic :).

My personal usecase is pretty similar to Jonas’, and I think my wishlist for a developer tool is covered here:

  • I should be able to develop/maintain multiple components simultaneously
  • I should be able to dogfood these as built from git all the time

A small detail that would be nice to make convenient is that development of core components sometimes do require too fresh bumps in dependencies, requiring (say) every GNOME Shell tinkerer to install libinput from git. There’s a point (weeks, months after) where that dependency already met its purpose and could even turn stale.

Perhaps it’s matter of getting used to the workflow, and systemd-systext list suffices to notice these after you suspect something is amiss, but it would be great to make these situations easier to track.

1 Like
13

Thanks for the feedback Carlos.

A small detail that would be nice to make convenient is that development of core components sometimes do require too fresh bumps in dependencies, requiring (say) every GNOME Shell tinkerer to install libinput from git. There’s a point (weeks, months after) where that dependency already met its purpose and could even turn stale.

That’s also a good point, and certainly an important detail. I don’t have a great answer at the moment, well, besides making a bigger extension out of these dependencies (or multiple extensions for each dependency).

I am actually thinking about the exact same issue for the CI scenario, e.g., how can we generate extensions that will run on a GNOME OS VM, when such dependencies bump are needed.

14

Perhaps for some, but not for me personally. I too often need to test on real hardware that a virtual machine can’t emulate, so virtual machines isn’t an option for the most part. The work flow is often also very 1. make change, 2. compile, 3. test, goto 1. and right now I can do those iterations without ever closing my text editor or terminals very quickly by just switching to a VT and pressing Up Enter, and having to sysext and log out, log back in, would make that a significantly more time consuming, disruptive and cumbersome work flow.

I do realize what I happen to do is extra difficult to target, since it both can’t be done with a virtual machine, but still need some level of os level environment, which for me right now is jhbuild and dbus-run-session.

15

This is useful info for setting requirements and expectations, thanks.

I think the most difficult set of requirements described here is this:

  1. Development and testing happening on the same device
  2. Hardware-specific code running that must be tested on bare metal
  3. System-level components being developed

Building with jhbuild against the host OS, installing into a prefix and running with dbus-run-session is a fine way to do this stuff if it works for you.

I’m curious what magic that dbus-run-session has that means you can test changes to system-level components on your host OS without logging out of the main session. Can you end up with multiple instances of systemd, GDM, gnome-session etc. that way on the same host ?

If so - it seems like that will remain the best workflow for you. (And to be clear, nobody is proposing that we take this away :- )

edit - i can see the dbus-run-session flow is documented at docs/building-and-running.md · main · GNOME / gnome-shell · GitLab, but the example there is running just gnome-shell. So the case where we think the sysext approach is most useful is where there are multiple things in flight, say you’re depending on an in-development feature in GDM or udev or something so you can’t just run against the /usr provided by your distro - is that something you currently deal with?

16

GNOME Shell and libmutter put a few D-Bus APIs on the session bus, so if you ever need to interact with those, you need to do your development within the same nested D-Bus session. This includes working on the display panel in Settings, some sandbox portal backends, gnome-remote-desktop, and other things. You can run gnome-session in it too, but then relies on --builtin being used.

Debugging multiple processes within the same nested D-Bus session is most easily done by running dbus-run-session somewhere, then using the $DBUS_SESSION_BUS_ADDRESS that it sets wherever you need to.

It definitely isn’t fool proof, and quite fragile. There are also issues with accidentally running applications that have active instances in the “real” session, that don’t expect two processes to write and read to/from the same files in $HOME.

I’m at the mercy of the people who keeps the light on in jhbuild : )

But I do from time to time explore alternatives, especially container based ones, since at least to some degree they can tick all those 3 boxes you mentioned, but nothing “perfect” has managed to get me to replace my current work flow, despite all the issues it has.

GDM, and really anything else that sits on the system bus that can’t easily just be temporarily shut down without too many consequences, are the things I don’t have a good work flow for, that also doesn’t really fit in a container, since it sits closer to the system, needs special privileges, doesn’t run as part of the session, etc, so it’s more difficult to “fool”, compared to all of the user session components. Here is what I think the work being done here will help the most, for me personally.

17

Ok - that’s useful to know. Perhaps we should use GDM as our test case rather than gnome-shell :slight_smile:

This is going a little off-topic, but are we close to being able to build smaller stacks of components using just Meson and subproject dependencies, without having to involve jhbuild to coordinate things?

18

I’ve got an out-there idea to propose, and I’m gonna dump a bunch of “we could do XYZ” here. I’m not saying it’s easy, or will happen soon, I’m just saying it’s possible.

I really don’t think VMs are entirely out of the question, and IMO they’re probably the best solution for iterating on stuff like GDM, gnome-shell, mesa, the kernel, etc in a realistic environment.

Does a GNOME session survive all GPUs being hot-un-plugged from the system, and then hotplugged back later? Presumably it goes headless for a while, and then when the GPU comes back it can re-initialize and put itself back on screen? I’m guessing support for this exists, since eGPUs have been a thing for a long time.

I bring that up because Single GPU Passthrough is a thing that exists. Basically it takes your actual display GPU, virtually unplugs it from the host system, then virtually plugs it into the VM. The VM can then directly draw onto displays physically connected to the GPU, with no SPICE, libmks, or anything like that sitting in the middle. Raw access to the GPU outputs. You’ve basically booted a second OS onto your system while leaving the first one still running in the background. Unless you’re working on the kernel’s PCIe/IOMMU drivers or other super-low-level drivers that would be aware of the VM, it should let you iterate on any part of the stack including the kernel. I suspect that for most anything that the GNOME community is involved in, this would be as good as booting on real hardware, just significantly more flexible.

We could do very similar things with trackpads, USB devices (wacom tablets), your bluetooth chipset, etc etc etc. I think most everything can be passed into a VM, and since most hardware is hotpluggable it can even be hotplugged back and forth between the VM and the host.

One of the few things we couldn’t pass through directly would probably be your keyboard: we’d need to virtualize it so that we can intercept some keyboard shortcut to allow you to return to the host system.

From there, we can build up tons of convenience. This is a very flexible solution.

  • Recent changes in systemd let us trivially and reliably take SSH control over the VM with 0 configuration
  • We can build sysexts on the host (pretty trivially: just meson install w/ DESTDIR=, then add the needed metadata file), then mount them directly from the host filesystem into the VM via virtiofs
  • Combining the virtiofs-sysexts with SSH lets us automate however much we want: you could make the GNOME Builder run button build gnome-shell, update the sysext, then tell the VM to refresh systemd-sysext and restart the session, then finally switch your hardware into the VM for you to test.
  • We can pass most things through a VM boundary nowadays, not just hardware. We could proxy through GDB, sysprof, and tools like that so that we can debug/profile/etc all the software in the guest from the host OS, and potentially visually with GNOME Builder.

Ultimately this sounds like a workflow that’s better than what you’re asking for: 1) Iterate on some component: gnome-shell, GDM, mesa, kernel, EFI, whatever. 2) Hit run in GNOME-builder. Compiled code is automatically reloaded in the VM. 3) VM takes over your computer. Test as if you’re actually booted on real hardware. Plug in displays into the HDMI port, etc. All in a fully real session, with the full suite of system services running. 4) When you’re done, hit some keyboard shortcut to switch back to your host system and repeat.

Point is, I think we shouldn’t discount VMs even for the use-cases that require access to hardware.

19

Same use-cases here as the ones Jonas and Carlos already mentioned before.

I used to use jhbuild back in the day, but these days I just go with the all-in option and sudo ninja install everything to /usr/local. That does break my setup (and even gdm) every once in a while, but it has the upside that

  1. it’s very easy to clean up in case you messed something up (rm -rf /usr/local/*)
  2. I get to dogfood all my changes, even in gdm
  3. it’s very quick to set up because I only need to dnf install a few -devel packages, then can build mutter and gnome-shell, and just relogin into the development session.

And while I agree with Adrians point that VMs could potentially be a solution to all these use-cases, all this seems very far in the future to me. And even then, a way to test things without VMs would still be nice, be it only to verify that a bug is not caused by improper handover to the VM.

I also like to hack on the kernel every now and then, so far I just make use of the kernels own install scripts (aka make bzImage/modules and sudo make install/modules_install) for that, and that works very nicely on Fedora. But I guess that part of the stack is kinda out of scope for this discussion…

20

So, you are building system extensions containing binary artifacts for a system component, with dependencies. Sounds a lot like a package to me… are we reinventing things here?

How is it different from your favorite package format, and why is it better?