GNOME openQA tests - looking back at 2023 and forwards to 2024

[sthursfield (Sam Thursfield)]

Hello everyone,

During 2023 we made quite some progress on end-to-end testing for GNOME with openQA.

Some specific highlights of 2023:

• 2754 tests run since we set up openqa.gnome.org in 2021
• Various issue reports to GNOME modules
• New tests: system shutdown, Loupe, Snapshot, Nautilus list view, and Dark Mode switch
• Updates for visual changes in GNOME - font rendering, Adwaita restyling and GTK4 ports. Thanks to Jordan for help with a lot of this
• An ongoing Outreachy internship to test accessibility features and GNOME on Mobile
• Documentation and developer experience improvements: a CONTRIBUTING.md guide, the ssam_openqa tool, the pipeline_report script and more
• Conference talks at FOSDEM, GUADEC, and XDC

We did not have any funded initiatives this year other than Outreachy - all this was done by volunteers and in bits & pieces of downtime.

Thanks to everyone who contributed in 2023:

• Tanjuate Achaleke
• Dorothy Kabarozi
• Abderrahim Kitouni
• Jordan Petridis
• Reece
• Riya
• Javier Jardón
• Mikkel Hansen
• Scott Clarke
• Valentin David

And thanks to Bart and Andrea for ongoing infrastructure support.

So what’s planned for 2024?

Investment

We have two Outreachy interns funded by the GNOME Foundation, who are funded until the end of February 2024

I aim to continue to spend ~1hr/wk of work time maintaining and improving the tests this year.

In principle, QA testing falls within the GNOME STF investment. There isn’t currently anyone funded by STF to work on openQA, partly because none of the people we know with existing experience are available for contracting. Let’s see if this gets anywhere in 2024.

I’d like to see more corporate investment and more volunteer contributions in GNOME’s openQA tests. Closer collaboration with downstream QA teams is one possible way to achieve this - see below. The bus factor of the test infra and the tests themselves has increased a little in 2023, and I’d like to see it increase further in 2024

Community

I’m hoping to meet some openQA core devs at FOSDEM this year and plan how we can collaborate more, ideally organising a hackfest further on in the year.

We have talked about a cross-distro QA hackfest, inviting folk from Ubuntu, Fedora, Debian etc. who use openQA to test GNOME, to see how we can share effort.

Within GNOME, the test maintainers need to know when failures correspond to bugs and when they are deliberate changes. There’s some technical work here to make tests more visible to everyone; but it’s mainly that we need module maintainers themselves to monitor the tests more closely, and proactively updating tests when landing visual changes.

I also want this to be the year where apps and modules can start using openQA to do pre-merge testing, with tests that live in-tree with the module source code. Running the tests is already possible at least for apps. The remaining gaps are mostly making it easy and fun to create and maintain openQA test suites.

Developer experience

As our interns Dorothy and Tanju will tell you, creating new tests and particularly new screenshots is not as simple as it should be. Some ideas for improving this:

• A workflow for running a local openQA instance, complete with developer mode (in progress)
• Integrate or build some other GUI tool needle editor, e.g. Qanvas, into the workflow

We also still need a way to quickly update needles after UI changes. OpenQA’s developer mode helps with this to an extent. We can’t actually use “developer mode” at openqa.gnome.org due to infrastructure limitations, though.

Since we don’t have a huge budget for running infrastructure at GNOME, my focus will be on making local QA workflows better rather than relying on shared infra.

Infrastructure

You can only gain Operator access to openqa.gnome.org with a GNOME LDAP account, which limits who can contribute. It would be nice to solve that.

Running the openQA web UI via Openshift would be appreciated by the infra team. If anyone has some idea how to do this, let me know.

Each time we run the openQA tests it takes 10-15 minutes of time on a Gitlab runner. Currently I don’t think we’re hitting any capacity limits. However I’m conscious that as we start running more tests, we start using more capacity.

Pre-merge testing

Currently the openQA tests can only run on gnome-build-meta’s master branch. This is due to some infrastructure changes which need to happen before we can push GNOME OS images from merge request pipelines.

Once pre-merge testing of MR’s is possible, it unlocks a great workflow for testing changes to system-level components (think gobject-introspection, GJS, etc): create a branch of gnome-build-meta with your changes, and the CI will build a new image and run a test cycle, finding any obvious breakages without you having to locally build and test every GNOME core component.

Release process

We need to branch the tests and clean up old screenshots so that the ‘master’ branch corresponds to the current, in-development GNOME release only. And we need stable branches for released GNOME versions to test the .1, .2, .3 releases of those. There’s a small technical blocker to this, and then the actual cleanup work to do.

More tests

Last but not least, extending the test suite further.

Tanjuate and Dorothy have recently added comprehensive tests for various GNOME accessibility features and will soon move onto testing mobile form factor. We’ll be showing this off separately.

I want my role this year to be empowering people who want to add tests. If you have ideas for what you want to test, let me know and I will show you how I think we can do it.

Hardware testing is also on the to-do list. This isn’t likely to come from a volunteer working on their Saturday afternoon, hopefully businesses will see the value of ongoing investment in testing GNOME on hardware and will make it happen. Spread the word!

Get involved

All of the above is stuff I’d like to see happen in 2024, of course at 1-2hrs/wk that gives about 80 hrs of my time over the course of the year, much of which might just go on keeping the tests running. So we need more contributors if we’re going to make the most of openQA for GNOME.

We have open Gitlab issues for everything I described above at Issues · GNOME / openQA Tests · GitLab. Tell me what you’re interested to work on and I’ll help you to get started.