I am happy to announce that GNOME now has an OpenQA service, which we can use to do automated testing of our nightly OS images.
There is an initial set of tests in the gnome-build-meta repository which cover the installer, gnome-initial-setup and a little of the Shell. The tests can be run manually on any gnome-build-meta merge request, and can be scheduled. Initial documentation is on the gnome-build-meta wiki.
I want to shout out James Thomas, Sudip Mukherjee and Javier Jardón for doing a lot of the implementation and setup work, and to Codethink for funding the time we spent. This is part of a wider testing initiative, which you can read more about in this article I published for Codethink.
What does it look like?
Here’s a screenshot of the latest test run as I write this:
OpenQA actually installs the GNOME OS image into a virtual machine and simulates user input, and the screenshots you see are taken from the virtual machine. It also records the whole test as a video, you can watch an example here.
Many thanks to the OpenQA team for such a powerful testing tool.
Can we run this on every merge request?
That’s not practical yet because building the installer is slow. We will likely set up a scheduled pipeline, and we can run it manually on any MR which could cause regressions in the VM images (such as updating the base Freedesktop SDK).
How can I get involved?
-
Join #gnome-os on IRC / #gnome-os:gnome.org on Matrix.
-
Enable notifications for gnome-build-meta and investigate any test failures that you see.
Can we add more tests? Can we add tests for $APP here?
We can test anything built in gnome-build-meta CI. Please do add tests, we just need to make sure the OpenQA test pipeline remains maintainable.
It’s going to take some time to figure out how we can get the most out of OpenQA, but I’m sure it’s going to lead to increased quality in our releases!