Hello,
Nearly 1 year passed since I last posted about OpenQA testing of GNOME OS.
A quick primer: OpenQA is a tool developed SuSE and used by several GNOME downstreams, including (of course) OpenSuSE. It allows us to boot a GNOME OS image in a VM, simulate clicks and key presses, and verify that the graphical output is what we expect.
GNOME’s OpenQA instance is still in a “alpha” level of deployment, you can browse it at openQA: Test results
The main announcement is that tests now live in a new GNOME/openqa-tests repo. The CI in this repo runs the tests against the last image published to https//os.gnome.org/. A full pipeline takes less than 5 minutes, so you can iterate and fix tests with a reasonable cycle time here.
You can still also test specific branches of gnome-build-meta, with the test-iso-installer-x86_64 job. This job will fetch the latest commit of openqa-tests.git on startup. The cycle time is slower - you will wait 30-40 minutes at least for the iso-installer-x86_64 job to run - but it allows us to check merge requests for regressions affecting GNOME OS before landing them.
Beyond that, not much has changed, in fact things bitrotted a bit over the last year. In some recent downtime I managed to fix a few things:
- Re-adding virtio VGA support to the upstream containers (OpenQA tests are failing: "qemu-system-x86_64: Virtio VGA not available" (#517) · Issues · GNOME / gnome-build-meta · GitLab)
- Updating all the tests for the UI changes in GNOME 42 + 43
- Finishing the openqa-tests.git split
This will stay in alpha status for the forseeable future unless more folk pick it up, the main blockers from my point of view to rolling this out further are:
- Theme / style changes can break a large number of tests, and I don’t know of a way to update all the screenshots (needles) in one go. (openqa: A way to update all needles after theme/toolkit changes (#1) · Issues · GNOME / openQA Tests · GitLab)
- There are intermittent failures during installation, which may point to issues in GNOME OS itself.
- It should be faster to run the tests against a gnome-build-meta pipeline, see openqa: CI should allow starting installer-iso-x86_64 task before build-x86_64 task completes (#430) · Issues · GNOME / gnome-build-meta · GitLab and iso-installer image is slow to build (#396) · Issues · GNOME / gnome-build-meta · GitLab
If you’re interested in exploring OpenQA, now is a good time to play with it, and you can get some useful automated testing done as long as you understand the limitations. Some docs are written up here: OpenQA for GNOME developers · Wiki · GNOME / gnome-build-meta · GitLab