openQA testing update, 2023/11 edition

Desktop
1

Hello,

The openQA tests for GNOME OS are running as ever. Since the last update we are now automatically gating https://os.gnome.org image using the test-s3-image job, so every new GNOME OS image is promoted to “stable” if the tests pass.

The testsuite itself is still small, as we only want tests that people will love and care about. We just have one test for each core app in the GNOME OS tests repo (openqa-tests.git). More detailed app tests need to be maintained in the app’s own Git repo, so they can run on feature branches and merge request pipelines before changes are merged. So we have been waiting for that to become possible.

And now it is!

App tests

This week I made the first CI pipeline which builds a Flatpak app (in this case Amberol), then runs an openQA test which installs and runs the Flatpak bundle that was just built, on the latest GNOME OS image from os.gnome.org.

image
image1024×768 77 KB

Here’s the Gitlab CI pipeline and the openQA test job. The whole build + test pipeline took just under 6 minutes.

There is plenty to improve here, and I would love to have some project teams become early adopters and help me figure out how we can make this useful for all GNOME teams. Let me know in the replies if you’re interested in helping. The only requirement is that your app publishes a Flatpak bundle in CI.

(Testing for system components like GNOME Shell is also planned, this requires using systemd-sysext rather than Flatpak to deploy the module into the GNOME OS image, and the remaining challenge is building ABI-compatible sysexts in CI pipelines).

Outreachy

It’s a particularly good time to get involved because we are participating in Outreachy this season. From December to February we expect to have two interns working full time on end-to-end testing. The participants will be confirmed next Monday 20th November.

The main goals are system-level tests around mobile form factor and accessibility features, but there will be time to create tests for some core apps as well. So act now and you may get some extra help to set up tests :slight_smile:

5 Likes
2

Hello Sam, I would love to help with the tests as am drawn more to accessibility features.
Great article!!

3

Thanks for the initiative @sthursfield!

4

How can I make use of openQA as a Core app maintainer?

For example, in Loupe, an unintended icon change happened and the invalid change just got accepted.

Is there a way to get notified of changes and errors?

5

I was hoping someone would ask this :slight_smile:

As of today, you can subscribe to notifications in gnome-build-meta. When a pipeline fails on branch ‘master’ in job `test-s3-image’, that means the openQA tests failed. Inside the job log for ‘test-s3-image’ is a link to openqa.gnome.org where you can inspect the test results and see if your app test passed or failed.

I haven’t advertised this workflow too much yet, as I am not sure how convenient it is. You will get notified on any failure in gnome-build-meta, not just the ones for your apps. Still - if you do try it, please tell me if you find it useful!

In practice, I check the openQA tests, and on failure I decide whether to update the screenshot. If something is obviously broken I will open an issue, otherwise I try to ask the app team. In your example above, I (incorrectly) assumed the app’s design changed and nobody updated the test.

Here’s an idea - what if we make a list of “owners” of different openQA tests? You could add a note for the app_loupe test saying “In case of test failure, content us in #whatever-your-matrix-room-is”, and I’ll promise to link test failures in that room whenever I see them.

Looking further to the future, the idea is you can now add openQA tests directly in your project repo so they run on every merge request, if you set that up. This has just become possible and still has plenty of rough edges, but it’ll mean that the app team has complete control over the tests.