Hello,
Back in August I announced openqa.gnome.org, which I hope will become a powerful tool for integration testing of GNOME.
In the last couple of weeks I had a bit more time to improve the setup, here’s what’s new:
Server updates: you shouldn’t notice a difference here, but rest assured the server is now up-to-date and config managed using Ansible. Let’s not talk about how it was set up before 
Needle updates through the web UI: if a test fails because an app’s UI has changed, you can use a point-and-click interface to update the screenshot. This can now also commit it and push to openqa-needles. Full instructions here.
- You need Operator permissions to use this feature. I would like to assign this to everyone with a GNOME account (issue), currently an OpenQA Administrator needs to assign it manually after you log in with your GNOME ID.
Six apps being tested: here’s the fun part! The plan is to add tests for all core apps, the first six are ready: https://openqa.gnome.org/tests/74#
MR is here, it’s not mergeable yet because of ongoing breakages in gnome-build-meta.
Please help the GNOME integration testing effort by checking gnome-build-meta after you add dependencies, rename your ‘master’ branch, switch build systems, etc.
Tests moving out of gnome-build-meta: the tests are moving to a separate repo openqa-tests.
I’d prefer the tests to live in gnome-build-meta, but every time I commit to them the full track → build → iso-installer CI pipeline has to run, adding a 2 hour delay. OpenQA is meant to be useful to all GNOME developers but the process for writing tests needs to be fun and quick for others to get involved.
To help people develop OpenQA tests, openqa-tests will have its own CI pipeline but using the prebuilt images from os.gnome.org (s3-image) instead of building from scratch each time. This is waiting on two things:
- a suitable Gitlab runner for the openqa-tests repo: gitlab runner with KVM for openqa-tests repo (#727) · Issues · Infrastructure / Infrastructure · GitLab
- a working gnome-build-meta build so we can merge openqa: Move tests into separate repository (!1362) · Merge requests · GNOME / gnome-build-meta · GitLab
I’d like to shout out Fabián Orccón who has been volunteering time to look at some further improvements to the OpenQA setup: openqa: Split up install from later tests (#450) · Issues · GNOME / gnome-build-meta · GitLab
And thanks to Codethink for letting me spend some work time on this.
Happy testing!