Hello everyone,
I am still poking periodically at automated testing of GNOME OS using OpenQA. In general the tests have been passing and occasionally catching real bugs. Nobody has been sponsoring any of this work for a while, so there is correspondingly not much development ongoing.
Recently I noticed the tests are failing in master due to a change in Adwaita that caused all app header bars to change colour. (Since OpenQA is doing screenshot comparisons, this change caused all the tests to start failing in master.) I spent a few hours this weekend trying to resolve that. Theme changes are a normal thing. If the OpenQA testing will ever make it to “production ready” status, we’ll need to be able to deal with them painlessly. So it’s important to figure out the best way to handle these cases.
It’s easy to see the issue in the OpenQA web UI (left is the old colour ,right is the new one):
Firstly, there is the communication aspect - I wasn’t sure if this change was intentional or not, until someone in IRC said it was deliberate. I still don’t have a link to the exact commit that caused the change. Where are these global theme changes decided? Where is a good place to check when I see a theming change in OpenQA?
Now, on to the technical details… by default, the testsuite fails at the first unmatched needle (screenshot). You can use the web UI to update it and restart the tests, which will take some minutes to fail at the next unmatched needle. This doesn’t scale when every needle needs to be updated.
OpenQA has a “developer mode” which is supposed to make this easier. I hacked this into working (details here) and tried it out today. It’s certainly more efficient than restarting the tests after each needle change. However, each needle change requires a number of clicks and waiting inbetween, so for the GNOME OS testsuite where more than 30 needles need updating, it is still more than an hour or work.
You can see an example of the process in a screencast here: openqa: A way to update all needles after theme/toolkit changes (#1) · Issues · GNOME / OpenQA Tests · GitLab
My ideal process would be more commandline based: you run the tests on your developer laptop (this is already possible), all the needles are updated based on whatever happens in the test run, and at the end you review them for correctness before commiting an update to openqa-needles.git. That way we could update the testsuite after a global theme change, in minutes. I don’t see a trivial way to implement this though, it might require changes to OpenQA itself, or at least re-implementing the protocol that the web UI uses to talk to the worker.
In the end, I didn’t manage to update all the tests because an infra issue appeared. Follow openqa: A way to update all needles after theme/toolkit changes (#1) · Issues · GNOME / OpenQA Tests · GitLab for an update whenever I have more time to look at this.
