The GNOME Infrastructure has been utilizing 2 different mirroring systems over the last few years, mirrorbrain (deprecated due to the underlying software being unmaintained) and later on mirrorbits. Both these tools allowed us to make good use of all the mirrors around the globe that donated bandwidth and storage to the GNOME Project. With the most recent infrastructure redesign (GNOME Infrastructure migration to AWS · Andrea Veri's Blog) we decided to review every single hosted service making sure the best architecture was in use. When we started looking at download.gnome.org we identified the following pain points:
Distributing tarballs that are then used to build RPM/DEB packages in Linux distributions is a possible attack vector, a malicious actor could gain access to one of the underlying mirrors and replace a tarball with one containing malware (see the xz case)
No control whatsoever on the underlying mirrors and their security practises
Requirement to manage yet another software, in this case mirrorbits
Having to manage new mirrors requests, deletions etc
Having to maintain an rsyncd instance for mirrors to remain in sync
For all the above reasons, we have decided to transition download.gnome.org to use CDN77, which has kindly been sponsoring the GNOME Infrastructure for around 3 years now. This will allow us to maintain a single source of truth and dramatically reduce the maintenance burden in operating GNOME’s mirroring system.
GNOME Release service will be deployed in production, this will allow GNOME module maintainers to build their tarballs from CI/CD directly, from there they will be able to upload their tarball directly to download.gnome.org
When 3. will be complete, master.gnome.org will be decommissioned
Hi,
there is a big difference between “will be allowed to” and “will be
forced to”, from my point of view. It can be a matter of habit, no
doubt, but forcing CI/CD for each project can be a burden.
Out of interest, without exposing personal secrets, how will one ensure
the project X installed tarballs only for project X by the maintainer
of the project X when CI/CD is executed and uploads random tarballs for
the release?
I suppose there will be a very detailed documentation how to setup it
in the CI/CD, because not everyone knows the internals enough (I’m one
of those whom do not, thus the learning curve would be enormous in my
case), covering all/most of the release specialties of the projects
under the GNOME umbrella (something one does on his/her own machine
these years). I also do not want to create a release from any commit,
this should be very specific place, not an automatic thing, not being
run by anyone passing around. A need to wait for the CI to finish after
a NEWS file update will be a very painful experience, not a better one.
The release process is a set of steps, sometimes non-trivial steps,
thus this proposal feels scary to me. Maybe it’ll make more sense when
the time comes, I do not know.
@mcrha I believe you understand that having you build a tarball that is consumed by distributions in your laptop is not what I consider a strong security practise. GitLab CI/CD is extremely flexible and customizable, and no you don’t have to build a release for every single commit, you can imagine that maintainers that are using CI today have better ways to handle this scenario.
The new GNOME Release services make uses of JWT tokens to only allow a specific module to install tarballs for that specific module and only when the request comes from GitLab CI.
We’ll be providing GitLab CI templates you can use to automate this and that will accept a tarball as the input source. The beauty of this process is that every module can build their own CI, there won’t be a unified building step process as it’d be hard to cover all cases, but there will be a process that has a tarball as input and the final output will be a successful tarball installation under download.gnome.org.
Huge kudos to the GNOME Infrastructure team for this. Being able to publish from tags in CI has been one of the things I’ve missed the most, both as a packager (like when people tag in gitlab but don’t upload the tarball), and as a maintainer. This will certainly be a remarkable improvement for many, looking forward for it to happen
Some instructions on how to undo and redo the steps (if possible) or corrective actions to be performed, if the tar from the wrong CI job gets published etc.
This will be part of the handbook, once the service has been set up and tested. We’re also going to include examples of CI jobs so that maintainers with no experience with GitLab CI can follow along—though becoming familiar with CI pipelines should really be a fundamental requirement for all GNOME maintainers, these days; it’s not 2004 any more.
There can’t be “the wrong CI job”: the CI job that generates the release has to inherit from a specific template, which takes care of most things outside the actual build instructions for generating the dist archive. Everything else is under the control of the template maintained by the GNOME release and infrastructure teams.