Changing the way GNOME modules are released

Community
,
1

This is a straw man proposal meant to jump start a conversation about changing the way modules in the GNOME core are released; it’s meant to be iterated upon, and discussed, and maybe even potentially dismissed. If the proposal is going to be accepted, there will be a future announcement.

Releasing is hard, we all know it. You have to find what changed, write it down; then you have to run the build and ensure that it passes the test suite; then you have to generate a release archive; tag the release commit; bump up the version; upload the release archive and run the ftpadmin script. Rinse, and repeat, for every module you maintain—and we all know a lot of us, through no fault of our own, maintain more than one project.

Now imagine having to wait for every module in the GNOME SDK and GNOME core to be released, on a deadline, and figuring out if things are still building when you throw them all together.

Welcome to the release team when a GNOME release day is upon them.

The problem

We are all overworked and stretched pretty thin. New maintainers don’t grow on trees, and we all end up either maintaining too many projects, or having to maintain things in our limited spare time. This means releases get delayed, which end up delaying other releases, which end up delaying the whole of GNOME, and then the release team has to pester people, and it’s not fun for anybody involved.

Existing mitigations

These issues have been there since time immemorial. I mean: they were meme worthy, once upon a time.

The release team has already tried to mitigate these issues over the years:

  1. releases have been moved to the weekend, to give more time to maintainers
  2. the new versioning scheme has reduced the amount of releases during the development phase of the cycle
  3. the release team remains on hand, as a “maintainer of last resort”, to spin up releases of projects

These mitigations are not enough, and we have had multiple cases of delayed releases—especially if things like rebasing the run time on top of a new freedesktop run time line up in the schedule.

Proposal

The long and short of it, is this:

For GNOME modules, the release team is going to be responsible of cutting a release needed by the GNOME project.

This includes:

  • development releases (alpha, beta, and release candidates)
  • stable releases (.0, .1, etc.)

Any additional release is left to the discretion of the extant maintainers.

The release-team would “freeze” every project on the Monday of the GNOME release, spin the releases, and then “thaw” the projects.

Additional requirements

For the release team

Releases should not involve a release archive. The gnome-build-meta project should build from a signed tag (using the a key owned by the GNOME release team).

Ideally, the only things the release team would do for these releases would be:

  1. verify that the current tip of the project (“release candidate”) builds in dist mode
  2. verify that any reverse dependency of the project builds against the release candidate
  3. tag the release candidate commit
  4. increment the version number of the project

Of course, in case of build issues, the release team would then have to fix them to satisfy steps 1 and 2. In general, these steps are managed through per-project and global CI pipelines.

The CI pipeline should be able to also do a release archive from the tag, for downstreams still using those archives. The release archive would not be the source used to release GNOME.

For the maintainers

Maintainers should always keep the NEWS and metainfo/appdata files up to date whenever they land things they wish to note inside a release. This would avoid the time consuming bit of the release process, and would allow the release team to do the least amount of changes in a project.

Additionally, maintainers of core GNOME modules should set up a CI pipeline to ensure that their project builds. If they don’t have time, they should ask the release team for help.

Pros and cons

The main “pro” is the whole point of the straw man: minimise the blockers for a GNOME release. Another item in the “pros” column is that it would remove a boring task from the maintainers plate, and leave them to deal with the actual work of maintaining their projects.

The “con” is that maintainers will have to relinquish some of their “powers” for the greater good. This will lead some module to be less maintained in the short run, because doing releases is typically the time when people try to cram the most work possible into a project.

6 Likes
2

Maintainers should always keep the NEWS and metainfo/appdata files up to date whenever they land things they wish to note inside a release.

What about translation updates? Currently most projects list the updated languages in NEWS, but they are not landed by maintainers.

3

Would this also be a good opportunity to standardise on pre/post-release version bumps? Right now it’s pretty much random depending on the module.

4

Considering that translation updates are not properly attributable (the committer is not always the author, and that information is not maintained in the commit log), I think having a “Translation updates” line item in the NEWS/appdata file is enough.

Alternatively, a simple script that looks up the changes in the po directory and generates a list of updated languages would be acceptable, I think.

5

Well, somebody still needs to add it to NEWS. It’s not impossible that a translation update lands e.g. a day before the release and the maintainer won’t notice it to update NEWS in time.

A script would probably be a better approach - regardless of whether it lists the languages or just mentions that there were translation updates.

6

I was trying to be accommodating in the announcement of the new versioning scheme:

Q: I do pre-release version bumps; what do I do, now?
A: The preferred model is to move to post-release version bumps.

Because I didn’t want to overload the discussion; I do consider post-release version bumps to be required, though.

You have two options, which mostly depend on your project’s own versioning scheme:

  1. Plain post-release version bump. You want your Git repository to always have a different version than the latest release, so that people can target Git instead of the release archive.
  2. Pre-release version bump followed by post-release version bump. This is typical of libraries following the Cairo versioning scheme: odd means “Git”, even means “tarball”.

Both options are predicated on people being able to track Git as opposed to tracking releases, and both options require post-release version bumps anyway.

1 Like
7

A couple more strawman variants on this proposal:

  • Release team automated releases are cut only for unstable and .0 releases, stable releases remain subject to maintainer discretion. (This is probably a good idea to avoid unnecessary stable release churn.)
  • Release team automated releases are cut only if you miss tarball deadline, allowing maintainers to opt-in to the old way of doing things if desired, with fallback to automated release whenever the tarball deadline is missed. (Not sure about this.)
  • Forget about crediting individual translations in the NEWS? I gave up on that long ago, because writing NEWS is slow and hard. (That said, translators work hard and deserve credit somewhere. But maybe not here?)
8

But see also: Even/odd versioning is confusing, let's stop doing it?

9

Would this proposed policy apply to all GNOME modules, or just libraries? For example, Calendar and Settings are leaf modules that don’t block other modules’ releases, would the release team still release them?

10

I mean unless you want to stop mentioning translations entirely not mentioning individual translations doesn’t solve much - maintainers still needs to add the

- Translation updates

line, and since translations land independently you’d have to check new commits in case there’a translation.

Moreover - if a module is unmaintained but there were translations - nobody would add that line even if other maintainers are tracking their commits.

Unless there is indeed a script adding it.

11

Considering that some apps are also providers of an API for plugins, the line is somewhat blurred.

I’d rather have the release team control all releases, to be quite honest with you. As I said, existing maintainer can do more releases if they so choose—just like they can do more than one alpha/beta/rc release during development, or they can release more often than once a month during the stable cycle.

The sneaky bit of the straw man is to ensure that the release team can switch over to a tag-based release model, instead of a tarball-based one; this requires figuring out which tags to trust, and using a GNOME release team key to sign a tag would help ensuring that the release team has at least verified that a tag is safe. I am not sure if Git allows two people to sign the same tag.

12

They already get credit in the About dialog.

The main problem, as I mentioned in my reply to @alicem, is that the author of the translation (especially when using Damned Lies) is not preserved in the commit log; this means that you cannot already credit the proper translators in your NEWS file.

13

That’s fine. I’m okay either way, just will need some time and mental bandwidth to eventually adapt to this new workflow.

14

I’m a big fan of this idea. :slight_smile:

@pwithnall has already written this script! gitlab-changelog script looks at commits since the last annotated tag and among other things emits output like this:

* Translation updates:
 - Czech
 - Dutch
 - French
 - Friulian
 - Georgian
 - Greek, Modern (1453-)
 - Icelandic
 - Latvian
 - Vietnamese

I think you could approximate the “whodunnit” bit by looking at the Last-Translator field in the po file header (including all changes to this field since the last annotated tag).

15

Well, it’s a strawman proposal, so everything subject to change. We could exclude leaf modules. But that wouldn’t really solve any grand problems. Calendar is part of core, so it does block the overall GNOME release. Some hypothetical examples of leaf modules causing problems:

  • Library you depend on breaks API, fix is committed to gnome-calendar git master a day or two later, but not released. Release team does not notice until trying to build release, when we’re in headless chicken mode.
  • Meson gets a bit stricter. Fix is committed to gnome-calendar git master, but not released. Release team again does not notice until headless chicken mode.
  • Vala or GCC changes something. Fix is committed to gnome-calendar git master, but not released. More release team headless chickens.

These scenarios should be familiar. Pretty sure Calendar has failed to build in releases more than once. There’s just no way to predict what will go wrong when modules miss the tarball deadline. We can’t really notice in advance because there’s just no way to predict what will break when building from the latest releases until we see what has released and what has not. We could set up a “tarball CI” to build the latest tarball of everything and verify that it works, but that would probably be useless because it will work fine until release week, then everything will break all at once and we’ll wind up exactly where we are right now.

16

I think automating as much as possible is a great idea, and maintainers probably need help with this. Not everyone has made the jump into the CI/containers/reproducibility world.

Case in point: of the accessibility modules, only atk had CI for the last major release; pyatspi2 and Orca still don’t. Yelp doesn’t have CI. I haven’t checked other core-desktop modules.

The CI schemes vary widely, and impose different responsibilities on maintainers and contributors. For librsvg and at-spi2-core I’m really happy with freedesktop ci-templates, but for example, GTK doesn’t use them (its container image must be built by hand), so it’s hard for me as a contributor to make an MR that needs an updated image.

For librsvg, which unfortunately still uses autotools, the CI pipeline runs a “distcheck” job, and when I do the merge request to update NEWS/etc., I get the tarball out of the distcheck job’s artifacts. I copy that to master.gnome.org, etc. I’d really like to be able to tag a commit by hand, and have the “make a tarball → upload it to FTP” step be automatic and triggered by the act of tagging a release. I think we could have a good session during GUADEC or soon enough to brainstorm all the supply chain and security implications of enabling that :slight_smile:

I suppose things are a lot easier for Meson modules, where there is no “make distcheck” step that potentially needs a lot of babysitting.

If the release team embarks on a “CI all the things” task, maybe we could look for things to standardize / automate a bit? Publishing generated docs to Gitlab pages (which destination path?), publishing other generated stuff (coverage reports, what else?). All those would be easier for maintainers to add if there were a list of “cool CI toys and what to cut&paste to enable them”.

We can totally go to the model of “keep everything in a release-worthy state all the time”, but tooling needs plenty of work. I think its worthwhile.

3 Likes
17

At the same time, it reduces work for individual maintainers, but it also makes releases harder to do (both for maintainers and the release team), in my opinion.

See for example this article that I liked:

It basically explains that the more frequent it is, the more we will want to automate it.

A basic example is for the NEWS file, if I have a huge backlog of commits to summarize, it’s harder. So your suggestion to regularly update the NEWS file outside of releases is a good idea for everybody I think, especially if there are less releases. (it’s something that I was already doing).

1 Like
18

By the way, a more general comment about trying to standardize more and more things between GNOME modules: it can be hard to achieve, and it’s more work for developers, to keep reminding “oh, I also need to do this that way”.

As already said, it’s where automation is useful (here for releasing), and code re-use for CI (but also for other global initiatives, when trying to have homogeneity).

19

I think in the end this could be a watershed moment for GNOME much like our 6 month release cadence. It will take a bit of getting used to, but in the end I think it’s very much worth it.

I know there are things as a maintainer I need to do better at, and I’ve always felt keeping NEWS up to date incrementally was the right thing to do (even though I generally don’t). But it also means we can push that work off into the merge request so that contributors are in the habit of both summarizing their changes in NEWS as well as documenting them as part of MRs (when necessary). Even less for the maintainer.

Of course, this effort seems like a lot of work to implement, but I would very much love to be doing less repetitive work as a maintainer.

20

Yes, yes and YES!

For gnome-shell, we have dist scripts that pre-generate some resources, namely stylesheets(*) and man pages(**), so I’d still like to generate those. This is currently also done in a CI job, but the “download locally, upload somewhere else, run ftpadmin install” dance is quite annoying.

*) so RHEL doesn’t need a sassc package
**) so Ubuntu doesn’t need their python2-based asciidoc package

I do consider post-release version bumps to be required, though.

I did try that when adapting the new version scheme, but went back to pre-release bumps when starting to include “meson dist” in the CI pipeline.

The current workflow is as follows:

  1. update NEWS, metainfo, …, bump version in meson.build
  2. open a merge request
  3. there is a “dist” job that is triggered by changes to the toplevel meson.build
  4. after merging, tag the release
  5. the tag pipeline triggers the “dist” job again, followed by a job that publishes the tarball as artifact

That makes sure that “meson dist” passes before tagging the release, but is also tied to pre-release bumps.

There’s the additional complication that “meson dist” checks that appstream metainfo has current release information (I used to forget that all the time).

Adding an empty release tag during a post-release version bump pretty much defeats the purpose of the test; also appstream-util only validates release tags with a timestamp, which must not be from the future.

While I had switched to post-release bumps, I “solved” that by only adding the tag at the time of the release, and ignoring the broken distcheck in the meantime. Not great, and not an option since dist was hooked up in CI.

That’s not meant as stop energy by any means. I’m happy to give post-release bumps another go, but need suggestions how to adjust the current setup to make them work (hopefully without losing the checks).

Maybe only include the distchecks when using the “release” buildtype? Is it possible to have a git hook that enforces that a corresponding pipeline must succeed before accepting a new tag?