Is hard code freeze useful?

Hi developers, I’m very strongly convinced that UI freeze, feature freeze, and API freeze are useful for us. But I’m much less sure about hard code freeze. I’m wondering whether the hard code freeze before the .0 stable release is useful to you. It’s certainly good to avoid committing last-minute mistakes, but I’m just not sure it really makes sense anymore. Nowadays we have CI which can catch many mistakes that we previously would have only discovered post-release. It’s also quite easy to circumvent because we have no rule preventing you from releasing .0 and then immediately adding a bunch of bugfixes and doing a quick .1… in fact, I would even encourage releasing a quick .1 if you have bugs that you want to fix. There is no hard freeze before other stable releases, and it seems weird to do a serious freeze and then relax it.

Opinions?

3 Likes

tl;dr: I think it’s worth it. It provides a forced brake on development speed at the end of the cycle, which is good for project stability and good for developer burnout. I say this without quantitative evidence though.

One related thing which could help smooth things would be to make the backport workflow in GitLab better, so that it’s less painful to branch a project early and start development on the next cycle, without causing a lot of pain for backporting fixes to stable. But improving that depends on getting GitLab improvements, which seems unlikely without sustained pressure on upstream GitLab.

Reasoning:

Downsides:

  • It’s a pain to request a freeze break request and queue MRs up waiting for freeze breaks to be approved.
  • It’s more work for the release team to handle those requests.
  • Largely pointless for projects with excellent code coverage in automated testing, or very diligent maintainers.

Upsides:

  • Any last-minute changes either get punted (which should increase the stability of the next release), or are forced to get more oversight than they might do otherwise.
  • Having a deadline (the freeze) before release means people are less likely to miss the release deadline, even if they miss the freeze deadline.
  • There are plenty of modules which don’t have excellent code coverage in automated testing (either because they don’t have tests, or because they are UI apps), and there are plenty of modules without diligent maintainers (i.e. without maintainers at all), where last-minute changes can benefit from increased oversight.

I intend to give a longer reply later, but I want to add context. Here is the current release schedule:

https://wiki.gnome.org/FortyFour

There is a 2 week hard code freeze between 44.rc and 44.0 followed by an entire month before we ask for 44.1 module releases.

The important thing is a timely release, to ensure that the person doing the GNOME release does not spend a week hunting down maintainers.

Side note: the original proposal for the release process did not have a “hard code freeze”. The “deep freeze” was a requirement for releasing GNOME 2.0, since people were still pushing last minute fixes near the release date.

The code freeze period was needed because in the past:

  • branching was hard (CVS and Subversion were a thing)
  • we only had garnome/jhbuild, which required building from tarballs decoupled from the source repositories
  • we did not have gnome-build-meta
  • we did not have per-project CI pipelines

So “hard code freeze” was a “pencils down” moment to let the release team assemble the build without people going around doing last minute changes.

I’d say that our workflow and infrastructure have changed a bit since 2002, so I’d be amenable to dropping the hard code freeze, as long as maintainers are made aware that the time between the rc release and the point-zero release isn’t a free-for-all, and all the other freezes apply. Ideally, last-minute integration of branches should not happen at the tail end of a development cycle, when there’s no time for QA.

1 Like

I second this, I think the forced breathing room after the freeze helps with stability, but of course I also say this without any evidence to back it up.

BTW this conversation was prompted when I observed several freeze breaks in both mutter and gnome-shell. (Hard freeze began at 23:59 UTC on March 4, but many commits landed on March 5 and March 6.) So if we decide to keep the freeze, we should find a better way to enforce it.

1 Like

A wild idea. Releases a collaborative effort, where maintainers are actually fairly invested in getting in the best .0 possible, and are working with you towards a goal.

Another wilder idea, people generally don’t like being enforced, and maintainers would also get unhappy if a release is cut in front of their faces, missing things they felt important. If a release missed the deadline, it’s not just because writing a NEWS entry is tedious.

While there might be theoretical benefit in a pencils down period, its usefulness gets quite watered down with CI, and I find a too long freeze period to be counterproductive. There’s already tens of patches in Mutter and GNOME Shell awaiting a freeze exception to be made and approved, or the freeze to be lifted. These are issues that maintainers do know may hinder the perception of the final release, and yet we have to make these requests, re-explain the issue and argue why we think it’s better merged than out.

Sure, “asking is not much”, “it’s hard to not get approved”, but truth is we drag our feet in this way for a full month each year split in 2 cycles, and issues do also not get approved (inexplicably, anything goes after .0, we even occasionally need brown bag stable module releases, but no biggie).

There’s also the question about what maintainers/developers do with that period. Some nothing, some turn to doing docs and tests, others branch early and never look back to the stable branch. Do any of these help with the public perception of the incoming .0?

I agree with Philip’s proposal, we should look at improving the management of stable branches (perhaps look at the infrastructure of other projects like Mesa). With early branching and (mostly) automatically managed backports, the freeze period would be pretty much moot.

Short of that, I think a deadline on a fixed timestamp just wishes delayed module releases away, and is vastly incompatible with them. I think it would create a lot less friction if you considered the .rc tag the moment the freeze applies to a module.

3 Likes

I think it has never been clear, but most maintainers do actually consider this to be the case already: the moment the freezes start is the moment the release is cut. We’re all human, and we are not all working on GNOME in our ${DAY_JOB}, so a hard deadline is unlikely to be met unless releases are done well before hand, which is not entirely realistic.

3 Likes

Isn’t it discouraged to do releases well beforehand anyway, since then you miss translations?

Yes, that’s one of the reasons why early releases are problematic—though you can always catch up with translations during the stable cycle.

For the .0 stable release, yes. Please do not release before the soft translation deadline.

For other releases, I think it’s fine to release whenever you please, although I suppose we could add soft translation deadlines for alpha/beta/rc if desired by translators.

Hm, we’ve been brainstorming about automating the .alpha/.beta/.rc/.0 releases: cut the tarball at a particular time regardless of readiness. Thing is, while the .1 and higher stable releases should be done by discretion of maintainers, the development releases really need to happen at predefined times because releases that are even slightly late cause major problems for release team, downstreams, and other GNOME developers. I suppose I’m really much more concerned about the tarball deadlines than I am about the freeze itself.

Anyway, it looks like we don’t have consensus on whether to keep hard code freeze or not. Maybe a halfway-in-between proposal would be to make it optional next release cycle, see how that goes, and check again for updated opinions in six months. (Not that we’ll actually remember to do such a thing.)

1 Like

I can see benefits to automating some of the releases for a module (we might eventually reach the holy grail of one-button releases!), but also the point about maintainers feeling like they are lacking control over the timing is very true.

A random idea which just came to me: Maybe a middle ground would be to automate some of the releases for a module as long as there are no open issues or MRs set for that release’s milestone in GitLab? For example, I guess I’d be happy* with someone else / a bot releasing GLib 2.76.1 any time from now, but only once this one open MR is merged. A release before then would not have all the right fixes in it.

This does require maintainers to consistently use the milestone feature on GitLab. I find it to be really useful, but I don’t know how other maintainers feel about it.

*I’m lying, I’d actually be ecstatic if someone else did a GLib release.

My 2¢ (as I’m not a code-heavy maintainer, but tend to support developers in various ways): a hard code freeze seems alien and archaic to me in the world of decentralized version control systems and automated testing. Lately, I saw it mostly as an arbitrary two-weeks window where you are artificially prevented from landing any bugfixes to the main branch, and the fixes instead just accumulate in a big pile of extra branches (that your testers running nightlies can’t test), that may or may not conflict with each other, and that create additional work for maintainers as they need to remember to go back to the pile of MRs and merge a bunch of them right after the freeze ends…

Also, looking at the releases schedules (for example, 44’s), I see the freeze happens at the same time as the “release candidate” is supposed to be published, and ends with the stable “.0” release… so unless you want to do paperwork for each fix, that would mean you can’t land any fixes for issues “found in the RC” in time for .0… That’s essentially a two-weeks time window for “opportunistic zero-day testing & bugfixes” going to waste. What gives?

That’s certainly the definition that I’ve been using, and not just when it suits me: I only did the 44.0 releases on Sunday at noon, so by a strictly time-based definition the hard-code freeze was already lifted.

Regarding to:

It’s also quite easy to circumvent because we have no rule preventing you from releasing .0 and then immediately adding a bunch of bugfixes and doing a quick .1… in fact, I would even encourage releasing a quick .1 if you have bugs that you want to fix.

It could also be unintentional because It is be prone to human error. Here an embarrassing moment:

For 44.0 I accidentally circumvented the rule. I branched Evince early on when the hard freeze started. I did so because I found myself with some time and I wanted to merge some patches for the next cycle.

Fast forward some days, I got caught by the release date while travelling. I did the release when the airplane was taxing to take off. I did not realized that I was on main instead of gnome-44 when preparing the release :man_facepalming:t5:. I only noticed it some days later. Instead of asking for a bunch of freeze breaks, I made a .1 release from the correct branch, with some less features than .0.

The morale of the story is that this is an honour system, and as such it is also prone to errors. Fortunately, these are exceptions rather than the rule.

The second morale of this story is to avoid doing releases when the airplane is taxing, and pay attention to the instructions of the flight attendants instead.

3 Likes

From a distro maintainer perspective, I think the part of the release schedule from hard code freeze to the first point release does not work as well.

I would like to get bugfixes to our users quicker. Having a 2 week hard code freeze followed by 4 or 5 weeks before the .1 release is a long time for people to wait to get their bugs fixed. The first point release is probably the most important since it is expected to include fixes for the most visible bugs.

It is mitigated by some maintainers know that they can ask for a Hard Code Freeze exception and will probably get it, but other maintainers may either not be aware that requests are likely to be approved or not want to do the extra paperwork. And maintainers can release a .1 early; some do.

Getting more bugs fixed between RC and the Stable Release is better for everyone. I think the only exception is if major changes are made then that could introduce new bugs or cause integration problems for distros, but it is just as disruptive if changes like that are made for .1 or .2. Therefore, I support eliminating the Hard Code Freeze milestone and replacing it with general advice about saving refactoring and major changes to the development series.

1 Like

So we received mixed feedback in this issue: several comments in favor of keeping the freeze, and several in favor of removing it. This makes it a little tough to decide what to do, but the freeze is enforced by release team and nobody from release team seems to be a proponent of it, so let’s try removing the hard freeze for GNOME 45 and see how it goes. We can always bring it back for 46 if maintainers feel it doesn’t go well.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.