A reflection on maintainership: what to do when the bus factor dwindles?

[azmeuk (Éloi Rivard)]

Hi,

I would like to put some light on gnome-online-accounts, and open a discussion about its maintainance. I have been watching the {Card|Cal}DAV support ticket for some times, and I suspects this ticket is symptomatic of a deeper problem in the GOA project.

The facts

The DAV ticket is the the oldest ticket for GOA on gitlab, and the most voted too (72 votes to this day). Far before the second most voted. Patches have been proposed for the CardDAV and the CalDAV parts for about a year now. I understand that there is interest in the subject, and hands willing to help. People on those tickets have been mentioning maintainers on gitlab, reaching them on IRC, but it seems they never succeed to catch enough attention.

Other less popular patches neither, even trivial ones and those submitted by trusted people. There are 24 waiting patches, but the commits in the past 6 months are only translations. In the 6 months before that nothing except translations and some janitoring. Bug reports are not always investigated, sorted or sometimes answered at all.

The problem

From a lot of metrics, the project looks unmaintained. And this is totally OK for a project to be unmaintained. I think everybody in the room is deeply grateful for the hard work that has been done so far. We know maintainers have tons of reason to do other things, have priorities that we cannot read on a gitlab project page, have family lives etc. All of those are good reasons.

With all the human factor in consideration, I think that not aknowledging the project being unmaintained, and not seeking for help, sends a negative message to the community. I am concerned by the tensions this situation creates, as we can see on that DAV ticket, where a lot of messages are “+1”, “is it ready yet?”, “please don’t +1” etc. As a maintainer of non GNOME projects, I know that unpatience messages can put a lot of stress and make maintainers just disable notifications and look away.

The solution?

How can we improve the situation?
First of all, is there facts that I read wrong or things that I do not know and that might make me see things otherwise?
Are there similar situations on other GNOME projects that have been solved?

What about:

• communicating on the fact that the project is unmaintained, and that it might be useless to submit patches at that point. The project Readme can be a good place for this. That would spare contributor human time, and waiting anxiety.
• calling for maintainers and reviewers. There seems to be willing people interested in this project, and when the bus factor diminishes, I do not know other ways than recruiting to save the life of both the maintainers and the project.
• If stepping up is time expensive, what about using Gitlab’s merge request approval mechanism - with some rules to define - to let the community merge trivial patches?

What do you think?

I hope this will bring a constructive discussion.

[ebassi (Emmanuele Bassi)]

Thanks for bringing this issue up for discussion.

From a 10000 feet perspective: the GNOME release team should be responsible for:

1. identifying modules that have fallen behind in terms of maintenance effort: small releases; missed development cycles
2. contacting the designated maintainers for a status report
3. placing modules into explicit deep maintenance mode/unmaintained states
4. acting as a reference point for potential new maintainers

The main problem is that any project that fits the bill for the first item is virtually undistinguishable from a project that has a low priority in the overall stack. For instance: GNOME Online Accounts may lack features, and may have bugs, but those missing features and those bugs might also very well be not blockers. The fact that DAV support has 70+ doesn’t imply that it’d be a fundamental feature, after all. I don’t doubt it’d be a desired one, but clearly it wasn’t a planned feature, or it’d be there already.

Not every GNOME module is equally maintained. Not every GNOME module is maintained at the same level for the entirety of its lifetime. Maintainers are stretched thin across the whole platform. This is not an excuse, and it’s not a way to discount feedback: it’s just the reality of any free and open source software project of the size and scope of GNOME. Getting new contributors to become maintainers is a complicated task in the best of cases; when it comes to projects that sit at the intersection of multiple other projects, it gets hairy pretty quickly, and we—as a project—lack onboarding resources.

The approach that the GNOME project has taken to limit the effects of a maintainer being overwhelmed and dropping their project has been to distribute the access to all GNOME source repositories to the whole of the GNOME development community. People with access to the GNOME group can (and often will) do releases when maintainers can’t; triage issues; review patches; and merge contributions; they will also pick up maintenance duties, if they are interested and have spare resources. The release team will mostly offer to spin releases and unbreak builds—assuming the maintainer wants to.

The obvious downside to this approach is that it requires anybody willing to pick up the mantle of maintainer to already be a part of the GNOME community. Of course, being a maintainer is a role that requires trust, and having the commit bit to all of the GNOME source code requires a lot of trust; that trust must come from somewhere, and it is usually best described in terms of previous involvement in existing projects. Triaging issues; reviewing code contributions; providing code and documentation fixes.

The good part is that you don’t need to be blocked on an unmaintained project: you can contribute to actively maintained projects, build your reputation, and get other maintainers to vouch for you. For instance, if the unmaintained project is a library or a service, you can work on the other GNOME projects that consume that library or that service. This will also help you into understanding how the pieces of the puzzle go together. Once you have acquired enough trust, you can get a GNOME account.

As I said above, it is my opinion that it is incumbent on the release team to figure out the state of the modules in GNOME, so I guess we (as in: the release team) will have to figure out a process for this.

[federico (Federico Mena Quintero)]

Thanks for bringing this up. GNOME is quite large these days, and like an old city, there are parts in everyday use that are in disrepair.

I definitely agree with @ebassi that the Release Team should identify modules which need help. This may help in gathering a few interested volunteers that we can intensively mentor to maintainer status. I’m offering my help to do the mentoring.

Now, if you’ll excuse me for changing topics to (ahem) talking about myself, but you asked about a similar situation for another GNOME module I took over librsvg’s maintenance in 2015, when it was more or less unmaintained, so I have some experience with that.

A little history

In February 2015, the Release Team posted to distributor-list about needing someone to fix security bugs which were outstanding in librsvg, which was unmaintained. I had casually worked on its code a couple of times, so I offered my help to fix them, and said that I wouldn’t do maintainership.

You know when someone nerd-snipes you, and you don’t even notice? Yeah, so a few days later I announced that indeed I would be taking over maintainership. In the interim, I had some private email discussions with people who were involved in the security bugs, the previous maintainer, and some friends in distros. That got me hooked.

Librsvg was in this state:

• Test suite was broken, and was not useful generally.
• Tons of compiler warnings.
• Plenty of untriaged bugs.
• Plenty of outstanding, untested patches.
• Extremely old README; minimal API documentation.

Etc. This was in the bad old days before Gitlab, so it didn’t even have CI.

There are people who give services to figure out what to do with this kind of situation. I contracted with Sumana Harihareswara to look over the state of librsvg and tell me what to do. She did a fantastic job and basically gave me a plan for how to revamp librsvg’s infrastructure, documentation, information for newcomers, bug triage, patch triage, test suite, how to establish relationships with interested people (e.g. distros, Wikimedia). I highly recommend that someone do this for gnome-online-accounts. I think the Foundation could pay for this kind of work; GOA may not be the most critical module in the platform, but it is a terrible user experience when it doesn’t work.

Sumana has been writing a book on getting open source projects unstuck. You can get an excerpt of the book from her page and follow its development there. Everything in there was applicable to librsvg, and I think it would be useful for gnome-online-accounts as well.

When people started to see movement in librsvg, it got more contributors. We succeeded in reviving it and porting the core library to Rust in about 3.5 years, and then another 2 years to port the rest. GOA’s code is not much bigger than librsvg was; reviving it is totally doable.

Quick analysis of gnome-online-accounts’s status

• No CI, which makes it hard for a maintainer to know if a patch breaks things. There is a merge request to add basic CI; it is better than nothing, and can be improved gradually. There are already of plenty low-level modules from which CI ideas can be copied.
• No test suite, which makes it hard for a maintainer to know if a patch breaks things. First merge the CI MR, and then start adding tests. GOA seems to be a lot of interfaces and a lot of JSON demarshaling for the various services it supports. Maybe write some mock implementations? Maybe paste some known-good JSON and test that it is parsed correctly?
• Old-looking READMEs and such. Turn them into Markdown, make them pretty, add the Gitlab badges so the project looks nice in gitlab.gnome.org.
• Untriaged MRs - there are some distro patches and others that should go in quickly. A good exercise would be to try to write tests for them, and get a test suite started.
• Untriaged bugs. Take a couple of days to quickly read through the bugs without trying to fix them; figure out which labels would be applicable to them (gitlab has a bunch of pre-made labels; maybe GOA needs custom ones).

Logistics and permissions

With librsvg it was easy to change maintainers, because I was already in the trusted group / etc. We need someone for GOA. Are you such a person (@azmeuk - you already have a bunch of activity in GOA)? Or do you know who of the people who have contributed patches we could ask to take on this job? What does Debarshi (the current maintainer) think?

[azmeuk (Éloi Rivard)]

I think I don’t fit the criterias since I have not done any significant contribution on the GNOME world at the moment. Neither do I know people invested in the project. I can at least try to review patches and issues if that can help.

Maybe we can summon @debarshir for clues?

[jfft (Jeff the accursed heisenbug hunter)]

I thought about this a bit last month, and one of the impressions I was under is that GNOME doesn’t have a clear procedure for situations like these, and managing the delicate social aspect of an unresponsive maintainer; Fedora has a policy for that, Mageia had a draft for it. Could be useful for the release team to come up with something for GNOME?

In some cases there are also maintainers who, with experience, are probably setting the bar for “new features” super high, because afterwards they’d often be “stuck with it”, including users not happy that XYZ new feature does not work well and oftentimes the person who brought a drive-by patch does not always stay around to maintain it afterwards. That, I understand, though there would need to be some sort of standard template text (part of the policy to be drafted?) that those people can use to plaster their project with it. Then they wouldn’t be getting in the weeds with newbies, but experienced hackers could still approach them with reasonable confidence.

In some rare cases, I’ve seen maintainers being downright hostile to new contributors, even when those new contributors are competent and bend over backwards to try to provide an improvement, and there is nothing you can do except witness the incredible damage these maintainers do to our community and image.
A true story to illustrate: roughly a year ago I saw a case where the official maintainer of “a well-known music player” ignored a contributor’s polite requests for comments on their merge request for 2½ years, then suddenly said, “I’m going to check back on this in about 3 months and if my concerns are still unaddressed I’m closing it.”… without having provided any indication of what their concerns might be (or that they even had concerns at all) then or over those years; then, when I politely and factually pointed out that they’re asking some impossible mindreading out of that contributor, they just deleted all the comments documenting their error (including my comment as well as their own) before locking and closing the issue. If the maintainer didn’t want the improvement (headerbar/client-side decorations) at all, then they could have just said No from the start and saved everyone years of grief and guesswork.
I thought that particular situation (and how it was handled) reflected extremely badly on the GNOME community for anyone who saw it happen; I found the maintainer’s petty behavior so toxic that I wondered, for a brief moment, about whether I wanted to remain part of GNOME—yet you know how long I’ve been here and how thick of a skin I have. The original contributor of that merge request has deleted their code out of sheer exasperation and disgust, and has since then been lost to that project forever. And that project also lost me forever as a QA & design person, because I want nothing to do with that kind of behavior.

The takeaway of the story is, in my opinion, that:

• A clear policy should exist to allow shared ownership/maintainership and, ideally, facilitate the spotting and transition of maintainership from jaded and burnt-out folks who don’t even realize or acknowledge they are being toxic to our community-building efforts.
• Admins & release team folks should be in close collaboration with “Engagement” folks so that we’re on the same page in terms of mediating such counterproductive behavior.

[matthew-salerno]

To speak more about GOA: I think there’s no lack of people willing to dig into the code here, in fact GOA in particular looks like a great place for new contributors familiar with C to get their feet wet. The issue seems largely to do with lack of trusted members to merge pull requests. I could be persuaded into adding new functionality or cleaning up issues if there was any reason to believe what I added would receive any feedback in the next month. As it stands I don’t believe any MR I might submit would receive feedback ever.
I also think it would be a mistake to dismiss GOA as something that isn’t used much, is unimportant, or doesn’t need updates. Considering it runs as a daemon and provides services many essential apps use, I would consider it a core service. And the amount of now popular missing services suggests that there is a need for updates. At this point in time I think GNOME should either consider finding a new maintainer or deprecate GOA in favor of something newer or nothing at all. Leaving it in its current state is telling developers of other apps that they can hook into a dbus service that simply isn’t cut out for the job in its current state.
Much like Federico said, I think it should be a priority to find anyone who can step in for the moment, even just to review MR’s to signal prospective devs that the project isn’t dead. I understand open source projects are run on volunteer time (well, open source desktop projects at least), and I appreciate the work that’s been done here. But it also seems irresponsible to be shipping Linux’s largest desktop environment with a project that has 3 year old merge requests for highly requested features and no feedback as to why it hasn’t been merged, with a whole year between now and when the last change to src was.
If it helps get the ball rolling I’d be willing to help in an effort to revitalize GOA, whether that’s code, organizing, or documentation. But I lack seniority to actually be able to maintain such a project.

[mcatanzaro (Michael Catanzaro)]

I think we should give maintainer access to anybody who is willing to step up and has at least a few previous commits to GNOME. Seniority is not required to maintain GNOME software: that’s too high a bar when we are in need of new contributors to keep projects healthy.

@debarshir, what do you think?

[1player (Stéphane Travostino)]

Given that the maintainer @debarshir has not responded once to a request for comments in the past 3 years since the MR was first opened nor in this thread, I think it’s time to move onto the next step and choose a new maintainer.

I don’t want to sound ungrateful to unpaid volunteers, but I honestly do not understand the reason one would completely ignore an issue for three years, even if it’s just to say “I don’t have time, leave me alone”, considering that their latest commit to GOA is on 30 March 2022. Free time is a precious resource, but telling people that one lacks time to look into an MR in three years is basic decency IMHO.

As others have said, the GNOME project needs a way to deal with these cases, or else it feels like some projects can be involuntarily or maliciously held hostage by unresponsive maintainers.

[pwithnall (Philip Withnall)]

@1player

I honestly do not understand the reason one would completely ignore an issue for three years, even if it’s just to say “I don’t have time, leave me alone”

One reason for a maintainer not saying that is because it means sticking their head above the parapet. The inevitable (perceived) followup to them commenting on an issue is someone asking for even more of their time or emotional energy.

Another reason might be that they receive more e-mails per day than they can read and reply to (even if they are perfectly organised and very efficient), and some of them just always drop off the bottom of the pile.

I have no idea whether either of these apply in this case, and I’m not suggesting that either of them do. But I think there are a number of plausible reasons why a maintainer might just never reply. Ascribing it to a lack of basic decency is perhaps a step too far.

[swilmet (Sébastien Wilmet)]

A reflection on maintainership: what to do when the bus factor dwindles?

After reading the title, here are my thoughts to avoid the “bus factor” problem, even if the number of developers reaches zero.

As a developer, make yourself replaceable. If another developer knows well the same programming language and underlying libraries, and if the codebase is well documented, this shouldn’t be a problem for another person to pick up the development and understand the code.

In a project like GNOME with many different modules, and with a lot of voluntary work, it regularly happens that no developers are still active on a module. This is in contrast to a project like LibreOffice with a monorepo: it doesn’t mean that some parts of the huge codebase of LibreOffice are basically unmaintained (even though it works fine). The difference with the monorepo setup is that if somebody proposes patches to an unloved area, chances are that it will be reviewed nevertheless.

A blog post that I found interesting also: Fostering a culture that values stability and reliability

[jfft (Jeff the accursed heisenbug hunter)]

For the specific case of GNOME Online Accounts, there is a BoF (workshop) happening on Saturday at GUADEC 2022. @ramcq, presuming one of you folks has access to a good microphone, could you add some sort of videoconferencing option for people in here that may be interested in participating remotely, if any?