IRC, Matrix, and thanks for all the kicks

Community
1

Hello GNOME community,

I’ve come to talk to you (again :musical_note: ) about instant messaging platforms and peaceful coexistence. This is a pretty heated issue, so I count on everyone to keep the conversation constructive so we can shape our future platform together in a positive way!

TL;DR

Our experience with instant messaging is suboptimal on IRC and Matrix. We can make it better, but not on both ends at the same time because of their very nature.

Matrix and IRC users are able to communicate thanks to a bridge. The bridge creates an IRC puppet for each Matrix user, so it can post messages there in their name. At each bridge restart, all puppets leave then join again the IRC channels. IRC sees this as a flood attack and kicks all the puppets.

Today when a puppet is kicked from IRC, the corresponding user on Matrix is kicked from the Matrix channel as well. It is done to avoid having inconsistent states between IRC and Matrix. We now have the possibility to enable an option to bypass this. Once enabled, getting a puppet kicked on IRC would not propagate the kick to Matrix.

The downside of this is that it would also prevent a few messages to make it from Matrix to IRC for a few minutes, on very rare occasions that will be known beforehand so admins can announce it. While not ideal, this seems to be less disruptive for the conversation than having half of people suddenly removed from it.

Please keep in mind that the solution described in this post corresponds to what we can do to improve what we have today and my personal opinion about which platform is the most suitable for our needs. It does not mean that the choice of our recommended platform has been made.

IRC, Matrix, and thanks for all the kicks

Let’s walk together through our history of instant messaging, what our needs are, before moving to some problems we have already solved, what else we can do to move forward, and what I consider to be the most realistic way to change platforms.

First, a bit of history

The GNOME contributors community never really had an official platform for instant messaging. Historically, developers used to hang on the GIMPnet IRC network. This platform is still very active today, and many of our core contributors still rely on this platform.

IRC is a very text-oriented platform, and some other teams didn’t feel at ease there. For example, the design team tried Telegram in the past and found the ability to easily share images and media to be very helpful. Foundation staff also felt more comfortable on Telegram than on IRC.

In the meantime, there were coordinated efforts to bring most of our tooling to modern standards so we keep attracting new contributors. GNOME moved from the sturdy and battle-tested Bugzilla to the more welcoming GitLab. GNOME moved away from the 90s vibe of mailing lists to the more engaging Discourse board.

IRC is the last system powering human interactions that hasn’t been updated to a more comfortable solution. Newcomers didn’t grow up with the customs of IRC, and have very different expectations for their instant messaging platforms. Persistent messages, rich media, edits, text formatting, or even the seemingly silly reactions: most of those features are supported by the tools we use to chat with our relatives.

Some people from the contributing community heard about the Matrix protocol. They got intrigued, then convinced, and we saw the first demands from people from the GNOME community to get a bridge between Matrix.org and GIMPnet IRC in December 2016. The Matrix Foundation entered in talks with the GNOME Foundation, and eventually on March 2, 2017 Matrix.org was bridged with GIMPnet IRC. The promise sounded good: you could keep in touch with people from IRC, all while benefiting from the goodness of a more modern protocol and client. No hard split, no migration. Everyone could live on the platform they loved in harmony.

The company Element, which was then called New Vector, offered to host a Matrix instance and to bridge it with GIMPnet. As a result, the dedicated GNOME instance was deployed on March 18, 2019. The bridge was moved from matrix.org to gnome.org in April 17, 2019. Even if the Foundation didn’t host the server, it could be in control of things since they were hosted behind the gnome.org domain name. That’s the instance many use and love today.

Reality was a tad disappointing in terms of peace and harmony: on the GNOME end we didn’t do our homework, the bridge was not properly tuned to our needs and we didn’t deploy everything to allow the remote Matrix instance to work properly. The experience ended up suboptimal on both ends of the bridge. Before commenting, remember the point is that every tool is unpleasant when it’s not used correctly.

Shortly after, the Foundation’s ED asked for a trial of a Rocket.Chat instance on August 26, 2019. Today, this Rocket.Chat instance is still online and mostly used by Foundation staff.

A global community

Locally, different teams of the GNOME contributing community have different requirements. Still, it’s important to have a single place where we can easily reach out, bond, and discover especially during a pandemic when we are deprived of in-person events. One of the goals of the Foundation is to foster collaboration inside the community. The looser the ties between our members, the more the platform they are creating is likely to crumble.

That crumbling is already starting to happen because of the fragmentation of our tooling. IRC and Matrix are bridged, so it’s more or a less a global community, but Rocket is completely on its own. @kprogri made the very welcome effort to join the IRC/Matrix world to keep in touch with the rest of the contributors. But even within the IRC/Matrix world there is some fragmentation. Invite-only IRC rooms are nearly impossible to bridge without being kicked all the time. And some people got so fed up of IRC spam and kicks they created Matrix-only rooms for their project. Plus, all the kicking, editing, code-pasting issues are upsetting people on both ends of the bridge.

GNOME is also part of a wider community: the desktop environment community. We are more than happy to organize the LAS with our friends from KDE every year. So far, the organisation of LAS happens on Telegram. KDE folks were invited on our Rocket.Chat instance, but unsurprisingly it never really caught on. An open-source federated system where both organisations can use their own identity would be a huge improvement over the current situation.

At an even larger scale, the GNOME project is a successful project used by many others. There is a global movement of organisations getting out of silos. The open source community is widely adopting a federated system: Mozilla, Fedora, KDE, Debian, Archlinux… all those organisations are either Matrix-first or are transitioning to Matrix. If GNOME decided to stand out for the sake of it and decided to use another platform, it would self-segregate itself from others.

What we have achieved

It’s important for the target solution to answer most people’s needs so they can feel at home here. We cherish our long time contributors as much as we want to keep attracting new ones. IRC can’t be shut down by force, its users can’t be coerced somewhere else and that’s a good thing.

We’ve been sitting with IRC users to understand what makes it so important to their eyes. One of the key points was that they rely on IRC for work or to hang out with other communities such as Freenode. We found out that there was a CLI client that more or less allows to see Matrix as another IRC server. This way people who really need IRC can keep browsing their usual channels using the IRC protocol, and browse the GNOME ones using the Matrix protocol.

We also took into consideration complaints from IRC and Matrix users to try to make the bridge easier to live with. We finished the deployment of the Matrix instance on the GNOME side to make it easier to connect to.

Another common complaint was the bridged room alias on the IRC end. They were quite convoluted, since #newcomers on IRC was #_gimpnet_#newcomers:gnome.org on Matrix. We updated the bridged room aliases to match what people are used to on IRC (e.g. #newcomers on GIMPnet is now #newcomers:gnome.org on Matrix).

In terms of discoverability and decluttering the environment, we curated the Matrix directory so rooms are easy to find, and documented how to use it for every Matrix user.

We also learnt from Mozilla how to set up a moderation bot so abuse can be handled properly on Matrix. We have already banned a few people and ensured it’s technically possible to perform moderation on both IRC and Matrix sides from the Matrix end to keep our community safe.

Finally, Matrix is already culturally well anchored in our community. Fractal, the GNOME client for Matrix has been around for quite some time and is in very active development. Julian Sparber received grants by the EU to fund his full time work on Fractal. We have two GSoC interns to make it even better this summer.

What we can do

A few more issues can still be fixed with a bit of work and tweaking, but in the end compromises have to be made at one or both ends of the bridge. The current situation is IRC users are mostly unhappy about Matrix because it forwards events not supported by IRC, such as message edits or quote replies. And Matrix users are mostly happy except at very specific points of time… when this happens:

Screenshot from 2021-04-27 14-05-06

This happens because of how the bridge works. When people join a bridged channel from the Matrix side, the bridge creates a puppet user for them on IRC. When the user sends a message on Matrix, the bridge controls their puppet and uses it to send the same message on IRC. When a second Matrix user joins, the same bridge, with the same address, creates a second puppet user on IRC.

You might have seen it coming: when the bridge restarts, all the puppets leave IRC at once, and they all rejoin at once. IRC sees a burst of joins happening in a short period of time: it enters in flood protection mode and kicks every joiner. This means every puppet is kicked.

Since the bridge tries to keep IRC and Matrix in the same state as much as possible, when a puppet is kicked from IRC it kicks the user from Matrix as well to avoid having a split-brain view.

People from Element have landed a MR offering an option (disabled by default) so the Matrix user is not kicked from the Matrix room when their puppet is kicked on IRC. This means that the source of truth when it comes to instant messaging would be on Matrix. During a short period of time, people on IRC might not see what people on Matrix post.

The issue here is not a technical one, but mostly a governance one. Today the experience is suboptimal on both ends of the bridge. It is possible to make the experience exellent on one end at a time, but not both. One of those ends has served very well for a long time, but doesn’t have a future. The other has the potential to be a welcoming platform to work with others, with modern features every newcomer could expect in 2021. We can make the experience great on Matrix if we toggle that switch, but we can’t do it if we don’t accept IRC is now a legacy platform that needs to slowly fade away.

What we can’t do

The analysis is clear: IRC and Matrix cannot be bridged fully in a user friendly way. IRC was never designed to be managed by another system. It doesn’t handle automatic registration on behalf of others. Its defense systems against spam make it kick every newcomer whenever too many of them join at the same time.

IRC used to be the top notch system for a while, and is still pretty reliable. There are issues here and there, but given how basic the whole protocol is, it is fairly sturdy. The frugality of the protocol that was once its strength has turned into its weakness in the modern world. People’s expectations have shifted. Most of our contributing community is already “working for the computer” when producing the great software we all enjoy. It’s time for instant messaging to get out of the way, and to allow the computer to work for us when it comes to talking to each other. No NickServ spells, no ChanServ incantations, no bouncer to rent or to host on a server. It’s time for us to just enter our credentials, browse the channels, and enjoy the conversation with the ones we like to work with.

Such a thing can’t happen without a transition plan. IRC has been around for a long time, and people have good reasons to love it. It’s difficult to change one’s workflow after many years of good service. We learnt it the hard way with the Rocket.Chat experiment: it never caught on because the step was too high. It required too much effort to move a whole community from one system to a completely different one.

That’s where the infamous bridges can finally help us. By shifting where we want the best experience to be, we can keep the bridge for a transition period. We can make the experience on Matrix better, keep in touch with people who really want to use IRC, incentivize them to progressively join us, and eventually, one day, remove the bridge entirely.

26 Likes
2

Thank you @thibaultamartin for the excellent write-up and the glimpse into our messenger history.

People from Element have landed a MR offering an option (disabled by default) so the Matrix user is not kicked from the Matrix room when their puppet is kicked on IRC. This means that the source of truth when it comes to instant messaging would be on Matrix.

To me, this seems like a very good compromise. It improves the user experience on the Matrix side while not significantly hurting the experience on the IRC side. And as you correctly mentioned: we only want the bridge during the transition period until we can completely move away from IRC.

7 Likes
3

Thank you for the detailed summary @thibaultamartin !

Unfortunately, I recently had the problem again of being kicked out of several bridge rooms (e.g. Design / Hackers). I know this error message too well by now. This behaviour isn’t really newcomers friendly :confused:

I think we should enable that option, as it seems to be a good compromise.

In my opinion we should adopt a transition plan, and in the long run switch to Matrix, as also suggested by @thibaultamartin. Currently, Matrix is getting unfair criticism from some people, but this is often not a problem with Matrix itself, but due to the use of the IRC<->Matrix brigde. The bridge is a great help for the transition phase, but should no longer be needed in the end.

8 Likes
4

I have already explained my preference for Matrix in the survey. So just a big thank you to @thibaultamartin and all the people that make it possible that the tooling gets “out of the way” as much as possible!

6 Likes
5

After reading about the usability problems explained here, I have two suggestions that might be helpful.

  1. Last time I checked the wiki (both GNOME and Matrix), the procedure to store the IRC credentials in the bot wasn’t explained or was too hidden in the docs instead of being something that appeared upfront. This useful to avoid having to re-login and re-enter all +r rooms after each bridge reboot. In this mode of operation the bot uses the registered nickname you provide it as the puppet user on the IRC side from the first login attempt since it starts. I’m not familiar with the IRC minutae but my guess is that those protection measures against flood attacks apply for unregistered nicks. If that’s the case (or if it can be set up that way), that might solve most of the problems short-term (provided you have a registered nickname on the IRC side) without these split-brain issues you mention.
  2. The best solution longer term is to run only the Matrix server and proxy IRC clients with something like this. It provides the most seamless integration with no compromises since we get to keep only a single source of truth. The downside is that this path involves developing over that project or starting another from scratch. The upside is that it’s very likely that other communities have similar transition needs as us and/or it can make transitions possible where it was deemed too involved.

If we commit to the last option (which I argue it’s a noble one to pursue for the net effect and unpredictable but positive impact it can have on us and our place and reach in the FLOSS community), I strongly recommend joining developer manpower from us and other communities and work over the project I just linked (I expect they will welcome the improvements). Note that none of the options are mutually exclusive.

Edit: Nevermind the first option, it is already well placed here. Definitely things have changed since last time I had to deal with all this stuff when I was a rookie.

1 Like
6

That’s a very valid point. We have a more general onboarding problem. It’s not very clear for newcomers which services we provide, which ones are connected to a GNOME account, what a GNOME account is in the first place or even how to get one. I need to make room in my schedule to finish our service catalogue, and what each user profiles (anonymous, registered user, GNOME Foundation member) can get you.

Since refactoring our onboarding process is a tedious task we need to address in the long run, I had updated the procedure to register a Matrix puppet against IRC is now part of the IRC/Matrix onboarding guide.

It would indeed be the cleanest way to do it because at least a single system would be authoritative on the accounts. As you said, it would even be a single base of accounts and it would “just” be proxified.

But people using this interface would still connect through the IRC protocol. Some limitations of IRC can’t be overcome. People using this proxy wouldn’t support edits, replies, and code formating. Those the main complaints IRC users have today when it comes to interactions with Matrix, and such a proxy couldn’t address them.

Deploying a non-trivial amount of paid and volunteer efforts to only address a tiny bit of the problem (the NickServ spells) is probably not worth it.

The MR Element has landed on the current bridge has a way better effort/benefit ratio. In practice it means every other month people from IRC might miss some messages during a few minutes. That’s quite an acceptable tradeoff if you consider it as a legacy system :slight_smile:

7

That’s a limitation of the IRC protocol, as you say. If IRC users complain about the lack of features of the service they use, well, I guess the solution is on their hands, not ours. The feature mismatch can be made nicer with some human-alike-but-trivial-enough translation rules tho (like using an external service for code blocks that exceed some length and sending a link through IRC), but that can be done both in the bridge and the client proxy.

If you aim to solve only that problem it looks like it is overkill. I don’t say “it is overkill” because some people might not be happy with this inconvenience. Legacy doesn’t mean unreliable where it wasn’t from a UX perspective. I’m personally OK with this tradeoff, but we have to consider other scenarios.

However, an enough-featured client proxy would change things at a more fundamental level, to the extent of making the experience of creating and managing rooms the same as full native Matrix. No complications or having to think about the bridge when doing those tasks. Then there’s the argument of thinking at the whole FLOSS world level and actively making IRC a thing of the past and all that it means for its future and newcomers everywhere by providing such strong means for it without leaving IRC users behind and forcing them to another UX, but that is a whole bigger topic, almost philosophical, and goes beyond us.

8

On the “Libera Chat” spam

First, of course, we know that despite what the spam indicated the spam did not actually come from Libera Chat teams. It comes from imbeciles who obviously wanted to give Libera Chat a bad image by flooding all sorts of disgusting messages.

What happened?

Let’s have a clear look at what actually happened.

  • Many IRC channels were flooded with spam messages. Given our Matrix instance and the IRC network we rely on are bridged, the flood messages were forwarded to Matrix
  • Those IRC channels were not in +R mode. This is the mode that only allows people who are registered on IRC and have cast the NickServ spells to join
  • Most of those channels were not set on +R mode during the flood
  • Since the flood matched simple expressions, it would have been easy to prevent it at server level but it didn’t happen
  • Several people have been granted privileges on Matrix and have been running scripts to clean up all the garbage left by IRC spam
  • New versions of our Matrix instance have been deployed an hour after the spam started, with more agressive spam filtering so the messages didn’t make it through anymore. Spam kept happening on IRC but didn’t make it through Matrix.

What can we learn?

As one of the persons who had to manage all the cleaning of the spam on the Matrix side, this was a very frustrating evening. It really felt like bailing out water of a boat with huge holes in the hull. And to be completely honnest: this was not a very sophisticated nor massive attack, making the struggle even more infuriating.

The easy solution could be to blame the computer. I could ragefully blame IRC, and actually did during the flood, out of exasperation. But the problem is not IRC. The problem boils down to two major issues:

  • The registration/authentication experience on IRC is really cumbersome: as a result, most IRC channels don’t have a lock on the door
  • We don’t have a proper moderation team with sufficient privileges to react quickly on IRC, and worse: we don’t have sovereignty over the IRC network we use to be able to set-up spam filters at server level. People who run it are volunteers who do their best on their spare time, and there is no real decision-making process nor way to enforce decisions on all servers of the network. It leaves the door open for spam on both platforms and is an effective risk for our users

The key takeaway after discussing with the GIMPnet administrators is that GIMPnet is running in maintenance mode. The administrators are keeping the network running, but don’t have the time or energy to carry out significant evolutions.

What is our way out?

The first solution that comes to mind would be why don’t you just put a lock on the doors then? And of course, like many of the why don’t you just questions the answer is not as simple as it seems. Today, GIMPnet is not making any exception for the IRC/Matrix bridge. Setting blindly all channels in +R mode on IRC would degrade the experience both on IRC and Matrix:

  • IRC users would need to cast the NickServ registration and and authentication spells
  • Matrix users who didn’t register on IRC and taught the bridge to play their credentials could just not join the conversation
  • It would require every existing and new channels ops to set-up the +R mode

Not great.

But I still have pretty high hopes that we can do better together. I had a very short and informal conversation with GIMPnet administrators who told me:

this recent upheaval has pointed out that there might be some need for a bit more modern processes and maybe getting some more active staff in that might have new and bright ideas

We still need to have a more formal conversation with the GIMPnet administrators and @averi to push the idea forward. If we can manage to have special privileges for the IRC/Matrix bridge on GIMPnet, we can find solutions to automate spam management without degrading the experience.

It is possible to set more agressive anti-spam settings on IRC, all while putting the bridge on an allow-list. This would require a bit of tweaking on the IRC side, but would definitely make the experience better at both ends.

9 Likes
9

As part of Fedora’s transition to Matrix, we’ve plumbed through all the rooms and we configured the rooms so that Matrix users are exempted from the registered nick requirement through the access list:

Excerpted from the #fedora-kde access list:

[20:11:51] ChanServ [ChanServ@services.]: Entry Nickname/Host          Flags
[20:11:51] ChanServ [ChanServ@services.]: ----- ---------------------- -----
[20:11:51] ChanServ [ChanServ@services.]: 8     *!*@gateway/shell/matrix.org/* +V [modified 29w 4d 2h ago]

Perhaps something similar could be done for GNOME as a transition measure?

2 Likes
10

It’s always nice to see what sister projects are doing, thanks for letting us know @Conan_Kudo!

Unfortunately, GIMPnet is powered by unrealIRCd which doesn’t allow such exceptions. I’ll bring the issue to GIMPnet administrators anyway to make sure I didn’t miss something in the docs :slight_smile:

11

Probably not a surprise to anyone, but I’d also advocate for fully switching our communications to Matrix in the long run.

It also makes sense to me to enable the option and prevent kicking Matrix users from bridged channels, those random kicks are the last major annoyance with Matrix in my opinion.

1 Like
12

Thanks for this excellent summary.

In case folk aren’t aware, I would like to highlight another solution / workaround for chat fragmentation. Most chat protocols are now accessible through the Web, and some apps provide specialized browser UIs suitable for managing multiple chat webapps. The two I know of are Tangram (GTK-based) and Ferdi (Electron-based).

To access IRC from a browser I use TheLounge web client, there are probably others too.

I would still love to see GNOME nominate an official platform, for reasons of community cohesion more than anything :slight_smile:

1 Like
13

@av and I had a really uplifting conversation with (some of) the GIMPnet administrators!

The key takeaways from the conversation are:

  • IRC opers are depleted and would happily welcome new staff to help fight spam
  • the IRC network will undergo routine maintenance for updates shortly
  • once everything is up to date, the admins are open to update the config to:
    • whitelisting the bridge so it doesn’t trigger flood protection (what cause the kicks right now)
    • exempting the bridge puppets (Matrix users created on IRC by the bridge) from having to be registered on GNOME channels when IRC channels are in +R mode

Really uplifting news :tada:

6 Likes
14

Thank you @thibaultamartin for the excellent write-up and the glimpse into our messenger history.

Very much agree. Thanks much for the excellent and well written write-up @thibaultamartin :slight_smile:

15

It was a pleasure to read and really gave me new interesting insights into the history of Instant messaging.

16

Speaking of it: I wonder how the “Chat Platforms Evaluation” is going :slight_smile:
@nmcgovern

2 Likes
17

Thank you for the detailed explanation of the problem space @thibaultamartin. I know how much work this represents :slight_smile:

I believe what is missing here now for people like me waiting for “dust to settle” before migrating would be a “plan & schedule” for the migration (if applicable) of GNOME’s Matrix instance and accounts/data from Element’s servers to GNOME’s. I would like to know when/where/how I would have to register an account to be all set for the next 100 years, without having to do it more than once…

Also, ignoring the whole IRC question for a minute: what would spammer control mechanisms look like in a hypothetical “pure Matrix” world?

18

Thanks for allowing me to introduce what’s left to do Jeff :grinning_face_with_smiling_eyes:

That’s a very fair question, which requires a solid answer to be acceptable. The initial post of this thread is the first part of a series. I also blogged something about how to explain Matrix with e-mails, to make it easier to understand for people less familiar with the subject. In its conclusion I say

A post on how an organisation can keep its sovereignty in a decentralised and open network as Matrix will follow in the coming weeks.

That’s another very important aspect: Matrix is an open network, but it doesn’t mean we can do anything we want and expect it to work like a charm. The migration from Element’s to GNOME’s infrastructure is one of the steps of a broader plan. We need to think it through first, and to set-up a set of rules to enforce our strategy. My mind is pretty clear on that point, but rather than throwing ideas as I’m sipping my first coffee of the morning, I’d rather wait for a detailed blog post to explain it all in depth.

Then comes the last part, the one you mention: we have a Matrix instance in a given state, we will have a target, we need a transition plan to reach that target. By a happy coincidence, @averi is re-working our identities management system. Most of the steps on the Matrix side are clear to me, but instead of throwing ideas in a mess I would rather wait for our identities management system to be advanced enough to finish up and formalise the transition plan.

Matrix has pretty solid, battle-tested defence mechanisms now. The Matrix.org instance has more than 100k Monthly Active Users (MAU) and of course has triggered spammers’ interest. The most immediate defence mechanism is against individual troublemakers: a moderator in a room can kick/ban a user from a room, and redact their messages so they are removed from the timeline.

More sophisticated attacks can happen, for example when a spammer has automated the spamming in several rooms. That’s where the hammer of Matrix justice comes to play. To keep things short, it creates a room in which a moderation team can send commands to a bot to ask it to kick/ban a user and redact their messages for all the rooms under its protection. Just a note: it doesn’t make sense in the Matrix universe to talk about “rooms belonging to a server”. What makes more sense is to talk about “rooms we control”: that’s why the moderation by mjolnir is opt-in. When people create a room and want to be protected by mjolnir, they need to grant it the administrator privileges to be placed under its protection.

Finally, for the more extreme cases there are two last resort mechanisms: putting an obviously maliciously crafted instance on a denylist, and for the most extreme cases shadow banning them.

That’s where things get really good: with the proper authorizations on the IRC side it is possible to kick and ban users from the Matrix side! For more sophisticated defence such as changing the modes of a channel, manual work has to be done on IRC. We’re still working with GIMPnet administrators to upgrade the network, and we’ll be able to retrieve proper rights to do it.

19

I see you’re not on Planet GNOME, and not DuckDuckGo-able either. Where is your blog? (You probably want to be on Planet GNOME!)

1 Like
20

The article I mention is Matrix for Instant Messaging :slight_smile:

I need to take the time to figure out tags with Hugo and I will open a MR to be added to Planet :grinning_face_with_smiling_eyes:

3 Likes