Managing a Public End to End Room on Matrix: Lessons Learned¶

As you can imagine, the team behind Gadgetbridge is motivated to use and spread the use of Free Software even in areas that are only tangential to the main scope of the project. Just to name a few examples, we are using Liberapay to collect donations (to all our donors: Thank you!), and the project's initiator even created the probably most widely known publicly available free software Git (and more!) hosting in Europe run by a collective.

It is in this spirit that we share our experience in managing and running a large public end-to-end encrypted (e2ee) room on matrix.org. Please be aware that this post is primarily intended for a technical audience and not for Gadgetbridge users.

Our Room¶

A Bit of History¶

The Gadgetbridge chat on matrix.org was created as an e2ee room in November 2016.

By early 2025, there were about 1700 users in the room. Periodically we were experiencing some errors related to e2ee - "Unable to Decrypt" or UTD in matrix jargon - mostly related to users' settings ("only send encrypted messages to verified sessions" being one of the most common issues) but also depending on the protocol itself (see the Resources section for a link to the Matrix.org github issue collecting those).

Digging Deeper (also: Being Kind to Your Hosts)¶

Besides the bugs, we wanted to make sure that our room would not be a burden on the matrix.org infrastructure: an e2ee room is inherently more resource-intensive than a normal room, due to the additional (meta)data and message exchange required to enable encryption, so we reached out to some members of the encryption spec team to understand their point of view on the matter.

We've gotten confirmation that our e2ee room is in line with their whole ethos as an "encrypted communications platform", and that our room is not the cause of any problems on Matrix's part: on the contrary, feedback on how e2ee scales is useful.

Admittedly, being (one of?) the largest e2ee rooms we're somewhat on the bleeding edge and more likely to suffer from UTD messages than most.

Over time Matrix deployed several new "room versions" (cfr. Resources section). Our room has been at version 1 up until May 2025 when it was upgraded to version 11 (current default).

The little pruning (April 2025)¶

With the information collected so far, we knew that the best way to approach the e2ee issues would be to remove as much inactive users from the channel as possible, unfortunately we did not find a way to do so for every user, but we thought of an alternative approach.

The Matrix protocol allows (and encourages) federation of multiple homeservers. It should come as no surprise that in our channel we had users from 150 homeservers (i.e. not matrix.org).

Suspecting that unreachable homeservers were contributing to the problems, we performed a manual check of each participating homeserver in the room using the Matrix Federation Tester. 50 servers were found to be out of the federation, hosting 55 users (~30% of the servers and ~3% of the users). Only two of these servers had more than one user (5 and 2, respectively).

With this list of servers, we checked the last activity of their users to confirm that they were long in the past.

Finally, we removed all these stale users by entering the reason "Your homeserver is not reachable according to the matrix federation tester."

This whole process was a bit cumbersome to do by hand, and we wonder if it could be automated and made part of either /devtools or an external utility, as it would be really helpful for (e2ee) rooms admins.

Commands Used in This Section¶

1. check homeservers: /devtools -> view servers in room
2. check activity of a member by homeserver: /devtools -> Explore room state -> m.room.member -> enter the server url in the search box
3. remove user from room: room info -> people -> enter the server in the search box -> choose user -> Remove from room

The Room Upgrade (May 2025)¶

Pruning helped reduce the errors, so it was clear that dormant accounts were the culprit. Unfortunately, we could not find a working approach to (quickly) identify these accounts, so we decided to do a room upgrade, as it would achieve the same result.

Unfortunately - as with killing a mosquito with a cannon - there are some side effects, and we have done our best to identify and work around them as much as possible. The fact that you are reading this document is our attempt to share our findings with other interested parties so that they can focus on improving the process without reinventing the wheel.

One thing you cannot avoid is that users will lose visibility to some messages from the moment the room is destroyed until the moment they join the new room (combination of points 1 and 4 listed in the Facts section below). What we did was to announce the upgrade a few hours in advance and schedule it for an evening (in our time zones) on a festive day (in our countries), hoping that most people would be able to join the new room quickly. In our case, about 50 people joined the upgraded room before the first message was posted..

Of course, when planning the upgrade, you should coordinate with the admin/mods teams and the people who run any bots you have in your room, as they may need manual intervention to enter the upgraded room.

How To (Effectively) Announce the Upgrade¶

We decided to proceed with a combination of @room mention (to reach all the users that were already in the room when we sent the message) and adding the information in the room description which can be seen by users when they join the room.

This is the message we posted 12 hours in advance (the notice added to the room description was very similar):

This evening, at approximately 20:00 CEST this room will be "upgraded". Practically this means that a new one will be created in which all participants will be automatically invited to enter. Because of the way e2ee works each user will not be able to read the messages posted in the new room until they have joined it. We have decided to give a few hours notice so that everyone can act promptly. Please note: It is NOT necessary to send a message to the new room, you just need to enter by following the link the system will show you!

Room Upgrade Task List¶

When the time comes, you can follow our upgrade script:

1. Make sure that a mod/admin user is connected through a client other than Element Web.
2. Clean up the room description if you have customized it to announce the upgrade.
3. Open developer tools (/devtools) and enable "Developer mode"
4. Use the command /upgraderoom <desired room version>
5. In the NEW room: get the link to the room (open "Room info", click "Copy link")
6. In the OLD room: rename the room to indicate that it is an old version (we added "archived on YYYY-MM-DD" to the name), consider changing the description as well (we set it to a link to the upgraded room)
7. In the OLD room: the mod/admin who is not connected via Element Web should post a link to the new room for clients that do not show the upgrade message with link automatically (you need to be a mod because normal users cannot post to archived rooms, and you need to use a client different than Element Web as it will prevent you from sending messages in archived rooms, regardless of your power level)
8. If you had bots that are not automatically following room upgrades, invite them in the new room. They will retain their power levels.

Questions and Answers about Room Upgrade (as Experienced in May 2025):¶

Facts¶

1. The Room History Visibility setting is ineffective for e2ee rooms because users only have the keys to decrypt messages from the moment they join the room, and all previous messages are undecipherable to them. (This might change in the future for invited users, see Resources section).

2. Later room versions have various improvements or features, but improved encryption is not one of them. Changing the room version won't directly help with UTD or other e2ee quirks.

3. The larger the room (read: the more users in it), the more likely there are to be e2ee quirks.

4. Upgrading the room (see the Resources section) means that the old room is deactivated, a new room is created, and the two are linked so that clients can access the new room from the old one. The new room is initially empty because users must explicitly join it.

5. The room upgrade procedure does not expose an encryption toggle, an encrypted room will be upgraded to an encrypted room.

Resources¶

1. "unable to decrypt" github issue with instructions for users on how to report (and a list of known/solved) e2ee issues: https://github.com/element-hq/element-meta/issues/245
2. e2ee matrix room: https://matrix.to/#/#e2e:matrix.org
3. matrix rooms versions (and more): https://spec.matrix.org/latest/rooms/
4. Sharing room keys when inviting new users issue (proposal): https://github.com/element-hq/element-meta/issues/39
5. Room upgrades spec: https://spec.matrix.org/v1.14/client-server-api/#room-upgrades
6. Matrix federation tester: https://federationtester.matrix.org/

Bonus Chapter: Simulating the Room Upgrade Process¶

If you made it this far, congratulations! In case you prefer to double check our findings (maybe some time has passed from this article to when you are performing an e2ee room upgrade), you can create a "guinea pig" room to test. Here is how we did it.

1. When creating a public room (through element web) its version will be the homeserver's room default version. Activate encryption (Room settings, "security and privacy") and "upgrade" the room to your original room's server version

   1. Open developer tools (/devtools) and enable "Developer mode"
   2. Use the command /roomupgrade <the_room_version_you_are_migrating_from>
   3. The old room should be archived and you should be in the new one. There should be a link on top that brings you to the old room.
   4. Try to replicate the rest of your room status (power levels, including custom ones), ban some users just to check if ban persist, custom permissions, add any bot you are using in the main room, ...

2. Perform a second room upgrade to the current default version.

   1. As previously the old room should be archived and you should be in the new one. There should be a link on top that brings you to the old room.
   2. Check that everything is working as expected/described in this document (e.g. check the power levels, banned user list, bots etc.)
   3. Verify if the limitations we mentioned still apply. E.g. that the new room's history is not visible to users that were in the old room until they actively join the new one.

3. You now should have all the information needed to successfully perform a room upgrade. If you found discrepancies between your experience and this document, please let us know!