After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 780408 - Increase connection limit from matrix.org to gimpnet
Increase connection limit from matrix.org to gimpnet
Status: RESOLVED FIXED
Product: sysadmin
Classification: Infrastructure
Component: IRC
unspecified
Other Mac OS
: High major
: ---
Assigned To: GNOME Sysadmins
GNOME Sysadmins
Depends on:
Blocks:
 
 
Reported: 2017-03-22 16:21 UTC by Matthew Hodgson
Modified: 2018-09-21 07:58 UTC
See Also:
GNOME target: ---
GNOME version: ---



Description Matthew Hodgson 2017-03-22 16:21:10 UTC
Hi,

As per https://mail.gnome.org/archives/desktop-devel-list/2017-March/msg00033.html we're hosting a matrix<->irc bridge to gimpnet.  However, it looks like we've maxed out the number of allowed connections to the network at 132 simultaneous connections from our IPv6 subnet (2001:470:1af1:104:*).  This is causing the bridge to cycle connections to IRC and generally misbehave.

Please can we urgently have an I:line equivalent from our subnet to increase the connection count?

thanks,

Matthew
Comment 1 Allan Day 2017-03-27 09:41:44 UTC
I heard on IRC that there has been some progress with this issue. Are there any updates?
Comment 2 Andrea Veri 2017-03-27 11:16:31 UTC
Both the servers running with IPv6 had an allow block added for the requested subnet. Matthew, can you please confirm, mark the bug as resolved accordingly?
Comment 3 Matthew Hodgson 2017-04-20 21:52:39 UTC
oops - sorry, I totally missed this. yup, we're all sorted now :)
Comment 4 Benjamin Berg 2018-06-15 13:45:24 UTC
Hey, it looks like we have been hitting this limit again for quite a while. Could someone look into it please?
Comment 5 Andrea Veri 2018-06-18 10:52:15 UTC
Benjamin, we currently have two IPv6 nodes in rotation and one of them was recently removed (the time you reported this matches with the one we pulled the node out from the round robin) from the pool for maintenance. It was re-added today, please let me know if you're still seeing any failure and mark the bug as resolved accordingly.

Thanks!
Comment 6 Benjamin Berg 2018-06-18 12:18:13 UTC
Hey AV,

aha! That does explain why so many people were quitting at the time.

However, we were having issues for much longer already, with some matrix people simply missing on the IRC side. Alexandre said that the matrix people confirmed that we were hitting a limit.

As a data point, right I am able to see 27 matrix users (based on IPv6 prefix) on IRC while Alexandre is saying he has 45 matrix users on his side.
Comment 7 Will Hunt 2018-06-18 13:05:25 UTC
Hi all,

I'm currently maintaining the Matrix IRC bridge project and matrix.org's instance. Sorry for the trouble, at the moment it looks like we are configured to allow no more than 2048 connections from our bridge to the IRC network. The number of connections has stayed constantly at the maximum we have allowed for on our side for weeks, and so we are unable to provision more for our Matrix users.

If Gimpnet is willing we'd like to increase the limit further?

Thanks for your help!

Half-Shot
Comment 8 Alexandre Franke 2018-06-23 18:57:40 UTC
(In reply to Will Hunt from comment #7)
> If Gimpnet is willing we'd like to increase the limit further?

irc.acc.umu.se has a limit set to 2000 connections from the matrix.org IP range and irc.poop.nl has 1000, so you should be fine increasing up to 3000.
Comment 9 Andrea Veri 2018-06-26 11:08:41 UTC
Will,

we're seeing an high amount of Matrix clients connecting to irc.acc.umu.se, is this server being used directly by the bridge instead of irc.gnome.org?
Comment 10 Will Hunt 2018-06-26 13:27:11 UTC
Andrea,

We're currently connecting to irc.gimp.org. Should we switch over to irc.gnome.org?
Comment 11 Will Hunt 2018-06-26 13:33:32 UTC
It's also worth noting that we restarted the IRC bridge at 8:40 AM UTC this morning so a large number of reconnects is expected. We've increased the limit to 3000 and noticed that we have more than 2048 connections which was the previous limit, so it should working as expected. I notice that irc.gnome.org is just pointing to irc.gimp.org so we should be fine.
Comment 12 Andrea Veri 2018-06-26 13:39:58 UTC
Yeah, irc.gnome.org points to irc.gimp.org behind the scenes, wondering what is causing 90% of the clients to end up on irc.acc.umu.se:

* |-irc.poop.nl                (234)
* |-irc.acc.umu.se             (2476)

maswan?
Comment 13 Andrea Veri 2018-06-26 13:41:01 UTC
And stats when the bridge was restarted this morning:

* |-irc.poop.nl                (239)
* |-irc.acc.umu.se             (984)
Comment 14 Will Hunt 2018-06-26 15:01:27 UTC
We're going to try to balance the connections between the two servers more carefully our side. Bridge has been restarted and we are monitoring.
Comment 15 Will Hunt 2018-06-26 15:56:37 UTC
Andrea,

Noticing that irc.acc.umu.se we connecting fine but irc.poop.nl is not allowing us to have more than 32 connections. We are using port 6697 for both hosts.

I found that trying to manually connect my IRC client using the matrix.org host to irc.poop.nl:6667 was fine, but irc.poop.nl:6697 closed the connection (while running the other 32 clients. Is it possible we are limited somehow by port?
Comment 16 Andrea Veri 2018-06-26 16:56:02 UTC
Will, thanks, forwarded your note to irc.poop.nl maintainer. He's looking at it. Will get back to you once he confirms we're fine, at that point it'd be ideal to kick the bridge again for proper balancing of users.
Comment 17 Andrea Veri 2018-06-27 12:23:05 UTC
Will, should be fixed on the irc.poop.nl side, can you please double check and eventually rebalance the bridge?

Thanks!
Comment 18 Will Hunt 2018-06-28 13:51:23 UTC
Andrea, We are going to restart the bridge tomorrow morning at 08:00 and hopefully it should balance out. Thanks for handing the stuff with irc.poop.nl!
Comment 19 Andrea Veri 2018-09-21 07:58:03 UTC
This should be fixed since several weeks already. Thanks!