It’s a mix between crawlers hitting us hard and this manifesting in high IO wait due to the storage backend being on spinning disks (until we migrate GitLab to the new platform) and GitLab’s webservice starting to misbehave around the same time every day since around a couple of GitLab upgrades ago. I have landed a cronjob that rolls out the webservice at a specific time interval, it’ll be smooth for users as Openshift handles this gracefully. We’ll migrate the service, revisit the rsyslog integration (which I also noticed stops receiving any forwarded logs around the same time) and see if this problem represents itself at that point and properly investigate.
There’s a known network degradation (multiple services flapping every few minutes) happening in RAL3 at this time and since around 23 UTC (24th of November), please keep an eye at https://status.gnome.org, we’ll be updating it as we know more.
I think hifron is lamenting the lack of a universal GNOME-wide authorization/login system, possibly because… well, I’m not sure. either because having to maintain a GitLab account is an added hardship, or because it’s not possible to register/sign-in to GitLab while it’s being DDOSed?
(FWIW, I use my GitHub account to login in to GNOME GitLab, and here, and my GitHub SSH identity is registered for access to the gitlab.g.o git-over-ssh server. If I have a separate GNOME account anymore, I’m not aware of it, tho I assume I did at least for GNOME bugzilla way back when.)
GNOME already has SSO and it had one for a long time now, if you are a GNOME contributor or GNOME Foundation member you are entitled to receive an account, anyone else should use other login mechanisms. Transitioning from other login mechanisms to a GNOME (unified, SSO) account is possible and usually happens when someone becomes a GNOME contributor and requests a new account as explained at Developer Access - GNOME Project Handbook.
Hi,
it seems it’s not it, I still cannot use GNOME GitLab in a sane way.
Either I’m lucky and it allows me to look on bugs/merge requests/…
for I do not know 30 seconds or less, then I just get “Error 503 No
healthy backends” or similar error message and that’s it. The status.gnome.org says “all is fine” with the GitLab, even there is not,
for a long time. The “Past Incidents” section should be removed there,
it does not reflect reality (possibly because it’s a manual thing and
nobody wants to confess problems).
Anyway, there had been mentioned AI scrapers or whatever. How do you
know it’s it? If you can identify it, can you block them? You cannot
block an IP, because it’s from a “random” range or from a place where
regular users may access the site? Can you slow them? Or slow down the
parts they use, thus the GitLab can do at least something, even in a
slow mode, than being able to do nothing. Sometimes the page refresh
takes few seconds to the 503, sometimes it’s instant. Does it mean that
the set timeout for the backend is in milliseconds? Using higher
timeout may help? I’m sure you did think of these basic things, you
know the internals and how the things work much better than me (it’s
easy, I know nothing about these). Does GitLab.com have any advice?
There had been mentioned that more GitLab instances are “affected”,
thus the Gitlba.com might think of something, I guess. You mentioned
migration to AWS. Are you sure it’ll solve it? Are the other GitLab
instances on “a very similar hardware as the GNOME GitLab”, none of
them using anything like the AWS? I probably misunderstood it, it
sounded like HDD versus SSD, but it sounds odd in its ground.
It’s pointless to report every single time somebody has issues with GitLab, at this juncture. The infrastructure team knows it, and they moved the migration to AWS up to the earliest possible time, which is Wednesday, December 4:
Until then, the situation won’t get better (or worse), because the infrastructure team does not have the ability to make changes in the places that are currently failing.
The only thing people can do is sit tight, and wait until next week. What people should actively avoid is constantly refreshing/reloading GitLab in the hope it’ll work, because that compounds the issue.
Sure. I thought GitLab was supposed to be operating with reduced QoS (which is how it’s been for me since the onset of the issue), but this is new information.
All load operations (and comment editing) were working with reduced QoS, but comment submission was failing 100%, which I thought was a different issue.
I have been trying here and there to explore the extensions of Gnome through the website Gnome Extensions but I’ve been having this error 2/3 of times I’m making a request.
The error looks like this :
Error 503 All backends failed or unhealthy
All backends failed or unhealthy
Error 54113
Details: cache-sof1510020-SOF 1733227929 1612651946
Varnish cache server
Seeing the “SOF” code and knowing that I live in Bulgaria, and that the server I’m trying to reach is most likely in Sofia, maybe this outage is only present in Bulgaria ?
I checked the Status and everything should be operational. When trying to go on Gnome’s GitLab I saw the same error so I am guessing the error is for the whole gnome.org domain and not only the extensions subdomain.
If anyone could tell me if they have the same error, and if so, if there is a way to use another mirror for the website ? I’m okay with long loading times as long as it loads eventually
Thanks, never knew that.
