I noticed two small problems that look correlated to me, but maybe they are not:
At a certain time of day saving edits in the fedoraproject wiki often (always?) fail with a "504 Gateway Timeout – The gateway did not receive a timely response from the upstream server or application." (quote from Firefox). I ran into this occasionally for quite a while already, always during my early mornings. Not sure when, seems to be around ~5:00 UTC? A little later things start to work as expected again.
I often kick coprs builds by tagging something in gitlab, which then sends a webhook to copr to kick of a build. That used to worked reliably, but since about a week gitlab sometimes fails to send the webhook – also during my early mornings . Error message in the gitlab UI: "Internal error occurred while delivering this webhook. – Error: Net::ReadTimeout"
Today I checked if both happened at the same time. And indeed that was the case. That was at about 5:15 UTC (the webhook had worked fine about an hour earlier)
This is not urgent, but somewhat annoying -- especially as I lost wiki changes due to this.
Metadata Update from @zlopez: - Issue tagged with: Needs investigation
Metadata Update from @phsmoura: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops
I am unsure what could be causing the copr end of things, but the wiki is very likely the daily db upgrade. ;(
That starts at 05:00UTC...
I am not sure why it's hitting harder than before. Perhaps we can do something to make it not starve things. Will need investigation on mariadb backups.
Thx for looking into this.
I am unsure what could be causing the copr end of things
As I said, it might be unrelated. I'll keep an eye out on things and update the status here, maybe it was a unrelated thing that with a bit of luck vanishes by itself again.
but the wiki is very likely the daily db upgrade. ;( That starts at 05:00UTC...
Ahh! FWIW, not sure if it really "hitting harder than before", as I noticed this before, just never complained.
@thl how have things been lately? Still an issue?
Sorry, had wanted to provide an update myself, but somehow forgot about it.
The copr problem is gone, guess it was just a fluke happening around the same time for a few days.
The wiki problem: haven't encountered it. Not sure if I was just lucky or not. Whatever. Let's close this then.
Thx for your help and sorry for the noise WRT to the copr problem
Metadata Update from @thl: - Issue close_status updated to: Insufficient data - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.