As of 2023-11-07 22:55 JST (=UTC+9), trying to use repodata on koji.fedoraproject.org , e.g. trying to download:
https://kojipkgs.fedoraproject.org/repos/f40-build/5576678/x86_64/repodata/2dd73446e7e6e318b22a8a7be0cdb178a95c4b2152137dc574ffb990382623e6-filelists.sqlite.bz2
from Tokyo (Japan) seems way slow, about 15kB/s.
On the other hand, trying to download repodata from mirror on Japan, e.g. http://ftp-srv2.kddilabs.jp/Linux/distributions/fedora/updates/39/Everything/x86_64/repodata/62d6f238e874df42fc150cbe6c4570f8f06456e486de0c73c5e998c6b61b4a18-filelists.sqlite.bz2
gets 5MB/s, so I guess something is wrong on koji.fedoraproject.org. Would you investigate? Thank you.
I'm not sure this is anything on our end... it sounds like something network wise between the datacenter kojipkgs is in and your client. ;(
Can you do a mtr to kojipkgs.fedoraproject.org and see if any hops are particularly slow?
I assume it's normally faster for you? or has it been this way all along?
There is a lot of traffic right now due to the f39 release, so that might be contributing here?
Metadata Update from @phsmoura: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, low-trouble, ops
Well, I don't know how to look at this, but anyway:
$ mtr -r -c 25 kojipkgs.fedoraproject.org Start: 2023-11-08T20:51:25+0900 HOST: localhost.localdomain Loss% Snt Last Avg Best Wrst StDev 1.|-- _gateway 0.0% 25 1.1 1.1 0.5 3.3 0.6 2.|-- mygateway.kaonbroadband.c 0.0% 25 4.9 5.6 3.9 13.4 1.9 3.|-- 123-98-236-1.jcomnet.j-cn 0.0% 25 30.8 23.3 15.2 40.5 7.0 4.|-- 10.202.119.52 0.0% 25 27.5 29.2 14.8 88.3 14.6 5.|-- 10.1.220.81 0.0% 25 19.2 21.8 15.0 30.4 4.8 6.|-- 10.1.11.241 0.0% 25 19.8 22.8 16.3 36.8 5.8 7.|-- gw7-be2.ot-dc.zaq.ad.jp 0.0% 25 16.6 21.0 16.0 30.4 4.1 8.|-- 210.130.154.169 0.0% 25 22.5 22.5 14.0 39.3 5.7 9.|-- tky001bb00.IIJ.Net 0.0% 25 21.2 24.1 13.3 38.8 6.3 10.|-- sea001bb00.IIJ.Net 0.0% 25 138.4 138.6 131.8 155.5 5.7 11.|-- be6098.rcr51.b042840-1.se 0.0% 25 137.8 144.8 136.5 155.6 6.3 12.|-- be3705.ccr22.sea02.atlas. 0.0% 25 142.4 143.1 136.5 157.9 5.7 13.|-- be3716.ccr21.sfo01.atlas. 0.0% 25 147.8 151.2 136.4 257.4 28.4 14.|-- be3109.ccr21.slc01.atlas. 0.0% 25 155.8 157.8 151.8 174.8 5.6 15.|-- be3037.ccr21.den01.atlas. 0.0% 25 172.5 169.4 161.2 202.9 8.3 16.|-- be3035.ccr21.mci01.atlas. 0.0% 25 182.1 179.0 173.1 196.1 5.6 17.|-- be2831.ccr41.ord01.atlas. 0.0% 25 188.6 192.7 184.1 207.4 6.8 18.|-- be2717.ccr21.cle04.atlas. 0.0% 25 196.4 198.0 192.3 205.4 3.6 19.|-- be2891.ccr41.dca01.atlas. 0.0% 25 203.7 217.1 201.7 333.0 30.6 20.|-- be3083.ccr41.iad02.atlas. 0.0% 25 209.5 209.7 202.7 221.8 4.4 21.|-- 38.32.106.90 12.0% 25 209.4 213.7 205.3 244.0 9.0 22.|-- 209.132.185.253 4.0% 25 226.2 217.4 208.2 235.2 6.9 23.|-- proxy-iad01.fedoraproject 20.0% 25 224.7 217.6 208.3 235.3 7.4
I assume it's normally faster for you?
Yes, normally it was enough fast, but for some these days it got very slow.
Is it any better now? Wondering if it's related to f39 release transfers?
Still very slow, ~12kB/s .
FYI downloading from https://download.copr.fedorainfracloud.org/ is as usual (i.e. enough fast for me), at least more than several MB/s .
Yeah, copr is in another datacenter entirely. ;(
So, I am not sure what we can do here... but... we are going to be updating/reboots basically all our servers on wed. If this is somehow a issue on our end, perhaps that will clear things up? Can I ask you to test after wed and see if theres any change?
If not, we will probibly have to try and ask networking providers...
Would it be useful to test if downloads to https://dl01.fedoraproject.org are as slow as downloads from koji? If they aren't then it is something to do with proxy01->kojiX.
If they are as slow then it is most likely network congestion on the Cogent link into the datacenter... it looks like 2 hops up.
Well, it's kojipkgs, not koji itself... but yeah, that would be a good datapoint as well...
Unfortunately, for example downloading https://dl01.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/x86_64/os/repodata/07f5ff534ef450ae24ecdd83bd3e04506f07c5de069d83937b8e667156579686-filelists.sqlite.xz is also slow, about ~20KB/s .
Can you try again today? (both dl01 and kojipkgs?)
Thanks
Tried yesterday and today (in Japan, UTC+9), and now it looks like working as normal as before, thank you. Once closing.
Metadata Update from @mtasaka: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
I'm very glad. I wish I knew exactly what caused it tho. ;(
Log in to comment on this ticket.