This task failed, because we were really slow when downloading a file from Koji: https://taskotron.fedoraproject.org/taskmaster/builders/x86_64/builds/41487 (and the same error happened on stg: https://taskotron.stg.fedoraproject.org/taskmaster/builders/x86_64/builds/87848)
We don't display progress bars in stdout, because it usually makes a lot of mess in text logs. But the downside is that we can't be extremely slow in downloading files, otherwise buildbot will kill the job because of inactivity (20 minutes).
In this case, the file has 230MB and it took us more than 20 minutes to download it. @mkrizek reproduced it on one of our dev buildslaves with wget, it really downloads the file with ~100-200kB/s. Which is really slow, considering all those machines should be in the same datacenter, having very thick network lines between them.
The file is here: http://koji.fedoraproject.org/koji/buildinfo?buildID=615368 https://kojipkgs.fedoraproject.org//packages/gammaray/2.2.1/4.fc23/src/gammaray-2.2.1-4.fc23.src.rpm
We can try to do some magic with progress bars in logs or buildbot timeouts, but my major question is - why on earth is it so slow? I guess we should ask Fedora Infrastructure to look into this. Or am I something missing? Maybe Koji packages server is not in the same datacenter as our buildslaves?
I wonder if we have /etc/hosts set up correctly. I know that kojipkgs has had some issues in the last couple of days but other than the fact that it went down for a bit on Tuesday, I don't know the details.
It'd be worth talking with #fedora-admin about whether or not we should be pointing at a different host for kojipkgs
apparently, this is a known intermittent issue that infra has been working on. we'll know more when they know more
From what I saw in discussion with kevin, their squid proxy slows down the download speeds immensely. After restarting it and some other adjustments, we saw an increase in download speed up to 25MB/s.
So, I think we don't need to make any adjustments in libtaskotron/buildbot right now. We just need to make sure that we notify fedora infra if we see this again, and make sure they somehow try to fix this if it occurs frequently.
I'm closing this ticket as resolved. Reopen if needed.
Metadata Update from @kparal: - Issue tagged with: infrastructure
Log in to comment on this ticket.