NOTE
If your issue is for security or deals with sensitive info please mark it as private using the checkbox below.
It appears that nl.mirrors.cicku.me is throttling downloads to an unusable degree. It starts out using my full bandwith (150Mbit/s), but then throttles to 30~40 kBit/s! I suspect that this also effects other mirrors by this individual, as they all appear hosted on cloudflare.
Here's a trace of a random large-ish package I tried to update. notice the rightmost column, current speed, decreasing over just a few seconds. It happens with all packages beyond a certain size, though.
~ % curl -o /dev/null https://nl.mirrors.cicku.me/fedora/linux/updates/39/Everything/x86_64/Packages/a/ansible-9.10.0-1.fc39.noarch.rpm |& stdbuf -oL tr '\r' '\n' | ts Sep 21 22:12:24 % Total % Received % Xferd Average Speed Time Time Time Current Sep 21 22:12:24 Dload Upload Total Spent Left Speed Sep 21 22:12:24 Sep 21 22:12:25 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 Sep 21 22:12:26 24 50.8M 24 12.4M 0 0 14.3M 0 0:00:03 --:--:-- 0:00:03 14.2M Sep 21 22:12:27 59 50.8M 59 30.0M 0 0 15.9M 0 0:00:03 0:00:01 0:00:02 15.9M Sep 21 22:12:28 72 50.8M 72 37.1M 0 0 12.8M 0 0:00:03 0:00:02 0:00:01 12.8M Sep 21 22:12:29 73 50.8M 73 37.1M 0 0 9710k 0 0:00:05 0:00:03 0:00:02 9709k Sep 21 22:12:31 73 50.8M 73 37.1M 0 0 7735k 0 0:00:06 0:00:04 0:00:02 7734k Sep 21 22:12:32 73 50.8M 73 37.1M 0 0 6226k 0 0:00:08 0:00:06 0:00:02 4832k Sep 21 22:12:33 73 50.8M 73 37.2M 0 0 5273k 0 0:00:09 0:00:07 0:00:02 1371k Sep 21 22:12:34 73 50.8M 73 37.2M 0 0 4707k 0 0:00:11 0:00:08 0:00:03 28221 Sep 21 22:12:34 73 50.8M 73 37.2M 0 0 4187k 0 0:00:12 0:00:09 0:00:03 28362 Sep 21 22:12:35 73 50.8M 73 37.3M 0 0 3871k 0 0:00:13 0:00:09 0:00:04 33085 Sep 21 22:12:36 73 50.8M 73 37.3M 0 0 3507k 0 0:00:14 0:00:10 0:00:04 34233 Sep 21 22:12:38 73 50.8M 73 37.3M 0 0 3172k 0 0:00:16 0:00:12 0:00:04 30516 Sep 21 22:12:38 73 50.8M 73 37.4M 0 0 2882k 0 0:00:18 0:00:13 0:00:05 31604 Sep 21 22:12:40 73 50.8M 73 37.4M 0 0 2727k 0 0:00:19 0:00:14 0:00:05 29891 Sep 21 22:12:40 73 50.8M 73 37.4M 0 0 2505k 0 0:00:20 0:00:15 0:00:05 27115 Sep 21 22:12:42 73 50.8M 73 37.4M 0 0 2411k 0 0:00:21 0:00:15 0:00:06 29414 Sep 21 22:12:42 73 50.8M 73 37.5M 0 0 2248k 0 0:00:23 0:00:17 0:00:06 32598 Sep 21 22:12:44 73 50.8M 73 37.5M 0 0 2151k 0 0:00:24 0:00:17 0:00:07 35710 Sep 21 22:12:44 73 50.8M 73 37.6M 0 0 2018k 0 0:00:25 0:00:19 0:00:06 39087 Sep 21 22:12:46 74 50.8M 74 37.6M 0 0 1924k 0 0:00:27 0:00:20 0:00:07 41680 Sep 21 22:12:46 74 50.8M 74 37.6M 0 0 1817k 0 0:00:28 0:00:21 0:00:07 40096 Sep 21 22:12:48 74 50.8M 74 37.7M 0 0 1758k 0 0:00:29 0:00:21 0:00:08 37067 Sep 21 22:12:48 74 50.8M 74 37.7M 0 0 1656k 0 0:00:31 0:00:23 0:00:08 36048 Sep 21 22:12:50 74 50.8M 74 37.7M 0 0 1610k 0 0:00:32 0:00:24 0:00:08 33199 Sep 21 22:12:51 74 50.8M 74 37.8M 0 0 1516k 0 0:00:34 0:00:25 0:00:09 32768 Sep 21 22:12:51 74 50.8M 74 37.8M 0 0 1475k 0 0:00:35 0:00:26 0:00:09 35687 Sep 21 22:12:52 74 50.8M 74 37.8M 0 0 1443k 0 0:00:36 0:00:26 0:00:10 39831 ^C
(EDIT: after downloading this specific file a bunch of times, it seems cloudflare has cached it better, and the throttling is gone. this is PER-FILE, though: trying the same package with aarch64 still exhibits this. so if you can't reproduce it, please pick another random large package)
This mirror was automatically picked by fastestmirror=True (as their ping, coming from cloudflare, is quite good), so this probably affects a number of people.
after adding minrate and timeout to my dnf.conf, it finally gave up on this mirror:
minrate
timeout
[MIRROR] gcc-gfortran-13.3.1-3.fc39.x86_64.rpm: Curl error (28): Timeout was reached for http://nl.mirrors.cicku.me/fedora/linux/updates/39/Everything/x86_64/Packages/g/gcc-gfortran-13.3.1-3.fc39.x86_64.rpm [Operation too slow. Less than 102400 bytes/sec transferred the last 10 seconds] [MIRROR] ansible-9.10.0-1.fc39.noarch.rpm: Curl error (28): Timeout was reached for http://nl.mirrors.cicku.me/fedora/linux/updates/39/Everything/x86_64/Packages/a/ansible-9.10.0-1.fc39.noarch.rpm [Operation too slow. Less than 102400 bytes/sec transferred the last 10 seconds] [MIRROR] gcc-13.3.1-3.fc39.x86_64.rpm: Curl error (28): Timeout was reached for http://nl.mirrors.cicku.me/fedora/linux/updates/39/Everything/x86_64/Packages/g/gcc-13.3.1-3.fc39.x86_64.rpm [Operation too slow. Less than 102400 bytes/sec transferred the last 10 seconds] [MIRROR] gcc-gfortran-13.3.1-3.fc39.x86_64.rpm: Curl error (28): Timeout was reached for https://nl.mirrors.cicku.me/fedora/linux/updates/39/Everything/x86_64/Packages/g/gcc-gfortran-13.3.1-3.fc39.x86_64.rpm [Operation too slow. Less than 102400 bytes/sec transferred the last 10 seconds] [MIRROR] ansible-9.10.0-1.fc39.noarch.rpm: Curl error (28): Timeout was reached for https://nl.mirrors.cicku.me/fedora/linux/updates/39/Everything/x86_64/Packages/a/ansible-9.10.0-1.fc39.noarch.rpm [Operation too slow. Less than 102400 bytes/sec transferred the last 10 seconds] [MIRROR] gcc-13.3.1-3.fc39.x86_64.rpm: Curl error (28): Timeout was reached for https://nl.mirrors.cicku.me/fedora/linux/updates/39/Everything/x86_64/Packages/g/gcc-13.3.1-3.fc39.x86_64.rpm [Operation too slow. Less than 102400 bytes/sec transferred the last 10 seconds]
somewhat soon, but realistically, whenever there's time.
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, ops
Metadata Update from @zlopez: - Issue tagged with: low-gain
nl.mirrors.cicku.me is throttling downloads to an unusable degree
I only throttle malicious bots which pull the site millions of times per day, they pull large files harder than any legitimate users here.
This mirror was automatically picked by fastestmirror=True (as their ping, coming from cloudflare, is quite good), so this probably affects a number of people. after adding minrate and timeout to my dnf.conf, it finally gave up on this mirror:
What happened here could be (I'm only guessing because I do not troubleshoot publicly) files absent in the NL location, when you are the first one to download the speed will be slower since the file is not sourced from a server in NL (this is contrary to your understanding that the slowness affects a lot of people). Once the formerly absent file is fully fetched to the NL location, you will be able to download with less issues (these file stay in the country location for over a month). You also have the freedom to manually use another mirror on your machine, which is also covered by the dnf.conf manpage.
The mirror is not only serving Fedora users and I have not received too many issue reports from my mirror admin address. Some issues are from the ISP, some issues are from the censorship and some are just network jitters. However, I must admit that the NL mirror was enabled for Fedora just from a week ago and you may be encountering issues higher than usual.
I kinda wish dnf would drop the 'fastest mirror' option... ping latency isn't really a very good indicator of download speed. ;(
So, what do we do here?
I have no idea.:smile:Checking my logs and I only received ~100 requests of last 14 days in NL...Whereas other hostnames have received 14M requests and nobody said anything for them.
All I can say is good luck, if nobody uses the NL location, then it won't be fast, so feel free to use other mirrors.
Closing this for now. I'm not sure what we can do in this case aside from drop the mirror, but it's working well for many others...
Metadata Update from @kevin: - Issue close_status updated to: Will Not/Can Not fix - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.