This is a reference to #4882 which is closed. Please, re-open.
https://pagure.io/fedora-infrastructure/issue/4882#comment-761196
Lets just track it here rather than opening an ancient ticket.
This looks like a routing issue between you and proxy06 or something on proxy06 itself. Needs investigating on our side.
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, medium-trouble, ops
Can you see if the issue is still happening now?
Date: Wed, 17 Nov 2021 14:03:39 +0000 (UTC) /etc/cron.hourly/0yum-hourly.cron: Could not get metalink https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64&infra=stock&content=centos error was 12: Timeout on https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64&infra=stock&content=centos: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Can you attach output of 'traceroute proxy06.fedoraproject.org' ?
Are you using ipv6 there or ipv4?
ipv4 only.
here are traceroutes from couple servers from different locations:
$ traceroute proxy06.fedoraproject.org traceroute to proxy06.fedoraproject.org (140.211.169.196), 30 hops max, 60 byte packets 1 unpublished 2 10.37.0.213 (10.37.0.213) 1.219 ms 1.177 ms 1.197 ms 3 10.37.131.209 (10.37.131.209) 1.034 ms 1.087 ms 1.038 ms 4 10.37.127.57 (10.37.127.57) 231.503 ms 231.477 ms 231.458 ms 5 88.201.160.38 (88.201.160.38) 0.883 ms 0.848 ms 0.836 ms 6 95.167.76.225 (95.167.76.225) 1.513 ms 1.389 ms 1.705 ms 7 87.226.133.56 (87.226.133.56) 16.290 ms 54.543 ms 87.226.183.61 (87.226.183.61) 21.300 ms 8 * ae1-500.cr1-stk3.ip4.gtt.net (77.67.90.96) 14.894 ms * 9 ae1-xcr2.six.cw.net (195.2.16.213) 15.793 ms * * 10 CENIC.ear1.SanJose1.Level3.net (4.15.122.46) 174.398 ms * * 11 be3376.ccr21.sto03.atlas.cogentco.com (130.117.50.225) 16.346 ms * lon--cenic-100ge.cenic.net (137.164.3.103) 186.228 ms 12 be3186.ccr41.fra03.atlas.cogentco.com (130.117.0.1) 36.737 ms be3187.ccr42.fra03.atlas.cogentco.com (130.117.1.118) 35.960 ms be2814.ccr42.ams03.atlas.cogentco.com (130.117.0.141) 142.010 ms 13 corv-p2-gw.nero.net (207.98.64.209) 191.265 ms be2558.rcr21.cph01.atlas.cogentco.com (154.25.10.245) 22.289 ms be2813.ccr41.ams03.atlas.cogentco.com (130.117.0.121) 38.970 ms 14 corv-car1-gw.nero.net (207.98.64.17) 190.455 ms 186.700 ms 190.231 ms 15 be2816.ccr42.ams03.atlas.cogentco.com (154.54.38.209) 34.037 ms proxy06.fedoraproject.org (140.211.169.196) 185.988 ms !X be2490.ccr42.jfk02.atlas.cogentco.com (154.54.42.85) 132.793 ms # traceroute proxy06.fedoraproject.org traceroute to proxy06.fedoraproject.org (140.211.169.196), 30 hops max, 60 byte packets 1 gateway (10.255.2.2) 0.120 ms 0.084 ms 0.084 ms 2 84.17.4.165 (84.17.4.165) 0.680 ms 0.748 ms 0.604 ms 3 176.241.97.41 (176.241.97.41) 0.760 ms 0.732 ms 1.072 ms 4 176.241.99.237 (176.241.99.237) 0.863 ms 0.812 ms 1.259 ms 5 oct-cr01-be2.78.spb.mts-internet.net (212.188.1.101) 1.209 ms 1.290 ms 1.438 ms 6 mmon-cr01-be4.78.spb.mts-internet.net (195.34.53.13) 1.441 ms 1.528 ms 1.737 ms 7 radio-cr01-ae3.78.hel.mts-internet.net (212.188.29.109) 7.818 ms 6.975 ms 6.943 ms 8 ae55.edge4.Stockholm2.Level3.net (213.242.110.181) 35.404 ms ae51.bar2.Helsinki1.Level3.net (62.67.19.237) 12.847 ms ae55.edge4.Stockholm2.Level3.net (213.242.110.181) 35.377 ms 9 * * * 10 CENIC.ear1.SanJose1.Level3.net (4.15.122.46) 181.458 ms 181.409 ms 181.326 ms 11 lon--cenic-100ge.cenic.net (137.164.3.103) 189.362 ms 189.601 ms 189.709 ms 12 eugn-p1-gw.nero.net (207.98.64.196) 191.534 ms 191.504 ms 191.617 ms 13 corv-p2-gw.nero.net (207.98.64.209) 192.730 ms 192.045 ms 191.348 ms 14 corv-car1-gw.nero.net (207.98.64.17) 190.073 ms 190.160 ms 190.521 ms 15 proxy06.fedoraproject.org (140.211.169.196) 190.795 ms !X 190.517 ms !X 190.661 ms !X
One more trace around the time when "Operation too slow" happened again:
Date: Wed, 1 Dec 2021 19:08:22 +0300 (MSK) /etc/cron.hourly/0yum-hourly.cron: Could not get metalink https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64&infra=stock&content=centos error was 12: Timeout on https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64&infra=stock&content=centos: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
# traceroute proxy06.fedoraproject.org traceroute to proxy06.fedoraproject.org (140.211.169.196), 30 hops max, 60 byte packets 1 cut 2 10.37.0.213 (10.37.0.213) 1.083 ms 1.090 ms 1.053 ms 3 10.37.131.209 (10.37.131.209) 0.992 ms 0.996 ms 0.993 ms 4 10.37.127.57 (10.37.127.57) 0.961 ms 0.953 ms 0.930 ms 5 88.201.160.38 (88.201.160.38) 0.906 ms 0.913 ms 0.872 ms 6 95.167.76.225 (95.167.76.225) 1.439 ms 1.343 ms 1.720 ms 7 87.226.133.56 (87.226.133.56) 13.573 ms 13.579 ms 87.226.183.61 (87.226.183.61) 14.413 ms 8 * ae1-500.cr1-stk3.ip4.gtt.net (77.67.90.96) 12.577 ms * 9 * * ae1-xcr2.six.cw.net (195.2.16.213) 16.547 ms 10 kbn-bb1-link.ip.twelve99.net (62.115.134.78) 846.587 ms CENIC.ear1.SanJose1.Level3.net (4.15.122.46) 182.971 ms xe-0-0-0-16-vpr1.sye.cwasia.net (195.2.18.86) 60.086 ms 11 be3377.ccr22.sto03.atlas.cogentco.com (154.54.36.89) 12.694 ms * lon--cenic-100ge.cenic.net (137.164.3.103) 186.076 ms 12 eugn-p1-gw.nero.net (207.98.64.196) 190.916 ms be3100.nr51.b015758-1.cph01.atlas.cogentco.com (130.117.15.117) 22.383 ms 22.547 ms 13 be12194.ccr41.lon13.atlas.cogentco.com (154.54.56.93) 134.428 ms be2496.ccr41.ham01.atlas.cogentco.com (154.54.61.221) 27.814 ms corv-p2-gw.nero.net (207.98.64.209) 188.278 ms 14 corv-car1-gw.nero.net (207.98.64.17) 186.591 ms be2317.ccr41.jfk02.atlas.cogentco.com (154.54.30.185) 131.637 ms be2816.ccr42.ams03.atlas.cogentco.com (154.54.38.209) 37.182 ms 15 be2889.ccr21.cle04.atlas.cogentco.com (154.54.47.49) 133.301 ms proxy06.fedoraproject.org (140.211.169.196) 190.264 ms !X 187.357 ms !X
proxy06 is somewhere around 9000 km from the first hop of the system. I can't tell if this is East or West Russia so assuming Moscow. I don't know why they are getting that as a DNS entry they are getting but they would probably have better support using a server in Europe.
After analyzing with tcpdump, it appears the slow speed happens not just with proxy06. I also saw proxy09's ip - 140.211.169.206. Is there anything I (or better you ;) can do to get that fixed? I'm getting a lot of such emails every single day. That's really annoying :((
Both proxy06 and proxy09 are in the same data location which depending on what part of Russia you are in are about as far away as you can get. That means you are travelling through a lot of congested data-hops and while icmp gets through (traceroute), UDP/TCP gets throttled at various places. The timeouts and slow data are caused by that and not the servers themselves.
The servers work fine for continental North America customers, so the part which can be fixed is why are eastern european or western asian customers getting sent to it. That is DNS related so we need to look at what resolvectl query mirrors.fedoraproject.org returns. I get the following from my NA system so the 140.211 ips are ok, but I would like to see what you are getting to see if DNS is thinking your ip address is in North America
resolvectl query mirrors.fedoraproject.org
$ resolvectl query mirrors.fedoraproject.org mirrors.fedoraproject.org: 2620:52:3:1:dead:beef:cafe:fed6 -- link: enp9s0u2u1u2 2600:2701:4000:5211:dead:beef:fe:fed3 -- link: enp9s0u2u1u2 2604:1580:fe00:0:dead:beef:cafe:fed1 -- link: enp9s0u2u1u2 2605:bc80:3010:600:dead:beef:cafe:feda -- link: enp9s0u2u1u2 2605:bc80:3010:600:dead:beef:cafe:fed9 -- link: enp9s0u2u1u2 38.145.60.21 -- link: enp9s0u2u1u2 8.43.85.73 -- link: enp9s0u2u1u2 152.19.134.198 -- link: enp9s0u2u1u2 152.19.134.142 -- link: enp9s0u2u1u2 67.219.144.68 -- link: enp9s0u2u1u2 140.211.169.196 -- link: enp9s0u2u1u2 140.211.169.206 -- link: enp9s0u2u1u2 38.145.60.20 -- link: enp9s0u2u1u2 209.132.190.2 -- link: enp9s0u2u1u2 (wildcard.fedoraproject.org)
here is what i get:
$ host mirrors.fedoraproject.org mirrors.fedoraproject.org is an alias for wildcard.fedoraproject.org. wildcard.fedoraproject.org has address 67.219.144.68 wildcard.fedoraproject.org has address 38.145.60.20 wildcard.fedoraproject.org has address 18.133.140.134 wildcard.fedoraproject.org has address 152.19.134.198 wildcard.fedoraproject.org has address 185.141.165.254 wildcard.fedoraproject.org has address 209.132.190.2 wildcard.fedoraproject.org has address 152.19.134.142 wildcard.fedoraproject.org has address 18.159.254.57 wildcard.fedoraproject.org has address 38.145.60.21 wildcard.fedoraproject.org has address 140.211.169.206 wildcard.fedoraproject.org has address 18.185.136.17 wildcard.fedoraproject.org has address 85.236.55.6 wildcard.fedoraproject.org has IPv6 address 2600:2701:4000:5211:dead:beef:fe:fed3 wildcard.fedoraproject.org has IPv6 address 2604:1580:fe00:0:dead:beef:cafe:fed1 wildcard.fedoraproject.org has IPv6 address 2a05:d01c:c6a:cc01:269:da52:9ae1:43e6 wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:fed9 wildcard.fedoraproject.org has IPv6 address 2001:4178:2:1269::fed2 wildcard.fedoraproject.org has IPv6 address 2a05:d014:10:7803:f774:4d7c:e277:a457
OK thanks. that found the problem. The EU DNS is getting a host that is too far away.
'140.211.169.196':{ |config_opts['def_proxy_ttl'] = '60' 'regions': ['DEFAULT', 'NA'], | 'names': ['proxy06', |# this is where you add/remove proxies or add them to a particular geoip region 'proxy06.fedoraproject.org']}, |# IN A 38.145.60.20 ; proxy-iad01 / iad2 (US) '140.211.169.206':{ |# IN A 38.145.60.21 ; proxy-iad02 / iad2 (US) 'regions': ['DEFAULT', 'NA', 'EU'], |# IN A 85.236.55.6 ; proxy02 / internetx (DE) 'names': ['proxy09', |# IN A 8.43.85.73 ; proxy03 / rdu-cc (US) 'proxy09.fedoraproject.org']}, |# IN A 152.19.134.142 ; proxy04 / ibiblio (US)
I will put the two servers to be NA/SA only.
Timeouts should go away within 24-48 hours as DNS caches timeout.
Awesome!! Thank you so much!
So, how have things been now?
No more issues, guys. Thank you again!
Metadata Update from @jazzl0ver: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
For some reason, the issue began to happen again on one of our servers (though, it's not in our datacenter, but still in our city - St.Petersburg, Russia):
# host mirrors.fedoraproject.org mirrors.fedoraproject.org is an alias for wildcard.fedoraproject.org. wildcard.fedoraproject.org has address 38.145.60.21 wildcard.fedoraproject.org has address 8.43.85.73 wildcard.fedoraproject.org has address 209.132.190.2 wildcard.fedoraproject.org has address 140.211.169.206 wildcard.fedoraproject.org has address 140.211.169.196 wildcard.fedoraproject.org has address 152.19.134.142 wildcard.fedoraproject.org has address 67.219.144.68 wildcard.fedoraproject.org has address 38.145.60.20 wildcard.fedoraproject.org has address 152.19.134.198 (removed ipv6 entries, since ipv6 is disabled on the server) # mtr 140.211.169.206 Host Loss% Snt Last Avg Best Wrst StDev 1. 10.255.2.2 0.0% 7 0.2 0.2 0.2 0.2 0.0 2. cut 0.0% 7 0.7 0.7 0.6 0.8 0.0 3. 176.241.97.41 0.0% 7 0.9 0.9 0.8 1.0 0.0 4. 176.241.99.237 0.0% 7 1.2 1.3 1.2 1.5 0.0 5. oct-cr01-be2.78.spb.mts-internet.net 0.0% 7 1.2 1.2 1.1 1.4 0.0 6. mmon-cr01-be4.78.spb.mts-internet.net 0.0% 7 1.8 1.7 1.6 1.9 0.0 7. radio-cr01-ae3.78.hel.mts-internet.net 0.0% 7 6.9 9.1 6.9 16.7 3.7 8. ae51.bar2.Helsinki1.Level3.net 0.0% 7 12.9 14.0 12.9 19.5 2.3 9. ae-1-3504.ear1.SanJose1.Level3.net 83.3% 7 178.1 178.1 178.1 178.1 0.0 10. CENIC.ear1.SanJose1.Level3.net 0.0% 7 178.1 181.4 178.0 197.6 7.9 11. lon--cenic-100ge.cenic.net 0.0% 6 189.5 199.5 189.3 250.0 24.7 12. eugn-p1-gw.nero.net 0.0% 6 191.6 191.6 191.2 191.9 0.0 13. corv-p2-gw.nero.net 0.0% 6 192.3 192.2 191.7 192.7 0.0 14. corv-car1-gw.nero.net 0.0% 6 190.3 190.3 190.3 190.4 0.0 15. proxy09.fedoraproject.org 0.0% 6 190.5 190.5 190.4 190.8 0.0
Please, help
Metadata Update from @jazzl0ver: - Issue status updated to: Open (was: Closed)
OK I checked the files and DNS servers we control. The only geoip located zones which get that data are ones which are in North America and South America. One of your DNS intermediaries (or the outbound ip in the snip ) is using an IP address/ARIN network which is supposed to be located in North America. The ips in the traceroute all get correct country codes from geoiplookup (RU , various EU countries, etc) so I am not sure what is causing this. Could you do the following:
for i in ns02.fedoraproject.org ns05.fedoraproject.org ns-iad01.fedoraproject.org ns-iad02.fedoraproject.org; do host mirrors.fedoraproject.org $i done
and see if one or more of them give the 'wrong' data to you. That will help us figure out if it is a DNS server we control which is stuck OR if the ip address you have is listed as NA for some reason OR if an intermediary DNS server you use is in the NA region.
https://paste.ee/p/5ZBcp
Is this problem still happening?
Yes, from time to time. Last occurrence was on Jan 11th, before that - Jan 5th and 6th.
So, how have things been lately?
Hi Kevin. No issues so far. Thank you very much!
Great. Please let us know if you see it again!
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.