I switched to a new ISP about 5 months ago and have been noticing slow/throttled downloads from Fedora mirrors. Speed is always extremely fast when pointing to local mirrors (e.g. mirror.ox.ac.uk, UK mirror service, anorien.warwick.ac.uk etc). It finally got annoying enough for me to check the actual repository source and it appears I am being redirected to APAC mirrors.
metalink repo mirror list...
$ curl -s 'https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64' <?xml version="1.0" encoding="utf-8"?> <metalink version="3.0" xmlns="http://www.metalinker.org/" type="dynamic" pubdate="Mon, 28 Mar 2022 13:28:27 GMT" generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager"> <files> <file name="repomd.xml"> <mm0:timestamp>1635226287</mm0:timestamp> <size>6285</size> <verification> <hash type="md5">8dcd06f0e4ca8f94b603d52022eee65f</hash> <hash type="sha1">3ca888dfb52dfb111ae8104e44d1f7bb0afe2c63</hash> <hash type="sha256">2b8e8f4786f4c8a6746f532735dc96ea8ef8c36cb4cde77f2aedb2af6905b136</hash> <hash type="sha512">bd3fcad0b5bd3e693b0eba28305e0652c687d76a2a3ca29501400d69b9b9888c46087e405a6f1d7f08d2beaf7865aee41eac18869bc1c905d12505a4800ec8d1</hash> </verification> <resources maxconnections="1"> <url protocol="http" type="http" location="IN" preference="100">http://fedora.mirror.net.in/fedora/linux/releases/35/Everything/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="JP" preference="99">http://ftp.jaist.ac.jp/pub/Linux/Fedora/releases/35/Everything/x86_64/os/repodata/repomd.xml</url> <url protocol="rsync" type="rsync" location="JP" preference="99">rsync://ftp.jaist.ac.jp/pub/Linux/Fedora/releases/35/Everything/x86_64/os/repodata/repomd.xml</url> <url protocol="https" type="https" location="CN" preference="98">https://mirrors.tuna.tsinghua.edu.cn/fedora/releases/35/Everything/x86_64/os/repodata/repomd.xml</url> <url protocol="http" type="http" location="CN" preference="98">http://mirrors.tuna.tsinghua.edu.cn/fedora/releases/35/Everything/x86_64/os/repodata/repomd.xml</url> <url protocol="rsync" type="rsync" location="CN" preference="98">rsync://mirrors.tuna.tsinghua.edu.cn/fedora/releases/35/Everything/x86_64/os/repodata/repomd.xml</url> <url protocol="rsync" type="rsync" location="ID" preference="97">rsync://fedora.mirror.angkasa.id/fedora-enchilada/linux/releases/35/Everything/x86_64/os/repodata/repomd.xml</url> ...
geolocation data returning correct in maxmind, ip2location
[bgp.tools] AS | IP | BGP Prefix | CC | Registry | Allocated | AS Name 208189 | 31.22.13.104 | 31.22.12.0/22 | GB | RIPE | 2019-02-25 | Swish Fibre Ltd
Prefix has been announced by AS208189 since 2020... https://stat.ripe.net/widget/routing-history#w.resource=31.22.12.0%2F22&w.starttime=2018-01-08T00%3A00%3A00&w.endtime=2022-03-28T00%3A00%3A00
Simulated the data that appears to be used by mirrormanager2 which contains the correct prefix-AS mapping... https://github.com/fedora-infra/mirrormanager2/blob/master/utility/mm2_get_global_netblocks
$ wget http://ftp.routeviews.org/dnszones/rib.bz2 $ bzcat rib.bz2 | perl zebra-dump-parser/zebra-dump-parser.pl | uniq >> list.out $ grep '^31\.22\.12\.' list.out 31.22.12.0/22 208189
Please could you confirm the source of geolocation data used for mirror detection and confirm that mirrormanager2 is using up-to-date information? Happy to submit a data correction request to the relevant data source if required.
Metadata Update from @mohanboddu: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, low-trouble, ops
I have the same issue of wrong IP location. I am in France. All IP location databases report my IP 212.114.28.25 as FR (see https://www.iplocation.net/ip-lookup).
But my IP is detected by fedora mirror manager in Russia and I am redirected to the server http://mirror.linux-ia64.org/.
Hmmm the GeoIP tool is giving the following:
$ geoiplookup 212.114.28.25 GeoIP Country Edition: RU, Russian Federation
Interesting. My IP isn't found from the geoiplookup tool...
$ geoiplookup 31.22.13.104 GeoIP Country Edition: IP Address not found $ geoiplookup -v 31.22.13.104 GeoIP Country Edition: GEO-106FREE 20180327 Build 1 Copyright (c) 2018 MaxMind Inc All Rights Reserved
Does that mean it's using a 4 year old database?
This tool is definitively giving wrong answers :(. Its database is totally outdated even in Fedora 35. It depends of GeoIP-GeoLite-data-2018.06-8.fc35.noarch package. It explains why my IP location (modified in 2020 or 2021) and bcfse1 one (2019) were wrong.
See the location of the up-to-date RIPE record: https://apps.db.ripe.net/db-web-ui/query?bflag=false&dflag=false&rflag=true&searchtext=212.114.28.25&source=RIPE
It could be that we are using an out of date database. Not sure it was ever updated since we switched to geoip2.
It seems an account is required to get updated databases. Not sure if there if GeoLite2 is free of charge or not.
Does Fedora already have an account to use GeoLite2 databases? For MirrorManager the country edition would be good enough. There is also the anaconda geolocation service which uses a GeoLite2 database.
GeoLite2 free databases were removed from usage due to GDPR and some lawsuits. I don't think we currently have an account with GeoLite2.
OK I can download the 'Lite' versions with my old account, but I do not have time to automate this. This is going to need to be copied to a lot of different places to make this work.
Metadata Update from @smooge: - Issue untagged with: low-gain, low-trouble - Issue tagged with: high-trouble, medium-gain
I am changing the priority of this for the following reasons: 1. Multiple applications are broken: * GNOME first boot placement for users * Mirrorlist * Fedora DNS zones 2. It is not going to be an easy fix. * A role needs to be written to autodownload the updates (either paid or unpaid) * The role needs to copy this to a place on various servers * The above parts all look in different places so may need code changes.
OK I have slapped a fix on for the GNOME and Mirrorlist tooling which should help clean up these errors for people. This was done by manually downloading new databases and putting them on sundries (for gnome geoip) and mm-backend01 (for mirrorlist).
We also need it on all proxies.
ah missed that. It is currently provided by the package geolite2-city-20191217-5.fc35.noarch and geolite2-city-20191217-5.fc35.noarch . Either we need to make an rpm which overwrites this or a different way to distribute the files.
Turning it over to someone else to do that patching.
The mirrorlist-server process has a parameter to use the GeoIP2 database from another location. So if you copy the file to another directory I can provide the change to adapt the mirrorlist-server service file.
Ok so there seems to be some sort of problem with the proxies. Some proxies have geolite2-country-20191217-5.fc35.noarch and geolite2-city-20191217-5.fc35.noarch installed but only due to older playbook issues. I have made https://pagure.io/fedora-infra/ansible/pull-request/1068 to try and 'fix' this issue for the newer proxies and such.
OK I have pushed this fix and run it on the proxies. I hope this will clear up bad location data in the next 2 to 24 hours.
Amazing, thank you all. I can confirm mirrormanager is now returning mirrors in WEUR... much better than downloading from CN!
Metadata Update from @smooge: - Issue assigned to smooge
So, we have the temp fix for now here... do we want to keep this open to track regularly updating this/putting place automation to do so?
Or open a new one for that?
If it is fixed for now, I would say close it. A larger ticket needs to be made where a login/password and account set up are needed for getting this done regularly. I think that should be a different ticket.
Metadata Update from @smooge: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Noticed this weekend that this has regressed, or load balancers in a pool have not been updated or a stale cache...
$ curl -s 'https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64' | grep -o 'location="[A-Z]\{1,2\}"' | sort | uniq location="CN" location="ID" location="JP" location="SG" location="TH" $ curl -s 'https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64' | grep -o 'location="[A-Z]\{1,2\}"' | sort | uniq location="CN" location="ID" location="JP" location="SG" location="TH" $ curl -s 'https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64' | grep -o 'location="[A-Z]\{1,2\}"' | sort | uniq location="AT" location="BE" location="BG" location="BY" location="CH" location="CY" location="CZ" location="DE" location="DK" location="FI" location="FR" location="GB" location="GR" location="HU" location="IS" location="LT" location="LU" location="MD" location="NL" location="PT" location="RS" location="RU" location="SE" location="SK" location="TR" location="UA" $ curl -s 'https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64' | grep -o 'location="[A-Z]\{1,2\}"' | sort | uniq location="AT" location="BE" location="BG" location="BY" location="CH" location="CY" location="CZ" location="DE" location="DK" location="FI" location="FR" location="GB" location="GR" location="HU" location="IS" location="LT" location="LU" location="MD" location="NL" location="PT" location="RS" location="RU" location="SE" location="SK" location="TR" location="UA" $ curl -s 'https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64' | grep -o 'location="[A-Z]\{1,2\}"' | sort | uniq location="CN" location="ID" location="JP" location="SG" location="TH"
The AS and geo-mapping of the IP in question hasn't changed in the last 3 months...
AS | IP | BGP Prefix | CC | Registry | Allocated | AS Name 208189 | 31.22.13.104 | 31.22.12.0/22 | GB | RIPE | 2019-02-25 | Swish Fibre Ltd
Metadata Update from @bcfse1: - Issue status updated to: Open (was: Closed)
Metadata Update from @smooge: - Assignee reset
I missed that this was re-opened.
Are you still seeing the issue? We have not changed these files in a while...
Please file a new issue if you are still seeing this?
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.