So I'll prefix this with our understanding of how MirrorManager is supposed to work:
The mirror admin notes a mirror, url, ways to access it, various other information. I've double checked and we don't have any way of hinting, in MirrorManager, of what architectures we support, etc. We are then expecting MirrorManager to do some basic verification of what's present, sync state, etc for the mirror before adding it to the appropriate mirrorlist for clients to query / snag content from. This has been my own longstanding understanding from when MirrorManager was first written, and was what we verified when we did the Micro Mirrors for FCIX, so if my assumptions above are incorrect this may be an us (FCIX) problem vs. a bug in MirrorManager.
We got a report from an aarch64 user that they were hitting the FCIX mirrors for content, since we don't have it they were seeing a lot of spurious 404s which is what I'd expect. Had the user get us their repo file to confirm it wasn't something weird, and it looked right, and was querying https://mirrors.fedoraproject.org/mirrorlist?repo=epel-9&arch=aarch64 appropriately (I've filled in the substitutions here). However you'll note that mirrorlist is returning a pile of FCIX machines, despite the fact we only carry x86_64 on the micros (most of the machines), but we do have it on mirror.fcix.net for example. We've been backfilling where we can to prevent the 404s but I'm unsure if the checker is stuck, or if it's only checking one site, or if it's getting confused based on sites having different content?
Anyway, need some guidance and/or help to figure out what's up and where/what needs fixing.
I cannot say much about the errors, but everything works as expected.
In you description to debug this you are using a wrong approach. Or you are using something from MirrorManager that does not work correctly. But which should not be relevant.
You are using the /mirrorlist interface and that is not working as one might expect. You need to look at /metalink as that is what most clients are using:
/mirrorlist
/metalink
$ curl -s "https://mirrors.fedoraproject.org/metalink?repo=epel-9&arch=aarch64&country=us" | grep fcix $ curl -s "https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=aarch64&country=us" | grep fcix <url protocol="https" type="https" location="US" preference="89">https://ziply.mm.fcix.net/epel/7/aarch64/repodata/repomd.xml</url> <url protocol="http" type="http" location="US" preference="89">http://ziply.mm.fcix.net/epel/7/aarch64/repodata/repomd.xml</url> <url protocol="rsync" type="rsync" location="US" preference="89">rsync://ziply.mm.fcix.net/fedora-epel/7/aarch64/repodata/repomd.xml</url> <url protocol="https" type="https" location="US" preference="87">https://mirror.fcix.net/epel/7/aarch64/repodata/repomd.xml</url> <url protocol="http" type="http" location="US" preference="87">http://mirror.fcix.net/epel/7/aarch64/repodata/repomd.xml</url> <url protocol="rsync" type="rsync" location="US" preference="87">rsync://mirror.fcix.net/fedora-buffet/epel/7/aarch64/repodata/repomd.xml</url>
None of your mirrors are carrying aarch64 for epel-9, but two seem to have content for epel-7. Looking at those mirrors I can see that the files are there.
The error/bug you are hitting with MirrorManager is about empty directories. MirrorManager cannot update the state of empty directories and if an empty directory was once on the mirror it will never disappear from the database as up to date. Which is only relevant for /mirrorlist. /metalink looks for up to date versions of repomd.xml which it cannot find and so the mirror will not be listed.
repomd.xml
So, with /metalink, what almost everything uses today, there should be no real problem.
I cannot say much about the actual errors people were reporting to you.
I hope that clears it up.
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: mirrorlists
Ok so this may be more of an issue that the user (and possibly the aarch64 epel package?) has mirrorlist instead of metalink. I'll kick that back to the user, since this seems like it's "working as expected given what's there, but not as what they wanted"
None of your mirrors are carrying aarch64 for epel-9, but two seem to have content for epel-7.
This surprises me. I'd expect "mirror.fcix.net" and "ziply.mm.fcix.net" to be tagged as carrying aarch64 for all three releases 7,8,9.
None of your mirrors are carrying aarch64 for epel-9, but two seem to have content for epel-7. This surprises me. I'd expect "mirror.fcix.net" and "ziply.mm.fcix.net" to be tagged as carrying aarch64 for all three releases 7,8,9.
Maybe at the time I looked the crawler did not yet pick up the latest state or the mirror was updated slowly. Now it seems to be there:
$ curl -s "https://mirrors.fedoraproject.org/metalink?repo=epel-9&arch=aarch64&country=us" | grep fcix <url protocol="https" type="https" location="US" preference="99">https://ziply.mm.fcix.net/epel/9/Everything/aarch64/repodata/repomd.xml</url> <url protocol="http" type="http" location="US" preference="99">http://ziply.mm.fcix.net/epel/9/Everything/aarch64/repodata/repomd.xml</url> <url protocol="rsync" type="rsync" location="US" preference="99">rsync://ziply.mm.fcix.net/fedora-epel/9/Everything/aarch64/repodata/repomd.xml</url> <url protocol="https" type="https" location="US" preference="84">https://mirror.fcix.net/epel/9/Everything/aarch64/repodata/repomd.xml</url> <url protocol="http" type="http" location="US" preference="84">http://mirror.fcix.net/epel/9/Everything/aarch64/repodata/repomd.xml</url> <url protocol="rsync" type="rsync" location="US" preference="84">rsync://mirror.fcix.net/fedora-buffet/epel/9/Everything/aarch64/repodata/repomd.xml</url>
We have no history about what mirror was up to date at what time. We can always just the current state.
You can always check yourself in MirrorManager and see if all directories are up to date. The important one is the repodata directory. If that is not up to date for a certain combination, it will not show up in the output of the metalink.
repodata
Looking at the latest version of the epel-release package you can see everything uses the metalink: https://src.fedoraproject.org/rpms/epel-release/blob/epel7/f/epel.repo#_6
Looks like the mirrorlink vs. metalink is coming in from an external chef module setting up epel, the original problem isn't in this direction.
Can this be closed? Are there any more open questions?
I think we're all set. PEBKAC
Metadata Update from @warthog9: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.