The f41 x86_64 and aarch64 (but not s390x and ppc64le) repos are returning a metalink without the current repomd.xml checksums:
✗ podman run -it registry.fedoraproject.org/fedora:41 bash [root@46a695fcf14e /]# dnf --refresh check-update Updating and loading repositories: Fedora 41 openh264 (From Cisco) - x86_64 100% | 3.0 KiB/s | 6.0 KiB | 00m02s Fedora 41 - x86_64 100% [==================] | 12.3 KiB/s | 385.2 KiB | -00m00s Fedora 41 - x86_64 100% [==================] | 12.3 KiB/s | 385.2 KiB | -00m00s Fedora 41 - x86_64 100% [==================] | 12.3 KiB/s | 385.2 KiB | -00m00s Fedora 41 - x86_64 100% [==================] | 12.3 KiB/s | 391.1 KiB | -00m00s >>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055 Fedora 41 - x86_64 100% [==================] | 12.3 KiB/s | 391.1 KiB | -00m00s ...snip... >>> Downloading successful, but checksum doesn't match. Calculated: f6dc87bb2fab2e324c8c8f3980 >>> Downloading successful, but checksum doesn't match. Calculated: f6dc87bb2fab2e324c8c8f3980 >>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055 >>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055 >>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055 >>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055 >>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055 >>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055 >>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055 >>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055 >>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055 >>> Downloading successful, but checksum doesn't match. Calculated: 48ea7729665cea44a1e7755055 >>> Librepo error: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirror Failed to download metadata (metalink: "https://mirrors.fedoraproject.org/metalink?repo=fedora-41&arch=x86_64") for repository "fedora" Librepo error: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
I see in the primary-mirror-fedora pod:
Syncing directories of Fedora Linux ━━━━━╸ 14% 0:00:46 Traceback (most recent call last): File "/opt/app-root/bin/mm2_update-master-directory-list", line 8, in <module> sys.exit(main()) ^^^^^^ File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.11/site-packages/mirrormanager2/utility/update_master_directory_list.py", line 760, in main check_category( File "/opt/app-root/lib64/python3.11/site-packages/mirrormanager2/utility/update_master_directory_list.py", line 810, in check_category syncer.sync(**extra_args) File "/opt/app-root/lib64/python3.11/site-packages/mirrormanager2/utility/update_master_directory_list.py", line 132, in sync self.sync_directories(category_directories) File "/opt/app-root/lib64/python3.11/site-packages/mirrormanager2/utility/update_master_directory_list.py", line 217, in sync_directories loader.load() File "/opt/app-root/lib64/python3.11/site-packages/mirrormanager2/lib/umdl.py", line 204, in load md5dict = self._checksums_from_globs(self.md5_globs, 32) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.11/site-packages/mirrormanager2/lib/umdl.py", line 169, in _checksums_from_globs for f in self._get_filenames(): File "/opt/app-root/lib64/python3.11/site-packages/mirrormanager2/lib/umdl.py", line 177, in _get_filenames yield from os.listdir(os.path.join(self.config["UMDL_PREFIX"], self.relative_dir_name)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/srv/pub/fedora/linux/development/41/Workstation/aarch64/iso' umdl for Fedora Linux returned non-zero. Something failed.Sun Sep 15 11:13:52 PM UTC 2024
Indeed there is no iso dir... there's an images/ dir for the raw image...
CC: @abompard @adrian
It also affects rawhide: https://download.copr.fedorainfracloud.org/results/solopasha/hyprland/fedora-rawhide-x86_64/08022788-hyprland-plugins-git/builder-live.log.gz
FTR, this is burning also for the Copr team because a big portion of builds there is broken completely now because of this (build failure for all chroots, not just f41), that's because we somewhat unfortunately picked the latest branched Fedora version for custom builds.
Not sure if it's the root cause but the permissions on the 41_Beta directory are wrong:
$ ls -l /srv/pub/fedora/linux/releases/ total 144 [...] drwxr-xr-x. 10 263 263 4096 Aug 30 20:13 39 drwxr-xr-x. 13 263 263 4096 Apr 19 19:23 40 drwxr-x---. 2 263 263 4096 Sep 13 17:46 41_Beta drwxr-xr-x. 2 99 99 4096 May 21 2016 7 [...]
As a result, mirrormanager can't update the repo data in the database.
Yes it affects Rawhide / F42 https://discussion.fedoraproject.org/t/rawhide-copr-failure-today/131345
And it affects Fedora CI, please fix ASAP, it is basically blocking all testing.
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, mirrorlists, ops
The permissions are not wrong. ;) They are set so that tier1 mirrors can sync content before the beta release tomorrow. :)
So, we need mm to either ignore that dir entirely or get it perms to read it (which may be tricky in openshift).
OVN-Kubernetes project is also hitting this in our CI: https://github.com/ovn-org/ovn-kubernetes/pull/4726 we tried rawhide and 41 both don't work as outlined in this ticket. This is basically blocking us from being able to merge anything :( I don't mean to add fuel to fire just coming here to see if we can bump up the priority on this one.
Crazy workaround idea: a facl on the dir allow the gid of the openshift project access?
I'm not sure this explains why rawhide is affected too?
rawhide
this is also blocking critpath updates from going stable, because it breaks openQA tests, so they all fail gating.
ok. I worked around things and got updated mirrormanager data pushed out.
I totally missed that that dir was in releases/ which was a mistake from the stage-release.sh script. ;(
I removed that dir, refreshed fullfiletimelists, opened the beta dir perms, rescanned things to get updated and closed it again.
So, everything should be back to working for now. We still need to fix the underlying issue, but it's not as urgent tomorrow morning.
if anyone still sees problems after a 'dnf --refresh' please let us know, but it should be fixed.
Seems to work for me!
Just chiming in, that seems to have cleared things up for me! Thank you! Looking forward to continuing to test 41 as it comes together.
Metadata Update from @zlopez: - Issue untagged with: Needs investigation - Issue tagged with: high-gain, medium-trouble
Metadata Update from @zlopez: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
I'm seeing some "Downloading successful, but checksum doesn't match." messages with both F41 and RPM Fusion repos currently. It looks like "Calculated: 7ab2587801b2" on the F41 Updates repo and "Calculated: 1493e728b93b" on RPM Fusion nonfree (both F41/RF repos not testing)
@espionage724 please open a new ticket, as than it will show up on the "new/open issues" page and will be seen by the right people , while this comment most likely will not -- and it's clearly a different issue given the time that went past.
Ohh, and btw: things work for me (FWIW, I should considered mostly as an ordinary user here in this context) on F41 .
Log in to comment on this ticket.