Hi,
The issue in #10914 seems to happen again. Currently (2023-04-11 9:35 UTC), download-ib01.fedoraproject.org is giving out inconsistent content in rsync://download-ib01.fedoraproject.org/fedora-epel/7/x86_64/repodata/
rsync rsync://download-ib01.fedoraproject.org/fedora-epel/7/x86_64/repodata/ ---------------------------- Fedora Master Mirror Servers ---------------------------- If you are using bare rsync to sync content, please stop and take a look at https://pagure.io/quick-fedora-mirror. See http://fedoraproject.org/wiki/Infrastructure/Mirroring for further instructions. ---------------------------- You have connected to download-ib01.fedoraproject.org drwxr-xr-x 4,096 2023/04/10 08:57:49 . -rw-r--r-- 331 2023/03/28 09:16:03 2762f5c8a7bede2413dd1dca4cf5837b2e5f4a3aa7d4f3174710bf4c18ca5e08-prestodelta.xml.gz -rw-r--r-- 3,521,760 2023/03/28 09:15:52 311972ed304fa09cb1a0fa59af30ed2b3f33d5c65e136dcaa54fef63d877f2d2-other.sqlite.bz2 -rw-r--r-- 1,092,209 2023/03/28 09:34:34 49978b01f0fae428acd830e1d6bc5db6a9f1c37ef52b9099184c004d619af9f3-updateinfo.xml.bz2 -rw-r--r-- 406,942 2023/04/10 08:42:16 58dca1bee60fc8c4daaadf4dc92072b89e206a1b86460501ce60709e0984fdd0-comps-Everything.x86_64.xml -rw-r--r-- 12,263,799 2023/03/28 09:15:56 8149599f2e301b234ca0b6b41c61cf64a2e8bac4e6bfe0d9c4ffd30ae589568c-filelists.sqlite.bz2 -rw-r--r-- 7,360,754 2023/03/28 09:15:53 976b7319aa484268735118362f9680fbf440251c8b4ab928e16fdefbceec3cc7-primary.sqlite.bz2 -rw-r--r-- 101,292 2023/04/10 08:42:33 97d2e9722dd5c0662a5c885a746a127e951f8860139076802e27d35eadb4aa45-comps-Everything.x86_64.xml.gz -rw-r--r-- 4,097,204 2023/03/28 09:15:50 c5052f189a998f5e4c089358a719cfb586d2b727168d3527eceeb41569a6863c-primary.xml.gz -rw-r--r-- 2,557,014 2023/03/28 09:15:50 d4eae01646d4b91d744f2e93978b5d0c0f282e20861f77099c697f3a37dafd22-other.xml.gz -rw-r--r-- 12,032,289 2023/03/28 09:15:50 dfbfded9c9c76540921e974a6d1dd24114ba8bf1a7a67e4de77138e0fbc921c7-filelists.xml.gz -rw-r--r-- 4,850 2023/04/10 08:57:49 repomd.xml
However, the content in dl.fedoraproject.org seems to be okay.
rsync rsync://dl.fedoraproject.org/fedora-epel/7/x86_64/repodata/ ---------------------------- Fedora Master Mirror Servers ---------------------------- If you are using bare rsync to sync content, please stop and take a look at https://pagure.io/quick-fedora-mirror. See http://fedoraproject.org/wiki/Infrastructure/Mirroring for further instructions. ---------------------------- You have connected to dl-iad01.fedoraproject.org drwxr-xr-x 8,192 2023/04/10 08:57:49 . -rw-r--r-- 1,091,229 2023/04/10 08:57:49 2179c9060af8e903ba2bf8459196aea9a0b2adf2966fe406eaa05519c7093e0d-updateinfo.xml.bz2 -rw-r--r-- 12,034,746 2023/04/10 08:42:33 3829d95e69d04e5fa68fb3a9823da4a4ec8d60006a513d83ec3b48acfaa0afbd-filelists.xml.gz -rw-r--r-- 689 2023/04/10 08:42:46 3c4c59ba89f4ff93fb29dd7836a24ebe5be397c18e02e256832ad3fdf9ac71e7-prestodelta.xml.gz -rw-r--r-- 12,256,263 2023/04/10 08:42:39 4f81843a3014e6101e49a42d0cbf44642eab49fa2eed1e085a394311a3aae701-filelists.sqlite.bz2 -rw-r--r-- 406,942 2023/04/10 08:42:16 58dca1bee60fc8c4daaadf4dc92072b89e206a1b86460501ce60709e0984fdd0-comps-Everything.x86_64.xml -rw-r--r-- 2,556,442 2023/04/10 08:42:33 67bf2891d3e35ee5d497e82b6344222c230da05a3f6d03073810af9a4bceeb78-other.xml.gz -rw-r--r-- 4,098,142 2023/04/10 08:42:33 78ecd399f01a2d13a29608ebe5e604c3f5bc2078bc556bd35313ad3a83bf0ce7-primary.xml.gz -rw-r--r-- 101,292 2023/04/10 08:42:33 97d2e9722dd5c0662a5c885a746a127e951f8860139076802e27d35eadb4aa45-comps-Everything.x86_64.xml.gz -rw-r--r-- 3,524,859 2023/04/10 08:42:35 a6b676c471786ccd721b8d585dfa8168bf34ac6db2131c89ffa7b20f4b60acaa-other.sqlite.bz2 -rw-r--r-- 7,361,914 2023/04/10 08:42:36 da47a6c6e3f666763394b611165ecb46a091958ff26b96000e805810b2bc9a02-primary.sqlite.bz2 -rw-r--r-- 4,850 2023/04/10 08:57:49 repomd.xml
It can be seen in the file list given by ib01 that repomd.xml is a newer version, the modification time of which is 2023/04/10, which is the same as that on iad0. However, the modification time of other xml and sqlite files on ib01 is 2023/03/28, which is early than those on iad0.
repomd.xml
The similar issue also happens on rsync://download-ib01.fedoraproject.org/fedora-enchilada/linux/updates/37/Everything/x86_64/repodata/
rsync rsync://download-ib01.fedoraproject.org/fedora-enchilada/linux/updates/37/Everything/x86_64/repodata/ ---------------------------- Fedora Master Mirror Servers ---------------------------- If you are using bare rsync to sync content, please stop and take a look at https://pagure.io/quick-fedora-mirror. See http://fedoraproject.org/wiki/Infrastructure/Mirroring for further instructions. ---------------------------- You have connected to download-ib01.fedoraproject.org drwxr-xr-x 4,096 2023/04/10 08:32:24 . -rw-r--r-- 1,682,460 2023/03/28 09:53:31 0302d15a7651407a09e5a31c2336a6e002be62e7f650e208529a4e813eae8826-updateinfo.xml.xz -rw-r--r-- 6,584,858 2023/03/28 08:27:04 1345182ed9e2f96e7f5c8d3c846ce1c3ef81268f99d0e423955f331e5e435055-primary.xml.zck -rw-r--r-- 4,578,877 2023/03/28 08:27:11 21680a5000969ea40c376c1703b7da9e1a7769a254d895de60a39cf566e59cc1-other.sqlite.bz2 -rw-r--r-- 24,212 2023/03/28 08:35:26 218d21b43562a752799edfed758ba524aab7a7761ccb880fecaeeed4806315cf-prestodelta.xml.gz -rw-r--r-- 1,832,947 2023/03/28 08:25:58 248e1cb496a86337b3b42ce86a6fd03a1b0deb13d8b040c356746345142e207a-comps-Everything.x86_64.xml -rw-r--r-- 6,578,369 2023/03/28 08:27:04 300f97f5e74057d49370466dc299a6629322e400ea92ceb97d926fe58c947a31-primary.xml.gz -rw-r--r-- 22,977,604 2023/03/28 08:27:04 4af854743041950adc801d67069644161e5f2b48935736e32ac3f4ce2c250419-filelists.xml.gz -rw-r--r-- 2,097,959 2023/03/28 09:53:34 4c2f0ffd786b3a3e4873f9740a02880bdcd96d3277f1a2fc9ed93442fe8da3fb-updateinfo.xml.zck -rw-r--r-- 13,529,840 2023/03/28 08:27:12 4c31bebb1fac1de6e6088253189f88bddc0c6b0a9215ee16d817d283afe0d152-primary.sqlite.bz2 -rw-r--r-- 1,833,041 2023/04/10 08:23:21 4e1f3a172b96d193431300e93f8b3c688c4613cd2401cef98a5ad3e34e3839a9-comps-Everything.x86_64.xml -rw-r--r-- 451,757 2023/03/28 08:27:05 58e4c2e288ab418309add41bcb9b4931e83c347b812fa1f64504cb8399781501-comps-Everything.x86_64.xml.gz -rw-r--r-- 2,639,590 2023/03/28 08:27:04 5d78c86c7f8b55fa3970dbd68f115dba8a5a7515d6070d4435320edbdc7d923f-other.xml.zck -rw-r--r-- 499,306 2023/03/28 08:27:22 796d08d101605349e04bf8d8c389b4023dd3eedf469558d9b0a6b7ddb5a2ddda-comps-Everything.x86_64.xml.zck -rw-r--r-- 87,396 2023/03/28 08:35:26 9235d1296a2f6173c1bc972ffc106f26c545015788b1aadf23e7f009d4ef3847-prestodelta.xml.zck -rw-r--r-- 19,975,043 2023/03/28 08:27:04 963ae43f1d69af6ecc4e64c41fd87576f694a0ef1d253c9c50a0d91b72242744-filelists.xml.zck -rw-r--r-- 2,564,978 2023/03/28 08:27:04 9f715080f47c9086b741a8af8a7f2b947ebd147e12fa26549a221fe4d71dc388-other.xml.gz -rw-r--r-- 23,291,736 2023/03/28 08:27:20 d90c3ee78f42baede9a22f43b6cdfbd45384617aa61b7fc83c3f9ed969490b8f-filelists.sqlite.bz2 -rw-r--r-- 499,371 2023/04/10 08:24:50 f2b273466eb70e538d00ae71f30bb5a00e386ac37d240dd8ee930e8566a6eb80-comps-Everything.x86_64.xml.zck -rw-r--r-- 451,780 2023/04/10 08:24:33 f59866fcdfc9fd88e07a9b70e5bebd5308376efeafac3fbb45bec56bd382a7fa-comps-Everything.x86_64.xml.gz -rw-r--r-- 8,313 2023/04/10 08:32:24 repomd.xml
rsync rsync://dl.fedoraproject.org/fedora-enchilada/linux/updates/37/Everything/x86_64/repodata/ ---------------------------- Fedora Master Mirror Servers ---------------------------- If you are using bare rsync to sync content, please stop and take a look at https://pagure.io/quick-fedora-mirror. See http://fedoraproject.org/wiki/Infrastructure/Mirroring for further instructions. ---------------------------- You have connected to dl-iad01.fedoraproject.org drwxr-xr-x 8,192 2023/04/11 09:32:37 . -rw-r--r-- 20,300,987 2023/04/11 08:31:32 26ceaa66fb9260262b317025c10f4d21c7d2ad07b35f9c56af5221d718d1e4e5-filelists.xml.zck -rw-r--r-- 6,748,266 2023/04/11 08:31:32 30612b456fa3398518548517306128a5eb8ef75bea816b2185c0771202b75cc9-primary.xml.zck -rw-r--r-- 15,189 2023/04/11 08:32:25 42584abe24535749f48ba8c2763880cc30a951ce8355ab2e5e7017a4e695de1f-prestodelta.xml.gz -rw-r--r-- 1,833,041 2023/04/11 08:30:20 4e1f3a172b96d193431300e93f8b3c688c4613cd2401cef98a5ad3e34e3839a9-comps-Everything.x86_64.xml -rw-r--r-- 13,851,453 2023/04/11 08:31:40 6431f62d7c4a5d53d83c31f311d74dd39905218a70745c375a70de21e72ff36a-primary.sqlite.bz2 -rw-r--r-- 23,310,235 2023/04/11 08:31:32 662b929e5f67f0ed5ebb8cca41189baf04fec0b564255283f2971ed5e5250a53-filelists.xml.gz -rw-r--r-- 6,734,483 2023/04/11 08:31:32 731f92520edc2ffb0a5f197aef8924c456934b658aeb20a6b6d4eadc3af6638f-primary.xml.gz -rw-r--r-- 23,631,179 2023/04/11 08:31:48 7a84ff6835e46dfd5678e5f0ec97fef85b46d20b6c1c19e42c8e7009055f35a9-filelists.sqlite.bz2 -rw-r--r-- 2,758,033 2023/04/11 08:31:32 8d8e8c7b486500cd2170f3feafa42d98aa96cf99ea014175f2d5f1dfc9bca750-other.xml.zck -rw-r--r-- 66,783 2023/04/11 08:32:25 aa9a0bd58aadd4ad5e3512603c3010cd9a7116fce9338c8f029930ec7ef5f7cb-prestodelta.xml.zck -rw-r--r-- 1,750,512 2023/04/11 09:32:33 b6541fbb72210423bd920716e02f4ff46b102c35afd37814d22b5c0ff68eddec-updateinfo.xml.xz -rw-r--r-- 2,680,767 2023/04/11 08:31:32 eee93ed82f38c96d4ca2118509aca614b7ec2b98c5c3fc472d86e96959f8ee04-other.xml.gz -rw-r--r-- 499,371 2023/04/11 08:31:50 f2b273466eb70e538d00ae71f30bb5a00e386ac37d240dd8ee930e8566a6eb80-comps-Everything.x86_64.xml.zck -rw-r--r-- 2,174,507 2023/04/11 09:32:37 f3f9e3db421af03f6942ac4ac91c8e819e53486eaa0670765f35d11bfd749e59-updateinfo.xml.zck -rw-r--r-- 451,780 2023/04/11 08:31:32 f59866fcdfc9fd88e07a9b70e5bebd5308376efeafac3fbb45bec56bd382a7fa-comps-Everything.x86_64.xml.gz -rw-r--r-- 4,758,710 2023/04/11 08:31:38 fdf4f60a23ce4685fc6e6784e0ecf7310494f0d9fa7d813a14369c00d1b8d2a2-other.sqlite.bz2 -rw-r--r-- 8,316 2023/04/11 09:32:37 repomd.xml
The symptom is basically the same as #10914.
Yes, that machine is having trouble. I am working on fixing it.
In the mean time I have disabled it from mirror lists. It should be out in an hour or so.
Metadata Update from @phsmoura: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops
Just a status update here.
I am still trying to get download-ib01 back fully in sync. It's better now, but still not 100% caught up yet.
We notified the kernel.org folks. They were syncing from download-ib01. We asked them to move and sync from dl.fedoraproject.org at least for now, so hopefully they will be back in sync again soon.
Duplicate ticket closed #11236 https://download-ib01.fedoraproject.org/pub/epel/8/Everything/x86_64/repodata/ is broken. repomd.xml was updated on 2023-04-10 and contains a link to a missing file.
Sorry for the lack of updates here. I have been poking at this off and on as time permits and hitting weird problems and fixing them.
download-ib01 is now a rhel9 instance, with more disk and memory. It now includes archives.
It's now been completely in sync for 2 days, so I am hopeful I have fixed all the various things I hit that were preventing it from staying in sync.
I've re-enabled it for mirrormanager and it should be open for business.
Please do file a new ticket if you all still see anything out of date with it.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.