fedora-infrastructure

Clone
Source Code
GIT

#11984 mirrors.centos.org centos stream 10 not setup correctly
Closed: Fixed 9 months ago by adrian. Opened 9 months ago by tdawson.

Closed: Fixed
Reopen Issue

NOTE

If your issue is for security or deals with sensitive info please
mark it as private using the checkbox below.

Describe what you would like us to do:

The stream 10 mirrormanager metadata/metalink does not seem to be setup correctly.

The biggest problem is that the verification hashes are not being updated. We've had a couple of pushes to the mirrors and the verification hashes haven't changed. They are also the same for all the arches. So if anyone uses the metalinks, dnf fails because the verification hash never matches what is downloaded from the mirrors.

$ curl 'https://mirrors.centos.org/metalink?repo=centos-baseos-10-stream&arch=x86_64&protocol=https,http' -s | grep sha512
    <hash type="sha512">c2f574a0f65a633b9d8541fec82a5cdb3d62ec3d033093d8e8617bc2ba2d6c02307ea18d4c9380da0c26dcd619780da0e3aff18139140c35629e6020070678f3</hash>
$ curl 'https://mirrors.centos.org/metalink?repo=centos-baseos-10-stream&arch=ppc64le&protocol=https,http' -s | grep sha512
    <hash type="sha512">c2f574a0f65a633b9d8541fec82a5cdb3d62ec3d033093d8e8617bc2ba2d6c02307ea18d4c9380da0c26dcd619780da0e3aff18139140c35629e6020070678f3</hash>

The other problem is that not all the arches are setup for all the repos. The following repo/arches are missing.

  • repo=centos-appstream-10-stream&arch=x86_64
  • repo=centos-baseos-10-stream&arch=s390x
  • repo=centos-highavailability-debug-10-stream&arch=aarch64
  • repo=centos-resilientstorage-10-stream&arch=ppc64le
  • repo=centos-resilientstorage-10-stream&arch=s390x
  • repo=centos-resilientstorage-10-stream&arch=x86_64

When do you need this to be done by? (YYYY/MM/DD)

2024/06/18

/cc: @adrian

Unfortunately I am a bit out of the loop with the current MirrorManager setup. The hosts I have access to seem to be no longer doing the work they used to. At this point I don't know how to help, sorry.

So, I looked a little bit in the database, and indeed the checksums here are what we have in the database:

mirrormanager2=# select filename,sha512 from file_detail fd join directory d on fd.directory_id = d.id where d.name = '10-stream/BaseOS/x86_64/os/repodata';
  filename  |                                                              sha512                                                              
------------+----------------------------------------------------------------------------------------------------------------------------------
 repomd.xml | c2f574a0f65a633b9d8541fec82a5cdb3d62ec3d033093d8e8617bc2ba2d6c02307ea18d4c9380da0c26dcd619780da0e3aff18139140c35629e6020070678f3
(1 row)

mirrormanager2=# select filename,sha512 from file_detail fd join directory d on fd.directory_id = d.id where d.name = '10-stream/BaseOS/ppc64le/os/repodata';
  filename  |                                                              sha512                                                              
------------+----------------------------------------------------------------------------------------------------------------------------------
 repomd.xml | c2f574a0f65a633b9d8541fec82a5cdb3d62ec3d033093d8e8617bc2ba2d6c02307ea18d4c9380da0c26dcd619780da0e3aff18139140c35629e6020070678f3
(1 row)

So I went and looked into the primary mirror scanner logs and I didn't find anything suspicious:

Step [600/600]: Reading http://mref1-priv.iad2.centos.org/10-stream/BaseOS/ppc64le/os/repodata/repomd.xml for checksum creation
 Step [601/601]: Reading http://mref1-priv.iad2.centos.org/10-stream/CRB/source/tree/repodata/repomd.xml for checksum creation
 Step [602/602]: Reading http://mref1-priv.iad2.centos.org/SIGs/10-stream/hyperscale/aarch64/packages-experimental/repodata/repomd.xml for checksum creation
 Step [603/603]: Reading http://mref1-priv.iad2.centos.org/10-stream/HighAvailability/s390x/debug/tree/repodata/repomd.xml for checksum creation
 Step [604/604]: Reading http://mref1-priv.iad2.centos.org/10-stream/CRB/ppc64le/debug/tree/repodata/repomd.xml for checksum creation
 Step [605/605]: Reading http://mref1-priv.iad2.centos.org/SIGs/9-stream/cloud/aarch64/openstack-bobcat/repodata/repomd.xml for checksum creation
 Step [606/606]: Reading http://mref1-priv.iad2.centos.org/10-stream/HighAvailability/x86_64/os/repodata/repomd.xml for checksum creation
 Step [607/607]: Reading http://mref1-priv.iad2.centos.org/10-stream/AppStream/aarch64/debug/tree/repodata/repomd.xml for checksum creation
 Step [608/608]: Reading http://mref1-priv.iad2.centos.org/10-stream/ResilientStorage/ppc64le/os/repodata/repomd.xml for checksum creation
 Step [609/609]: INSERT INTO "repository" ("name", "category_id", "version_id", "arch_id", "directory_id", "prefix", "disabled") VALUES ($1, $2, $3, $4, $5, $6, $7) -- binds: ["10-stream/ResilientStorage/ppc64le/os", 17, 934245, 15, 705701, "centos-resilientstorage-10-stream", false]
Created Repository(prefix=centos-resilientstorage-10-stream, version=934245, arch=15, category=17) -> Directory 705701
 Step [610/610]: Reading http://mref1-priv.iad2.centos.org/SIGs/10-stream/hyperscale/aarch64/packages-facebook/repodata/repomd.xml for checksum creation
 Step [611/611]: Reading http://mref1-priv.iad2.centos.org/10-stream/RT/x86_64/debug/tree/repodata/repomd.xml for checksum creation
 Step [612/612]: Reading http://mref1-priv.iad2.centos.org/10-stream/AppStream/x86_64/os/repodata/repomd.xml for checksum creation
 Step [613/613]: INSERT INTO "repository" ("name", "category_id", "version_id", "arch_id", "directory_id", "prefix", "disabled") VALUES ($1, $2, $3, $4, $5, $6, $7) -- binds: ["10-stream/AppStream/x86_64/os", 17, 934245, 3, 705714, "centos-appstream-10-stream", false]
Created Repository(prefix=centos-appstream-10-stream, version=934245, arch=3, category=17) -> Directory 705714
 Step [614/614]: Reading http://mref1-priv.iad2.centos.org/10-stream/AppStream/s390x/debug/tree/repodata/repomd.xml for checksum creation
 Step [615/615]: Reading http://mref1-priv.iad2.centos.org/10-stream/BaseOS/aarch64/debug/tree/repodata/repomd.xml for checksum creation
 Step [616/616]: Reading http://mref1-priv.iad2.centos.org/SIGs/9-stream/cloud/x86_64/openstack-antelope/repodata/repomd.xml for checksum creation
 Step [617/617]: Reading http://mref1-priv.iad2.centos.org/10-stream/BaseOS/ppc64le/debug/tree/repodata/repomd.xml for checksum creation
 Step [618/618]: Reading http://mref1-priv.iad2.centos.org/SIGs/9-stream/cloud/ppc64le/openstack-caracal/debug/repodata/repomd.xml for checksum creation
 Step [619/619]: Reading http://mref1-priv.iad2.centos.org/10-stream/RT/x86_64/os/repodata/repomd.xml for checksum creation
 Step [620/620]: Reading http://mref1-priv.iad2.centos.org/SIGs/9-stream/cloud/x86_64/openstack-caracal/repodata/repomd.xml for checksum creation
 Step [621/621]: Reading http://mref1-priv.iad2.centos.org/SIGs/10-stream/hyperscale/aarch64/packages-experimental/debug/repodata/repomd.xml for checksum creation
 Step [622/622]: Reading http://mref1-priv.iad2.centos.org/SIGs/10-stream/hyperscale/aarch64/packages-facebook/debug/repodata/repomd.xml for checksum creation
 Step [623/623]: Reading http://mref1-priv.iad2.centos.org/SIGs/9-stream/cloud/ppc64le/openstack-bobcat/debug/repodata/repomd.xml for checksum creation
 Step [624/624]: Reading http://mref1-priv.iad2.centos.org/10-stream/HighAvailability/x86_64/debug/tree/repodata/repomd.xml for checksum creation
 Step [625/625]: Reading http://mref1-priv.iad2.centos.org/10-stream/ResilientStorage/x86_64/os/repodata/repomd.xml for checksum creation
 Step [626/626]: INSERT INTO "repository" ("name", "category_id", "version_id", "arch_id", "directory_id", "prefix", "disabled") VALUES ($1, $2, $3, $4, $5, $6, $7) -- binds: ["10-stream/ResilientStorage/x86_64/os", 17, 934245, 3, 705750, "centos-resilientstorage-10-stream", false]
Created Repository(prefix=centos-resilientstorage-10-stream, version=934245, arch=3, category=17) -> Directory 705750
 Step [627/627]: Reading http://mref1-priv.iad2.centos.org/10-stream/CRB/s390x/os/repodata/repomd.xml for checksum creation
 Step [628/628]: Reading http://mref1-priv.iad2.centos.org/10-stream/ResilientStorage/s390x/debug/tree/repodata/repomd.xml for checksum creation
 Step [629/629]: Reading http://mref1-priv.iad2.centos.org/10-stream/AppStream/s390x/os/repodata/repomd.xml for checksum creation
 Step [630/630]: Reading http://mref1-priv.iad2.centos.org/SIGs/10-stream/hyperscale/source/packages-facebook/repodata/repomd.xml for checksum creation
 Step [631/631]: Reading http://mref1-priv.iad2.centos.org/10-stream/HighAvailability/s390x/os/repodata/repomd.xml for checksum creation
 Step [632/632]: Reading http://mref1-priv.iad2.centos.org/10-stream/NFV/x86_64/os/repodata/repomd.xml for checksum creation
 Step [633/633]: Reading http://mref1-priv.iad2.centos.org/10-stream/HighAvailability/aarch64/os/repodata/repomd.xml for checksum creation
 Step [634/634]: Reading http://mref1-priv.iad2.centos.org/10-stream/HighAvailability/ppc64le/debug/tree/repodata/repomd.xml for checksum creation
 Step [635/635]: Reading http://mref1-priv.iad2.centos.org/SIGs/9-stream/cloud/aarch64/openstack-caracal/repodata/repomd.xml for checksum creation
 Step [636/636]: Reading http://mref1-priv.iad2.centos.org/10-stream/CRB/aarch64/debug/tree/repodata/repomd.xml for checksum creation
 Step [637/637]: Reading http://mref1-priv.iad2.centos.org/10-stream/AppStream/ppc64le/os/repodata/repomd.xml for checksum creation
 Step [638/638]: Reading http://mref1-priv.iad2.centos.org/SIGs/9-stream/cloud/source/openstack-antelope/repodata/repomd.xml for checksum creation
 Step [639/639]: Reading http://mref1-priv.iad2.centos.org/10-stream/BaseOS/x86_64/os/repodata/repomd.xml for checksum creation
 Step [640/640]: Reading http://mref1-priv.iad2.centos.org/SIGs/9-stream/cloud/source/openstack-caracal/repodata/repomd.xml for checksum creation
 Step [641/641]: Reading http://mref1-priv.iad2.centos.org/SIGs/9-stream/cloud/x86_64/openstack-bobcat/debug/repodata/repomd.xml for checksum creation
 Step [642/642]: Reading http://mref1-priv.iad2.centos.org/10-stream/AppStream/source/tree/repodata/repomd.xml for checksum creation
 Step [643/643]: Reading http://mref1-priv.iad2.centos.org/SIGs/9-stream/cloud/source/openstack-bobcat/repodata/repomd.xml for checksum creation
 Step [644/644]: Reading http://mref1-priv.iad2.centos.org/10-stream/BaseOS/s390x/debug/tree/repodata/repomd.xml for checksum creation

The following command returned nothing:

oc -n mirrormanager logs job/primary-mirror-centos-28639152 | grep -i error

I tried to retrieve the file that has been read in the logs to check the sha512 sum, but I got a 404:

wget http://mref1-priv.iad2.centos.org/10-stream/BaseOS/x86_64/os/repodata/repomd.xml
--2024-06-14 07:22:09--  http://mref1-priv.iad2.centos.org/10-stream/BaseOS/x86_64/os/repodata/repomd.xml
Resolving mref1-priv.iad2.centos.org (mref1-priv.iad2.centos.org)... 10.3.163.235
Connecting to mref1-priv.iad2.centos.org (mref1-priv.iad2.centos.org)|10.3.163.235|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-06-14 07:22:09 ERROR 404: Not Found.

I'm surprised, it's the URL that is in the log as the file being read for checksum creation. Is there some prefix that does not appear in the logs? http://mref1-priv.iad2.centos.org/10-stream/ is also a 404.

@adrian, you should have access to:
https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/ns/mirrormanager/cronjobs/primary-mirror-centos/jobs
That's the last cronjob runs to scan the centos primary mirror.

I'll be off this afternoon and Monday as well, but feel free to ask questions here and I may have time to answer in the next hours.

For the record, the mirrormanager images build with version 0.4.3 of scan-primary-mirror as specified here (from the branch with the same name in the scan-primary-mirror's git repo).

 
# curl -s 'https://mirrors.centos.org/metalink?repo=centos-baseos-10-stream&arch=x86_64&protocol=https,http' | grep md5
    <hash type="md5">234fee33caea89251f7281aee17d4100</hash>
# curl -s https://mirror.stream.centos.org/10-stream/BaseOS/x86_64/os/repodata/repomd.xml | md5sum
b3143a9cad30acd398cd8ae6a1d63e40

So the primary mirror is also returning the wrong checksum.
Edited 9 months ago by adrian

```

curl -s 'https://mirrors.centos.org/metalink?repo=centos-baseos-10-stream&arch=x86_64&protocol=https,http' | grep md5

<hash type="md5">234fee33caea89251f7281aee17d4100</hash>

curl -s https://mirror.stream.centos.org/10-stream/BaseOS/x86_64/os/repodata/repomd.xml | md5sum

b3143a9cad30acd398cd8ae6a1d63e40
```

So the primary mirror is also returning the wrong checksum.

All of the actual mirrors share the same checksums. It's the mirror link checksum that disagrees.

I actually had a look and the fact that mirrormanager was returning the same checksum for all arches was .. interesting.
So I chatted with @adrian and we found that it was the computed checksum from a returned 404 page !
While Mirrormanager is using rsync to discover new files/repositories, it still uses http to compute repomd.xml checksum and there was a thing specific for that vhost/setup in IAD2 that wasn't added for 10-stream Alias / path .. so that internal IAD2 mirror was returning 404, that mirrormanager was computing as the real checksum

It should be fixed soon when crawler will validate against internal iad2 server

Room for improvement at the mirrormanager side : check the http returned code (200 vs 404) and only store checksum against real returned repomd.xml and not 404 page (or else)

I started a run of the primary mirror scanner just now manually on the old RHEL7 host. Not sure how to do it in OpenShift.
Edited 9 months ago by adrian

Should be fixed now,

Upstream I also added a check to avoid situation like this in the future: https://github.com/adrianreber/scan-primary-mirror/pull/224

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: high-gain, medium-trouble, ops
9 months ago

@adrian Does this ticket still need something to be done or could we close it now?

I would say this is done. Closing.

Metadata Update from @adrian:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
9 months ago

I have verified that this is now working for CentOS Stream 10 machines. Thank you all for your work in fixing this.

Log in to comment on this ticket.

Metadata
None

medium-trouble ops high-gain

None
None
Waiting on Assignee
Boards 1
ops Status: Backlog
Powered by Pagure 5.14.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.