fedora-ci / general

#128 Zuul on EPEL 8: Failed to download packages: Status code: 403 for https://infrastructure.fp.o/repo/rhel/rhel8/koji/latest/x86_64/RHEL-8-001/non_modular/glusterfs-api-6.0-37.el8.x86_64.rpm
Closed by churchyard. Opened by churchyard.

Closed
Reopen Issue

Hello, I am trying to run a CI test on Zuul on EPEL 8:

https://src.fedoraproject.org/rpms/python-tox/pull-request/35

The test repeatedly errors with:

Failed to download packages: Status code: 403 for https://infrastructure.fedoraproject.org/repo/rhel/rhel8/koji/latest/x86_64/RHEL-8-001/non_modular/glusterfs-api-6.0-37.el8.x86_64.rpm (IP: 38.145.60.16)

Is is some misconfiguration or a network problem?

cc @fbo @kevin

I can reproduce with:

podman run --user root -it centos:8

[root@668bac90e7a8 /]# cat /etc/yum.repos.d/epel8.repo
[epel8-build]
name=epel8-build
baseurl=https://kojipkgs.fedoraproject.org/repos/epel8-build/latest/x86_64/
gpgcheck=0
enabled=1

dnf install -y standard-test-roles
...
[MIRROR] glusterfs-6.0-37.el8.x86_64.rpm: Status code: 403 for https://infrastructure.fedoraproject.org/repo/rhel/rhel8/koji/latest/x86_64/RHEL-8-001/non_modular/glusterfs-6.0-37.el8.x86_64.rpm (IP: 38.145.60.16)
[MIRROR] glusterfs-api-6.0-37.el8.x86_64.rpm: Status code: 403 for https://infrastructure.fedoraproject.org/repo/rhel/rhel8/koji/latest/x86_64/RHEL-8-001/non_modular/glusterfs-api-6.0-37.el8.x86_64.rpm (IP: 38.145.60.16)
[MIRROR] glusterfs-6.0-37.el8.x86_64.rpm: Status code: 403 for https://infrastructure.fedoraproject.org/repo/rhel/rhel8/koji/latest/x86_64/RHEL-8-001/non_modular/glusterfs-6.0-37.el8.x86_64.rpm (IP: 38.145.60.16)
[MIRROR] glusterfs-api-6.0-37.el8.x86_64.rpm: Status code: 403 for https://infrastructure.fedoraproject.org/repo/rhel/rhel8/koji/latest/x86_64/RHEL-8-001/non_modular/glusterfs-api-6.0-37.el8.x86_64.rpm (IP: 38.145.60.16)
[MIRROR] glusterfs-6.0-37.el8.x86_64.rpm: Status code: 403 for https://infrastructure.fedoraproject.org/repo/rhel/rhel8/koji/latest/x86_64/RHEL-8-001/non_modular/glusterfs-6.0-37.el8.x86_64.rpm (IP: 38.145.60.16)
[MIRROR] glusterfs-api-6.0-37.el8.x86_64.rpm: Status code: 403 for https://infrastructure.fedoraproject.org/repo/rhel/rhel8/koji/latest/x86_64/RHEL-8-001/non_modular/glusterfs-api-6.0-37.el8.x86_64.rpm (IP: 38.145.60.16)
[MIRROR] glusterfs-6.0-37.el8.x86_64.rpm: Status code: 403 for https://infrastructure.fedoraproject.org/repo/rhel/rhel8/koji/latest/x86_64/RHEL-8-001/non_modular/glusterfs-6.0-37.el8.x86_64.rpm (IP: 38.145.60.16)
[FAILED] glusterfs-6.0-37.el8.x86_64.rpm: Status code: 403 for https://infrastructure.fedoraproject.org/repo/rhel/rhel8/koji/latest/x86_64/RHEL-8-001/non_modular/glusterfs-6.0-37.el8.x86_64.rpm (IP: 38.145.60.16)
...

But why ??

So, the problem here is that we don't expose rhel binaries to everyone. We are not going to distribute RHEL to the world via our copy. :)

That said, we could possibly enable the ci somehow.

Are these accesses from known ip's? Can you provide a list of those?

(Note that it used to work before.)

According to repodata there is a lot of packages from RHEL in the epel-build tag.
[fabien@ceres tmp]$ grep infrastructure.fedoraproject.org 43b636abe04d4d4f48d40f420d2b5dd65d9e033a73ad8bdef486521fe0b37b87-primary.xml | wc -l
6804

So should we avoid the use of that repo ? And simply use epel-release ?

@kevin the CI nodes are ephemeral VMs running on a dedicated cloud at vexxhost. Our IP range for the CI nodes is: 38.102.83.0/24

(Note that it used to work before.)

Huh, did it need to pull any rhel packages before? perhaps the tests changed?

According to repodata there is a lot of packages from RHEL in the epel-build tag.

Yes, they are all in 'external' koji repos.

[fabien@ceres tmp]$ grep infrastructure.fedoraproject.org 43b636abe04d4d4f48d40f420d2b5dd65d9e033a73ad8bdef486521fe0b37b87-primary.xml | wc -l
6804
So should we avoid the use of that repo ? And simply use epel-release ?
@kevin the CI nodes are ephemeral VMs running on a dedicated cloud at vexxhost. Our IP range for the CI nodes is: 38.102.83.0/24

Could you just use centos here? epel should layer on top of centos fine...

Huh, did it need to pull any rhel packages before? perhaps the tests changed?

No they didn't, but perhaps an attempted update od packages now fails :/

@churchyard should we avoid the use of epel8-build and simply stick to centos + epel-release ?

The default epel8 mock config is:

[BaseOS]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=8&arch=$basearch&repo=BaseOS&infra=$infra
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
gpgcheck=1
skip_if_unavailable=False
[AppStream]
name=CentOS-$releasever - AppStream
mirrorlist=http://mirrorlist.centos.org/?release=8&arch=$basearch&repo=AppStream&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/AppStream/$basearch/os/
gpgcheck=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
[PowerTools]
name=CentOS-$releasever - PowerTools
mirrorlist=http://mirrorlist.centos.org/?release=8&arch=$basearch&repo=PowerTools&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/PowerTools/$basearch/os/
gpgcheck=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
[Devel]
name=CentOS-$releasever - Devel WARNING! FOR BUILDROOT USE ONLY!
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=Devel&infra=$infra
#baseurl=http://mirror.centos.org/$contentdir/$releasever/Devel/$basearch/os/
gpgcheck=1
enabled=0
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
[centosplus]
name=CentOS-$releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=8&arch=$basearch&repo=centosplus&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/os/
gpgcheck=1
enabled=0
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
[cr]
name=CentOS-$releasever - cr
baseurl=http://mirror.centos.org/centos/8/cr/$basearch/os/
gpgcheck=1
enabled=0
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
[base-debuginfo]
name=CentOS-$releasever - Debuginfo
baseurl=http://debuginfo.centos.org/8/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
[extras]
name=CentOS-$releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=8&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
[fasttrack]
name=CentOS-$releasever - fasttrack
mirrorlist=http://mirrorlist.centos.org/?release=8&arch=$basearch&repo=fasttrack&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/fasttrack/$basearch/os/
gpgcheck=1
enabled=0
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
[BaseOS-source]
name=CentOS-$releasever - BaseOS Sources
baseurl=http://vault.centos.org/centos/8/BaseOS/Source/
gpgcheck=1
enabled=0
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
[AppStream-source]
name=CentOS-$releasever - AppStream Sources
baseurl=http://vault.centos.org/centos/8/AppStream/Source/
gpgcheck=1
enabled=0
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
[PowerTools-source]
name=CentOS-$releasever - PowerTools Sources
baseurl=http://vault.centos.org/centos/8/PowerTools/Source/
gpgcheck=1
enabled=0
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
[extras-source]
name=CentOS-$releasever - Extras Sources
baseurl=http://vault.centos.org/centos/8/extras/Source/
gpgcheck=1
enabled=0
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
[centosplus-source]
name=CentOS-$releasever - Plus Sources
baseurl=http://vault.centos.org/centos/8/centosplus/Source/
gpgcheck=1
enabled=0
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
[epel]
name=epel
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-8&arch=$basearch
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/epel/RPM-GPG-KEY-EPEL-8
gpgcheck=1
skip_if_unavailable=False
[testing]
name=epel-testing
enabled=0
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=testing-epel8&arch=$basearch
failovermethod=priority
skip_if_unavailable=False
[local]
name=local
baseurl=https://kojipkgs.fedoraproject.org/repos/epel8-build/latest/$basearch/
cost=2000
enabled=0
skip_if_unavailable=False
[epel-debuginfo]
name=epel-debug
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-debug-8&arch=$basearch
failovermethod=priority
enabled=0
skip_if_unavailable=False
[epel-source]
name=epel-source
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-source-8&arch=$basearch
failovermethod=priority
enabled=0
skip_if_unavailable=False
[epel-modular]
name=epel-modular
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-modular-8&arch=$basearch
failovermethod=priority
enabled=0
skip_if_unavailable=False
[epel-modular-debuginfo]
name=epel-modular-debug
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-modular-debug-8&arch=$basearch
failovermethod=priority
enabled=0
skip_if_unavailable=False
[epel-modular-source]
name=epel-modular-source
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-modular-source-8&arch=$basearch
failovermethod=priority
enabled=0
skip_if_unavailable=False

I suggest to use the same, with local enabled. Not sure about epel-modular but I guess that one needs to be enabled as well.
Edited by churchyard

local isn't going to work here... thats going to point to the same RHEL content we can't distribute. ;(

Could you add a centos-base.repo ?

Indeed the local repo loads itself from mock, but fails to download packages with error 403.

With https://pagure.io/fedora-zuul-jobs/pull-request/69 installation of standard-test-roles happens w/o epel8-build repo setup on the test node. Let's see how this behave by adding that PR as depends-on.

I was expecting to be able to use nobest option of dnf but it looks like dnf does not fallback on the regular epel repo to fetch gluster. Even not sure dnf can react to 403 errors.

But glusterfs is not in epel. It's in base RHEL and/or CentOS...

Thanks for the fix.

Metadata Update from @churchyard:
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.