I see this with another build https://fedora.softwarefactory-project.io/zuul/build/d4b90665fe004c04b537a71988d49509 too. From my laptop I also got forbidden to access this url. Should it be reported as an issue on fedora-infra https://pagure.io/fedora-infrastructure/issues ?

I thought about that, since the name kinda suggests it's an infra managed server. I'll ask on Matrix and report back here.

Here is the answer I've got from @kevin on Matrix:

well, I guess we need to talk to Zuul folks... we don't provide free rhel to everyone. ;) If there's a way to allowlist just specific ips from them we may be able to do that...

Since I'm neither part of Zuul folks nor of Infra folks, I think it would be more productive to eliminate the man in the middle (c'est moi).

So to add a bit of context here... this the fedora infrastructure PRIVATE mirror of RHEL. It's supposed to be 403 for everyone but specific whitelisted addresses.
We don't want to provide rhel binaries for the world. That would be very bad. ;)

We may be able to whitelist zuul ip's... as long as some things are agreed, like no way for people to download buildroots/binaries from the rhel packages, no other things using those ip's, those ip's not changing without us knowing, etc.

It may also be possible to just use alma or rocky, or something else... perhaps cloud instances?

Anyhow, happy to help here, but realize we don't want to just expose rhel binaries out...

I think we need EPEL folks involved here.

From my point of view we can just switch test environment to use CentOS Stream 9. Whatever is built in EPEL must work on CentOS Stream even if to build a package EPEL used RHEL buildroot.

If that's ok from EPEL point of view - we would just need to adjust the pipeline slightly.

--

If we really want to test EPEL packages against RHEL, than we need a separate conversation with EPEL Steering Committee.

It is doable, the pool of machines which is used by Zuul CI is actually provided to Zuul folks by the Fedora Infrastructure AWS account, and we can setup separate networking rules for the dedicated EPEL testing nodes, but I am not sure if EPEL community really needs that.

And in the future (EPEL 10) as I understood by default we will build EPEL packages against CentOS Stream 10.

CC @carlgeorge @tdawson

I think we need EPEL folks involved here.

From my point of view we can just switch test environment to use CentOS Stream 9. Whatever is built in EPEL must work on CentOS Stream even if to build a package EPEL used RHEL buildroot.

This works, until it doesn't.
CentOS Stream is upstream of RHEL and often has newer libraries, that sometimes are not 100% backwards compatible
One library that always keeps people on their toes is LLVM, it get's updated each release, and seems to break older packages here and there. qt5 is another package, but supposedly it's gotten better. But it's different for each RHEL release.

If we really want to test EPEL packages against RHEL, than we need a separate conversation with EPEL Steering Committee.

I don't see what the Steering Committee can do here, or what is really being asked.
We, the EPEL Steering Committee can't grant you access to a RHEL repository. That's Fedora infrastructure, or possible zuule people, or Red Hat subscription people.

It is doable, the pool of machines which is used by Zuul CI is actually provided to Zuul folks by the Fedora Infrastructure AWS account, and we can setup separate networking rules for the dedicated EPEL testing nodes, but I am not sure if EPEL community really needs that.

And in the future (EPEL 10) as I understood by default we will build EPEL packages against CentOS Stream 10.

Yes and no.
The latest EPEL builds, on epel10, will build against CentOS Stream 10.
But, we will also have the epel10.N (or maybe it's 10_N) that will be built against the current stable RHEL release.

Let me ask the obvious question: What is the setup for EPEL8 jobs in Zuul?

As I pointed out, the Zuul job for EPEL8 just works™.

I don't see what the Steering Committee can do here, or what is really being asked.
We, the EPEL Steering Committee can't grant you access to a RHEL repository.

"Zuul people" don't need to access Red Hat private repos. Zuul provides an open service for the Fedora project and its subprojects to test packages using standard AWS virtual machines. If EPEL wants to be able to test EPEL packages against RHEL repos in Zuul CI, someone from EPEL needs to request access to RHEL repos for Zuul CI.

Technically speaking Zuul is a CI orchestrator, which can schedule a job on a vm from the pre-defined pool. You can add your own pools of VMs to Zuul.

So I'd say that if EPEL project needs RHEL-based CI, than EPEL project has to get an AWS pool from Fedora Infra with the agreement that this specific pool will have access to private RHEL repos.

Then we need to add this pool to Zuul CI and setup Zuul pipelines so that EPEL jobs will be scheduled on machines from this pool.

The owner of the pool (and access to the RHEL repos) will be the EPEL project. Zuul will be scheduling certain specific jobs on it.

I guess that comes back to a more basic question.
Who is requesting this? What are the parameters that were asked?
Maybe the question has the answers in it.

Who is requesting this? What are the parameters that were asked?

I'm not sure who you are asking. In case that question wasn't meant for me, please ignore.

From my perspective, this is something that comes built-in in dist-git. I submit a PR and that triggers CI jobs. If those are run, I expect them to have meaningful results. A failure due to an inaccessible repo is useless.

I'm fine with CI jobs not being triggered for certain branches. In which case I would need to take care of some testing before pushing a package update. I prefer the automation, though, since it takes that load of my shoulders and I can spend that time on other things.

Basically, the workflow for the job rpm-install-test is:
- 1. wait for the dependent job rpm-scrach-build (this job build on Fedora's Koji the package(s) for the given PR/branch)
- 2. on a stream9 VM (provided by our infra - not Fedora AWS currently)
- 3. install epel-release
- 4. install the zuul-built repo (it contains the built packages fetched from Koji)
- 5. install the built context repo from Koji https://kojipkgs.fedoraproject.org/repos/epel9-build/latest/x86_64/ repo
- 6. run the installation test

So my understanding is that the build context repo fallback to an internal infra to serve rhel packages. Then repo added to the dnf config on step 5 is actually causing trouble and perhaps we could simply remove that. All installation deps for a given built package should be resolved from the base system packages or epel repo.

Here is the job definition where we set an additional variable that trigger the extra repo to be added to the dnf config: https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/jobs.yaml#_99

I'll propose a patch to remove this for epel9

@gui1ty I've merged https://pagure.io/fedora-zuul-jobs/pull-request/186 and your change pass the test.

Let me know if we can close this issue.

LGTM. Thanks!