fedora-infrastructure

#8979 iad2/openshift: coreos-cincinnati builds are very slow and always fail
Closed: Fixed by kevin. Opened by lucab.

Closed: Fixed
Reopen Issue

I've deployed the existing coreos-cincinnati playbook to the new IAD2 OpenShift cluster, however builds never succeed there. The build-configuration has a 30mins sanity-timeout, and builds always hit that ceiling without making much progress.

Excerpt from the latest attempted build:

Cloning "https://github.com/coreos/fedora-coreos-cincinnati.git" ...
Commit: 96b5e815f09a1be9c6660aa8f45823c32c71e048
Step 1/11 : FROM fedora:31
 ---> adfbfa4a115a
Step 2/11 : RUN dnf -y install g++ openssl-devel
 ---> Running in 522ecb0ab910
Fedora Modular 31 - x86_64                       89 kB/s | 5.2 MB     01:00    
Fedora Modular 31 - x86_64 - Updates             69 kB/s | 4.1 MB     01:00
[eventual 30mins timeout]

The same playbook and build-configuration "reliably" works on the existing cluster.

It looks like the dnf install is being insanely slowed down or throttled, possibly on network access.

I don't have access to other namespaces/pods to verify whether this is a problem specific to the project, or cluster-wide.

Metadata Update from @pingou:
- Issue tagged with: iad2

Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: groomed, medium-gain, medium-trouble

Short answer: It was the egress policy we had set on your application.

Longer answer: In phx2 we setup mirrormanager to point anything coming from our net to our master mirrors. We have not yet done this in iad2, so you basically get the normal mix of random mirrors from mirrors.fedoraproject.org. Unfortunately in our egress policy we locked things down to just the master mirrors, so all the other ones were blocked.

For now, I just set your egress policy to ALlow at the end.

We need to setup mirrormanager for iad2 and adjust the egress policy a bit and then it should be fine to put back the deny.

In the mean time if looks like it's building... take a look and see if that fixes the issue for you.

Indeed, that was it.

Builds confirmed working now (at least the one you triggered quickly completed without issues).

The application got deployed and is confirmed working too.

Unless you want to re-purpose this ticket to track the mirrormanager work, from my side it can be closed.

Great! Thanks.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata
None

iad2 medium-gain medium-trouble

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.