fedora-qa / os-autoinst-distri-fedora

#481 podman tests frequently fail due to quay.io connectivity issues(?)
Opened by adamwill. Modified

Open
Close Issue

In the last day or two, the podman test is frequently failing (on both arches, on both instances). There are various failures, but they all seem to happen when a test is trying to pull something from the real quay.io for some reason. One common one is a timeout doing a pull, like this:

not ok 96 [070] build with copy-from referencing the base image
# (from function `run_podman' in file /usr/share/podman/test/system/helpers.bash, line 598,
#  in test file /usr/share/podman/test/system/070-build.bats, line 890)
#   `run_podman build --jobs 1 -t ${target} -f ${containerfile2} ${tmpdir}' failed
#
# [13:44:38.416312159] # podman  image exists quay.io/libpod/alpine:latest
# [13:44:38.445043131] [ rc=1 (expected) ]
#
# [13:44:38.451284745] # podman  build --jobs 1 -t derived-t96-aecwt3ct -f /tmp/podman_bats.7wWEDM/build-test/Containerfile2 /tmp/podman_bats.7wWEDM/build-test
# [13:46:38.475138625] [2/2] STEP 1/2: FROM quay.io/libpod/alpine AS final
# Trying to pull quay.io/libpod/alpine:latest...
# Getting image source signatures
# Copying blob sha256:9d16cba9fb961d1aafec9542f2bf7cb64acfc55245f9e4eb5abecd4cdc38d749
# timeout: sending signal TERM to command ‘podman’
# [13:46:38.484258602] [ rc=124 (** EXPECTED 0 **) ]
# *** TIMED OUT ***
#
# [13:46:38.499217824] # podman  rm -t 0 -a -f
#
# [13:46:38.546122560] # podman  rmi -f b-t96-aecwt3ct
#
# [13:46:38.579566655] # podman  image prune -f
# # [teardown]

another I've seen is an auto-update test (from 255-auto-update.bats) getting a 504 response from quay.io and retrying; the retry worked, but the test was considered 'failed' because there was a warning logged for the 504 and the test fails on unexpected warnings.

This started around noon UTC on 2026-01-20. Before that failures were very rare, now 80-90% of attempts are failing.

I've rebooted the entire prod instance (server and workers) but that doesn't seem to be helping.

For now I've set the test to auto-retry 5 times, but that's only a mitigation and might hide genuine bugs. It'd be good to figure out what's going on here but I'm really not sure.

@lsm5 any ideas? Do you happen to know who we could ask about this on the quay.io end?

Paul Holzinger suggested using a caching proxy to mitigate this, that's probably a good idea anyway to reduce unnecessary repeated traffic to quay.io for every run of this test. We already implemented one for Flathub, so we could more or less copy that for this.

Metadata Update from @adamwill:
- Custom field story_points adjusted to 5

Sorry I was out yesterday. Paul's suggestion would be the way to go.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/issues/481

Please continue any further discussion there.
Metadata

bug

None
None
5
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.