fedora-ci / general

#364 fedora-ci.koji-build.tier0.functional environment for F37 is missing gsl-2.6-7.fc37
Closed by mvadkert. Opened by ppisar.

Closed
Reopen Issue

gsl-2.6-7.fc37 which provides libgsl.so.25()(64bit) in in Fedora 37 since 2022-07-25. Yet a fedora-ci.koji-build.tier0.functional test for unpaper-7.0.0-3.fc37 https://artifacts.dev.testing-farm.io/9263acc7-2562-4c9a-9093-09453c16829f/ reports:

Stage: 14-Install-packages.txt
Command:
---v---v---v---v---v---
dnf --allowerasing -y install $(cat rpms-list)
---^---^---^---^---^---
Stdout:
---v---v---v---v---v---
Last metadata expiration check: 0:00:30 ago on Tue 30 Aug 2022 09:03:44 AM UTC.
(try to add '--skip-broken' to skip uninstallable packages)
---^---^---^---^---^---
Stderr:
---v---v---v---v---v---
Warning: Permanently added '18.191.214.156' (ECDSA) to the list of known hosts.
Error: 
 Problem: conflicting requests
  - nothing provides libgsl.so.25()(64bit) needed by siril-1.0.3-4.fc37.x86_64
---^---^---^---^---^---

Instaling the packages from the update https://bodhi.fedoraproject.org/updates/FEDORA-2022-9a555d4c74 locally works for me. I conclude TMT infrastructure has outdated repositories.

Metadata Update from @mvadkert:
- Issue assigned to mvadkert

Metadata Update from @mvadkert:
- Assignee reset
- Issue tagged with: Testing Farm

it is weird, we are using an image from 4 days ago, Fedora-Cloud-Base-37-20220826.n.0.x86_64-hvm-us-east-2-gp2-0 plus we should be adding koji buildroot to the mix chm

Metadata Update from @mvadkert:
- Issue assigned to mvadkert

hmm, looking at fedora ci channel, there were some repo problems. trying to rerun.

Today lsm5 on fedora-devel IRC channel complained that https://artifacts.dev.testing-farm.io/0628a454-b994-46e1-944e-f962ab8de983/ is missing systemd-251.7-611.fc37.

Today lsm5 on fedora-devel IRC channel complained that https://artifacts.dev.testing-farm.io/0628a454-b994-46e1-944e-f962ab8de983/ is missing systemd-251.7-611.fc37.

occurred today as well https://artifacts.dev.testing-farm.io/29b62da9-ca5c-46b6-97a4-4e830f088860/work-tests.ymlHEwPja/tests-ekSYfH/

also unable to find systemd 251.7 today on conmon gating tests https://artifacts.dev.testing-farm.io/7c5a8555-81e8-4fc3-8bb7-29446604e8f4/

@lsm5 does podman require that systemd version? Obviously it was around in the repos:

We add this repo:

TASK [Check if tag repository https://kojipkgs.fedoraproject.org/repos/f37-build/latest/x86_64/ exists] ***

Which has:

curl https://kojipkgs.fedoraproject.org/repos/f37-build/latest/x86_64/pkglist | grep systemd
...
toplink/packages/systemd/251.7/611.fc37/x86_64/systemd-251.7-611.fc37.x86_64.rpm

But this was not installed:
https://artifacts.dev.testing-farm.io/7c5a8555-81e8-4fc3-8bb7-29446604e8f4/guest-setup-83833097-2a3d-4513-bcb0-569526edc0b3/artifact-installation-83833097-2a3d-4513-bcb0-569526edc0b3/5-Install-packages.txt

@lsm5 looks to me your problem is different.

Anyway, seems the expectation in Fedora is that people have fully updated system? We do this in Rawhide:

https://gitlab.com/testing-farm/infrastructure/-/blob/testing-farm/ranch/public/citool-config/guest-setup/pre-artifact-installation/update-and-restart.yml

Maybe we should do that for all Fedora releases ... the only drawback is it adds to the test execution time around 2-3 minutes.

Once we start updating images daily, this problem will be less visible

So maybe we

@santiago PTAL ^

@mvadkert yes it does need systemd 251.7 and if you look in the gating test logs under Important package versions, it lists 251.6. i wouldn't mind 2-3 minutes for podman and related packages. Or maybe we could do this for each rpm as need arises, so it's not a noticeable burden on all packages?

V=C2=A0Fri, Nov 11, 2022 at 12:52:53PM +0000,=C2=A0Miroslav Vadkerti napsal=
(a):

Anyway, seems the expectation in Fedora is that people have fully updated=
system?

Yes.

We do this in Rawhide:
=20
https://gitlab.com/testing-farm/infrastructure/-/blob/testing-farm/ranch/=
public/citool-config/guest-setup/pre-artifact-installation/update-and-resta=
rt.yml
=20
Maybe we should do that for all Fedora releases ... the only drawback is =
it adds to the test execution time around 2-3 minutes.
=20
Once we start updating images daily, this problem will be less visible

I have no idea how images are updated and cached in CI infrastructure,
however, I believe running "dnf upgrade" does have to mean a 3-minute delay.

Pristine, released Fedora images have empty DNF cache. If you invoke "dnf
upgrade" for the first time, you will spend minutes by downloading 70 MB of
repository metadata. Usually only to recognize that there is no update
available.

So first optimization is taking a pristine released Fedora image, lower
"metadata_expire" YUM repository configuration to dozen of seconds, run
"dnf upgrade", save the changes as a new image layer and then use this layer
for executing the test. A difference between the pristine and the updated
image is the populated DNF cache. Then, as a result, if you run "dnf
upgrade" before each test, "dnf upgrade" will usually quickly recognize that
the repository has not changed (see /repomd/revision number in
https://dl.fedoraproject.org/pub/fedora/linux/updates/37/Everything/x86_64= /repodata/repomd.xml).

Second optimizition is how often to regenerate the updated layer.

IMHO gating environment should resemble end-user system as much as possible.
Hence adding buildroot repositories from Koji is not the right approach. The
environment should use updates-testing Fedora repositories. Fedora
repositories are updated daily. Updating them more frequently won't help
because there simply won't be any new package. Watch e.g.
https://dl.fedoraproject.org/pub/fedora/linux/updates/37/Everything/state
content. It doesn't usually change more often than once in a day. If you
observe Last-Modified HTTP header ("Fri, 11 Nov 2022 01:00:23 GMT" now), you
will notice that it updates in the first hour after UTC midnight. So if you
schedule generating the updatated layer at 2 AM UTC, you will have a good
chance that you do one full update and then no test won't need updating any
packages.

Of course there are people who insist that gating infrastructure needs
to utilize Koji buildroot repositories because that's what they use when
building their packages. I think that answer to those people is "use
side-tags".

A problem with Koji buildroots is they change with every built package. But
not more often than createrepo_c can produce a compose. In reality it's eve=
ry
8, 10, 15 minutes. Here it's difficult to optimize. An approach could be
speculatively starting and dnf-upgrading a container when a build starts in
Koji and then using it for gating that build. Assuming most builds finishes=
in
few minutes, it could amortize the waiting time.

-- Petr

We decided to ditch tag repository, add the updating and work on updating the CI images daily.

Closing in favor of:
https://pagure.io/fedora-ci/general/issue/376

Metadata Update from @mvadkert:
- Issue status updated to: Closed (was: Open)
Metadata

Testing Farm

None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.