fedora-ci / general

#378 When (only) a downgrade satisfies a dependency, CI runs without the tested build of the package
Opened by churchyard. Modified

Open
Close Issue

I've realized in https://src.fedoraproject.org/rpms/python-virtualenv/pull-request/83 that something was wrong and reproduced the issue in https://src.fedoraproject.org/rpms/python-gear/pull-request/56

tl;dr if a pull request:

  • adds Requires: (nonexisting-thing if existing-thing) to a package
  • has existing-thing in the CI dependencies (e.g. STI required_packages:)

The CI will downgrade the tested package to the version from Fedora and happily report the success of testing while not testing the package built from that CI.

You can see in the logs from tests of https://src.fedoraproject.org/rpms/python-gear/pull-request/56 -- python3-gear-0.16.0-1.fc37.noarch was installed, but the PR updated the release to 666.

This is not very common to get but it is extremely dangerous. I tend to trust the CI: when it is green I suppose that the build was tested and it works. But this shows that we cannot trust the CI result at all because in certain circumstances, it is not the build from the PR that is tested.

Metadata Update from @mvadkert:
- Issue tagged with: STI

Some things that come to my mind:

  1. why package repo with highest priority did not help here
  2. investigate package version locking (might not be feasible)
  3. after pipeline run, run again the sanity check verifying the installed packages
  4. check if it would also happen with tmt

why package repo with highest priority did not help here

because the package from the highest priority repository had broken dependencies

investigate package version locking

This one? https://dnf-plugins-core.readthedocs.io/en/latest/versionlock.html

after pipeline run, run again the sanity check verifying the installed packages

+100

check if it would also happen with tmt

Do you know about a simple enough package with tmt I can use to repeat the experiment?
Edited by churchyard

Do you know about a simple enough package with tmt I can use to repeat the experiment?

https://src.fedoraproject.org/rpms/pipenv/pull-request/57

The issue is not specific to STI, tmt has the same flaw.

@churchyard thanks for looking into my comments.

So feels like we are down to 2 improvements:

  1. introduce https://dnf-plugins-core.readthedocs.io/en/latest/versionlock.html, seems this is something anyway we do inside RHEL in the legacy /distrubution/install/brew-build/task

  2. Run verification on pipeline cleanup

The 1. should be easier to do as 2.

We will track this improvement here and also in RH Jira: https://issues.redhat.com/browse/TFT-1790 (accessible only to RH)
Edited by mvadkert

Ok, generic version locking is not something generally accepted as a good solution:

from @ksrot 

IMHO this is not a step in a good direction. I believe TF should not do any assumptions what a tester is going to do in his TP and add obstacles this way. If a tester wants to know details about a tested artifact TF should expose it somehow and if a specific NVR should be protected it should be configured on a TP level (TBH). Could be a tmt internal/pluggable test, such as the proposed SELinux denial check.

I don't know what half of the abbreviations mean, but in my mind, the CI framewrok must ensure the tested thing is the expected thing and fail explicitly if that's impossible. Individual tests should not try to ensure that the tested thing is actually installed.

I think that is not disputed, we should not let the test run if the tested artifact was not installed correctly, and that should be already happening, so I need to look at this again what is happening and why.

Investigating: https://src.fedoraproject.org/rpms/pipenv/pull-request/57

Testing Farm correctly installed the expected package versions:

https://artifacts.dev.testing-farm.io/806d9eff-2860-4209-94ea-bbc40fa7dfb1/guest-setup-9092cc0b-287a-4208-b5ce-70fea47dccb9/artifact-installation-9092cc0b-287a-4208-b5ce-70fea47dccb9/4-Verify-all-packages-installed.txt

But tmt then downgraded it from tag-repository, during the prepare step.

  • we have tag repository with priority 9
  • we have test artifact repository with priority 1

It is currently unclear why dnf still downgrades the package with this setup.

From tmt log:

https://artifacts.dev.testing-farm.io/806d9eff-2860-4209-94ea-bbc40fa7dfb1/work-basic099tt07s/log.txt

17:47:13         Execute command 'export KOJI_TASK_ID=99808429; export TMT_PLAN_DATA=/var/ARTIFACTS/work-basic099tt07s/plans/basic/data; export TMT_TREE=/var/ARTIFACTS/work-basic099tt07s/plans/basic/tree; rpm -q --whatprovides pipenv rpm python3-nose python3 || dnf install -y pipenv rpm python3-nose python3' on guest '3.17.109.179'.
17:47:13         Run command: ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oIdentitiesOnly=yes -p22 -i /etc/citool.d/id_rsa_artemis -S/tmp/tmpakli4_qy root@3.17.109.179 'export KOJI_TASK_ID=99808429; export TMT_PLAN_DATA=/var/ARTIFACTS/work-basic099tt07s/plans/basic/data; export TMT_TREE=/var/ARTIFACTS/work-basic099tt07s/plans/basic/tree; rpm -q --whatprovides pipenv rpm python3-nose python3 || dnf install -y pipenv rpm python3-nose python3'
17:47:13             cmd: rpm -q --whatprovides pipenv rpm python3-nose python3 || dnf install -y pipenv rpm python3-nose python3
17:47:13         environment: None
17:47:13             out: pipenv-2022.10.25-666.fc39.noarch
17:47:13             out: rpm-4.18.1-1.fc39.x86_64
17:47:13             out: no package provides python3-nose
17:47:13             out: python3-3.11.3-1.fc39.x86_64
17:47:14             out: Last metadata expiration check: 0:00:07 ago on Tue 11 Apr 2023 05:47:06 PM UTC.
17:47:14             out: Package pipenv-2022.10.25-666.fc39.noarch is already installed.
17:47:14             out: Package rpm-4.18.1-1.fc39.x86_64 is already installed.
17:47:14             out: Package python3-3.11.3-1.fc39.x86_64 is already installed.
17:47:14             out: Dependencies resolved.
17:47:14             err: 
17:47:14             err:  Problem: package pipenv-2022.10.25-666.fc39.noarch requires (xxxxxx if python3-nose), but none of the providers can be installed
17:47:14             err:   - cannot install the best candidate for the job
17:47:14             out: ================================================================================
17:47:14             out:  Package       Arch    Version               Repository                    Size
17:47:14             out: ================================================================================
17:47:14             out: Installing:
17:47:14             out:  python3-nose  noarch  1.3.7-40.fc38         testing-farm-tag-repository  327 k
17:47:14             out: Downgrading:
17:47:14             out:  pipenv        noarch  2022.10.25-4.fc39     testing-farm-tag-repository  4.7 M
17:47:14             out: 
17:47:14             out: Transaction Summary
17:47:14             out: ================================================================================
17:47:14             out: Install    1 Package
17:47:14             out: Downgrade  1 Package
17:47:14             out: 
17:47:14             out: Total download size: 5.0 M
17:47:14             out: Downloading Packages:
17:47:14             out: (1/2): python3-nose-1.3.7-40.fc38.noarch.rpm    1.6 MB/s | 327 kB     00:00    
17:47:14             out: (2/2): pipenv-2022.10.25-4.fc39.noarch.rpm       22 MB/s | 4.7 MB     00:00    
17:47:14             out: --------------------------------------------------------------------------------
17:47:14             out: Total                                            23 MB/s | 5.0 MB     00:00     
17:47:14             out: Running transaction check
17:47:14             out: Transaction check succeeded.
17:47:14             out: Running transaction test
17:47:14             out: Transaction test succeeded.
17:47:14             out: Running transaction
17:47:14             out:   Preparing        :                                                        1/1 
17:47:15             out:   Installing       : python3-nose-1.3.7-40.fc38.noarch                      1/3 
17:47:15             out:   Downgrading      : pipenv-2022.10.25-4.fc39.noarch                        2/3 
17:47:15             out:   Cleanup          : pipenv-2022.10.25-666.fc39.noarch                      3/3 
17:47:16             out:   Running scriptlet: pipenv-2022.10.25-666.fc39.noarch                      3/3 
17:47:16             out:   Verifying        : pipenv-2022.10.25-4.fc39.noarch                        1/3 
17:47:16             out:   Verifying        : pipenv-2022.10.25-666.fc39.noarch                      2/3 
17:47:16             out:   Verifying        : python3-nose-1.3.7-40.fc38.noarch                      3/3 
17:47:16             out: 
17:47:16             out: Downgraded:
17:47:16             out:   pipenv-2022.10.25-4.fc39.noarch                                               
17:47:16             out: Installed:
17:47:16             out:   python3-nose-1.3.7-40.fc38.noarch                                             
17:47:16             out: 
17:47:16             out: Complete!
17:47:16         Command event: 2.696 waiting for stream readers
17:47:16         Command event: 2.696 stdout reader done
17:47:16         Command event: 2.696 stderr reader done

More investigation is needed

Looking closer at the log:

17:47:14             err: 
17:47:14             err:  Problem: package pipenv-2022.10.25-666.fc39.noarch requires (xxxxxx if python3-nose), but none of the providers can be installed
17:47:14             err:   - cannot install the best candidate for the job

I guess tmt should not continue in this case?

It is planned that tmt will take over the responsiblity from Testing Farm on this, so I believe that will ultimately resolve problems like these, but it would be good to find a workaround until that happens.

https://issues.redhat.com/browse/TT-224
Metadata
None

STI

None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.