fedora-ci / general

#538 fedora-ci.koji-build.installability.functional update test fails for dnf5 with libdnf5-plugin-local: Signature verification failed
Opened by ppisar. Modified

Open
Close Issue

fedora-ci.koji-build.installability.functional update/downgrade/remove tests fail for dnf5-5.3.0.0-7.fc44 because of unsigned packages in _local_dnf repository https://artifacts.dev.testing-farm.io/99345f41-486f-4f14-a3d7-c1230db0ef3c/work-installabilityf367iu22/installability/execute/data/guest/default-0/installability-2/data/viewer.html#

 ---------------------------------------------------------
|                                                         |
|   Running update test for dnf5-0:5.3.0.0-7.fc44.x86_64  |
|                                                         |
---------------------------------------------------------
(process:8883): librhsm-WARNING **: 12:41:10.829: Found 0 entitlement certificates
(process:8883): librhsm-WARNING **: 12:41:10.829: Found 0 product certificates
Updating and loading repositories:
 Tag repository for rawhide             100% |  28.9 KiB/s |   3.5 KiB |  00m00s
 Local libdnf5 plugin repo              100% |  12.6 KiB/s |   1.5 KiB |  00m00s
 fedora-44 Buildroot                    100% |  28.9 KiB/s |   3.5 KiB |  00m00s
 Copr repo for mini-tps owned by @osci  100% |  15.0 KiB/s |   1.8 KiB |  00m00s
 Repo for 141342397 Brew build          100% |  43.1 KiB/s |   3.0 KiB |  00m00s
Repositories loaded.
Package                                            Arch   Version          Repository                       Size
Upgrading:
 dnf5                                              x86_64 0:5.3.0.0-7.fc44 brew-141342397                2.9 MiB
   replacing dnf5                                  x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository   2.9 MiB
 dnf5-plugin-automatic                             x86_64 0:5.3.0.0-7.fc44 brew-141342397              186.5 KiB
   replacing dnf5-plugin-automatic                 x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository 174.5 KiB
 dnf5-plugin-manifest                              x86_64 0:5.3.0.0-7.fc44 brew-141342397              313.3 KiB
   replacing dnf5-plugin-manifest                  x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository 394.9 KiB
 dnf5-plugins                                      x86_64 0:5.3.0.0-7.fc44 brew-141342397                1.4 MiB
   replacing dnf5-plugins                          x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository   1.4 MiB
 dnf5daemon-server                                 x86_64 0:5.3.0.0-7.fc44 brew-141342397              753.2 KiB
   replacing dnf5daemon-server                     x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository 745.5 KiB
 dnf5daemon-server-polkit                          noarch 0:5.3.0.0-7.fc44 brew-141342397              326.0   B
   replacing dnf5daemon-server-polkit              noarch 0:5.3.0.0-4.fc44 testing-farm-tag-repository 326.0   B
 libdnf5                                           x86_64 0:5.3.0.0-7.fc44 brew-141342397                3.9 MiB
   replacing libdnf5                               x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository   3.9 MiB
 libdnf5-cli                                       x86_64 0:5.3.0.0-7.fc44 brew-141342397              983.3 KiB
   replacing libdnf5-cli                           x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository 955.1 KiB
 libdnf5-cli-devel                                 x86_64 0:5.3.0.0-7.fc44 brew-141342397              195.8 KiB
   replacing libdnf5-cli-devel                     x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository 195.8 KiB
 libdnf5-devel                                     x86_64 0:5.3.0.0-7.fc44 brew-141342397              709.3 KiB
   replacing libdnf5-devel                         x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository 709.3 KiB
 libdnf5-plugin-actions                            x86_64 0:5.3.0.0-7.fc44 brew-141342397              286.5 KiB
   replacing libdnf5-plugin-actions                x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository 274.5 KiB
 libdnf5-plugin-appstream                          x86_64 0:5.3.0.0-7.fc44 brew-141342397               32.1 KiB
   replacing libdnf5-plugin-appstream              x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository  32.1 KiB
 libdnf5-plugin-expired-pgp-keys                   x86_64 0:5.3.0.0-7.fc44 brew-141342397               86.3 KiB
   replacing libdnf5-plugin-expired-pgp-keys       x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository  82.4 KiB
 libdnf5-plugin-local                              x86_64 0:5.3.0.0-7.fc44 brew-141342397               62.4 KiB
   replacing libdnf5-plugin-local                  x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository  58.4 KiB
 libdnf5-plugin-rhsm                               x86_64 0:5.3.0.0-7.fc44 _dnf_local                   42.6 KiB
   replacing libdnf5-plugin-rhsm                   x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository  42.6 KiB
 perl-libdnf5                                      x86_64 0:5.3.0.0-7.fc44 _dnf_local                    8.2 MiB
   replacing perl-libdnf5                          x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository   8.2 MiB
 perl-libdnf5-cli                                  x86_64 0:5.3.0.0-7.fc44 _dnf_local                  127.3 KiB
   replacing perl-libdnf5-cli                      x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository 127.3 KiB
 python3-libdnf5                                   x86_64 0:5.3.0.0-7.fc44 brew-141342397                9.6 MiB
   replacing python3-libdnf5                       x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository   9.3 MiB
 python3-libdnf5-cli                               x86_64 0:5.3.0.0-7.fc44 _dnf_local                  133.0 KiB
   replacing python3-libdnf5-cli                   x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository 133.0 KiB
 python3-libdnf5-python-plugins-loader             x86_64 0:5.3.0.0-7.fc44 _dnf_local                   80.9 KiB
   replacing python3-libdnf5-python-plugins-loader x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository  72.9 KiB
 ruby-libdnf5                                      x86_64 0:5.3.0.0-7.fc44 _dnf_local                    7.0 MiB
   replacing ruby-libdnf5                          x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository   6.9 MiB
 ruby-libdnf5-cli                                  x86_64 0:5.3.0.0-7.fc44 _dnf_local                   93.9 KiB
   replacing ruby-libdnf5-cli                      x86_64 0:5.3.0.0-4.fc44 testing-farm-tag-repository  89.9 KiB
Transaction Summary:
 Upgrading:         22 packages
 Replacing:         22 packages
Total size of inbound packages is 10 MiB. Need to download 10 MiB.
After this operation, 670 KiB extra will be used (install 37 MiB, remove 36 MiB).
[ 1/22] dnf5-0:5.3.0.0-7.fc44.x86_64    100% | 102.8 MiB/s | 947.5 KiB |  00m00s
[ 2/22] libdnf5-0:5.3.0.0-7.fc44.x86_64 100% |  79.6 MiB/s |   1.3 MiB |  00m00s
[ 3/22] libdnf5-cli-0:5.3.0.0-7.fc44.x8 100% |  17.1 MiB/s | 368.8 KiB |  00m00s
[ 4/22] dnf5-plugins-0:5.3.0.0-7.fc44.x 100% |  68.2 MiB/s | 489.0 KiB |  00m00s
[ 5/22] dnf5-plugin-manifest-0:5.3.0.0- 100% |  14.9 MiB/s | 167.4 KiB |  00m00s
[ 6/22] dnf5-plugin-automatic-0:5.3.0.0 100% |   7.9 MiB/s | 130.1 KiB |  00m00s
[ 7/22] python3-libdnf5-0:5.3.0.0-7.fc4 100% | 175.4 MiB/s |   1.8 MiB |  00m00s
[ 8/22] libdnf5-plugin-local-0:5.3.0.0- 100% |   6.5 MiB/s |  79.5 KiB |  00m00s
[ 9/22] libdnf5-plugin-expired-pgp-keys 100% |   5.3 MiB/s |  92.0 KiB |  00m00s
[10/22] libdnf5-plugin-appstream-0:5.3. 100% |  16.2 MiB/s |  66.4 KiB |  00m00s
[11/22] libdnf5-plugin-actions-0:5.3.0. 100% |  17.9 MiB/s | 165.3 KiB |  00m00s
[12/22] libdnf5-devel-0:5.3.0.0-7.fc44. 100% |  11.5 MiB/s | 164.2 KiB |  00m00s
[13/22] dnf5daemon-server-0:5.3.0.0-7.f 100% |  50.0 MiB/s | 307.4 KiB |  00m00s
[14/22] libdnf5-cli-devel-0:5.3.0.0-7.f 100% |   8.7 MiB/s |  89.6 KiB |  00m00s
[15/22] dnf5daemon-server-polkit-0:5.3. 100% |   4.0 MiB/s |  53.4 KiB |  00m00s
[16/22] libdnf5-plugin-rhsm-0:5.3.0.0-7 100% |  11.5 MiB/s |  70.6 KiB |  00m00s
[17/22] perl-libdnf5-0:5.3.0.0-7.fc44.x 100% | 121.7 MiB/s |   1.6 MiB |  00m00s
[18/22] perl-libdnf5-cli-0:5.3.0.0-7.fc 100% |   5.6 MiB/s |  91.6 KiB |  00m00s
[19/22] python3-libdnf5-cli-0:5.3.0.0-7 100% |  15.9 MiB/s |  97.8 KiB |  00m00s
[20/22] python3-libdnf5-python-plugins- 100% |   8.4 MiB/s |  86.3 KiB |  00m00s
[21/22] ruby-libdnf5-0:5.3.0.0-7.fc44.x 100% |  86.7 MiB/s |   1.6 MiB |  00m00s
[22/22] ruby-libdnf5-cli-0:5.3.0.0-7.fc 100% |  20.6 MiB/s |  84.4 KiB |  00m00s
--------------------------------------------------------------------------------
[22/22] Total                           100% |  70.9 MiB/s |   9.6 MiB |  00m00s
Running transaction
Transaction failed: Signature verification failed.
OpenPGP check for package "libdnf5-plugin-rhsm-5.3.0.0-7.fc44.x86_64" (/var/cache/libdnf5/_dnf_local-71c913707df56d1b/packages/libdnf5-plugin-rhsm-5.3.0.0-7.fc44.x86_64.rpm) from repo "_dnf_local" has failed: The package is not signed.
Warning: skipped OpenPGP checks for 15 packages from repository: brew-141342397

What is _dnf_local repository? Shouldn't the test only use testing-farm-tag-repository and brew-141342397 repositories? Why the packages in the repository are not signed, or the repository is configured to require signatures?

My conjecture is that _dnf_local contains a package manager used to perform the test, which coincides when a subject of the test is dnf5 component. And that packages are not signed because only very recently RPM tried to switch on mandatory signing. (Though I believe RPM team has not yet enabled the mandatory signing).

What is _dnf_local repository? Shouldn't the test only use testing-farm-tag-repository and brew-141342397 repositories? Why the packages in the repository are not signed, or the repository is configured to require signatures?

This is a repository created on demand from the koji build (not actually brew, that's just the name format defined in the script). This is hidden underneath mtps-get-task --createrepo command (called here). I do not think it would ever get signing unless we sign it with a locally generated and trusted key? Is it not possible to disable signing requirement for these ephemeral repos? Also who is the point of contact for that change proposal? It will affect all tmt and testing-farm jobs also.

What is _dnf_local repository? Shouldn't the test only use testing-farm-tag-repository and brew-141342397 repositories? Why the packages in the repository are not signed, or the repository is configured to require signatures?

This is a repository created on demand from the koji build (not actually brew, that's just the name format defined in the script). This is hidden underneath mtps-get-task --createrepo command (called here).

I think you've just described brew-141342397 repository. But the problem is with _dnf_local repository.

I do not think it would ever get signing unless we sign it with a locally generated and trusted key? Is it not possible to disable signing requirement for these ephemeral repos?

Setting pkg_gpgcheck or gpgcheck option to 0 in a configuration file for the repository. That should be enough. I can see that mtps-get-task already does that.

Is _dnf_local repository also created with it? The failure indicates it isn't.

Also who is the point of contact for that change proposal? It will affect all tmt and testing-farm jobs also.

The https://fedoraproject.org/wiki/Changes/Enforcing_signature_checking_by_default change is driven by @pmatilai. I believe he postponed the change for Fedora 45 https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/DECC3OHO5LIP3HWNNZIAC6KWP4QKFHES/. Nevertheless, the change is not supposed to override how a repository is configured.

I think you've just described brew-141342397 repository. But the problem is with _dnf_local repository.

I don't know. It is either something internal to dnf5 or a side-effect of create_repo. For the former checkup with the current change of dnf5 if something fishy is happening there (although it should still be using the older dnf5 repo iiuc). For the latter, it's the same mini-tps repo that needs to adapt.

On https://github.com/fedora-ci/installability-pipeline/ there is an example of how to reproduce the test, could you try to run the following from the git checkout of that repo:

$ tmt run -a \
  -e TASK_ID=141342397 \
  -e PROFILE_NAME=fedora-44 \
  provision --how virtual --image fedora-rawhide \
  login

This should re-run the test similar to how it is done in that test and also put you in an ssh connection to debug what is going on. I will also try to do that and see if I can find anything obvious, otherwise could use more eyes on it.

@ppisar I confirmed that _dnf_local is created by dnf5 itself (after installing libdnf5-plugin-local). Could you ping the people involved to take a close look? See the following log

 --------------------------------------------------------------------------
|                                                                          |
|   Running install test for libdnf5-plugin-local-0:5.3.0.0-7.fc44.x86_64  |
|                                                                          |
--------------------------------------------------------------------------
Updating and loading repositories:
Repositories loaded.
Package               Arch   Version          Repository          Size
Installing:
 libdnf5-plugin-local x86_64 0:5.3.0.0-7.fc44 brew-141342397  62.4 KiB
Transaction Summary:
 Installing:         1 package
Total size of inbound packages is 79 KiB. Need to download 79 KiB.
After this operation, 62 KiB extra will be used (install 62 KiB, remove 0 B).
[1/1] libdnf5-plugin-local-0:5.3.0.0-7. 100% |  25.9 MiB/s |  79.5 KiB |  00m00s
--------------------------------------------------------------------------------
[1/1] Total                             100% |  19.4 MiB/s |  79.5 KiB |  00m00s
Running transaction
[1/3] Verify package files              100% | 500.0   B/s |   1.0   B |  00m00s
[2/3] Prepare transaction               100% |  19.0   B/s |   1.0   B |  00m00s
[3/3] Installing libdnf5-plugin-local-0 100% | 417.3 KiB/s |  63.4 KiB |  00m00s
Warning: skipped OpenPGP checks for 1 package from repository: brew-141342397
Complete!
 -----------------------------------------------------------------------
|                                                                       |
| TEST                                                                  |
| ====                                                                  |
|                                                                       |
|   TYPE:        install                                                |
|   NEVRA:       libdnf5-plugin-local-debuginfo-0:5.3.0.0-7.fc44.x86_64 |
|   SELINUX:     Enforcing                                              |
|   dnf5 HISTORY:                                                       |
|                                                                       |
-----------------------------------------------------------------------
 ------------------------------------------------------------------------------------
|                                                                                    |
|   Preparing for installing libdnf5-plugin-local-debuginfo-0:5.3.0.0-7.fc44.x86_64  |
|                                                                                    |
------------------------------------------------------------------------------------
 ------------------------------------------------------------------------------------
|                                                                                    |
|   Running install test for libdnf5-plugin-local-debuginfo-0:5.3.0.0-7.fc44.x86_64  |
|                                                                                    |
------------------------------------------------------------------------------------
Updating and loading repositories:
 Tag repository for rawhide             100% |  25.7 KiB/s |   3.5 KiB |  00m00s
 fedora-44 Buildroot                    100% |  25.7 KiB/s |   3.5 KiB |  00m00s
 Copr repo for mini-tps owned by @osci  100% |  13.3 KiB/s |   1.8 KiB |  00m00s
 Repo for 141342397 Brew build          100% |  29.0 KiB/s |   3.0 KiB |  00m00s
 Local libdnf5 plugin repo              ???% |   0.0   B/s |  -1.0   B |  ?     
>>> Curl error (37): Could not read a file:// file for file:///var/lib/dnf/plugins/local/repodata/repomd.xml [Could not open file /var/lib/dnf/plugins/local/repodata/repomd.xml] - file:///var/lib/dnf/plugins/local/repodata/repomd.xml
>>> Usable URL not found

At no point has the rpm enforcing signature change been active in Fedora, not even briefly. This is something else.

@lecris, thanks for finding the cause. The libdnf5-plugin-local subpackage exists since dnf5-5.3.0.0-1.fc44. I incidentally waived it for dnf5-5.3.0.0-3.fc44 because that build also touched the signature verification. So this failure is not a new bug.

libdnf5-plugin-local documents that:

Note that the repository has pkg_gpgcheck verification enabled by default but doesn't specify any gpgkey, it assumes all required keys were already imported.

I opened a DNF5 upstream ticket to make the signature check configurable https://github.com/rpm-software-management/dnf5/issues/2580. Though it won't fix this Fedora CI issue.

I propose disabling the _dnf_local repository in this CI. At the end we want packages to be installed from Koji, not from some cache. The hard-coded repository name is documented as _dnf_local:

After each libdnf5 transaction copy all downloaded packages to a _dnf_local repository on the local filesystem and generate repository metadata.

Do you think it would be possible to change this installability-pipeline to always execute dnf or dnf5 command with --disablerepo _dnf_local option?

Metadata Update from @ppisar:
- Issue status updated to: Closed (was: Open)

Metadata Update from @ppisar:
- Issue status updated to: Open (was: Closed)

Do you think it would be possible to change this installability-pipeline to always execute dnf or dnf5 command with --disablerepo _dnf_local option?

I would like to not do that, this seems like a genuine issue that would affect users and it should be fixed on the dnf side.

What I am not certain, is this opt-in only after installing libdnf5-plugin-local or does this occur regardless. If the latter then something really must be done here, before it gets to the users.

This happens only if libdnf5-plugin-local is installed. That it is automatically enabled is a feature. That is requires signatures by default is a security feature.

Another option could be using dnf --installroot option to perform the testing installation/upgrades/downgrades isolated from the host system. But it would make the test less alike to real installations (e.g. no /boot).

That is requires signatures by default is a security feature.

Iiuc the issues is that it enabled the gpg check for that repo specifically even though it doesn't have signed rpms. It's more an implementation detail that they need to figure out.

Another option could be using dnf --installroot option to perform the testing installation/upgrades/downgrades isolated from the host system. But it would make the test less alike to real installations (e.g. no /boot).

mini-tps is in charge of the details of how the tests installation are run, and personally I wish to get rid of it as soon as possible. If the builds need to be signed, I think we could handle it on the specific test implementations, and more globally on tmt side if needed. But we will wait for some more feedback on the CP to see what exactly should we do.

DNF team discussed this issue and stated that some plugins change a system behavior and and it's not reasonable to expect that installing all of them won't break some use cases, e.g. this one.

The team recommends the CI infrastructure to inhibit the effect of the the local plugin by disabling the plugin at run-time:

That can be done either by adding --disable-plugin=local argument to dnf5 invocation, or more broadly with --no-plugins option.

The --no-plugins option can also be specified as plugins=0 line in a /etc/dnf/dnf.conf configuration file.

The local plugin can also be disabled by creating/overwriting its /etc/dnf/libdnf5-plugins/local.conf configuration with this content:

[main]
name = local
enabled = false

DNF team will try to improve the plugin to cover this CI use case automatically, but until that the team would welcome applying one of the workarounds in Fedora CI.
Metadata
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.