Thanks for working on this, panu.

The Sequoia backend should eventually get updated to allow this as long as the system policy does

Version 1.1.0 or later of rpm-sequoia already respects the system policy. An alternate policy can be specified using the SEQUOIA_CRYPTO_POLICY environment variable, as described in the README.

The current version of fedora crypto policies does not include a sequoia policy, but there is an MR to add it. I believe they are waiting for more feedback before merging it. Note: the proposed Fedora 38 policy disables SHA-1.

On a related note, https://src.fedoraproject.org/rpms/rpm/pull-request/29 also points out that testing-farm is using an EOL version of ansible, so maybe this is partially related to https://pagure.io/fedora-ci/general/issue/330

Note: a possible workaround to make some progress would be to use a crypto policy that allows SHA-1, which, as I mention above, can be set via an environment variable.

Version 1.1.0 or later of rpm-sequoia already respects the system policy.

I know, but that's not in rawhide yet. That's what I meant with "updated eventually".

Right, AIUI, that is blocked on rust-sequoia-policy-config being reviewed.

Okay, seems since yesterday somebody updated (copied?) the koji package in https://kojipkgs.fedoraproject.org/repos/rawhide/ to current rawhide build, which is unsigned, making the issue go away. Thanks to whoever it was.

That an unsigned package is better than a (relatively) weakly signed one is cuckoo, but that's another topic...

That an unsigned package is better than a (relatively) weakly signed one is cuckoo, but that's another topic...

I agree that this is "cuckoo", and that it's another topic. Is there someplace where that other topic is being discussed?

Okay, seems since yesterday somebody updated (copied?) the koji package in https://kojipkgs.fedoraproject.org/repos/rawhide/ to current rawhide build, which is unsigned, making the issue go away. Thanks to whoever it was.

https://kojipkgs.fedoraproject.org/repos/rawhide/ is the koji buildroot repo. Nothing in it is signed in any way ever.

builds are built, then pushed to rawhide after CI runs. The last koji package was https://bodhi.fedoraproject.org/updates/FEDORA-2022-55a2ce55f7 which was stable 2 weeks ago.

Nothing was done yesterday to the koji package that I know of.

Where is ci downloading from? Whats testing-farm-tag-repository ?

Where is ci downloading from? Whats testing-farm-tag-repository ?

[testing-farm-tag-repository]
name=Tag repository for rawhide
baseurl=https://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/
gpgcheck=0
enabled=1
priority=9
skip_if_unavailable=True
# See TFT-847 - glibc32*
# See TFT-1284 - beakerlib
exclude=glibc32* beakerlib

(i.e. the rawhide buildroot repo)

so yeah, those should never end up being signed.

Okay, seems since yesterday somebody updated (copied?) the koji package in https://kojipkgs.fedoraproject.org/repos/rawhide/ to current rawhide build, which is unsigned, making the issue go away. Thanks to whoever it was.

https://kojipkgs.fedoraproject.org/repos/rawhide/ is the koji buildroot repo. Nothing in it is signed in any way ever.

Wherever it may have come from, the previous koji package in there certainly was signed, see
https://artifacts.dev.testing-farm.io/a1c1cc94-0f2b-41f0-a0c2-2456dd50359a/ , in particular:

   "stderr": "error: rpmdbNextIterator: skipping h#     495 \nHeader RSA signature: BAD (header tag 268: invalid OpenPGP signature)\nHeader SHA256 digest: OK\nHeader SHA1 digest: OK\nerror: rpmdbNextIterator: skipping h#     495 \nHeader RSA signature: BAD (header tag 268: invalid OpenPGP signature)\nHeader SHA256 digest: OK\nHeader SHA1 digest: OK\nError: An rpm exception occurred: package not installed",
    "stderr_lines": [
        "error: rpmdbNextIterator: skipping h#     495 ",
        "Header RSA signature: BAD (header tag 268: invalid OpenPGP signature)",
        "Header SHA256 digest: OK",
        "Header SHA1 digest: OK",
        "error: rpmdbNextIterator: skipping h#     495 ",
        "Header RSA signature: BAD (header tag 268: invalid OpenPGP signature)",
        "Header SHA256 digest: OK",
        "Header SHA1 digest: OK",
        "Error: An rpm exception occurred: package not installed"
    ],

Those two numbers map to the following, previously installed packages (so the thing is failing trying to update these older packages to the current, unsigned version, a subtlety that I kinda missed yesterday in the excitement of it suddenly working):

494 python3-koji-1.28.1-1.fc38.noarch (not an OpenPGP signature)
495 koji-1.28.1-1.fc38.noarch (not an OpenPGP signature)

builds are built, then pushed to rawhide after CI runs. The last koji package was https://bodhi.fedoraproject.org/updates/FEDORA-2022-55a2ce55f7 which was stable 2 weeks ago.

Nothing was done yesterday to the koji package that I know of.

Right - see above, what actually changed is the base install on that host. Not the koji package in the repo. Apologies for the confusion.

Sorry, this was a workaround we applied due to a bug we hit and was blocking all testing.

https://gitlab.com/testing-farm/infrastructure/-/merge_requests/90/diffs

We should have reverted it sooner.
We apologize for the trouble this caused.
Back in that time, seemed like a quick workaround that would solve our problem we had with koji.

We will be more transparent about such workarounds in the future.
Actually, the whole setup we currently own in the repo will be in the future more easily discoverable and contributable by the community.

Oh, we all know what it's like: you do a temporary workaround to put out a fire and then while putting out the next five fires, you forget all about the first one. No worries from my side.
Just good to hear there was a perfectly rational explanation behind this "mystery package" which makes you wonder if there's a security breach, and a big +1 for making it more transparent.

Ah ha. That explains it.. Thanks for the info!