This is caused by a koji regression: https://bugzilla.redhat.com/show_bug.cgi?id=2118011

The regression has a fix in testing, could you please pull from there?

@fbo @tdecacqu hi, I believe this needs in the image Zuul has? I would love to learn how to fix it myself

@mvadkert hi, that's great to hear. So looking at the job output, you can get it's image label from the emit-job-header : Print node information task, which prints zuul-worker-f36.

Then you can check in the main config project (https://softwarefactory-project.io/cgit/config/) what is this label refer to, and you'll find: https://softwarefactory-project.io/cgit/config/tree/nodepool/static_config/zs.yaml#n264

Looking at the image, it does not seems like it contains koji: podman run -it --rm quay.io/software-factory/zuul-worker-f36 rpm -qa | grep -i koji

Thus there may be something in the job task that install koji, and the task could be adjusted to ensure it pick/pin the right version.

It seems like koji is installed as a dependency of fedpkg here: https://pagure.io/zuul-distro-jobs/blob/master/f/roles/fedpkg-srpm-build/tasks/main.yaml . It's also ensured here: https://pagure.io/zuul-distro-jobs/blob/master/f/roles/fedpkg-build-ng/tasks/main.yaml#_13

Thus what you can do to fix the job is:
- open a PR to adjust one of these tasks.
- open a PR on one of the distgit by using a Depends-On the zuul-distro-jobs PR.

If the second PR works, then we can land the fix until koji is properly updated.

@tdecacqu thanks for the info! I am trying something here:

https://pagure.io/zuul-distro-jobs/pull-request/133

And trying to verify via:

https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/312

@tdecacqu stupid question, how can I find out why Depends-On does not work?

@mvadkert That's a good question. On the status page, you can see the depends-on relation ship ( https://fedora.softwarefactory-project.io/zuul/status ). Otherwise you can check the inventory variables, in the zuul-info/inventory.yaml, the depends-on appears in the zuul.items list.

@tdecacqu thanks, so according to the status the depends on is ok, hmm .. so it must be something else then

@tdecacqu also inventory.yaml looks good afaict. As that job has secrets for kerberos, isn't that it again, cannot test dependencies with project which uses secrets??

Oh I see, my bad, rpm-scratch-build is defined in a config project, as you can see in https://fedora.softwarefactory-project.io/zuul/job/rpm-scratch-build . That means the job doesn't take into account unmerged change (for security reason, otherwise a malicious author could try to print the koji secret).

To test that, we would need to either make the job untrusted, by removing the use of secret. Or the usual strategy is to duplicate the role and create a copy of the job to use the new role.

But since rpm-scratch-build seems to be broken already, i guess we could merge your zuul-distro-job fix.

@tdecacqu let's try it, I believe my change is ok

I suspect this constraint is what led to the creation of fedpkg-build-ng, and for the same reason, we could create fedpkg-build-testing with your change. Then we create a new job in fedora-project-config, e.g. rpm-scratch-build-testing with a new playbook playbooks/koji/build-testing.yaml to use the new role. And finally we can attach this new job to a test repository to make sure it works, and when we are satisfied, we can replace the current job playbook with the new one. That would be the recommended solution to introduce such change safely. But I don't think that is necessary for this issue, and I think we can simply try to hot fix the bug.

It seems like https://pagure.io/zuul-distro-jobs/pull-request/133 fixed this issue. Perhaps we should queue a change to remove the workaround?

I am tracking it downstream, I will submit it in few days ...

Jira TFT-1518

HI I see that last run https://src.fedoraproject.org/rpms/python3.11/pull-request/74 passed the rpm-scratch-build job successfully. It seems the 133 fix did the job. Thanks !
We need to take care of removing the workaround when koji-1.29.1-4.fc36 lands in main stream repo.

HI I see that last run https://src.fedoraproject.org/rpms/python3.11/pull-request/74 passed the rpm-scratch-build job successfully. It seems the 133 fix did the job. Thanks !
We need to take care of removing the workaround when koji-1.29.1-4.fc36 lands in main stream repo.

We need to take care of removing the workaround when koji-1.29.1-4.fc36 lands in main stream repo.

I would wait for koji-1.30 because more fixes are coming, I have it tracked in https://issues.redhat.com/browse/TFT-1518, so will do it once things get moving. If you have a better place, I am fine with that.

Ok +1

I am resolving this, we are still waiting for 1.30 ...

I will be checking it from time to time, it is a blocked item in our kanban