Yes, this is possible. We could as you are suggesting configure the CI to run rpm-scratch-build on a container version that match the target branch to ensure RPM macro and features are up to date.

However we don't have a rawhide zuul-worker container (we have a worker as VM) as of now and I'm wondering if this is a safe thing to do for rawhide branch in the way that that could bring instability to the rawhide CI testing. We could also use the role https://pagure.io/zuul-distro-jobs/blob/master/f/roles/mock-srpm-build but that would require that this job run on a VM.

Do you thing that migrating to f35 container instead of f34 might solve your issue ?

@churchyard any thoughts on this ?

I'm wondering if this is a safe thing to do for rawhide branch in the way that that could bring instability to the rawhide CI testing.

I see. Your concern makes sense.

Do you thing that migrating to f35 container instead of f34 might solve your issue ?

Yes, the f35 container fixes the issue in our case, as RPM version in f35 is 4.17. The issue happens on RPM < 4.17.
https://src.fedoraproject.org/projects/rpms/rpm

Note after migrating to f35 container, the rpmlint version 1 on f34 will be changed to the version 2 on f35. https://src.fedoraproject.org/rpms/rpmlint

The current command syntax is changed, and the result by the rpmlint is changed.

rpmlint version 1

$ rpmlint  --file ruby.rpmlintrc  ./*.spec ./*.rpm

rpmlint version 2

$ rpmlint  -r ruby.rpmlintrc  ./*.spec ./*.rpm

I'm wondering if this is a safe thing to do for rawhide branch in the way that that could bring instability to the rawhide CI testing.

I got an idea. I think ideally we want to run the rawhide container in the case of the PR to the rawhide branch. But it's not clear how much the rawhide environment is unstable to run the checks on Zuul CI.

My suggestion is to create an option for users to specify a container version to run in Zuul CI by themselves by setting the following syntax on a PR's first comment.

The case of using Fedora rawhide container.

Target: rawhide

The case of using Fedora f35 container.

Target: f35

It's like the following syntax.

Depends-on: URL

https://pagure.io/zuul-distro-jobs/pull-request/127#comment-167669

I'm wondering if this is a safe thing to do for rawhide branch in the way that that could bring instability to the rawhide CI testing.

I see. Your concern makes sense.

Do you thing that migrating to f35 container instead of f34 might solve your issue ?

Yes, the f35 container fixes the issue in our case, as RPM version in f35 is 4.17. The issue happens on RPM < 4.17.
https://src.fedoraproject.org/projects/rpms/rpm

Ok then I'll prepare a change to use the f35 contrainer for that job.

Note after migrating to f35 container, the rpmlint version 1 on f34 will be changed to the version 2 on f35. https://src.fedoraproject.org/rpms/rpmlint

The current command syntax is changed, and the result by the rpmlint is changed.

rpmlint version 1

$ rpmlint --file ruby.rpmlintrc ./*.spec ./*.rpm

rpmlint version 2

$ rpmlint -r ruby.rpmlintrc ./*.spec ./*.rpm

Thanks for the info that will be useful when bumping rpmlint container. However I won't change the container to run rpmlint now as no bug is reported. However the bump of rpmlint to benefit version2 might be great soon, would you like to open a separate issue for this ?

I'm wondering if this is a safe thing to do for rawhide branch in the way that that could bring instability to the rawhide CI testing.

I got an idea. I think ideally we want to run the rawhide container in the case of the PR to the rawhide branch. But it's not clear how much the rawhide environment is unstable to run the checks on Zuul CI.

My suggestion is to create an option for users to specify a container version to run in Zuul CI by themselves by setting the following syntax on a PR's first comment.

The case of using Fedora rawhide container.

Target: rawhide

The case of using Fedora f35 container.

Target: f35

It's like the following syntax.

Depends-on: URL
https://pagure.io/zuul-distro-jobs/pull-request/127#comment-167669

In Zuul, the test node selection and preparation is defined in the CI config and cannot be selected/changed based on the commit message.
The "Depends-on" is really the only setting that Zuul extracts from commit messages.

I'm wondering if this is a safe thing to do for rawhide branch in the way that that could bring instability to the rawhide CI testing.

I see. Your concern makes sense.

Do you thing that migrating to f35 container instead of f34 might solve your issue ?

Yes, the f35 container fixes the issue in our case, as RPM version in f35 is 4.17. The issue happens on RPM < 4.17.
https://src.fedoraproject.org/projects/rpms/rpm

Ok then I'll prepare a change to use the f35 contrainer for that job.

Here is the change https://pagure.io/fedora-zuul-jobs-config/pull-request/153 as this cannot be validated speculatively I'll merge when there no changes in the queue and revert if that cause any trouble.

I'd appreciate is long term strategy, not just one time fix. IOW I'd love if the goal is e.g.:

Zuul always uses an appropriate environment for test.

While the non goal is:

We have updated our container from F34 to F35 which fixes your current issues.

Bonus points would be if Zuul:

1) can sync/lock the host updates to the same versions as Fedora infrastructure uses to host Koji.
2) Use Mock to setup the test environment.

The outcome would be that as long as Koji works, Zuul should work as well.

I'm wondering if this is a safe thing to do for rawhide branch in the way that that could bring instability to the rawhide CI testing.

I see. Your concern makes sense.

Do you thing that migrating to f35 container instead of f34 might solve your issue ?

Yes, the f35 container fixes the issue in our case, as RPM version in f35 is 4.17. The issue happens on RPM < 4.17.
https://src.fedoraproject.org/projects/rpms/rpm

Ok then I'll prepare a change to use the f35 contrainer for that job.

Ok I've merged the change and it seems to behave correctly and I see

2022-03-15 12:00:10.912859 | TASK [fedpkg-srpm-build : Show RPM version]
2022-03-15 12:00:12.271918 | container | RPM version 4.17.0
2022-03-15 12:00:12.418610 | container | ok: Runtime: 0:00:00.005646

However the bump of rpmlint to benefit version2 might be great soon, would you like to open a separate issue for this ?

I opened the ticket here.
https://pagure.io/fedora-ci/general/issue/332

I'd appreciate is long term strategy, not just one time fix. IOW I'd love if the goal is e.g.:

There is an on going discussion here https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/thread/24QTCZLZ635UDCOEPAPVP3LG3ML54CAT/

~~~
Zuul always uses an appropriate environment for test.
~~~

That should be the case and that should be fixed if not.

While the non goal is:

~~~
We have updated our container from F34 to F35 which fixes your current issues.
~~~

Bonus points would be if Zuul:

1) can sync/lock the host updates to the same versions as Fedora infrastructure uses to host Koji.
2) Use Mock to setup the test environment.

The outcome would be that as long as Koji works, Zuul should work as well.

rpm-scratch-build(-*) job relies on Fedora Koji to run scratch builds, only the SRPM is built on the Zuul test node (a container which was switched to f35 some minutes ago).

The fedpkg srpm might fails in case of macro missing or rpm outdated version. So solution could be to ensure:
- we run the rpm-scratch-build job on a container that match the release/branch -> This can be done. However for the rawhide branch I fear that the "instability" of a rawhide container could prevent the CI to trigger the scratch-build on Koji from time to time. I feel more comfortable to rely on a stable version.
- we run the rpm-scratch-build job on a VM and use mock to build the srpm by using the right target release. However this solution requires far more compute time and we prefer to avoid using VM when possible.

Other jobs (installability/functional tests) are either performed on Testing Farm or Zuul nodes.
- rpm-install-test: Zuul node (use a VM that match the target release)
- rpm-test: Zuul node (use a VM that match the target release)
- rpm-(sti|tmt)-test: use Testing Farm (TF uses the right VM image according to the target branch)

rpmlint and rpm-rpminspect jobs run on f34 (should be updated to f35)

To avoid such issues in the future, I see only two approaches that are robust enough:

1. The SRPMs are always created from within target mock (e.g. via fedpkg srpm --srpm-mock) -- this is a bit slower than creating the SRPM with local rpmbuild (e.g. via fedpkg srpm) but works nicely even for very distant environments (e.g. epel7-only spec files).

2. The virtual machines are "native". This is much harder to achieve because you need to maintain 7+ environments including both very old (and hence problematic) and very fresh (and hence unstable) systems.