Thanks for the report

It seems that's because rpminspect is run with the '-t BAD' parameter meaning that VERIFY level of failure does not trigger rpminspect failure. https://rpminspect.readthedocs.io/en/latest/usage.html#reporting-levels

Loading the report in the CI web app shows that the issue are reported as VERIFY (https://fedora.softwarefactory-project.io/rpminspect-report/?url=https://fedora.softwarefactory-project.io/logs//4/4/64cabec16a79cd36fa1fe90a62952c24f7b5a5b5/check/rpm-rpminspect/d0f0d31/result.json)

AFAIR we set that option because the VERIFY level failures might happen quite often. So either we revert to default (VERIFY) or we check with rpminspect upstream if such validation should raise 'BAD' level failure.

How about reporting the number of VERIFY tests in the PR message? My understanding is that those issues would require the attention of the author, so at least it makes sense to have it reported somewhere. Maybe it would look nice to get the report from testing-farm, e.g. https://artifacts.dev.testing-farm.io/adf8c134-9e29-4f66-b7d5-ad3d17c887e3/

But the main issue I have is with the SOVERSION change which should be covered by some tests. But I see that that one is reported in rpmdeplint which is not present in this configuration? Could these tests be added on the zuul side by default, or should they be configured on the package's tmt side? I think it can capture quite some severe issues so it would be nice to be on all zuul. Same with installability CI on bohdi.

How about reporting the number of VERIFY tests in the PR message? My understanding is that those issues would require the attention of the author, so at least it makes sense to have it reported somewhere. Maybe it would look nice to get the report from testing-farm, e.g. https://artifacts.dev.testing-farm.io/adf8c134-9e29-4f66-b7d5-ad3d17c887e3/

It is actually not as easy to make Zuul reports such data as a PR comment. It will require that the rpminspect Zuul job is itself able to call the Pagure API to push comments.

There is an issue open to integrate this reporting system (however I don't yet have time to handle that task). https://pagure.io/fedora-ci/general/issue/417. The current reporting system (see rpminspect report in https://fedora.softwarefactory-project.io/zuul/build/2a747a4ee91e49b097e2407256c18a66/artifacts) displays the amount of issues in VERIFY.

How is it handled in testing-farm, is the rpminspect job is set to fail for VERIFY issues ?

But the main issue I have is with the SOVERSION change which should be covered by some tests. But I see that that one is reported in rpmdeplint which is not present in this configuration? Could these tests be added on the zuul side by default, or should they be configured on the package's tmt side? I think it can capture quite some severe issues so it would be nice to be on all zuul. Same with installability CI on bohdi.

The installability jobs is the rpm-install-test in the Fedora Zuul CI. Here is an issue to discuss this subject: https://pagure.io/fedora-ci/general/issue/369

For rpmdeplint, yes Fedora Zuul CI does not run it. Perhaps @mvadkert knows how to enable it on PRs via Fedora CI ?

There is an issue open to integrate this reporting system (however I don't yet have time to handle that task). https://pagure.io/fedora-ci/general/issue/417

I am working on something similar for packit+copr. If you have an idea of a common interface, I think I can integrate to work for both. It will require to change the rpminspect job to call the tmt plan, but I think it would be good to have some consistency across all of these.

I am trying to address the other 2 jobs over there as well. I will be using the artifacts in /var/share/test-artifacts/*.rpm, so if that is available in the testing-farm triggers from zuul as well, it should be a drop-in replacement.

The current reporting system displays the amount of issues in VERIFY.

Yeah, but not having it in the PR part makes it a bit hard to detect. Is it possible to set the job result as something besides PASS/FAIL? Or to hack in additional messages to the overall zuul message on the PR?

@msrb could we continue adding rpminspect and rpmdeplint next to installability?

@mvadkert I am afraid rpmdeplint is in a very poor shape and nothing but Rawhide works there (it's a problem of the wrapper, not the test itself). @jpopelka is currently actively working on rpmdeplint, so once things are better, we can try to enable it.
rpminspect should work, I think.

@msrb cool, also just trigger is missing?