We have recently been having a problem with test failures in gating on Fedora 36 for our project, stratisd[0].
We have a very good hypothesis about why this is happening although I'ld like to point out that something like #348 would help to confirm this, and would help with many other things as well.
When stratisd is compiled, some code is auto-generated from various header files that are available on the system on which it is compiled, including the header file, dm-ioctl.h[1]. This file hard-codes several version numbers[2]. These version numbers are passed on every devicemapper ioctl that stratisd issues. If the version numbers are higher than those that the devicemapper kernel module recognizes, then the ioctl is rejected. stratisd is implemented so that it shuts down if the first ioctl it sends is rejected (this ioctl is special and vital).
We believe that the build and test infrastructure in the Fedora packaging system is in such a state that this is what is happening. Our hypothesis is that some sort of dnf update call is made before stratisd is compiled and tested, but that no subsequent reboot is done. This means that the headers are updated, but without this reboot, not the relevant kernel module. As a consequence all our tests that require stratisd to be running as a daemon can not be run.
We can not absolutely confirm this, although we could if #348 were implemented, as stratisd does extensive logging, and this problem would be readily apparent in the log.
At this point we've reduced our gating tests to the tiny subset of tests that do not test stratisd as a daemon. But obviously that is not the point of the gating tests, which are supposed to test the functioning of the installed package.
We're asking if you can confirm whether our hypothesis is correct and if you can correct this problem, assuming our hypothesis is correct.
I think that the best way to go about confirming the hypothesis would be to implement #348.
[0] https://bodhi.fedoraproject.org/updates/?packages=stratisd [1] https://github.com/torvalds/linux/blob/master/include/uapi/linux/dm-ioctl.h [2] https://github.com/torvalds/linux/blob/master/include/uapi/linux/dm-ioctl.h#L288
We confirmed that our hypothesis as to the root cause of the problem was correct, and that is now fixed in devicemapper Rust crate v 0.32.1, stratisd 3.3.0. So I think it's fair to close this.
Metadata Update from @mulhern: - Issue status updated to: Closed (was: Open)