fedora-ci / general

#348 Run sosreports on test failure and upload the result as an artifact attached to the test
Closed by mvadkert. Opened by mulhern.

Closed
Reopen Issue

The functional tests associated with an update[1] are failing in Fedora 36. This is odd, because the same tests succeeded in rawhide[2].

We have tried to repeat the failure using a Fedora 36 VM and have not succeeded, i.e., the tests pass.

We have tried to induce failures by various techniques, and none of the failure modes we have caused have resembled the one that is in the Fedora 36 tests[3].

We are going to add some extra checks to give us a little more information about what may have gone wrong[4].

But we feel that it could take us quite a long time to get to the bottom of this be repeatedly extending our test scripts to get better information about whatever it is in their respective environments that is causing all the tests to error out in f36 but to succeed in rawhide.

There is already a facility for gathering this kind of information and it is called sosreports[5].

I'm not sure if it is available by default, but it should be easy to make it available in the test environment.

What I would like is a trigger that, on failure of a test, runs sosreports and uploads the resulting gzipped tarfile as an artifact attached to the failing test.

An alternative, which is a little more configurable, is a way to specify that sosreports should be run on failure in a project's tests.yml[6].

[1] https://bodhi.fedoraproject.org/updates/FEDORA-2022-32f6f5c246
[2] https://bodhi.fedoraproject.org/updates/FEDORA-2022-c6e008e3d4
[3] https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/149968/testReport/(root)/tests/simple/
[4] https://github.com/stratis-storage/testing/pull/138
[5] https://github.com/sosreport
[6] https://src.fedoraproject.org/rpms/stratisd/blob/f36/f/tests/tests.yml

Metadata Update from @mvadkert:
- Issue assigned to mvadkert

@mulhern sounds like a neat idea to gather sosreport in case of failed tests. Let me think a bit about the implementation, and get back to you soon

Metadata Update from @mvadkert:
- Issue tagged with: Testing Farm

We have this on our board, and know what the implementation will look like. Unfortunately, we plan to do it in Q4 as it requires additional work in our framework.

It is tracked inside RH here: https://issues.redhat.com/browse/TFT-1469

@mulhern hi, seems this can be now achieved with adding this to your plan:

finish:
    how: shell
    script: dnf install -y sos && sos report --batch --tmp-dir $TMT_PLAN_DATA

While this will do the step always, I consider that an sub-optimal, but should not matter that much.

Here is an example of our sanity plan MR with the above:

https://artifacts.dev.testing-farm.io/44effaa5-6e42-487f-8798-3faed2aae937/

The sos report can be downloaded from here:

https://artifacts.dev.testing-farm.io/44effaa5-6e42-487f-8798-3faed2aae937/work-sanitykw0ftk1o/testing-farm/sanity/data/

@mulhern would this solution be sufficient to you? it is quite straight forward, works now, I would maybe think so ...

Metadata Update from @mvadkert:
- Issue status updated to: Closed (was: Open)

Metadata Update from @mvadkert:
- Issue status updated to: Open (was: Closed)

closed by mistake ...
Edited by mvadkert

@mulhern closing for now, please let us know if the solution would be insufficient

Metadata Update from @mvadkert:
- Issue status updated to: Closed (was: Open)

@mvadkert Thanks! We will try it out.
Metadata

Testing Farm

None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.