fedora-ci / general

#352 /check-conflicts-x86_64 and /check-conflicts-aarch64 are timing out for the latest grafana build
Closed by msrb. Opened by jkurik.

Closed
Reopen Issue

The currently latest grafana build grafana-8.5.6-1.fc37 has two failing tests: /check-conflicts-x86_64 and /check-conflicts-aarch64 . Both tests are timing out after 90mins resp. 180mins. However when I run those tests manually on a local machine, both tests ( rpmdeplint check-conflicts ) are successfully passing after approx. 2 seconds.
I was not able to trace down what is the root cause due to my lack on CI knowledge and the environment those tests are running in. Can someone please have a look at it ? I do not expect the issue is with the grafana build it self.

I tried to run manually the whole fedora-ci.koji-build.rpmdeplint.functional in testing-farm.io and here are durations of each individual tests:

            00:01:10 /check-sat-x86_64 [1/8]
            00:11:43 /check-repoclosure-x86_64 [2/8]
            00:01:11 /check-upgrade-x86_64 [3/8]
            04:32:14 /check-conflicts-x86_64 [4/8]
            00:01:04 /check-sat-aarch64 [5/8]
            00:08:28 /check-repoclosure-aarch64 [6/8]
            00:01:00 /check-upgrade-aarch64 [7/8]
            01:57:38 /check-conflicts-aarch64 [8/8]

So, in total the overall duration is almost 7 hours. As such, may I ask please to prolong the maximum time for /check-conflicts-x86_64 resp. /check-conflicts-aarch64 to some more relevant time (~ 8 hours each) ?

The same is happening for grafana-9.0.2-1.fc37.

The maximum durations for individual rpmdeplint tests are configured here. We could extend them but as @msrb mentioned, such a long execution time is weird. @bgoncalv, any idea what might have gone wrong with the check-conflicts test that it takes that long? What is a reasoanble maximum time for these tests?

Note we are currently seeing all rpmdeplint tests taking waaay too long for some reason: https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/rpminspect-pipeline/job/master/buildTimeTrend

The first time we had a trouble with rpmdeplint test was on Jul 4, 2022 on grafana-8.5.6-1.fc37 build and it is always reproducible when tests are restarted/re-triggered.

@psss , @bgoncalv : I collected log file from the test and IMO the test looks fine. The build contains 13518 files the test is checking for potential conflicts. Every file check takes approx. 1/2 second, which looks legit to me.

Metadata Update from @msrb:
- Issue assigned to msrb

Test with the increased timeout: https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/rpmdeplint-pipeline/job/master/61425/

The test /check-conflicts-x86_64 is still timing out after 4 hours: https://artifacts.dev.testing-farm.io/437ae0b6-42d4-47fe-9f03-1843014d6800/

[32m[09:33:20] [+] [worker_0] [stdout] [/rpmdeplint]     execute[0m
[32m[09:34:22] [+] [worker_0] [stdout] [/rpmdeplint]         how: tmt[0m
[32m[09:43:52] [+] [worker_0] [stdout] [/rpmdeplint]             00:01:02 /check-sat-x86_64 [1/8][0m
[32m[09:44:45] [+] [worker_0] [stdout] [/rpmdeplint]             00:09:30 /check-repoclosure-x86_64 [2/8][0m
[32m[13:44:45] [+] [worker_0] [stdout] [/rpmdeplint]             00:00:53 /check-upgrade-x86_64 [3/8][0m
[32m[13:46:09] [+] [worker_0] [stdout] [/rpmdeplint]             04:00:00 /check-conflicts-x86_64 [4/8] (timeout)[0m
[32m[14:02:41] [+] [worker_0] [stdout] [/rpmdeplint]             00:01:23 /check-sat-aarch64 [5/8][0m
[32m[14:04:29] [+] [worker_0] [stdout] [/rpmdeplint]             00:16:31 /check-repoclosure-aarch64 [6/8][0m

The test /check-conflicts-aarch64 timed out as well, after 90 minutes.

@msrb any update on this ?
Can we set the timeout temporarily i.e. to 20 hours to measure how long the test takes in the CI environment and modify the timeout then to an acceptable value accordingly.

12h here: https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/rpmdeplint-pipeline/view/change-requests/job/PR-42/2/

There are probably some timeouts on testing farm side as well, but 12 hours should still be within the range. Let's see...

 
/check-conflicts-x86_64:
    result: pass
    log:
      - data/check-conflicts-x86_64/output.txt
    duration: 04:35:14

Patch merged, re-running tests for the latest grafana build now: https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/rpmdeplint-pipeline/job/master/62698/
The results should be visible in Bodhi once testing is done.

Metadata Update from @msrb:
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.