#102 Fedora CI job output success but hangs and timeout
Closed by jimbair. Opened by sergesanspaille.

Hi folks,

this job

https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2951/pipeline/

shows a successful run, but doesn't stop and then reports a timeout. The tail of the log is

[2020-03-23T18:18:25.316Z]     Testing Time: 1322.37s
[2020-03-23T18:18:25.316Z]       Expected Passes    : 2365
[2020-03-23T18:18:25.316Z]     Run test 'llvm-test-suite/test-suite': done. Test's exit code: 0
[2020-03-23T18:18:25.316Z]   stdout_lines: <omitted>

This timeout doesn't make sense to me :-/


I'm taking a look now to see what I can find

You can see the full run in the console here:
https://jenkins-continuous-infra.apps.ci.centos.org/view/Fedora%20All%20Packages%20Pipeline/job/fedora-rawhide-pr-pipeline/2951/artifact/package-tests/logs/console.log

At the end of the test, it hits cleanup:
++ clean_up
https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/blob/master/config/Dockerfiles/fedoraci-runner/package-test/package-test.sh#L106-L124

The long timeout is in the Jenkinsfile (I believe) which explains why it takes so long (some tests do take a very long time)
https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/blob/master/Jenkinsfile#L166-L168

What I find odd is that the test (in console) seems fine, except that it hangs:

    Testing Time: 1322.37s
      Expected Passes    : 2365
    Run test 'llvm-test-suite/test-suite': done. Test's exit code: 0
  stdout_lines: <omitted>
++ clean_up
++ ret=124

I'm still looking around, but that's what I've sorted out so far...

While looping through test yamls, we're getting stuck on tests.yaml on the second test:

[2020-03-23T18:18:25.250Z] changed: [/workDir/workspace/fedora-rawhide-pr-pipeline/images/test_subject.qcow2] => (item=llvm-test-suite/test-suite) => changed=true

The last change to that file was in September, so that isn't part of the PR, so either something in that test suite changed or something in our qcow changed (or some other odd piece in the chain).

Still poking around...

Metadata Update from @jimbair:
- Issue assigned to jimbair

So I see that this PR was merged after a second re-run:
https://src.fedoraproject.org/rpms/clang/pull-request/58

https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2956/pipeline/

then walking through the other tests, I found they also ran as expected:
https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2976/pipeline/

I'm not sure if we had an infra hiccup or something else, but this appears to be working now. I do see a new PR currently in flight, but I'll consider this issue good for now, but please re-open if you see this behavior again.

Metadata Update from @jimbair:
- Issue status updated to: Closed (was: Open)

Metadata