#75 pipeline execution gets stuck
Closed by bgoncalv. Opened by bgoncalv.

it is not often, but it appears sometimes the pipeline gets stuck when trying to execute command in a container.

ex: https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-build-pipeline/6515/console

13:22:19  Starting nvr-verify
[Pipeline] echo
13:22:20  running pipeline step: nvr-verify
[Pipeline] withCredentials
21:18:35  Cancelling nested steps due to timeout

https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-f31-build-pipeline/109/console

22:18:34  + awk -F= '{print "env."$1"=\""$2"\""}' /workDir/workspace/fedora-f31-build-pipeline/koji-build/logs/job.props
[Pipeline] load
[Pipeline] { (/workDir/workspace/fedora-f31-build-pipeline/job.props.groovy)
[Pipeline] }
[Pipeline] // load
[Pipeline] echo
22:18:36  running pipeline step: koji-build
[Pipeline] withCredentials
06:17:13  Cancelling nested steps due to timeout

@jbieren I don't know if this could cause the problem, but do we really need this function to have handlePipelineStep and withCredentials?

https://github.com/openshift/contra-lib/blob/master/vars/executeInContainer.groovy#L24

I'd assume they should be called before running executeInContainer, if necessary.

@bgoncalv I wasn't involved with the writing of that function, so I can't speak to its design. However, I am not sure what harm they would cause by being there if they aren't being used. I know it is the last thing you see in your console output when this is hanging, but nothing after it is going to give any output until
https://github.com/openshift/contra-lib/blob/master/vars/executeInContainer.groovy#L39
so it is hard to be sure what is hanging

we hit it again:

02:01:39  + awk -F= '{print "env."$1"=\""$2"\""}' /workDir/workspace/fedora-f30-build-pipeline/koji-build/logs/job.props
[Pipeline] load
[Pipeline] { (/workDir/workspace/fedora-f30-build-pipeline/job.props.groovy)
[Pipeline] }
[Pipeline] // load
[Pipeline] echo
02:01:40  running pipeline step: koji-build
[Pipeline] withCredentials
10:00:13  Cancelling nested steps due to timeout
[Pipeline] // withCredentials

https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-f30-build-pipeline/924/console

Let's see if https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/pull/185 fixes the issue

@jbieren the PR above seems to solve the issue, but we still run some functions the depend on executeInContainer from contra-lib, such as the functions from the jobs to test qcow2 https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/blob/cde23212a27e843c082e8d93a6e60f7ba7ae8107/vars/fedoraImageTest.groovy would be possible to change the contra-lib function? Otherwise I will need to update the job that validates the qcow2.

@bgoncalv I am not the only person who works with contra-lib. You are welcome to raise a PR to the repo to make the change you want to executeInContainer, and it will need approval from others (not just me) to be accepted and merged. The library was created to be used by numerous teams, and I believe other teams are using the executeInContainer function, so we cannot break them

Metadata Update from @bgoncalv:
- Issue status updated to: Closed (was: Open)

Metadata