It seems since the plugins update on Jenkins master the oc tool also got updated, the new version does not work well with the old openshift cluster running jenkins.
sh-4.2$ oc version oc v3.11.0+d545883-301 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.30.0.1:443 openshift v3.6.1+008f2d5 kubernetes v1.6.1+5115d708d7
For example oc get pods fails
oc get pods
sh-4.2$ oc get pods No resources found. Error from server (NotAcceptable): unknown (get pods) sh-4.2$
This impact our job than clean up old PODs, like https://jenkins-continuous-infra.apps.ci.centos.org/view/all/job/recycle-openshift-builds/663/console
+ oc get pods + egrep 'Error|Evicted' No resources found. Error from server (NotAcceptable): unknown (get pods) [Pipeline] sh ++ oc get pods ++ grep ' Completed ' ++ awk '{print $1}' ++ grep '\-build' No resources found. Error from server (NotAcceptable): unknown (get pods) + oc delete pod error: resource(s) were provided, but no name, label selector, or --all flag specified
@astepano @jimbair could you help investigate this?
@bgoncalv use older oc version, I use oc-3.7 to interact with the cluster, that works ....
I believe there is nothing to fix here, please note that you have oc version 3.11, but the cluster is 3.6
I can second this. Version mishmash between cli and cluster can lead to really weird errors. Using the same cli version as the cluster is definitely the way to go.
I agree, but why are we using the new version now and what is the best way to install the older one? :)
https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/blob/master/config/s2i/jenkins/slave/Dockerfile#L1
We note this on the slave, but not on the master - do we need a dockerfile for the master?
the jenkins master version seems to be build from https://github.com/CentOS-PaaS-SIG/ci-pipeline/blob/master/config/s2i/jenkins/jenkins-persistent-buildconfig-template.yaml#L78
As we discussed in IRC:
https://github.com/CentOS-PaaS-SIG/ci-pipeline/commit/60dfb9b512e0c47dc876aeed032061d27b9533e0#diff-579465d3c50a19bd07dfe333deee4083R78
This change appears to be what's causing the updated binary
As noted here: https://hub.docker.com/r/openshift/jenkins-2-centos7/
"This image is now maintained at quay.io/openshift/origin-jenkins"
However, I don't see any 3.x tags in quay, so it's all 4.0+. I assume we would need to update the yaml (maybe only for our instance) to point specifically at 3.6 until we migrate to communishift.
I've pinged @jbieren in IRC (and here now) to see if maybe this shared ci-pipeline repo is being used in other 3.11 clusters, while ours is still on 3.6? Or if we're the only ones using it, maybe we move it back down to 3.6.
As noted here: https://hub.docker.com/r/openshift/jenkins-2-centos7/ "This image is now maintained at quay.io/openshift/origin-jenkins" However, I don't see any 3.x tags in quay, so it's all 4.0+. I assume we would need to update the yaml (maybe only for our instance) to point specifically at 3.6 until we migrate to communishift. I've pinged @jbieren in IRC (and here now) to see if maybe this shared ci-pipeline repo is being used in other 3.11 clusters, while ours is still on 3.6? Or if we're the only ones using it, maybe we move it back down to 3.6.
I don't know for certain of any other projects using the repo, unless the downstream one still is - @bgoncalv is it?
@jbieren do you know which instance the downstream one is? I can take a look from our side (or we can roll back the config and see if we break something else). I also pinged you on IRC. =)
My IRC is on my desktop in the office I am not allowed to go into =\
I don't understand the question of what instance the downstream one is
@jbieren oof - sorry about IRC. =\ Is Google Hangouts good? Email?
As for my question, I was asking as a follow up to your question:
My assumption is you mean the RHEL build pipelines?
Both work great
As for my question, I was asking as a follow up to your question: I don't know for certain of any other projects using the repo, unless the downstream one still is - @bgoncalv is it? My assumption is you mean the RHEL build pipelines?
Yeah, I was referring to the rhel8 rpm build pipelines
the rhel8 pipeline doesn't depend on https://github.com/CentOS-PaaS-SIG/ci-pipeline any more.
Proposed fix here: https://github.com/CentOS-PaaS-SIG/ci-pipeline/pull/829