Ohai,
our Jenkins (https://jenkins-foreman.apps.ocp.ci.centos.org/) seems to have lost the connection to the underlying OpenShift, as there are no more clouds defined in "Manage Jenkins → Manage Nodes and Clouds → Configure Clouds".
If I understood @arrfab correctly on IRC last week, this is most probably due to a version mismatch between the running OpenShift and the Jenkins OpenShift/Kubernetes plugins.
There are several similar reports of such behaviour on the internet:
However, judging by those, the combination of plugins we have should work?
Kubernetes 1.31.0 Kubernetes Client API 5.10.1-171.vaa0774fb8c20 OpenShift Sync 1.0.53
Could you please double-check against a working Jenkins which versions we should stick to?
The logs look fine to me too?
2022-04-04 11:12:14 INFO io.fabric8.jenkins.openshiftsync.GlobalPluginConfiguration configChange OpenShift Sync Plugin processing a newly supplied configuration 2022-04-04 11:12:14 INFO io.fabric8.jenkins.openshiftsync.OpenShiftUtils shutdownOpenShiftClient Stopping openshift client: null 2022-04-04 11:12:14 INFO io.fabric8.jenkins.openshiftsync.OpenShiftUtils initializeOpenShiftClient Current OpenShift Client Configuration: io.fabric8.openshift.client.OpenShiftConfig@7e2ba938[oapiVersion=v1,openShiftUrl=https://kubernetes.default:443/oapi/v1/,buildTimeout=300000,openshiftApiGroupsEnabled=false,disableApiGroupCheck=false,trustCerts=true,disableHostnameVerification=false,masterUrl=https://kubernetes.default:443/,apiVersion=v1,namespace=foreman,caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,caCertData=<null>,clientCertFile=<null>,clientCertData=<null>,clientKeyFile=<null>,clientKeyData=<null>,clientKeyAlgo=<null>,clientKeyPassphrase=changeit,trustStoreFile=<null>,trustStorePassphrase=<null>,keyStoreFile=<null>,keyStorePassphrase=<null>,authProvider=<null>,requestConfig=io.fabric8.kubernetes.client.RequestConfig@3a462db2,contexts=[],currentContext=<null>,username=<null>,password=<null>,oauthToken=<null>,watchReconnectInterval=1000,watchReconnectLimit=-1,connectionTimeout=10000,uploadConnectionTimeout=10000,uploadRequestTimeout=120000,requestRetryBackoffLimit=0,requestRetryBackoffInterval=0,requestTimeout=10000,rollingTimeout=900000,scaleTimeout=600000,loggingInterval=20000,websocketTimeout=5000,websocketPingInterval=30000,maxConcurrentRequests=64,maxConcurrentRequestsPerHost=5,impersonateUsername=<null>,oauthTokenProvider=<null>,impersonateGroup=<null>,impersonateGroups=<null>,impersonateExtras=<null>,http2Disable=false,httpProxy=<null>,httpsProxy=<null>,proxyUsername=<null>,proxyPassword=<null>,noProxy={},userAgent=<null>,tlsVersions={TLS_1_2},errorMessages={401=Unauthorized! Configured service account doesn't have access. Service account may have been revoked., 403=Forbidden!Configured service account doesn't have access. Service account may have been revoked.},customHeaders=<null>,autoConfigure=false,file=<null>] 2022-04-04 11:12:15 INFO io.fabric8.jenkins.openshiftsync.OpenShiftUtils initializeOpenShiftClient New OpenShift client initialized: io.fabric8.openshift.client.DefaultOpenShiftClient@3a0386be 2022-04-04 11:12:16 INFO io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask doRun Confirming Jenkins is started 2022-04-04 11:12:16 INFO io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask waitForJenkinsStartup Jenkins not ready... 2022-04-04 11:12:16 INFO jenkins.InitReactorRunner$1 onAttained System config adapted 2022-04-04 11:12:16 INFO jenkins.InitReactorRunner$1 onAttained Loaded all jobs 2022-04-04 11:12:16 INFO jenkins.InitReactorRunner$1 onAttained Configuration for all jobs updated 2022-04-04 11:12:16 INFO hudson.model.AsyncPeriodicWork lambda$doRun$1 Started Download metadata 2022-04-04 11:12:16 INFO hudson.model.AsyncPeriodicWork lambda$doRun$1 Finished Download metadata. 3 ms 2022-04-04 11:12:16 INFO jenkins.util.groovy.GroovyHookScript execute Executing /var/lib/jenkins/init.groovy.d/update-center-init.groovy 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask waitForJenkinsStartup Jenkins not ready... 2022-04-04 11:12:17 INFO jenkins.InitReactorRunner$1 onAttained Completed initialization 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask stop Stopping all informers ... 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask stop Stopped all informers 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.BuildConfigInformer start Starting BuildConfig informer for {} !!foreman 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.BuildConfigManager reconcileJobsAndBuildConfigs Reconciling jobs and build configs 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.BuildConfigManager reconcileJobsAndBuildConfigs Reconciling jobs and build configs 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.BuildConfigInformer start BuildConfig informer started for namespace: {}foreman 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.BuildInformer start Starting Build informer for {} !!foreman 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.BuildManager reconcileRunsAndBuilds Reconciling job runs and builds 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.BuildManager reconcileRunsAndBuilds Reconciling job runs and builds 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.BuildInformer start Build informer started for namespace: {}foreman 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.ConfigMapInformer start Starting configMap informer for {} !!foreman 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.ConfigMapInformer start ConfigMap informer started for namespace: {}foreman 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.ImageStreamInformer start Starting ImageStream informer for {} !!foreman 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.ImageStreamInformer start ImageStream informer started for namespace: {}foreman 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.SecretInformer start Starting secret informer {} !!foreman 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.SecretInformer start Secret informer started for namespace: {}foreman 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask start All the namespaced informers have been registered!! ... starting all registered informers 2022-04-04 11:12:17 INFO io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask start All registered informers have been started 2022-04-04 11:12:18 INFO io.fabric8.jenkins.openshiftsync.ConfigMapInformer onAdd ConfigMap informer received add event for: {}cico-workspace 2022-04-04 11:12:18 INFO io.fabric8.jenkins.openshiftsync.PodTemplateUtils addAgents Adding PodTemplate(s) for foreman
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: centos-ci-infra, high-gain, high-trouble
I have tried various combinations of plugins but as you have mentioned the ones there look to be the correct version. I also tried to rollout the pod to see if that would pick up the correct configuration but that also made no difference.
My next plan would be to recreate the project from scratch to bring it back to a "default" state and see if the issue still persists. Is this something you would be ok with me doing?
Please do. All configs are in JJB.
Metadata Update from @mobrien: - Issue assigned to mobrien
Metadata Update from @zlopez: - Issue priority set to: Medium Priority (was: Waiting on Assignee)
@evgeni Sorry for the delay on this. We rolled out a cluster upgrade on Friday which had a new Jenkins image along with it. This appears to have fixed the issue.
Could you test it when you get a chance and let us know if all is working ok?
It seems to work fine again, thanks!
I didn't dare to perform a plugin upgrade round yet.
One thing that I noticed, the user created in https://pagure.io/centos-infra/issue/569 is missing now :(
From irc chat this looks to be resolved so I will close out the ticket
Metadata Update from @mobrien: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.