fedora-ci / general

#174 [osci-jenkins-1] 0/2 nodes are available: 1 Insufficient cpu, 1 Insufficient memory
Closed by astepano. Opened by msrb.

Closed
Reopen Issue

It looks like osci-jenkins-1 is having trouble creating pods: https://osci-jenkins-1.ci.fedoraproject.org/job/eln-build-pipeline/47890/console

Many (all?) builds seem to be affected.

Metadata Update from @msrb:
- Issue assigned to astepano

@astepano could you please take a look what is going on? thanks :wink:

I think this is because Java on master process eats all available resources.
Jenkins fails check jenkins-runners.
We didn't find out why Java process is so high.
We have a new VPC in new AWS region.
Currently I cannot scale K8S resources.
Need to wait until we move to new VPC + make some optimization to existing configuration.

Note the problem is that kubernetes plugin is unable to create any pods. Currently there is 0 pods attached to Jenkins master. This started on Friday (evening I think). This was not a problem before.

And just to be clear: since no pods can be created, ELN + generic + functional pipelines will all either block (and builds pile up, waiting indefinitely), or timeout after couple of hours. So the Jenkins instance is basically unusable at the moment.

I believe we found cause for this. I sent PR/merged ELN pipelines. The reason for this 40K + 60K history length + EBS as storage backend for k8s nodes.

Metadata Update from @astepano:
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.