fedora-infrastructure

#8858 Unavailability of the-new-hotness
Closed: Fixed by kevin. Opened by zlopez.

Closed: Fixed
Reopen Issue

Describe what you would like us to do:

Because of the issue in the-new-hotness I started digging in the Kibana to find out what happened and I noticed a few minutes downtimeon 22nd April 22:24 UTC. When I looked at the last log entry in previous pod, I didn't saw any error. I just wanted to ask if there was some downtime or it is possible that Kibana doesn't capture stderr?

Kibana link: https://kibana.app.os.fedoraproject.org/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2020-04-22T20:24:00.000Z',mode:absolute,to:'2020-04-22T20:35:00.000Z'))&_a=(columns:!(kubernetes.container_name,message),index:'project.the-new-hotness.4d37d9b5-7c9a-11e9-97f8-525400a18f5e.*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'kubernetes.namespace_name:%22the-new-hotness%22')),sort:!('@timestamp',asc),uiState:(vis:(legendOpen:!f)))

When do you need this to be done by? (YYYY/MM/DD)

N/A

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: OpenShift, medium-gain, medium-trouble

So, this was almost surely this discussion in #fedora-noc: 

pr 22 13:28:47 <zodbot>        PROBLEM - rabbitmq01.phx2.fedoraproject.org/Check queue the-new-hotness is WARNING: RABBITMQ_QUEUE WARNING - messages WARNING (29), messages_ready OK (0) messages_unacknowledged OK (29) consumers OK (1) (noc01)
Apr 22 13:38:47 <zodbot>        PROBLEM - rabbitmq01.phx2.fedoraproject.org/Check queue the-new-hotness is CRITICAL: RABBITMQ_QUEUE CRITICAL - messages CRITICAL (120), messages_ready OK (0) messages_unacknowledged OK (120) consumers OK (2) (noc01)
Apr 22 13:39:03 <smooge>        thats not good
Apr 22 13:39:36 <smooge>        is it?
Apr 22 13:40:12 <nirik> not likely. I can look. 
Apr 22 13:40:39 <zodbot>        PROBLEM - rabbitmq01.phx2.fedoraproject.org/Check queue bodhi is WARNING: RABBITMQ_QUEUE WARNING - messages WARNING (15), messages_ready OK (0) messages_unacknowledged OK (15) consumers OK (3) (noc01)
Apr 22 13:41:37 <nirik> huh
Apr 22 13:41:39 <nirik> NAME                       READY     STATUS    RESTARTS   AGE
Apr 22 13:41:39 <nirik> the-new-hotness-68-2wv5j   1/1       Unknown   0          5d
Apr 22 13:41:39 <nirik> the-new-hotness-68-h98w4   1/1       Running   0          11m
Apr 22 13:41:46 <nirik> Unknown? never seen that one before
Apr 22 13:41:58 <nirik> its processing tho. 
Apr 22 13:42:03 <nirik> so it should finish
Apr 22 13:42:12 <smooge>        nirik, maybe that is the equivalent of yak farming
Apr 22 13:42:34 <smooge>        it has a lot of yaks before it can get to running
Apr 22 13:43:15 <smooge>        ok I work up at 4am and started work at 5.. so I am going to call it soon
Apr 22 13:43:29 <smooge>        work up.. man I am tireder than I thought
Apr 22 13:43:34 <nirik> os-node03 seems... non resposinve
Apr 22 13:43:53 <smooge>        not pinging?
Apr 22 13:45:02 <nirik> slow to login
Apr 22 13:45:37 <nirik> pegged on memory
Apr 22 13:48:00 <smooge>        reboot?
Apr 22 13:48:32 <nirik> I as going to try and give it more memory. 
Apr 22 13:48:43 <nirik> but it isn't setup to increase on the fly
Apr 22 13:48:49 <zodbot>        RECOVERY - rabbitmq01.phx2.fedoraproject.org/Check queue the-new-hotness is OK: RABBITMQ_QUEUE OK - messages OK (0) messages_ready OK (0) messages_unacknowledged OK (0) consumers OK (1) All queues under the thresholds (noc01)
Apr 22 13:49:56 <nirik> it looks like it killed everything somehow... 
Apr 22 13:50:00 *       bowlofeggs passes smooge a vat of coffee to keep him going another 48-72 hours or so
Apr 22 13:50:39 <zodbot>        RECOVERY - rabbitmq01.phx2.fedoraproject.org/Check queue bodhi is OK: RABBITMQ_QUEUE OK - messages OK (0) messages_ready OK (0) messages_unacknowledged OK (0) consumers OK (2) All queues under the thresholds (noc01)
Apr 22 13:51:08 <nirik> I think it's repaired itself

Basically what appeared to happen is that os-node03 just OOM killed something very important like cri-o or kubed, which in turn killed all the pods on that compute node.
It came back up a bit later after restarting everything, but things were down for a bit.

Also, note that part of the logging pods were on that node, so thats why there were no logs for this time. :(

Anyhow, I am pretty sure thats what happened. I think once we get the dc move done and things back up and running, I would like to double the memory on our openshift nodes or add some more...

If there's anything to still investigate here let me know.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Thanks for the response @kevin, I wasn't aware there was something like this going on. This also explains the weird response in bugzilla that was reported in the-new-hotness issue.
Metadata
None

OpenShift medium-gain medium-trouble

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.