Issue #7735: [greenwave] Greenwave not sending messages on fedmsg or fedora-messaging - fedora-infrastructure

fedora-infrastructure

#7735 [greenwave] Greenwave not sending messages on fedmsg or fedora-messaging

Closed: Fixed 5 years ago by kevin. Opened 5 years ago by tdawson.

Greenwave hasn't sent a message via fedmsg or fedora-messaging for two weeks.
https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.greenwave.decision.update

I've locally logged all fedora-messaging messages, to double check, incase they weren't being duplicated on both fedora-messaging and fedmsg. Also without success.
As far as I can see, greenwave isn't sending out messages.

smooge commented 5 years ago

@gnaponie @ralph I think this is somewhere on your teams workflow?

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: greenwave

5 years ago

ralph commented 5 years ago

Linking #7111 from the last time something like this came up.

ralph commented 5 years ago

Looks like the last deployment was 12 days ago. The pods starts up and connects to the public gateway at tcp://hub.fedoraproject.org:9940 and then just sits there. It probably got connected to a gateway with broken vpn and doesn't know that it needs to reconnect. Will try to roll it out afresh with oc rollout latest dc/greenwave-fedmsg-consumers.

ralph commented 5 years ago

Wait - I should've checked this first. Greenwave sends messages in response to resultsdb messages.... but @tdawson, it looks like resultsdb isn't sending any. At least, I don't see any in this datagrepper query. Any idea what's up there?

edit: it looks like the most recent one is from ~14 days ago: 2019-04-09 17:17:39 UTC.

Edited 5 years ago by ralph

ralph commented 5 years ago

/var/log/httpd/error_log on resultsdb01.qa.fedoraproject.org is full of pika connection refused errors.

Looks like the code was introduced in https://pagure.io/taskotron/resultsdb/pull-request/130

Maybe just missing config to go with it? @cverna, can you have a look?

Metadata Update from @ralph:
- Issue assigned to cverna

5 years ago

cverna commented 5 years ago

The fedora-messaging config was setup to be applied in stg only.

I have opened an FBR to fix that --> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/JPQCZ3MT7AJZJLSLJDC36V3CVVZ3YUPT/

cverna commented 5 years ago

I have also sent an FBR to add monitoring for resultsdb messages.

https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/U4P4SAE6JG7ISJOYQNUE6L6QXCMAG6WX/

smooge commented 5 years ago

This has been deployed can someone test?

cverna commented 5 years ago

The greenwave fedmsg consumer is now getting messages from resultsdb. But I have not seen yet a greenwave message sent, let's give it enough time to receive a results that would trigger a message.