The koji fedora-messaging plugin seems to be having some issues.
Some folks report that it never sent them build completed or failed messages.
When I was tagging the mass rebuild in, it stalled in the middle with a bunch of timeouts for sending fedora-messaging messages.
I wonder if this could be related to the large badges queue somehow?
Some errors in the koji error_log:
[Mon Jul 22 21:28:57.844982 2024] [wsgi:error] [pid 3441885:tid 3441885] [client 10.3.163.77:33030] 2024-07-22 21:28:57,844 [ERROR] m=host.failBuild u=buildvm-ppc64le-11.iad2.fedoraproject.org p=3441885 r=10.3.163.77:33030 koji._koji_plugin__koji-fedoramessaging: Schema for buildsys.build.state.change message (id 87b145e2-0c9a-4334-95d9-70053018df7e) from Koji not valid trying to send message as generic fedoramessaging message. Error: None is not of type 'number'
and
[Mon Jul 22 19:39:41.014691 2024] [wsgi:error] [pid 3427875:tid 3427875] [client 10.3.163.76:60874] 2024-07-22 19:39:41,013 [ERROR] m=host.openTask u=buildvm-x86-20.iad2.fedoraproject.org p=3427875 r=10.3.163.76:60874 koji._koji_plugin__koji-fedoramessaging: Un-expected error sending fedora-messaging message
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, high-gain, ops
Looking at the error, it doesn't seem to be related to the badges queue, but rather some change in koji that is not providing value for the message we expect.
I've fixed the schema, rebuilt it and deployed it on koji01 and koji02.
koji01
koji02
Metadata Update from @abompard: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.