Issue #11411: noggin traceback in fedora-messaging - fedora-infrastructure

fedora-infrastructure

#11411 noggin traceback in fedora-messaging

Closed: Fixed 2 years ago by abompard. Opened 2 years ago by kevin.

noggin seems to be having tracebacks sending messages:

Traceback (most recent call last):
--- <exception caught here> ---
  File "/opt/app-root/lib64/python3.9/site-packages/fedora_messaging/api.py", line 262, in _tw
isted_publish
    yield _twisted_service._service.factory.publish(message, exchange=exchange)
  File "/opt/app-root/lib64/python3.9/site-packages/fedora_messaging/twisted/factory.py", line
 238, in publish
    protocol = yield self.when_connected()
  File "/opt/app-root/lib64/python3.9/site-packages/fedora_messaging/twisted/factory.py", line
 203, in when_connected
    yield self._client_deferred
fedora_messaging.exceptions.ConnectionException: 

[2023-07-12 17:06:48,228] ERROR in messaging: Publishing message failed. Giving up. ['  File "
/opt/app-root/src/noggin/utility/messaging.py", line 33, in publish\n    _publish(message)\n',
 '  File "/opt/app-root/lib64/python3.9/site-packages/backoff/_sync.py", line 105, in retry\n 
   ret = target(*args, **kwargs)\n', '  File "/opt/app-root/src/noggin/utility/messaging.py", 
line 23, in _publish\n    api.publish(message)\n', '  File "/opt/app-root/lib64/python3.9/site
-packages/fedora_messaging/api.py", line 316, in publish\n    eventual_result.wait(timeout=tim
eout)\n', '  File "/opt/app-root/lib64/python3.9/site-packages/crochet/_eventloop.py", line 19
8, in wait\n    result.raiseException()\n', '  File "/opt/app-root/lib64/python3.9/site-packag
es/twisted/python/failure.py", line 504, in raiseException\n    raise self.value.with_tracebac
k(self.tb)\n']

CC: @abompard

Metadata Update from @phsmoura:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

2 years ago

Metadata Update from @abompard:
- Issue assigned to abompard

2 years ago

abompard commented 2 years ago

The RabbitMQ certificate for Noggin has expired on June 25th, I'll renew it.

abompard commented 2 years ago

OK, in the ansible private repo I've created a script in files/rabbitmq/renew-certs.sh that will automatically renew all the certificates that expire within 7 days. It's certainly not a standalone solution as corresponding apps should be redeployed, but at least it's saving a bit of work, and can maybe be built upon.

abompard commented 2 years ago

And it works now.

Metadata Update from @abompard:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago