This update cannot be pushed to stable:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-3f6760cc65
The web UI does nothing, and the command line fails with a lengthy Python error indicating some sort of time-out:
$ bodhi updates request FEDORA-2020-3f6760cc65 stable Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 249, in send_request data = output.json() File "/usr/lib/python3.9/site-packages/requests/models.py", line 898, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib64/python3.9/site-packages/simplejson/__init__.py", line 525, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.9/site-packages/simplejson/decoder.py", line 370, in decode obj, end = self.raw_decode(s) File "/usr/lib64/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/bin/bodhi", line 11, in <module> load_entry_point('bodhi-client==5.2.2', 'console_scripts', 'bodhi')() File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/usr/lib/python3.9/site-packages/bodhi/client/__init__.py", line 263, in wrapper method(*args, **kwargs) File "/usr/lib/python3.9/site-packages/bodhi/client/__init__.py", line 690, in request resp = client.request(update, state) File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 117, in wrapper result = method(*args, **kwargs) File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 297, in request return self.send_request(f'updates/{update}/request', File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 252, in send_request raise ServerError( fedora.client.ServerError: ServerError(https://bodhi.fedoraproject.org/updates/FEDORA-2020-3f6760cc65/request, 504, Error returned from json module while processing b'https://bodhi.fedoraproject.org/updates/FEDORA-2020-3f6760cc65/request': b'Expecting value: line 1 column 1 (char 0)' b"<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n")
Ideally before F33 comes out.
The bodhi client seems to fail to authenticate against FAS (which I find rather slow on my side as well)
Can you try again?
Sorry for the delay. It is giving the same error as above.
Is this using krb for auth? Anyway I have renewed my krb ticket, and the bodhi client should also have access to the Fedora ssh key, and neither seems to help ...
@rjones Can you try it with the bodhi cli?
Metadata Update from @mohanboddu: - Issue tagged with: low-gain, low-trouble, ops
Metadata Update from @mohanboddu: - Issue priority set to: Waiting on Reporter (was: Needs Review)
bodhi cli means bodhi-client-5.2.2-2.fc33.noarch? That's what I'm testing above. I tested it again a second ago, but same error.
bodhi-client-5.2.2-2.fc33.noarch
Sorry for the delay. It is giving the same error as above. Is this using krb for auth? Anyway I have renewed my krb ticket, and the bodhi client should also have access to the Fedora ssh key, and neither seems to help ...
Bodhi client does not use krb nor ssh keys to authenticate to FAS, you need to provide your FAS password. Once you authenticate successfully once your openid session is cached under the ~/.fedora directory.
Could you try removing the files under ~/.fedora and try again ?
I removed fedora_session and openidbaseclient-sessions.cache from that directory. When rerunning the command I had to enter my username and password which I didn't before. However it still failed with the same error.
fedora_session
openidbaseclient-sessions.cache
Why is the status "waiting on reporter"?
Metadata Update from @pingou: - Issue priority set to: Waiting on Assignee (was: Waiting on Reporter)
Because it wasn't changed this morning after you answered
One idea on this one: nginx and openshift both send 504 errors if the server takes too long to answer.
This is a large update (166 builds). So if bodhi takes a little while (more than 60 seconds for nginx, I haven't found the value for openshift yet) to respond, either nginx or openshift will raise that 504 error.
Err, it's not nginx, it's gunicorn that we're using there...
So both gunicorn and openshift have been instructed to wait 180s before killing the request (cf: https://pagure.io/fedora-infra/ansible/c/116987403f154266f17c8eb604b8b3b603cf31cb?branch=master and https://pagure.io/fedora-infra/ansible/c/97db4e4c5f6b3f7276833493ce879ac868659509?branch=master )
Bodhi has been rebuilt in openshift.
@rjones could you give it a try now?
Metadata Update from @pingou: - Issue priority set to: Waiting on Reporter (was: Waiting on Assignee)
bodhi - 2020-09-16 13:47:54 (karma 0) This update has been submitted for stable by rjones.
Looks good! I guess we can close this now thanks.
Metadata Update from @rjones: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Arf, I forgot to ask if you could check how long it took the command to run. Any guess estimate?
Metadata Update from @pingou: - Issue untagged with: low-gain, low-trouble - Issue assigned to pingou - Issue tagged with: medium-gain, medium-trouble
Not sure. It certainly wasn't instant, so maybe a few mins.
Log in to comment on this ticket.