I cannot work on a bodhi update that has close to 300 packages. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-d7c1181e05
If this update were to just go through the system, this wouldn't be a problem. But it is for EPEL8 and I need to time it so that it goes out around the same time as CentOS 8.2.
Because I wasn't able to update it, it started to get pushed to stable last week. The only command that went through was to "unpush" ... which I thought would unpush it from stable, but instead unpushed it from epel-testing.
So now I am stuck with the packages in epel-testing-candidate. I could create several smaller updates (about 50 packages each), but then they would have to wait the mandatory 2 weeks, which might be too long. It's getting close to a month since RHEL 8.2 was released.
What do I want: 1 - The packages currently in that update to get pushed back into epel-testing. 2 - Be able to release those packages to stable when it is time to release them, not having to wait a mandatory 2 weeks.
2020/05/27 (At least for part 1)
Webpage error: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-d7c1181e05 504 Gateway Time-out The server didn't respond in time.
Command line error: bodhi updates query --updateid FEDORA-EPEL-2020-d7c1181e05 Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/fedora/client/openidbaseclient.py", line 249, in send_request data = output.json() File "/usr/lib/python3.6/site-packages/requests/models.py", line 897, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib64/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/bin/bodhi", line 11, in <module> load_entry_point('bodhi-client==4.0.2', 'console_scripts', 'bodhi')() File "/usr/lib/python3.6/site-packages/click/core.py", line 721, in call return self.main(args, kwargs) File "/usr/lib/python3.6/site-packages/click/core.py", line 696, in main rv = self.invoke(ctx) File "/usr/lib/python3.6/site-packages/click/core.py", line 1065, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3.6/site-packages/click/core.py", line 1065, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3.6/site-packages/click/core.py", line 894, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/lib/python3.6/site-packages/click/core.py", line 534, in invoke return callback(args, kwargs) File "/usr/lib/python3.6/site-packages/bodhi/client/init.py", line 232, in wrapper method(*args, kwargs) File "/usr/lib/python3.6/site-packages/bodhi/client/init.py", line 578, in query resp = client.query(rows_per_page=rows, kwargs) File "/usr/lib/python3.6/site-packages/bodhi/client/bindings.py", line 109, in wrapper result = method(*args, kwargs) File "/usr/lib/python3.6/site-packages/bodhi/client/bindings.py", line 402, in query return self.send_request('updates/', verb='GET', params=kwargs) File "/usr/lib/python3.6/site-packages/fedora/client/openidbaseclient.py", line 258, in send_request 'output': to_bytes(output.text), fedora.client.ServerError: ServerError(https://bodhi.fedoraproject.org/updates/, 504, Error returned from json module while processing b'https://bodhi.fedoraproject.org/updates/': b'Expecting value: line 1 column 1 (char 0)' b"<html><body>
Metadata Update from @cverna: - Issue assigned to cverna
Metadata Update from @cverna: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-trouble, medium-gain
Bodhi takes too long to answer that request so the OpenShift haproxy kills it. I have manually updated the config on os-master01 to allow longer requests.
oc -n bodhi annotate route bodhi-web --overwrite haproxy.router.openshift.io/timeout=120s
I ll update the bodhi playbook to add that setting since I think it does not hurt to have it.
@tdawson you should be able to load that page now (It might takes some time and maybe a refresh to warm the cache).
Ansible was updated in https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=d685007fd63643b8652ee1cf760e4498e129d4ff
Closing this now, please feel free to reopen if needed.
Metadata Update from @cverna: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
It looks like things have slowed down so much that the current timeout isn't long enough. Others have remarked on it for EPEL, so I know it isn't just me. I am getting the same webpage error as before. But even a simple one package override is timing out as well.
Command Line error: $ bodhi overrides save kf5-knewstuff-5.68.0-1.el8 Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/fedora/client/openidbaseclient.py", line 249, in send_request data = output.json() File "/usr/lib/python3.6/site-packages/requests/models.py", line 897, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib64/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last): File "/usr/bin/bodhi", line 11, in <module> load_entry_point('bodhi-client==4.0.2', 'console_scripts', 'bodhi')() File "/usr/lib/python3.6/site-packages/click/core.py", line 721, in call return self.main(args, kwargs) File "/usr/lib/python3.6/site-packages/click/core.py", line 696, in main rv = self.invoke(ctx) File "/usr/lib/python3.6/site-packages/click/core.py", line 1065, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3.6/site-packages/click/core.py", line 1065, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3.6/site-packages/click/core.py", line 894, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/lib/python3.6/site-packages/click/core.py", line 534, in invoke return callback(args, kwargs) File "/usr/lib/python3.6/site-packages/bodhi/client/init.py", line 232, in wrapper method(*args, kwargs) File "/usr/lib/python3.6/site-packages/bodhi/client/init.py", line 935, in save_buildroot_overrides kwargs) File "/usr/lib/python3.6/site-packages/bodhi/client/init.py", line 261, in _save_override expired=kwargs.get('expire', False)) File "/usr/lib/python3.6/site-packages/bodhi/client/bindings.py", line 109, in wrapper result = method(*args, kwargs) File "/usr/lib/python3.6/site-packages/bodhi/client/bindings.py", line 462, in save_override 'overrides/', verb='POST', auth=True, data=data) File "/usr/lib/python3.6/site-packages/fedora/client/openidbaseclient.py", line 258, in send_request 'output': to_bytes(output.text), fedora.client.ServerError: ServerError(https://bodhi.fedoraproject.org/overrides/, 504, Error returned from json module while processing b'https://bodhi.fedoraproject.org/overrides/': b'Expecting value: line 1 column 1 (char 0)' b"<html><body>
Metadata Update from @tdawson: - Issue status updated to: Open (was: Closed)
It's unclear if this is/was caused by the same timeout issue or not... ;(
We tried a number of things the other day to restore service after this was re-opened... and I think that it's working again, but the root cause of the issue is not known.
Is it working now for you?
Yes and no. If I try several times I am able to look at the bodhi webpage. I am also able to change the status ... sometimes. I am unable to edit.
This has become rather critical for me. I need this update to be pushed to stable, today. But things are taking forever for me to even see what is happening. And each time I see, it's taking a step backwards instead of forwards.
I am changing this request. Please, whoever has permission, please push this update to stable. Today.
Yes and no. If I try several times I am able to look at the bodhi webpage. I am also able to change the status ... sometimes. I am unable to edit. This has become rather critical for me. I need this update to be pushed to stable, today. But things are taking forever for me to even see what is happening. And each time I see, it's taking a step backwards instead of forwards. I am changing this request. Please, whoever has permission, please push this update to stable. Today.
I have changed the update to set the stable request, so hopefully it will get pushed today if nothing else breaks.
Thank You
Currently all of the packages in that update only have the tag "epel8-testing" Bodhi if failing the push with this error:
FEDORA-EPEL-2020-d7c1181e05 ejected from the push because "Cannot find relevant tag for accounts-qml-module-0.7-1.el8. None of ['epel8-testing'] are in ['f32-container-updates-candidate', 'f33-updates-candidate', 'f32-modular-updates-candidate', 'f32-flatpak-updates-candidate', 'dist-6E-epel-testing-candidate', 'epel7-testing-candidate', 'dist-5E-epel-testing-candidate', 'f27-modular-updates-candidate', 'f33-container-updates-candidate', 'eln-updates-candidate', 'f30-modular-updates-candidate', 'f28-modular-updates-candidate', 'f28-container-updates-candidate', 'f30-container-updates-candidate', 'epel8-testing-candidate', 'f30-flatpak-updates-candidate', 'f31-modular-updates-candidate', 'f31-container-updates-candidate', 'f31-flatpak-updates-candidate', 'f29-modular-updates-candidate', 'f29-container-updates-candidate', 'f29-flatpak-updates-candidate', 'f22-updates-candidate', 'f21-updates-candidate', 'f25-updates-candidate', 'f24-updates-candidate', 'f23-updates-candidate', 'f26-updates-candidate', 'f27-updates-candidate', 'f28-updates-candidate', 'f32-updates-candidate', 'f30-updates-candidate', 'f31-updates-candidate', 'f29-updates-candidate', 'el8-modular-updates-candidate']."
Should I just be patient? Or is there something wrong with their tags?
Yes the builds were not tag properly, I have fixed that so hopefully the update will be pushed to stable later today.
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-d7c1181e05 is not yet pushed , the status is pending
Ok this was finally pushed to epel-8 stable, sorry it took so long. Please reopen the ticket if needed.
I have verified that it's fixed and available on the mirrors. Thanks for getting this done for me.
Login to comment on this ticket.