https://bodhi.fedoraproject.org/updates/FEDORA-CONTAINER-2020-f45dce1436 seems to be breaking bodhi updates pushes.
It's not in candidate registry for some reason, so it fails when trying to copy it to registry...
Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: [2020-11-26 00:34:18,260: ERROR/ForkPoolWorke r-27] Exception in ComposerThread(f34-container-updates-testing) Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: Traceback (most recent call last): Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: File "/usr/lib/python3.8/site-packages/bodh i/server/tasks/composer.py", line 395, in work Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: self._compose_updates() Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: File "/usr/lib/python3.8/site-packages/bodh i/server/tasks/composer.py", line 857, in _compose_updates Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: copy_container(build, destination_tag=dtag) Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: File "/usr/lib/python3.8/site-packages/bodhi/server/util.py", line 1231, in copy_container Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: cmd(skopeo_cmd, raise_on_error=True) Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: File "/usr/lib/python3.8/site-packages/bodhi/server/util.py", line 795, in cmd Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: raise RuntimeError(msg) Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: RuntimeError: /usr/bin/bodhi-skopeo-lite copy docker://candidate-registry.fedoraproject.org/trojan:0-2 docker://registry.fedoraproject.org/trojan:0-2 returned a non-0 exit code: 1 Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: [2020-11-26 00:34:18,263: INFO/ForkPoolWorker-27] Compose object updated. Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: [2020-11-26 00:34:18,263: INFO/ForkPoolWorker-27] Thread(f34-container-updates-testing) finished. Success: False Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: [2020-11-26 00:34:18,276: INFO/ForkPoolWorker -27] Compose object updated. Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: [2020-11-26 00:34:18,277: ERROR/ForkPoolWorker-27] ComposerThread failed. Transaction rolled back. Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: Traceback (most recent call last): Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: File "/usr/lib/python3.8/site-packages/bodhi/server/tasks/composer.py", line 325, in run Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: self.work() Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: File "/usr/lib/python3.8/site-packages/bodhi/server/tasks/composer.py", line 395, in work Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: self._compose_updates() Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: File "/usr/lib/python3.8/site-packages/bodhi/server/tasks/composer.py", line 857, in _compose_updates Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: copy_container(build, destination_tag=dtag) Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: File "/usr/lib/python3.8/site-packages/bodh i/server/util.py", line 1231, in copy_container Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: cmd(skopeo_cmd, raise_on_error=True) Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: File "/usr/lib/python3.8/site-packages/bodh i/server/util.py", line 795, in cmd Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: raise RuntimeError(msg) Nov 26 00:34:18 bodhi-backend01.iad2.fedoraproject.org celery-3[1313]: RuntimeError: /usr/bin/bodhi-skopeo-lite copy docker://candidate-registry.fedoraproject.org/trojan:0-2 docker://registry.fedoraproject.org/trojan:0-2 returned a non-0 exit code: 1
Should f34 container updates be going thru bodhi?
In any case we need to fix this asap, because all updates flow is blocked until it is.
CC: @mohanboddu @cverna @pingou @humaton
The build linked from that update is from august 2020, and I believe that we are pruning the candidate registry to delete builds that are older than 30 days.
That would explain why the image is not there anymore, there is a more recent build Nov 21st that was done --> https://koji.fedoraproject.org/koji/buildinfo?buildID=1644398 (trojan:0-4 instead of trojan:0-2) so I think we need to modify that update to use the latest build.
So I have unlocked the update and reset the request status to None so that the update would not be included in the push.
None
I have also asked the packager to modify his update to use the latest build.
So updates are not blocked anymore.
Also created an upstream issue to handle that exception https://github.com/fedora-infra/bodhi/issues/4164
Closing as upstream issue
Metadata Update from @zlopez: - Issue close_status updated to: Upstream - Issue status updated to: Closed (was: Open)
After some discussion it will be better to leave the ticket here
Issue status updated to: Open (was: Closed)
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: dev, medium-gain, medium-trouble
This happened again today with https://bodhi.fedoraproject.org/updates/FEDORA-CONTAINER-2020-a3e5dcfe88
It was an update from 2 months ago that decided to go stable today...
I unlocked it and unpushed it.
Looks like its gating status was changed from 'Waiting' (set two months ago), to 'ignore' three days ago. So for 2 months it was waiting on gating and 3 days ago something changed and unblock the update.
I remember Clément working on a new bodhi release, but iirc it hasn't been pushed yet.
My guess is greenwave, I see it was re-deployed 3 days ago, so that likely fixed something that was broken before.
Yep. I added f34 containers/flatpak/modules to greenwave... so that was likely it.
So are we missing something in the branching SOP or did we just miss that step?
So, I am really not sure.
When we branched @humaton added fedora-34 there... but to the 'taskotron' sections, which I am pretty sure are ignored now that taskotron is gone.
I attempted to update greenwave config in ansible f9e7d727668321146139b1ac3b65a91dca49ff58 but I am not sure I got it right.
Whats the difference there between subject_type: koji_build and subject_type: bodhi_update ? Should we have no requirement for koji_build (thats the way it is now, but wasn't before I changed it), or should we not have it there?
"Whats the difference there between subject_type: koji_build and subject_type: bodhi_update ?"
Subject types are to do with the resultsdb queries greenwave runs. So if you look at a resultsdb result, one of the fields is "type". When you do a Greenwave query for "subject_type" "koji_build", it will...usually...look up ResultsDB results with "koji_build" as the type. When you do a Greenwave query for subject_type "bodhi_update", it will look up ResultsDB results with "bodhi_update" as the type.
There is some icky abstraction and configurability around this stuff at the Greenwave level that's basically to do with dealing with poor choices we have made in various CI systems when reporting to resultsdb. Various config options for Greenwave subject types cause different resultsdb queries to happen (including possibly multiple ones). You can find all of those in the greenwave source.
Practically speaking, for Fedora, we need policies for the koji_build and bodhi_update subject types because that's what Bodhi asks Greenwave for when it wants decisions and results. It expects the koji_build query to find results from Fedora CI (and formerly from Taskotron), and the bodhi_update query to find results from openQA. CI tests at the package level, so it submits results with koji_build as the type and an identifier that identifies the package that was tested. openQA tests at the update level, so it files results with bodhi_update as the type and an identifier that identifies the update that was tested. Bodhi is interested in results both for tests of the update and of any of the packages included in the update, so it asks Greenwave for both, and our Greenwave policy needs to cover that.
koji_build
bodhi_update
ok, it all becomes more clear and I think we know what happened here now. :)
Thanks @adamwill for all the Pr's and comments.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.