bugzilla.redhat.com admins are making some changes to the way auth works.
See: https://listman.redhat.com/archives/bugzilla-announce-list/2022-February/msg00000.html
for details. Basically all the old auth methods will stop working and will in fact revoke access if you are using them. ;(
We have a number of things doing this we need to fix asap:
roles/bodhi2/base/templates/production.ini.j2:bugzilla_api_key = {{ bodhi_bugzilla_api_key }} roles/distgit/pagure/templates/pagure-sync-bugzilla.py.j2:BZPASS = '{{ bugzilla_password }}' roles/openshift-apps/the-new-hotness/templates/config.toml:password = "{{ upstream_release_bugzilla_password }}" roles/openshift-apps/the-new-hotness/templates/config.toml:api_key = "{{ upstream_release_bugzilla_api_token }}" roles/openshift-apps/toddlers/templates/fedora-messaging.toml:bugzilla_username = "{{ bugzilla_user }}" roles/openshift-apps/toddlers/templates/fedora-messaging.toml:bugzilla_password = "{{ bugzilla_password }}" roles/releng/templates/ftbfs.cfg.j2:username={{ ftbfs_bugzilla_account }} roles/releng/templates/ftbfs.cfg.j2:password={{ ftbfs_bugzilla_password }} roles/relvalconsumer/templates/bugzillarc.j2:api_key={{ relvalconsumer_bugzilla_api_key }}
and possibly more. :(
Metadata Update from @zlopez: - Issue tagged with: announcement, high-gain, high-trouble
We should aim to be in production with these changes before the beta freeze on 2022-02-22
Metadata Update from @kevin: - Issue priority set to: Waiting on Assignee (was: Needs Review)
The way I understood this is that you can still authenticate with username+password or api_key, providing you are using an updated python-bugzilla (3.2.0).
So, for example, review-stats or Bodhi, which both use python-bugzilla as middle layer, should have no problems with the bugzilla change. Am I wrong?
Replying to myself: yes, I was wrong.
I've updated review-stats dockerfile to use the updated python-bugzilla 3.2.0, but the staging instance has started to fail. Reading also the discussion on the devel mailing list, all the apps which are using username+password authentication need to be migrated to api keys. Those which are already using api key authentication through python-bugzilla should be fine.
So, please add review-stats to the above list of apps that need to be fixed. I've created https://pagure.io/fedora-infra/ansible/pull-request/970
So, looking a bit more:
bodhi is ok in prod, but still uses a password in staging? Anyone know why?
bugzilla2fedmsg is going to need changes. Anyone want to take a stab at it?
review-stats has the above pr. Thanks @mattia!
the-new-hotness is going to need changes. @zlopez can you make those?
toddlers is going to need changes. @pingou? or @abompard?
Finally, the ftbfs script is going to need changes. @churchyard can you do those, or would you prefer someone else do them?
@mohanboddu and @humaton are you on the releng changes? I only see the ftbfs script, but there might be others?
I can adapt the ftbfs script itself, but I need somebody to help me provide the token to it where it runs.
So, looking a bit more: bodhi is ok in prod, but still uses a password in staging? Anyone know why?
I assume no one has never created an api key in staging bugzilla to be used with staging Bodhi. There should be no problems to switch bodhi staging to bz api key authentication.
@kevin The ticket for the-new-hotness is here.
I already have everything prepared on production, so this is for the staging only. I need somebody from the infra ops to generate an API key for staging and then I just update the config for staging.
There is also a new version of hotness ready for release that is waiting till this is done.
The-new-hotness is now prepared for the change on both staging and production.
For those who want to try it now the staging Bugzilla already rejects unauthorized authentication methods.
https://pagure.io/releng/pull-request/10665
https://pagure.io/fedora-infra/ansible/pull-request/982
Updates:
I have created a staging bodhi bugzilla api key. The variable is bodhi_bugzilla_api_key_stg and the user is updates@fedoraproject.org (not updates@stg.fedoraproject.org as the bodhi config currently has). I can whip up a pr for this, or someone else can if they have time before I do. :)
review-stats vars are in and thats ready to push whenever it's updated to do stg also.
ftbfs needs a freeze break, but hopefully we have that and @humaton can push that tomorrow. Variables are in and the pr is ready to go.
That leaves us with 2:
More updates:
We just got a bit more time:
"Hi, due to a bug in Red Hat Bugzilla some users have not been able to create API keys, to give these users adequate time to prepare for this change it will be postponed to Monday the 14th of March 2022 at UTC 00:00.
Additionally the automated disablement of API keys and passwords has itself been disabled."
That leaves us with 2: bugzilla2fedmsg is going to need changes. Anyone want to take a stab at it? toddlers is going to need changes. @pingou? or @abompard?
I gave a look to bugzilla2fedmsg code, I'm not familiar with STOMP, but it seems to me that it only authenticates to the STOMP broker. I can't find anything in the code which uses the bugzilla.username and bugzilla.password set into the config file...
bugzilla.username
bugzilla.password
Commit 349238d2 relates to this ticket
Huh, yeah, I thought bugzilla2fedmsg just got the bug # and then looked it up for data, but it appears the STOP message actually contains the data? @abompard can you confirm?
ftbfs is done
That leaves us with just toddlers I think?
I create pull requests for toddlers today: Ansible PR - marked as freeze-break-request Toddlers PR
The new version of toddlers is now deployed in production. I will monitor it for a while, but everything seems OK for now.
Awesome. Then I think we are done.
Finger crossed.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)