As CentOS and Fedora are using shared IPA backend for authentication, I'd request that nothing touching enrolled centos machines in IPA would be done (manually or through scripts) I had to just waste my time this morning investigating why ipsilon (https://id.stg.centos.org) wasn't allowing anyone to auth (and so no openidc for services using our ipsilon instance)
Someone (who ? or a script ?) removed ipsilon.stg.iad2.centos.org from the ipsilon HBAC rule, denying so all auth requests .
ipsilon.stg.iad2.centos.org
Can you identify the root cause and ensure it wouldn't happen again please ?
Thanks a lot
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, high-gain, ops
seems related to https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_92 ...
Metadata Update from @arrfab: - Issue untagged with: Needs investigation, high-gain, ops - Issue priority set to: Needs Review (was: Waiting on Assignee)
The change you are referring to happened 4 years ago. So I assume that didn't caused the machine to be removed.
Metadata Update from @kevin: - Issue assigned to kevin
It's actually https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101
It was using the wrong hostname... but that was set in 2021? b8e6754f97c (Aurélien Bompard 2021-03-22 17:07:45 +0100 101) host: "{{ (env == 'production')|ternary('ipsilon.iad2.centos.org', 'centos-ipa-client02.stg.iad2.fedoraproject.org') }}"
anyhow, I changed it to ipsilon.stg.iad2.centos.org
If you can confirm it's fixed / working?
forgot to give feedback (PTO) but yes, now working again ... Is there a way to ensure that ansible would not remove hosts not managed by itself ? Otherwise, just ensure that it's documented somewhere as that means that on my centos ansible infra side I'll never be autonomous to deploy/enroll a machine as then Fedora ansible would remove it :/
Well, I mean we could remove that task from our playbooks and you could depend on yourself to create that ?
Otherwise ansible is going to setup the thing thats defined.
Do you want us to remove that setup from our side?
@kevin : even if you remove that block, I guess that the other one about Fedora host would then remove the existing centos machines again .. so isn't there a parameter for that ipahbacrule to not purge things not declared ? if not then we'll have to still hard-code centos machines there but with dc move and so multiple machines that will need to be enrolled, I'd like that to be documented at least :)
ipahbacrule
well, my reading / understanding of it is that:
https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_85 creates the 'ipsilon' hbac rule and
https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101 has action: member, so it adds those members/hosts.
So, I guess we could change the first one to just confirm/add the fedora ipsilon servers and then it wouldn't recreate the rule... but then the rule would have to be... manually created or something?
Would this be high trouble, medium trouble, low trouble or a timeboxed investigation?
What if we do this:
diff --git a/playbooks/groups/ipsilon.yml b/playbooks/groups/ipsilon.yml index 8c95c4344a..1f4636ece3 100644 --- a/playbooks/groups/ipsilon.yml +++ b/playbooks/groups/ipsilon.yml @@ -89,6 +89,14 @@ hbacsvc: - ipsilon usercategory: all + ipaadmin_password: "{{ ipa_admin_password }}" + tags: + - ipsilon + + - name: Allow login through ipsilon for the Fedora instance + ipahbacrule: + name: ipsilon + action: member host: "{{ (env == 'production')|ternary(groups['ipsilon'], groups['ipsilon_stg']) }}" ipaadmin_password: "{{ ipa_admin_password }}" tags:
ie, we remove the host call from the one that makes the hbrbac rule, so it should just create it if it doesn't exist. Then we populate the fedora hosts in a seperate host level task.
just FYI : it happened again and so I guess someone rolled out the unmodified ansible for rdu3, and so it removed access for enrolled rdu3 hosts for centos :/ .... Can we fix that so that I don't have to manually add back myself enrolled servers, removed by fedora automation ? TIA :)
I think I see the issue here, there is an exception for centos instances, but only for iad2 centos instances.
- name: Allow login through ipsilon for the CentOS instance ipahbacrule: name: ipsilon action: member host: "{{ (env == 'production')|ternary('ipsilon.iad2.centos.org', 'ipsilon.stg.iad2.centos.org') }}" ipaadmin_password: "{{ ipa_admin_password }}" tags: - ipsilon
And I will check if the rdu3 machines are in the ipsilon and ipsilon_stg groups.
ipsilon
ipsilon_stg
I see that even the ipsilon and ipsilon_stg doesn't contain iad2.centos.org machines, because we don't want to deploy them, so that is OK.
iad2.centos.org
Here is the PR that adds the rdu3 machines to hbacrule.
I merged the PR and run the ipsilon playbook and the machine seems to be correctly added to HBAC rule.
Let me close this as fixed. @arrfab feel free to reopen it, if we missed something.
Metadata Update from @zlopez: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)