While logged into batcave I try to run:
sudo rbac-playbook -l os_control_stg -t delete openshift-apps/fedora-ostree-pruner.yml sudo rbac-playbook -l os_control_stg openshift-apps/fedora-ostree-pruner.yml
Both fail. I also tried the above for coreos-koji-tagger and coreos-cincinnati with exactly same results.
Play recap:
[c4rt0@batcave01 ~][PROD-IAD2]$ sudo rbac-playbook -l os_control_stg -t delete openshift-apps/fedora-ostree-pruner.yml First Factor: Second Factor: NOTIFY: [rbac-playbook] SUCCESS c4rt0 ran openshift-apps/fedora-ostree-pruner.yml ('Details: \n' "limit: ['os_control_stg'] \n" 'check: False \n' "tags: ['delete'] \n" 'user: None \n' 'start_at_task: None \n' 'Sha256: 4f017baeac0d50bb4cd58919808a76f086a907ca7ea5a20430759358b2669620') EXECV: /usr/bin/sudo -i /bin/bash -i -c cd /srv/web/infra/ansible ; /usr/bin/python3 /usr/bin/ansible-playbook /srv/web/infra/ansible/playbooks/openshift-apps/fedora-ostree-pruner.yml -l os_control_stg -t delete PLAY [Make the app be real] *************************************************************************************************** TASK [openshift/object-delete : Delete object file ({{tmpfile.path}})] ******************************************************** Friday 09 May 2025 13:35:29 +0000 (0:00:01.196) 0:00:01.196 ************ Friday 09 May 2025 13:35:29 +0000 (0:00:01.195) 0:00:01.195 ************ skipping: [os-control01.stg.iad2.fedoraproject.org] TASK [openshift/object-delete : Delete project files ({{tmpfile.path}})] ****************************************************** Friday 09 May 2025 13:35:29 +0000 (0:00:00.077) 0:00:01.273 ************ Friday 09 May 2025 13:35:29 +0000 (0:00:00.077) 0:00:01.273 ************ changed: [os-control01.stg.iad2.fedoraproject.org] TASK [openshift/object-delete : Call `oc delete` on the object] *************************************************************** Friday 09 May 2025 13:35:30 +0000 (0:00:00.619) 0:00:01.893 ************ Friday 09 May 2025 13:35:30 +0000 (0:00:00.619) 0:00:01.892 ************ fatal: [os-control01.stg.iad2.fedoraproject.org]: FAILED! => {"changed": true, "cmd": "oc -n fedora-ostree-pruner delete project/fedora-ostree-pruner", "delta": "0:00:00.160224", "end": "2025-05-09 13:35:30.590958", "msg": "non-zero return code", "rc": 1, "start": "2025-05-09 13:35:30.430734", "stderr": "error: You must be logged in to the server (Unauthorized)", "stderr_lines": ["error: You must be logged in to the server (Unauthorized)"], "stdout": "", "stdout_lines": []} PLAY RECAP ******************************************************************************************************************** os-control01.stg.iad2.fedoraproject.org : ok=1 changed=1 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 logs written to: /var/log/ansible/fedora-ostree-pruner/2025/05/09/13.35.28 Friday 09 May 2025 13:35:31 +0000 (0:00:01.140) 0:00:03.034 ************ =============================================================================== openshift/object-delete ------------------------------------------------- 1.84s ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ total ------------------------------------------------------------------- 1.84s Friday 09 May 2025 13:35:31 +0000 (0:00:01.140) 0:00:03.033 ************ =============================================================================== openshift/object-delete : openshift/object-delete : Call `oc delete` on the object ------------------------------------- 1.14s openshift/object-delete : openshift/object-delete : Delete project files ({{tmpfile.path}}) ---------------------------- 0.62s openshift/object-delete : openshift/object-delete : Delete object file ({{tmpfile.path}}) ------------------------------ 0.08s
AFAIK this worked previously, it would be nice if it would work again. I'm afraid I can't provide logs due to insufficient permissions.
Ideally as soon as possible, but it isn't critical.
Odd. Do you mind if I re-run it to debug?
Nevermind. I see it.
Someone changed the kube config on os-control01.stg... I am not sure who or how it's not working, but it's not working. ;)
@dkirwan do you know who made those changes, or whats going on there?
Just in case anything like this happens in the future, I won't mind.
Thanks for looking into this!
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-trouble, medium-gain, ops
@c4rt0 can you please retest to see fi this is resolved now.
Metadata Update from @dkirwan: - Issue assigned to dkirwan
Thanks folks! I can confirm this is now working!
Metadata Update from @c4rt0: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Metadata Update from @dkirwan: - Issue status updated to: Open (was: Closed)
Metadata Update from @dkirwan: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.