I'm currently documenting the list of Fedora Copr credentials and how to maintain it, including adding new items and rotation. Currently, we use the ansible-private.git repo where we have no access.
ansible-private.git
Some credentials are relatively challenging to rotate, requiring tight coordination and timing to avoid extended downtimes. Considering the need for frequent rotations, we would have to ensure that a sysadmin-main member is on call with us frequently for a reasonable amount of time.
sysadmin-main
I'm curious about the preferred solution to this problem. Would it be acceptable to have a service similar to Hashi Vault within Fedora Infra, where each team has its namespace with read-write access? We currently have a private in-VPN deployment of Vault maintained by Red Hat IT, and it works reasonably well.
Alternatively, would it be acceptable to add Ansible Vault support to rbac-playbook? In this case, sysadmin-main members would only store a single team-specific credential into ansible-private.git, and teams could store ansible-vault-encrypted credentials into ansible.git. Thus, rbac-playbook still wouldn't require any Vault credentials at runtime.
ansible-vault
ansible.git
rbac-playbook
Is there any other, ideally better way to deal with this problem?
Metadata Update from @phsmoura: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, low-trouble, ops
So, I can try a more verbose answer later, but short answer: no, not right now, but we plan to move to AWX, and once we start doing that we likely will look at using vault or something else.
This might be better as a discussion thread on discuss?
But anyhow, can try and answer more later.
Do we actually have to wait for https://pagure.io/fedora-infrastructure/issue/11377?
I can start the discussion (if you prefer it). Though not sure it will help, this is rather an issue for us to solve.
Another option that comes to my mind is to use external credential provider from ansible.git, e.g. Bitwarden, and store just the token into ansible-private.git.
Well, now I see bitwarden-cli review #1 and bitwarden-cli review #2, both staled. It's a Node app, some libraries bundled -- this is not something I could personally help with and maintain. The thing is that /bin/bw is a pre-requisite for Bitwarden Ansible plugin. So i think that using a free Vault through https://portal.cloud.hashicorp.com would be the easiest option (well integrated into Fedora and Ansible Community General). We could in theory also start our own separate credentials cluster (the portal provides some ready-made button for starting a custom vault in EC2) but the default password store could be "OK" enough?
/bin/bw
I am not really in favor of any external 3rd party system off hand. That means we depend on them, and the network between us.
I suppose if copr wanted to switch to something, they could... but of course we will probibly have to have the creds to access that in our ansible-private for now.
Would it help here if we promoted you to sysadmin-main (and thus ansible-private access). Currently that repo is all or nothing, ie, if you have access you can see all passwords/tokens/auth/etc.
I was hoping to get our test AWX up and running and then consider how we wanted to move things, I don't want to be in a rush and decide on something thats insecure or has dependencies we don't like down the road. The AWX move may also move our ansible repo to a new setup (to decouple local paths), but that also needs to be discussed determined.
Good point with the connectivity requirement.
Yes, this is one of the options I touched (we need to allow automatic decryption for those who run the playbooks).
Yes, having access (temporary) access would minimize the barrier for us to do the rotation.
Understood. For the record, AWX actually has its own credential storage, but I'm not certain about the future policy within Fedora Infra. In the Red Hat 'in-VPN' use-case, we use AWX more or less as a 'credentials proxy,' and the original passwords are stored in external vault. I assume the reason is to prevent placing too many expectations on the AWX architecture itself (due to potential data losses during re-deployment).
ok. I can propose adding you to main. I honestly would have suggested that before, but I thought you wouldn't be very interested since you are focused on copr... ;)
Well, the point in deploying and testing AWX is so that we can actually see how things work and how we want our workflow to work. It's hard to speculate on things before we have a test setup we can actually look at and see whats possible/painfull, etc.
Thank you. I'm interested, but I wish I could allocate some time to other infrastructure tasks to balance the rights that would be granted to me.
I believe the level of convenience with AWX will help everyone. There are some issues (for me that would be the "live" playbook log/stdout isn't live enough, and that playbook environment has some weird caching mechanisms). But the Copr team will be a happy early adopter I'm sure.
Ok, closing. At least till we have AWX up&running.
Metadata Update from @praiskup: - Issue close_status updated to: Initiative Worthy - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.