to do a delete all the images that are older than one month

I assume there will also be a condition that the image is not referenced as :latest, right?

@ttomecek While I can't answer that question (not sure how it's implemented), things should not need to stay long on the candidate registry: users really should be pulling from registry.fp.o, and so should builds.
Having users or CI pull from candidate-registry regularly is not sustainable.

Yes the candidate-registry should be used only during testing of an bodhi update and then when the update is stable the container will be available in registry.fp.o.

So I think 1 month is long enough to keep things on the candidate-registry

@cverna if that is just about the candidate registry I don't see any issues with that as anything that "matters"(all promoted images) will be preserved in the "production" registry, right?

would be possible to prune them faster once they are pushed to stable?

Globally, the image TTL on candidate is 1 month, but images that are pushed to registry.fp.o are pruned 3 days after they landed registry.fp.o or something similar

@cverna if that is just about the candidate registry I don't see any issues with that as anything that "matters"(all promoted images) will be preserved in the "production" registry, right?

Yes I have opened another ticket to discuss the production registry. I agree with you anything that matters should not be on the candidate registry.

would be possible to prune them faster once they are pushed to stable?

I guess we could listen to bodhi fedmsg and trigger to cleanup but I am not sure this is worth the effort tbh.

Globally, the image TTL on candidate is 1 month, but images that are pushed to registry.fp.o are pruned 3 days after they landed registry.fp.o or something similar

Yes that's a good idea, I think this is worth keeping in mind if we still have issues with once the 1 month old pruning is in place.

Seems this project might be useful https://github.com/andrey-pohilko/registry-cli

+1 for the 1 month then

@cverna the project looks interesting! Would be awesome if skopeo knew all of that.

I have created an Ansible Module [0] and Ansible playbook [1] to do the cleanup. I have also added a nagios check on /srv/registry for disk space so we should get an nagios alert when we are under 20% free disk space.

I am considering adding the playbook to the master playbook which is ran every night, that way we should always have enough space on the registries.

Closing this ticket.

[0] - https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/library/delete_old_oci_images.py
[1] - https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/playbooks/manual/oci-registry-prune.yml