Our cancidate and production container registries are growing without much bound currently.
See:
https://admin.fedoraproject.org/collectd/bin/graph.cgi?hostname=oci-registry01.iad2.fedoraproject.org;plugin=df;plugin_instance=srv-registry;type=df_complex;begin=-9022400
and
https://admin.fedoraproject.org/collectd/bin/graph.cgi?hostname=oci-candidate-registry01.iad2.fedoraproject.org;plugin=df;plugin_instance=srv-registry;type=df_complex;begin=-11622400
We have a manual script in playbooks/manual/oci-registry-prune.yml but I'm not convinced it's understanding the new ostree containers and flatpak containers. It seems to operate on tags and deletes everything older than 30days, which isn't likely what we want anyhow.
So, We need to:
Come up with a retention plan. What do we keep and for how long?
Implement something that enforces that. cron job or whatever, just not anything manual. ;)
Note that even if we move to quay.io, we likely want to prune some things anyhow, not keep everything forever.
Any input welcome!
CC: @cverna @otaylor @dustymabe @siosm
The current playbook we have is for the candidate registry, we had the discussion in the container SIG (https://pagure.io/ContainerSIG/container-sig/issue/33) and thought that 1 months was good enough there.
I think that the ansible module should work for both ostree and flatpak containers since it is using the registry APIs and nothing specific to the actual container image itself. But folks could take a look at it here https://pagure.io/fedora-infra/ansible/blob/main/f/library/delete_old_oci_images.py
For the production registry, I think we should look in terms of if a Fedora version is EOL or not. Something like we keep container images up to 2 EOL version (for example we would have now 35, 36, 37, 38, 39, 40) and when 37 gets EOL we would prune the images based on F35.
Metadata Update from @phsmoura: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops
Please keep at least the latest base image for all Fedora release. All other container images for EOL'ed images can be pruned.
For Quay.io, we should enforce a default lifetime for all image by default: https://docs.projectquay.io/use_quay.html#tag-expiration
We could set this to 2 years for example for all images and override it on a per image basis.
Just to go over what we have/produce... each day we:
for each release rawhide f39 f38-updates f37-updates f39-updates-testing f38 updates-testing f37-updates-testing a fedora-base image a fedora-base-minimal image a fedora-toolbox image
for rawhide and 39 we also do the 4 ostree desktop variants right?
Are all of those useful all the time? Can they be good for rolling back/bisecting problems?
It's unclear to me how much space things are using (is there any way to get that sort of information from a registry?) or how often people want to use older images for whatever reason.
I would think it might be useful to keep eol initial images at release and the last updated one for historical reasons?
[backlog refinement] The discussion for this ticket is still open.
Login to comment on this ticket.