[backlog refinement]
We didn't have capacity to investigate this yet, but it's still something we want to do.

The comment linked above: https://pagure.io/fedora-infrastructure/issue/10373#comment-765006 is correct and current.

We were blocked on Flatpaks because of the lack of support for OCI images/manifests, which has been resolved now for quay.io, and in addition Flatpak since switched to using the older Docker v2 image format.

We'd still need to keep flatpak-indexer running and expose its URL at https://registry.fedoraproject.org/index (this URl is in the configuration of Fedora systems so can't change), but the images could hosted on quay.io. [The index points to the registry - it doesn't have to be on the same toplevel URL]

The Fedora CoreOS team is interested in this migration happening. We have a few "Fedora" containers we are going to start to publish. We met with the team previously and all decided that since quay.io was the end goal for Fedora containers that we (FCOS) would start to publish our Fedora CoreOS containers in quay.io from their inception (more context in https://pagure.io/fedora-infrastructure/issue/10702). Needless to say all of Fedora's containers should be accessible from a single source (reducing confusion), so the quicker the migration the better so that we don't have some containers coming from one registry and others from another.

The fedora-toolbox images for Toolbx are also currently hosted on registry.fedoraproject.org. We are open to moving to quay.io but we need some migration period because the URL to the image is baked into the toolbox RPM.

It looks the above comment was written by a bot.

Yeah, and a spam link. ;( Deleting it/blocking.

[backlog_refinement]
We still want to do this, but lacking free cycles.

I'm currently looking for stakeholders in this investigation to gather all the requirements. Till now I identified these users of registry.fp.o (with corresponding point of contact):

• Fedora Flatpaks @otaylor

EDIT: Removing OSTree variants as those are already on quay.io
EDIT2: Removing CoreOS and Toolbx as they already moved

I can do Silverblue/Kinoite/Sericea/Onyx. We're "already" on Quay.io so nothing to do for us: https://quay.io/repository/fedora/fedora-silverblue

@siosm In this case this investigation doesn't affect you. I see the last update on registry.fp.o for fedora-kinoite, fedora-silverblue and fedora-sericea is from May or March, so I assume these are old data and we don't need to preserve them.

Yes, those repos are not officially used yet (neither on r.fp.o or quay.io).

For the Container SIG (https://fedoraproject.org/wiki/Container_SIG), this would "solve":
- https://pagure.io/ContainerSIG/container-sig/issue/57
- https://pagure.io/ContainerSIG/container-sig/issue/58

CoreOS is also already on Quay.io

It seems that even the Toolbx is already on quay.io. So this means that he only user of registry.fp.o right now are Fedora Flatpaks.

The only users of the current registry I know of are:
- Fedora base images containers
- Fedora toolbox containers
- Flatpaks

All of those are still on registry.fp.o

@rishi Could you confirm that the Toolbox is still using registry.fp.o?

@siosm Who I can contact about Fedora base images containers?

CC @cverna for the base container image

Some background/info:

• The Fedora Base image and Minimal Base and Toolbox are all made via pungi / releng. For rawhide we push to BOTH registry.fedoraproject.org and quay.io. For releases, it's manual still I think, and we also push to both places. So I would not say they moved, more that that are in both could easily adjust to registry.fp.o going away. ;)

• In addition to registry.fedoraproject.org we have candidate-registry.fedoraproject.org. This is used by bodhi (ie, things are pushed to candidate-registry for testing and then registry for release). We would need to adjust bodhi in a post registry.fedoraproject.org world.

• Is there any way to redirect registry.fedoraproject.org to quay.io? Or are we going to have to try and get docker/podman/users to update over a long time before just taking it down?

Thanks for looking into this @zlopez !

Some background/info:
[...]
* In addition to registry.fedoraproject.org we have candidate-registry.fedoraproject.org. This is used by bodhi (ie, things are pushed to candidate-registry for testing and then registry for release). We would need to adjust bodhi in a post registry.fedoraproject.org world.

IIRC: OSBS pushes to candidate-registry.fedoraproject.org, then bodhi copies to to registry.fedoraporject.org, using the 'testing' label for testing, and the 'latest' label for stable releases.

• Is there any way to redirect registry.fedoraproject.org to quay.io? Or are we going to have to try and get docker/podman/users to update over a long time before just taking it down?

I bet this could be done with mod_proxy / mod_rewrite.

• Proxy requests to https://registry.fedoraproject.org/v2/REPO/ to https://quay/v2/fedora/REPO/
• Except return a 307 for https://registry.fedoraproject.org/v2/REPO/blobs/* - this is documented as being required to be supported by the distribution spec.

Where we really only care about REPO of fedora and fedora-toolbox (and will stop carrying about the latter once current releases go EOL and we stop rebuilding the toolbox for them.)

So to summarize all this, we have three stakeholders that are currently using registry.fp.o:

• Fedora Flatpaks @otaylor
• Fedora base/minimal images @cverna
• Fedora toolbox @rishi

And we would need to redirect OSBS and Bodhi to use quay.io when we move.

So to summarize all this, we have three stakeholders that are currently using registry.fp.o:

• Fedora Flatpaks @otaylor
• Fedora base/minimal images @cverna

I'm not sure how much time cverna has anymore. I would say the containers sig, but thats not really active. It might be just releng here.

• Fedora toolbox @rishi

And we would need to redirect OSBS and Bodhi to use quay.io when we move.

yes, and any external users trying to go to it also.

I finished the ARC investigation for migration. Looking forward for any feedback :-)

Looks like we have some on the list. ;)

We should probibly contact quay.io admins and ask them if there's any issues with moving there or any plans they have that would make that a bad idea.

I noticed the feedback, let me contact quay.io admins and ask them.

I sent e-mail to a contact in quay.io with these questions:

1) Mock switched to "--use-bootstrap-image" (podman pulling images
from various registries by default) and we had no single issue reported
against the Fedora's registry, but CentOS (on quay.io) gives us random
"pull" failures:

https://github.com/rpm-software-management/mock/issues/1191

Are you aware of this issue?

2) Quay.io is moving into console.redhat.com[2], which makes it even less
fun since RH accounts for the console require giving a lot more
information.

Do we need to be Red Hat customers to access that? Could it be possible to
allow Fedora Account System login?

3) There is a rate limiting enabled for pulling on quay.io [3]. Could it be possible to
remove that if some Fedora services start hitting that?

Let's wait for response.

And I already got the response:

Thanks for reaching out- we'd certainly like to support your migration.
Fedora makes perfect sense as a tenant on quay.io. Let me try to answer your questions:

1) Not aware of this issue- I don't believe anyone has raised a support ticket with us on it.
Wasn't clear to me from the GH issue if you had a stable reproducer. If you do,
please feel free to raise a bug report at https://issues.redhat.com/projects/PROJQUAY
and we can take a look.

2) Our long term plan is to move all authenticated web UI access to console.redhat.com
but we will keep our quay.io web UI available for unauthenticated access 
(e.g. google search results linking to public images). So only users who need authenticated
access to your namespace(s)- for example to administer a Team, etc.. would need to sign up
for a Red Hat Account. Robot account / docker CLI access will still work directly and not require
RH SSO- so your automation can still push images, etc..

We have no plans to integrate the Fedora Account System login- but open to discuss what that
could look like (esp. if it supports OIDC).

3) We can disable the rate limiting on your namespace(s)- it's usually not a problem, we do this
for other Red Hat teams (e.g. Openshift). I would be interested to understand more of your
expected traffic loads for push/pull so we can plan accordingly on our side.

FWIW the FCOS team occasionally has trouble with quay, but only on our aarch64 builder in AWS. There is also some report from the podman team having trouble.

The issues always appear to be intermittent/flakiness and not a consistent failure. Nevertheless, it causes failures.

I created a new ticket for actual migration as the investigation is now finished and I'm closing this one as finished.

I also had meeting with quay.io folks last week. I pointed them to mock issue and we discussed the potential risk of rate limiting (according to statistics I gathered from registry.fedoraproject.org, we shouldn't hit that at all), but just in case we do, we can request to add the service to quay.io whitelist. Otherwise there doesn't seem to be anything blocking the amigration and they will help us if we hit any roadblock.

It seems that even the Toolbx is already on quay.io.

I don't know what that repository on quay.io is. It's definitely not the default or canonical source of the fedora-toolbox images.

There's https://quay.io/organization/toolbx but it's currently only for images outside the Fedora family.

@rishi Could you confirm that the Toolbox is still using registry.fp.o?

Yes, Toolbx still uses registry.fedoraproject.org for its fedora-toolbox images. These images are now produced by Fedora's nightly composes.