#11555 Remove PDC queries from ansible
Opened 7 months ago by humaton. Modified 21 days ago

Hi, Fedora project plans to sunset the PDC project.
In Ansible we are currently querying PDC to get active Fedora releases, in multiple places.
Please use Bodhi API to get the list of active releases.

/roles/mirrormanager/backend/templates/handle_propagation.sh
/roles/mirrormanager/crawler/files/check_propagation.sh
/roles/bodhi2/backend/templates/owner-sync-pagure.j2
/roles/distgit/pagure/templates/pagure-sync-bugzilla.py.j2

Active releases in bodhi:
curl -XGET --header "Content-Type: application/json" https://bodhi.fedoraproject.org/releases/?state=current| jq

Pending releases in bodhi:
curl -XGET --header "Content-Type: application/json" https://bodhi.fedoraproject.org/releases/?state=pending| jq


I will try to look into
loop @kevin @zlopez could you assign me the ticket ?

Metadata Update from @phsmoura:
- Issue assigned to seddik
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

7 months ago

I think perhaps some or all of these could just use our ansible variables for releases... under vars/all/

Those should describe the current and pending releases, etc.

Do you know why i'm getting this status ?? something strange ?

[seddik@log01 /][PROD-IAD2]$ curl -s -XGET --header "Content-Type: application/json" https://bodhi.fedoraproject.org/releases/?state=active | python -m json.tool  
{
    "status": "error",
    "errors": [
        {
            "location": "querystring",
            "name": "state",
            "description": "\"active\" is not one of disabled, pending, frozen, current, archived"
        }
    ]
}

@kevin at least you think that the bodhi API url should be defined as variable for templates ??

@seddik Sorry the URL has the wrong param, it should be current in the first query.

I suppose these all call the pdc url and could just use bodhi, but what I was meaning was that we have ansible variables for the current fedora releases. Like right now:

{{ FedoraCycleNumber }} = 38
{{ FedoraBranched }} = True, so that means that {{ FedoraBranchedNumber }} = 39
{{ FedoraRawhideNumber }} = 40
{{ FedoraPreviousCycleNumber }} = 37

With that you can construct the current non eol fedora releases.

However, in this case it might indeed be easier to use bodhi I guess, depending on what info it's looking for.
You might check the current data and see what best would recreate it.

Hope that helps.

[backlog_refinement]
@seddik Are you still working on this? Do you need some help?

@zlopez sorry for the delay
Yes, I syncing with @lenkaseg on this change

There are some differences between pdc and bodhi api ...
For example we don't have release_type key in pdc api, so the question what would be good to select from bodhi ??

Json output from bodhi api

{
    "page": 1,
    "pages": 1,
    "releases": [
        {
            "branch": "epel7",
            "candidate_tag": "epel7-testing-candidate",
            "composed_by_bodhi": true,
            "create_automatic_updates": null,
            "dist_tag": "epel7",
            "eol": null,
            "id_prefix": "FEDORA-EPEL",
            "long_name": "Fedora EPEL 7",
            "mail_template": "fedora_epel_legacy_errata_template",
            "name": "EPEL-7",
            "override_tag": "epel7-override",
            "package_manager": "unspecified",
            "pending_signing_tag": "epel7-signing-pending",
            "pending_stable_tag": "epel7-pending",
            "pending_testing_tag": "epel7-testing-pending",
            "released_on": null,
            "setting_status": null,
            "stable_tag": "epel7",
            "state": "current",
            "testing_repository": null,
            "testing_tag": "epel7-testing",
            "version": "7"
        }

Metadata Update from @zlopez:
- Issue untagged with: medium-gain
- Issue tagged with: high-gain

3 months ago

As I've commented in https://pagure.io/fedora-infra/ansible/pull-request/1599#comment-199496 I don't think this is something of simple solution.
The main problem is that PDC is queried to list active branches regarding each specific package, while Bodhi can only list active branches by releases, but that doesn't mean that each package is active in all releases...

I think we first need to write something to map active branches per package, as Bodhi cannot be queried for that. Or do we already have something?

Branches of a package can be retrieved from "https://src.fedoraproject.org/api/0/rpms/<package>/git/branches", though I cannot see any option to filter out branches on which a package is active. One option is to check in a lookaside cache, those retired_in_<release>.json, if the package is retired on a particular active release.
It's the output of this script: https://pagure.io/releng/blob/main/f/scripts/get_retired_packages.sh.
So making an intersection between active releases from bodhi and package branches from src.fp.o, filtered for packages that are retired (means present in a json file on the lookaside cache).
I wonder if there is a less complicated way to check active branches per package.

Branches of a package can be retrieved from https://src.fedoraproject.org/api/0/rpms/<package>/git/branches, though I cannot see any option to filter out branches on which a package is active. One option is to check in a lookaside cache, those retired_in_<release>.json, if the package is retired on a particular active release.
It's the output of this script: https://pagure.io/releng/blob/main/f/scripts/get_retired_packages.sh.
I wonder if there is a less complicated way to check active branches per package.

Dist-git no longer have the concept of active/inactive branch, PDC held that
info.
So at this point Bodhi is likely the best source of info for this, iiuc that's
what the pagure-dist-git auth script that's run on git push queries now, no?

So at this point Bodhi is likely the best source of info for this, iiuc that's
what the pagure-dist-git auth script that's run on git push queries now, no?

Yup, well, it's still in a stage of open PR :)
Not sure if it's transferable here. dist_git_auth.py runs a git command on the same machine as are the git repos of the packages. Would it work for mirrormanager and bodhi roles here?

So at this point Bodhi is likely the best source of info for this, iiuc that's
what the pagure-dist-git auth script that's run on git push queries now, no?

Yup, well, it's still in a stage of open PR :)
Not sure if it's transferable here. dist_git_auth.py runs a git command it on the same machine as are the git repos of the packages. Would it work for mirrormanager and bodhi roles here?

You could retrieve from Bodhi the list of active releases, the question then is
where does the mapping bodhi active releases <-> dist-git branches happen?
Is that hard-coded in dist_git_auth.py or is it part of the info Bodhi returns?
If it's the later, you can start by getting all the active releases from bodhi
and then query pagure for the git branches info and you should be able to
compute from there what is active where.

An alternative is have a cron job that would cache this info like we do for
bugzilla with: https://pagure.io/pagure-dist-git/blob/master/f/scripts which are
made available in: https://src.fedoraproject.org/extras/

The alternative sounds good. It would be nice to have up-to-date info about active branches per package somewhere.

The alternative sounds good. It would be nice to have up-to-date info about active branches per package somewhere.

You have it, but not in 1 place :s
That is until/if we move to gitlab and create pkgdb3 to manage it :p

So at this point Bodhi is likely the best source of info for this, iiuc that's
what the pagure-dist-git auth script that's run on git push queries now, no?

Yup, well, it's still in a stage of open PR :)
Not sure if it's transferable here. dist_git_auth.py runs a git command it on the same machine as are the git repos of the packages. Would it work for mirrormanager and bodhi roles here?

You could retrieve from Bodhi the list of active releases, the question then is
where does the mapping bodhi active releases <-> dist-git branches happen?
Is that hard-coded in dist_git_auth.py or is it part of the info Bodhi returns?
If it's the later, you can start by getting all the active releases from bodhi
and then query pagure for the git branches info and you should be able to
compute from there what is active where.

An alternative is have a cron job that would cache this info like we do for
bugzilla with: https://pagure.io/pagure-dist-git/blob/master/f/scripts which are
made available in: https://src.fedoraproject.org/extras/

Each bodhi release has the "branch" property which maps the dist-git branch.

Related PR https://pagure.io/fedora-infra/ansible/pull-request/1957# - removing pdc calls from /roles/bodhi2/backend/templates/owner-sync-pagure.j2

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog