#11723 Remove PDC from package retirement process
Closed: Fixed 2 months ago by jnsamyak. Opened a year ago by humaton.

As part of PDC retirement rewrite logic to stop querying PDC for required information. The source of truth if the package is retired should be the presence of dead.package on its release branch in dist-git. The source of truth about active release branches should be bodhi API.

./scripts/block_retired.py
./scripts/pdc/get-retired-packages.py
./scripts/fedretire


Metadata Update from @lenkaseg:
- Issue assigned to lenkaseg

a year ago

Proposed solution:

There are three scripts in releng that check for retired packages, not package by package, but all retired at once.
PDC has the information about the retired packages (active=False) because the information about a retired package is sent to PDC at the event of retiring. Without PDC there would be no place to have names of all retired packages stored.

We can extract the information about retired packages from /srv/git/rpms/*.git on pkgs01 machine - those that contain dead.package(checked or each branch).

There is a simple bash script that would retrieve the retired packages per branch (#PR11757) nightly and output a json in this format: {"branch1": [package01, package02]¸branch2: [package01, package02]}. The json would then be exposed on some API endpoint somewhere (distgit?). Releng scripts would then check the retired packages there.

First thanks for working on this, sorry for the delay in feedback. ;(

This seems like it would work, but oof... thats a LOT of io. We have 38262 packages x (right now) 4 active fedora branches x ( right now) 4 active epel branches, which means we need to check something like 306096 branches... and per day? that would put a ton of load on pkgs01. ;(

So, perhaps we could ponder on what those scripts do and if we can figure a better way to do it.

block_retired.py is run even day with the rawhide compose to block packages in koji that were retired in the last day. If we move this retirement to a toddler, we don't need to run this script anymore?

I am not sure what the other two are for. @humaton any ideas?

would having the equivalent data stored in bodhi instead of pdc work? I think this was one of the items that was flagged as something bodhi could do as it sort of covered things it already did?

Perhaps.

Also the dead.package commits are all in datagrepper / on the fedora-messaging bus... but I don't think it's super easy to get that information, you have to get all commit messages and unpack them to see if dead.package was created as part of the commit. ;(

So I played with the script Lenka created. It took:

real    18m47.324s
user    3m25.859s
sys     10m47.095s

I didn't notice any particular spike in load, but monitoring might have.

block_retired.py is run even day with the rawhide compose to block packages in koji that were retired in the last day. If we move this retirement to a toddler, we don't need to run this script anymore?

We will extend this toddler in the process of retiring PDC. https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/plugins/pdc_retired_packages.py

after the toddlers update, there will be no need to keep running the block_retired.py.

Related PR on ansible which shold schedule the script nightly on pkgs01.
https://pagure.io/fedora-infra/ansible/pull-request/1650#

Related PR #11796, removes pdc from get-retired-packages.py
https://pagure.io/releng/pull-request/11796

I didn't find fedretire neither in ansible nor toddlers. Is there any other place where it could be used?

Related PR#12024
Fedretire script removed, it's not used anymore

This issue can be considered fixed when the block-retired.py script is solved.

@amedvede Would you have an update on block-retired.py please?

@amedvede's PR of a toddler which blocks retired packages in koji is merged, maybe we could close this issue? @humaton

https://pagure.io/fedora-infra/toddlers/pull-request/201

Most of the work here is completed, one quick thing is to clean it up from the nightly,sh (60 and 61 line)

https://pagure.io/pungi-fedora/blob/main/f/nightly.sh#_60

Metadata Update from @jnsamyak:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 months ago

Log in to comment on this ticket.

Metadata