This is definitely not always fine.
https://bugzilla.redhat.com/show_bug.cgi?id=2262526 is one case where it caused a significant issue. The other case I constantly run into is that a retirement removes a package listed in comps, or kickstarts, or workstation-ostree-config, or FCOS manifests, or lorax's lists of packages to install into the installer environment (...yeah, we have too many package manifests, what else is new).
We really should have some kind of check on package retirement operations. Ideally the checks would be gating, but we really at least need informational ones. We should know if retiring a package will cause a capability that some other package depends on to disappear from the distribution (i.e. the retired package provided something that something else depends on, and which no other package provides). We should know if retiring a package will cause an entry in comps or kickstarts or (rest of long laundry list) to no longer be installable.
I don't think any of this is particularly easy, but I at least figured we should have an issue for it, because it keeps coming up.
The example case does illustrate a situation where there might be a bit of a catch-22, of course. If we prevented the retirement of kernel-tools because it would result in python3-perf no longer being provided by anything, would that prevent the build of a kernel source package containing a python3-perf binary package because kernel-tools already has a python3-perf binary package? That obviously would be an issue we'd have to figure out somehow. It almost feels like we'd have to represent package retirements as "builds" that can be added to Bodhi updates, or something, which feels weird. Maybe there's a better way.
kernel
python3-perf
kernel-tools
When do you need this? (YYYY/MM/DD) No particular date.
When is this no longer needed or useful? (YYYY/MM/DD) When we finally give up on Fedora Linux and pivot to yak farming.
If we cannot complete your request, what is the impact? We'll continue to encounter breakages caused by package retirements, which will continue to make us dream wistfully of the simple joys of yak farming.
I guess another concurrency issue is "what if package B depends on package A, and you want to retire packages A and B together"? We don't want to block to retirement of A because B depends on it. So you'd have to be able to group retirements...
I guess a wacky idea: what if doing a package retirement had to involve a build of the source package which produced no binary packages? That would kinda provide a slightly goofy mechanism for bundling retirements, and for grouping retirements together with appropriate changes to other packages (like in the 'retire kernel-tools and move the build of its binary packages into the kernel source package' case, for instance). But...yeah, it's pretty goofy. :D
Ideally, retirements would happen via bodhi and could be grouped with builds.
That way, you can retire 2 packages together. Or you can retire kernel-tools and add python3-perf to kernel at the same time.
EDIT: I pretty much repeated what you already said in the comment above. I was so eager to say this that I posted the comment before reading all of yours, sorry about that.
Metadata Update from @phsmoura: - Issue tagged with: high-trouble, medium-gain, ops
My first thought would be to handle this like we do new branches (and soon unretirements). ie, ticket that gets processed by a toddler. That would have the ordering problem unless you could submit all of them in one request.
I don't like to argue bureaucracy, but I'd honestly kinda want to pitch for this to be high-gain. It's definitely high-trouble, though.
Metadata Update from @adamwill: - Issue untagged with: high-trouble, medium-gain, ops
sorry, I didn't mean to literally change the tags, that's just a pagure bug, I think.
Metadata Update from @adamwill: - Issue tagged with: high-trouble, medium-gain, ops
CC: @amedvede, is working with automating the unretirement process through toddlers, looping him in to see if we can include these as part of the automation.
Whatever happens will also need to account for the automated Orphaned Packages Process and FTBFS/FTI retirements. An implicit part of that process is that the maintainers of dependent packages that don't take action within the time window will have their packages broken and have to take action to avoid the eventual retirement of their packages. For better or for worse, occasionally breaking packages ensures that these processes actually fulfill their purpose of removing unmaintained packages from the distribution.
Metadata Update from @amedvede: - Issue assigned to amedvede
Hello guys, I want to implement those checks so I assigned this ticket to myself, but I'm on PTO right now so I'll do it next week after my return.
The solution might be to make fedpkg command which will check if some packages are dependent on the requested package and suggest including it in the request, and toddler that will process this request. What do you think about a suggested solution, can it work?
fedpkg
For manual retirements, yeah (see gotmax's note above on automated processes). It should also check if any subpackage of the source package is listed in comps or kickstarts.
There's a lot of detail to getting this 100% right, but it might not be necessary at first if it's just going to print a warning that will be manually handled...if we want to block requests that 'fail' the check we would need to get as accurate as possible. A specific case I can think of - what if something else in the distro also provides the capability that's being removed?
The implementation is now getting a list of branches provided by a package that is trying to be retired. It will be called to create a dependency tree of the package and notify the user about all packages that are affected. It's not the ideal solution because other packages can provide the same packages, so now I'm thinking about what the best solution will be with Aurelien. https://pagure.io/fedpkg/pull-request/579
PR is updated, now it's getting a list of co-packages and getting list of packages provided by them. This list is checked if any of those packages are required by others, if not it will be removed from list. After that it will check each package if its provided by others, if so it will also be removed from the list. So at the end, there will be a list of requested package and packages provided by it that are required by others and not provided by others. The next step is to check if those packages in comps, kickstart and so on. An example of this process for postfix here https://paste.centos.org/view/17effec9
postfix
hum... that paste link is 404 here... expired? xan you stick it in your fedorapeople space?
Functionality is added, now it checks if package or other packages that need to be retired with it are in comps, workstation-ostree, FCOS manifests. No need to check if it is in lorax's and kickstarts manifest because it uses comps to get list of packages. https://pagure.io/fedpkg/pull-request/579
The example of processing https://amedvede.fedorapeople.org/example_retirement_checks.txt If that's okay and PR seems good, I can start writing tests and soon it will be done.
Wow... thats a large output. ;) Could we supress some of that that doesn't matter? Or I guess it's there for debugging/completeness?
How long does that take to run?
One downside to checking this at fedpkg is that someone could bypass/disable it locally, but thats fine I guess, as we are no worse off than now and would see that someone bypassed this if it caused problems.
Yeah this output is from my testing script and its just showing a processing for some package in my case posfix
posfix
Total time: 62.643925189971924 Quite long time but it for such package as posfix, for smaller packages the time consume will be lower
I don't see how someone can disable it locally, I probably didn't get what you mean
If that seems okay, I can start to cover it with tests
Wow... thats a large output. ;) Could we supress some of that that doesn't matter? Or I guess it's there for debugging/completeness? Yeah this output is from my testing script and its just showing a processing for some package in my case posfix How long does that take to run? Total time: 62.643925189971924 Quite long time but it for such package as posfix, for smaller packages the time consume will be lower
That seems like a long time. It seems like the main issue here is that the code makes several independent calls to dnf repoquery which requires creating a new process and loading all the repodata into memory each time. Might I suggest the single fedrq command that accomplishes this purpose: fedrq whatrequires-src --branch <BRANCH NAME> <SOURCE PACKAGE NAME? Or you can make the repoquery calls more efficient. You don't need to query for the provides separately; you can just pass package names to --whatrequires and it'll resolve them for you. Also, I think you can pass --whatrequires multiple times with different package names so you don't need separate calls for each subpackage (co-package). The code should also work with EPEL and branched Fedora releases prior to the final freeze. It seems the current repoquery calls just use whatever repos are enabled on the same, regardless of what branch is being retired.
dnf repoquery
fedrq whatrequires-src --branch <BRANCH NAME> <SOURCE PACKAGE NAME
--whatrequires
One downside to checking this at fedpkg is that someone could bypass/disable it locally, but thats fine I guess, as we are no worse off than now and would see that someone bypassed this if it caused problems. I don't see how someone can disable it locally, I probably didn't get what you mean
Well, there would have to be a way to disable this for purposes of the orphaned packages process, other automated process, or in the case that output has false positives (e.g., multiple packages Provide the same capability as @adamwill mentioned).
@gotmax23 Thank you for advice! I created two files that show time spent before and after optimization: - before: https://amedvede.fedorapeople.org/timestamps_1_version.txt - after: https://amedvede.fedorapeople.org/timestamps_2_version.txt
I didn't use fedrq whatrequires-src --branch <BRANCH NAME> <SOURCE PACKAGE NAME because processing takes a longer time than dnf repoquery. "Or you can make the repoquery calls more efficient. You don't need to query for the provides separately; you can just pass package names to --whatrequires and it'll resolve them for you." - it works for second point of processing, unfortunately it couldn't be applied in 3. and 4. point because it needs to check each package name by going through the list. If this time is still too slow, there is a space in 5. 6. and 7. points, I can create a small API that contains a database with packages listed in different manifests. It allows to use of 1 single request instead of parsing each manifest individually. The flag is added https://pagure.io/fedpkg/pull-request/579#_2__32, do I also need to check if a package is orphaned, for example, or if the whole processing will be based on this flag, it will be fine?
I didn't use fedrq whatrequires-src --branch <BRANCH NAME> <SOURCE PACKAGE NAME because processing takes a longer time than dnf repoquery.
It shouldn't once you have a fresh cache... What exact command are you running? In any case, that command won't be sufficient if you also want to tackle the issue of provides available from multiple packages. Also, this still doesn't address
The code should also work with EPEL and branched Fedora releases prior to the final freeze. It seems the current repoquery calls just use whatever repos are enabled on the same, regardless of what branch is being retired.
It seems the correct solution here is to use the dnf or libdnf5 Python API, possibly through fedrq's wrapper API. Spawning multiple repoquery processes is just going to take too long.
If this time is still too slow, there is a space in 5. 6. and 7. points, I can create a small API that contains a database with packages listed in different manifests. It allows to use of 1 single request instead of parsing each manifest individually.
Are the manifests available in the repo metadata comps groups?
Log in to comment on this ticket.