Hello all, thanks @jgrulich for opening the ticket. Yes, I am happy to help. Just a small context:

• The reason why I am asking now is also that the Packit team is revisiting the priorities for the next quarter so I would like to know if we should take a look at something important or blocking. (We are collecting feedback also in this GitHub poll/discussion.)
• To be clear from the very start, we/you don't need to use this for all the packages. If it helps with a simple subset and saves some time for the more complex one, this would be a win I think.

František

I think it would be interesting for us to explore. I had asked a couple years ago for Packit to get set up with KDE Invent and earlier this year we discussed the need to support telling Packit about generating sequences of packages to update and chain-build automatically from a known list of packages.

Are these things stuff that could be done now?

Hi @ngompa !

Yes, Packit now supports side tags. It's covered here if you are interested.

it's quite fresh so we are still polishing and fixing corner cases. It would be nice to know if something is still missing.

Do you have automatic build group and dependency sorting in your roadmap yet?

Not yet (no one has asked so far), but we can try to think about it and see how this could work.

You can see what we do now here (which is quite primitive): https://pagure.io/fedora-kde/kde-update-scripts

A generalizable solution would probably involve using libdnf to sort packages, similar to what Koschei and OBS' perl-BSSolv do.

Yeah the script is quite primitive, and sometimes feels like it's held together with spit and dreams. But hey it works!

For the sake of transparency, this is usually what is involved in doing a kde stack update:

1- clone_packages.py (With what you need to update, i.e. 'clone_packages.py plasma'. Right now, we have plasma, gear, pim and frameworks implemented. We're looking to add maui to it as well.

2- Update.py (That basically updates the specs to the new version, and downloads the sources from download.kde.org/release ftp)

3- build.py --copr (We use a copr to build our internal tests before pushing over to rawhide. For packages that fail to build, we fix them, and simply use the same command, but using --resume-from)

(Once the COPR build is successful)

4- We upload the sources to the cache (update.py --upload)

4.5- All changes for rawhide gets pushed to distgit with a one-liner bash command (fedpkg commit -m "blahblah", fedpkg push. Similar to Step 8.)

5- We make a side-tag (fedpkg request-side-tag)

6- build.py --target (To build on the sidetag, again fixing stuff as it happens)

7- Once successfully built, create the update in bodhi

8- Use a one-liner bash script pushd/popd that goes through all dist-git folders to switch-branch and merge from rawhide, and repeat from step 5 until we did all branches.

Thanks @farchord for the workflow. This helped.

Do I understand it correctly that as part of (2), you are pushing the changes directly to rawhide?
Would pull requests work for you as well? Packit can technically push to rawhide but we would like to avoid having write access to the packages so we don't allow it for the service. (And also, in our opinion, pull requests have other benefits.)

Btw. should I/we deep dive into the details here or should I join Monday's meeting to discuss this? (I can also give a short intro to those not experienced with Packit.)
I am fine with both.

Btw. should I/we deep dive into the details here or should I join Monday's meeting to discuss this? (I can also give a short intro to those not experienced with Packit.)
I am fine with both.

I think we should do both. If folks can provide info here, it helps, and we can also have a high-bandwidth discussion on Monday's meeting including an introduction to Packit for everyone.

Btw. should I/we deep dive into the details here or should I join Monday's meeting to discuss this? (I can also give a short intro to those not experienced with Packit.)
I am fine with both.

I think we should do both. If folks can provide info here, it helps, and we can also have a high-bandwidth discussion on Monday's meeting including an introduction to Packit for everyone.

Makes sense. It can also help me prepare a bit.

Thanks @farchord for the workflow. This helped.

Do I understand it correctly that as part of (2), you are pushing the changes directly to rawhide?
Would pull requests work for you as well? Packit can technically push to rawhide but we would like to avoid having write access to the packages so we don't allow it for the service. (And also, in our opinion, pull requests have other benefits.)

Btw. should I/we deep dive into the details here or should I join Monday's meeting to discuss this? (I can also give a short intro to those not experienced with Packit.)
I am fine with both.

Nah, step 2 only updates specs and downloads sources. Update.py only updates the specs, downloads sources and (optionally, if --upload is specified) uploads the sources to the lookaside cache.

The pushing to the dist-git repos happens outside of the script. I missed that part, that should be step 4.5. I'll add it.

Thanks all for the meeting!

For those interested, here are slides with some useful links.

Here are the ones that might be the most relevant:

• Onboarding guide.
• Documentation for the releasing a group of (dependent) packages in a side-tag
  • I am not sure if I made myself clear but you can specify an order.
  • For a bigger adoption, we need to find a way how to get this info from elsewhere. I've created an issue for this.

We found right away, that in this number of packages, the pull requests will be tough to review manually => here's an issue covering this and the current alternative in the form of a fedora-bot

For transparency, we agreed to try this on a bunch of packages to see how this works and find (and hopefully fix..;) possible issues blocking the mass onboarding.

Let me know if you need any help or join our Matrix channel (#packit:fedora.im), I'll try to provide updates if there is something relevant.

František

We discussed the dependency issue today and we are not sure what would be the best place/time to load/generate this information.

Loading this on every dist-git/Koji event of the packages might be quite resource-intensive (and wasteful).

Does the dependency tree change often? Do you know when this happens?

Thanks!
František

We discussed the dependency issue today and we are not sure what would be the best place/time to load/generate this information.

Loading this on every dist-git/Koji event of the packages might be quite resource-intensive (and wasteful).

Does the dependency tree change often? Do you know when this happens?

Thanks!
František

In our case, the dependancy tree is generated using the spec files, using the BuildRequires.

So imagine that we have 2 packages, Package1 and Package2. If Package2 requires Package1, then our script knows that Package1 has to be put in a group before Package2.

Now, I believe it's @aleasto that mentionned this in the meeting monday, but this specific method doesn't work with packages that have a cmake or pkgconfig buildrequire set (As the name differs from the actual package name in most cases). In that case, the script does dnf queries (dnf repoquery) to resolve the cmake names to actual packages to build it's dependancy tree.

Thanks @farchord for the details. How often does this dependency tree change (i.e. can we cache this information)? (=We need to figure out when/where to calculate the dependencies.)

Regarding the calculation itself, I hope we can use/mimic exactly what you do.

Thanks @farchord for the details. How often does this dependency tree change (i.e. can we cache this information)? (=We need to figure out when/where to calculate the dependencies.)

Regarding the calculation itself, I hope we can use/mimic exactly what you do.

It can virtually change any release. I don't know how long the current script caches the result, as I know it does. But it's not long.

@aleasto or @marcdeop can probably answer this one.

Building the tree on a pre-defined set of packages is a relatively inexpensive operation (I wouldn't know the time complexity off the top of my head, but the scale is low since we're dealing with < 1000 packages).

The expensive part of the algorithm is the name resolution from the virtual-provides name to the SRPM name, for which we invoke repoquery. For every package X in the set we cache the list of its dependencies as SRPM names, and invalidate it whenever the specfile of X changes.
This is technically inaccurate, because a virtual-provides might be moved from one SRPM to another without the specfile of the dependant package changing, but it's rare that this happens within our limited package set. Also, because KDE updates are released as bundles all specfiles of our package set are changed at the same time, so for a normal update all caches are invalidated (the cache is still useful for when the chainbuild fails, you go fix one package, and fire a new chainbuild).

Also, because we use repoquery we can only know about the currently published packages, so we wouldn't detect if the update we are packaging changes the list of virtual-provides. This is an unsolvable problem with this algorithm, because we may only know the list of virtual-provides after the package has been built.

In an ideal case, we actually re-sort the whole graph after every successful build and submit new builds into the side tag iteratively until the whole group is fully resolved and built.

This is kind of a weakness of the chain-build mechanism: we have to pre-sort the whole graph and hope that the sequence of builds doesn't mutate (or need a secondary rebuild somewhere in the middle and do it again). For most of the KDE stack, it's probably fine. It's more of a problem if Qt + Frameworks + Plasma need to be done at once.

I mentioned earlier about OBS (Open Build Service): I used to run an OBS deployment for almost a decade and have significant experience with it. One of the interesting attributes about how OBS handles this is that it all the sources are read, sequenced, and as each build completes, the sequence is recomputed for the whole scope of the namespace and builds occur until the graph stabilizes and builds stop changing. This makes simple loops automatic to resolve as long as the binary artifacts aren't unstable (ie. non-reproducible).

Unfortunately, to be able to go that far, we need a way to encode build counters. There's a long discussion about this that is worth reading to understand this topic.

Hello all,

a few updates from Packit:

• We are currently prioritising the work on a global config that might be beneficial for mass adoption and to be able to have the whole setup in one place. (With global configs, I can see some options to handle the dependencies.)
• When doing a regular check of old issues, we've come across an issue created by @ngompa to support invent.kde.org GitLab instance. (This is not needed for downstream automation. It's to be able to use Packit as a CI on invent.kde.org.) Sorry, it is taking us so long. Is this still relevant? Would you be interested in having it supported?
• The non-divergent branching for release syncing is now fully supported: https://packit.dev/posts/non-divergent-dist-git-branches.

I am not sure if you've had time to try Packit on a few projects. Let me know if you need any help with that.

Hello all,

a few updates from Packit:

• We are currently prioritising the work on a global config that might be beneficial for mass adoption and to be able to have the whole setup in one place. (With global configs, I can see some options to handle the dependencies.)

I do suggest you implement dependency resolution within Packit using libsolv in some way (either through python-libdnf5 or through python-solv directly). You should be able to read spec files, sort dependencies, and re-sort if needed after each build by reading the data.

• When doing a regular check of old issues, we've come across an issue created by @ngompa to support invent.kde.org GitLab instance. (This is not needed for downstream automation. It's to be able to use Packit as a CI on invent.kde.org.) Sorry, it is taking us so long. Is this still relevant? Would you be interested in having it supported?

There are some efforts where it would be useful, such as @sfaulken and I working on firstboot wizard work in KDE.