Hi, welcome to my life!

So, that idea won't work for...oh, lots of reasons.

1. Some tests are already grouped with parents and children. Making all of them run after another grandparent test would be difficult/impossible (not sure which).
2. Tests on different flavors can't have relationships, so you'd need one parent test per flavor.
3. You'd need at least one parent test per flavor anyway because the problem might occur or not occur depending on what's already installed on the system.
4. It won't catch problems openQA would otherwise catch only in an image build test. These do exist - it's why I did the work of extending the 'is the package from the update actually used' code to the image build tests (which isn't straightforward).
5. Probably some others I forgot!

But more importantly, this is not worth doing because catching dependency errors in openQA is the wrong way to do it. We only do it because nothing else is gating on it (or even doing it reliably), so if we don't do it in openQA, we will let updates with dependency issues through testing and blow up subsequent openQA runs (this is a real thing that was happening all the time before I added these checks).

The right way to do it is rmdepcheck in Fedora CI, which we're working on in https://pagure.io/fedora-ci/general/issue/522 , when I have the time. That has a ton of advantages - like it runs in about a minute flat, so we can run it on every package build, and it should catch every case where a package has dependency problems, which openQA definitely doesn't. And the output is relatively easy to understand, and right there.

I don't think it's worth spending much/any engineering effort on this as opposed to spending it on getting rmdepcheck moved along. I think it's easier just to have you and lruzicka and anyone else who's interested learn how to find the errors. FWIW I usually look for a test where it failed in _advisory_update, then look at serial_terminal.txt, which should always have the errors. But getting them from the red frame is fine too. The idea is that we should promptly investigate failures and interpret them on the Bodhi update page, so the maintainer doesn't have to.

For install_default_update_live, BTW, there is literally no easy way to debug it. If that test fails in this check, it means the packages from the update weren't baked into the live image in the parent live image build test. So you can go to the parent test and look at the kiwi logs and confirm that, yes, it put in the older packages for some reason. Why? You don't know! If you're lucky, other tests will also fail and you can find the error from their logs. But if openQA is only encountering the dep issue in an image build test, it's a huge PITA.

In the past, when this happened, I would swear a lot, then manually look at what the new and old package provide and which of those things other packages depend on and try to figure out what the problem was. Maybe I'd set up a mock chroot and try to replicate the live image package set, then try and update it to the package from the update and look for an error, and then not get one, and then swear a lot more and bitch about it on Matrix. It was the worst.

Now what I do is go to staging openQA and look at the rmdepcheck result, because rmdepcheck usually finds the problem. Much nicer. :D (if staging isn't on the rmdepcheck branch, I change it to the rmdepcheck branch and rerun the everything-boot-iso tests for the update).

This is actually a large part of why I wrote rmdepcheck; this was the situation that made me think "heyy, if we could somehow edit the repo metadata so the old packages aren't available to the depsolver, it'd be way easier to see what's going wrong." originally I was going to do it inside openQA till I realized that was dumb.

btw, for investigating these issues it's also useful to know how openQA handles repositories.

Currently, for Rawhide and Branched before Bodhi activation, openQA adds the buildroot repository. So all packages that have been 'pushed to stable' in Bodhi at the time the test runs (give or take a few minutes for repo regeneration) should be available to the test.

For Branched after Bodhi activation and stable releases, openQA does not add the buildroot repository. So only packages which have actually been in a compose are available to the tests. So if update B depends on something from update A, tests for update B will only pass after update A has been pushed stable and there has been a compose - for Branched, a nightly compose; for stable, an updates compose.

The openQA 'workarounds' mechanism can be used to short-circuit the wait if you're in a hurry for some reason - add the update that's gone stable but not yet been in a compose to the workarounds hash in utils.pm somewhere, roll that out, re-run the tests.

We could potentially enable the buildroot repo for all tests. It's a pretty subjective choice. The tradeoff is that then we miss problems if they're masked by a buildroot override.

This has been very helpful, thanks for writing it all down.

But more importantly, this is not worth doing because catching dependency errors in openQA is the wrong way to do it

I agree that focusing on rmdepcheck (I always read it as rpmdepcheck) is the better way to go. Thanks for working on it.

For Branched after Bodhi activation and stable releases, openQA does not add the buildroot repository

Right, I forgot about this part again. That was the reason for the packager's confusion.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/issues/438

Please continue any further discussion there.