for the record, the idea of "universal" is that it can potentially run on any installer image. One issue with fixing this is going to be that that is very slightly a lie: a few of the tests that are in there actually fail if you use a netinst, because they aren't set up to get an external network connection and they need one if you use a netinst.

The other problem with fixing this is that we do prefer to run universal on a DVD image if possible because when run using a netinst the tests are subject to the "actually install yesterday's packages" problem. When I set up this design we had both the 'everything' DVD and the server DVD, but we don't build the everything DVD any more so server DVD is the only game in town.

What we can probably do here is move tests from universal that really don't work unless the image is a DVD to the server DVD flavor - technically the correct thing to do would be to create a new universal-dvd flavor that required a DVD image and have fedora_openqa use any available DVD image to run it, but I'm not fking around with that since it's very unlikely we're ever gonna invent another DVD installer image again, so it's just busywork - and then perhaps move the tests listed above out of universal and run them on both, say, Server DVD and Everything netinst, or something.

Aside from install_updates_img_local, I think all of those should work fine with the netinst ISO as well?

I'll figure it out next week. Trial and error...

OK, so, I didn't figure it out next week :D But now we have some time to work on this.

Initial thoughts:

• The 'universal' concept is kind of...much less useful than it used to be. We came up with it In A World Where composes didn't block on images at all, so we didn't know for sure that a 'successful' compose would contain certain images. Now they do, and Server DVD is a fatal image, so we can be pretty sure any successful compose has a Server DVD image, and we probably don't really need this concept any more.

• It'd actually be nice to run some of these tests on certain updates (ones containing installer-related packages) too.

• Tentative plan: let's dump 'universal' (this will also simplify the scheduler quite a lot, so yay) and put all the tests from it into the Server DVD flavor. We can then also run selected tests, something like the list @ngompa gave, on the Everything netinst.

How does that sound?

Sounds good to me :wine_glass:

So I'm working on this. Not sending PRs yet as I want to do more, but I have universal-overhaul branches set up in this repo and fedora_openqa. For now this moves most tests out of universal and into Server-dvd-iso. Turns out we do need to keep universal around, because of upgrade tests: these are not associated with any compose image asset (the things we represent as flavors) at all, they boot from base disk images and only test upgrades using the repos from the compose tree, which we don't represent as a 'flavor'.

We could rename 'universal' to something else like 'noimage' or 'repoonly', but I don't really see the point in the churn, so I didn't. I just simplified the scheduler to drop all the "score" stuff and schedule 'universal' for all arches it saw at least one image for (unless overridden by an arg).

I'm testing this on my pet instance ATM; here's how it looks as of now. Tests take a while to run as I only have four workers, and some will fail because I don't have a full set of base disk images, but it should be enough to sanity test this before I try it out on lab.

so an interesting, non-obvious issue about this: it creates a big cluster of dependent jobs. There are jobs in both Server-dvd-iso and universal that use the support_server. Combining them together gives us a big cluster of 9 jobs. That means you need a worker host with 9 worker processes or openQA will just never schedule the cluster and get stuck forever. Our official deployments are big enough, but my pet deployment isn't and probably other pet deployments aren't either. That's why happyassassin is stuck not scheduling anything right now (it's probably an openQA bug that the scheduler doesn't skip the cluster and schedule something else after a while, but meh).

I'll think on this. Simplest "fix" would just be to have multiple support_servers.

Working on the above issue, I've split out PXE and iSCSI jobs in recent commits (untested yet). I think the remaining jobs are all NFS-related so a bit less natural to split up. I'll try it out tomorrow.

update here - the support server split isn't working well in testing :( will try and figure out the bugs tomorrow.