We should have a 'basic video mode install' test (or multiple, for live and regular installers). We didn't do this before because the test case specifies it must be done on bare metal, but it would still be useful to test it in openQA because it can catch some failures; we just shouldn't report a pass to the wiki when the test passes. This should be pretty easy to do.
Hmm, poked at this a bit today and it's really not trivial. The problem is getting a VM to behave anything like a real system under these conditions; they don't, really. nomodeset's effect on the qemu graphics 'devices' and drivers is rather different from how it acts on 'real' hardware and drivers.
nomodeset
With the 'qxl' device, qemu happened to recreate the specific 'basic graphics mode' bug we saw in Fedora 26 Alpha, but that was kinda dumb luck. The bug wasn't really in anything 'basic mode'-specific, it was actually in the Wayland -> X11 fallback code combined with gdm autologin. It just so happens that when booted on a VM with the 'qxl' device, Fedora / GNOME will wind up in Wayland mode on a 'regular' boot and X11 mode on a 'nomodeset' boot. But it uses the 'qxl' driver when it falls back to X11, not 'vesa'. I'm not even sure why the qxl+nomodeset combination actually triggers the X11 fallback, and thus if that's even behaviour we can rely on.
So...I'm not sure we can do anything useful here. We're just gonna need to test this manually earlier. (Or hook up openQA bare metal testing, of course...)
This issue has been migrated to Fedora Forge: https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/issues/50
Please continue any further discussion there.