I noticed that we basically have no gathered logs in there. Why? Could openqa try to switch to tty to gather logs?

It does. If we don't get logs, it means switching to tty didn't work.

The code is in https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/lib/installedtest.pm#_35 . It has special handling for dracut emergency mode, otherwise it switches to tty6 and tries to upload logs (if networking works and tar is available) or dump them out on the serial console (if not).

In this case, switching to tty6 doesn't work, the system is hung too hard for some reason. The second red thumbnail is openQA checking for either a logged-in console or a login prompt, and finding neither. You can see the record of this in https://openqa.fedoraproject.org/tests/3643039/logfile?filename=autoinst-log.txt , from lines 1326 (where the initial failure happens) onwards - we see select_console(testapi_console="tty6-console") to try and get to tty6, then we try and match the login prompt or a logged-in console, but fail.

A human in this case would likely reboot to runlevel 3 to get logs. We could try and enhance post_fail_hook to do that, I guess, but it might get a bit complicated.

Ok, if there's an automated step to try to gather logs from TTY6, that's great. In this particular case, TTY switching really doesn't work, I confirmed manually.

I wonder what OpenQA does in cases when no system user is available (after Workstation install but before gnome-initial-setup)? In that case, the systemd.debug-shell approach might be useful.

it doesn't use the system user, it uses root. We create the root account on almost all test paths.

There are definitely occasional cases where the current approach doesn't work, but addressing them is complex. "just always add systemd.debug-shell to the kernel cmdline" has several problems: it's actually quite hard to reliably add kernel args such that they will be present in every test. In some situations the bootloader timeout is very short so we can't reliably do it at boot time. We can inject args after install for tests which involve an anaconda install, but some tests don't.

And beyond the practical problems, there's the philosophical one that we want our tests to resemble real-world usage as closely as possible, and it's not typical to boot with systemd.debug-shell in real-world usage; it's possible that doing it could cause a bug that otherwise wouldn't happen, or hide one that otherwise would.

So far I've preferred in such cases to reproduce the problem manually, or hack up the tests temporarily on the staging instance to get the logs out in some way. We can definitely consider ways to improve this, but I don't think it's easy.

it's actually quite hard to reliably add kernel args such that they will be present in every test.

Yeah, I was more thinking about the cases where you boot from a pre-created disk image, but obviously that's just some fraction of the tests.

I think we can close this ticket, but I still want to ask about this:

We create the root account on almost all test paths.

How do you do this for Workstation installation?

Like this. It's a divergence from a stock install, yeah - but one we live with, because just a lot of the tests are designed around having a root account available and it'd be a lot of work to rejig them all to work without one.

(I mean...we could look at having root_console do sudo, but...mehhh...)

I'm surprised it works fine without forcing a SELinux relabel. Ok, thanks. I think we can close this.

"just always add systemd.debug-shell to the kernel cmdline" has several problems

For future reference, if this is really needed one day - editing the kernel args is not needed, this can be also enabled by running systemctl enable debug-shell in the installed system, or just by creating the appropriate symlink in etc. Source

sure, but the general issue is 'there's no single, surefire way to do this for all tests'. all the tests have different circumstances.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/issues/409

Please continue any further discussion there.