I've recently overhauled https://fedoraproject.org/wiki/QA:Testcase_USB_fmw and simplified the verification phase quite a bit - currently we just need to run checkisomd5 or cmp (or ideally both). At this point, it would be nice if we could automate it. Manually running the test on Fedora 41/42/43/Flathub (and also Windows/macOS) is tiring, and I do it in a Fedora VM anyway, just redirecting my USB stick into it. In OpenQA, we wouldn't be redirecting a physical usb stick, it would be an emulated device, but it could do 90% of verification for us, and the remaining 10% "bare metal" bits could then be tested on any of 41/42/43/Flathub, just once, saving us a lot of manual time.
checkisomd5
cmp
OpenQA could: 1. install FMW 2. test writing the image (or even possibly multiple, let's say one Live and one netinst-style) 3. verify the written image 4. test flash drive restoration (if this was difficult for any reason, it's OK to keep it to the manual test version instead)
I'm not sure how easy or difficult this might be. We would definitely need a virtual drive to act like a flash drive. I think some params can be set in qemu/libvirt to make it look like one. If hotplugging in/out is possible, it would be even better (for example test unplugging after writing and replugging before restoration), but if that's hard to do, then just booting the system with the flash drive already plugged in and not ejecting it is still a valuable automated test.
I think we should definitely be able to do a 'permanently attached disk' version of this. I took a quick look into the possibility of hotplugging. openQA (os-autoinst, really) mostly uses qemu's QMP for the task of 'doing stuff to the VM while it's running'. The code is in backend/qemu.pm. From a quick look, though, I don't see anything about hotplugging storage devices. You can use the misleadingly-named eject_cd in os-autoinst to send a QMP eject command to any device, though I'm not sure what happens if you try to 'eject' a USB stick - whether it works, or whether eject is strictly for "drives with separate media", like optical drives or floppy drives.
backend/qemu.pm
eject_cd
eject
It looks like the device_add and device_del QMP commands can be used for hotplugging devices (at least that's what AI said, and a quick look at the doc seems to confirm that), but os-autoinst doesn't currently provide a wrapper for those AFAICT.
device_add
device_del
Possibly test code can get to the backend's generic handle_qmp_command function, I'm not sure. If so we could do something with that. If not we'd have to write a wrapper and submit it upstream.
handle_qmp_command
For the permanently attached disk scenario, I think it would be good to 1. write the image 2. close FMW 3. verify the image 4. start FMW 5. restore the drive
The closing&reopening FMW would make it closer to a real-world scenario, when the user wants to restore their flash drive a few hours/days later after using it. It means FMW needs to be able to recognize the drive again after a fresh start, and offer the restore functionality. It's not as good as hotplugging the flash drive after FMW is started (and verifying that the hardware change is detected and option actually pops up dynamically), but it's close enough, and hotplugging can be tested by a human.
It would be 80% work done for 20% of time (compared to implementing missing features in handle_qmp_command), which sounds good to me.
Metadata Update from @lruzicka: - Issue assigned to lruzicka
Metadata Update from @lruzicka: - Custom field story_points adjusted to 8
I will try to make this happen.
A note on this: I only implemented a disconnect function upstream using QMP device_del so far, but QMP does also have a device_add which I guess could potentially be used to connect the USB device after system boot, simulating hotplug and making this test more accurate to likely real-world use. I'll try and find a bit of time to poke into whether I can figure out how to use that. (It would also let us automate https://fedoraproject.org/wiki/QA:Testcase_desktop_automount from the desktop matrix, which would be nice).
I am finishing this without the hot-plugging stuff. Incorporating this into the PR would another amount of time and added complexity to the ticket which I have not expected. I suggest we close the original version first and have another ticket to add hot-plugging to this later, when we know how we should do it exactly. In between, we could have tests running with the original requirements.
https://pagure.io/fedora-qa/os-autoinst-distri-fedora/pull-request/462
This issue has been migrated to Fedora Forge: https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/issues/450
Please continue any further discussion there.