It might be a good idea to add an admon box to those test cases and mention that there is an IoT component in Bugzilla which can be used for reporting problems which don't clearly fall under a specific component (systemd, kernel, etc). Also mention that it's good to mark reported bugs as Blocks: IoT (regardless of the component they are filed under), so that the IoT team have an overview of reported issues. See the IoT tracker bug.

Now that the release criteria regarding IoT have been looked through [0], I'll start working on getting test cases going. I have talked with @pwhalen in the IoT meeting on 26 February 2020 [1] and have agreed that we'll work on blocking test cases first, then depending on what resources are left, optional test cases.

[0] https://teams.fedoraproject.org/project/fedora-iot/us/76
[1] https://meetbot.fedoraproject.org/teams/fedora_iot_working_group_meeting/fedora_iot_working_group_meeting.2020-02-26-14.01.log.html

we'll also need result tracking.

I have worked a little on porting some essential test cases to Fedora IoT locally, however, I am wondering if putting these on the Wiki is where we want them? Is our plan to have the test cases on the wiki with everything else, or to set up some kind of TCMS before F32 launch and use IoT as our test-bed for working out bugs with the TCMS before we port everything else over? @sumantrom, how is the Kiwi setup coming? @kparal, can you chime in here?

@coremodule IMHO, it will be good to have wiki test cases for this release. As I try to finish the work around Kiwi, it's just POC and we aren't sure if we want to use it at all. FAS is still under development and will take some time. I feel it will be too ambitious of us to use Kiwi before F32 launch. Also, given the base stuff still is OS-Tree + ign, and we already have test cases for these in wikitcms. I believe moving with wikitcms makes sense for now (atleast this release).
Another part of my concern is, deploying things in communishift (which will have a significant downtime) will make things hard anyway.
@kparal WDYT?

I agree with @sumantrom. Let's place those test cases in wiki, as usual. We can then extend the existing release validation matrices with IoT test cases (where it makes sense, like Installation) and/or create a separate "IoT" set of matrices for the rest (where it makes sense). That would be added to the existing list of Desktop/Server/Cloud/etc matrices we already have:
https://fedoraproject.org/wiki/QA:Release_validation_test_plan#Current_and_recent_validation_test_events

If KiwiTCMS is ready before F32, we can convert some test cases and place it there (or even better, we don't need to convert them, just link to the wiki test cases, and we're done in 10 minutes -- good enough for PoC) and have it as a testbed. But I wouldn't rely on that at all, so let's do the wiki approach now.

We cannot extend the existing matrices as long as we have the problem of IoT composes being separate. There's no realistic way to filter the matrices on event creation, and I don't want to create matrices for main validation events which include references to IoT when those composes don't have any IoT images.

We can create a separate wiki matrix, though, because we can just not use that matrix for regular validation events, but only for IoT validation events. When I manage to get back around to coding that whole thing into wikitcms.

OK, let's create only the separate IoT matrix, and put all IoT test cases in there (including installation). In the worst case, we can spawn the individual compose matrices manually, out of the template, it's not that difficult.

@pwhalen already did it and is using it. Now Beta's signed off I should be able to get back to working on wikitcms next week, and do all the necessary work on that side; then we can create a proper matrix template (pwhalen's template is slightly 'bigger' than a matrix template, it's more like the old all-in-one 'templates' we used to use for manual event creation before I wrote wikitcms) and start creating IoT events automatically.

We talked with @coremodule about this last week. Pwhalen's IoT QA template contains a good deal of IoT test cases now. But we need to make them a bit more official. We should move the template out of Pwhalen's userspace to the same level and naming syntax as our other release validation templates. The same with test cases, most of them as in Pwhalen's userspace. Let's move them to the same naming scheme as our other test cases. We can have "IoT" in their title, at least for those that are relevant just to IoT (use your best judgement). Also, let's add them to proper category. I propose "IoT Acceptance Test Cases", because that will have the same format as e.g. Server Acceptance Test Cases.

@coremodule Can you please make this happen? (If the test cases' content is not 100% yet, that doesn't matter).

The new official IoT relval template is now here:
https://fedoraproject.org/wiki/Template:General_IoT_test_matrix

More background here:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/thread/HV6R6K6MD3TFHHHYHFCP5YFA6QUEAGOG/

The test cases are all moved to the QA namespace now too, so I think we can close this.