@kparal I am looking at this. I am willing to take this. As I communicate with the ARM team and put down the results on this ticket. Thanks for bringing this up.

@pbrobinson should probably be tagged on this...

@kparal I am looking at this. I am willing to take this.

Don't be afraid to set yourself as the assignee then :-) Thanks.

Spoke to @pwhalen . The list of devices that's blocking are https://docs.fedoraproject.org/en-US/iot/release-criteria/. This page has a list of blocking hardware. I will be working towards writing the release criteria and then expanding the test cases.

@sumantrom For the record, I have a Dragonboard 410c as well as a Rasberry Pi 3B+ for test purposes.

Spoke to @pwhalen . The list of devices that's blocking are https://docs.fedoraproject.org/en-US/iot/release-criteria/. This page has a list of blocking hardware.

And I wrote this in the ticket description:

Note: This is not about Fedora IoT, that has specific release criteria. This is about general ARM32/64 composes.

So, @sumantrom , is pwhalen saying the release blocking devices are the same for IoT and general ARM composes?

Spoke to @pwhalen . The list of devices that's blocking are https://docs.fedoraproject.org/en-US/iot/release-criteria/. This page has a list of blocking hardware. I will be working towards writing the release criteria and then expanding the test cases.

That list is specific to IoT devices. For Arm our list is here:
https://fedoraproject.org/wiki/Architectures/ARM/Supported_Platforms

I will update the ARM list to reflect AArch64 boards and which are capable of running Workstation (should include most devices with 2GB+ of ram).

Thanks @pwhalen . Just a note, there's another list present at:
https://fedoraproject.org/wiki/Architectures/ARM#Supported_Hardware_and_Devices

So it might make sense to eventually merge them into a single list. Or at least provide additional link to the page you provided, because I haven't found a link to it anywhere.

How is this handled on x86_64? I don't believe we have an explicit list of blocking devices there but rather each HW issue is considered whether it's worthy of blocking/FE depending on likelyhood of it being an issue for users and the number of users it may affect.

EG there's no way I would block F-32 on an issue with RPi4 as it doesn't yet have upstream accelerated graphics but likely will in F-33 and being a popular device we'd likely consider issues around that at that point something we'd want fixed.

Yes, for x86_64 we don't have an explicit list of supported devices. For ARM we always have done so far and IIRC this was at the request of ARM team.

If we don't have a specific list of supported devices, the decision essentially gets made on a case-by-case basis by whoever shows up to release blocker review meetings - this is what happens when hardware-specific x86_64 issues are proposed as blockers. For Intel that's really the only plausible option, but as the range of available ARM hardware is more limited, having ARM SIG specify what we consider 'supported' in advance has always been possible (and, as I said, desired).

If you think this should change to be more similar to the x86_64 approach we can certainly do that, but it may well result in somewhat haphazard decisions given that I don't think the normal group of blocker bug reviewers are that familiar with the ARM hardware that's available in the market or how popular each model is...

BTW, the list the criteria cite is the one at https://fedoraproject.org/wiki/Architectures/ARM/Supported_Platforms . Dunno where the https://fedoraproject.org/wiki/Architectures/ARM#Supported_Hardware_and_Devices one comes from, but it is not hooked into the release process AFAICS, only the Supported_Platforms one is.

If you think this should change to be more similar to the x86_64 approach we can certainly do that, but it may well result in somewhat haphazard decisions given that I don't think the normal group of blocker bug reviewers are that familiar with the ARM hardware that's available in the market or how popular each model is...

I would like to see ARM/AArch64 be more similar to x86_64 as well. I can help highlight the important devices to the other blocker bug reviewers when needed.

Dunno where the https://fedoraproject.org/wiki/Architectures/ARM#Supported_Hardware_and_Devices one comes from

I found it by google search. I believe it's important to not have confusing desynced duplicated stuff, that's why I suggested those two places to be merged.

I would like to see ARM/AArch64 be more similar to x86_64 as well. I can help highlight the important devices to the other blocker bug reviewers when needed.

So, if the affected board was not listed among Supported Platforms as supported, we would automatically reject it, correct? And if it was listed in there, we would make some estimate of how popular that board is, and whether it's OK to keep it broken or whether we need to block on it, correct?

What about Workstation support? Do we expect all boards to be functional enough to run Workstation without issues? Or are there some which should get excluded automatically? If that's the case, we should extend the Supported Platforms table to mark this.

What about some other use cases? For example serial console might be important for Server/Cloud. Are all boards expected to support it completely, or some are not? Are there any other similar areas that we should consider?

Do we have the same expectations for armhfp as for aarch64? (For armhfp, we only block on Minimal installer, which doesn't say at all which use cases, like desktop or server, are blocking).

With x86_64, things are way easier because almost everything is expected to work on any hardware (with some notable differences, e.g. BIOS vs UEFI support). ARM is a completely different world, so we need to define what is expected where. Or we can say that everything is expected everywhere, but in that case, the supported platforms list might be very short. Or, we can perhaps have a very short list of "fully supported platforms" and the rest would go to "partially supported platforms" with much lower requirements. Thoughts?

Talked with @pwhalen and @pbrobinson about getting an updated list for F33. This should happen in the near future.

I see a possible contingency we should plan for: If indeed the list of IoT Reference Platforms are blocking, we should establish a way to generate a list of what will be considered a blocking group of hardware for a certain release to prevent the list being updated at the end of a release cycle and then a bug found on the newly-added hardware causing us to block release. This could be as simple as getting together with the IoT team after the end of one release cycle and adding any new boards to the list of "Release Blocking Hardware" for the next release cycle. This would be enough to prevent this particular incident.

After doing a bit of digging, it looks like we do this already for ARM-releases, but it doesn't look like we intend that all of the supported ARM hardware be supported IoT hardware. Please correct me if I am misinterpreting that.

Why do we need explicit release blocking devices? Does Workstation for x86_64 have a list of devices they block on? Not that I'm aware of.

Why do we need explicit release blocking devices?

We don't need to have explicit release-blocking devices if that's what the IoT-team decides. The QA team would treat hardware-specific IoT/ARM issues the same way we treat hardware-specific x86 issues; each issue considered individually. See @adamwill 's comment above for more clarity on what that looks like.

Does Workstation for x86_64 have a list of devices they block on? Not that I'm aware of.

No, x86 Workstation does not. But then, x86 workstation doesn't have a list of known-working "Reference Platforms" associated with it, where IoT and ARM do. Having the list of "IoT Reference Platforms" is confusing if they are not intended to be blocking hardware. (At least, without an explicit statement that the hardware is not blocking, something like "These reference platforms have been tested and are known to work with Fedora IoT, however their inclusion on this list does not necessarily constitute a release-blocking issue should a bug that is specific to this hardware be found." or something like that anyway.) If the intention is to not have specific release-blocking hardware, then we should make that clearer than it currently is. (I have, until now, personally interpreted that list to mean that if one of those boards has an issue with Fedora IoT, we will block the release.)

I have submitted a PR to the IoT-docs pagure [0] adding a note that the "Reference Platforms" are only known-working hardware and an issue with them does not necessarily constitute a release-blocker.

If @pbrobinson and @pwhalen intend for the list of IoT Reference Platforms to not be a list of explicit release-blocking hardware, I will close this ticket as "Worksforme".

[0] https://pagure.io/fedora-iot/iot-docs/pull-request/56#

Let's resolve this ticket already. Currently our release criteria say:

Supported ARM platforms are those listed by the ARM team at Architectures/ARM/Supported_Platforms.

Supported IoT platforms are those listed by the IoT team here.

So let's do the following:
[ ] Make sure Architectures/ARM/Supported_Platforms gets updated with Fedora 33/34 information. Alternatively, we can remove that reference from our release criteria, if the ARM team wants to have the same approach as with x86_64 (case by case decisions).
[ ] Make sure Architectures/ARM#Supported_Hardware_and_Devices is merged with the previous page, or removed, so that we don't have duplicate and inconsistent information on the wiki.
[ ] Figure out if all supported devices are expected to run all release-blocking images (Workstation, Server, etc), and if not, decide if it makes sense to specify that on the page above.

Giving the ticket to @coremodule . Thanks.

Talked with @pbrobinson at the ARM meeting today; he's been swamped lately and hasn't gotten around to finishing the "official" list, but appreciated the reminder.

Well...both lists still exist. https://fedoraproject.org/wiki/Architectures/ARM/Supported_Platforms says "Last Revised: Tuesday Feb 20, 2020", and looks like it. I was the last one to touch https://fedoraproject.org/wiki/Architectures/ARM#Supported_Hardware_and_Devices substantially (dropping some 32-bit only platforms, as best I could). jlinton has added a few things to it piecemeal over the last few years. I'm not sure if it can be considered "current", though - can @pbrobinson , @pwhalen and / or @jlinton comment?

as discussed above, we can, if desired, drop the whole 'supported platforms' concept as far as the release process is concerned and handle ARM hardware issues on a case-by-case basis with the blocker review stakeholders doing their best "how many people is this going to hurt" evaluation. but as none of the usual blocker voting folks is really an expert on ARM hardware, the results may be unpredictable.

From my perspective anything that is Systemready ES or better (ex, VE, SR) is "supported" to the extent that I will attempt to fix things. With various odd devices like the x13s and solidrun honeycomb, old thunderX2's, etc thrown in for good measure.

And the VE/SR platforms are largely "silently" working. There are a few glitches here/there (like the fedora release on the yt710's are ????). But hyperv, aws, parallels, utm, etc and all the released server hardware platforms are largely in the works without issues categories.

I think all this is best effort anyway, so I'm in the drop the "supported" nomenclature, with the idea that keeping "known issues"/"How to get this working" pages around is quite helpful.

Although that said, we probably should be tracking things like driver/module enablement against the platforms using them so we can make decisions about enabling/disabling drivers based on when we officially enable/drop support for something.

We can tweak the wording on the pages, I guess, but the key question for the release process is - do we still want to have a defined list of ARM platforms which are expected to work (not just "jlinton will try and fix problems if you tell him about them"), and where it's a release-blocking bug if they don't?

(and should that list just be "raspberry pi"? :>)