Initial re-prioritization according to the updated [https://fedoraproject.org/wiki/Fedora_Release_Criteria Fedora Release Criteria] complete -> [https://fedoraproject.org/wiki/QA:Fedora_13_Install_Results_Template#Test_Matrix QA:Fedora_13_Install_Results_Template#Test_Matrix].

I'll take discussion of the remaining gaps to test@l.fp.org.

The example table needs to be updated (it still has 'Tier1' and 'Tier2'). Maybe we should re-do the default test order too?

Replying to [comment:5 adamwill]:

The example table needs to be updated (it still has 'Tier1' and 'Tier2').

Good catch, I've updated the 'key'. I'm not dead-set on using the terms {Alpha,Beta,Final} in those columns yet. But it will do for now.

Maybe we should re-do the default test order too?

I plan to tackle this last. I'm leaving tests in their previous sort order until everything is completed. It's just easier for me to grok the changes and ensure I'm not lowering the priority of something that should get some review first.

After reorganizing the test cases using the release criteria as a guide, the following tests I was unable to easily fit into the Alpha, Beta or Final priority.

Package Set
* [https://fedoraproject.org/wiki/QA/TestCases/PackageSetsMinimalPackageInstall QA/TestCases/PackageSetsMinimalPackageInstall]

I'd actually like to remove this test. It's original intent was to capture what the critical path package set is better capturing now. I don't think we lose anything since we'll continue to have the ''Default Package Install'', repoclosure and conflict tests on the list.

User Interface
* [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Cmdline QA:Testcase_Anaconda_User_Interface_Cmdline]
* [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_serial_console QA:Testcase_Anaconda_User_Interface_serial_console]
* [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Telnet QA:Testcase_Anaconda_User_Interface_Telnet]

I know the installer team would like to stop supporting some of these funky options (e.g. telnet), I'd like to keep this on the list as long as they continue to be supported. Any thoughts on the appropriate priority?

Upgrade system
* [https://fedoraproject.org/wiki/QA:Testcase_Preupgrade_from_older_release QA:Testcase_Preupgrade_from_older_release]

Upgrading from a the CURRENT-2 release is not explicitly listed in the release criteria. Does that mean this can be dropped?

Partitioning
* [https://fedoraproject.org/wiki/QA/TestCases/PartitioningUninitializedDisks QA/TestCases/PartitioningUninitializedDisks]

This test correctly captures the specific scenario when anaconda does not recognize the disks and prompts to initialize them. I'd like to propose moving this to Alpha. The use case captured is when you are installing on a new blank hard drive. Thoughts?

Recovery
* [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_updates.img_via_installation_source QA:Testcase_Anaconda_updates.img_via_installation_source]
* [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_updates.img_via_local_media QA:Testcase_Anaconda_updates.img_via_local_media]
* [https://fedoraproject.org/wiki/QA:Testcase QA:Testcase] Anaconda traceback debug mode

Again, I think these are important tests, but I'm unclear on the priority. Just picking something ... beta?

We probably need a new priority label for tests we want to do but which are not release critical at any stage. The cmdline, serial, telnet, upgrade from -2 should go in there.

Yes, PUD should be Alpha, I think.

traceback debug mode is Alpha, isn't it? From Alpha requirements: "The installer must be able to report failures to Bugzilla, with appropriate information included"

updates.img working is probably Beta or Final, I'd think.

Thanks for the feedback Adam.

Replying to [comment:8 adamwill]:

We probably need a new priority label for tests we want to do but which are not release critical at any stage. The cmdline, serial, telnet, upgrade from -2 should go in there.

I'll double check with the installer/virt teams. I suspect serial support is more important as it's used by our install automation and by the virt teams. Cmdline targets installation in environments without a console (typically virt, ppc and s390x).

You make a good point. I keep going back and forth on ... if the use cases aren't release critical, should we continue testing them? Anyone have a suggested phrasing for tests we want to run, but they aren't release critical?

Yes, PUD should be Alpha, I think.

Okay, thanks. Updated wiki.

traceback debug mode is Alpha, isn't it? From Alpha requirements: "The installer must be able to report failures to Bugzilla, with appropriate information included"

Maybe. As I read it, the criteria captures the test [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_save_traceback_to_bugzilla QA:Testcase_Anaconda_save_traceback_to_bugzilla]. I feel that might need to be expanded or a new criteria is added.

How do folks feel about adding a criteria to the [https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria Fedora_13_Beta_Release_Criteria] ... something like ... ''The installer must be able to copy failures logs to a remote system and offer a debug shell for investigating failures, and support copying failure information to a remote system.''

updates.img working is probably Beta or Final, I'd think.

Agreed. We currently have [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_updates.img_via_URL QA:Testcase_Anaconda_updates.img_via_URL] listed as an Alpha priority case. I think this is correct since the use case offers the only mechanism to work around issues once the installer is released as Alpha. Unfortunately, I can't see a Alpha (or Beta) release criteria to link this back to. Any suggested updates to the criteria to accommodate this?

With regards to [https://fedoraproject.org/wiki/QA/TestCases/PackageSetsMinimalPackageInstall QA/TestCases/PackageSetsMinimalPackageInstall]. Any concerns around obsoleting this test?

"With regards to QA/TestCases/PackageSetsMinimalPackageInstall. Any concerns around obsoleting this test? "

I think it could use more discussion. Minimal install is important to some people and I don't see anything that replaces it. We'd need some kind of definite commitment to providing a functional minimal installation, though, I guess. Something to take to devel group?

Replying to [comment:9 jlaska]:

<skip>

You make a good point. I keep going back and forth on ... if the use cases aren't release critical, should we continue testing them? Anyone have a suggested phrasing for tests we want to run, but they aren't release critical?

Wow, I keep thinking this for a looong time but still can't find a good solution. I think it depends on the goal for installation tests. We test it whether to meet the release criteria or to fully check the installation. In my view so far we design it for a full check, or we won't create the repo cases. Then I think we can have extra label(s) for our own. But it's really tough to think out one. ：）

Yes, PUD should be Alpha, I think.

Okay, thanks. Updated wiki.

traceback debug mode is Alpha, isn't it? From Alpha requirements: "The installer must be able to report failures to Bugzilla, with appropriate information included"

Maybe. As I read it, the criteria captures the test [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_save_traceback_to_bugzilla QA:Testcase_Anaconda_save_traceback_to_bugzilla]. I feel that might need to be expanded or a new criteria is added.

How do folks feel about adding a criteria to the [https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria Fedora_13_Beta_Release_Criteria] ... something like ... ''The installer must be able to copy failures logs to a remote system and offer a debug shell for investigating failures, and support copying failure information to a remote system.''

Agree.

updates.img working is probably Beta or Final, I'd think.

Agreed. We currently have [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_updates.img_via_URL QA:Testcase_Anaconda_updates.img_via_URL] listed as an Alpha priority case. I think this is correct since the use case offers the only mechanism to work around issues once the installer is released as Alpha. Unfortunately, I can't see a Alpha (or Beta) release criteria to link this back to. Any suggested updates to the criteria to accommodate this?

With regards to [https://fedoraproject.org/wiki/QA/TestCases/PackageSetsMinimalPackageInstall QA/TestCases/PackageSetsMinimalPackageInstall]. Any concerns around obsoleting this test?

Sometimes when sth wrong happens during default package installation, minimal install is another choice. So I think we can just keep it.

Replying to [comment:10 adamwill]:

"With regards to QA/TestCases/PackageSetsMinimalPackageInstall. Any concerns around obsoleting this test? "

I think it could use more discussion. Minimal install is important to some people and I don't see anything that replaces it. We'd need some kind of definite commitment to providing a functional minimal installation, though, I guess. Something to take to devel group?

Ah good point. I believe this might be a new use case slightly different from the original intent of this case. But still worth keeping. Thanks!

Thanks to clumens and dcantrell for their [https://www.redhat.com/archives/anaconda-devel-list/2010-February/msg00059.html feedback]. I've updated the test priority for the telnet, cmdline, serial console and traceback debug test cases.

• Alpha - serial and cmdline
• Final - telnet, traceback debug

If no other objections or concerns, I'll close this ticket out tomorrow.

Thank you James!

Thanks for the feedback folks, closing this out.