There's a current proposal for specific release criteria for graphical package managers. Once that proposal is finalized, assuming we do actually adopt some new criteria, we should write some test cases to cover them. Once the test cases are written, we can automate them - I'll file another ticket for that.
Assigning to @kparal for now as he's drafting the criteria.
https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/254 is the ticket for automating the test cases once they're made.
I am happy to write the test cases, if anybody hasn't taken it yet!
I was figuring @kparal or I could do it...you already got most of the tickets :D
The criterion is now live: https://fedoraproject.org/wiki/Fedora_36_Final_Release_Criteria#software-install-remove-update
Any volunteers for writing the test case (or perhaps several of them)? I certainly welcome anyone who wants to do it :-)
Metadata Update from @kparal: - Assignee reset
Can I volunteer ? @kparal
No, you've been a naughty boy and certainly can't write this test case! :-D Thanks, Sumantro. Assigned to you.
Metadata Update from @kparal: - Issue assigned to sumantrom
Oh yeah, we had this ticket. :D Thanks sumantro!
Hey @sumantrom , let me know before you start writing the cases, we should make sure they are easily testable in OpenQA :D
@sumantrom ping? what's the status here? thanks!
Note: the new test cases should subsume https://fedoraproject.org/wiki/QA:Testcase_Gnome_Software .
@kparal , maybe do you want to take this back? or @lruzicka ?
I talked with Sumantro about this on Monday, and he said he would work on this in 2-3 weeks, IIRC. Ping @sumantro.
Thanks Kamil and Adam. I am working on it already. I will keep posting the updates on @test. Thanks a lot!
Gnome Software Test Cases : https://fedoraproject.org/wiki/User:Sumantrom/Draft_Install_Remove_Gnome_Software https://fedoraproject.org/wiki/User:Sumantrom/Draft_Update_Gnome_Software https://fedoraproject.org/wiki/User:Sumantrom/Draft_list_software_details_Gnome_Software https://fedoraproject.org/wiki/User:Sumantrom/Draft_software_launch_Gnome_Software https://fedoraproject.org/wiki/User:Sumantrom/Draft_software_repo_gnome_software https://fedoraproject.org/wiki/User:Sumantrom/Draft_system_update_notification_Gnome_Software
My following comment will have KDE and followed by other default graphical package managers
KDE https://fedoraproject.org/wiki/User:Sumantrom/Draft_Install_Remove_KDE_Discover https://fedoraproject.org/wiki/User:Sumantrom/Draft_Update_Software_Discover https://fedoraproject.org/wiki/User:Sumantrom/Draft_list_software_details_Discover https://fedoraproject.org/wiki/User:Sumantrom/Draft_launch_software_Discover https://fedoraproject.org/wiki/User:Sumantrom/Draft_software_repo_discover
@kparal @adamwill ,please bless with feedbacks and scopes of improvements. I have figured out some corner cases, as I was writing through the criteria. Installing any package from Koji (downgrading a package) and then going to GNOME software, trying to check for updates. The updates tab does list an update displaying the old version installed and new version available. But when instantly searching for the same package on the search bar of Gnome Software, it tells me, installed and the version reflects the NEW version which we started from and not the one which the system has currently. Any idea if this would be nominated as blocker?
https://fedoraproject.org/wiki/User:Sumantrom/Draft_Install_Remove_Gnome_Software https://fedoraproject.org/wiki/User:Sumantrom/Draft_Install_Remove_KDE_Discover
These test cases are almost the same. If you look at https://fedoraproject.org/wiki/Template:Desktop_test_matrix , we have universal test cases which apply both to GNOME and KDE. Sometimes they can point out necessary differences (e.g. that in GNOME you start Software and in KDE you start Discover), but then the steps can be fairly generic, they don't need to specify exact button labels. Can you please unify all the test cases to be generic instead of separate for gnome-software/plasma-discover?
Also, most of the content should be in "How to test", so that you can logically follow them step by step. "Expected results" should only contain instructions which don't make sense as an individual step, e.g. something that must be true the whole time.
So this particular generic test case could look something like this (a quick draft):
= Description = This tests installing and removing packages using the default graphical package manager. (no Setup section, because no setup is necessary) = How to test = 1. Launch the default graphical package manager ("Software" in GNOME, "Discover" in KDE). 2. If there's a Featured app section (Editor's choice, app carousel, etc), pick one of the promoted apps and install it. 3. If there are app categories (like Graphics, Development, Games, etc), pick one app from any category and install it. 4. Use the search bar to find an app and install it. * If there are multiple ways to install an app, e.g. from the category/search view as well as from the app details page, try to test all available ways of installation. 5. Use the search bar to find all apps which you just installed, and verify that they're marked as installed (there's some graphical tick, or there's a button to remove it instead of installing it, etc). 6. If there's an "Installed" section, verify that the newly installed apps are listed in there and marked as installed. 7. If there's a start button (Open, Launch, etc) to start an app, use it to start all newly installed apps. 8. Verify that the newly installed apps are present using low-level tools. * For RPM packages, verify with rpm -q $packagename. * For Flatpak apps, search inflatpak list --app output. 9. Remove all newly installed apps. * If there are multiple ways to remove an app, e.g. from the category/search view as well as from the app details page, try to test all available ways of removal. 10. Verify that the removed apps are not marked as installed (in the search view, in the Installed section, in app details page, etc). 11. Verify that the removed apps are no longer present using low-level tools (see step 8). = Expected Results = All attempted actions should function properly. ## This is a bit redundant, the section could be empty as well. But it doesn't hurt. https://fedoraproject.org/wiki/User:Sumantrom/Draft_software_launch_Gnome_Software https://fedoraproject.org/wiki/User:Sumantrom/Draft_launch_software_Discover
= Description = This tests installing and removing packages using the default graphical package manager.
(no Setup section, because no setup is necessary)
= How to test = 1. Launch the default graphical package manager ("Software" in GNOME, "Discover" in KDE). 2. If there's a Featured app section (Editor's choice, app carousel, etc), pick one of the promoted apps and install it. 3. If there are app categories (like Graphics, Development, Games, etc), pick one app from any category and install it. 4. Use the search bar to find an app and install it. * If there are multiple ways to install an app, e.g. from the category/search view as well as from the app details page, try to test all available ways of installation. 5. Use the search bar to find all apps which you just installed, and verify that they're marked as installed (there's some graphical tick, or there's a button to remove it instead of installing it, etc). 6. If there's an "Installed" section, verify that the newly installed apps are listed in there and marked as installed. 7. If there's a start button (Open, Launch, etc) to start an app, use it to start all newly installed apps. 8. Verify that the newly installed apps are present using low-level tools. * For RPM packages, verify with rpm -q $packagename. * For Flatpak apps, search inflatpak list --app output. 9. Remove all newly installed apps. * If there are multiple ways to remove an app, e.g. from the category/search view as well as from the app details page, try to test all available ways of removal. 10. Verify that the removed apps are not marked as installed (in the search view, in the Installed section, in app details page, etc). 11. Verify that the removed apps are no longer present using low-level tools (see step 8).
rpm -q $packagename
flatpak list --app
= Expected Results = All attempted actions should function properly. ## This is a bit redundant, the section could be empty as well. But it doesn't hurt.
https://fedoraproject.org/wiki/User:Sumantrom/Draft_software_launch_Gnome_Software https://fedoraproject.org/wiki/User:Sumantrom/Draft_launch_software_Discover
I included this in the draft above. It's a simple operation, I don't think it deserves a standalone test case :)
https://fedoraproject.org/wiki/User:Sumantrom/Draft_Update_Gnome_Software https://fedoraproject.org/wiki/User:Sumantrom/Draft_Update_Software_Discover
We already have https://fedoraproject.org/wiki/QA:Testcase_desktop_update_graphical . But it could be extended with instructions on what to do when no update is available. I think that instead of guiding users to download specific builds from koji, it's much easier to ask them to enable updates-testing, if it's not enabled already. If it is enabled and the system is fully updates, I think the easiest approach is to run sudo dnf distrosync --disablerepo=updates-testing --assumeno, pick one package from the list and downgrade it using sudo dnf distrosync --disablerepo=updates-testing $package. Then reboot and try to update using gnome-software.
sudo dnf distrosync --disablerepo=updates-testing --assumeno
sudo dnf distrosync --disablerepo=updates-testing $package
https://fedoraproject.org/wiki/User:Sumantrom/Draft_system_update_notification_Gnome_Software
We already have https://fedoraproject.org/wiki/QA:Testcase_desktop_update_notification . It could link to instructions on how to downgrade a package, in case no packages updates are available at this moment.
https://fedoraproject.org/wiki/User:Sumantrom/Draft_software_repo_gnome_software https://fedoraproject.org/wiki/User:Sumantrom/Draft_software_repo_discover
The repo states should also be checked by low-level tools - dnf and flatpak.
Also, please add "Associated release criterion" banners to all test cases (see our existing test cases in the Desktop matrix). Thanks!
I have figured out some corner cases, as I was writing through the criteria. Installing any package from Koji (downgrading a package) and then going to GNOME software, trying to check for updates. The updates tab does list an update displaying the old version installed and new version available. But when instantly searching for the same package on the search bar of Gnome Software, it tells me, installed and the version reflects the NEW version which we started from and not the one which the system has currently. Any idea if this would be nominated as blocker?
You haven't rebooted after the downgrade, right? I don't think gnome-software is expected to automatically pick up package changes, if you perform them using other tools (like dnf/rpm). That's why I included rebooting in proposed instructions for the update test case.
I will do the needful. Thanks for the comments!
Yeah, I agree with @kparal . On the whole I think it's better for test cases to be generic, with app-specific notes only where really necessary. When you write a test case specifically to the UI of a current application, it tends to go stale very quickly; GNOME used to have a completely different GUI package manager, KDE currently has two and has changed them out multiple times in the past. Generic instructions are less subject to bitrot, and allow us to use the same test case for multiple desktops (including the non-blocking ones).
It's quite tricky to reliably use the "downgrade a package then check for updates" approach without a reboot. openQA does all this to make it work on GNOME:
https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/master/f/tests/desktop_notifications.pm#_28
for KDE, we have to do a pkcon refresh after the downgrade, IIRC.
pkcon refresh
I do think the behaviour you noticed should be reported as a bug, though. It sounds like when you refresh updates and find the update available, Software should also update its cache of installed versions, but it doesn't do that. I don't think it's a blocker, really, but it does seem like a bug.
@sumantrom did you ever get to revising these?
Login to comment on this ticket.