PR#355: Implement test for SELinux Crash notifications. - fedora-qa/os-autoinst-distri-fedora

fedora-qa / os-autoinst-distri-fedora

#355 Implement test for SELinux Crash notifications.

Closed 3 months ago by lruzicka. Opened 3 months ago by lruzicka.

error_reporting into main

Implement test for SELinux Crash notifications.

Lukas Ruzicka • 3 months ago

2bc171f

lib/utils.pm

file modified

+73 -1

		`@@ -6,7 +6,7 @@`
		`use Exporter;`
		`use lockapi;`
		`use testapi qw(is_serial_terminal :DEFAULT);`
		- our @EXPORT = qw/run_with_error_check type_safely type_very_safely desktop_vt boot_to_login_screen console_login console_switch_layout desktop_switch_layout console_loadkeys_us do_bootloader boot_decrypt check_release menu_launch_type setup_repos repo_setup get_workarounds disable_updates_repos cleanup_workaround_repo console_initial_setup handle_welcome_screen gnome_initial_setup anaconda_create_user check_desktop quit_firefox advisory_get_installed_packages acnp_handle_output advisory_check_nonmatching_packages start_with_launcher quit_with_shortcut disable_firefox_studies select_rescue_mode copy_devcdrom_as_isofile get_release_number check_left_bar check_top_bar check_prerelease check_version spell_version_number _assert_and_click is_branched rec_log repos_mirrorlist register_application get_registered_applications desktop_launch_terminal solidify_wallpaper check_and_install_git download_testdata make_serial_writable set_update_notification_timestamp kde_doublek_workaround dm_perform_login/;
		+ our @EXPORT = qw/run_with_error_check type_safely type_very_safely desktop_vt boot_to_login_screen console_login console_switch_layout desktop_switch_layout console_loadkeys_us do_bootloader boot_decrypt check_release menu_launch_type setup_repos repo_setup get_workarounds disable_updates_repos cleanup_workaround_repo console_initial_setup handle_welcome_screen gnome_initial_setup anaconda_create_user check_desktop quit_firefox advisory_get_installed_packages acnp_handle_output advisory_check_nonmatching_packages start_with_launcher quit_with_shortcut disable_firefox_studies select_rescue_mode copy_devcdrom_as_isofile get_release_number check_left_bar check_top_bar check_prerelease check_version spell_version_number _assert_and_click is_branched rec_log repos_mirrorlist register_application get_registered_applications desktop_launch_terminal solidify_wallpaper check_and_install_git download_testdata make_serial_writable set_update_notification_timestamp kde_doublek_workaround dm_perform_login check_errors_notifications check_errors_cli/;


		`# We introduce this global variable to hold the list of applications that have`
		`@@ -1838,4 +1838,76 @@`
		`send_key "ret";`
		`}`

		`+ # This subroutine tests the desktop notifications.`
		`+ # It opens the notifications, checks the notifications are empty,`
		`+ # closes them, waits some time and repeats three times.`
		`+`
		`+ sub check_errors_notifications {`
		`+ my $desktop = shift;`
		`+`
		`+ my @count = (1 .. 3);`
		`+ foreach (@count) {`
		`+ # Show notifications, check that there is no problem reported,`
		`+ # wait a couple seconds, then unshow notifications, wait again`
		`+ # and repeat times.`
		`+ if ($desktop eq "kde") {`
		`+ assert_and_click("kde_show_hidden_icons");`
		`+ assert_and_click("kde_open_notifications");`
		`+ }`
		`+ else {`
		`+ send_key("super-v");`
		`+ }`
		`+ sleep(10);`
		`+ unless (check_screen("desktop_no_notifications")) {`
		`+ record_soft_failure("Notifications are not empty!");`
		`+ last;`
		`+ }`
		`+ if ($desktop eq "kde") {`
		`+ send_key("esc");`
		`+ }`
		`+ else {`
		`+ send_key("super-v");`
		`+ }`
		`+ sleep(30);`
		`+ }`
		`+ }`
		`+`
		`+ # This subroutine checks`
		`+ sub check_errors_cli {`
		`+ my ($desktop, $term) = @_;`
		`+ my $sudo = get_var("ROOT_PASSWORD", "weakpassword");`
		`+ my $term = "terminal";`
		`+ if ($desktop eq "kde") { $term = "konsole"; }`
		`+`
		`+ # Start terminal application and check whether something has been reported.`
		`+ # Clear the terminal between commands to avoid double checking one result.`
		`+ menu_launch_type($term);`
		`+ wait_still_screen(5);`
		`+`
		`+ type_very_safely("coredumpctl --all\n");`
		`+ assert_screen("coredump_no_coredumps");`
		`+ type_very_safely("clear\n");`
		`+`
		`+ type_very_safely("abrt-cli list\n");`
		`+ assert_screen("abrt_no_problems");`
		`+ type_very_safely("clear\n");`
		`+`
		`+ type_very_safely("sudo abrt-cli list\n");`
		`+ sleep(2);`
		`+ type_string("$sudo\n");`
		`+ assert_screen("abrt_no_problems");`
		`+ type_very_safely("clear\n");`
		`+`
		`+ type_very_safely("exit\n");`
		`+`
		`+ # On KDE where SETroubleshoot is still available,`
		`+ # check its results.`
		`+ if ($desktop eq "kde") {`
		`+ menu_launch_type("selinux");`
		`+ assert_screen("selinux_no_alerts");`
		`+ }`
		`+ }`
		`+`
		`+`
		`+`
		`1;`

needles/error_checking/abrt_no_problems-20250127.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "xpos": 188,`
		`+ "ypos": 220,`
		`+ "width": 94,`
		`+ "height": 21,`
		`+ "type": "match"`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "abrt_no_problems"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/error_checking/abrt_no_problems-20250127.png

file added

empty or binary file added

needles/error_checking/abrt_no_problems-kde-20250127.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "xpos": 58,`
		`+ "ypos": 163,`
		`+ "width": 94,`
		`+ "height": 20,`
		`+ "type": "match"`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "abrt_no_problems"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/error_checking/abrt_no_problems-kde-20250127.png

file added

empty or binary file added

needles/error_checking/coredump_no_coredumps-20250121.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "xpos": 188,`
		`+ "ypos": 187,`
		`+ "width": 154,`
		`+ "height": 16,`
		`+ "type": "match"`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "coredump_no_coredumps"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/error_checking/coredump_no_coredumps-20250121.png

file added

empty or binary file added

needles/error_checking/coredump_no_coredumps-kde.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "xpos": 59,`
		`+ "ypos": 164,`
		`+ "width": 153,`
		`+ "height": 18,`
		`+ "type": "match"`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "coredump_no_coredumps"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/error_checking/coredump_no_coredumps-kde.png

file added

empty or binary file added

needles/error_checking/desktop_no_notifications-gnome-20250121.json

file added

+16

		`@@ -0,0 +1,16 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "xpos": 281,`
		`+ "ypos": 386,`
		`+ "width": 155,`
		`+ "height": 30,`
		`+ "type": "match"`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "DESKTOP-gnome",`
		`+ "desktop_no_notifications"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/error_checking/desktop_no_notifications-gnome-20250121.png

file added

empty or binary file added

needles/error_checking/desktop_no_notifications-kde-20250127.json

file added

+16

		`@@ -0,0 +1,16 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "ypos": 550,`
		`+ "xpos": 692,`
		`+ "type": "match",`
		`+ "height": 17,`
		`+ "width": 209`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "DESKTOP-kde",`
		`+ "desktop_no_notifications"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/error_checking/desktop_no_notifications-kde-20250127.png

file added

empty or binary file added

needles/error_checking/kde_open_notifications-20250127.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "xpos": 590,`
		`+ "ypos": 317,`
		`+ "width": 111,`
		`+ "height": 28,`
		`+ "type": "match"`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "kde_open_notifications"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/error_checking/kde_open_notifications-20250127.png

file added

empty or binary file added

needles/error_checking/kde_open_notifications-bell.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "xpos": 728,`
		`+ "ypos": 724,`
		`+ "width": 23,`
		`+ "height": 25,`
		`+ "type": "match"`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "kde_open_notifications"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/error_checking/kde_open_notifications-bell.png

file added

empty or binary file added

needles/error_checking/kde_open_notifications-selected.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "xpos": 590,`
		`+ "ypos": 316,`
		`+ "width": 110,`
		`+ "height": 28,`
		`+ "type": "match"`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "kde_open_notifications"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/error_checking/kde_open_notifications-selected.png

file added

empty or binary file added

needles/error_checking/kde_show_hidden_icons-20250127.json

file added

+16

		`@@ -0,0 +1,16 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "ypos": 726,`
		`+ "xpos": 876,`
		`+ "height": 22,`
		`+ "width": 23,`
		`+ "type": "match",`
		`+ "match": 90`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "kde_show_hidden_icons"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/error_checking/kde_show_hidden_icons-20250127.png

file added

empty or binary file added

needles/error_checking/selinux_no_alerts-20250127.json

file added

+22

		`@@ -0,0 +1,22 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "xpos": 441,`
		`+ "ypos": 238,`
		`+ "width": 139,`
		`+ "height": 20,`
		`+ "type": "match"`
		`+ },`
		`+ {`
		`+ "xpos": 146,`
		`+ "ypos": 271,`
		`+ "width": 69,`
		`+ "height": 21,`
		`+ "type": "match"`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "selinux_no_alerts"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/error_checking/selinux_no_alerts-20250127.png

file added

empty or binary file added

templates.fif.json

file modified

+29

		`@@ -1458,6 +1458,24 @@`
		`"START_AFTER_TEST": "%DEPLOY_UPLOAD_TEST%"`
		`}`
		`},`
		`+ "error_checking": {`
		`+ "profiles": {`
		`+ "fedora-KDE-live-iso-x86_64-*-64bit": 32,`
		`+ "fedora-Workstation-live-iso-ppc64le-*-ppc64le": 30,`
		`+ "fedora-Workstation-live-iso-x86_64-*-64bit": 30,`
		`+ "fedora-Workstation-live_osbuild-iso-x86_64-*-64bit": 70,`
		`+ "fedora-Workstation-upgrade-x86_64-*-64bit": 40,`
		`+ "fedora-Workstation-upgrade-aarch64-*-aarch64": 40,`
		`+ "fedora-Workstation-raw_xz-raw.xz-aarch64-*-aarch64": 32,`
		`+ "fedora-Workstation-live-iso-x86_64-*-64bit": 30`
		`+ },`
		`+ "settings": {`
		`+ "BOOTFROM": "c",`
		`+ "HDD_1": "disk_%FLAVOR%_%MACHINE%.qcow2",`
		`+ "POSTINSTALL": "error_checks",`
		`+ "START_AFTER_TEST": "%DEPLOY_UPLOAD_TEST%"`
		`+ }`
		`+ },`
		`"i3_tiling": {`
		`"profiles": {`
		`"fedora-i3-live-x86_64-*-64bit": 70`
		`@@ -1785,6 +1803,17 @@`
		`"PARTITIONING": "guided_delete_all"`
		`}`
		`},`
		`+ "install_error_checking": {`
		`+ "profiles": {`
		`+ "fedora-KDE-live-iso-x86_64-*-bios": 16,`
		`+ "fedora-Workstation-live-iso-x86_64-*-bios": 11`
		`+ },`
		`+ "settings": {`
		`+ "PACKAGE_SET": "default",`
		`+ "POSTINSTALL": "_collect_data",`
		`+ "ERROR_CHECK": "1"`
		`+ }`
		`+ },`
		`"install_european_language": {`
		`"profiles": {`
		`"fedora-Server-dvd-iso-aarch64-*-aarch64": 40,`

tests/_do_install_and_reboot.pm

file modified

		`@@ -108,6 +108,14 @@`
		`assert_screen "anaconda_install_done";`
		`# wait for transition to complete so we don't click in the sidebar`
		`wait_still_screen 3;`
		`+ # If error checking is ordered, perform the routines to see`
		`+ # whether any notifications or error reports have been triggered`
		`+ # during the installation.`
		`+ if (get_var("ERROR_CHECK")) {`
		`+ my $desktop = get_var("DESKTOP");`
		`+ check_errors_notifications($desktop);`
		`+ check_errors_cli(get_var($desktop));`
		`+ }`
		`# if this is a live install, let's go ahead and quit the installer`
		`# in all cases, just to make sure quitting doesn't crash etc.`
		`# not on web UI, as it immediately reboots`

tests/error_checks.pm

file added

+32

		`@@ -0,0 +1,32 @@`
		`+ use base "installedtest";`
		`+ use strict;`
		`+ use testapi;`
		`+ use utils;`
		`+`
		`+`
		`+ # This test checks that no errors are reported through automated tools after installation.`
		`+ # Currently, this checks for anything in system notifications, coredumpctl, and abrt-cli.`
		`+ # The test fails if anything is reported by these tools.`
		`+ #`
		`+`
		`+ sub clear {`
		`+ type_very_safely("clear\n");`
		`+ sleep(2);`
		`+ }`
		`+`
		`+ sub run {`
		`+ my $self = shift;`
		`+ my $desktop = get_var("DESKTOP", "gnome");`
		`+`
		`+ check_errors_notifications($desktop);`
		`+`
		`+ check_errors_cli($desktop);`
		`+ }`
		`+`
		`+ sub test_flags {`
		`+ return {fatal => 1};`
		`+ }`
		`+`
		`+ 1;`
		`+`
		`+ # vim: set sw=4 et:`

lruzicka commented 3 months ago

This test checks if any errors are reported using automatic error
reporting mechanisms. It checks whether there are any errors
reported through Desktop Notifications, Abrt, and CoredumpCTL.
In KDE, where the SETroubleshoot application is installed,
the test also checks whether there are any SELinux notifications.

The test case requires the above is checked within the Live environment,
and also post install, so the structure of this test consists of two parts:
- there are subroutines in utils.pm that are called within
_do_install_and_reboot.pm, right after the installation
completes.
- there is a error_checking.pm that utilizies the subroutines, but runs
after the installed system has been booted.

In order not to fiddle with install_default test cases, we introduce
another installation test, install_error_checking, where the
checks are triggered.

This fixes https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/354

zuul commented 3 months ago

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/652cc84ae6734c4084ce6d59fd18c9d5

os-autoinst-tox : SUCCESS in 3m 49s
os-autoinst-perl : SUCCESS in 10m 07s

rebased onto bd687aa

3 months ago

zuul commented 3 months ago

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/e56b121ab5c541aab6f25c345d31d8fc

os-autoinst-tox : SUCCESS in 3m 27s
os-autoinst-perl : SUCCESS in 9m 10s

adamwill commented 3 months ago

I think this is mostly redundant with the existing desktop_notifications tests, the stuff in the post-fail hook for uploading coredumps on failures, and _console_avc_crash.pm. Maybe we can enhance some of those with bits from this, but I don't think we should merge it as-is. sorry!

lruzicka commented 3 months ago

I think this is mostly redundant with the existing desktop_notifications tests, the stuff in the post-fail hook for uploading coredumps on failures, and _console_avc_crash.pm. Maybe we can enhance some of those with bits from this, but I don't think we should merge it as-is. sorry!

Sure, no problem, why don't we fill in the corresponding test case matrix then?

adamwill commented 3 months ago

We do, at least we're configured to? QA:Testcase_desktop_update_notification and QA:Testcase_desktop_error_checks should be marked passed if both desktop_notifications_live and desktop_notifications_postinstall pass. In the current matrix they're marked passed for GNOME but not KDE (because one of the tests failed).

lruzicka commented 3 months ago

Ok, when I checked the matrix, it was not filled in although other tests were. I am closing this.

Pull-Request has been closed by lruzicka

3 months ago

adamwill commented 3 months ago

yeah, remember it won't be filled if the tests fail :D It's probably safest to always check conf_test_suites.py. And of course if a test case is listed there but isn't getting filled on the wiki even if the associated tests pass, that's probably a fedora_openqa/wikitcms bug...