standard-test-roles

#32 Beakerlib Tests can result in "FAIL" even if the test is a "PASS"
Closed Opened by rasibley.

Closed
Reopen Issue

In some cases, beakerlib tests will result in a failure when invoked using the new standard test interface, since it's not running in a "real" beakerlib environment.

For example, running the following test standalone results in a PASS, yet when invoked using the standard test interface it will result in a FAIL:
https://upstreamfirst.fedorainfracloud.org/iproute/blob/master/f/ip-neigh-sanity-test

Logfile for above example test that actually passes, yet reports as failed:

TESTOUT.log

Likely fix here: https://pagure.io/standard-test-roles/pull-request/34#

Unfortunately the current version of the PR #34 results in false UNKNOWNs being reported.

Here are two examples of tests and associated logfiles that previously PASSed, but now report as UNKNOWN:

  • https://upstreamfirst.fedorainfracloud.org/sed/blob/master/f/bz217720-uppercase-operand/runtest.sh

  • test.bz217720-uppercase-operand.log

  • https://upstreamfirst.fedorainfracloud.org/sed/blob/master/f/bz203339

  • test.bz203339.log

This issue should be fixed by PR #34, included in package update standard-test-roles-2.1 which is currently making its way to the rawhide and updates-testing repos.

Verified it's now working with the lastest version:

[vagrant@localhost iproute]$ sudo ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory) TEST_SUBJECTS="" TEST_ARTIFACTS=$PWD/artifacts ansible-playbook --tags classic tests.yml
[WARNING]: provided hosts list is empty, only localhost is available

PLAY [localhost] *****************

TASK [Gathering Facts] *****************
ok: [localhost]

TASK [standard-test-beakerlib : Add executor host] *************
changed: [localhost]

TASK [standard-test-beakerlib : Execute beakerlib tests] *************
changed: [localhost] => (item=ip-neigh-sanity-test)

TASK [standard-test-beakerlib : Make the master test summary log artifact] *********
changed: [localhost] => (item=ip-neigh-sanity-test)

TASK [standard-test-beakerlib : Pull out the logs] *************
changed: [localhost]

TASK [standard-test-beakerlib : Check the results] *************
changed: [localhost]

PLAY RECAP *******************
localhost : ok=16 changed=10 unreachable=0 failed=0

[vagrant@localhost iproute]$ cat artifacts/test.log
PASS ip-neigh-sanity-test

[vagrant@localhost iproute]$ rpm -q standard-test-roles
standard-test-roles-2.1-1.fc26.noarch
Edited by rasibley

Metadata Update from @rasibley:
- Issue status updated to: Closed (was: Open)

@merlinm It seems like we should revert my commit: 43ce1143e78d4f1abd242c9ca692354ceba7baa6

@stefw, your previous comment appears to be out of place here. Did you intend to include it with https://upstreamfirst.fedorainfracloud.org/bash/pull-request/2 ?

@merlinm I think my commit which clears the artifacts directory is at fault here. If so, it should be reverted. It was simply for convenience.

Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.