From 92cc40221855676d50bb22fa25c46499bf41800b Mon Sep 17 00:00:00 2001 From: Petr Šplíchal Date: May 05 2022 10:00:34 +0000 Subject: Update obsoleted information on the main page Add links to the `tmt` documentation and mark it as recommended. Mark Standard Test Interface and Standard Test Roles as obsoleted. Remove some old outdated stuff and broken links. --- diff --git a/modules/ROOT/pages/index.adoc b/modules/ROOT/pages/index.adoc index 85830ac..76b3dd3 100644 --- a/modules/ROOT/pages/index.adoc +++ b/modules/ROOT/pages/index.adoc @@ -34,15 +34,14 @@ There are three main pieces of the puzzle to get this nicely working: A process === Process === -==== Standard Test Interface ==== +==== Enabling Tests ==== -In order to clearly distinguish test from the CI system running it the *Standard Test Interface* was introduced. -It clearly defines essential terms such as test, test subject, test suite, test framework, test result, test artifact, test system and describes what are their responsibilities and requirements. +Tests can be enabled using the *Test Metadata Specification* implemented by the <> +which provides a number of features to efficiently work with tests and is now the recommended way. +The *Standard Test Interface* is still supported but has been obsoleted. -This general approach gives a nice flexibility as it does not enforce any specific tools or frameworks to be used. -Basically it only describes how tests are discovered and where the testing results should be stored to be processed by the automation. - -* xref:standard-test-interface.adoc[Standard Test Interface] +* https://tmt.readthedocs.io/en/latest/spec.html[Test Metadata Specification] (recommended) +* xref:standard-test-interface.adoc[Standard Test Interface] (obsoleted) ==== Gating ==== @@ -75,10 +74,21 @@ Consistent format of the CI messages allows to build simpler services which will === Tools === +==== Test Management Tool ==== + +The ``tmt`` tool provides a user-friendly way to work with tests. +You can comfortably create new tests, safely and easily run tests across different environments, +review test results, debug test code and enable tests in the CI using a consistent and concise config. + +* xref:tmt.adoc[Test Management Tool] +* https://tmt.readthedocs.io/en/latest/guide.html[Upstream Guide] + ==== Standard Test Roles ==== Standard Test Roles were implemented to enable both automation tools and developers in their local environments to easily execute tests. This set of ansible roles supports various frameworks and allows to execute tests against different test subjects (such as classic rpm package, docker container or Atomic Host). +Standard Test Roles have been obsoleted by the <>. + * xref:standard-test-roles.adoc[Standard Test Roles] * https://pagure.io/standard-test-roles[pagure.io/standard-test-roles] @@ -91,15 +101,6 @@ Here are some examples: * https://github.com/beakerlib/beakerlib/wiki/man[BeakerLib] and it's https://pagure.io/beakerlib-libraries[Libraries] * https://github.com/avocado-framework/avocado[Avocado] -==== Metadata ==== - -Standard Test Interface defines only a very simple metadata for selecting which tests should be run. -For more complex scenarios *Flexible Metadata Format* can be used: - -* Fedora CI https://pagure.io/fedora-ci/metadata[Metadata Specification] -* https://fedoraproject.org/wiki/Flexible_Metadata_Format[Flexible Metadata Format] -* https://github.com/psss/fmf[github.com/psss/fmf] - ==== Pipeline ==== The testing *Pipeline* detects tests for enabled packages, executes the test coverage and gathers the results. @@ -135,41 +136,29 @@ However, in some cases it might be worth to fetch tests for Fedora CI from the u ==== Test Code ==== -Tests may be written all sorts of different ways, but have to be exposed and invoked in a standard way. -Tests are enabled by including the `tests/tests.yml` file in the package dist-git repository as defined by the Standard Test Interface. -Test code itself can be stored directly in the dist-git or fetched from another repository. -Shared tests namespace can be used for storing test code relevant for multiple packages. - -* xref:quick-start-guide.adoc[Quick Start Guide] ... quick intro for the impatient -* xref:tests.adoc[Tests] ... how to run, write and wrap tests -* xref:share-test-code.adoc[Share Test Code] ... shared tests namespace -* xref:pull-requests.adoc[Pull Requests] ... how to create pull requests -* xref:examples.adoc[Examples] ... real-life success stories of enabled tests - -==== Test Execution ==== +Test code can be stored directly in the dist-git or fetched from another repository. +The minimal config to enable a simple smoke test looks like this: -Executing a test written with the use of Standard Test Roles is as simple as running an ansible playbook + execute: + script: foo --version - ansible-playbook tests.yml +Tests from a shared repository can be enabled in this way: -However, a set of environment variables needs to be set properly in order to execute the test against the desired test subject. -The *Tests* wiki contains detailed instructions about running tests and adding new test coverage. + discover: + how: fmf + url: https://src.fedoraproject.org/tests/shell + execute: + how: tmt -* xref:tests.adoc[Tests] ... How to run, write and wrap tests +See the xref:tmt.adoc[Test Management Tool] documentation for more details and examples. -==== Test Coverage ==== - -There are statistics pages to track how many Fedora dist-git repositories have tests in the Standard Test Format. -These pages are updated automatically. - -* https://fedoraproject.org/wiki/CI/Tests/stat[Basic Operating System Status] +==== Test Execution ==== -==== Test Porting ==== +Executing tests is as simple as running a single command: -There is an active effort to open source existing internal Red Hat tests to Fedora called Upstream First. -Tests being migrated can be found in the upstream first repository: + tmt run -* https://upstreamfirst.fedorainfracloud.org[Upstream First] +Learn more in the xref:tmt.adoc#_execute_tests[Execute Tests] section. ==== Shared Responsibility ==== @@ -191,9 +180,9 @@ If you have questions or would like to get involved: Here are some additional related links: +* xref:tmt.adoc[Test Management Tool] ... Fedora CI docs for tmt +* https://tmt.readthedocs.io/en/latest/guide.html[tmt Guide] ... upstream tmt guide * xref:standard-test-interface.adoc[Standard Test Interface] ... definition of the process * xref:standard-test-roles.adoc[Standard Test Roles] ... set of ansible roles -* https://pagure.io/fedora-ci/workshop[Workshop] ... workshop materials * xref:tests.adoc[Tests] ... executing and adding tests -* https://upstreamfirst.fedorainfracloud.org[Upstream First] ... tests to be ported to dist-git * https://fedoraproject.org/wiki/SIGs/CI[CI SIG] ... Continuous Integration Special Interest Group