From c5c055cc4cf632847435bc3872b9a29f14f59ef2 Mon Sep 17 00:00:00 2001 From: Miroslav Vadkerti Date: Apr 10 2019 10:46:53 +0000 Subject: [PATCH 1/3] WIP: Split to Test & CI metadata Signed-off-by: Miroslav Vadkerti --- diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..b25c15b --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +*~ diff --git a/README.md b/README.md index ccfcf30..ca04b07 100644 --- a/README.md +++ b/README.md @@ -1,63 +1,2 @@ - -Metadata -======== - -Fedora CI Metadata Specification. - - -Requirements ------------- - -In order to use the [Flexible Metadata Format][fmf] effectively for the CI -testing we need to agree on the essential set of attributes to be used. -For each attribute we need to standardize: - -* name ... clear, unique, well chosen -* type ... expected type: string, number, list, dictionary -* purpose ... description of the attribute purpose - -Each attribute definition should contain at least one apt example of the -usage and user stories illustrating the motivation. - - -Attributes ----------- - -Here is the list of attributes proposed so far. Material for discussion. -Nothing final for now. - -* [summary](/fedora-ci/metadata/blob/master/f/summary.fmf) -* [description](/fedora-ci/metadata/blob/master/f/description.fmf) -* [tags](/fedora-ci/metadata/blob/master/f/tags.fmf) -* [test](/fedora-ci/metadata/blob/master/f/test.fmf) -* [path](/fedora-ci/metadata/blob/master/f/path.fmf) -* [environment](/fedora-ci/metadata/blob/master/f/environment.fmf) -* [duration](/fedora-ci/metadata/blob/master/f/duration.fmf) -* [relevancy](/fedora-ci/metadata/blob/master/f/relevancy.fmf) -* [contact](/fedora-ci/metadata/blob/master/f/contact.fmf) -* [component](/fedora-ci/metadata/blob/master/f/component.fmf) -* [tier](/fedora-ci/metadata/blob/master/f/tier.fmf) -* [provision](/fedora-ci/metadata/blob/master/f/provision.fmf) -* [disabled](/fedora-ci/metadata/blob/master/f/disabled.fmf) -* [result](/fedora-ci/metadata/blob/master/f/result.fmf) - -See [examples](/fedora-ci/metadata/blob/master/f/examples.md) to learn -how "path" and "test" attributes can be used together. - - -Status ------- - -Each attribute has its current status defined in the following way: - -* draft ... attribute definition has been proposed -* approved ... proposal has been discussed and approved -* production ... attribute is supported in the tooling - - -Links ------ - -* [Flexible Metadata Format][fmf] - -[fmf]: https://fedoraproject.org/wiki/Flexible_Metadata_Format += Metadata = +Fedora CI Metadata Specification. \ No newline at end of file diff --git a/ci/README.md b/ci/README.md new file mode 100644 index 0000000..6694846 --- /dev/null +++ b/ci/README.md @@ -0,0 +1,52 @@ += Fedora CI metadata = + +== Why? == +Specification of CI pipeline, trying to be CI agnostic and user friendly. + +== Artifacts == +- Specifies the artifact / artifacts which we are testing +- Supported artifacts: + - build + - pull-request + - update + - (?compose, ?container) +- Provides constraints for the test environment: + - architecture + - base compose + +== Steps == + +=== discover === +- which test metadata storage to use? + - provides constraints for test enviroment + - architectures supported + - disk/memory constraints + - product relevancy +- environment var + +=== provision === +- provision test environemnt according to contstraints from artifact and discover +- grouping based on testcase relevancy +- output: list of environments with list of tests which to run on them +- might fail that is cannot provision accoring to the constraints + +=== prepare === +- install artifact + - make it possible to customize according to user cases + - example: + - conflicts between rpms + - optionally add debuginfo + - install with devel module +- additional setup required + - inject arbitrary commends + - before/after +- how: + - playbook + - shell script + +=== execute === +- according to the provision output does the execution +- paralelization? + +=== report === +- email knobs \ No newline at end of file diff --git a/ci/examples/.fmf/version b/ci/examples/.fmf/version new file mode 100644 index 0000000..d00491f --- /dev/null +++ b/ci/examples/.fmf/version @@ -0,0 +1 @@ +1 diff --git a/ci/examples/artifacts.fmf b/ci/examples/artifacts.fmf new file mode 100644 index 0000000..bc153ec --- /dev/null +++ b/ci/examples/artifacts.fmf @@ -0,0 +1,19 @@ +# Basic structure of artifacts and related testsets +/test: + /pull-request: + /pep: + summary: All code must comply with the PEP8 style guide + /lint: + summary: Run pylint to catch common problems (no gating) + /build: + /smoke: + summary: Basic smoke test (Tier1) + /features: + summary: Verify important features + /update: + /basic: + summary: Run all Tier1, Tier2 and Tier3 tests + /security: + summary: Security tests (extra job to get quick results) + /integration: + summary: Integration tests with related components diff --git a/ci/examples/bed.fmf b/ci/examples/bed.fmf new file mode 100644 index 0000000..e63d80b --- /dev/null +++ b/ci/examples/bed.fmf @@ -0,0 +1,20 @@ +# Tests which need a fresh test bed for each +/test/build/smoke: + summary: Basic smoke test + # List tests manually + discover: + how: list + tests: + - test/one + - test/two + - test/three + # Setup with ansible + prepare: + how: ansible + playbooks: + - 'dependencies.yml' + - 'selenium.yml' + # Use restraint with isolation enabled + execute: + how: restraint + isolate: true diff --git a/ci/examples/gating.fmf b/ci/examples/gating.fmf new file mode 100644 index 0000000..05e5d6d --- /dev/null +++ b/ci/examples/gating.fmf @@ -0,0 +1,30 @@ +# Turn on gating for selected testsets (defined inline) +/test: + /pull-request: + /pep: + summary: All code must comply with the PEP8 style guide + # Do not allow ugly code to be merged into master + gate: + - merge-pull-request + /lint: + summary: Run pylint to catch common problems (no gating) + /build: + /smoke: + summary: Basic smoke test (Tier1) + # Basic smoke test is used by three gates + gate: + - merge-pull-request + - add-build-to-update + - add-build-to-compose + /features: + summary: Verify important features + /update: + # This enables the 'release-update' gate for all three testsets + gate: + - release-update + /basic: + summary: Run all Tier1, Tier2 and Tier3 tests + /security: + summary: Security tests (extra job to get quick results) + /integration: + summary: Integration tests with related components diff --git a/ci/examples/rpmdiff.fmf b/ci/examples/rpmdiff.fmf new file mode 100644 index 0000000..ccdfcde --- /dev/null +++ b/ci/examples/rpmdiff.fmf @@ -0,0 +1,20 @@ +# Basic example of an rpmdiff per-package config +/test: + /build: + /rpmdiff-essential: + summary: Essential rpmdiff tests (used for gating) + execute: + how: rpmdiff + tests: + - ABI symbols + - File permissions + gate: + - add-build-to-update + - add-build-to-compose + /rpmdiff-additional: + summary: Additional rpmdiff tests (informational) + execute: + how: rpmdiff + tests: + - File list + - Specfile checks diff --git a/ci/examples/separate.fmf b/ci/examples/separate.fmf new file mode 100644 index 0000000..ac94ee8 --- /dev/null +++ b/ci/examples/separate.fmf @@ -0,0 +1,43 @@ +# Turn on gating for selected testsets (separate gate section) +/test: + /pull-request: + /pep: + summary: All code must comply with the PEP8 style guide + /lint: + summary: Run pylint to catch common problems (no gating) + /build: + /smoke: + summary: Basic smoke test (Tier1) + /features: + summary: Verify important features + /update: + /basic: + summary: Run all Tier1, Tier2 and Tier3 tests + /security: + summary: Security tests (extra job to get quick results) + /integration: + summary: Integration tests with related components + +/gate: + # Gate blocking the pull request merge into master + /merge-pull-request: + test: + - /test/pull-request/pep + - /test/build/smoke + + # Gate blocking the build to be added to an update/erratum + /add-build-to-update: + test: + - /test/build/smoke + + # Gate blocking the build to be added to compose + /add-build-to-compose: + test: + - /test/build/smoke + + # Gate blocking release of an update/erratum + /release-update: + test: + - /test/update/basic + - /test/update/security + - /test/update/integration diff --git a/ci/examples/simple.fmf b/ci/examples/simple.fmf new file mode 100644 index 0000000..2fce84f --- /dev/null +++ b/ci/examples/simple.fmf @@ -0,0 +1,16 @@ +# Simple use cases should be super simple to write +/test: + /pull-request: + /validate: + prepare: + requires: python3-pep8 + execute: + how: shell + command: pep8 *.py + /build: + /smoke: + discover: + how: fmf + repo: https://src.fedoraproject.org/tests/tar.git + execute: + how: restraint diff --git a/ci/examples/tooling.fmf b/ci/examples/tooling.fmf new file mode 100644 index 0000000..29b2a01 --- /dev/null +++ b/ci/examples/tooling.fmf @@ -0,0 +1,32 @@ +# Multiple tool support for test discovery & execution +/test: + /build: + # Standard Test Inteface + /sti: + execute: + how: sti + playbooks: + - 'tests.yml' + tags: + - 'classic' + # Simple shell script + /shell: + execute: + how: shell + path: 'tests' + test: './test.sh' + environment: + PYTHON: 'python2' + # Workflow Tomorrow + /wow: + execute: + how: wow + options: '--plan 1234 --tags Tier1' + # Flexible Metadata Format + Restraint + /fmf: + discover: + how: fmf + filter: 'tier: 1, 2' + repository: https://src.fedoraproject.org/tests/python + execute: + how: restraint diff --git a/ci/examples/workflow.fmf b/ci/examples/workflow.fmf new file mode 100644 index 0000000..a40115e --- /dev/null +++ b/ci/examples/workflow.fmf @@ -0,0 +1,59 @@ +# Workflow steps: discover, provision, prepare, execute, report +/test: + # Discover relevant tests + discover: + how: fmf + # System requirements + provision: + memory: + min: '1 GB' + arch: + - 'x86_64' + # Machine configuration + prepare: + ansible: + - 'setup.yml' + # Test execution + execute: + how: restraint + # Result reporting + report: + contact: email@address.org + + # Build testing + /build: + /smoke: + summary: Basic smoke test (Tier1) + discover+: + filter: 'tier: 1' + /features: + summary: Verify important features + discover+: + filter: 'tag: functional' + + # Errata testing + /update: + provision+: + arch: + - x86_64 + - ppc64 + - s390x + /basic: + summary: Run all Tier1, Tier2 and Tier3 tests + discover+: + filter: 'tier: 1, 2, 3' + /security: + summary: Security tests (extra job to get quick results) + discover+: + filter: 'tag: security' + /integration: + summary: Integration tests with related components + discover+: + filter: 'tag: integration' + provision+: + memory: + min: '4 GB' + max: '8 GB' + prepare: + ansible: + - 'integration.yml' diff --git a/component.fmf b/component.fmf deleted file mode 100644 index 85b5763..0000000 --- a/component.fmf +++ /dev/null @@ -1,24 +0,0 @@ -name: component -type: list of strings -purpose: relevant fedora/rhel source package names -status: production - -motivation: - It's useful to be able to easily select all tests relevant for given - component or package. As they do not always have to be stored in the - same repository and because many tests cover multiple components a - dedicated field is needed. - -stories: - - As a SELinux tester testing the "checkpolicy" component I want to - run Tier1 tests for all SELinux components plus all checkpolicy - tests. - -examples: | - component: [libselinux, checkpolicy] - -notes: > - The following fmf command can be used to select test set described - by the user story above: - - fmf --key test --filter 'tags: Tier1 | component: checkpolicy' diff --git a/contact.fmf b/contact.fmf deleted file mode 100644 index 8a4c640..0000000 --- a/contact.fmf +++ /dev/null @@ -1,19 +0,0 @@ -name: contact -type: string or list of strings (name with email address) -purpose: person(s) maintaining the test -status: production - -motivation: - When there are several people collaborating on tests it's useful to - have a way how find who is responsible for what. - -stories: - - As a developer reviewing a complex failed test I would like to - contact the person who maintains the code and understands it well. - -examples: | - contact: Name Surname - - contact: - - First Person - - Second Person diff --git a/description.fmf b/description.fmf deleted file mode 100644 index cacecac..0000000 --- a/description.fmf +++ /dev/null @@ -1,24 +0,0 @@ -name: description -type: string (multi line, plain text) -purpose: detailed description of what the test does -status: production - -motivation: - For complex tests it makes sense to provide more detailed - description to better clarify what is covered by the test. - -stories: - - As a tester I come to a test code I wrote 10 years ago (so I have - absolutely no idea about it) and would like to quickly understand - what it does. - - As a developer I review existing test coverage for my component - and would like to get an overall idea what is covered without - having to read the whole test code. - -examples: | - description: | - This test checks all available wget options related to - downloading files recursively. First a tree directory - structure is created for testing. Then a file download - is performed for different recursion depth specified by - the "--level=depth" option. diff --git a/disabled.fmf b/disabled.fmf deleted file mode 100644 index d7a2095..0000000 --- a/disabled.fmf +++ /dev/null @@ -1,19 +0,0 @@ -name: disabled -type: boolean -purpose: allow disabling individual tests -status: production - -motivation: - When a test is broken and it is not possible to fix it immediately - there should be an easy way how to temporarily mark it as disabled - so that it's skipped during the test suite execution. - -stories: - - As a developer/tester I want selected tests to be skipped during - test execution by marking them as disabled. - -examples: | - disabled: yes - -notes: - The default value is "disabled: no". diff --git a/duration.fmf b/duration.fmf deleted file mode 100644 index a3ae536..0000000 --- a/duration.fmf +++ /dev/null @@ -1,21 +0,0 @@ -name: duration -type: string -purpose: maximum time for test execution -status: approved - -motivation: - In order to prevent stuck tests consuming resources we define a - maximum time for test execution. If the limit is exceeded the - running test is killed by the test harness. - -stories: - - As a developer/tester I want to prevent resource wasting by stuck - tests. - - As a test harness I need to know after how long time I should kill - test if it is still running. - -notes: - Use the same format as the "sleep" command. For example: 3m, 2h, 1d. - -examples: | - duration: 5m diff --git a/environment.fmf b/environment.fmf deleted file mode 100644 index 9f6c047..0000000 --- a/environment.fmf +++ /dev/null @@ -1,22 +0,0 @@ -name: environment -type: dictionary -purpose: environment variables to be set before running the test -status: approved - -motivation: - Test scripts might require certain environment variables to be set. - Although this can be done on the shell command line as part of the - "test" attribute it makes sense to have a dedicated field for this, - especially when the number of parameters grows. This might be useful - for virtual test cases as well. - -stories: - - As a tester I need to pass environment variables to my test script - to properly execute the desired test scenario. - - As a tester I'm using a single test script for testing different - Python implementations specified by environment variable PYTHON. - -examples: | - environment: - PACKAGE: python37 - PYTHON: python3.7 diff --git a/examples.md b/examples.md deleted file mode 100644 index 71b367c..0000000 --- a/examples.md +++ /dev/null @@ -1,95 +0,0 @@ -Examples -======== - -Below you can find some basic examples using the metadata which -have been already defined. - -BeakerLib Tests ---------------- - -Three beakerlib tests, each in it's own directory: - -main.fmf - - test: ./runtest.sh - - /one: - path: /tests/one - /two: - path: /tests/two - /three: - path: /tests/three - -fmf - - /one - path: tests/one - test: ./runtest.sh - - /two - path: tests/two - test: ./runtest.sh - - /three - path: tests/three - test: ./runtest.sh - -Three Scripts -------------- - -Three different script residing in a single directory: - -main.fmf - - path: /tests - - /one: - test: ./one - /two: - test: ./two - /three: - test: ./three - -fmf - - /one - path: /tests - test: ./one - - /two - path: /tests - test: ./two - - /three - path: /tests - test: ./three - -Virtual Tests -------------- - -Thre virtual test cases based on a single test script: - -main.fmf - - path: /tests/virtual - - /one: - test: ./script --one - /two: - test: ./script --two - /three: - test: ./script --three - -fmf - - /one - path: /tests/virtual - test: ./script --one - - /two - path: /tests/virtual - test: ./script --two - - /three - path: /tests/virtual - test: ./script --three diff --git a/examples/.fmf/version b/examples/.fmf/version deleted file mode 100644 index d00491f..0000000 --- a/examples/.fmf/version +++ /dev/null @@ -1 +0,0 @@ -1 diff --git a/examples/artifacts.fmf b/examples/artifacts.fmf deleted file mode 100644 index bc153ec..0000000 --- a/examples/artifacts.fmf +++ /dev/null @@ -1,19 +0,0 @@ -# Basic structure of artifacts and related testsets -/test: - /pull-request: - /pep: - summary: All code must comply with the PEP8 style guide - /lint: - summary: Run pylint to catch common problems (no gating) - /build: - /smoke: - summary: Basic smoke test (Tier1) - /features: - summary: Verify important features - /update: - /basic: - summary: Run all Tier1, Tier2 and Tier3 tests - /security: - summary: Security tests (extra job to get quick results) - /integration: - summary: Integration tests with related components diff --git a/examples/bed.fmf b/examples/bed.fmf deleted file mode 100644 index e63d80b..0000000 --- a/examples/bed.fmf +++ /dev/null @@ -1,20 +0,0 @@ -# Tests which need a fresh test bed for each -/test/build/smoke: - summary: Basic smoke test - # List tests manually - discover: - how: list - tests: - - test/one - - test/two - - test/three - # Setup with ansible - prepare: - how: ansible - playbooks: - - 'dependencies.yml' - - 'selenium.yml' - # Use restraint with isolation enabled - execute: - how: restraint - isolate: true diff --git a/examples/gating.fmf b/examples/gating.fmf deleted file mode 100644 index 05e5d6d..0000000 --- a/examples/gating.fmf +++ /dev/null @@ -1,30 +0,0 @@ -# Turn on gating for selected testsets (defined inline) -/test: - /pull-request: - /pep: - summary: All code must comply with the PEP8 style guide - # Do not allow ugly code to be merged into master - gate: - - merge-pull-request - /lint: - summary: Run pylint to catch common problems (no gating) - /build: - /smoke: - summary: Basic smoke test (Tier1) - # Basic smoke test is used by three gates - gate: - - merge-pull-request - - add-build-to-update - - add-build-to-compose - /features: - summary: Verify important features - /update: - # This enables the 'release-update' gate for all three testsets - gate: - - release-update - /basic: - summary: Run all Tier1, Tier2 and Tier3 tests - /security: - summary: Security tests (extra job to get quick results) - /integration: - summary: Integration tests with related components diff --git a/examples/rpmdiff.fmf b/examples/rpmdiff.fmf deleted file mode 100644 index ccdfcde..0000000 --- a/examples/rpmdiff.fmf +++ /dev/null @@ -1,20 +0,0 @@ -# Basic example of an rpmdiff per-package config -/test: - /build: - /rpmdiff-essential: - summary: Essential rpmdiff tests (used for gating) - execute: - how: rpmdiff - tests: - - ABI symbols - - File permissions - gate: - - add-build-to-update - - add-build-to-compose - /rpmdiff-additional: - summary: Additional rpmdiff tests (informational) - execute: - how: rpmdiff - tests: - - File list - - Specfile checks diff --git a/examples/separate.fmf b/examples/separate.fmf deleted file mode 100644 index ac94ee8..0000000 --- a/examples/separate.fmf +++ /dev/null @@ -1,43 +0,0 @@ -# Turn on gating for selected testsets (separate gate section) -/test: - /pull-request: - /pep: - summary: All code must comply with the PEP8 style guide - /lint: - summary: Run pylint to catch common problems (no gating) - /build: - /smoke: - summary: Basic smoke test (Tier1) - /features: - summary: Verify important features - /update: - /basic: - summary: Run all Tier1, Tier2 and Tier3 tests - /security: - summary: Security tests (extra job to get quick results) - /integration: - summary: Integration tests with related components - -/gate: - # Gate blocking the pull request merge into master - /merge-pull-request: - test: - - /test/pull-request/pep - - /test/build/smoke - - # Gate blocking the build to be added to an update/erratum - /add-build-to-update: - test: - - /test/build/smoke - - # Gate blocking the build to be added to compose - /add-build-to-compose: - test: - - /test/build/smoke - - # Gate blocking release of an update/erratum - /release-update: - test: - - /test/update/basic - - /test/update/security - - /test/update/integration diff --git a/examples/simple.fmf b/examples/simple.fmf deleted file mode 100644 index 2fce84f..0000000 --- a/examples/simple.fmf +++ /dev/null @@ -1,16 +0,0 @@ -# Simple use cases should be super simple to write -/test: - /pull-request: - /validate: - prepare: - requires: python3-pep8 - execute: - how: shell - command: pep8 *.py - /build: - /smoke: - discover: - how: fmf - repo: https://src.fedoraproject.org/tests/tar.git - execute: - how: restraint diff --git a/examples/tooling.fmf b/examples/tooling.fmf deleted file mode 100644 index 29b2a01..0000000 --- a/examples/tooling.fmf +++ /dev/null @@ -1,32 +0,0 @@ -# Multiple tool support for test discovery & execution -/test: - /build: - # Standard Test Inteface - /sti: - execute: - how: sti - playbooks: - - 'tests.yml' - tags: - - 'classic' - # Simple shell script - /shell: - execute: - how: shell - path: 'tests' - test: './test.sh' - environment: - PYTHON: 'python2' - # Workflow Tomorrow - /wow: - execute: - how: wow - options: '--plan 1234 --tags Tier1' - # Flexible Metadata Format + Restraint - /fmf: - discover: - how: fmf - filter: 'tier: 1, 2' - repository: https://src.fedoraproject.org/tests/python - execute: - how: restraint diff --git a/examples/workflow.fmf b/examples/workflow.fmf deleted file mode 100644 index a40115e..0000000 --- a/examples/workflow.fmf +++ /dev/null @@ -1,59 +0,0 @@ -# Workflow steps: discover, provision, prepare, execute, report -/test: - # Discover relevant tests - discover: - how: fmf - # System requirements - provision: - memory: - min: '1 GB' - arch: - - 'x86_64' - # Machine configuration - prepare: - ansible: - - 'setup.yml' - # Test execution - execute: - how: restraint - # Result reporting - report: - contact: email@address.org - - # Build testing - /build: - /smoke: - summary: Basic smoke test (Tier1) - discover+: - filter: 'tier: 1' - /features: - summary: Verify important features - discover+: - filter: 'tag: functional' - - # Errata testing - /update: - provision+: - arch: - - x86_64 - - ppc64 - - s390x - /basic: - summary: Run all Tier1, Tier2 and Tier3 tests - discover+: - filter: 'tier: 1, 2, 3' - /security: - summary: Security tests (extra job to get quick results) - discover+: - filter: 'tag: security' - /integration: - summary: Integration tests with related components - discover+: - filter: 'tag: integration' - provision+: - memory: - min: '4 GB' - max: '8 GB' - prepare: - ansible: - - 'integration.yml' diff --git a/path.fmf b/path.fmf deleted file mode 100644 index dee533b..0000000 --- a/path.fmf +++ /dev/null @@ -1,25 +0,0 @@ -name: path -type: string -purpose: filesystem directory to be entered before executing the test -status: approved - -motivation: - As the object hierarchy does not need to copy the filesystem - structure (e.g. when using virtual test cases) we need a way how to - define where the test is located. - -stories: - - As a test writer I define two virtual test cases, both using the - same script for executing. - -notes: - Automation is expected to change directory to provided path starting - from the fmf tree root directory before executing the test. Use - absolute path starting with slash. If path is not defined, object - name (identifier) is used by default. - -examples: | - path: /protocols/https - -links: - - http://fmf.readthedocs.io/en/latest/features.html#virtual diff --git a/provision.fmf b/provision.fmf deleted file mode 100644 index 7e06bd2..0000000 --- a/provision.fmf +++ /dev/null @@ -1,31 +0,0 @@ -name: provision -type: dictionary -purpose: set of environment requirements -status: production - -motivation: - In some cases tests have special requirements for the environment in - order to run successfully. For now just simple qemu options for the - standard-inventory-qcow2 provisioner are supported. - -stories: - - As a tester I want to specify amount of the memory which needs to - be available for the test. - - As a tester I want to specify network interface card to be used in - qemu. - -examples: | - provision: - standard-inventory-qcow2: - qemu: - m: 3G - net_nic: - model: e1000 - -notes: - Memory size is specified in megabytes. Optionally, a suffix of “M” - or “G”. Use qemu-system-x86_64 -net nic,model=help for a list of - available devices. See links for real life example usage. - -links: - - https://pagure.io/pschindl-osci/blob/master/f/provision.fmf diff --git a/relevancy.fmf b/relevancy.fmf deleted file mode 100644 index ee161f3..0000000 --- a/relevancy.fmf +++ /dev/null @@ -1,25 +0,0 @@ -name: relevancy -type: list -purpose: Test Case Relevancy rules used for filtering relevant test cases -status: approved - -motivation: - Sometimes a test case is only relevant for specific environment. - Test Case Relevancy allows to filter irrelevant test cases out. - -stories: - - As a tester I want to skip execution of a particular test case in - given test environment. - -examples: | - relevancy: - - "distro < f-28: False" - - "distro = rhel-7 & arch = ppc64: False" - -notes: > - Environment is defined by one or more environment dimensions such as - product, distro, collection, variant, arch, component. Relevancy - consists of a set of rules of the form "condition: decision". - -links: - - https://fedoraproject.org/wiki/CI/Test_Case_Relevancy diff --git a/result.fmf b/result.fmf deleted file mode 100644 index 1fb4b02..0000000 --- a/result.fmf +++ /dev/null @@ -1,29 +0,0 @@ -name: result -type: string -purpose: specify how test result should be interpreted -status: approved - -motivation: - Even if a test fails it might makes sense to execute it to be able - to manually review the results (ignore test result) or ensure the - behaviour has not unexpectedly changed and the test is still failing - (expected fail). - -stories: - - As a developer I have a test, I know it's failing for now, yet I - want to have it for future reference. - - As a tester I want to regularly execute the test but temporarily - ignore test result until more investigation is done and the test - can be fixed properly. - -notes: - The following values should be supported: - - respect: test result is respected (fails when test failed) - - ignore: ignore the test result (test always passes) - - xfail: expected fail (pass when test fails, fail when test passes) - -examples: | - result: ignore - -notes: - The default value is "result: respect". diff --git a/summary.fmf b/summary.fmf deleted file mode 100644 index 78b081a..0000000 --- a/summary.fmf +++ /dev/null @@ -1,18 +0,0 @@ -name: summary -type: string (one line, up to 50 characters) -purpose: concise summary of what the test does -status: production - -motivation: - In order to efficiently collaborate on test maintenance it's - crucial to have a short summary of what the test does. - -stories: - - As a developer reviewing multiple failed tests I would like - to get quickly an idea of what my change broke. - -examples: | - summary: Test wget recursive download options - -links: - - https://stackoverflow.com/questions/2290016/git-commit-messages-50-72-formatting diff --git a/tags.fmf b/tags.fmf deleted file mode 100644 index 398f0f7..0000000 --- a/tags.fmf +++ /dev/null @@ -1,20 +0,0 @@ -name: tags -type: list -purpose: free-form tags for easy filtering -status: production - -motivation: - Throughout the years, free-form tags proved to be useful for many, - many scenarios. Primarily to provide an easy way how to select a - subset of objects. - -stories: - - As a developer/tester I would like to run only a subset of - available tests. - -examples: | - tags: [Tier1, fast] - -notes: - Tags are case-sensitive. - Using lowercase is recommended. diff --git a/test.fmf b/test.fmf deleted file mode 100644 index a1dc049..0000000 --- a/test.fmf +++ /dev/null @@ -1,16 +0,0 @@ -name: test -type: string -purpose: shell command which executes the test -status: approved - -motivation: - This attribute defines how the test is to be executed. - -stories: - - As a developer/tester I want to easily execute all available tests - with just one command. - - As a test writer I want to run a single test script in multiple - ways (e.g. providing different parameters) - -examples: | - test: ./runtest.sh diff --git a/test/README.md b/test/README.md new file mode 100644 index 0000000..ccfcf30 --- /dev/null +++ b/test/README.md @@ -0,0 +1,63 @@ + +Metadata +======== + +Fedora CI Metadata Specification. + + +Requirements +------------ + +In order to use the [Flexible Metadata Format][fmf] effectively for the CI +testing we need to agree on the essential set of attributes to be used. +For each attribute we need to standardize: + +* name ... clear, unique, well chosen +* type ... expected type: string, number, list, dictionary +* purpose ... description of the attribute purpose + +Each attribute definition should contain at least one apt example of the +usage and user stories illustrating the motivation. + + +Attributes +---------- + +Here is the list of attributes proposed so far. Material for discussion. +Nothing final for now. + +* [summary](/fedora-ci/metadata/blob/master/f/summary.fmf) +* [description](/fedora-ci/metadata/blob/master/f/description.fmf) +* [tags](/fedora-ci/metadata/blob/master/f/tags.fmf) +* [test](/fedora-ci/metadata/blob/master/f/test.fmf) +* [path](/fedora-ci/metadata/blob/master/f/path.fmf) +* [environment](/fedora-ci/metadata/blob/master/f/environment.fmf) +* [duration](/fedora-ci/metadata/blob/master/f/duration.fmf) +* [relevancy](/fedora-ci/metadata/blob/master/f/relevancy.fmf) +* [contact](/fedora-ci/metadata/blob/master/f/contact.fmf) +* [component](/fedora-ci/metadata/blob/master/f/component.fmf) +* [tier](/fedora-ci/metadata/blob/master/f/tier.fmf) +* [provision](/fedora-ci/metadata/blob/master/f/provision.fmf) +* [disabled](/fedora-ci/metadata/blob/master/f/disabled.fmf) +* [result](/fedora-ci/metadata/blob/master/f/result.fmf) + +See [examples](/fedora-ci/metadata/blob/master/f/examples.md) to learn +how "path" and "test" attributes can be used together. + + +Status +------ + +Each attribute has its current status defined in the following way: + +* draft ... attribute definition has been proposed +* approved ... proposal has been discussed and approved +* production ... attribute is supported in the tooling + + +Links +----- + +* [Flexible Metadata Format][fmf] + +[fmf]: https://fedoraproject.org/wiki/Flexible_Metadata_Format diff --git a/test/component.fmf b/test/component.fmf new file mode 100644 index 0000000..85b5763 --- /dev/null +++ b/test/component.fmf @@ -0,0 +1,24 @@ +name: component +type: list of strings +purpose: relevant fedora/rhel source package names +status: production + +motivation: + It's useful to be able to easily select all tests relevant for given + component or package. As they do not always have to be stored in the + same repository and because many tests cover multiple components a + dedicated field is needed. + +stories: + - As a SELinux tester testing the "checkpolicy" component I want to + run Tier1 tests for all SELinux components plus all checkpolicy + tests. + +examples: | + component: [libselinux, checkpolicy] + +notes: > + The following fmf command can be used to select test set described + by the user story above: + + fmf --key test --filter 'tags: Tier1 | component: checkpolicy' diff --git a/test/contact.fmf b/test/contact.fmf new file mode 100644 index 0000000..8a4c640 --- /dev/null +++ b/test/contact.fmf @@ -0,0 +1,19 @@ +name: contact +type: string or list of strings (name with email address) +purpose: person(s) maintaining the test +status: production + +motivation: + When there are several people collaborating on tests it's useful to + have a way how find who is responsible for what. + +stories: + - As a developer reviewing a complex failed test I would like to + contact the person who maintains the code and understands it well. + +examples: | + contact: Name Surname + + contact: + - First Person + - Second Person diff --git a/test/description.fmf b/test/description.fmf new file mode 100644 index 0000000..cacecac --- /dev/null +++ b/test/description.fmf @@ -0,0 +1,24 @@ +name: description +type: string (multi line, plain text) +purpose: detailed description of what the test does +status: production + +motivation: + For complex tests it makes sense to provide more detailed + description to better clarify what is covered by the test. + +stories: + - As a tester I come to a test code I wrote 10 years ago (so I have + absolutely no idea about it) and would like to quickly understand + what it does. + - As a developer I review existing test coverage for my component + and would like to get an overall idea what is covered without + having to read the whole test code. + +examples: | + description: | + This test checks all available wget options related to + downloading files recursively. First a tree directory + structure is created for testing. Then a file download + is performed for different recursion depth specified by + the "--level=depth" option. diff --git a/test/disabled.fmf b/test/disabled.fmf new file mode 100644 index 0000000..d7a2095 --- /dev/null +++ b/test/disabled.fmf @@ -0,0 +1,19 @@ +name: disabled +type: boolean +purpose: allow disabling individual tests +status: production + +motivation: + When a test is broken and it is not possible to fix it immediately + there should be an easy way how to temporarily mark it as disabled + so that it's skipped during the test suite execution. + +stories: + - As a developer/tester I want selected tests to be skipped during + test execution by marking them as disabled. + +examples: | + disabled: yes + +notes: + The default value is "disabled: no". diff --git a/test/duration.fmf b/test/duration.fmf new file mode 100644 index 0000000..a3ae536 --- /dev/null +++ b/test/duration.fmf @@ -0,0 +1,21 @@ +name: duration +type: string +purpose: maximum time for test execution +status: approved + +motivation: + In order to prevent stuck tests consuming resources we define a + maximum time for test execution. If the limit is exceeded the + running test is killed by the test harness. + +stories: + - As a developer/tester I want to prevent resource wasting by stuck + tests. + - As a test harness I need to know after how long time I should kill + test if it is still running. + +notes: + Use the same format as the "sleep" command. For example: 3m, 2h, 1d. + +examples: | + duration: 5m diff --git a/test/environment.fmf b/test/environment.fmf new file mode 100644 index 0000000..9f6c047 --- /dev/null +++ b/test/environment.fmf @@ -0,0 +1,22 @@ +name: environment +type: dictionary +purpose: environment variables to be set before running the test +status: approved + +motivation: + Test scripts might require certain environment variables to be set. + Although this can be done on the shell command line as part of the + "test" attribute it makes sense to have a dedicated field for this, + especially when the number of parameters grows. This might be useful + for virtual test cases as well. + +stories: + - As a tester I need to pass environment variables to my test script + to properly execute the desired test scenario. + - As a tester I'm using a single test script for testing different + Python implementations specified by environment variable PYTHON. + +examples: | + environment: + PACKAGE: python37 + PYTHON: python3.7 diff --git a/test/examples.md b/test/examples.md new file mode 100644 index 0000000..71b367c --- /dev/null +++ b/test/examples.md @@ -0,0 +1,95 @@ +Examples +======== + +Below you can find some basic examples using the metadata which +have been already defined. + +BeakerLib Tests +--------------- + +Three beakerlib tests, each in it's own directory: + +main.fmf + + test: ./runtest.sh + + /one: + path: /tests/one + /two: + path: /tests/two + /three: + path: /tests/three + +fmf + + /one + path: tests/one + test: ./runtest.sh + + /two + path: tests/two + test: ./runtest.sh + + /three + path: tests/three + test: ./runtest.sh + +Three Scripts +------------- + +Three different script residing in a single directory: + +main.fmf + + path: /tests + + /one: + test: ./one + /two: + test: ./two + /three: + test: ./three + +fmf + + /one + path: /tests + test: ./one + + /two + path: /tests + test: ./two + + /three + path: /tests + test: ./three + +Virtual Tests +------------- + +Thre virtual test cases based on a single test script: + +main.fmf + + path: /tests/virtual + + /one: + test: ./script --one + /two: + test: ./script --two + /three: + test: ./script --three + +fmf + + /one + path: /tests/virtual + test: ./script --one + + /two + path: /tests/virtual + test: ./script --two + + /three + path: /tests/virtual + test: ./script --three diff --git a/test/path.fmf b/test/path.fmf new file mode 100644 index 0000000..dee533b --- /dev/null +++ b/test/path.fmf @@ -0,0 +1,25 @@ +name: path +type: string +purpose: filesystem directory to be entered before executing the test +status: approved + +motivation: + As the object hierarchy does not need to copy the filesystem + structure (e.g. when using virtual test cases) we need a way how to + define where the test is located. + +stories: + - As a test writer I define two virtual test cases, both using the + same script for executing. + +notes: + Automation is expected to change directory to provided path starting + from the fmf tree root directory before executing the test. Use + absolute path starting with slash. If path is not defined, object + name (identifier) is used by default. + +examples: | + path: /protocols/https + +links: + - http://fmf.readthedocs.io/en/latest/features.html#virtual diff --git a/test/provision.fmf b/test/provision.fmf new file mode 100644 index 0000000..7e06bd2 --- /dev/null +++ b/test/provision.fmf @@ -0,0 +1,31 @@ +name: provision +type: dictionary +purpose: set of environment requirements +status: production + +motivation: + In some cases tests have special requirements for the environment in + order to run successfully. For now just simple qemu options for the + standard-inventory-qcow2 provisioner are supported. + +stories: + - As a tester I want to specify amount of the memory which needs to + be available for the test. + - As a tester I want to specify network interface card to be used in + qemu. + +examples: | + provision: + standard-inventory-qcow2: + qemu: + m: 3G + net_nic: + model: e1000 + +notes: + Memory size is specified in megabytes. Optionally, a suffix of “M” + or “G”. Use qemu-system-x86_64 -net nic,model=help for a list of + available devices. See links for real life example usage. + +links: + - https://pagure.io/pschindl-osci/blob/master/f/provision.fmf diff --git a/test/relevancy.fmf b/test/relevancy.fmf new file mode 100644 index 0000000..ee161f3 --- /dev/null +++ b/test/relevancy.fmf @@ -0,0 +1,25 @@ +name: relevancy +type: list +purpose: Test Case Relevancy rules used for filtering relevant test cases +status: approved + +motivation: + Sometimes a test case is only relevant for specific environment. + Test Case Relevancy allows to filter irrelevant test cases out. + +stories: + - As a tester I want to skip execution of a particular test case in + given test environment. + +examples: | + relevancy: + - "distro < f-28: False" + - "distro = rhel-7 & arch = ppc64: False" + +notes: > + Environment is defined by one or more environment dimensions such as + product, distro, collection, variant, arch, component. Relevancy + consists of a set of rules of the form "condition: decision". + +links: + - https://fedoraproject.org/wiki/CI/Test_Case_Relevancy diff --git a/test/result.fmf b/test/result.fmf new file mode 100644 index 0000000..1fb4b02 --- /dev/null +++ b/test/result.fmf @@ -0,0 +1,29 @@ +name: result +type: string +purpose: specify how test result should be interpreted +status: approved + +motivation: + Even if a test fails it might makes sense to execute it to be able + to manually review the results (ignore test result) or ensure the + behaviour has not unexpectedly changed and the test is still failing + (expected fail). + +stories: + - As a developer I have a test, I know it's failing for now, yet I + want to have it for future reference. + - As a tester I want to regularly execute the test but temporarily + ignore test result until more investigation is done and the test + can be fixed properly. + +notes: + The following values should be supported: + - respect: test result is respected (fails when test failed) + - ignore: ignore the test result (test always passes) + - xfail: expected fail (pass when test fails, fail when test passes) + +examples: | + result: ignore + +notes: + The default value is "result: respect". diff --git a/test/summary.fmf b/test/summary.fmf new file mode 100644 index 0000000..78b081a --- /dev/null +++ b/test/summary.fmf @@ -0,0 +1,18 @@ +name: summary +type: string (one line, up to 50 characters) +purpose: concise summary of what the test does +status: production + +motivation: + In order to efficiently collaborate on test maintenance it's + crucial to have a short summary of what the test does. + +stories: + - As a developer reviewing multiple failed tests I would like + to get quickly an idea of what my change broke. + +examples: | + summary: Test wget recursive download options + +links: + - https://stackoverflow.com/questions/2290016/git-commit-messages-50-72-formatting diff --git a/test/tags.fmf b/test/tags.fmf new file mode 100644 index 0000000..398f0f7 --- /dev/null +++ b/test/tags.fmf @@ -0,0 +1,20 @@ +name: tags +type: list +purpose: free-form tags for easy filtering +status: production + +motivation: + Throughout the years, free-form tags proved to be useful for many, + many scenarios. Primarily to provide an easy way how to select a + subset of objects. + +stories: + - As a developer/tester I would like to run only a subset of + available tests. + +examples: | + tags: [Tier1, fast] + +notes: + Tags are case-sensitive. + Using lowercase is recommended. diff --git a/test/test.fmf b/test/test.fmf new file mode 100644 index 0000000..a1dc049 --- /dev/null +++ b/test/test.fmf @@ -0,0 +1,16 @@ +name: test +type: string +purpose: shell command which executes the test +status: approved + +motivation: + This attribute defines how the test is to be executed. + +stories: + - As a developer/tester I want to easily execute all available tests + with just one command. + - As a test writer I want to run a single test script in multiple + ways (e.g. providing different parameters) + +examples: | + test: ./runtest.sh diff --git a/test/tier.fmf b/test/tier.fmf new file mode 100644 index 0000000..e1b809c --- /dev/null +++ b/test/tier.fmf @@ -0,0 +1,19 @@ +name: tier +type: string +purpose: name of the tier set this test belongs to +status: production + +motivation: + It's quite common to organize tests into "tiers" based on their + importance, stability, duration and other aspects. For this tags + have been used quite often as there was not corresponding attribute + available. It might make sense to have a dedicated field for this + functionality as well. + +stories: + - As a tester testing a security advisory I want to run the stable + set of important tests which cover the most essential + functionality and can provide test results in a short time. + +examples: | + tier: 1 diff --git a/tier.fmf b/tier.fmf deleted file mode 100644 index e1b809c..0000000 --- a/tier.fmf +++ /dev/null @@ -1,19 +0,0 @@ -name: tier -type: string -purpose: name of the tier set this test belongs to -status: production - -motivation: - It's quite common to organize tests into "tiers" based on their - importance, stability, duration and other aspects. For this tags - have been used quite often as there was not corresponding attribute - available. It might make sense to have a dedicated field for this - functionality as well. - -stories: - - As a tester testing a security advisory I want to run the stable - set of important tests which cover the most essential - functionality and can provide test results in a short time. - -examples: | - tier: 1 From 459ed289abdfff2ea45521b66f672525b996dbd3 Mon Sep 17 00:00:00 2001 From: Petr Šplíchal Date: Apr 11 2019 14:43:39 +0000 Subject: [PATCH 2/3] Main README, L2 README, simplified structure --- diff --git a/.fmf/version b/.fmf/version deleted file mode 100644 index d00491f..0000000 --- a/.fmf/version +++ /dev/null @@ -1 +0,0 @@ -1 diff --git a/README.md b/README.md index ca04b07..ad49789 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,46 @@ -= Metadata = -Fedora CI Metadata Specification. \ No newline at end of file + +Metadata +======== + +This is Fedora CI Metadata Specification which defines that all +data needed for test execution in the CI system are stored as a +plain text information in [Flexible Metadata Format][fmf] under +version control in the git repository close to the test code or +source code. There are two levels of metadata defined: + + +L1 Metadata +----------- + +These are test metadata closely related to individual test cases +for which it make sense to store them directly with the test code. +Examples of such metadata are `summary`, `description`, `duration` +or `tags`. See Level 1 Metadata section for more detailed +information and list of already defined attributes. + +* [Level 1 Metadata][l1] + + +L2 Metadata +----------- + +These are CI metadata which contain information for execution of +multiple test cases such as how the environment for testing should +be prepared, which set of test cases is relevant for testing +specific artifact or which frameworks should be used for +execution. See Level 2 Metadata section for detailed +specification. + +* [Level 2 Metadata][l2] + + +Links +----- + +Links for further reading: + +* [Flexible Metadata Format][fmf] + +[fmf]: https://fedoraproject.org/wiki/Flexible_Metadata_Format +[l1]: https://pagure.io/fedora-ci/metadata/blob/master/f/l1 +[l2]: https://pagure.io/fedora-ci/metadata/blob/master/f/l2 diff --git a/ci/README.md b/ci/README.md deleted file mode 100644 index 6694846..0000000 --- a/ci/README.md +++ /dev/null @@ -1,52 +0,0 @@ -= Fedora CI metadata = - -== Why? == -Specification of CI pipeline, trying to be CI agnostic and user friendly. - -== Artifacts == -- Specifies the artifact / artifacts which we are testing -- Supported artifacts: - - build - - pull-request - - update - - (?compose, ?container) -- Provides constraints for the test environment: - - architecture - - base compose - -== Steps == - -=== discover === -- which test metadata storage to use? - - provides constraints for test enviroment - - architectures supported - - disk/memory constraints - - product relevancy -- environment var - -=== provision === -- provision test environemnt according to contstraints from artifact and discover -- grouping based on testcase relevancy -- output: list of environments with list of tests which to run on them -- might fail that is cannot provision accoring to the constraints - -=== prepare === -- install artifact - - make it possible to customize according to user cases - - example: - - conflicts between rpms - - optionally add debuginfo - - install with devel module -- additional setup required - - inject arbitrary commends - - before/after -- how: - - playbook - - shell script - -=== execute === -- according to the provision output does the execution -- paralelization? - -=== report === -- email knobs \ No newline at end of file diff --git a/ci/examples/.fmf/version b/ci/examples/.fmf/version deleted file mode 100644 index d00491f..0000000 --- a/ci/examples/.fmf/version +++ /dev/null @@ -1 +0,0 @@ -1 diff --git a/ci/examples/artifacts.fmf b/ci/examples/artifacts.fmf deleted file mode 100644 index bc153ec..0000000 --- a/ci/examples/artifacts.fmf +++ /dev/null @@ -1,19 +0,0 @@ -# Basic structure of artifacts and related testsets -/test: - /pull-request: - /pep: - summary: All code must comply with the PEP8 style guide - /lint: - summary: Run pylint to catch common problems (no gating) - /build: - /smoke: - summary: Basic smoke test (Tier1) - /features: - summary: Verify important features - /update: - /basic: - summary: Run all Tier1, Tier2 and Tier3 tests - /security: - summary: Security tests (extra job to get quick results) - /integration: - summary: Integration tests with related components diff --git a/ci/examples/bed.fmf b/ci/examples/bed.fmf deleted file mode 100644 index e63d80b..0000000 --- a/ci/examples/bed.fmf +++ /dev/null @@ -1,20 +0,0 @@ -# Tests which need a fresh test bed for each -/test/build/smoke: - summary: Basic smoke test - # List tests manually - discover: - how: list - tests: - - test/one - - test/two - - test/three - # Setup with ansible - prepare: - how: ansible - playbooks: - - 'dependencies.yml' - - 'selenium.yml' - # Use restraint with isolation enabled - execute: - how: restraint - isolate: true diff --git a/ci/examples/gating.fmf b/ci/examples/gating.fmf deleted file mode 100644 index 05e5d6d..0000000 --- a/ci/examples/gating.fmf +++ /dev/null @@ -1,30 +0,0 @@ -# Turn on gating for selected testsets (defined inline) -/test: - /pull-request: - /pep: - summary: All code must comply with the PEP8 style guide - # Do not allow ugly code to be merged into master - gate: - - merge-pull-request - /lint: - summary: Run pylint to catch common problems (no gating) - /build: - /smoke: - summary: Basic smoke test (Tier1) - # Basic smoke test is used by three gates - gate: - - merge-pull-request - - add-build-to-update - - add-build-to-compose - /features: - summary: Verify important features - /update: - # This enables the 'release-update' gate for all three testsets - gate: - - release-update - /basic: - summary: Run all Tier1, Tier2 and Tier3 tests - /security: - summary: Security tests (extra job to get quick results) - /integration: - summary: Integration tests with related components diff --git a/ci/examples/rpmdiff.fmf b/ci/examples/rpmdiff.fmf deleted file mode 100644 index ccdfcde..0000000 --- a/ci/examples/rpmdiff.fmf +++ /dev/null @@ -1,20 +0,0 @@ -# Basic example of an rpmdiff per-package config -/test: - /build: - /rpmdiff-essential: - summary: Essential rpmdiff tests (used for gating) - execute: - how: rpmdiff - tests: - - ABI symbols - - File permissions - gate: - - add-build-to-update - - add-build-to-compose - /rpmdiff-additional: - summary: Additional rpmdiff tests (informational) - execute: - how: rpmdiff - tests: - - File list - - Specfile checks diff --git a/ci/examples/separate.fmf b/ci/examples/separate.fmf deleted file mode 100644 index ac94ee8..0000000 --- a/ci/examples/separate.fmf +++ /dev/null @@ -1,43 +0,0 @@ -# Turn on gating for selected testsets (separate gate section) -/test: - /pull-request: - /pep: - summary: All code must comply with the PEP8 style guide - /lint: - summary: Run pylint to catch common problems (no gating) - /build: - /smoke: - summary: Basic smoke test (Tier1) - /features: - summary: Verify important features - /update: - /basic: - summary: Run all Tier1, Tier2 and Tier3 tests - /security: - summary: Security tests (extra job to get quick results) - /integration: - summary: Integration tests with related components - -/gate: - # Gate blocking the pull request merge into master - /merge-pull-request: - test: - - /test/pull-request/pep - - /test/build/smoke - - # Gate blocking the build to be added to an update/erratum - /add-build-to-update: - test: - - /test/build/smoke - - # Gate blocking the build to be added to compose - /add-build-to-compose: - test: - - /test/build/smoke - - # Gate blocking release of an update/erratum - /release-update: - test: - - /test/update/basic - - /test/update/security - - /test/update/integration diff --git a/ci/examples/simple.fmf b/ci/examples/simple.fmf deleted file mode 100644 index 2fce84f..0000000 --- a/ci/examples/simple.fmf +++ /dev/null @@ -1,16 +0,0 @@ -# Simple use cases should be super simple to write -/test: - /pull-request: - /validate: - prepare: - requires: python3-pep8 - execute: - how: shell - command: pep8 *.py - /build: - /smoke: - discover: - how: fmf - repo: https://src.fedoraproject.org/tests/tar.git - execute: - how: restraint diff --git a/ci/examples/tooling.fmf b/ci/examples/tooling.fmf deleted file mode 100644 index 29b2a01..0000000 --- a/ci/examples/tooling.fmf +++ /dev/null @@ -1,32 +0,0 @@ -# Multiple tool support for test discovery & execution -/test: - /build: - # Standard Test Inteface - /sti: - execute: - how: sti - playbooks: - - 'tests.yml' - tags: - - 'classic' - # Simple shell script - /shell: - execute: - how: shell - path: 'tests' - test: './test.sh' - environment: - PYTHON: 'python2' - # Workflow Tomorrow - /wow: - execute: - how: wow - options: '--plan 1234 --tags Tier1' - # Flexible Metadata Format + Restraint - /fmf: - discover: - how: fmf - filter: 'tier: 1, 2' - repository: https://src.fedoraproject.org/tests/python - execute: - how: restraint diff --git a/ci/examples/workflow.fmf b/ci/examples/workflow.fmf deleted file mode 100644 index a40115e..0000000 --- a/ci/examples/workflow.fmf +++ /dev/null @@ -1,59 +0,0 @@ -# Workflow steps: discover, provision, prepare, execute, report -/test: - # Discover relevant tests - discover: - how: fmf - # System requirements - provision: - memory: - min: '1 GB' - arch: - - 'x86_64' - # Machine configuration - prepare: - ansible: - - 'setup.yml' - # Test execution - execute: - how: restraint - # Result reporting - report: - contact: email@address.org - - # Build testing - /build: - /smoke: - summary: Basic smoke test (Tier1) - discover+: - filter: 'tier: 1' - /features: - summary: Verify important features - discover+: - filter: 'tag: functional' - - # Errata testing - /update: - provision+: - arch: - - x86_64 - - ppc64 - - s390x - /basic: - summary: Run all Tier1, Tier2 and Tier3 tests - discover+: - filter: 'tier: 1, 2, 3' - /security: - summary: Security tests (extra job to get quick results) - discover+: - filter: 'tag: security' - /integration: - summary: Integration tests with related components - discover+: - filter: 'tag: integration' - provision+: - memory: - min: '4 GB' - max: '8 GB' - prepare: - ansible: - - 'integration.yml' diff --git a/l1/.fmf/version b/l1/.fmf/version new file mode 100644 index 0000000..d00491f --- /dev/null +++ b/l1/.fmf/version @@ -0,0 +1 @@ +1 diff --git a/l1/README.md b/l1/README.md new file mode 100644 index 0000000..2f4ebfe --- /dev/null +++ b/l1/README.md @@ -0,0 +1,57 @@ + +L1 Metadata +=========== + +Fedora Test Metadata Specification + + +Requirements +------------ + +In order to use the [Flexible Metadata Format][fmf] effectively for the CI +testing we need to agree on the essential set of attributes to be used. +For each attribute we need to standardize: + +* name ... clear, unique, well chosen +* type ... expected type: string, number, list, dictionary +* purpose ... description of the attribute purpose + +Each attribute definition should contain at least one apt example of the +usage and user stories illustrating the motivation. + + +Attributes +---------- + +Here is the list of attributes proposed so far. Material for discussion. +Nothing final for now. + +* [summary](/fedora-ci/metadata/blob/master/f/l1/summary.fmf) +* [description](/fedora-ci/metadata/blob/master/f/l1/description.fmf) +* [tags](/fedora-ci/metadata/blob/master/f/l1/tags.fmf) +* [test](/fedora-ci/metadata/blob/master/f/l1/test.fmf) +* [path](/fedora-ci/metadata/blob/master/f/l1/path.fmf) +* [environment](/fedora-ci/metadata/blob/master/f/l1/environment.fmf) +* [duration](/fedora-ci/metadata/blob/master/f/l1/duration.fmf) +* [relevancy](/fedora-ci/metadata/blob/master/f/l1/relevancy.fmf) +* [contact](/fedora-ci/metadata/blob/master/f/l1/contact.fmf) +* [component](/fedora-ci/metadata/blob/master/f/l1/component.fmf) +* [tier](/fedora-ci/metadata/blob/master/f/l1/tier.fmf) +* [provision](/fedora-ci/metadata/blob/master/f/l1/provision.fmf) +* [disabled](/fedora-ci/metadata/blob/master/f/l1/disabled.fmf) +* [result](/fedora-ci/metadata/blob/master/f/l1/result.fmf) + +See [examples](/fedora-ci/metadata/blob/master/f/l1/examples.md) to learn +how "path" and "test" attributes can be used together. + + +Status +------ + +Each attribute has its current status defined in the following way: + +* draft ... attribute definition has been proposed +* approved ... proposal has been discussed and approved +* production ... attribute is supported in the tooling + +[fmf]: https://fedoraproject.org/wiki/Flexible_Metadata_Format diff --git a/l1/component.fmf b/l1/component.fmf new file mode 100644 index 0000000..85b5763 --- /dev/null +++ b/l1/component.fmf @@ -0,0 +1,24 @@ +name: component +type: list of strings +purpose: relevant fedora/rhel source package names +status: production + +motivation: + It's useful to be able to easily select all tests relevant for given + component or package. As they do not always have to be stored in the + same repository and because many tests cover multiple components a + dedicated field is needed. + +stories: + - As a SELinux tester testing the "checkpolicy" component I want to + run Tier1 tests for all SELinux components plus all checkpolicy + tests. + +examples: | + component: [libselinux, checkpolicy] + +notes: > + The following fmf command can be used to select test set described + by the user story above: + + fmf --key test --filter 'tags: Tier1 | component: checkpolicy' diff --git a/l1/contact.fmf b/l1/contact.fmf new file mode 100644 index 0000000..8a4c640 --- /dev/null +++ b/l1/contact.fmf @@ -0,0 +1,19 @@ +name: contact +type: string or list of strings (name with email address) +purpose: person(s) maintaining the test +status: production + +motivation: + When there are several people collaborating on tests it's useful to + have a way how find who is responsible for what. + +stories: + - As a developer reviewing a complex failed test I would like to + contact the person who maintains the code and understands it well. + +examples: | + contact: Name Surname + + contact: + - First Person + - Second Person diff --git a/l1/description.fmf b/l1/description.fmf new file mode 100644 index 0000000..cacecac --- /dev/null +++ b/l1/description.fmf @@ -0,0 +1,24 @@ +name: description +type: string (multi line, plain text) +purpose: detailed description of what the test does +status: production + +motivation: + For complex tests it makes sense to provide more detailed + description to better clarify what is covered by the test. + +stories: + - As a tester I come to a test code I wrote 10 years ago (so I have + absolutely no idea about it) and would like to quickly understand + what it does. + - As a developer I review existing test coverage for my component + and would like to get an overall idea what is covered without + having to read the whole test code. + +examples: | + description: | + This test checks all available wget options related to + downloading files recursively. First a tree directory + structure is created for testing. Then a file download + is performed for different recursion depth specified by + the "--level=depth" option. diff --git a/l1/disabled.fmf b/l1/disabled.fmf new file mode 100644 index 0000000..d7a2095 --- /dev/null +++ b/l1/disabled.fmf @@ -0,0 +1,19 @@ +name: disabled +type: boolean +purpose: allow disabling individual tests +status: production + +motivation: + When a test is broken and it is not possible to fix it immediately + there should be an easy way how to temporarily mark it as disabled + so that it's skipped during the test suite execution. + +stories: + - As a developer/tester I want selected tests to be skipped during + test execution by marking them as disabled. + +examples: | + disabled: yes + +notes: + The default value is "disabled: no". diff --git a/l1/duration.fmf b/l1/duration.fmf new file mode 100644 index 0000000..a3ae536 --- /dev/null +++ b/l1/duration.fmf @@ -0,0 +1,21 @@ +name: duration +type: string +purpose: maximum time for test execution +status: approved + +motivation: + In order to prevent stuck tests consuming resources we define a + maximum time for test execution. If the limit is exceeded the + running test is killed by the test harness. + +stories: + - As a developer/tester I want to prevent resource wasting by stuck + tests. + - As a test harness I need to know after how long time I should kill + test if it is still running. + +notes: + Use the same format as the "sleep" command. For example: 3m, 2h, 1d. + +examples: | + duration: 5m diff --git a/l1/environment.fmf b/l1/environment.fmf new file mode 100644 index 0000000..9f6c047 --- /dev/null +++ b/l1/environment.fmf @@ -0,0 +1,22 @@ +name: environment +type: dictionary +purpose: environment variables to be set before running the test +status: approved + +motivation: + Test scripts might require certain environment variables to be set. + Although this can be done on the shell command line as part of the + "test" attribute it makes sense to have a dedicated field for this, + especially when the number of parameters grows. This might be useful + for virtual test cases as well. + +stories: + - As a tester I need to pass environment variables to my test script + to properly execute the desired test scenario. + - As a tester I'm using a single test script for testing different + Python implementations specified by environment variable PYTHON. + +examples: | + environment: + PACKAGE: python37 + PYTHON: python3.7 diff --git a/l1/examples.md b/l1/examples.md new file mode 100644 index 0000000..71b367c --- /dev/null +++ b/l1/examples.md @@ -0,0 +1,95 @@ +Examples +======== + +Below you can find some basic examples using the metadata which +have been already defined. + +BeakerLib Tests +--------------- + +Three beakerlib tests, each in it's own directory: + +main.fmf + + test: ./runtest.sh + + /one: + path: /tests/one + /two: + path: /tests/two + /three: + path: /tests/three + +fmf + + /one + path: tests/one + test: ./runtest.sh + + /two + path: tests/two + test: ./runtest.sh + + /three + path: tests/three + test: ./runtest.sh + +Three Scripts +------------- + +Three different script residing in a single directory: + +main.fmf + + path: /tests + + /one: + test: ./one + /two: + test: ./two + /three: + test: ./three + +fmf + + /one + path: /tests + test: ./one + + /two + path: /tests + test: ./two + + /three + path: /tests + test: ./three + +Virtual Tests +------------- + +Thre virtual test cases based on a single test script: + +main.fmf + + path: /tests/virtual + + /one: + test: ./script --one + /two: + test: ./script --two + /three: + test: ./script --three + +fmf + + /one + path: /tests/virtual + test: ./script --one + + /two + path: /tests/virtual + test: ./script --two + + /three + path: /tests/virtual + test: ./script --three diff --git a/l1/path.fmf b/l1/path.fmf new file mode 100644 index 0000000..dee533b --- /dev/null +++ b/l1/path.fmf @@ -0,0 +1,25 @@ +name: path +type: string +purpose: filesystem directory to be entered before executing the test +status: approved + +motivation: + As the object hierarchy does not need to copy the filesystem + structure (e.g. when using virtual test cases) we need a way how to + define where the test is located. + +stories: + - As a test writer I define two virtual test cases, both using the + same script for executing. + +notes: + Automation is expected to change directory to provided path starting + from the fmf tree root directory before executing the test. Use + absolute path starting with slash. If path is not defined, object + name (identifier) is used by default. + +examples: | + path: /protocols/https + +links: + - http://fmf.readthedocs.io/en/latest/features.html#virtual diff --git a/l1/provision.fmf b/l1/provision.fmf new file mode 100644 index 0000000..7e06bd2 --- /dev/null +++ b/l1/provision.fmf @@ -0,0 +1,31 @@ +name: provision +type: dictionary +purpose: set of environment requirements +status: production + +motivation: + In some cases tests have special requirements for the environment in + order to run successfully. For now just simple qemu options for the + standard-inventory-qcow2 provisioner are supported. + +stories: + - As a tester I want to specify amount of the memory which needs to + be available for the test. + - As a tester I want to specify network interface card to be used in + qemu. + +examples: | + provision: + standard-inventory-qcow2: + qemu: + m: 3G + net_nic: + model: e1000 + +notes: + Memory size is specified in megabytes. Optionally, a suffix of “M” + or “G”. Use qemu-system-x86_64 -net nic,model=help for a list of + available devices. See links for real life example usage. + +links: + - https://pagure.io/pschindl-osci/blob/master/f/provision.fmf diff --git a/l1/relevancy.fmf b/l1/relevancy.fmf new file mode 100644 index 0000000..ee161f3 --- /dev/null +++ b/l1/relevancy.fmf @@ -0,0 +1,25 @@ +name: relevancy +type: list +purpose: Test Case Relevancy rules used for filtering relevant test cases +status: approved + +motivation: + Sometimes a test case is only relevant for specific environment. + Test Case Relevancy allows to filter irrelevant test cases out. + +stories: + - As a tester I want to skip execution of a particular test case in + given test environment. + +examples: | + relevancy: + - "distro < f-28: False" + - "distro = rhel-7 & arch = ppc64: False" + +notes: > + Environment is defined by one or more environment dimensions such as + product, distro, collection, variant, arch, component. Relevancy + consists of a set of rules of the form "condition: decision". + +links: + - https://fedoraproject.org/wiki/CI/Test_Case_Relevancy diff --git a/l1/result.fmf b/l1/result.fmf new file mode 100644 index 0000000..1fb4b02 --- /dev/null +++ b/l1/result.fmf @@ -0,0 +1,29 @@ +name: result +type: string +purpose: specify how test result should be interpreted +status: approved + +motivation: + Even if a test fails it might makes sense to execute it to be able + to manually review the results (ignore test result) or ensure the + behaviour has not unexpectedly changed and the test is still failing + (expected fail). + +stories: + - As a developer I have a test, I know it's failing for now, yet I + want to have it for future reference. + - As a tester I want to regularly execute the test but temporarily + ignore test result until more investigation is done and the test + can be fixed properly. + +notes: + The following values should be supported: + - respect: test result is respected (fails when test failed) + - ignore: ignore the test result (test always passes) + - xfail: expected fail (pass when test fails, fail when test passes) + +examples: | + result: ignore + +notes: + The default value is "result: respect". diff --git a/l1/summary.fmf b/l1/summary.fmf new file mode 100644 index 0000000..78b081a --- /dev/null +++ b/l1/summary.fmf @@ -0,0 +1,18 @@ +name: summary +type: string (one line, up to 50 characters) +purpose: concise summary of what the test does +status: production + +motivation: + In order to efficiently collaborate on test maintenance it's + crucial to have a short summary of what the test does. + +stories: + - As a developer reviewing multiple failed tests I would like + to get quickly an idea of what my change broke. + +examples: | + summary: Test wget recursive download options + +links: + - https://stackoverflow.com/questions/2290016/git-commit-messages-50-72-formatting diff --git a/l1/tags.fmf b/l1/tags.fmf new file mode 100644 index 0000000..398f0f7 --- /dev/null +++ b/l1/tags.fmf @@ -0,0 +1,20 @@ +name: tags +type: list +purpose: free-form tags for easy filtering +status: production + +motivation: + Throughout the years, free-form tags proved to be useful for many, + many scenarios. Primarily to provide an easy way how to select a + subset of objects. + +stories: + - As a developer/tester I would like to run only a subset of + available tests. + +examples: | + tags: [Tier1, fast] + +notes: + Tags are case-sensitive. + Using lowercase is recommended. diff --git a/l1/test.fmf b/l1/test.fmf new file mode 100644 index 0000000..a1dc049 --- /dev/null +++ b/l1/test.fmf @@ -0,0 +1,16 @@ +name: test +type: string +purpose: shell command which executes the test +status: approved + +motivation: + This attribute defines how the test is to be executed. + +stories: + - As a developer/tester I want to easily execute all available tests + with just one command. + - As a test writer I want to run a single test script in multiple + ways (e.g. providing different parameters) + +examples: | + test: ./runtest.sh diff --git a/l1/tier.fmf b/l1/tier.fmf new file mode 100644 index 0000000..e1b809c --- /dev/null +++ b/l1/tier.fmf @@ -0,0 +1,19 @@ +name: tier +type: string +purpose: name of the tier set this test belongs to +status: production + +motivation: + It's quite common to organize tests into "tiers" based on their + importance, stability, duration and other aspects. For this tags + have been used quite often as there was not corresponding attribute + available. It might make sense to have a dedicated field for this + functionality as well. + +stories: + - As a tester testing a security advisory I want to run the stable + set of important tests which cover the most essential + functionality and can provide test results in a short time. + +examples: | + tier: 1 diff --git a/l2/README.md b/l2/README.md new file mode 100644 index 0000000..18774b8 --- /dev/null +++ b/l2/README.md @@ -0,0 +1,147 @@ +# L2 Metadata + +Fedora CI Metadata Specification + + +## Motivation + +Specification of CI pipeline, trying to be CI agnostic and user friendly. +Here are the main points which motivated this effort: + +* Additional metadata for test execution needed, stored on various places +* The tests.yml file defined by Standard Test Interface is limited +* Human readable configuration for gating (replace gating.yaml) +* Multiple gates in the release pipeline (pull request, build, update/errata) +* Clearly separate testing steps (discover, provision, prepare, execute, report) +* Support multiple tools implementing individual testing steps + + +## Artifacts + +Specifies the artifact to be tested. Currently defined artifacts: + +* `pull-request` ... code check (static analysis, conventions...) +* `build` ... functional testing of rpm build from koji or brew +* `update` ... one or more builds grouped in an update/erratum + +Based on the artifact it is possible to detect some environment +constraints such as architecture or base compose. + + /test: + /pull-request: + /pep: + summary: All code must comply with the PEP8 style guide + /lint: + summary: Run pylint to catch common problems (no gating) + /build: + /smoke: + summary: Basic smoke test (Tier1) + /features: + summary: Verify important features + + +## Hierarchy + +There are several levels of test execution data configuration: + +* default ... global default settings (common for most instances) +* detect ... detect from previous steps output (e.g. distro from build) +* define ... allow to override value by explicit user configuration + + +## Steps + +There are five separate steps defined for the test execution. +Each step can be supported by multiple implementations. +Special keyword `how` defines which implementation should be used. + +### Discover + +Gather information about the test cases which are supposed to be run. +From the test case metadata constraints for test enviroment can be detected: + +* Architectures supported +* Disk and memory constraints +* Product relevancy +* Environment variables + +Examples of metadata storage: + +* `list` ... Manual list of test caess +* `fmf` ... Flexible Metadata Format +* `tcms` ... Test Case Management System + +Example config: + + discover: + how: 'fmf' + filter: 'tier: 1' + +### Provision + +Provision test environemnt according to constraints from artifact and discover. + +* Grouping based on test case relevancy +* Output: list of environments with list of tests which to run on them +* Might fail that is cannot provision accoring to the constraints + +Examples of provisioning implementation: + +* `localhost` ... run directly on the local machine +* `openstack` ... create vm in using OpenStack +* `beaker` ... reserve machine using Beaker + +Example config: + + provision: + memory: + min: '1 GB' + arch: + - 'x86_64' + +### Prepare + +Additional configuration of the provisioned environment needed for testing. + +* Install artifact (customizable according to user needs) + * Conflicts between rpms + * Optionally add debuginfo + * Install with devel module +* Additional setup possible if needed + * Inject arbitrary commands + * Before/after TODO: What does that mean? + +Examples of preparation implementation: + +* `shell` ... execute arbitratry shell commands to set up the system +* `ansible` ... apply ansible playbook to get the desired final state + +Example config: + + prepare: + how: ansible + playbook: setup.yml + +### Execute + +Specification of the testing framework which should execute tests. + +* According to the provision output does the execution +* Paralelization? + +Example config: + + execute: + how: restraint + +### Report + +Adjusting notifications about the test progress and results. + +* `email` ... send email notification +* `irc` ... notify on irc chat + +Example config: + + report: + contact: email@address.org diff --git a/l2/artifacts.fmf b/l2/artifacts.fmf new file mode 100644 index 0000000..bc153ec --- /dev/null +++ b/l2/artifacts.fmf @@ -0,0 +1,19 @@ +# Basic structure of artifacts and related testsets +/test: + /pull-request: + /pep: + summary: All code must comply with the PEP8 style guide + /lint: + summary: Run pylint to catch common problems (no gating) + /build: + /smoke: + summary: Basic smoke test (Tier1) + /features: + summary: Verify important features + /update: + /basic: + summary: Run all Tier1, Tier2 and Tier3 tests + /security: + summary: Security tests (extra job to get quick results) + /integration: + summary: Integration tests with related components diff --git a/l2/bed.fmf b/l2/bed.fmf new file mode 100644 index 0000000..e63d80b --- /dev/null +++ b/l2/bed.fmf @@ -0,0 +1,20 @@ +# Tests which need a fresh test bed for each +/test/build/smoke: + summary: Basic smoke test + # List tests manually + discover: + how: list + tests: + - test/one + - test/two + - test/three + # Setup with ansible + prepare: + how: ansible + playbooks: + - 'dependencies.yml' + - 'selenium.yml' + # Use restraint with isolation enabled + execute: + how: restraint + isolate: true diff --git a/l2/gating.fmf b/l2/gating.fmf new file mode 100644 index 0000000..05e5d6d --- /dev/null +++ b/l2/gating.fmf @@ -0,0 +1,30 @@ +# Turn on gating for selected testsets (defined inline) +/test: + /pull-request: + /pep: + summary: All code must comply with the PEP8 style guide + # Do not allow ugly code to be merged into master + gate: + - merge-pull-request + /lint: + summary: Run pylint to catch common problems (no gating) + /build: + /smoke: + summary: Basic smoke test (Tier1) + # Basic smoke test is used by three gates + gate: + - merge-pull-request + - add-build-to-update + - add-build-to-compose + /features: + summary: Verify important features + /update: + # This enables the 'release-update' gate for all three testsets + gate: + - release-update + /basic: + summary: Run all Tier1, Tier2 and Tier3 tests + /security: + summary: Security tests (extra job to get quick results) + /integration: + summary: Integration tests with related components diff --git a/l2/rpmdiff.fmf b/l2/rpmdiff.fmf new file mode 100644 index 0000000..ccdfcde --- /dev/null +++ b/l2/rpmdiff.fmf @@ -0,0 +1,20 @@ +# Basic example of an rpmdiff per-package config +/test: + /build: + /rpmdiff-essential: + summary: Essential rpmdiff tests (used for gating) + execute: + how: rpmdiff + tests: + - ABI symbols + - File permissions + gate: + - add-build-to-update + - add-build-to-compose + /rpmdiff-additional: + summary: Additional rpmdiff tests (informational) + execute: + how: rpmdiff + tests: + - File list + - Specfile checks diff --git a/l2/separate.fmf b/l2/separate.fmf new file mode 100644 index 0000000..ac94ee8 --- /dev/null +++ b/l2/separate.fmf @@ -0,0 +1,43 @@ +# Turn on gating for selected testsets (separate gate section) +/test: + /pull-request: + /pep: + summary: All code must comply with the PEP8 style guide + /lint: + summary: Run pylint to catch common problems (no gating) + /build: + /smoke: + summary: Basic smoke test (Tier1) + /features: + summary: Verify important features + /update: + /basic: + summary: Run all Tier1, Tier2 and Tier3 tests + /security: + summary: Security tests (extra job to get quick results) + /integration: + summary: Integration tests with related components + +/gate: + # Gate blocking the pull request merge into master + /merge-pull-request: + test: + - /test/pull-request/pep + - /test/build/smoke + + # Gate blocking the build to be added to an update/erratum + /add-build-to-update: + test: + - /test/build/smoke + + # Gate blocking the build to be added to compose + /add-build-to-compose: + test: + - /test/build/smoke + + # Gate blocking release of an update/erratum + /release-update: + test: + - /test/update/basic + - /test/update/security + - /test/update/integration diff --git a/l2/simple.fmf b/l2/simple.fmf new file mode 100644 index 0000000..2fce84f --- /dev/null +++ b/l2/simple.fmf @@ -0,0 +1,16 @@ +# Simple use cases should be super simple to write +/test: + /pull-request: + /validate: + prepare: + requires: python3-pep8 + execute: + how: shell + command: pep8 *.py + /build: + /smoke: + discover: + how: fmf + repo: https://src.fedoraproject.org/tests/tar.git + execute: + how: restraint diff --git a/l2/tooling.fmf b/l2/tooling.fmf new file mode 100644 index 0000000..29b2a01 --- /dev/null +++ b/l2/tooling.fmf @@ -0,0 +1,32 @@ +# Multiple tool support for test discovery & execution +/test: + /build: + # Standard Test Inteface + /sti: + execute: + how: sti + playbooks: + - 'tests.yml' + tags: + - 'classic' + # Simple shell script + /shell: + execute: + how: shell + path: 'tests' + test: './test.sh' + environment: + PYTHON: 'python2' + # Workflow Tomorrow + /wow: + execute: + how: wow + options: '--plan 1234 --tags Tier1' + # Flexible Metadata Format + Restraint + /fmf: + discover: + how: fmf + filter: 'tier: 1, 2' + repository: https://src.fedoraproject.org/tests/python + execute: + how: restraint diff --git a/l2/workflow.fmf b/l2/workflow.fmf new file mode 100644 index 0000000..a40115e --- /dev/null +++ b/l2/workflow.fmf @@ -0,0 +1,59 @@ +# Workflow steps: discover, provision, prepare, execute, report +/test: + # Discover relevant tests + discover: + how: fmf + # System requirements + provision: + memory: + min: '1 GB' + arch: + - 'x86_64' + # Machine configuration + prepare: + ansible: + - 'setup.yml' + # Test execution + execute: + how: restraint + # Result reporting + report: + contact: email@address.org + + # Build testing + /build: + /smoke: + summary: Basic smoke test (Tier1) + discover+: + filter: 'tier: 1' + /features: + summary: Verify important features + discover+: + filter: 'tag: functional' + + # Errata testing + /update: + provision+: + arch: + - x86_64 + - ppc64 + - s390x + /basic: + summary: Run all Tier1, Tier2 and Tier3 tests + discover+: + filter: 'tier: 1, 2, 3' + /security: + summary: Security tests (extra job to get quick results) + discover+: + filter: 'tag: security' + /integration: + summary: Integration tests with related components + discover+: + filter: 'tag: integration' + provision+: + memory: + min: '4 GB' + max: '8 GB' + prepare: + ansible: + - 'integration.yml' diff --git a/test/README.md b/test/README.md deleted file mode 100644 index ccfcf30..0000000 --- a/test/README.md +++ /dev/null @@ -1,63 +0,0 @@ - -Metadata -======== - -Fedora CI Metadata Specification. - - -Requirements ------------- - -In order to use the [Flexible Metadata Format][fmf] effectively for the CI -testing we need to agree on the essential set of attributes to be used. -For each attribute we need to standardize: - -* name ... clear, unique, well chosen -* type ... expected type: string, number, list, dictionary -* purpose ... description of the attribute purpose - -Each attribute definition should contain at least one apt example of the -usage and user stories illustrating the motivation. - - -Attributes ----------- - -Here is the list of attributes proposed so far. Material for discussion. -Nothing final for now. - -* [summary](/fedora-ci/metadata/blob/master/f/summary.fmf) -* [description](/fedora-ci/metadata/blob/master/f/description.fmf) -* [tags](/fedora-ci/metadata/blob/master/f/tags.fmf) -* [test](/fedora-ci/metadata/blob/master/f/test.fmf) -* [path](/fedora-ci/metadata/blob/master/f/path.fmf) -* [environment](/fedora-ci/metadata/blob/master/f/environment.fmf) -* [duration](/fedora-ci/metadata/blob/master/f/duration.fmf) -* [relevancy](/fedora-ci/metadata/blob/master/f/relevancy.fmf) -* [contact](/fedora-ci/metadata/blob/master/f/contact.fmf) -* [component](/fedora-ci/metadata/blob/master/f/component.fmf) -* [tier](/fedora-ci/metadata/blob/master/f/tier.fmf) -* [provision](/fedora-ci/metadata/blob/master/f/provision.fmf) -* [disabled](/fedora-ci/metadata/blob/master/f/disabled.fmf) -* [result](/fedora-ci/metadata/blob/master/f/result.fmf) - -See [examples](/fedora-ci/metadata/blob/master/f/examples.md) to learn -how "path" and "test" attributes can be used together. - - -Status ------- - -Each attribute has its current status defined in the following way: - -* draft ... attribute definition has been proposed -* approved ... proposal has been discussed and approved -* production ... attribute is supported in the tooling - - -Links ------ - -* [Flexible Metadata Format][fmf] - -[fmf]: https://fedoraproject.org/wiki/Flexible_Metadata_Format diff --git a/test/component.fmf b/test/component.fmf deleted file mode 100644 index 85b5763..0000000 --- a/test/component.fmf +++ /dev/null @@ -1,24 +0,0 @@ -name: component -type: list of strings -purpose: relevant fedora/rhel source package names -status: production - -motivation: - It's useful to be able to easily select all tests relevant for given - component or package. As they do not always have to be stored in the - same repository and because many tests cover multiple components a - dedicated field is needed. - -stories: - - As a SELinux tester testing the "checkpolicy" component I want to - run Tier1 tests for all SELinux components plus all checkpolicy - tests. - -examples: | - component: [libselinux, checkpolicy] - -notes: > - The following fmf command can be used to select test set described - by the user story above: - - fmf --key test --filter 'tags: Tier1 | component: checkpolicy' diff --git a/test/contact.fmf b/test/contact.fmf deleted file mode 100644 index 8a4c640..0000000 --- a/test/contact.fmf +++ /dev/null @@ -1,19 +0,0 @@ -name: contact -type: string or list of strings (name with email address) -purpose: person(s) maintaining the test -status: production - -motivation: - When there are several people collaborating on tests it's useful to - have a way how find who is responsible for what. - -stories: - - As a developer reviewing a complex failed test I would like to - contact the person who maintains the code and understands it well. - -examples: | - contact: Name Surname - - contact: - - First Person - - Second Person diff --git a/test/description.fmf b/test/description.fmf deleted file mode 100644 index cacecac..0000000 --- a/test/description.fmf +++ /dev/null @@ -1,24 +0,0 @@ -name: description -type: string (multi line, plain text) -purpose: detailed description of what the test does -status: production - -motivation: - For complex tests it makes sense to provide more detailed - description to better clarify what is covered by the test. - -stories: - - As a tester I come to a test code I wrote 10 years ago (so I have - absolutely no idea about it) and would like to quickly understand - what it does. - - As a developer I review existing test coverage for my component - and would like to get an overall idea what is covered without - having to read the whole test code. - -examples: | - description: | - This test checks all available wget options related to - downloading files recursively. First a tree directory - structure is created for testing. Then a file download - is performed for different recursion depth specified by - the "--level=depth" option. diff --git a/test/disabled.fmf b/test/disabled.fmf deleted file mode 100644 index d7a2095..0000000 --- a/test/disabled.fmf +++ /dev/null @@ -1,19 +0,0 @@ -name: disabled -type: boolean -purpose: allow disabling individual tests -status: production - -motivation: - When a test is broken and it is not possible to fix it immediately - there should be an easy way how to temporarily mark it as disabled - so that it's skipped during the test suite execution. - -stories: - - As a developer/tester I want selected tests to be skipped during - test execution by marking them as disabled. - -examples: | - disabled: yes - -notes: - The default value is "disabled: no". diff --git a/test/duration.fmf b/test/duration.fmf deleted file mode 100644 index a3ae536..0000000 --- a/test/duration.fmf +++ /dev/null @@ -1,21 +0,0 @@ -name: duration -type: string -purpose: maximum time for test execution -status: approved - -motivation: - In order to prevent stuck tests consuming resources we define a - maximum time for test execution. If the limit is exceeded the - running test is killed by the test harness. - -stories: - - As a developer/tester I want to prevent resource wasting by stuck - tests. - - As a test harness I need to know after how long time I should kill - test if it is still running. - -notes: - Use the same format as the "sleep" command. For example: 3m, 2h, 1d. - -examples: | - duration: 5m diff --git a/test/environment.fmf b/test/environment.fmf deleted file mode 100644 index 9f6c047..0000000 --- a/test/environment.fmf +++ /dev/null @@ -1,22 +0,0 @@ -name: environment -type: dictionary -purpose: environment variables to be set before running the test -status: approved - -motivation: - Test scripts might require certain environment variables to be set. - Although this can be done on the shell command line as part of the - "test" attribute it makes sense to have a dedicated field for this, - especially when the number of parameters grows. This might be useful - for virtual test cases as well. - -stories: - - As a tester I need to pass environment variables to my test script - to properly execute the desired test scenario. - - As a tester I'm using a single test script for testing different - Python implementations specified by environment variable PYTHON. - -examples: | - environment: - PACKAGE: python37 - PYTHON: python3.7 diff --git a/test/examples.md b/test/examples.md deleted file mode 100644 index 71b367c..0000000 --- a/test/examples.md +++ /dev/null @@ -1,95 +0,0 @@ -Examples -======== - -Below you can find some basic examples using the metadata which -have been already defined. - -BeakerLib Tests ---------------- - -Three beakerlib tests, each in it's own directory: - -main.fmf - - test: ./runtest.sh - - /one: - path: /tests/one - /two: - path: /tests/two - /three: - path: /tests/three - -fmf - - /one - path: tests/one - test: ./runtest.sh - - /two - path: tests/two - test: ./runtest.sh - - /three - path: tests/three - test: ./runtest.sh - -Three Scripts -------------- - -Three different script residing in a single directory: - -main.fmf - - path: /tests - - /one: - test: ./one - /two: - test: ./two - /three: - test: ./three - -fmf - - /one - path: /tests - test: ./one - - /two - path: /tests - test: ./two - - /three - path: /tests - test: ./three - -Virtual Tests -------------- - -Thre virtual test cases based on a single test script: - -main.fmf - - path: /tests/virtual - - /one: - test: ./script --one - /two: - test: ./script --two - /three: - test: ./script --three - -fmf - - /one - path: /tests/virtual - test: ./script --one - - /two - path: /tests/virtual - test: ./script --two - - /three - path: /tests/virtual - test: ./script --three diff --git a/test/path.fmf b/test/path.fmf deleted file mode 100644 index dee533b..0000000 --- a/test/path.fmf +++ /dev/null @@ -1,25 +0,0 @@ -name: path -type: string -purpose: filesystem directory to be entered before executing the test -status: approved - -motivation: - As the object hierarchy does not need to copy the filesystem - structure (e.g. when using virtual test cases) we need a way how to - define where the test is located. - -stories: - - As a test writer I define two virtual test cases, both using the - same script for executing. - -notes: - Automation is expected to change directory to provided path starting - from the fmf tree root directory before executing the test. Use - absolute path starting with slash. If path is not defined, object - name (identifier) is used by default. - -examples: | - path: /protocols/https - -links: - - http://fmf.readthedocs.io/en/latest/features.html#virtual diff --git a/test/provision.fmf b/test/provision.fmf deleted file mode 100644 index 7e06bd2..0000000 --- a/test/provision.fmf +++ /dev/null @@ -1,31 +0,0 @@ -name: provision -type: dictionary -purpose: set of environment requirements -status: production - -motivation: - In some cases tests have special requirements for the environment in - order to run successfully. For now just simple qemu options for the - standard-inventory-qcow2 provisioner are supported. - -stories: - - As a tester I want to specify amount of the memory which needs to - be available for the test. - - As a tester I want to specify network interface card to be used in - qemu. - -examples: | - provision: - standard-inventory-qcow2: - qemu: - m: 3G - net_nic: - model: e1000 - -notes: - Memory size is specified in megabytes. Optionally, a suffix of “M” - or “G”. Use qemu-system-x86_64 -net nic,model=help for a list of - available devices. See links for real life example usage. - -links: - - https://pagure.io/pschindl-osci/blob/master/f/provision.fmf diff --git a/test/relevancy.fmf b/test/relevancy.fmf deleted file mode 100644 index ee161f3..0000000 --- a/test/relevancy.fmf +++ /dev/null @@ -1,25 +0,0 @@ -name: relevancy -type: list -purpose: Test Case Relevancy rules used for filtering relevant test cases -status: approved - -motivation: - Sometimes a test case is only relevant for specific environment. - Test Case Relevancy allows to filter irrelevant test cases out. - -stories: - - As a tester I want to skip execution of a particular test case in - given test environment. - -examples: | - relevancy: - - "distro < f-28: False" - - "distro = rhel-7 & arch = ppc64: False" - -notes: > - Environment is defined by one or more environment dimensions such as - product, distro, collection, variant, arch, component. Relevancy - consists of a set of rules of the form "condition: decision". - -links: - - https://fedoraproject.org/wiki/CI/Test_Case_Relevancy diff --git a/test/result.fmf b/test/result.fmf deleted file mode 100644 index 1fb4b02..0000000 --- a/test/result.fmf +++ /dev/null @@ -1,29 +0,0 @@ -name: result -type: string -purpose: specify how test result should be interpreted -status: approved - -motivation: - Even if a test fails it might makes sense to execute it to be able - to manually review the results (ignore test result) or ensure the - behaviour has not unexpectedly changed and the test is still failing - (expected fail). - -stories: - - As a developer I have a test, I know it's failing for now, yet I - want to have it for future reference. - - As a tester I want to regularly execute the test but temporarily - ignore test result until more investigation is done and the test - can be fixed properly. - -notes: - The following values should be supported: - - respect: test result is respected (fails when test failed) - - ignore: ignore the test result (test always passes) - - xfail: expected fail (pass when test fails, fail when test passes) - -examples: | - result: ignore - -notes: - The default value is "result: respect". diff --git a/test/summary.fmf b/test/summary.fmf deleted file mode 100644 index 78b081a..0000000 --- a/test/summary.fmf +++ /dev/null @@ -1,18 +0,0 @@ -name: summary -type: string (one line, up to 50 characters) -purpose: concise summary of what the test does -status: production - -motivation: - In order to efficiently collaborate on test maintenance it's - crucial to have a short summary of what the test does. - -stories: - - As a developer reviewing multiple failed tests I would like - to get quickly an idea of what my change broke. - -examples: | - summary: Test wget recursive download options - -links: - - https://stackoverflow.com/questions/2290016/git-commit-messages-50-72-formatting diff --git a/test/tags.fmf b/test/tags.fmf deleted file mode 100644 index 398f0f7..0000000 --- a/test/tags.fmf +++ /dev/null @@ -1,20 +0,0 @@ -name: tags -type: list -purpose: free-form tags for easy filtering -status: production - -motivation: - Throughout the years, free-form tags proved to be useful for many, - many scenarios. Primarily to provide an easy way how to select a - subset of objects. - -stories: - - As a developer/tester I would like to run only a subset of - available tests. - -examples: | - tags: [Tier1, fast] - -notes: - Tags are case-sensitive. - Using lowercase is recommended. diff --git a/test/test.fmf b/test/test.fmf deleted file mode 100644 index a1dc049..0000000 --- a/test/test.fmf +++ /dev/null @@ -1,16 +0,0 @@ -name: test -type: string -purpose: shell command which executes the test -status: approved - -motivation: - This attribute defines how the test is to be executed. - -stories: - - As a developer/tester I want to easily execute all available tests - with just one command. - - As a test writer I want to run a single test script in multiple - ways (e.g. providing different parameters) - -examples: | - test: ./runtest.sh diff --git a/test/tier.fmf b/test/tier.fmf deleted file mode 100644 index e1b809c..0000000 --- a/test/tier.fmf +++ /dev/null @@ -1,19 +0,0 @@ -name: tier -type: string -purpose: name of the tier set this test belongs to -status: production - -motivation: - It's quite common to organize tests into "tiers" based on their - importance, stability, duration and other aspects. For this tags - have been used quite often as there was not corresponding attribute - available. It might make sense to have a dedicated field for this - functionality as well. - -stories: - - As a tester testing a security advisory I want to run the stable - set of important tests which cover the most essential - functionality and can provide test results in a short time. - -examples: | - tier: 1 From 6f93363c6cf81623377b3d5362f808e57e98f9bc Mon Sep 17 00:00:00 2001 From: Petr Šplíchal Date: Apr 25 2019 16:00:29 +0000 Subject: [PATCH 3/3] Address issues raised in comments * Added a short "FMF is just an improved YAML" sentence * Steps extended with discover-only and execute-only example * Included DDD example with memory and compose * Extended steps section description * Added a new step `finish` for cleanup actions * Plus several other minor changes and typo fixes --- diff --git a/README.md b/README.md index ad49789..9491fdc 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,10 @@ This is Fedora CI Metadata Specification which defines that all data needed for test execution in the CI system are stored as a plain text information in [Flexible Metadata Format][fmf] under version control in the git repository close to the test code or -source code. There are two levels of metadata defined: +source code. FMF uses YAML to store data in a concise human and +machine readable way plus adds a few nice features like virtual +hierarchy, inheritance and elasticity to minimize data duplication +and maintenance. There are two levels of metadata defined: L1 Metadata diff --git a/l2/README.md b/l2/README.md index 18774b8..667baab 100644 --- a/l2/README.md +++ b/l2/README.md @@ -48,16 +48,27 @@ There are several levels of test execution data configuration: * detect ... detect from previous steps output (e.g. distro from build) * define ... allow to override value by explicit user configuration +Let's demonstrate these on a simple example with a distribution compose (C) and the amount of memory (M): + +* CI system is configured by `default` to use compose `C1` installed on machines with `M1` GB of memory. +* When inspecting an artifact CI can `detect` that for this particular build target, compose `C2` is a much better choice, and memory is fine as it is. +* User can still explicitly `define` in the configuration that at least `M2` GB of memory is needed for successful test execution which overrides both default and detected values. + ## Steps -There are five separate steps defined for the test execution. +There are several separate steps defined for the test execution. +Clearly separating testing stages gives users control over their individual aspects. +Each step makes it clear what and how can be influenced for that particular stage of the process. +This approach also allows to run only selected steps when desired. +For example run `discover` only to see which tests will be executed or skip `provision` and `prepare` when quickly testing on localhost. Each step can be supported by multiple implementations. Special keyword `how` defines which implementation should be used. ### Discover Gather information about the test cases which are supposed to be run. +This includes list of test cases with corresponding L1 metadata. From the test case metadata constraints for test enviroment can be detected: * Architectures supported @@ -67,8 +78,8 @@ From the test case metadata constraints for test enviroment can be detected: Examples of metadata storage: -* `list` ... Manual list of test caess -* `fmf` ... Flexible Metadata Format +* `list` ... Manual list of test cases +* `fmf` ... Flexible Metadata Format filter * `tcms` ... Test Case Management System Example config: @@ -79,17 +90,20 @@ Example config: ### Provision -Provision test environemnt according to constraints from artifact and discover. +Describes what environment is needed for testing and how it should provisioned. +Provides a generic and extensible way to write down essential hardware requirements. +For example one consistent way how to specify "at least 2 GB of RAM" for all supported provisioners. -* Grouping based on test case relevancy -* Output: list of environments with list of tests which to run on them -* Might fail that is cannot provision accoring to the constraints +* Possible grouping of test cases based on test case relevancy +* Might fail if cannot provision according to the constraints Examples of provisioning implementation: * `localhost` ... run directly on the local machine * `openstack` ... create vm in using OpenStack * `beaker` ... reserve machine using Beaker +* `podman` ... create a container using podman +* `qemu-kvm` ... run a vm using qemu-kvm Example config: @@ -109,7 +123,7 @@ Additional configuration of the provisioned environment needed for testing. * Install with devel module * Additional setup possible if needed * Inject arbitrary commands - * Before/after TODO: What does that mean? + * Before/after artifact installation Examples of preparation implementation: @@ -126,8 +140,8 @@ Example config: Specification of the testing framework which should execute tests. -* According to the provision output does the execution -* Paralelization? +* Execute discovered tests on provisioned boxes +* With optional support for parallelization Example config: @@ -145,3 +159,14 @@ Example config: report: contact: email@address.org + +### Finish + +Additional actions to be performed after the test execution has been completed. +Counterpart of the `prepare` step useful for various cleanup actions. + +Example config: + + finish: + how: shell + command: upload-logs.sh