Yeah, I think the fact that STI takes precedence over TMT was only mentioned in the email when we introduced new pipelines+testing farm+tmt support. Better docs are needed for sure.

https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/thread/6CKHT5OH4EZNNQXOIAPMKXUHSRHSROHO/

There are these possiblities:

1. consider those tests as separate test entities, so report them under diffrent test name. The current usage of tier0 is broken anyway. Something like

fedora-ci.koji-build.tmt.functional
fedora-ci.koji-build.sti.functional

For default test names

1. Change testing farm to support two different tests and report it in one result. Technicall possible, but not working right now with TFT. And mvadkert is not fan of this approach of mixing two different test definitions together. Keep reporting to fedora-ci.koji-build.tier0.functional

2. Mark as unsupported, refuse to run.

I am personally fan of 1. But needs more discussion and agreement.

We now have this situation in the annobin dist git where a STI test was added to already existing FMF test. The result is that the CI system now only executes the STI test and skips the FMF one. Fixing this issue would be much appreciated.

@mcermak can you agree on team level using one format please? Ideally tmt? Tests can be easily moved to tmt from STI:

https://cockpit-project.org/blog/fmf-unified-testing.html

@mvadkert , the newly added STI test is referring to a remote repo using a URL, it is a test that is supposed to be shared across various components.

Can I refer to such kind of test within the existing FMF configuration? Or should I transform that test to FMF and store that elsewhere (e.g. in the annobin dist git)? That would certainly work, but it's a testcase duplication.

Thoughts?

Yeah, it would be needed to transform the referenced test:

https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/rawhide/f/tests

To tmt also, and then include it as a "remote" test to the plan.

@mvadkert imagine other happy users of the current version of the shared test ...

IMHO the right thing to do is to allow the users to mix and match the STI and FMF tests.

Thoughts?

There are these possiblities:

1. consider those tests as separate test entities, so report them under diffrent test name. The current usage of tier0 is broken anyway. Something like

fedora-ci.koji-build.tmt.functional
fedora-ci.koji-build.sti.functional

This would need some broader coordination as there are probably people who gate on their0.functional already. And once we do it in Fedora, I think there is a chance that people will ask for the same behavior downstream (=more coordination).

I've worked this around by dropping the STI test and instead introducing a local clone of it in the FMF format. (@tstellar FYI)

I still believe that allowing users to mix the STI and FMF tests is meaningful and has reasonable real life use cases.

+1 on this one. With the attempts to use tmt for testing Python packages I hit this issue every now and then. From the new maintainer point of view, I'd prefer to keep what's been working so far as is, and incrementally add new stuff with modern and supported framework. I've read there are no plans for deprecating STI so far while tmt gains more attention, so it'd be only beneficial to the package maintainers to have a seamless experience when they want to test their packages.
The @msrb's concern about gating is in place: python-packaging, where we'd like to add tmt configuration, sets a rule for fedora-ci.koji-build.tier0.functional.

Hi,
seems like this is still not resolved? Tried to add gating.yaml for rsync and build in rawhide but it seems like only tests from the source dist-git were run and not those that are pointed to by plans/ci.fmf. ci.fmf points here.

Should I work around this by explicitly specifying the ci.fmf in the !PassingTestCaseRule {test_case_name: fedora-ci.koji-build.tier0.functional } by changing i to something like - fedora-ci.koji-build./plans/ci.functional or is this expected to work any time soon?

@mruprich no, OSCI is not very fond on implementing this.

@msrb can we find some middle ground? If there is only one format, user tier0, if both use tmt and sti and test types?

@mvadkert @msrb I tried to follow the docs on gating to specify the file.fmf that should be used but that does not seem to work either - rsync build where the test is missing.

In the rsync repo there is the plans/ci.fmf file that specifies where the tests should be and according to the docs it should be used like this - test_case_name: fedora-ci.koji-build./plans/ci.functional. So instead it used the tier0 again.

Do I have to delete the tests/ directory from rsync repo to make this work?

hmm, seems so ... :( @msrb any idea why tmt is not the preferred test here?

as for the testcase name, you need to enable that reporting in plans ...

@psss did we test the separate plan reporting also in Fedora pls?

Do I have to delete the tests/ directory from rsync repo to make this work?

@mruprich I've renamed tests.yml to disable running STI tests [1].

1. https://src.fedoraproject.org/rpms/httpd/c/a2204eef190301180f6362bc8ff8f4dc4beb83ae?branch=rawhide

@mvadkert @msrb I tried to follow the docs on gating to specify the file.fmf that should be used but that does not seem to work either - rsync build where the test is missing.

In the rsync repo there is the plans/ci.fmf file that specifies where the tests should be and according to the docs it should be used like this - test_case_name: fedora-ci.koji-build./plans/ci.functional. So instead it used the tier0 again.

@mruprich plan names in gating.yaml should be only used if separate test plan reporting is used. Agree it's not very clear but It's stated in documentation "Before the above-mentioned rules can be used, separate plan reporting has to be enabled. See the Multiple Plans section for details." Maybe this should be mentioned at the beginning of that section @psss?

@bnater thx, I did both - disabled tests.yml AND used the ci.fmf in the main repo with the 'resultsdb-testcase: separate' statement and still the result is missing

On a related topic - why does it say '1 of 1 required test results missing' and 'no tests are required' at the same time in Bodhi?

Seems like I commented too soon, now it actually passed and it used test from the url. But all the tests are duplicated, any idea why?