So far, I handle the following exit codes in task-modularity-testing-framework:

# MTF exit codes
MTF_EXIT_FAIL='125'
MTF_EXIT_SKIP='126'

(and 0, obviously). Is there any documentation of additional exit codes?

Because 2 is the same as 1 for me at the moment, the task will crash anyway. What do you want to actually happen?

So actually MTF should exit with "MTF_EXIT_FAIL='125'".

We don't want to store fake results in resultsdb (crashed tasks that are masked as failed tests). Crashes are different from failed tests. That being said, we don't really have a good way to notify you about crashed tasks (because resultsdb sends fedmsgs, so if we don't store it there, no notification gets sent), and I'm sorry about that.

For this particular issue, the root problem should be solved instead. When debugging this with @psabata on IRC, he found out this is likely a race condition, because the module is in PDC now:
https://pdc.fedoraproject.org/rest_api/v1/unreleasedvariants/?active=True&variant_id=base-runtime&variant_version=f26&variant_release=20170629141656

So PDC should be fixed. Until that is done, here's a suggestion:

is the "active: true" constraint relevant in this context? could you just drop it?
I don't know, we can ask jscotka

@bgoncalv @kparal thans for clarification.
I think we have to have there this constrain "active: true"
because it is used also to look for dependencies. So it is not easy to remove that. This caused me issues in past several times, when I've removed that.

As an example is: memcached is dependent on baseruntime. so that I'll look for baseruntime, stream: f26, -> without active: true, it can also find some failed one so that, bad packages there (maybe they are already not stored in koji?)

Just one solution is, to use active:true just when specific version is not selected. so that I'm looking for the lastest one. so this could solve this issue. But this is little bit tricky and can cause sideeffects

@jscotka do you think it this case of missing PDC build MTF should crash? I'm not sure if the crash is correct or it should exit with MTF_EXIT_FAIL.

@bgoncalv You need to distinguish test suite failure from test system/environment failure. You can't group them up under a single exit code.

Hi,
closing this issue.
In future taskotron will solve it in more elegant way. MTF is not able to improve this anyhow.