The corresponding fedpkg work can be found at https://pagure.io/fedpkg/pull-request/309.
The corresponding fm-orchestrator work can be found at https://pagure.io/fm-orchestrator/pull-request/1136.

@cqi, @lsedlar, @onosek, could you please review this? It is needed to enable scratch-builds for modules and this is starting to be a real problem.

We have also MBS PR open and this pyrpkg change is needed in order to actually use the MBS part of this change...

New keyword argument should be added at the end for better backwards compatibility.

Do you think both spellings are needed?

Is there a situation where both srpms and scm_url/branch would be used? If yes, would be better to use --srpm foo.rpm --srpm bar.rpm instead of --srpm foo.rpm bar.rpm to avoid requiring particular order of the options?

I added a couple inline comments, but overall the code change looks fine to me. Let me summarize my understanding of module scratch builds will work based on the commands:

There are two things I can override in the scratch build: I can replace the modulemd file with --file, or I can use particular version of a modular package by specifying --srpm (even multiple times).

Is that accurate?

Just running *pkg module-scratch-build is not actually going to do use any locally modified file (so it will be equivalent to rpm scratch build from scm). Adding one of the options will turn it into an equivalent of rpm scratch build from srpm.

@lsedlar Thank you for the feedback. I'll respond as soon as I get a bit caught up following my PTO.

That is an excellent point I had overlooked. Yes, both srpms and scm_url/branch could be used at the same time. I'll revise to allow only the singular spelling, potentially appearing multiple times.

I added a couple inline comments, but overall the code change looks fine to me. Let me summarize my understanding of module scratch builds will work based on the commands:
There are two things I can override in the scratch build: I can replace the modulemd file with --file, or I can use particular version of a modular package by specifying --srpm (even multiple times).
Is that accurate?

Yes. But it's also possible to provide both a custom modulemd file AND one or more custom srpms for module scratch builds.

Just running *pkg module-scratch-build is not actually going to do use any locally modified file (so it will be equivalent to rpm scratch build from scm). Adding one of the options will turn it into an equivalent of rpm scratch build from srpm.

Right. But a scratch rpm build consists of only one package. With a scratch module build, there is a modulemd and srpm(s). Whatever is not specified on the command line will be pulled from scm--so you can pick and choose just what you want to override.

I just rebased this PR to the latest master. The corresponding fm-orchestrator work (https://pagure.io/fm-orchestrator/pull-request/1136) is expected to be merged on Monday (March 4), so we'll be ready for this to be merged soon.

@merlinm: I've tried this with MBS staging and there are few issues. Note that none of them probably blocks this PR and can be addressed by follow-up PRs if people decide that. Just listing them here:

• The --file should be able to work even outside of git repository, or at least you should be able to run it with uncommitted changes which is currently not possible.
• Even if you commit and push the changes, --file``fails withCould not execute module_build: is not JSON serializable`.

Note: the CI failure for the latest commits appears to be unrelated to this PR, but rather, perhaps, the appearance of a new version of PyYAML?

17:39:57 Collecting PyYAML (from rpkg==1.57)
...
17:39:57   Downloading https://files.pythonhosted.org/packages/9f/2c/9417b5c774792634834e730932745bc09a7d36754ca00acf1ccd1ac2594d/PyYAML-5.1.tar.gz (274kB)
17:39:58 PyYAML requires Python '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*' but the running Python is 2.6.9

I revised my latest two commits and force-pushed them, but Pagure says I added 12 new commits. Go figure.

In any case, CI still fails. I just created rpkg PR #423 as a potential workaround for that issue.

I just rebased this PR to the latest master, and I added two additional commits to show scratch status to module-build-info output, and fix module-build-watch so it exits as expected when a scratch module build completes.

@merlinm I can see you are working hard on it. Is WIP label still valid? Or it is finished and you just adding new features?
In case it is finished we can go further and prepare it for a merge until it will grow to huge dimensions. I can see 14 commits. It seems quite a lot. Are they all relevant? Of course, it doesn't have to be one only commit, but a lower number will probably make this code nicer.

@onosek Hi. Yes, the work should be finished--except for some possible fine tuning. The server side of this in fm-orchestrator has been merged for some time. I'll squash the commits down to a smaller and more sensible set and remove the WIP label shortly. Thanks!

@merlinm, I reviewed the code and I like it.
But some unit-tests are failing. This is the list:

Python3 environment:
- test_module_build_with_scratch_option

Python2 environment:
- test_module_build_scratch
- test_module_build_scratch_with_modulemd_and_srpms
- test_module_build_with_scratch_option

It seems it is one similar error in all methods above.
Currently, there is also conflict with previously merged PR, but it looks trivial. I will merge it afterwards. Thank you.

@onosek, I rebased and resolved the merge conflicts. I could have sworn the unit tests passed after my previous commit--but they're definitely passing now after adjusting the expected output to accomodate the change introduced by PR #427. Thank you!

Unit tests are OK.