cc @jlebon

Two things:

1. yes, maybe there is something to fix - we could move latest symlink only if there is no newer repo already. Anyway, we want to have these parallel tasks, as users may have a reason to create 'obsoleted' repo (e.g. for reconstructing some older build).
2. this looks like flawed workflow. Why are you running so many distrepos, if you care only about last? kojira (when doing normal buildroot repos) doesn't spawn duplicate tasks if they are some already running for given repo. So only one (dist)repo task should be running in one moment. Maybe ideal for your situation is to check, if there is running distrepo task owned by you, cancel it and trigger new one. Risk is, that if you're doing a lot of builds, it could cancel/trigger forever and no new distrepo will be created.

Isn't it more an issue for tag2distrepo? Maybe, there could be an option to toggle between current behaviour and cancel/retrigger. @puiterwijk ?

tag2distrepo was initially developed for the Infra tags, where we have on average one tag action per week, I'd estimate, with sometimes 4 per day, and then 0 in a week.
In that context, the possibility of a race condition is small enough that it's not really a problem, and the duplicate tasks problem would also not appear.
From what I had heard, the same approximate amount would've been the case for CoreOS, but I guess the tagging behaviour there changed significantly.

If anyone wants improvements to tag2distrepo, they can open a Releng ticket, or submit PRs.

tag2distrepo was initially developed for the Infra tags, where we have on average one tag action per week, I'd estimate, with sometimes 4 per day, and then 0 in a week.
In that context, the possibility of a race condition is small enough that it's not really a problem, and the duplicate tasks problem would also not appear.
From what I had heard, the same approximate amount would've been the case for CoreOS, but I guess the tagging behaviour there changed significantly.

I think the problem is not really how many per week, but rather how many can be run simultaneously. In CoreOS we checkin a file that contains a list of builds to tag and if there are 4 or 5 builds in there they all get tagged at the same time which means we can get into this race condition. So even if we only tag 4 builds in a week, but we do it all at the same time we could still end up in this condition.

If anyone wants improvements to tag2distrepo, they can open a Releng ticket, or submit PRs.

I found the code at pagure.io/releng/tag2distrepo. If that is where a fix for this would be applied I can see if I can find an obvious place in the code. Thanks

I think the problem is not really how many per week, but rather how many can be run simultaneously. In CoreOS we checkin a file that contains a list of builds to tag and if there are 4 or 5 builds in there they all get tagged at the same time which means we can get into this race condition. So even if we only tag 4 builds in a week, but we do it all at the same time we could still end up in this condition.

So is it doing koji tag-pkg in a loop? Perhaps it could just do them all in one call?

So is it doing koji tag-pkg in a loop? Perhaps it could just do them all in one call?

We are running a koji multicall to tag all builds using a single call into an intermediate signing tag, which then gets signed by robosignatory and then tagged into coreos-pool. So it's actually the robosignatory process that's doing the final tagging into coreos-pool.

It looks like the code within tag2distrepo runs within a koji plugin postTag callback. I don't know if we can appropriately serialize a callback. I see a few possible approaches:

• Make the tag2distrepo code inspect to see if an existing distrepo is running for this tag and wait until there is no distrepo running for this tag.. If we tag in 10 builds we'd have 9 waiting, then 8, then 7. etc..
• Serialize the distrepo code within koji itself (is there value in multiple disrepos on a tag in parallel?)
• Add logic to the repoDone function in kojihub.py to make it only update the latest symbolic link if the current distrepo task is the one with the highest repoid or koji task id? i.e. This will guarantee the latest link points to the latest version of the repo.

@tkopecek @puiterwijk @kevin - do any of those seem reasonable?

Make the tag2distrepo code inspect to see if an existing distrepo is running for this tag and wait until there is no distrepo running for this tag.. If we tag in 10 builds we'd have 9 waiting, then 8, then 7. etc..

-1 - This call is hub plugin, so it is happening in httpd. Waiting there means exhausting hub resources and potentially blocking everything.

Serialize the distrepo code within koji itself (is there value in multiple disrepos on a tag in parallel?)

0 - Not sure here, if it usable. theoretically we can have an option in tag, dedicated to distrepo only (normal repos definitely needs to be runnable in parallel). In such case we can extend distrepo task to wait for all (at most one in this case) distrepo to be finished, before task itself is run. It could be done quite simply by prepending 'waitRepo' subtask.

Other option would be to have it as a distRepo task option. So, it wouldn't be in tag configuration, but you can request "actual" repo, which is triggered after all currenlty running distRepo tasks for given tag. It is a little bit more clean. Note, that this can also be done client-side (which is not best if distRepo is triggered by plugin). I've gone similar way in #1619, so it should be consistent with this behaviour.

Add logic to the repoDone function in kojihub.py to make it only update the latest symbolic link if the current distrepo task is the one with the highest repoid or koji task id? i.e. This will guarantee the latest link points to the latest version of the repo.

This one looks best to me. It makes sense to have 'latest' pointing to repo which was created latest, not finished. @mikem?

I see a few possible approaches:

IIUC here based on https://pagure.io/koji/issue/1630#comment-593345, would another approach be to enhance robosignatory to also use the multicall interface to tag all the packages in one shot?

I see a few possible approaches:

IIUC here based on https://pagure.io/koji/issue/1630#comment-593345, would another approach be to enhance robosignatory to also use the multicall interface to tag all the packages in one shot?

It would be nice to know if that would actually work. IIUC even though we're using the multicall interface we still end up with N tasks for N builds to tag (i.e. a task for each build to tag). So I'm not even sure that would solve the problem.

I have a concern that there are a number of unnecessary dist repo tasks being created. With 10 builds, there are 10 dist repo tasks but only one is needed. That may not be so bad (9 wasted tasks), but is there a chance that we may have 50 or 100 builds in the future? 100 dist repo tasks with 99 thrown away seems problematic. Is that a legitimate concern?

Is that a legitimate concern?

It is. Typically I'd say the number will be more around 10 (i.e. it's a set of a bout 400 packages and the new builds we tag are the ones that have reached stable in the past day), but it is a concern. If there is a way to batch koji tagging tasks and make only one distrepo get called using the tag2distrepo plugin, then we can do that.

Alternatively we can stop using tag2distrepo for this tag and start calling distrepo directly (if someone will give us permission to do so), but I think the race condition is a legitimate race condition for anyone using tag2distrepo. If you tag more than one package (even if it's just 2) then there is a possibility for the latest symlink to point to the wrong directory if the latest distrepo task finishes after the earlier ones.

and start calling distrepo directly

Assuming by this you mean via a bot, and not humans, then +1. E.g. coreos-koji-tagger knows when all packages are tagged in, so it could easily queue up a single distRepo operation after.

and start calling distrepo directly

Assuming by this you mean via a bot, and not humans, then +1.

Correct.

E.g. coreos-koji-tagger knows when all packages are tagged in, so it could easily queue up a single distRepo operation after.

Exactly. The only problem here is that we have tagged packages in manually and it seemed better to make the distrepo be automatic (via tag2distrepo). That automation is probably something we can live without.

Any more thoughts on this? As an example, the tagger just tagged a few packages in, and it resulted in 19 concurrent distRepo tasks. :(

Exactly. The only problem here is that we have tagged packages in manually and it seemed better to make the distrepo be automatic (via tag2distrepo). That automation is probably something we can live without.

Yeah, that's a good point. Seems like an OK compromise in the short-term. Though definitely would be nice to sort this out properly long-term.

What about tag option saying, that there should be only one distrepo task running. In such case, distrepo task will look, if there is something running already and cancel it. So, there will be only one task running in any moment. Of course, there needs to be some privileged uncancellable tasks (spawned by admin/repo permissions) which needs to be run anyway.
?

What about tag option saying, that there should be only one distrepo task running. In such case, distrepo task will look, if there is something running already and cancel it.

Sounds good to me. It sounds like this is a variation of:

• Make the tag2distrepo code inspect to see if an existing distrepo is running for this tag and wait until there is no distrepo running for this tag.. If we tag in 10 builds we'd have 9 waiting, then 8, then 7. etc..

from https://pagure.io/koji/issue/1630#comment-593487 ?

Yep, it is a variation, but embedded to base koji instead of plugin, so maybe little bit more general.

+1 - I'm in favor of anything that solves this problem and isn't prone to races.

PR #1869

Commit 74f525a7 fixes this issue

Commit 4d037946 fixes this issue

a small fix: #1912

Commit eb3a8360 fixes this issue

Commit 75f4c24b fixes this issue

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/1630

Please continue any further discussion there.