pungi

#492 add retry policy for failed tasks
Opened by dustymabe. Modified

Open
Close Issue

Often when debugging failed nightly prod atomic composes (like this one) we end up with a resolution of: "this was a transient failure". For example this task is what caused the earlier linked compose to fail, but the same task passed when it was resubmitted by dennis later.

Could we try to implement a "retry once" policy for failed tasks? I know it won't necessarily get rid of all transient failures as some of them are related to prolonged outages of other infrastructure, but it may help get past some things.

I will add I had restarted libvirt and kojid on the builder that did the image build. which may have been what was needed for it to succeed

It would be relatively easy to add the retries, however how likely is this to avoid problems versus just make the compose slower (by having to wait for the same failure twice)?

I'm inclined to closing this as Won't Fix as the price we would pay for this seems too high compared to the very little benefit it would present.

If you still think this is a desirable feature, feel free to reopen.

Metadata Update from @lsedlar:
- Issue status updated to: Closed (was: Open)

Metadata Update from @lsedlar:
- Issue close_status updated to: Invalid

Just wondering if we could enable retry option on non-failable artifacts? Also we could add some logic of identifying the failures, like if pungi thinks its a network glitch sorta thing then retry it if not skip it.

Metadata Update from @mohanboddu:
- Issue status updated to: Open (was: Closed)

it could be a sort of 'operator opt-in': after an image fails an alert is sent and admins have, say, an hour to hit a 'just retry it' button, otherwise the compose dooms.

Not sure how complex that would be to implement...

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/pungi/pungi/issues/492

Please continue any further discussion there.
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.