When the module build is canceled or resumed, the state is changed in the database via the REST API, then it sends a message that the state changed. The backend then receives the message and runs the appropriate module handler. If the backend is already running another handler on the module, it could then transition the module to another state. The failed handler would then run with the module in the incorrect state and cause the failed handler to change the module build state in the database to failed again, which causes another message to be published, which causes the failed handler to run again.

I have no idea why it was written this way, but it'd be better if the state changes in the database happened serially only by the backend.

In module_build_service.views:patch is where the state transition occurs when it is canceled. In module_build_service.utils.submit:submit_module_build is where the build is resumed. Both of those change the state of the module build in the database and send a message stating the state changed. The backend will run the handlers for those in order.

Part of the problem might be that the backend still runs the failed handler even if the state doesn't match in module_build_service.scheduler.handlers.modules:failed. It then transitions it to failed regardless of the state again at the end of module_build_service.scheduler.handlers.modules:failed.

There are some improvements that could be made to the current workflow, but the best approach would be to let the backend handle the state change in the database so that the transitions happen serially in a predictable manner.