#1359 Module builds stall when resumed when a Koji error occurs in components._finalize
Opened 4 years ago by mprahl. Modified 4 years ago

The module build 5153 failed when the following Koji communication error occurred in components._finalize at the end of a batch.

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/module_build_service/scheduler/consumer.py", line 256, in process_message
    further_work = handler(conf, session, msg) or []
  File "/usr/lib/python2.7/site-packages/module_build_service/scheduler/handlers/components.py", line 167, in complete
    return _finalize(config, session, msg, state=koji.BUILD_STATES["COMPLETE"])
  File "/usr/lib/python2.7/site-packages/module_build_service/scheduler/handlers/components.py", line 99, in _finalize
    session, parent, config
  File "/usr/lib/python2.7/site-packages/module_build_service/builder/base.py", line 147, in create_from_module
    components=components,
  File "/usr/lib/python2.7/site-packages/module_build_service/builder/base.py", line 124, in create
    owner=owner, module=module, config=config, **extra)
  File "/usr/lib/python2.7/site-packages/module_build_service/utils/general.py", line 282, in wrapper
    return function(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/module_build_service/builder/KojiModuleBuilder.py", line 181, in __init__
    self.koji_session = self.get_session(config)
  File "/usr/lib/python2.7/site-packages/module_build_service/utils/general.py", line 151, in inner
    return function(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/module_build_service/builder/KojiModuleBuilder.py", line 510, in get_session
    koji_session.krb_login(principal=principal, keytab=keytab, ctx=ctx, ccache=ccache)
  File "/usr/lib/python2.7/site-packages/module_build_service/builder/koji_backports.py", line 76, in krb_login
    (rep_enc, sinfo_enc, addrinfo) = self.callMethod('krbLogin', req_enc, proxyuser)
  File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 2410, in callMethod
    return self._callMethod(name, args, opts)
  File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 2527, in _callMethod
    return self._sendCall(handler, headers, request)
  File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 2449, in _sendCall
    return self._sendOneCall(handler, headers, request)
  File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 2489, in _sendOneCall
    r.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 834, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 502 Server Error: Proxy Error

After the failure, the user resumed the module build, but it just stalled since there was no event to trigger MBS to tag all the components.

Retries could be added to that part of the code, the scheduler could handle this issue when the build is resumed, or code could be added in the poller to find modules stuck in this situation.


Log in to comment on this ticket.

Metadata