#11509 Bodhi API is down, http 500 (a cross-service IODC failure?)
Closed: Fixed with Explanation 8 months ago by zlopez. Opened 8 months ago by kparal.

At the moment, Bodhi seems to crash on any API call. Web UI seems to work fine.

Example 1:

$ fedora-easy-karma 
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 812, in <module>
    fek = FedoraEasyKarma()
          ^^^^^^^^^^^^^^^^^
  File "/usr/bin/fedora-easy-karma", line 463, in __init__
    self.bc = BodhiClient()
              ^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/bodhi/client/bindings.py", line 206, in __init__
    self._build_oidc_client(client_id, id_provider)
  File "/usr/lib/python3.11/site-packages/bodhi/client/bindings.py", line 209, in _build_oidc_client
    self.oidc = OIDCClient(
                ^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/bodhi/client/oidcclient.py", line 66, in __init__
    self._build_client(client_id, scope, id_provider)
  File "/usr/lib/python3.11/site-packages/bodhi/client/oidcclient.py", line 76, in _build_client
    self._get_provider_metadata(id_provider)
  File "/usr/lib/python3.11/site-packages/bodhi/client/oidcclient.py", line 72, in _get_provider_metadata
    raise OIDCClientError(response.text)
bodhi.client.oidcclient.OIDCClientError: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at 
 root@localhost to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>

Example 2:

$ bodhi releases list
Traceback (most recent call last):
  File "/usr/bin/bodhi", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/bodhi/client/cli.py", line 269, in wrapper
    method(*args, **kwargs)
  File "/usr/lib/python3.11/site-packages/bodhi/client/cli.py", line 1316, in list_releases
    client = bindings.BodhiClient(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/bodhi/client/bindings.py", line 206, in __init__
    self._build_oidc_client(client_id, id_provider)
  File "/usr/lib/python3.11/site-packages/bodhi/client/bindings.py", line 209, in _build_oidc_client
    self.oidc = OIDCClient(
                ^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/bodhi/client/oidcclient.py", line 66, in __init__
    self._build_client(client_id, scope, id_provider)
  File "/usr/lib/python3.11/site-packages/bodhi/client/oidcclient.py", line 76, in _build_client
    self._get_provider_metadata(id_provider)
  File "/usr/lib/python3.11/site-packages/bodhi/client/oidcclient.py", line 72, in _get_provider_metadata
    raise OIDCClientError(response.text)
bodhi.client.oidcclient.OIDCClientError: 
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
  <head>
    <link rel="stylesheet" type="text/css" media="all" href="//fedoraproject.org/static/css/fedora.css">
    <link rel="stylesheet" type="text/css" media="print" href="//fedoraproject.org/static/css/print.css">
    <link rel="shortcut icon" href="//fedoraproject.org/favicon.ico" type="image/vnd.microsoft.icon">
    <!--[if lt IE 7]>
    <style type="text/css">
      #wrapper
      {
      height: 100%;
      overflow: visible!important;
      }
      .download-sidebar a {
      background-image: none!important;
      }
    </style>
    <![endif]-->
    <title>Fedora Project</title>
  </head>
  <body class="503">
    <div id="wrapper">
      <div id="head">
        <h1><a href="/en/index">Fedora</a></h1>
      </div>
      <div id="sidebar">
        <div id="nav">
          <h2>Navigation</h2>
          <ul>
            <li id="nav-home"><a href="//fedoraproject.org/index">Home</a></li>
            <li id="nav-get"><a href="//fedoraproject.org/get-fedora">Get Fedora</a></li>
            <li id="nav-join"><a href="//fedoraproject.org/join-fedora">Join Fedora</a></li>
            <li id="nav-help"><a href="//fedoraproject.org/get-help">Get Help</a></li>
          </ul>
          <h2>Tools</h2>
          <ul>
            <li><a href="//docs.fedoraproject.org/">Docs</a> Fedora documentation.</li>
            <li><a href="//fedoraproject.org/wiki/">Wiki</a> Collaborative knowledge.</li>
            <li><a href="//fedoraplanet.org/">Planet</a> The voices of the Fedora community.</li>
            <li><a href="//fedoraproject.org/wiki/Communicate">Communicate</a> Speak with Fedora.</li>
            <li><a href="//fedoraproject.org/wiki/FedoraEvents">Events</a> Meet the Fedora community.</li>
          </ul>
        </div>
      </div>
      <div id="content">
        <h2>Sorry!  This service is currently unavailable.</h2>
        <p>The service that you are trying to access is currently unavailable.  Please try refreshing this page in a couple of minutes.  If you still see this message, then please follow the steps below:</p>
        <ol>
          <li>Check on <a href="https://www.fedorastatus.org/">the status page</a> if there are any known outages for our services.</li>
          <li>If it is accessible, please check the <a href="https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/">how to work with our team document</a> for more information.</li>
        </ol>
      </div>
    </div>
    <div id="bottom">
      <div id="footer">
        <p class="copy">
        &copy; 2010 Red Hat, Inc. and others.  For comments or queries, please <a href="https://fedoraproject.org/wiki/Communicating_and_getting_help">contact us</a>.
        </p>
        <p class="disclaimer">
        The Fedora Project is maintained and driven by the community and sponsored by Red Hat.  This is a community maintained site.  Red Hat is not responsible for content.
        </p>
        <ul>
          <li class="first"><a href="//fedoraproject.org/sponsors">Sponsors</a></li>
          <li><a href="//fedoraproject.org/wiki/Legal">Legal</a></li>
          <li><a href="//fedoraproject.org/wiki/Legal/TrademarkGuidelines">Trademark Guidelines</a></li>
        </ul>
      </div>
    </div>
  </body>
</html>

bodhi-client-7.2.1-1.fc38.noarch


The issue might be more general. Logging in from pagure.io just says "discoveryfailure". Logging in from fedora wiki says "Fatal error authenticating user".

Koji is down completely with http 500.

Metadata Update from @zlopez:
- Issue priority set to: None (was: Needs Review)
- Issue tagged with: high-gain, low-trouble, ops

8 months ago

Metadata Update from @zlopez:
- Issue untagged with: low-trouble
- Issue tagged with: Needs investigation

8 months ago

Bodhi API should be back up, but the koji is still returning 500

Both of them should run now. It seems that there was some update for python package that they are using and it didn't restarted the httpd service.

After restarting the httpd service it started working again.

Metadata Update from @zlopez:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

8 months ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog