#7606 I think Bodhi's koji kerberos ticket is expired
Closed: Fixed 5 years ago by kevin. Opened 5 years ago by bowlofeggs.

  • Describe what you need us to do:
    The backend sent this e-mail to me:

Message

[2019-02-28 00:00:16][requests_kerberos.kerberos_ ERROR]

(('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Ticket expired', -1765328352))

Process Details

  • host: bodhi-backend01.phx2.fedoraproject.org
  • PID: 134769
  • name: fedmsg-hub-3
  • command: /usr/bin/python3 /usr/bin/fedmsg-hub-3
  • msg_id:

Callstack that lead to the logging statement

  File "/usr/lib64/python3.7/threading.py", line 885 in _bootstrap

  File "/usr/lib64/python3.7/threading.py", line 917 in _bootstrap_inner

  File "/usr/lib/python3.7/site-packages/bodhi/server/consumers/masher.py", line 344 in run

  File "/usr/lib/python3.7/site-packages/bodhi/server/consumers/masher.py", line 408 in work

  File "/usr/lib/python3.7/site-packages/bodhi/server/consumers/masher.py", line 76 in wrapper

  File "/usr/lib/python3.7/site-packages/bodhi/server/consumers/masher.py", line 570 in determine_and_perform_tag_actions

  File "/usr/lib/python3.7/site-packages/bodhi/server/consumers/masher.py", line 588 in _determine_tag_actions

  File "/usr/lib/python3.7/site-packages/bodhi/server/models.py", line 1455 in get_tags

  File "/usr/lib/python3.7/site-packages/bodhi/server/buildsys.py", line 639 in get_session

  File "/usr/lib/python3.7/site-packages/bodhi/server/buildsys.py", line 672 in get_koji_login

  File "/usr/lib/python3.7/site-packages/bodhi/server/buildsys.py", line 596 in koji_login

  File "/usr/lib/python3.7/site-packages/koji/__init__.py", line 2120 in krb_login

  File "/usr/lib/python3.7/site-packages/koji/__init__.py", line 2258 in gssapi_login

  File "/usr/lib/python3.7/site-packages/koji/__init__.py", line 2475 in _callMethod

  File "/usr/lib/python3.7/site-packages/koji/__init__.py", line 2397 in _sendCall

  File "/usr/lib/python3.7/site-packages/koji/__init__.py", line 2436 in _sendOneCall

  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 572 in post

  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 524 in request

  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 644 in send

  File "/usr/lib/python3.7/site-packages/requests/hooks.py", line 31 in dispatch_hook

  File "/usr/lib/python3.7/site-packages/requests_kerberos/kerberos_.py", line 404 in handle_response

  File "/usr/lib/python3.7/site-packages/requests_kerberos/kerberos_.py", line 400 in handle_response

  File "/usr/lib/python3.7/site-packages/requests_kerberos/kerberos_.py", line 286 in handle_401

  File "/usr/lib/python3.7/site-packages/requests_kerberos/kerberos_.py", line 261 in authenticate_user

  File "/usr/lib/python3.7/site-packages/requests_kerberos/kerberos_.py", line 243 in generate_request_header

Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/requests_kerberos/kerberos_.py", line 227, in generate_request_header
channel_bindings=self.cbt_struct)
kerberos.GSSError: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Ticket expired', -1765328352))

  • When do you need this? (YYYY/MM/DD)
    ASAP

  • When is this no longer needed or useful? (YYYY/MM/DD)
    If we stop shipping updates.

  • If we cannot complete your request, what is the impact?
    Bodhi cannot compose updates.


I marked urgent because we cannot ship updates right now. If you disagree, feel free to lower priority.

Metadata Update from @smooge:
- Issue assigned to smooge

5 years ago

Random observation:

This really just mean that your Kerberos TGT expired, not the kerberos principal itself. Can you run these commands in the env?

$ export KRB5_TRACE=/dev/stdout
$ klist -k ./the.keytab
$ kinit -k -t ./the.keytab <bodhi-identity>

[root@bodhi-backend01 ~][PROD]# klist -t -k /etc/krb5.bodhi_bodhi.fedoraproject.org.keytab
Keytab name: FILE:/etc/krb5.bodhi_bodhi.fedoraproject.org.keytab
KVNO Timestamp Principal


1 2018-11-05 20:52:45 bodhi/bodhi.fedoraproject.org@FEDORAPROJECT.ORG
1 2018-11-05 20:52:45 bodhi/bodhi.fedoraproject.org@FEDORAPROJECT.ORG
1 2018-11-05 20:52:45 bodhi/bodhi.fedoraproject.org@FEDORAPROJECT.ORG
1 2018-11-05 20:52:45 bodhi/bodhi.fedoraproject.org@FEDORAPROJECT.ORG
1 2018-11-05 20:52:45 bodhi/bodhi.fedoraproject.org@FEDORAPROJECT.ORG
1 2018-11-05 20:52:45 bodhi/bodhi.fedoraproject.org@FEDORAPROJECT.ORG
1 2018-11-05 20:52:45 bodhi/bodhi.fedoraproject.org@FEDORAPROJECT.ORG
1 2018-11-05 20:52:45 bodhi/bodhi.fedoraproject.org@FEDORAPROJECT.ORG

Removed out of date kerberos ticket and restarted services. I think this recovered. Can we close as fixed?

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

5 years ago

Log in to comment on this ticket.

Metadata