In the last couple of weeks I have been seeing Bodhi API timeouts occurring multiple times for a period of time.
I hope someone could look into it and find out what is going on.
I am primarily using the Releases endpoint (for example it is used by fbrnch and also dl-fedora, etc). I think I need to make them more robust for this, but this hasn't really been a problem at all until now.
Funny thing is it seems to depend on the calling client: curl seems okay, but not /usr/bin/bodhi or /usr/bin/bodhi-cli.
Is it rejecting connections in some circumstances?
$ bodhi releases info FEDORA-42 <no response>
I mentioned these timeouts a few times on matrix including #devel
Thanks for asking in matrix about cookies etc.
I think fbrnch is accessing the bodhi API directly without any authentication or cookies. But I think it is using http/1.1 and not http/2 like curl
However it also happens with the python bodhi client: does that use cookies or auth?
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, bodhi, medium-gain
The bodhi cli can auth with oauth... but if you are seeing it with curl, the auth angle isn't very likely.
When you get timeouts could you get curl -v output? that would tell us what proxy it was connecting to and possibly provide some hints.
I think there were some recent infra OS upgrades? I am wondering if it could relate to that.
It may not just be Bodhi indeed: right now download is timing out for me too (via Haskell). edit: also pagure.io right now
Just to be clear curl always seems to work, but not python/haskell apparently it seems.
Here is a simple reproducer for me with python:
import requests print(requests.get("https://bodhi.fedoraproject.org/releases/F42").json())
which always hangs.
Compare with: curl https://bodhi.fedoraproject.org/releases/F42
curl https://bodhi.fedoraproject.org/releases/F42
Or is this some kind of client filtering?
Both fedpkg request-branch and fbrnch request-branches are also timing out for me currently. edit: finally fedpkg completed (after 3 min).
fedpkg request-branch
fbrnch request-branches
Can you try and identify what proxy you are hitting when this happens?
either from ss -an output or tcpdump or there might be some way in requests.
I am really thinking it is somehow a proxy misbehaving as we are not seeing this issue very widely. ;(
Oh, you could also try just overriding the proxy in /etc/hosts:
38.145.60.20 bodhi.fedoraproject.org
does doing that work?
We also fixed a mailman query that was saturating I/O on the database server.
Are you still seeing this issue now?
Hopefully things are better/solved. Please re-open if you still are seeing anything.
Metadata Update from @kevin: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.