@cverna can you help on this and also let us know who to hand this to in future.

I think I have fixed it for now with https://github.com/fedora-infra/bodhi/commit/8f7d7859330e1b370bbeb02e9c9f6f7a53f94dc3.

I am mostly away from keyboard this weekend but i ll keep monitoring this ticket, and look for a proper fix next week.

Ok happens again :( so going to try reverting back to bodhi 5.4.1

Ok so this is still happening in 5.4.1 so I reverted back to 5.5 we are getting the following traceback

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/pyramid/tweens.py", line 41, in excview_tween
    response = handler(request)
  File "/usr/lib/python3.8/site-packages/pyramid/router.py", line 147, in handle_request
    response = _call_view(
  File "/usr/lib/python3.8/site-packages/pyramid/view.py", line 667, in _call_view
    response = view_callable(context, request)
  File "/usr/lib/python3.8/site-packages/pyramid/config/views.py", line 169, in __call__
    return view(context, request)
  File "/usr/lib/python3.8/site-packages/pyramid/config/views.py", line 188, in attr_view
    return view(context, request)
  File "/usr/lib/python3.8/site-packages/pyramid/config/views.py", line 214, in predicate_wrapper
    return view(context, request)
  File "/usr/lib/python3.8/site-packages/pyramid/viewderivers.py", line 436, in rendered_view
    result = view(context, request)
  File "/usr/lib/python3.8/site-packages/pyramid/viewderivers.py", line 144, in _requestonly_view
    response = view(request)
  File "/usr/lib/python3.8/site-packages/cornice/service.py", line 502, in wrapper
    response = view_(request)
  File "/usr/local/lib/python3.8/site-packages/bodhi/server/services/releases.py", line 327, in query_releases_json
    total = db.execute(count_query).scalar()
  File "/usr/lib64/python3.8/site-packages/sqlalchemy/orm/scoping.py", line 163, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/usr/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 1291, in execute
    return self._connection_for_bind(bind, close_with_result=True).execute(
  File "/usr/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1014, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1127, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
    util.raise_(
  File "/usr/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/usr/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    self.dialect.do_execute(
  File "/usr/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) lost synchronization with server: got message type "e", length 1937009006

[SQL: SELECT count(DISTINCT releases.id) AS count_1 
FROM releases]
(Background on this error at: http://sqlalche.me/e/13/e3q8 )

I have also attached the pod logs

Could it be caused by some dependency change? I see the infra repo was added to the container two days ago, maybe something was upgraded to a version that causes problems.

Cross reference for the GitHub issue: https://github.com/fedora-infra/bodhi/issues/4105

Every single time I load almost any bodhi page I get

I don't seem to be able to reproduce this, though I don't see a resolution here or in the upstream ticket.

What's the current status? Can someone still reproduce this?

~30 minutes ago over the API:

$ fedora-update-feedback 
Username: churchyard
FAS Password: 
Authenticating with bodhi ...
Querying dnf for installed packages ...
Querying bodhi for updates ...
 Updates (testing): [ =======================================                                                                                                                                                             ]  20% Error: "Remote bodhi instance returned an error message: BodhiError {\n    errors: [\n        {\n            \"location\": \"body\",\n            \"description\": \"\\\"Could not locate column in row for column \\\'comments.id\\\'\\\"\",\n            \"name\": \"NoSuchColumnError\",\n        },\n    ],\n    status: \"error\",\n}"

The web interface does not give the error that often (possibly not at all) now.

Actually, just now:

I think perhaps @cverna fixed this? Is anyone still seeing it?

Still seeing this :-( - a couple of minutes ago [Aug 25 around 15:50 UTC]

curl 'https://bodhi.fedoraproject.org/updates/?rows_per_page=10&releases=F32F'

Gave me a response:

{"status": "error", "errors": [{"location": "body", "name": "NoSuchColumnError", "description": "\"Could not locate column in row for column 'testcases.id'\""}]}

Might be a different problem, previously I always got comments.id.

The underlying traceback that @cverna pasted: https://pagure.io/fedora-infrastructure/issue/9234#comment-671224 seems to indicate a problem with the connection from postgresql getting in a bad state - so I doubt the exact reported error matters.

From some quick google searching, it seems likely that the problem is either:
a) a concurrency problem - perhaps the two concurrent threads in the WSGI daemon process are sharing resources but not locking them [or same with Bodhi's composer threads]
b) the postgresql server unexpectedly closing the connections to Bodhi - because of query timeouts or something

Temporarily changing the WSGI configuration to a single thread per process might one place to start.

Submitted https://pagure.io/fedora-infra/ansible/pull-request/227 as a freeze break request.

However, before that I scaled the web pod to 1 instance instead of 2 and that seems to be helping?

So, yeah, when there's just 1 web pod it seems to be happy?

So what should we do here? Does that help us isolate what the problem might be?

The error I got now:

500 Internal Server Error

"Could not locate column in row for column 'builds.type'"

So, this has not happened in a long while. I am going to close it. Please let us know if anyone sees it again.

This is back with 5.7.1.

Theory: this shows up for a while whenever we update production Bodhi, then gradually eases off? Some kind of caching effect?

https://openqa.fedoraproject.org/tests/1059650#step/_advisory_update/27 - openQA test attempt to download with Bodhi failing on "Could not locate column in row" three times in a row.

Ping @asaleh this seems to have started after your bodhi upgrade. ;)

@adamwill I looked into it, but once again, can't reproduce, i.e. when trying to download the same update you mention in the advisory update 27, it just downloads.

It's intermittent, and not related to any particular update. It just happens on some attempts. To reproduce it you have to just run bodhi updates download --updateid (some update) a bunch of times and it'll happen on some of them. At least, two days ago it was. As I said above it may be one of those effects where a cache or something similar is being filled after the version upgrade / restart, and once that's done, the bug doesn't happen any more...

I think this is fixed now?

well, "fixed", or "gone away again until the next time we update bodhi"...

well, "fixed", or "gone away again until the next time we update bodhi"...

So far, so good. After 12 hours Bodhi logs don't show any sign of 500 Errors.
I think we found the root cause, it was a change in joins I made some time ago [1]... @asaleh has hot patched the prod instance, provided everything runs smoothly we will land the patch in Bodhi code permanently, so next updates should not start again going crazy.

[1] https://github.com/fedora-infra/bodhi/issues/4302#issuecomment-982433941

Awesome! Thanks @asaleh and @mattia

Did someone update bodhi again?

Is this still happening? We were doing a mass update/reboot cycle, so it might have done this while things like the database rebooted?

Yeah...so far it seems to have been a one-off, I guess it was just a coincidence that it was this error again.

I've just restarted Bodhi because since yesterday ansible change was throwing a lot of errors again...

Shall we look at switching bodhi workers to async workers as proposed in https://pagure.io/fork/mattia/fedora-infra/ansible/c/2a918955d91d982b466ae9c153668c52d43c98c1?branch=bodhi_workers ? Last time it was said that my proposed change was to be applied during the last Bodhi upgrade, but apparently didn't.

We could sure. We are however in freeze... is this worth making a freeze break out of?

If so, can you make a PR and propose it?

We could sure. We are however in freeze... is this worth making a freeze break out of?

If so, can you make a PR and propose it?

I'm going to propose a PR to start changing staging (I assume staging is not in freeze?) and wait the freeze end for prod.

Yeah, staging is never frozen. :)

I merged your pr and pushed it out.