We have duplicate results in resultsdb at dev: http://taskotron-dev.fedoraproject.org/resultsdb/results?&testcases=dist.rpmlint
It looks like resultsdb_api returns error 500 at every first attempt to report something (and is retried then).
10.5.124.209 - - [14/May/2018:12:41:11 +0000] "GET /resultsdb_api/api/v2.0/testcases/dist.rpmgrill HTTP/1.1" 200 194 "-" "python-requests/2.18.4" 10.5.124.209 - - [14/May/2018:12:41:11 +0000] "POST /resultsdb_api/api/v2.0/results HTTP/1.1" 500 291 "-" "python-requests/2.18.4" 10.5.124.209 - - [14/May/2018:12:41:11 +0000] "POST /resultsdb_api/api/v2.0/results HTTP/1.1" 201 754 "-" "python-requests/2.18.4"
Every first attempt is failing with:
[Mon May 14 12:42:27.034612 2018] [wsgi:error] [pid 17712:tid 140448298104576] [remote 10.5.124.209:33516] publish_taskotron_message(result) [Mon May 14 12:42:27.034618 2018] [wsgi:error] [pid 17712:tid 140448298104576] [remote 10.5.124.209:33516] File "/usr/lib/python2.7/site-packages/resultsdb/messaging.py", line 100, in publish_taskotron_message [Mon May 14 12:42:27.034624 2018] [wsgi:error] [pid 17712:tid 140448298104576] [remote 10.5.124.209:33516] fedmsg.publish(topic='result.new', modname='taskotron', msg=msg) [Mon May 14 12:42:27.034630 2018] [wsgi:error] [pid 17712:tid 140448298104576] [remote 10.5.124.209:33516] File "/usr/lib/python2.7/site-packages/fedmsg/__init__.py", line 81, in _wrapper [Mon May 14 12:42:27.034635 2018] [wsgi:error] [pid 17712:tid 140448298104576] [remote 10.5.124.209:33516] return func(*args, **kw) [Mon May 14 12:42:27.034641 2018] [wsgi:error] [pid 17712:tid 140448298104576] [remote 10.5.124.209:33516] File "/usr/lib/python2.7/site-packages/fedmsg/__init__.py", line 93, in publish [Mon May 14 12:42:27.034646 2018] [wsgi:error] [pid 17712:tid 140448298104576] [remote 10.5.124.209:33516] return __local.__context.publish(topic, msg, **kw) [Mon May 14 12:42:27.034652 2018] [wsgi:error] [pid 17712:tid 140448298104576] [remote 10.5.124.209:33516] File "/usr/lib/python2.7/site-packages/fedmsg/core.py", line 327, in publish [Mon May 14 12:42:27.034658 2018] [wsgi:error] [pid 17712:tid 140448298104576] [remote 10.5.124.209:33516] self.publisher.send_multipart( [Mon May 14 12:42:27.034663 2018] [wsgi:error] [pid 17712:tid 140448298104576] [remote 10.5.124.209:33516] AttributeError: 'FedMsgContext' object has no attribute 'publisher'
easy pre-fix would be disabling the message spawning in config on dev, before we have time to debug. On the other hand, this is not working for quite some time, is anybody actually missing the messages? :)
I don't know about fedmsg error, but the duplicated results seem to present only since 2018-05-12 (around 5 AM UTC). I don't know what changed, but I see no result duplication before. The last dnf transaction is from 05-08. I'm puzzled.
Don't know either. I manually turned the fedmsg production off for now, we can debug this later on.
Thanks. Please put it into ansible, so that we can know how to easily revert it once we figure out how to fix this.
Turns out we don't want fedmsgs to be published from dev, we just added a new config option that was not properly disabled on dev and that caused the confusion. I fixed this in ansible: https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=8561ed8031baeed432cd918311410af0417c5f2e
For future reference, I believe the traceback was caused by resultsdb-dev.yml playbook not containing the fedmsg roles as our stg/prod playbooks, which caused the fedmsg certificates not being generated in /etc/pki/fedmsg.
resultsdb-dev.yml
/etc/pki/fedmsg
Metadata Update from @kparal: - Issue assigned to kparal (was: jskladan)
Metadata Update from @kparal: - Issue close_status updated to: Fixed
Log in to comment on this ticket.