Can't find the messages sent by the pipeline on datagrepper:
https://apps.fedoraproject.org/datagrepper/raw?topic=org.centos.prod.ci.koji-build.test.queued&delta=360000
https://apps.fedoraproject.org/datagrepper/raw?topic=org.centos.prod.ci.koji-build.test.running&delta=360000
https://apps.fedoraproject.org/datagrepper/raw?topic=org.centos.prod.ci.koji-build.test.complete&delta=360000
It could be due to plug-in upgrade https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/thread/KTE4EV6JLO45EIS5SHH35LHEFBCVCLII/
Example of missing message:
{"msg":{"artifact":{"component":"mariadb-connector-odbc","dependencies":[],"id":"39011579","issuer":"ljavorsk","nvr":"mariadb-connector-odbc-3.1.5-1.fc32","scratch":false,"source":"UNKNOWN","type":"koji-build"},"contact":{"docs":"https://pagure.io/standard-test-roles","email":"ci@lists.fedoraproject.org","irc":"#fedora-ci","name":"fedora-ci","team":"fedora-ci","url":"https://jenkins-continuous-infra.apps.ci.centos.org/"},"generated_at":"2019-11-15T12:00:27.903862Z","pipeline":{"build":"7285","id":"57df177e-ac2e-4ca9-b51f-9f074a33eebf","name":"fedora-rawhide-build-pipeline","stage":{"name":"package-tests"}},"run":{"log":"https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-build-pipeline/7285/console","rebuild":"https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-build-pipeline/7285/rebuild/parameterized","url":"https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-build-pipeline/detail/fedora-rawhide-build-pipeline/7285/pipeline/"},"system":[{"architecture":"x86_64","label":"upstream-fedora-pipeline","os":"master","provider":"CentOS CI","variant":"Cloud"}],"test":{"category":"functional","namespace":"fedora-ci.koji-build","result":"passed","type":"tier0"},"version":"0.2.1"},"msg_id":"2019-5c9f7ba1-3c29-4f82-8799-233a90a9a796","timestamp":1573819234114,"topic":"org.centos.prod.ci.koji-build.test.complete"}
@astepano what do you think could have been caused by the plug-in update?
@bgoncalv the same plugins is used for receiving messages and sending. The subscriber/trigger works as expected as you can see. I added logs for the JMS Messaging plugin. It shows that messages are sent:
pub address: tcp://172.19.4.24:9941 Nov 15, 2019 2:13:45 PM FINE com.redhat.jenkins.plugins.ci.messaging.FedMsgMessagingWorker sendMessage JSON message body: {"msg":{"branch":"master","build_id":"1413174","build_url":"https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-build-pipeline-trigger/detail/fedora-build-pipeline-trigger/417234/pipeline/","ci_topic":"org.centos.prod.ci.pipeline.allpackages-build.package.queued","comment_id":"","namespace":"rpms","nvr":"pyparted-3.11.3-1.fc32","original_spec_nvr":"","ref":"x86_64","repo":"pyparted","rev":"kojitask-39012848","scratch":false,"status":"SUCCESS","test_guidance":"''","username":"dcantrel"},"msg_id":"2019-a3536c81-a5da-4844-8ed5-401ef4402d22","timestamp":1573827225211,"topic":"org.centos.prod.ci.pipeline.allpackages-build.package.queued"} Nov 15, 2019 2:13:51 PM FINE com.redhat.jenkins.plugins.ci.messaging.FedMsgMessagingWorker sendMessage
However, https://apps.fedoraproject.org/datagrepper/raw?topic=org.centos.prod.ci.pipeline.allpackages-build.package.queued&delta=360000 Doesn't show sent messages.
https://jenkins-continuous-infra.apps.ci.centos.org/log/MESSAGING%20LOG%20XXX%20ASTEPANO/
Asked on IRC @bstinson to check 172.19.4.24 side
15:33:33 <bstinson> i see them showing up on the relay 15:33:42 <bstinson> which means jenkins is sending them to the right place 15:34:57 <bgoncalv> interesting, I can't find the test result in bodhi when I check the tests for a build 15:34:58 <astepano> magic :-D 15:37:59 <bstinson> http://resultsdb.ci.centos.org/resultsdb/results 15:38:27 <bstinson> do those timestamps line up with recent jobs? 14:34 was around 4 minutes ago 15:42:43 <astepano> bgoncalv: https://apps.fedoraproject.org/datagrepper/raw?topic=org.centos.prod.ci.pipeline.allpackages-build.package.complete&delta=360000 and received message: http://resultsdb.ci.centos.org/resultsdb/results/528497
note, we don't have any new results in this resultsdb...
https://taskotron.fedoraproject.org/resultsdb/results?testcases=fedora-ci.koji-build.tier0.functional
From IRC:
<pingou> there are two instances of resultsdb, one in rdu and one in phx2 <pingou> we see the results in rdu but not in phx2 <bookwar> so what is the impact of the issue currently? we test things, test results land into one of the resultsdb instances, but datagrepper/bodhi/greenwave can not see them? <pingou> correct
Troubleshooting continues
Test showing that messages are going through: https://paste.centos.org/view/975ce2f1
How to replicate if needed while we continue troubleshooting the issue:
<jbair> bstinson: can you share your method for the above test? <jbair> I'd like to document it in case anyone needs to work around the issue if it persists longer than today <bstinson> jbair: steps are: 'dnf install fedmsg', make /etc/fedmsg.d/endpoints.py look like this: https://paste.centos.org/view/f4bd834d, run 'fedmsg-tail'
Escalated to Fedora Infra here: https://pagure.io/fedora-infrastructure/issue/8389
this is fixed.
Metadata Update from @bgoncalv: - Issue status updated to: Closed (was: Open)