fedora-infrastructure

Clone
Source Code
GIT

#9820 Fedora meetings mailing list posts all have subject "fedmsg notification" instead of something useful as of March 26
Closed: Fixed 3 years ago by kevin. Opened 4 years ago by mattdm.

Closed: Fixed
Reopen Issue

See https://lists.fedoraproject.org/archives/list/meetingminutes@lists.fedoraproject.org/.

Previously, meetings would have titles like:

  • Subject: nirik's meeting titled "Fedora Infrastructure Ops Daily Standup Meeting" ended in #fedora-meeting-3
  • Subject: geppetto's meeting titled "fpc" ended in #fedora-meeting-1
  • Subject: tdawson's meeting titled "EPEL (2021-03-24)" ended in #fedora-meeting

But now it is all:

  • fedmsg notification
  • fedmsg notification
  • fedmsg notification

This seems to have started on March 26th.

Metadata Update from @humaton:
- Issue tagged with: dev, low-gain, medium-trouble
4 years ago

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
4 years ago

So this is already reported. Its been more than 1 month and many many emails on meetingminutes list with the same subject name. Can someone please look into this issue please?

@mohanboddu at least the other ticket I reported got some attention. I don't see anyone discussing here this problem.

We have more things to work on than we can work on... and this is... not high priority.

We will hopefully get to it, but please do be patient?

This issue as @pingou noted is likely in fedmsg meta rules somewhere.

I know it seems trivial, but this actually has a meaningful impact on my ability to keep up with summaries of what all is going on. I know there is a lot of work, but could it please not be bottom-of-the-barrel priority?

@ryanlerch perhaps you could dig into this one as it's irc bot related and you have been working on those lately?

Metadata Update from @ryanlerch:
- Issue assigned to ryanlerch
3 years ago

Ok, it's was a little hard to track down where these email were coming from, but i think they are being sent by FWN. From (https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/fmn.html#mailing-lists)

We use FMN as a way to forward certain kinds of messages 
to mailing lists so people can read them the good old fashioned 
way that they like to. To accomplish this, we create ‘bot’ FAS 
accounts with their own FMN profiles and we set their email 
addresses to the lists in question.

and i found this user in Fedora Accounts, which i assume is the bot user for sending messages from FWN to the meetingminutes mailing list:

https://accounts.fedoraproject.org/user/fmnmeetingminutes/

However, i can't jump into this account to check if this is the case, or to fix the filter, as i am not an admin, Is it possible for me to be added as a FMN admin?

Pretty sure this is the config file that controls this:

https://pagure.io/fedora-infra/ansible/blob/main/f/roles/notifs/frontend/templates/fmn.web.cfg
Edited 3 years ago by ryanlerch

Okay, so i found out where the "fedmsg notification" text is coming from

FWN creates it when it hits an exception trying to generate the title:

https://github.com/fedora-infra/fmn/blob/develop/fmn/formatters.py#L394

It also logs the error, but not sure how to find that log...

 
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]: [2021-06-16 02:54:31][   fmn.lib   ERROR] 'module' object has no attribute 'get'
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]: Traceback (most recent call last):
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/lib/__init__.py", line 86, in matches
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     value = fn(config, message, **arguments)
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/rules/generic.py", line 21, in user_filter
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return fasnick in fmn.rules.utils.msg2usernames(message, **config)
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/rules/utils.py", line 451, in msg2usernames
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return _cache.get_or_create(key, creator)
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 833, in get_or_create
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     async_creator) as value:
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/dogpile/lock.py", line 154, in __enter__
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return self._enter()
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/dogpile/lock.py", line 94, in _enter
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     generated = self._enter_create(createdtime)
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/dogpile/lock.py", line 145, in _enter_create
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     created = self.creator()
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 800, in gen_value
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     created_value = creator()
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/rules/utils.py", line 450, in <lambda>
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     creator = lambda: fedmsg.meta.msg2usernames(msg, **config)
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fedmsg/meta/__init__.py", line 152, in __wrapper
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return f(msg, **config)
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fedmsg/meta/__init__.py", line 171, in __wrapper
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return f(msg, processor=processor, **config)
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fedmsg/meta/__init__.py", line 295, in msg2usernames
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return processor.usernames(msg, **config)
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fedmsg_meta_fedora_infrastructure/supybot.py", line 132, in usernames
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     if nick not in blacklisted_people
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/fmn_fasshim.py", line 185, in nick2fas
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     update_nick(nickname)
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/fmn_fasshim.py", line 107, in update_nick
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]:     if config.get('fasjson'):
Jun 16 02:54:31 notifs-backend01.iad2.fedoraproject.org celery[9997]: AttributeError: 'module' object has no attribute 'get'

So it's something in ansible 821209cb26 (the hotfix for fasjson support).

Ok, after looking at https://pagure.io/fedora-infra/ansible/c/821209cb26?branch=main i think i found that issue there, and tried to fix it with this PR:

https://pagure.io/fedora-infra/ansible/pull-request/644

that was not the exception that i was expecting to see here, but it looks like that one is filling up the logs it seems...

I merged that, but not clear if thats the problem here or not. :)

There's other tracebacks there also, I think nb added you so you can look on notifs-backend01? Just look in the journal there...

I think we made it worse. :) It doesn't appear to be sending them at all now. ;( 

n.consumer    INFO] FMNConsumer received 2021-90881338-968e-4782-a502-921a5c63c74c org.fedoraproje
ct.prod.meetbot.meeting.complete
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org fedmsg-hub[9984]: [2021-06-16 18:28:30][mo
ksha.hub   DEBUG] 'FMNConsumer' thread 140498513688320 | Going back to waiting on the incoming que
ue.
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/s
ite-packages/dogpile/lock.py", line 94, in _enter
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     generated = self._enter_
create(createdtime)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/s
ite-packages/dogpile/lock.py", line 145, in _enter_create
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     created = self.creator()
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/s
ite-packages/dogpile/cache/region.py", line 800, in gen_value
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     created_value = creator(
)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/s
ite-packages/fmn/rules/utils.py", line 450, in <lambda>
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     creator = lambda: fedmsg
.meta.msg2usernames(msg, **config)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/s
ite-packages/fedmsg/meta/__init__.py", line 152, in __wrapper
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return f(msg, **config)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/s
ite-packages/fedmsg/meta/__init__.py", line 171, in __wrapper
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return f(msg, processor=
processor, **config)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/s
ite-packages/fedmsg/meta/__init__.py", line 295, in msg2usernames
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return processor.usernam
es(msg, **config)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/s
ite-packages/fedmsg_meta_fedora_infrastructure/supybot.py", line 132, in usernames
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     if nick not in blacklisted_people
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/fmn_fasshim.py", line 185, in nick2fas
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     update_nick(nickname)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/fmn_fasshim.py", line 107, in update_nick
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     if config.app_conf.get('fasjson'):
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]: AttributeError: 'module' object has no attribute 'get'
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]: [2021-06-16 18:28:30][   fmn.lib   ERROR] 'module' object has no attribute 'get'
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]: Traceback (most recent call last):
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/lib/__init__.py", line 86, in matches
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     value = fn(config, message, **arguments)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/rules/generic.py", line 21, in user_filter
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return fasnick in fmn.rules.utils.msg2usernames(message, **config)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/rules/utils.py", line 451, in msg2usernames
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return _cache.get_or_create(key, creator)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 833, in get_or_create
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     async_creator) as value:
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/dogpile/lock.py", line 154, in __enter__
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return self._enter()
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/dogpile/lock.py", line 94, in _enter
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     generated = self._enter_create(createdtime)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/dogpile/lock.py", line 145, in _enter_create
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     created = self.creator()
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 800, in gen_value
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     created_value = creator()
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/rules/utils.py", line 450, in <lambda>
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     creator = lambda: fedmsg.meta.msg2usernames(msg, **config)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fedmsg/meta/__init__.py", line 152, in __wrapper
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return f(msg, **config)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fedmsg/meta/__init__.py", line 171, in __wrapper
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return f(msg, processor=processor, **config)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fedmsg/meta/__init__.py", line 295, in msg2usernames
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     return processor.usernames(msg, **config)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fedmsg_meta_fedora_infrastructure/supybot.py", line 132, in usernames
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     if nick not in blacklisted_people
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/fmn_fasshim.py", line 185, in nick2fas
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     update_nick(nickname)
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:   File "/usr/lib/python2.7/site-packages/fmn/fmn_fasshim.py", line 107, in update_nick
Jun 16 18:28:30 notifs-backend01.iad2.fedoraproject.org celery[9997]:     if config.app_conf.get('fasjson'):

@ryanlerch let me know if you can't login to notifs-backend01 and I can get you setup to do so so you can look at logs... and you should be able to run the playbook as well.

So, I got it to start sending again, and on some meetings its working! But on others it's not.

I think it's now working on meetings where the person starting the meeting is not in the account system.

Actually it's working on all of them now. Might have just been older ones that were queued when the fix was landed. :)

Do let us know if you see anything further.

many thanks to @ryanlerch for the fix!

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
3 years ago

Thank you so much for fixing this @kevin and @ryanlerch !

Log in to comment on this ticket.

Metadata

dev low-gain medium-trouble

None
None
Waiting on Assignee
Boards 1
dev Status: Done
Powered by Pagure 5.14.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.