We should track and finish moving everything from fedmsg to fedora-messaging.
We have at least:
releng/scripts/fedmsg-functions.sh (used by branched/rawhide composes) roles/nagios_server/files/nagios/commands/notify.cfg (used by nagios)
and all the following playbooks still calling fedmsg/base:
playbooks/groups/autocloud-backend.yml: - fedmsg/base playbooks/groups/autocloud-web.yml: - fedmsg/base playbooks/groups/badges-backend.yml: - fedmsg/base playbooks/groups/badges-web.yml: - fedmsg/base playbooks/groups/bugyou.yml: - fedmsg/base playbooks/groups/bugzilla2fedmsg.yml: - fedmsg/base playbooks/groups/busgateway.yml: - fedmsg/base playbooks/groups/datagrepper.yml: - fedmsg/base playbooks/groups/elections.yml: - fedmsg/base playbooks/groups/fas.yml: - fedmsg/base playbooks/groups/fedimg.yml: - fedmsg/base playbooks/groups/fedocal.yml: - fedmsg/base playbooks/groups/freshmaker.yml: - fedmsg/base playbooks/groups/github2fedmsg.yml: - fedmsg/base playbooks/groups/hubs.yml: - fedmsg/base playbooks/groups/kerneltest.yml: - fedmsg/base playbooks/groups/koji-hub.yml: - fedmsg/base playbooks/groups/koschei-backend.yml: - fedmsg/base playbooks/groups/loopabull.yml: - fedmsg/base playbooks/groups/mailman.yml: - fedmsg/base playbooks/groups/mbs.yml: - fedmsg/base playbooks/groups/mdapi.yml: - fedmsg/base playbooks/groups/mirrormanager.yml: - fedmsg/base playbooks/groups/modernpaste.yml: - fedmsg/base playbooks/groups/noc.yml: - fedmsg/base playbooks/groups/notifs-backend.yml: - fedmsg/base playbooks/groups/notifs-web.yml: - fedmsg/base playbooks/groups/nuancier.yml: - fedmsg/base playbooks/groups/odcs.yml: - fedmsg/base playbooks/groups/odcs.yml: - role: fedmsg/base playbooks/groups/openqa.yml: - { role: fedmsg/base, tags: ['fedmsg_base', 'fedmsg'] } playbooks/groups/packages.yml: - fedmsg/base playbooks/groups/pdc.yml: - fedmsg/base playbooks/groups/pdc.yml: - fedmsg/base playbooks/groups/people.yml: - fedmsg/base playbooks/groups/pkgs.yml: - fedmsg/base playbooks/groups/releng-compose.yml: - fedmsg/base playbooks/groups/resultsdb.yml: - { role: fedmsg/base, playbooks/groups/retrace.yml:# - fedmsg/base playbooks/groups/sundries.yml: - role: fedmsg/base playbooks/groups/taskotron.yml: - { role: fedmsg/base } playbooks/groups/value.yml: - fedmsg/base playbooks/groups/wiki.yml: - fedmsg/base playbooks/groups/zanata2fedmsg.yml: - fedmsg/base playbooks/hosts/happinesspackets-stg.fedorainfracloud.org.yml: - fedmsg/base playbooks/hosts/happinesspackets.fedorainfracloud.org.yml: - fedmsg/base
Koschei migration from fedmsg to fedora-messaging in production is planned on 2019-09-26. Staging instance already uses fedora-messaging, eg. this message ("username": "amqp-bridge")
"username": "amqp-bridge"
@karsten you were going to look at the infrastructure scripts moving right? Could you perhaps focus on/start with planet? It seems to somehow be broken right now, and I can't seem to figure out why. Moving it to fedora-messaging would hopefully fix it and get us a bit further along too at the same time. :)
It runs on people02.fedoraproject.org... the venus-planet package will need adjusting I guess: https://koji.fedoraproject.org/koji/buildinfo?buildID=942792
Or if thats not your cup o tea, we can look for someone else. :)
Also, we need a plan for notifs.
playbooks/groups/resultsdb.yml: - { role: fedmsg/base,
I thought that we already ported this to fedora-messaging.
playbooks/groups/taskotron.yml: - { role: fedmsg/base }
As taskotron is in maintenance mode, I'm not really keen on putting time into porting the one sub-component which still uses fedmsg to fedora-messaging unless there's no other option
playbooks/groups/resultsdb.yml: - { role: fedmsg/base, I thought that we already ported this to fedora-messaging.
We did, this is likely just a clean up that we didn't do :(
Same for elections, koji-hub and pkgs I think.
Also Ansible callback plugins need to be ported:
callback_plugins/fedmsg_callback.py callback_plugins/fedmsg_callback2.py
Metadata Update from @cverna: - Issue tagged with: backlog
Metadata Update from @kevin: - Issue assigned to karsten
I'm dropping the following from the list:
autocloud retires with f29:
playbooks/groups/autocloud-backend.yml: - fedmsg/base playbooks/groups/autocloud-web.yml: - fedmsg/base
we are giving away badges:
playbooks/groups/badges-backend.yml: - fedmsg/base playbooks/groups/badges-web.yml: - fedmsg/base
bugyou never took off/should be retired now.
playbooks/groups/bugyou.yml: - fedmsg/base
elections we are giving away:
playbooks/groups/elections.yml: - fedmsg/base
fedocal we are giving away:
playbooks/groups/fedocal.yml: - fedmsg/base
hubs is gone.
playbooks/groups/hubs.yml: - fedmsg/base
going away:
playbooks/groups/modernpaste.yml: - fedmsg/base playbooks/groups/nuancier.yml: - fedmsg/base
done: playbooks/groups/koji-hub.yml: - fedmsg/base playbooks/groups/pkgs.yml: - fedmsg/base playbooks/groups/koschei-backend.yml: - fedmsg/base playbooks/groups/openqa.yml: - { role: fedmsg/base, tags: ['fedmsg_base', 'fedmsg'] } playbooks/groups/resultsdb.yml: - { role: fedmsg/base,
Unknown????????????
playbooks/groups/fedimg.yml: - fedmsg/base playbooks/groups/freshmaker.yml: - fedmsg/base
That leaves:
playbooks/groups/bugzilla2fedmsg.yml: - fedmsg/base playbooks/groups/busgateway.yml: - fedmsg/base playbooks/groups/datagrepper.yml: - fedmsg/base playbooks/groups/fas.yml: - fedmsg/base playbooks/groups/github2fedmsg.yml: - fedmsg/base playbooks/groups/kerneltest.yml: - fedmsg/base playbooks/groups/loopabull.yml: - fedmsg/base playbooks/groups/mailman.yml: - fedmsg/base playbooks/groups/mbs.yml: - fedmsg/base playbooks/groups/mdapi.yml: - fedmsg/base playbooks/groups/mirrormanager.yml: - fedmsg/base playbooks/groups/noc.yml: - fedmsg/base playbooks/groups/notifs-backend.yml: - fedmsg/base playbooks/groups/notifs-web.yml: - fedmsg/base playbooks/groups/odcs.yml: - fedmsg/base playbooks/groups/odcs.yml: - role: fedmsg/base playbooks/groups/packages.yml: - fedmsg/base playbooks/groups/pdc.yml: - fedmsg/base playbooks/groups/pdc.yml: - fedmsg/base playbooks/groups/people.yml: - fedmsg/base playbooks/groups/releng-compose.yml: - fedmsg/base playbooks/groups/retrace.yml:# - fedmsg/base playbooks/groups/sundries.yml: - role: fedmsg/base playbooks/groups/taskotron.yml: - { role: fedmsg/base } playbooks/groups/value.yml: - fedmsg/base playbooks/groups/wiki.yml: - fedmsg/base playbooks/groups/zanata2fedmsg.yml: - fedmsg/base ansible callbacks
playbooks/groups/bugzilla2fedmsg.yml: - fedmsg/base
Being ported by @abompard
playbooks/groups/fas.yml: - fedmsg/base
going away (sooner or later :()
playbooks/groups/kerneltest.yml: - fedmsg/base
Being handed over to Jeremy
playbooks/groups/loopabull.yml: - fedmsg/base
Ported to fedora-messaging upstream, we can migrate when we want
playbooks/groups/packages.yml
I think @cverna was saying yesterday that this no longer works, and thus we should be able to remove it.
Is there any sense of what should be prioritized higher in this list? Just curious for some expert opinion @kevin @pingou
There are a few angles and I don't think any good answer to this question:
Some that we could look at porting today would be:
Once these are done we'll likely ran into the more complex ones.
There are a few angles and I don't think any good answer to this question: We want to retire fedmsg so all of these should be ported to fedora-messaging We will need to migrate datagrepper, busgateway and FMN from fedmsg to fedora-messaging but both of them require the features that today fedmsg_meta_fedora_infrastructure provides and these features are now handled as part of the message schemas that each application should provide To make the migration from fedmsg faster, we have not added support to message schemas to the application we've migrated, so we'll need to go back to a number of our applications to add these schemas. Some that we could look at porting today would be: wiki sundries (not quite sure what uses fedmsg there though)
We want to retire fedmsg so all of these should be ported to fedora-messaging We will need to migrate datagrepper, busgateway and FMN from fedmsg to fedora-messaging but both of them require the features that today fedmsg_meta_fedora_infrastructure provides and these features are now handled as part of the message schemas that each application should provide To make the migration from fedmsg faster, we have not added support to message schemas to the application we've migrated, so we'll need to go back to a number of our applications to add these schemas.
wiki sundries (not quite sure what uses fedmsg there though)
I am not sure here either. It's using fedmsg/base, but might be only for the user id or something.
retrace odcs mbs people (here as well, not sure what uses fedmsg there)
It's fedoraplanet. It emits a fedmsg on blog posts.
releng-compose noc Once these are done we'll likely ran into the more complex ones.
releng-compose noc
We should consider doing some design work on the replacement for FMN and datagrepper, since I think they will take the longest because we have to design new ones that avoid the mistakes of the current ones.
There's been some work going on by @jcline for a FMN replacement, I'm going to pick it up. About Datagrepper, my guess is that we could port it pretty easily by just changing the listener and ignoring message headers at first, but if there are design issues with the current one that we want to iron out, then yeah it's the right moment to think about a new design. Is there a document or a page that list the current issues with datanommer / datagrepper?
@karsten you were going to look at the infrastructure scripts moving right? Could you perhaps focus on/start with planet? It seems to somehow be broken right now, and I can't seem to figure out why. Moving it to fedora-messaging would hopefully fix it and get us a bit further along too at the same time. :) It runs on people02.fedoraproject.org... the venus-planet package will need adjusting I guess: https://koji.fedoraproject.org/koji/buildinfo?buildID=942792 Or if thats not your cup o tea, we can look for someone else. :)
Just a note here that planet is in fact sending fedmsgs again. I have no idea what happened to fix it, but perhaps it's not the highest priority now. (I thought I noted this here, but I guess not, many appologies).
Also, if planet is proving hard we could revisit the software we use for that (if there's any better ones now) or some other way forward.
Not that I know of, but there are some that I think we could/should solve:
After learning about them for koji, I think we might want to look at partitioning the main messages table. Split it out per month or something, then make sure queries default to just the current partition unless you specifically want to query older ones. This would make both inserts and queries a lot faster.
Additionally, we may want to talk to internal RH folks about this as I think they running a datanommer/datagrepper set and might be interested in our changes and wanting to move to them.
So, nothing super radical, but good to address.
I haven't found any retrace messages in datagrepper, either I'm using the wrong category or retrace doesn't emit anything.
So, retrace server is kind of an umbrella for 2 related projects:
I am not sure who the current developers are on faf/abrt/retrace. @msuchy is the admin contact. @msuchy who should we ask about porting this app to fedora-messaging?
I think perhaps the compose scripts might be highest priority here. Those are run from pungi-fedora and the nightly.sh script.
Why is playbooks/groups/busgateway.yml on this list ? Afaik thats for setting up the fedmsg relay and will get deleted once everything got moved to fedora-messaging / rabbitmq ? Is there anything else that needs to be done here ?
Yeah, you are right, busgateway just goes away when fedmsg does. I got the initial list by looking at what playbooks called any fedmsg role in ansible...
I've posted patches to fedora-infra, waiting to get them merged before this can be closed
I think this got merged, right ? I don't have ACL to close this
Yeah, thanks for the patches!
Did we get all the applications? Or are there still some to move?
Metadata Update from @cverna: - Assignee reset
Metadata Update from @pingou: - Issue tagged with: dev
We've set this board: https://github.com/orgs/fedora-infra/projects/7 to track progress on adding fedora-messaging message schemas to apps.
I'm looking forward the day we'll do some serious clean up on our ansible repo :)
I've distilled information from this ticket and packages we have in Fedora (including -infra tags) into a summary doc. I'm not sure I've caught everything, though :wink:. Many of the items I found also need to be triaged, i.e. whether an item is ours to take care of, someone else's or should go away.
@kevin @pingou @abompard, would you please give it a quick look over, correct mistakes you find and fill in info you have (e.g. move items into their correct places)? Thanks!
Thanks for doing that!
I moved freshmaker, zanata2fedmsg and packages to dead... packages we need to figure out what we are replacing it with, the others are already gone.
kerneltest we moved the old one as the new re-write one needs some more work before its ready. We will want to talk to justin about that.
@nphilipp I've looked at it and moved some things around but I'm a little confused on the difference between that doc and the board at: https://github.com/orgs/fedora-infra/projects/7 which is ordered more or less by priority and includes some of the items you've listed there (though it also misses some items that you've listed in hackmd I think).
@pingou as I understand it (@abompard correct me if I'm wrong), the board tracks the "has fedora-messaging, needs schemas" items, i.e. it was missing at least some items that need to be ported from fedmsg to fedora-messaging.
I started the document initially to compile the list of items that still need porting, then thought that when I'm touching each one of these, I can put in all the info I find, too. I.e. it should describe the initial status of hopefully all items we need to look at, so we can say e.g. "this item needs to be ported and schemas added" or "the schemas of this item need to be completed". it's not supposed to track work like the board, but should help us verify that we track all the items we should in it. Does that make sense?
Mind, I'm not sure if we should extend the scope of the board to cover porting work, too, or add a separate one for that.
Metadata Update from @smooge: - Issue tagged with: medium-gain, medium-trouble, rabbitmq
Here's whats left today:
badges-backend.yml: - fedmsg/base badges-web.yml: - fedmsg/base bugzilla2fedmsg.yml: - fedmsg/base busgateway.yml: - fedmsg/base datagrepper.yml: - fedmsg/base fedimg.yml: - fedmsg/base fedocal.yml: - fedmsg/base github2fedmsg.yml: - fedmsg/base kerneltest.yml: - fedmsg/base mailman.yml: - role: fedmsg/base mbs.yml: - fedmsg/base mirrormanager.yml: - role: fedmsg/base notifs-backend.yml: - fedmsg/base notifs-web.yml: - role: fedmsg/base nuancier.yml: - fedmsg/base pdc.yml: - role: fedmsg/base resultsdb.yml: - fedmsg/base sundries.yml: - role: fedmsg/base value.yml: - fedmsg/base wiki.yml: - fedmsg/base
fedocal moved to openshift and fedora-messaging. I guess it still shows because https://pagure.io/fedora-infra/ansible/pull-request/530 didn't get in yet
Current list:
playbooks/groups/badges-backend.yml: - fedmsg/base playbooks/groups/badges-web.yml: - fedmsg/base playbooks/groups/busgateway.yml: - fedmsg/base playbooks/groups/datagrepper.yml: - fedmsg/base playbooks/groups/fedimg.yml: - fedmsg/base playbooks/groups/github2fedmsg.yml: - fedmsg/base playbooks/groups/kerneltest.yml: - fedmsg/base playbooks/groups/mailman.yml: - role: fedmsg/base playbooks/groups/mbs.yml: - fedmsg/base playbooks/groups/mirrormanager.yml: - role: fedmsg/base playbooks/groups/notifs-backend.yml: - fedmsg/base playbooks/groups/notifs-web.yml: - role: fedmsg/base playbooks/groups/nuancier.yml: - fedmsg/base playbooks/groups/pdc.yml: - role: fedmsg/base playbooks/groups/resultsdb.yml: - fedmsg/base playbooks/groups/sundries.yml: - role: fedmsg/base playbooks/groups/value.yml: - { role: fedmsg/base, playbooks/groups/wiki.yml: - fedmsg/base
playbooks/groups/badges-backend.yml: - fedmsg/base playbooks/groups/badges-web.yml: - fedmsg/base playbooks/groups/busgateway.yml: - fedmsg/base playbooks/groups/datagrepper.yml: - fedmsg/base playbooks/groups/fedimg.yml: - fedmsg/base playbooks/groups/github2fedmsg.yml: - fedmsg/base playbooks/groups/kerneltest.yml: - fedmsg/base playbooks/groups/mailman.yml: - role: fedmsg/base playbooks/groups/mbs.yml: - fedmsg/base playbooks/groups/mirrormanager.yml: - role: fedmsg/base playbooks/groups/notifs-backend.yml: - fedmsg/base playbooks/groups/notifs-web.yml: - role: fedmsg/base playbooks/groups/nuancier.yml: - fedmsg/base playbooks/groups/pdc.yml: - role: fedmsg/base playbooks/groups/sundries.yml: - role: fedmsg/base playbooks/groups/value.yml: - { role: fedmsg/base, playbooks/groups/wiki.yml: - fedmsg/base
[backlog_refinement] We need to check with @abompard about datagrepper, which should be migrated to openshift. Confirmed on the meeting, datagrepper is done.
playbooks/groups/badges-backend.yml: - fedmsg/base playbooks/groups/badges-web.yml: - fedmsg/base playbooks/groups/busgateway.yml: - fedmsg/base playbooks/groups/fedimg.yml: - fedmsg/base playbooks/groups/github2fedmsg.yml: - fedmsg/base playbooks/groups/kerneltest.yml: - fedmsg/base playbooks/groups/mailman.yml: - role: fedmsg/base playbooks/groups/mbs.yml: - fedmsg/base playbooks/groups/mirrormanager.yml: - role: fedmsg/base playbooks/groups/notifs-backend.yml: - fedmsg/base playbooks/groups/notifs-web.yml: - role: fedmsg/base playbooks/groups/nuancier.yml: - fedmsg/base playbooks/groups/pdc.yml: - role: fedmsg/base playbooks/groups/sundries.yml: - role: fedmsg/base playbooks/groups/value.yml: - { role: fedmsg/base, playbooks/groups/wiki.yml: - fedmsg/base
[backlog_refinement] No update on this ticket
[backlog refinement] FMN is currently being rewritten
[backlog refinement] There is no progress on this ticket, but the fedmsg is being work on to be ported to EPEL8. So at least we don't need to run it on the soon unsuported OS.
[backlog_refinement] With the new FMN with fedora messaging available, we can remove it from the list.
playbooks/groups/badges-backend.yml: - fedmsg/base playbooks/groups/badges-web.yml: - fedmsg/base playbooks/groups/busgateway.yml: - fedmsg/base playbooks/groups/fedimg.yml: - fedmsg/base playbooks/groups/github2fedmsg.yml: - fedmsg/base playbooks/groups/kerneltest.yml: - fedmsg/base playbooks/groups/mailman.yml: - role: fedmsg/base playbooks/groups/mbs.yml: - fedmsg/base playbooks/groups/mirrormanager.yml: - role: fedmsg/base playbooks/groups/nuancier.yml: - fedmsg/base playbooks/groups/pdc.yml: - role: fedmsg/base playbooks/groups/sundries.yml: - role: fedmsg/base playbooks/groups/value.yml: - { role: fedmsg/base, playbooks/groups/wiki.yml: - fedmsg/base
[backlog refinement] The list is still same, there are few services we would like to replace or get rid off + there is some work being done on badges.
I've converted the Mediawiki emitter to fedora-messaging. One less on the list! :-)
I've looked at the fedmsg configuration for sundries in ansible and on the host, and I don't think it's used anymore. There are traces of a test IRC bot but that's old and unused. I'll remove the fedmsg conf from there.
sundries
The value01 host is where the fedmsg IRC gateway runs, it's a bot that sends all incoming messages to the #fedora-fedmsg channel. Do we want to keep that running? I would be in favor of shutting it down, if one wants to see all incoming messages one can run a proper consumer instead of an IRC client, and if one wants historical data one can use datagrepper.
value01
#fedora-fedmsg
I see that there's also a second gateway on value01 that sends some messages to a #fedora-releng channel. It's filtered by topic and message body (regexps). This makes more sense to me, even though the regexps are pretty outdated. Are the releng folks using this? That said, if the releng folks want to have IRC notifications on compose events, FMN can do that.
#fedora-releng
I'm for shutting them down.
So, I think we already talked about this a while back. ;) (I think when new FMN was being developed).
I'm fine retiring the fedmsg-irc. I liked it because I could have my irc client in #fedora-fedmsg and log it and then it made grepping events very easy. :) But I can figure something else out...
The releng one I think @humaton really liked and I think it's kind of nice for some things too, like knowing when a rawhide compose finishes (if failed or not) or that updates went out correctly. I suppose if we really want this we could make a small consumer that does what we want?
So, I'm ok retiring it, but would be good to get a ack from @humaton
So, here's an update status, if I'm not mistaken:
I think it might be nice to have some configurable way to notify things on matrix for groups, but that I think requires some scoping and brainstorming and deciding things, and we could do that later, so I am ok with just dropping it nowish.
Yeah, we could improve FMN to let group sponsors defined rules for their group, that would send notifications to the group's list or IRC channel, as we do for users. We may need to allow for additional channels if they want to send the flood somewhere else. We may need specific rules for groups, because user rules may not be relevant. This is a good bit of work, but not undoable, and I think it's better and more accessible to do it in FMN than in toddlers actually (which would pretty much end up being "for sysadmins only")
That could work. One thing the existing fedmsg-irc does though is allowing for keywords anywhere in the message to be caught. But perhaps that would be a feature we could just add to fmn? Would be pretty cpu intensive tho. ie (I want any message with 'Xfce' in it or any message with 'releng' in it).
Nuancier is now decommissioned.
Metadata Update from @zlopez: - Issue untagged with: medium-gain - Issue assigned to zlopez - Issue tagged with: high-gain
Updated status as of today:
kerneltest.yml is done
Login to comment on this ticket.