Questions from the meeting. Could this be done in staging first?

Sure thing, though iirc we don't have it in staging as otherwise we would end up sending notifications for even on the staging bus.

I'm fine either way tbh

I could take on this task if it is not urgent. It will take me a few days to sort all things out since these will be my first VMs built.

@bodanel you still want to take this on? Basically look at where in inventory/ and inventory/host_vars/ we have notifs-web01 and notifs-backend01 defined. Copy that for 02.

For ip addresses:

notifs-backend02 IN A 10.3.163.107
notifs-web02 IN A 10.3.163.108

Once you have a PR for that I can merge it and run it (or @pingou can)

Yep, I'll do it later today/tomorrow.

ok. I took the liberty of doing these...

they are both up and installed.

notifs-web02 -> rhel8
notifs-backend02 -> fedora34

notifs-backend02 playbook gets to installing fedmsg-hub and fails (we never built fedmsg-hub for newer fedora, could switch to fedmsg-hub-3 / python3 version tho)
notifs-web02 playbook gets to installing fedmsg and fails (same as above).

Let me know if you need anything more on them.

So, I'll rebuild notifs-web02 as F34 as we apparently did not branch fedmsg in epel8.

Checking backend02 first though

Alright, so the ansible playbooks are finishing for both backend02 and web02.

Services on backend02 almost all fail to start with the error:

   File "/usr/lib/python3.9/site-packages/fmn/tasks.py", line 41, in <module>
     from . import fmn_fasshim
   File "/usr/lib/python3.9/site-packages/fmn/fmn_fasshim.py", line 30, in <module>
     client = Client(url=fasjson.get('url', default_url))
   File "/usr/lib/python3.9/site-packages/fmn/fasjson_client.py", line 26, in __init__
     gssapi_auth = HTTPSPNEGOAuth(opportunistic_auth=True, creds=creds)
 UnboundLocalError: local variable 'creds' referenced before assignment

@scoady maybe you have an idea of what's going on here? I wonder if this is related to keytab since I don't see one being created in the playbook.

On web02, apache starts but it needs some more testing before we can confirm it works.

I tested web02 a little more using port-forwarding.

I could log in fine, toggle settings on and off, I even received the confirmation email to confirm my email address (that didn't work on IRC though, I suppose because backend02 isn't working properly).

So from a basic testing it seems to work ok.

I've pulled the different fixes I've had to do server side (be that on backend02 or web02) in this PR: https://github.com/fedora-infra/fmn/pull/320 (together with the spec file)

So, I guess I can close this as we have made the instances? Please feel free to re-open if there's further work here to track here.

I was originally hoping to keep that ticket around to track progress on this as I won't have much time to work on it and it's not really functional (yet).

ok, so lets reopen and use this to track moving notifs-backend to python3 and a supported os.

https://github.com/fedora-infra/fmn/blob/develop/fmn/fasjson_client.py

was a stop-gap as FMN was only on python2 -- we should be able to remove it now, and just use fasjson_client

I think @scoady and/or @zlopez were going to look into this more this quarter? With hopefully some time from @pingou to show them around.

Is that still the plan?

I plan to look into FMN this quarter, but didn't have time yet.

Any news here? ;)

I had some time to look at the FMN, but didn't got to this ticket yet.

I created a PR for removing fasjson_client.py and replacing it with fasjson-client library https://github.com/fedora-infra/fmn/pull/332

I tested it briefly in vagrant and all the tox tests were passing. I also tried to login in the FMN and it worked.

I would be glad if somebody could guide me through deployment process for staging or at least point me to a place, where this is documented.

After finally fixing the CI tests and see them passing on https://github.com/fedora-infra/fmn/pull/332 I merged the PR.

Next step would be the deployment in staging. If anybody knows how this is done for FMN, let me know.

So, we currently have no fmn in staging because we couldn't deploy a new one because the os was EOL. ;(

We do have 2 of them in prod:

notifs-web01 / notifs-backend01 -> old f27 based, python2 version (ie, "live" prod)

notifs-web02 / notifs-backend02 -> new f34 based python3 version, but not 'live'

So, options:

1. We could just deploy to the existing 02 prod ones and test as best you can before we switch to it.
2. We could deploy a notifs-backend01.stg/notifs-web01.stg with f35 and the python3 version, test there, then move to prod
3. Something else

I think this could be a fine thing to move to openshift too, but thats probibly too many steps at once? Or does that make it easier?

Thoughts?

ok. I have deployed the python3 version in staging now (on f35).

So, now there's a notifs-web01.stg and a notifs-backend01.stg.

They are using python-fmn-2.2.2-3.fc35 that is built in f35-infra tag (currently in staging only).

notifs-web01 is crashing tho, so the web interface isn't available:

[Thu Mar 03 00:55:45.143752 2022] [wsgi:error] [pid 98384:tid 98543] [remote 10.3.166.74:45646] mod_wsgi (pid=98384): Exception occurred processing WSGI script '/usr/share/fmn/fmn.web.wsgi'.
[Thu Mar 03 00:55:45.143814 2022] [wsgi:error] [pid 98384:tid 98543] [remote 10.3.166.74:45646] Traceback (most recent call last):
[Thu Mar 03 00:55:45.143829 2022] [wsgi:error] [pid 98384:tid 98543] [remote 10.3.166.74:45646]   File "/usr/share/fmn/fmn.web.wsgi", line 17, in <module>
[Thu Mar 03 00:55:45.143832 2022] [wsgi:error] [pid 98384:tid 98543] [remote 10.3.166.74:45646]     from fmn.web.app import app as application
[Thu Mar 03 00:55:45.143835 2022] [wsgi:error] [pid 98384:tid 98543] [remote 10.3.166.74:45646]   File "/usr/lib/python3.10/site-packages/fmn/web/app.py", line 18, in <module>
[Thu Mar 03 00:55:45.143837 2022] [wsgi:error] [pid 98384:tid 98543] [remote 10.3.166.74:45646]     from flask.ext.openid import OpenID
[Thu Mar 03 00:55:45.143845 2022] [wsgi:error] [pid 98384:tid 98543] [remote 10.3.166.74:45646] ModuleNotFoundError: No module named 'flask.ext'

I haven't enabled anything on the backend yet either, but the playbook has run. It doesn't appear to be running any of the fmn services yet though.

What do we want to do about db? There is one... from I don't know when or whats in it. :)

@zlopez I can get you access to those vm's, perms to run the playbook, whatever needed to bring it up. :)

We should make sure not to spam people from this staging version. :)

[Thu Mar 03 00:55:45.143835 2022] [wsgi:error] [pid 98384:tid 98543] [remote 10.3.166.74:45646] File "/usr/lib/python3.10/site-packages/fmn/web/app.py", line 18, in <module> [Thu Mar 03 00:55:45.143837 2022] [wsgi:error] [pid 98384:tid 98543] [remote 10.3.166.74:45646] from flask.ext.openid import OpenID

This is the old way to import flask extensions, it should be replaced simply by: from flask_openid import OpenID

I will try to work on the FMN today and take it to state that it could be started on F35.

@pingou
Probably part of the code not covered by tests, because I didn't saw this in CI.

@kevin I'm not sure what are you running on the VM, but this is already fixed in latest develop.

And I see that there is 2.2.3 already tagged in repo and plenty of fixes happened till then.

We should release a new version. @pingou Do you know how the new version is released for FMN?

Yeah, we are running python-fmn-2.2.2-3.fc35 as noted.

@kevin
python-fmn-2.3.0-1.fc35 is built in f35-infra tag
Let's see if this works.

Problem: cannot install the best update candidate for package python3-fmn-2.2.2-3.fc35.noarch
- nothing provides python3.10dist(datanommer-models) >= 1~a1 needed by python3-fmn-2.3.0-1.fc35.noarch

So, I guess we need a newer python3-datanommer-models?

Hm, I didn't change anything regarding dependencies. So I'm not sure where this came from.

Looking at it, the current version in Fedora is 0.9.1 and there is 1.0.2 already in upstream.
We just need to rebuild the package, @infra-sig has commit access to repository, so somebody from Infra team should be able to do that.

ok. I cleaned up a bunch of things but 1.0.2 doesn't build because it requires:

fedora-messaging<3.0.0,>=2.1.0

and we have 3.0.0 in all fedora branches. ;(

@abompard can you help here? What does python-datanommer-models require any version of fedora-messaging except 3.0.0 (The one we have) ? :)

Oh, that should be fixed indeed. For precaution the dependencies we set in poetry don't span major versions but in this case datanommer is compatible with fedora-messaging 3.x.
I'll update it.

@abompard Did you had time to update the datanommer?

I changed it in the rpm package level... so notifs-backend01.stg should be all set, but I haven't looked to see whats broken after that update.

This is still being troubleshooted. The remaining issue now is that the staging instance doesn't sent any other e-mail than confirmation e-mail.

Unfortunately there is no obvious error that we were able to find and it looks like the incoming messages are being processed, but no e-mail is sent.

The most pressing issues were already solved and the FMN is running in production. There are still some issues with handling the queue, but the ticket could be closed as fixed.