#10952 mini-initiative proposal: skeleton for bridge from fasjson groups to discourse
Closed: Fixed a year ago by dkirwan. Opened 2 years ago by mattdm.

Background

This is a follow-up to https://pagure.io/fedora-infrastructure/issue/9580, inspired by my discovery of the Communishift Authorization Operator, which is very similar to (at least part of) what I want.

As Kevin notes at the end of that ticket, upstream doesn't seem to be going in a helpful-for-our-use-case direction with this, so a little system to keep things in sync seems best to me.

Summary

I want to one-way sync FAS group membership information to Discourse (specifically, https://discussion.fedoraproject.org/). If a user is added to or removed from a group in FAS, and that user has a Discourse account and there is a Discourse group of the same name, the Discourse user should be added to (or removed from) the corresponding Discourse group.

As a bonus, this can solve another problem: if an account is locked, disabled, or deleted from FAS, the Discourse account should also be locked.

Not even pseudocode

Here's what I envision:

  1. A webhook Fedora Message Bus topic about group membership change is received (see below), giving us the username to check
  2. fasjson: validate that the account is active and in good standing
  3. fasjson: get list of groups for the user
  4. discourse api: disable the account if it should be disabled
  5. discourse api: get list of all groups
  6. take intersection of the fas and discourse groups (groups will be manually created on Discussion, so only groups that exist will be synced)
  7. discourse api: get list of groups user belongs to
  8. discourse api: remove the user from groups that are in 7 (discourse) but not 6 (fas)
  9. discourse api: add the user to any group that is in 6 (fas) but not 7 (discourse)

But wait! Before you think "that's too many steps for a mini-initiative!" —

I can do the Discourse part!

I am familiar with the Discourse API. But I'm not with fasjson. It doesn't look complicated, but then I realized there's a whole bunch of yak-shaving to do around getting a keytab and setting it up so that's available and authenticated and running in infrastructure. And it looks like you have already shaved that yak! — that is, all of these things are very close to what Communishift Authorization Operator already does.

So,

What would really help me:

  1. Steps 1-3 running somewhere -- maybe as a toddler. That should pass either "this account should be disabled" or the resulting grouplist to....
  2. A stub that does nothing. (This will be quicker for me with python, but I can do this in whatever language you prefer.)
  3. A place to put a secret available to that stub (an API key — I can either provide the key, or you can provide a way for me to add it securely)
  4. Something where when I push to a git repo on a deploy branch (or whatever), that commit gets deployed, replacing the stub (or my previous version).
  5. And access to logs would be a nice bonus

Because most of this is what Communishift Authorization Operator already does, I really hope that this will not be a lot of effort.

Triggering

I had originally imagined this would trigger on webhooks from discourse. We could still do that, but I think it makes more sense to listen to Fedora Message Bus messages for group changes (https://fedmsg2.readthedocs.io/en/latest/topics.html#fas).

However, there is something I don't understand -- there is fas.group.member.remove, but no fas.group.member.add. (There are .sponsor and .approve, though.) Hopefully you know what's going on here.

We'd also want to trigger on the account being deactivated or disabled — fas.user.update, I guess.

These could trigger different paths in my part of the code (remove, add, disable) but that has more risk of getting out of sync. I think it's better to just send user and grouplist and be done.

In case a message gets lost or out of sync, there probably should also be some occasional out-of-band double-check.

The connection to the discourse part

This could all live in the same codebase (maybe all running as one toddler), or the first part (the fas listener) could make a remote call of some sort -- a JSON http POST with the username and a list of groups. (And some auth token!). Maybe something running in OpenShift? I can work with whatever you think makes sense.

The actual code should be pretty simple, and I'll make sure to comment it well so that anyone else can, in the future, fix any issues or make updates.

Okay, but, why?

We're finally getting Enterprise hosting, which will let us combine the Ask and Discussion instances. Having group information will allow us to link permissions for posting in certain areas to group membership, allow teams to polls of just their team, allow teams to get notifications, and so much more. We could also make it so Fedora contributors in certain groups get automatically put to "not a newbie" level (which avoids some frustrations with limits on image uploads, etc.). This will be important as more teams move from mailing lists.


Note: I've edited this a bit since first filed!


Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: dev, high-gain, high-trouble, mini-initiative

2 years ago

Hey, so, I've tried to make this as not-high-trouble-at-all as possible:

  1. basically a duplicate of something just implemented
  2. can fit into the existing "toddlers" framework rather than being some new service
  3. clearly spec'ing out whta i
  4. leave the part that is out of existing Fedora Account system / fasjson expertise to me"

Is there anything else I can do for y'all, or do differently, to possibly get this down to "medium-trouble" at least? What are the next steps?

This is looking very similar to what we did with the communishiftauthorization operator.

@mattdm would you be able to provide an API key on the Discourse side, that has enough CRUD permissions for users and groups that we can start testing against?

Just confirming here that you are working on this now @dkirwan ? :)

This is being worked on by @dkirwan and @lenkaseg.

Metadata Update from @zlopez:
- Issue assigned to dkirwan

a year ago

Hi all, we are pretty close to being feature complete, we do need to test this operator running in staging with something resembling a production load.

We want to add the staging discourse to staging ipa/noggin/ipsilon and while we have the required information to update the openidc.staging.static in the ansible-private repo, it looks like we are missing a plugin on the Discourse side to handle oauth2. If possible, can @lenkaseg and I @dkirwan be added as administrators on the production Discourse instance? We'd like to examine the configuration on Discourse side, and figure out which plugins might be required to be installed on the staging side.

ping @mattdm

Hi all,

We've installed the latest version of fas2discourse operator on our staging OpenShift and configured it to point at the staging Discourse instance:
https://fedoraproject.staged-by-discourse.com.

This instance is now hooked into staging IPA/Noggin and Ipsilon. So community members can login and create their user accounts on this staging Discourse.

Think we are ready to do some proper testing now. Start making the groups that we want, and invite users in these groups to login and see if they are correctly given access.

Sync time set is currently set to run every 20 minutes, this can be lowered to 5 minutes or maybe even 2 minutes but we run the risk of hitting Discourse rate limiting if we are making large updates and go any lower.

https://pagure.io/infra-docs-fpo/pull-request/203# PR for the fas2discourse operator SOPs, reviewing in team before merging.

That was the final step btw, the operator appears to be working as expected in staging!

The groups in staging now match what is in production, and what is in IPA/Noggin/FAS

Hoping to deploy to production soon, with a version that is running with functionality for adding/removing users from groups disabled, but logging proposed changes will debug for a few days before enabling full functionality and calling this ticket complete.

We've pushed this into production this morning. Just to be aware, FAS/IPA/Noggin is the single source of truth, so please make sure that we add users to groups there, rather than in the Discourse side.

Closing as complete, to raise bugs against this service please open a ticket at: https://pagure.io/cpe/fas2discourse/issues

Metadata Update from @dkirwan:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a year ago

Login to comment on this ticket.

Metadata
Boards 2
dev Status: Backlog
mini-initative Status: Backlog