Hello.

Some background. We have a single AWS community account. It's used for fedora-infrastructure, centos, copr, fedora-ci, and many other groups. So, we do not provide access to the entire account, instead we create a role and grant only those permissions needed to each role. We also use tagging to prevent one role from doing anything with the resources of another one.
The only OpenShift cluster that Fedora infrastructure is running in aws is our communishift instance. It's for incubating community/new things before they are ready for production.

We do run two other clusters: a staging one and a production one, but we run those on our own physical hardware, not in aws. Those are all managed from our public ansible repo.

I think we would like from our side more information, like what sort of resources and access you need, etc.

Thanks for reaching out. :)

I think we would like from our side more information, like what sort of resources and access you need, etc.

Currently we only have an internal Red Hat deployment. It's done on virtual machines running on physical hardware inside Red Hat network. However, we would like to move to Openshift for our public deployments. Our current use case has following requirements:

• Our core deployment would be done on an Openshift 4. It does not matter if the Openshift 4 cluster is on AWS or physical hardware, as long as the cluster is maintained by the Fedora infrastructure team and is accessible publicly.
• We would need to dynamically spin up virtual machines to run workers. We need to run workers in virtual machines because the csmock tool internally uses mock and it requires root privileges. So, this can not be done inside Openshift containers.

Our use case is very similar to the Copr team and we would like have similar resources and support as the fedora copr deployment. The only difference is that the Copr team uses only virtual machines for their deployments, and the OpenScanHub team would like to use both Openshift and virtual machines.

This case also seems very similar to Fedora CoreOS folks (They use our openshift cluster for their build pipeline, and do actual builds on vm's.)
The difference is that they don't dynamically provision them (I don't think). They have some builders and just use those for build jobs (some in AWS and some in our info for arches aws doesn't provide).

A few more questions:

When you say "Integrate with Fedora project" do you mean just provide the service? Or is the intent to run over all or some builds in fedora as they happen? Whats the short term/long term here?

Do you need dynamically provisioned vm's? Or would some static ones do? Could they be in AWS / only reachable via ssh? Or do they need more access than that?

Perhaps we should look at scheduling a short meeting to discuss and also bring in our Product owner/management chair to prioritize things. :)

Thanks for the info so far, hopefully we can help make this all happen.

When you say "Integrate with Fedora project" do you mean just provide the service? Or is the intent to run over all or some builds in fedora as they happen? Whats the short term/long term here?

All source packages that are shipped in the Fedora project should be scanned through OpenScanHub. The short term goal is to at least build a prototype of public deployment that we can demonstrate, and the long term goal is to scale the scans to all Fedora packages.

Do you need dynamically provisioned vm's? Or would some static ones do? Could they be in AWS / only reachable via ssh? Or do they need more access than that?

OpenScanHub depends on the kobo[1] package and it currently does not support dynamic provisioning of VMs. In the short term, having fixed number of VMs (workers) is acceptable for us, but our long term goal is to have dynamically provisioned VMs in the cloud. I believe it would be fine to have just ssh access to them.

Perhaps we should look at scheduling a short meeting to discuss and also bring in our Product owner/management chair to prioritize things. :)

That's a good idea. I will follow up on it through e-mail. Thanks!

[1] https://github.com/release-engineering/kobo

I am CC'ing @kdudka on this ticket as he maintains internal OpenScanHub deployments and has better understanding of the requirements.

@kevin Would I still have to wait for it? Or this can be resolved now.

CC @smilner

So, I wasn't at the last meeting, but my understanding from Steve is that you want to stand up a proof of concept.

So, for the control part you have communishift (and let us know if you need more resources), and for the worker part you need a VM.

Can you let me know what stats you need for the vm and I can stand it up in aws and hand it off to you. What OS/Size/type/etc?

Will that be sufficent to get you going?

Can you let me know what stats you need for the vm and I can stand it up in aws and hand it off to you. What OS/Size/type/etc?

OS - CentOS Stream 9
RAM - 64 GB
CPU Cores - 16

It may look higher than average but running analyzers takes a lot of resources.

Just to be clear, I am going to build a prototype for the Fedora leadership team and we would be blocked on dynamic provisioning of workers before can do an implementation for the community.

Sorry for the slow reply here. I was out last thur/friday and swamped the rest of the week. ;(

Do you have a preferred region? Or doesn't matter?

How about disk space?

Looking and I see several 16 core instances, but they have 32gb memory, is that ok? or do you need more memory?

Understood on the poc... using copr resalloc is a great idea!

Kevin,

I discussed this issue with @praiskup and he suggested us to not use static virtual machines due to security reasons. We should follow the same pattern as the Copr team is using, that is, a virtual machine should not be reused to do more than one build. It should be destroyed once the build completes and a new one should come up to replace it.

ok, that makes sense.

Do you have a preferred region? Or doesn't matter?

I mostly do my experiments in us-west-2 region, but it does not matter for experimental purpose.

How about disk space?

Compiling small packages takes less disk space, but for more resource consuming packages like kernel, we would need 100+ GB disk space.

Looking and I see several 16 core instances, but they have 32gb memory, is that ok? or do you need more memory?

They should be fine for experimental purpose.

My main concern is that I do not need static virtual machines. I need the ability to dynamically spin virtual machines on demand, as mentioned in my previous comment.

32 GiB RAM should be sufficient for most of the scans if we run one scan at a time on each worker.

I have already created a public prototype of this service and shared it with the leadership team. I hope that would help in getting approval on this ticket.

I've setup a large part of this, but a bit is left:

• Need to merge https://pagure.io/fedora-infra/ansible/pull-request/1614 which needs a freeze break or to wait until after f39 is out the door.

and

• I need to get you a cli token. Whats the best way to get that to you? gpg email, bitwarden send?

Got gpg key info out of band, will try and generate a user/token soon.

For deployment to openshift see: https://docs.fedoraproject.org/en-US/infra/developer_guide/openshift/#_openshift which is kind of bare bones, but basically the way we deploy things is to have a playbook in ansible and that in turn uses a role for the application with templates and files. The playbook copies the templates and files to a control host and then uses 'oc apply -f' or the like to load them into openshift. You can see lots of examples in our infra-ansible repo under roles/openshift_apps/ and playbooks/openshift/
The playbook lists stg and prod control hosts so it deploys to both, but for an initial version you can just specify the staging control host only deploy to staging.
Usually we setup a 'sysadmin-$name' account system group, then give the people in that group permission to run the playbook for that application from our control host (batcave01.iad2.fedoraproject.org) but we are also in the process of rolling out AWX.

aws key sent out of band.

I setup the web interface/role as well now that we are out of freeze. See https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/aws-access/ for the saml2 link to login to the console with that role using your fedora account.

Let us know via additional tickets if you need any perms adjusted, etc.

Shall we close this now?

Shall we close this now?

Shall I open a separate ticket for getting access to staging and production Openshift clusters?

Access to projects on the clusters is controlled by the ansible scripts.

For example:

https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/openshift-apps/bodhi.yml#_40

So, if you could make a PR that adds a playbook, and roles/openshift-apps/<yourapp>/{tasks|templates} we can review it, get it merged and then add your group as able to run the playbook. Running that will deploy the project/app in openshift and give any appowners perms to login and look at their project (but changes should be made via ansible).

Does that make sense? It may be an oddball way to deploy, but it works for us. ;)

Shall we close this now?

I can confirm that I am able to access AWS console through my FAS account. We can close this ticket now. Thank you!