Describe what you would like us to do:

I will take this one and start working on it. I will create everything for staging first, we can later do it for production.

One question about OIDC entry. What e-mail contact should be used there?

I will take this one and start working on it. I will create everything for staging first, we can later do it for production.

Thanks a ton!

One question about OIDC entry. What e-mail contact should be used there?

Let's make it fzatlouk[at]redhat.com

All the secrets for staging should be in place. I created the db and db user as well (hopefully I set the correct db permissions as well) and executed the ipsilon playbook to reflect the changes in OIDC.

So everything should be ready to deploy the kanban app on staging. Let me know if there will be something missing.

It seems there are some missing acls:

$ sudo rbac-playbook openshift-apps/kanban.yml -l staging
NOTIFY: [rbac-playbook] FAILURE frantisekz ran openshift-apps/kanban.yml
('Details:  \n'
 "limit: ['staging']  \n"
 'check: False  \n'
 'tags: None  \n'
 'user: None  \n'
 'start_at_task: None  \n'
 'Sha256: 6d996ffa33c60fe74466c26c66f40b0b3e58ae326565b12aa27f0e1a87bd679e')
user frantisekz is not authorized to run openshift-apps/kanban.yml

Could you try it now? I added the acls for sysadmin-qa, as it needs a group to be specified.

Could you try it now? I added the acls for sysadmin-qa, as it needs a group to be specified.

Nope :(, did try it (after ssh re-loggin), ends with the same issue.

I probably need to run some playbook as well, let me look.

Try it now, I ran the batcave playbook with rbac tag.

Try it now, I ran the batcave playbook with rbac tag.

Play is running now, thanks!

And obviously failing ... :D

fatal: [os-control01.stg.iad2.fedoraproject.org]: FAILED! => {"changed": true, "cmd": "oc -n kanban apply --validate=strict -f /etc/openshift_apps/kanban/deploymentconfig.yml", "delta": "0:00:00.368934", "end": "2024-05-28 14:13:24.327423", "msg": "non-zero return code", "rc": 1, "start": "2024-05-28 14:13:23.958489", "stderr": "error: error parsing /etc/openshift_apps/kanban/deploymentconfig.yml: error converting YAML to JSON: yaml: line 41: did not find expected key", "stderr_lines": ["error: error parsing /etc/openshift_apps/kanban/deploymentconfig.yml: error converting YAML to JSON: yaml: line 41: did not find expected key"], "stdout": "", "stdout_lines": []}

Which points to in our dc:

        env: |-
          {{ load_file('envvars.jinja') | indent(8) }}

I'd guess there are missing vars in the secret storage?

Edited 18 days ago by frantisekz

According to the error it's expecting JSON and the file has YAML structure.

I checked the ansible-private and all the keys are there. You can try to run it with -vv to get more info. I would like to know which key is actually missing.

jskladan commented 18 days ago

According to the error it's expecting JSON and the file has YAML structure.

@zlopez to me, it does not read as "I'm expecting json, and it is yaml" but "Encountered an error when converting the /etc/openshift_apps/kanban/deploymentconfig.yml YAML file to JSON", and the error is "did not find expected key (on line 41)"

The {{ load_file .. }} Jinja line reads the envvars.jinja file, renders it, and outputs the rendered outcome to the current template (indenting it 8 spaces).

Since the deploymentconfig is yaml, I'd think the rendered contents should be YAML too.

Looking at the template, we still might have an error there - the |- should probably not be there, since the contents will be rendered as string. Maybe it is the root cause? Would also make sense with the error, since the parser is probably expecting the env being a dictionary, and not a string.

But I'm no expert here :/

Edited 18 days ago by jskladan

@jskladan That could be the cause, but it's always hard to tell when working with templates. We can try to remove it and see if it helps.

jskladan commented 18 days ago

I don't have access, sadly. Hopefully @frantisekz will get back online soon :D

The issues in the ansible were resolved.

@zlopez is it possible that the psql user was created wrong? mid-hook and deployment is failing at CREATE TABLE with no schema has been selected :

INFO [alembic.runtime.migration] Will assume transactional DDL.
Traceback (most recent call last):
File "/opt/app-root/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1971, in _exec_single_context
self.dialect.do_execute(
File "/opt/app-root/lib64/python3.11/site-packages/sqlalchemy/engine/default.py", line 919, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.InvalidSchemaName: no schema has been selected to create in
LINE 2: CREATE TABLE alembic_version (

Edited 18 days ago by frantisekz

Yeah, @jskladan hacked a small python-psql shell (as I am unable to connect to db01 or db01.stg from batcave via psql as I used to do), and:

on our other app on stg:

SELECT current_schema();
('public',)

and for kanban user/db:

SELECT current_schema();
(None,)

That is possible, this was the first time I created a new db with users in our postgres DB, let me check what I missed.

I added a few more permissions to kanban_user, could you try it now?

I added a few more permissions to kanban_user, could you try it now?

Now, it ended with:

psycopg2.errors.InsufficientPrivilege: permission denied for schema public
LINE 2: CREATE TABLE alembic_version

We're getting somewhere, I still do believe we're missing some perms?

Yeah, let me check

I hope I fixed that now (granted CREATE privilege on public schema to kanban_user). Could you try again?

I hope I fixed that now (granted CREATE privilege on public schema to kanban_user). Could you try again?

Yes, it worked fine now, thanks!

We're now crash-looping for our reasons, will investigate.

Nice, let me know if you need anything else from me.

From: "https://kanban.qa{{env_suffix}}.fedoraproject.org/oidc_callback"
To: "https://kanban.qa{{env_suffix}}.fedoraproject.org/flask_oidc/authorize"

Nice, let me know if you need anything else from me.

Okay, we've resolved the deployment issues on our side, some more things if I may ask:

We have changed oidc endpoint:

I believe this would need to be changed auth server side?

And apart from that, can you proceed with production env preparation (db, oidc)?

@kevin Would you have time/cycles to add dns? Production would be ready asap once @zlopez has time to prepare the env (I do believe route needs to be up first, if I am not mistaken).

Edited 17 days ago by frantisekz

Let me change the OIDC endpoint first and I will start preparing everything for production as well.

The OIDC endpoint is now changed. I will start preparing the secrets for production and I will look at the DNS as well.

The DB, OIDC and secrets are now available for production. I will now look at the DNS.

So I looked in our DNS repository and can't find how this is done for Openshift as I don't see anything in the documentation and can't find anything hosted on OpenShift here.

@kevin How the DNS is done for OpenShift, I probably missed something.

kevin commented 17 days ago

So, what dns exactly. ;)

Do you want:

kanban.qa.fedoraproject.org ?

qa.fedoraproject.org/kanban

Currently blockerbugs is under like the second one. I think it would be consistent to just add this under there too? but I guess it's not a big deal either way.

For the first one we would have to add a dns entry in fedoraproject.org.template for kanban.qa and kanban.qa.stg and then add a website to playbooks/includes/proxies-websites and then a reverseproxy to playbooks/includes/proxies-reverseproxy.

For the second one we don't need dns or a new website, just a reverseproxy addition with qa.fedoraproject.org website.