On ipa we have 3 nodes, I'm not sure who's the master ..
maybe try with ipa02..3 ??

So, ipa by default redirects that to a primary server, but... all our servers are behind a proxy, so that 'iad2' network is not resolvable or reachable. ;)

We have an ansible task to disable this rewrite and let it use the external proxy host.

See "- name: Disable rewrites" in roles/ipa/server/tasks/main.yml

Unfortunately, ipa puts the default one back every time it updates, so the playbook has to be run. ;(

I've done that now, and I expect it should be working for you now?

I've done that now, and I expect it should be working for you now?

Yes, that appears to work now, thanks! Should we have (do we have?) a ticket to fix that permanently? Low prio, I guess, but seems like we should deal with it...

Changing gain from high to low, as I didn't noticed that this is staging only.

It would be great to fix this, but I am not sure how off hand?

I guess we could ask ipa packagers to mark that redirect as a config file so it doesn't get overwritten if it changes?

This is /etc/httpd/conf.d/ipa-rewrite.conf

rpm -qf /etc/httpd/conf.d/ipa-rewrite.conf

ipa-server-common-4.12.2-1.el9_5.3.noarch

Or is there some other way?

CC: @abbra

IPA does not officially support putting itself beyond the proxy. We resisted (and continue to do so) for the whole life of the project, for a variety of reasons, primarily due to a fragility of that support and lack of official support for aliasing of the host principals.

You have to continue adjusting your custom configuration.

@kevin Could we setup some job that will trigger playbook run with -t config each time ipa-server-common gets updated?

Well, with dnf5 we could use an action config.

I don't think we want to build some elaborate mousetrap / rube roldberg device here.

I guess we could make a cronjob that looked at the conf file and if it's changed, copy our version over it?

Thats the best I can come up with.

I think the best candidate to check idempotency is ansible ;)
As you said Let's put a cron job (to run playbook with the tag config) , so i don't know about frequency.

Sorry for the lag, travelling for conferences.

So I agree the obvious fix is to let Ansible keep it straight. Idempotence seem easy if it's just file content, we can just ensure it is what we want.

Seems like a good issue to ease me into our Ansible setup, would you agree @kevin? If so, feel free to assign it to me and I'll take a closer look.

well, I was suggesting just a simple cron job on the ipa servers.

Just look if that file has changed or has diffs from a stored copy.
If not, exit, if so, copy the stored one over and reload httpd

If we try and put ansible in the mix, that means we have to either just run it always on a timer/cron from batcave01 and that seems like a waste.

So, sure if you want to work on it thats great! But I'd advise just a simple shell script or whatever on ipa servers.

Normally I'd suggest putting that into the cfg mgmt to make sure it stays fixed - but I'll bow to the historical weight of Fedora here :)

I can write a cronjob - do we have a place to review such a script, @kevin?

@gwmngilfen That would be probably the ansible repository.

So, I had another thought... can we use apache loading rules here?

What if we left the upstream one alone, but added another one that sorted lexically before the ipa one. Could we have that just have it's rules take precidence?

Shall we leave this issue open to track a better fix? Or close it and make a new ticket once we figure out what the fix is going to be?

I'm willing to deal with it - it's low-prio but good practice. I will see if I can get the Apache thing to work in a test env, if not a cronjob is an easy workaround.

EDIT: However, I don't appear to be able to self-assign, so feel free to put it on me :)

I can confirm we should be able to overwrite the rewrite rule with lexical scoping. I spun up a clean httpd install, and added two files to conf.d:

/etc/httpd/conf.d/test.conf 
# Rewrite rule to serve the same file on a different URL
RewriteEngine On
RewriteRule ^/static/test$ /textfile.txt [L]

[root@n27-15-162 conf.d]# cat /etc/httpd/conf.d/zz_test.conf 
# Rewrite rule to serve the same file on a different URL
RewriteEngine On
RewriteRule ^/static/test$ /textfile2.txt [L]

so zz_test comes after test, and thus if we curl, we get textfile.txt:

# curl http://localhost/static/test
Failure - Old file

but if i move the file to 01, then it works

[root@n27-15-162 conf.d]# mv /etc/httpd/conf.d/{zz,01}_test.conf 
[root@n27-15-162 conf.d]# systemctl restart httpd
[root@n27-15-162 conf.d]# curl http://localhost/static/test
Success - New file

So that works :tada:

The catch is that right now we just comment out the rule (as I understand it, I can't access the ipa server to check, and the rewrite rules aren't in the rpm). So we'll need to construct a new rule to replace the one that is written (that I can't see), and place it higher in the lexical scope.

Having downgraded and re-upgraded ipa-server in staging (thanks to @zlopez!) it would appear the package does not touch ipa-rewrites.conf at all - we even tried moving the file out of the way, and it was not re-created.

We did see a lot of backups that appear to be some form of automation:

ipa-rewrites.conf
ipa-rewrites.conf.bak
ipa-rewrites.conf.bak.7
ipa-rewrites.conf.bak.7.7
ipa-rewrites.conf.bak.7.7.7
ipa-rewrites.conf.bak.7.7.7.7
etc

It seems unlikely that a human would add that many 7s, but in any case, they all appear to have the same content, which is also the same as the ipa-rewrites.conf from Ansible.

@kevin Any ideas what else could be writing the file that we're overwriting with Ansible? Right now it looks like it's not created by anything....

IPA owns the configuration file (ipa-rewrite.conf, not ipa-rewrites.conf) but it is empty in the beginning and then is populated when ipa-server-install or ipa-replica-install is run:

%ghost %attr(0644,root,root) %config(noreplace) %{_sysconfdir}/httpd/conf.d/ipa-rewrite.conf

After that it is only updated when the original template in /usr/share/ipa/ipa-rewrite.conf.template is updated. The last change for that happened in 2020.

Thanks @abbra for getting me back on-track! I found the ghost but didn't see the install process. Can confirm the template is on-disk on my test host, I can dig into it from here.