Still investigating. As a work around, for now, I added logrotate to the root crontab

ok, so I added a -v to the service and let it run and:

...
Mar 22 00:00:00 proxy31.fedoraproject.org logrotate[3068400]: rotating pattern: /var/log/httpd/*log  after 1 days (7 rotations)
Mar 22 00:00:00 proxy31.fedoraproject.org logrotate[3068400]: empty log files are rotated, old logs are removed                                                    
Mar 22 00:00:00 proxy31.fedoraproject.org logrotate[3068400]: No logs found. Rotation not needed.

So, I think whats happening there is that the cron one you added runs at 01:00... and the logrotate is set to run 'daily', but it checks /var/lib/logrotate/logrotate.status to see the last time it was rotated and it's 23hours, not yet a day, so it says... nope, not yet.

So, I've disabled the cron added one and left the -v on the service, so we should be able to check back on this tomorrow and my theory is that it should rotate then, or at least tell us in logs why not. ;)

My theory turned out to be...wrong. ;)

I put the cron back.

So, for some reason it's saying 'no logs found'... why can't it see the httpd/*log files?

Could it be SELinux issue?

No denials and the labels on the files seem right. ;(

I also had trouble to find the root cause, worked on that before going on PTO (about 2 weeks ago) and here are a few things that I checked:

• Config files in /etc/logrotate.d/httpd and /etc/logrotate.conf and they seem right
• Disk still had about 25% free space
• If another process was keeping logfiles open. There wasn't
• Running logrotate command wasn't throwing any errors
• Service status was pointing it was up and running without issues

The only error I found was an overlaping alias in httpd.conf pointed by error.log, but it doesn't seem to be related.

Our infra is coded right? Would be a bad idea to provision that machine again?

Yeah, we could just redeploy it. But that should probibly wait until after freeze.

I'm pretty stumped as to what could be going on.

I wonder if it could be a noaudit selinux thing... and doing a 'fixfiles onboot' and rebooting it (forcing a selinux relabel) is worth trying?

Shouldn't the SELinux errors be visible somewhere even without audit turned on?

Not always. selinux has what it calls 'don't audit' rules, that don't log ever, even when they are hit. This is usually for things that have a lot of denials...

You can turn on logging of noaudit rules too, but it really spews a lot. It's usually easier to just relabel and see if that fixes it.

Then I agree that relabel sounds like a good option. Should we wait for end of freeze?

Yep.

@kevin I think with the freeze over, sssd giving issues, and logrotate being an issue, let just do a clean rebuild and if the issue persists we can take a deeper look.