fedora-ci / general

#535 Non standard root "/" mode 775 on testing farm
Opened by jskarvad. Modified

Open
Close Issue

The / mode is usually 755, but for some reason unknown to me it's 775 when running Bodhi gating tests on testing farm. This causes sendmail to stop working and most of the sendmail tests fails which blocks sendmail from entering the stable Fedora repo (e.g. it was stuck in the testing for 7 months last time). The failure:

:: [ 13:26:42 ] :: [   LOG    ] ::   Dec 10 13:26:42 ip-172-31-25-16.us-east-2.compute.internal systemd[1]: Starting sendmail.service - Sendmail Mail Transport Agent...
:: [ 13:26:42 ] :: [   LOG    ] ::   Dec 10 13:26:42 ip-172-31-25-16.us-east-2.compute.internal (sendmail)[8594]: sendmail.service: Referenced but unset environment variable evaluates to an empty string: SENDMAIL_OPTARG
:: [ 13:26:42 ] :: [   LOG    ] ::   Dec 10 13:26:42 ip-172-31-25-16.us-east-2.compute.internal sendmail[8594]: 451 4.0.0 /etc/mail/sendmail.cf: line 91: fileclass: cannot open '/etc/mail/local-host-names': Group writable directory
:: [ 13:26:42 ] :: [   LOG    ] ::   Dec 10 13:26:42 ip-172-31-25-16.us-east-2.compute.internal sendmail[8594]: NOQUEUE: SYSERR(root): /etc/mail/sendmail.cf: line 91: fileclass: cannot open '/etc/mail/local-host-names': Group writable directory
:: [ 13:26:42 ] :: [   LOG    ] ::   Dec 10 13:26:42 ip-172-31-25-16.us-east-2.compute.internal sendmail[8594]: NOQUEUE: SYSERR(root): /etc/mail/sendmail.cf: line 633: fileclass: cannot open '/etc/mail/trusted-users': Group writable directory
:: [ 13:26:42 ] :: [   LOG    ] ::   Dec 10 13:26:42 ip-172-31-25-16.us-east-2.compute.internal sendmail[8594]: 451 4.0.0 /etc/mail/sendmail.cf: line 633: fileclass: cannot open '/etc/mail/trusted-users': Group writable directory
:: [ 13:26:42 ] :: [   LOG    ] ::   Dec 10 13:26:42 ip-172-31-25-16.us-east-2.compute.internal systemd[1]: sendmail.service: Control process exited, code=exited, status=71/OSERR
:: [ 13:26:43 ] :: [   LOG    ] ::   Dec 10 13:26:42 ip-172-31-25-16.us-east-2.compute.internal systemd[1]: sendmail.service: Failed with result 'exit-code'.
:: [ 13:26:43 ] :: [   LOG    ] ::   Dec 10 13:26:42 ip-172-31-25-16.us-east-2.compute.internal systemd[1]: Failed to start sendmail.service - Sendmail Mail Transport Agent.

Original test run:
https://artifacts.dev.testing-farm.io/fa7bfd1f-ecd0-45ff-9cd1-a9f51a64cf5d/

This wasn't reproducible locally nor in the 1minutetip, so it took me some time to debug. Explicitly running:

# chmod 755 /

In the first test fixed the test and all consequent tests.

If this isn't the correct place where to report it, please give me a hint where to report.

It was originally reported here:
https://pagure.io/fedora-infrastructure/issue/12974

Howdy, this is also good place, otherwise if you do not know next time:

https://docs.testing-farm.io/Testing%20Farm/0.1/issues.html

I am looking what is going on here. We do not touch this piece at all we run instances directly from Fedora Cloud Base images.

I see 775 in a vanilla Fedora Cloud Base images, so nothing to do with Testing Farm or Fedora CI:

❯ tmt run provision -h virtual -i https://kojipkgs.fedoraproject.org/compose/42/latest-Fedora-42/compose/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-42-1.1.x86_64.qcow2 login
/var/home/thrix/.local/share/tmt/run-105
/default/plan
    provision
        queued provision.provision task #1: default-0
        provision.provision task #1: default-0
        how: virtual
        ansible: GuestAnsible(group=None, vars={})
        image: https://kojipkgs.fedoraproject.org/compose/42/latest-Fedora-42/compose/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-42-1.1.x86_64.qcow2
        progress: downloading...
        memory: 2048 MB
        disk: 40 GB
        progress: booting...
        multihost name: default-0
        arch: x86_64
        distro: Fedora Linux 42 (Cloud Edition)
        queued provision.action task #1: login
        provision.action task #1: login
        login: Starting interactive shell
# ls -ld /
drwxrwxr-x. 1 root root 228 Apr  9  2025 /
# stat /
  File: /
  Size: 228         Blocks: 0          IO Block: 4096   directory
Device: 0,34    Inode: 256         Links: 1
Access: (0775/drwxrwxr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:root_t:s0
Access: 2025-12-11 19:10:10.592676861 +0000
Modify: 2025-04-09 12:10:06.587194056 +0000
Change: 2025-04-09 12:10:13.304298371 +0000
 Birth: 2025-04-09 12:09:39.888317616 +0000

https://github.com/OSInside/kiwi/issues/2920

@jskarvad thanks for reporting, based on the Fedora devel Matrix discussion:

https://matrix.to/#/!DMptnKvPTdgoULzUTV:fedoraproject.org/$VOQVDecMMQe84-kbh02gJ4BTN_yHSg7SS9-sDZUlWLs?via=fedoraproject.org&via=fedora.im&via=matrix.org

Seems like an inconsistency with the unix standard done by kiwi, the tool which builds Fedora images.
Edited by mvadkert

So this is actually a Mock bug: https://github.com/rpm-software-management/mock/issues/1675

Pull request to fix Mock submitted: https://github.com/rpm-software-management/mock/pull/1676

I checked today and the latest Rawhide still has it wrong.

I guess we are waiting until the changes propagate?

❯ tmt run provision -h virtual -i https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/compose/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-Rawhide-20260109.n.0.x86_64.qcow2 login
/var/home/thrix/.local/share/tmt/run-003
/default/plan
    provision
        queued provision.provision task #1: default-0
        provision.provision task #1: default-0
        how: virtual
        ansible: GuestAnsible(group=None, vars={})
        image: https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/compose/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-Rawhide-20260109.n.0.x86_64.qcow2
        progress: downloading...
        memory: 2048 MB
        disk: 40 GB
        progress: booting...
        multihost name: default-0
        arch: x86_64
        distro: Fedora Linux 44 (Cloud Edition Prerelease)
        queued provision.action task #1: login
        provision.action task #1: login
        login: Starting interactive shell
root@default-0:/var/home/thrix/.local/share/tmt/run-003/default/plan/tree[root@default-0 tree]# ls -ld /
drwxrwxr-x. 1 root root 166 Jan  9 06:00 /

The mock fix hasn't been released yet, probably should go talk to @msuchy about that.
Metadata
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.