fedora-infrastructure

Clone
Source Code
GIT

#11725 random koji buildroot failure due to some wrong arguments occasionally passed to dnf5??
Closed: Fixed a year ago by kevin. Opened a year ago by mtasaka.

Closed: Fixed
Reopen Issue

First look at this:

https://koji.fedoraproject.org/koji/taskinfo?taskID=111915408
On x86_64 (failure):

DEBUG package_manager.py:289:  ['/usr/bin/dnf5', '--installroot', '/var/lib/mock/f40-build-48241989-5748672/root/', 'groupinstall', 'build', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes', '--allowerasing']
DEBUG util.py:636:  child environment: None
DEBUG util.py:553:  Using nspawn with args ['--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.4b3ys_hb:/etc/resolv.conf']
DEBUG util.py:558:  Executing command: ['/usr/bin/systemd-nspawn', '-q', '-M', '10de1e78548b49359ba23a3572a16e3f', '-D', '/var/lib/mock/f40-build-48241989-5748672-bootstrap/root', '-a', '--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.4b3ys_hb:/etc/resolv.conf', '--console=pipe', '--setenv=TERM=vt100', '--setenv=SHELL=/bin/bash', '--setenv=HOME=/var/lib/mock/f40-build-48241989-5748672/root/installation-homedir', '--setenv=HOSTNAME=mock', '--setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin', '--setenv=PROMPT_COMMAND=printf "\\033]0;<mock-chroot>\\007"', '--setenv=PS1=<mock-chroot> \\s-\\v\\$ ', '--setenv=LANG=C.UTF-8', '--setenv=LC_MESSAGES=C.UTF-8', '--resolv-conf=off', '/usr/bin/dnf5', '--installroot', '/var/lib/mock/f40-build-48241989-5748672/root/', 'groupinstall', 'build', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes', '--allowerasing', '--setopt=tsflags=nocontexts'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/var/lib/mock/f40-build-48241989-5748672/root/installation-homedir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'C.UTF-8', 'LC_MESSAGES': 'C.UTF-8', 'SYSTEMD_NSPAWN_TMPFS_TMP': '0', 'SYSTEMD_SECCOMP': '0'} and shell False
DEBUG util.py:461:  Unknown argument "groupinstall" for command "dnf5". Add "--help" for more information about the arguments.

On ppc64le (successful):

DEBUG package_manager.py:289:  ['/usr/bin/dnf5', '--installroot', '/var/lib/mock/f40-build-48241999-5748672/root/', 'group', 'install', 'build', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes']
DEBUG util.py:636:  child environment: None
DEBUG util.py:553:  Using nspawn with args ['--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.1560uiw8:/etc/resolv.conf']
DEBUG util.py:558:  Executing command: ['/usr/bin/systemd-nspawn', '-q', '-M', '5a2517106e6f43f091bf5a7e28eb3e4f', '-D', '/var/lib/mock/f40-build-48241999-5748672-bootstrap/root', '-a', '--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.1560uiw8:/etc/resolv.conf', '--console=pipe', '--setenv=TERM=vt100', '--setenv=SHELL=/bin/bash', '--setenv=HOME=/var/lib/mock/f40-build-48241999-5748672/root/installation-homedir', '--setenv=HOSTNAME=mock', '--setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin', '--setenv=PROMPT_COMMAND=printf "\\033]0;<mock-chroot>\\007"', '--setenv=PS1=<mock-chroot> \\s-\\v\\$ ', '--setenv=LANG=C.UTF-8', '--setenv=LC_MESSAGES=C.UTF-8', '--resolv-conf=off', '/usr/bin/dnf5', '--installroot', '/var/lib/mock/f40-build-48241999-5748672/root/', 'group', 'install', 'build', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes', '--setopt=tsflags=nocontexts'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/var/lib/mock/f40-build-48241999-5748672/root/installation-homedir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'C.UTF-8', 'LC_MESSAGES': 'C.UTF-8', 'SYSTEMD_NSPAWN_TMPFS_TMP': '0', 'SYSTEMD_SECCOMP': '0'} and shell False
DEBUG util.py:463:  Updating and loading repositories:
DEBUG util.py:463:   build                                  100% |  57.2 MiB/s |  16.6 MiB |  00m00s

So for x86_64, groupinstall is passed to dnf5, on the other hand on ppc64le group install is passed.

Next, https://koji.fedoraproject.org/koji/taskinfo?taskID=111912530
On x86_64 (successful):

DEBUG file_util.py:21:  created dir: /var/lib/mock/f40-build-48240062-5748626/root/installation-homedir
DEBUG package_manager.py:289:  ['/usr/bin/dnf5', '--installroot', '/var/lib/mock/f40-build-48240062-5748626/root/', 'group', 'install', 'build', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes']
DEBUG util.py:636:  child environment: None
DEBUG util.py:553:  Using nspawn with args ['--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.fdt5pbne:/etc/resolv.conf']
DEBUG util.py:558:  Executing command: ['/usr/bin/systemd-nspawn', '-q', '-M', '69286198251d4cbebe3bbba5ac862e76', '-D', '/var/lib/mock/f40-build-48240062-5748626-bootstrap/root', '-a', '--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.fdt5pbne:/etc/resolv.conf', '--console=pipe', '--setenv=TERM=vt100', '--setenv=SHELL=/bin/bash', '--setenv=HOME=/var/lib/mock/f40-build-48240062-5748626/root/installation-homedir', '--setenv=HOSTNAME=mock', '--setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin', '--setenv=PROMPT_COMMAND=printf "\\033]0;<mock-chroot>\\007"', '--setenv=PS1=<mock-chroot> \\s-\\v\\$ ', '--setenv=LANG=C.UTF-8', '--setenv=LC_MESSAGES=C.UTF-8', '--resolv-conf=off', '/usr/bin/dnf5', '--installroot', '/var/lib/mock/f40-build-48240062-5748626/root/', 'group', 'install', 'build', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes', '--setopt=tsflags=nocontexts'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/var/lib/mock/f40-build-48240062-5748626/root/installation-homedir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'C.UTF-8', 'LC_MESSAGES': 'C.UTF-8', 'SYSTEMD_NSPAWN_TMPFS_TMP': '0', 'SYSTEMD_SECCOMP': '0'} and shell False
DEBUG util.py:463:  Updating and loading repositories:
DEBUG util.py:463:   build                                  100% |  97.2 MiB/s |  17.6 MiB |  00m00s

Okay, group install is used, and no error.

But on i686 (failure):

DEBUG util.py:553:  Using nspawn with args ['--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.4fqzpd1k:/etc/resolv.conf']
DEBUG util.py:558:  Executing command: ['/usr/bin/systemd-nspawn', '-q', '-M', '41961b9ee3b140eda2ee2c91559a676a', '-D', '/var/lib/mock/f40-build-48240060-5748626-bootstrap/root', '-a', '--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.4fqzpd1k:/etc/resolv.conf', '--console=pipe', '--setenv=TERM=vt100', '--setenv=SHELL=/bin/bash', '--setenv=HOME=/var/lib/mock/f40-build-48240060-5748626/root/installation-homedir', '--setenv=HOSTNAME=mock', '--setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin', '--setenv=PROMPT_COMMAND=printf "\\033]0;<mock-chroot>\\007"', '--setenv=PS1=<mock-chroot> \\s-\\v\\$ ', '--setenv=LANG=C.UTF-8', '--setenv=LC_MESSAGES=C.UTF-8', '--resolv-conf=off', '/usr/bin/dnf5', '--installroot', '/var/lib/mock/f40-build-48240060-5748626/root/', 'group', 'install', 'build', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes', '--allowerasing', '--setopt=tsflags=nocontexts'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/var/lib/mock/f40-build-48240060-5748626/root/installation-homedir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'C.UTF-8', 'LC_MESSAGES': 'C.UTF-8', 'SYSTEMD_NSPAWN_TMPFS_TMP': '0', 'SYSTEMD_SECCOMP': '0'} and shell False
DEBUG util.py:461:  Unknown argument "--allowerasing" for command "install". Add "--help" for more information about the arguments.

Now extra --allowerasing option is passed and dnf5 is complaining.

So looks like some random wrong argument is occuasionally passed to dnf5??

Would someone look at this? I think (if this is the real issue) this should be resolved before mass rebuild.

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: Needs investigation, high-gain, ops
a year ago

@jnsamyak Could you look into that, it seems that the issue is more related to build scripts than problem with builders.

From the first glance it seems that groupinstall is now group install. Not sure about the --allowerasing argument.

Oops: for failure i686 creating buildroot:
https://koji.fedoraproject.org/koji/taskinfo?taskID=111912575

DEBUG file_util.py:21:  created dir: /var/lib/mock/f40-build-48240060-5748626/root/installation-homedir
DEBUG package_manager.py:289:  ['/usr/bin/dnf5', '--installroot', '/var/lib/mock/f40-build-48240060-5748626/root/', 'group', 'install', 'build', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes', '--allowerasing']
DEBUG util.py:636:  child environment: None
DEBUG util.py:553:  Using nspawn with args ['--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.4fqzpd1k:/etc/resolv.conf']
DEBUG util.py:558:  Executing command: ['/usr/bin/systemd-nspawn', '-q', '-M', '41961b9ee3b140eda2ee2c91559a676a', '-D', '/var/lib/mock/f40-build-48240060-5748626-bootstrap/root', '-a', '--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.4fqzpd1k:/etc/resolv.conf', '--console=pipe', '--setenv=TERM=vt100', '--setenv=SHELL=/bin/bash', '--setenv=HOME=/var/lib/mock/f40-build-48240060-5748626/root/installation-homedir', '--setenv=HOSTNAME=mock', '--setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin', '--setenv=PROMPT_COMMAND=printf "\\033]0;<mock-chroot>\\007"', '--setenv=PS1=<mock-chroot> \\s-\\v\\$ ', '--setenv=LANG=C.UTF-8', '--setenv=LC_MESSAGES=C.UTF-8', '--resolv-conf=off', '/usr/bin/dnf5', '--installroot', '/var/lib/mock/f40-build-48240060-5748626/root/', 'group', 'install', 'build', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes', '--allowerasing', '--setopt=tsflags=nocontexts'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/var/lib/mock/f40-build-48240060-5748626/root/installation-homedir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'C.UTF-8', 'LC_MESSAGES': 'C.UTF-8', 'SYSTEMD_NSPAWN_TMPFS_TMP': '0', 'SYSTEMD_SECCOMP': '0'} and shell False
DEBUG util.py:461:  Unknown argument "--allowerasing" for command "install". Add "--help" for more information about the arguments.

I was looking upstream to see if this is a known issue, and I stumbled upon this: https://github.com/rpm-software-management/dnf5/issues/548; I will debug things more related to this;

There is a similar discussion going on here: https://pagure.io/releng/issue/11888#comment-891934

The patches we tried to do with @kevin were assuring that:
- group install is always used, never groupinstall, globally for all chroots, not just F40, this has been somehow hotfixed?
- and that --allowerasing is not used with group command, but this is F40 only because only F40 uses dnf5
Edited a year ago by praiskup

dnf5 group install --allowerasing is not yet supported in dnf5 5.1.11. The support is only in upstream git tree https://github.com/rpm-software-management/dnf5/commit/57fe937a7049072fde9acb9406d730fd250ae65a now. I can back port the support if it helps you.

I don't know what it means by "if it helps you" because it is now directly specified by me or other reporter. Me (or other reporter) just did fedpkg build but koji build used dnf5 group install --allowerasing .

But looking at https://pagure.io/releng/issue/11737 , looks like this was because of koji builder setting issue.

That was addressed to relengs. Thanks for the the link. I noticed it there.

Yes, this was a builder that was down when the change was applied yesterday that then came back up. ;(

Sorry about that.

Please let us know if you still see this issue anywhere.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
a year ago

Log in to comment on this ticket.

Metadata
None

high-gain ops Needs investigation

None
None
Waiting on Assignee
Boards 1
ops Status: Backlog
Powered by Pagure 5.14.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.