fedora-infrastructure

#9589 rpmbuild crashes on s390x just after "Checking for unpackaged file": Child return code was: -11
Closed: Upstream by kevin. Opened by ppisar.

Closed: Upstream
Reopen Issue

All builds on s390x seems to crash when rpmbuild writes the resulting RPM packages:

First known failure https://koji.fedoraproject.org/koji/buildinfo?buildID=1671655 at Wed, 20 Jan 2021 04:41:38 UTC:

Processing files: dummy-test-package-crested-0-2563.noarch
Executing(%license): /bin/sh -e /var/tmp/rpm-tmp.wBt9N3
+ umask 022
+ cd /builddir/build/BUILD
+ cd dummy-test-package-crested-0
+ LICENSEDIR=/builddir/build/BUILDROOT/dummy-test-package-crested-0-2563.noarch/usr/share/licenses/dummy-test-package-crested
+ export LC_ALL=C
+ LC_ALL=C
+ export LICENSEDIR
+ /usr/bin/mkdir -p /builddir/build/BUILDROOT/dummy-test-package-crested-0-2563.noarch/usr/share/licenses/dummy-test-package-crested
+ cp -pr LICENSE /builddir/build/BUILDROOT/dummy-test-package-crested-0-2563.noarch/usr/share/licenses/dummy-test-package-crested
+ RPM_EC=0
++ jobs -p
+ exit 0
Provides: dummy-test-package-crested = 0-2563
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Checking for unpackaged file(s): /usr/lib/rpm/check-files /builddir/build/BUILDROOT/dummy-test-package-crested-0-2563.noarch
Child return code was: -11
EXCEPTION: [Error()]
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/util.py", line 600, in do_with_status
    raise exception.Error("Command failed: \n # %s\n%s" % (command, output), child.returncode)
mockbuild.exception.Error: Command failed: 
 # bash --login -c /usr/bin/rpmbuild -bb --target noarch --nodeps /builddir/build/SPECS/dummy-test-package-crested.spec

Wonder if it could be the new rpm-4.16.1.2-3.fc34 with dwarf5 support, but its build finished 2021-01-19 18:43:43

CC @fweimer

thread from the devel list is https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/XQWO72AZ7MKVC3CSICKKYF77EHNUVJ6U/

The last passing build is for ELN https://koji.fedoraproject.org/koji/buildinfo?buildID=1671651 which shows these upgrades-to:

    gdb-minimal-10.1
    rpm-4.16.1.2
    rpm-build-4.16.1.2
    rpm-build-libs-4.16.1.2
    rpm-libs-4.16.1.2

The last passing build for F34 https://koji.fedoraproject.org/koji/buildinfo?buildID=1671644 shows these updates-to:

    libgcc-11.0.0-0.15.fc34
    libgomp-11.0.0-0.15.fc34
    libstdc++-11.0.0-0.15.fc34

Note that gcc-11.0.0-0.15.fc34 changelog says:

- switch to DWARF 5 by default

Actually when rebuilding some srpm with gcc-11.0.0-0.15.fc34 and old rpm-4.16.1.2-2.fc34, this shows another error that /usr/lib/rpm/debugedit complains about something related to DWARF 5.

I'm trying to see if I can fix this in a side tag with an rpm rebuild.

gcc-11.0.0-0.14.fc34.s390x and rpm-4.16.1.2-1.fc34.s390x together do not trigger this issue; it is possible to rebuild rpm and compsize (with -gdwarf-4 in CFLAGS, due to the older rpm).

Unaware of this issue I "fixed" rpm so it builds fine on eln now and might have introduced the same issue in eln with rpm-4.16.1.2-3.eln108. Sigh. Sorry.

@mjw A quick test showed that rpm-4.16.1.2-3.eln108 doesn't have this problem on its own. But we definitely need to watch out for this issue if we bring in the new gcc version.

@mtasaka

Note that gcc-11.0.0-0.15.fc34 changelog says:

  • switch to DWARF 5 by default

Actually when rebuilding some srpm with gcc-11.0.0-0.15.fc34 and old rpm-4.16.1.2-2.fc34, this
shows another error that /usr/lib/rpm/debugedit complains about something related to DWARF 5.

Note this is part of the upcoming mass rebuild proposed in https://pagure.io/releng/issue/9858

Could you post the exact complaints that /usr/lib/rpm/debugedit produces and possibly the actual srpm/builds/logs? We aren't aware of any debugedit bugs at the moment related to DWARF5 (although there are some in gcc and dwz - see the releng issue).

Builds which are mentioned in the email thread on devel list

https://koji.fedoraproject.org/koji/taskinfo?taskID=60085705
https://koji.fedoraproject.org/koji/taskinfo?taskID=60089433
https://koji.fedoraproject.org/koji/taskinfo?taskID=60083125

Downgrade to gcc-11.0.0-0.14.fc34.s390x fixes the issue, but then DWARF 5 is also gone.

On an up-to-date rawhide system with gcc-11.0.0-0.15.fc34.s390x and rpm-4.16.1.2-3.fc34.s390x, downgrading just libgomp fixes the issue:

rpm --nodeps -U --oldpackage libgomp-11.0.0-0.14.fc34.s390x.rpm

@jakub This looks like libgomp has been miscompiled.

@mtasaka

Note that gcc-11.0.0-0.15.fc34 changelog says:

  • switch to DWARF 5 by default

Actually when rebuilding some srpm with gcc-11.0.0-0.15.fc34 and old rpm-4.16.1.2-2.fc34, this
shows another error that /usr/lib/rpm/debugedit complains about something related to DWARF 5.

Note this is part of the upcoming mass rebuild proposed in https://pagure.io/releng/issue/9858

Could you post the exact complaints that /usr/lib/rpm/debugedit produces and possibly the actual srpm/builds/logs? We aren't aware of any debugedit bugs at the moment related to DWARF5 (although there are some in gcc and dwz - see the releng issue).

Sorry, actually with gcc-11.0.0-0.15.fc34 and old rpm-4.16.1.2-1.fc34, I meant. With this combination, I see (for example I rebuild annobin-9.59-1.fc34),

+ /usr/lib/rpm/find-debuginfo.sh -j6 --strict-build-id -m -i --build-id-seed 9.59-1.fc34 --unique-debug-suffix -9.59-1.fc34.x86_64 --unique-debug-src-base annobin-9.59-1.fc34.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /builddir/build/BUILD/annobin-9.59
explicitly decompress any DWARF compressed ELF sections in /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/bin/annocheck
explicitly decompress any DWARF compressed ELF sections in /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/lib64/clang/11.0.1/annobin-for-clang.so
explicitly decompress any DWARF compressed ELF sections in /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/lib/gcc/x86_64-redhat-linux/11/plugin/annobin.so.0.0.0
explicitly decompress any DWARF compressed ELF sections in /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/lib64/clang/11.0.1/annobin-for-llvm.so
extracting debug info from /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/bin/annocheck
extracting debug info from /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/lib64/clang/11.0.1/annobin-for-llvm.so
extracting debug info from /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/lib64/clang/11.0.1/annobin-for-clang.so
extracting debug info from /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/lib/gcc/x86_64-redhat-linux/11/plugin/annobin.so.0.0.0
/usr/lib/rpm/debugedit: /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/lib/gcc/x86_64-redhat-linux/11/plugin/annobin.so.0.0.0: Unknown debugging section .debug_loclists
/usr/lib/rpm/debugedit: /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/lib/gcc/x86_64-redhat-linux/11/plugin/annobin.so.0.0.0: Unknown debugging section .debug_rnglists
/usr/lib/rpm/debugedit: /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/lib/gcc/x86_64-redhat-linux/11/plugin/annobin.so.0.0.0: Unknown debugging section .debug_line_str
/usr/lib/rpm/debugedit: /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/lib/gcc/x86_64-redhat-linux/11/plugin/annobin.so.0.0.0: DWARF version 5 unhandled
/usr/lib/rpm/debugedit: /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/bin/annocheck: Unknown debugging section .debug_loclists
/usr/lib/rpm/debugedit: /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/bin/annocheck: Unknown debugging section .debug_rnglists
/usr/lib/rpm/debugedit: /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/bin/annocheck: Unknown debugging section .debug_line_str
/usr/lib/rpm/debugedit: /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/bin/annocheck: DWARF version 5 unhandled

I think this is expected.

@mtasaka
Sorry, actually with gcc-11.0.0-0.15.fc34 and old rpm-4.16.1.2-1.fc34, I meant. With this combination, I see (for example I rebuild annobin-9.59-1.fc34),

```
+ /usr/lib/rpm/find-debuginfo.sh -j6 --strict-build-id -m -i --build-id-seed 9.59-1.fc34 --unique-debug-suffix -9.59-1.fc34.x86_64 --unique-debug-src-base annobin-9.59-1.fc34.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /builddir/build/BUILD/annobin-9.59
[...]
/usr/lib/rpm/debugedit: /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/bin/annocheck: Unknown debugging section .debug_loclists
/usr/lib/rpm/debugedit: /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/bin/annocheck: Unknown debugging section .debug_rnglists
/usr/lib/rpm/debugedit: /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/bin/annocheck: Unknown debugging section .debug_line_str
/usr/lib/rpm/debugedit: /builddir/build/BUILDROOT/annobin-9.59-1.fc34.x86_64/usr/bin/annocheck: DWARF version 5 unhandled

```

I think this is expected.

O, yes, that is indeed expected with old rpm and new gcc. Thanks for clarifying.

Upstream commit identified: https://gcc.gnu.org/pipermail/gcc-patches/2021-January/563928.html

gcc build that should fix this, thanks jakub: https://koji.fedoraproject.org/koji/buildinfo?buildID=1671837

Do we need to follow up the gcc build with further builds of other packages? If so I can help if you let me know which packages need to be rebuilt.

So, I think this can be closed? Do let us know if there's anything to do from the infrastructure side... otherwise we will just wait for the gcc build.

Metadata Update from @kevin:
- Issue close_status updated to: Upstream
- Issue status updated to: Closed (was: Open)

If my maths is right, the previous GCC build took just over 13 hours, so we're expecting the current GCC build to finish in the next hour or so.

Well I shouldn't make predictions, but at least it is now writing out the packages on armv7.

gcc-11.0.0-0.16.fc34 should be in the buildroot now

I kicked off this build which was failing before:
https://koji.fedoraproject.org/koji/taskinfo?taskID=60150161
but has already succeeded on s390x, so looking good.
Metadata
None
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.