Learn more about these different git repos.
Other Git URLs
Reported by @luminoso, being debugged by @frostyx now: https://download.copr.fedorainfracloud.org/results/luminoso/Signal-Desktop/fedora-33-x86_64/01645182-signal-desktop/build-01645182.rsync.log
Warning: Permanently added '3.235.14.207' (ECDSA) to the list of known hosts. receiving incremental file list ./ build.log.gz configs.tar.gz hw_info.log.gz root.log.gz signal-desktop-1.35.1-1.fc33.src.rpm signal-desktop-1.35.1-1.fc33.x86_64.rpm signal-desktop-1.35.1-1.fc34.src.rpm signal-desktop-1.35.1-1.fc34.x86_64.rpm signal.spec state.log.gz success chroot_scan/ chroot_scan/var/ chroot_scan/var/lib/ chroot_scan/var/lib/mock/ chroot_scan/var/lib/mock/fedora-33-x86_64-1599089051.833892/ chroot_scan/var/lib/mock/fedora-33-x86_64-1599089051.833892/root/ chroot_scan/var/lib/mock/fedora-33-x86_64-1599089051.833892/root/var/ chroot_scan/var/lib/mock/fedora-33-x86_64-1599089051.833892/root/var/log/ chroot_scan/var/lib/mock/fedora-33-x86_64-1599089051.833892/root/var/log/dnf.librepo.log chroot_scan/var/lib/mock/fedora-33-x86_64-1599089051.833892/root/var/log/dnf.log chroot_scan/var/lib/mock/fedora-33-x86_64-1599089051.833892/root/var/log/dnf.rpm.log chroot_scan/var/lib/mock/fedora-33-x86_64-1599089153.657560/ chroot_scan/var/lib/mock/fedora-33-x86_64-1599089153.657560/root/ chroot_scan/var/lib/mock/fedora-33-x86_64-1599089153.657560/root/var/ chroot_scan/var/lib/mock/fedora-33-x86_64-1599089153.657560/root/var/log/ chroot_scan/var/lib/mock/fedora-33-x86_64-1599089153.657560/root/var/log/dnf.librepo.log chroot_scan/var/lib/mock/fedora-33-x86_64-1599089153.657560/root/var/log/dnf.log chroot_scan/var/lib/mock/fedora-33-x86_64-1599089153.657560/root/var/log/dnf.rpm.log chroot_scan/var/lib/mock/fedora-rawhide-x86_64-1599088708.098724/ chroot_scan/var/lib/mock/fedora-rawhide-x86_64-1599088708.098724/root/ chroot_scan/var/lib/mock/fedora-rawhide-x86_64-1599088708.098724/root/var/ chroot_scan/var/lib/mock/fedora-rawhide-x86_64-1599088708.098724/root/var/log/ chroot_scan/var/lib/mock/fedora-rawhide-x86_64-1599088708.098724/root/var/log/dnf.librepo.log chroot_scan/var/lib/mock/fedora-rawhide-x86_64-1599088708.098724/root/var/log/dnf.log chroot_scan/var/lib/mock/fedora-rawhide-x86_64-1599088708.098724/root/var/log/dnf.rpm.log sent 487 bytes received 276,793,420 bytes 110,717,562.80 bytes/sec total size is 276,723,874 speedup is 1.00
(it's preferred to track this, so I formally filled the issue)
Backed-up results: https://download.copr.fedorainfracloud.org/results/luminoso/Signal-Desktop/backup-issue-1501-results.tar.gz
I spent a couple of hours on this issue and I got nowhere.
I wanted to go through the backend logs but some of them did break for some reason.
$ gzip -fd *.gz gzip: modifyrepo.log-20200830.gz: unexpected end of file
Do you have any idea if this can be workarounded? I tried zcat and zgrep that somebody on stackoverflow recommended but didn't help. What is strange to me is that about half of the logs decompressed sucessfully and only about half didn't (unfortunatelly the one with worker logs)
zcat
zgrep
I went through the code and tried to figure out some reproducer and the only way I could get e.g. EL7 artifacts into F33 chroot was by running copr-rpmbuild without the --drop-resultdir option. Which IMHO didn't cause the issue because I didn't see any way how copr-backend could run it without the parameter. Also it don't fit into the fact that this issue happened (AFAIK) only for one build but for all chroots in that build.
copr-rpmbuild
--drop-resultdir
Would it be possible for us to add a check (I have it WIP) that we would run before createrepo, to see whether e.g. EPEL7 chroot has only .el7 packages? Or this could break someone's use-cases? If possible, I would probably go this way instead of hunting the bug. If the check finds out this happens repeatedly, we might see some patterns.
Ad decompression, I'd try bsdtar or something else than zcat/zgrep, because those tools just use gzip internall I believe. Anyways, the fact that we have corrupt logs is probably even more severe system problem than what happened in copr.
bsdtar
Would it be possible for us to add a check (I have it WIP) that we would run before createrepo
Well, it is not possible to guess the %dist tag easily - at least not for the chroots that will come in future. I think that simple script going through all the build results would be helpful ... but strictly speaking, there may be "noarch" packages mixed up as well.
I wrote a script for finding all occurrences of this issue across all projects, please see PR#1613
I executed the script from PR#1613 on production so within few days (or possibly weeks? I have no idea how long it is going to take to walk through all of our data) we should see whether we had only one occurrence of this issue or many of them. See more information in our trello card.
Commit 89b97af relates to this ticket
Metadata Update from @praiskup: - Issue assigned to praiskup (was: frostyx)
Commit 44c4ce8 fixes this issue
Log in to comment on this ticket.