Learn more about these different git repos.
Other Git URLs
Finish: clean chroot Traceback (most recent call last): File "/usr/libexec/mock/mock", line 1060, in <module> exitStatus = main() File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 826, in main result = run_command(options, args, config_opts, commands, buildroot, state) File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 940, in run_command mockbuild.rebuild.do_rebuild(config_opts, commands, buildroot, options, srpms) File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace result = func(*args, **kw) File "/usr/lib/python3.9/site-packages/mockbuild/rebuild.py", line 85, in do_rebuild rebuild_generic(srpms, commands, buildroot, config_opts, cmd=build, File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace result = func(*args, **kw) File "/usr/lib/python3.9/site-packages/mockbuild/rebuild.py", line 24, in rebuild_generic commands.init(prebuild=not config_opts.get('short_circuit')) File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace result = func(*args, **kw) File "/usr/lib/python3.9/site-packages/mockbuild/backend.py", line 166, in init self.bootstrap_buildroot.initialize(**kwargs) File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace result = func(*args, **kw) File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 134, in initialize self._init(prebuild=prebuild, do_log=do_log) File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace result = func(*args, **kw) File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 169, in _init self.plugins.call_hooks('preinit') File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace result = func(*args, **kw) File "/usr/lib/python3.9/site-packages/mockbuild/plugin.py", line 78, in call_hooks hook(*args, **kwargs) File "/usr/lib/python3.9/site-packages/mockbuild/plugins/hw_info.py", line 60, in _PreInitHook self._unprivPreInitHook() File "/usr/lib/python3.9/site-packages/mockbuild/plugins/hw_info.py", line 55, in _unprivPreInitHook out.close() OSError: [Errno 28] No space left on device
See e.g. here: https://copr.fedorainfracloud.org/coprs/g/copr/copr-dev/build/2828877/
I tested locally, and I don't seem to have problems ... with:
rpm-4.16.1.3-1.fc34.x86_64 dnf-4.8.0-1.fc34.noarch mock-2.12-1.git.14.ed5231a.fc34.noarch mock-core-configs-36-1.git.0.2851ad7.fc34.noarch qemu-user-static-5.2.0-8.fc34.x86_64
Do we need the builder images' update?
Metadata Update from @praiskup: - Issue priority set to: High
I am looking into this right now. I executed:
# copr-rpmbuild --verbose --drop-resultdir --build-id 2831750 --chroot fedora-rawhide-ppc64le
Source
The build is progressing nicely even though I'm confused about the filesystem organization. I think all the data should be placed in /var/lib/rpm-coprbuild where the space is abundant:
/var/lib/rpm-coprbuild
Filesystem Size Used Avail Use% Mounted on /dev/vdb1 16G 71M 15G 1% /var/lib/copr-rpmbuild
But instead, the workplace of the build is the standard /var/lib/mock where the space is limited:
/var/lib/mock
[root@fedora ~]# cd /var/lib/mock [root@fedora mock]# ls -lha celkom 16K drwxrwsr-x. 4 root mock 4.0K Sep 21 11:50 . drwxr-xr-x. 29 root root 4.0K Jul 13 12:08 .. drwxrwsr-x. 3 root mock 4.0K Sep 21 11:50 fedora-rawhide-ppc64le-1632224894.965045 drwxrwsr-x. 4 root mock 4.0K Sep 21 11:50 fedora-rawhide-ppc64le-bootstrap-1632224894.965045 [root@fedora mock]# df -h . Filesystem Size Used Avail Use% Mounted on /dev/vda2 5.5G 2.4G 2.9G 45% /
I also don't understand the need for ~140G of swap:
[root@fedora copr-rpmbuild]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1M 0 rom /config vda 252:0 0 5.6G 0 disk ├─vda1 252:1 0 4M 0 part └─vda2 252:2 0 5.6G 0 part / vdb 252:16 0 156.5G 0 disk ├─vdb1 252:17 0 16G 0 part /var/lib/copr-rpmbuild └─vdb2 252:18 0 140.5G 0 part [SWAP] [root@fedora copr-rpmbuild]# free -h total used free shared buff/cache available Mem: 4.0Gi 703Mi 159Mi 3.0Gi 3.1Gi 248Mi Swap: 140Gi 1.5Gi 139Gi
I executed 4 different builds on the same builder, sequentially and wasn't able to reproduce.
In the end, none of the disks was running out of space, nor memory.
Indeed, it seems it has begun working again, per recent builds in: https://pagure.io/copr/copr/pull-request/1921
I can't explain what happened honestly. Hopefully, it will not regress back.
Btw., there still were many allocated ppc64le builders from the last week (last week I dropped only the orphaned x86_64 builders, not ppc64le). So I removed them now (shoudl increase the ppc64le throughput). But that doesn't seem to be related to this issue honestly. Dunno.
Thank you for the help Tomáš anyway. May I deallocate the testing VM?
I'd have no idea what to try next (probably a stress test?), so if things are getting back to normal let's treat this as fixed.
Thanks for the reply Pavel!
Metadata Update from @praiskup: - Issue close_status updated to: Insufficient data - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.