I am trying to update 0ad flatpak but the builds are running out of disk space (one of the rpms involved, 0ad-data is 1.3 GB). Would it be possible to increase the disk space available in the OSBS builders, please?
$ fedpkg flatpak-build Created task: 73543190 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=73543190 Watching tasks (this may be safely interrupted)... 73543190 buildContainer (noarch): free 73543190 buildContainer (noarch): free -> open (buildvm-x86-01.iad2.fedoraproject.org) 73543190 buildContainer (noarch): open (buildvm-x86-01.iad2.fedoraproject.org) -> FAILED: Fault: <Fault 2001: 'Image build failed. Error in plugin orchestrate_build: {"x86_64": {"flatpak_create_oci": "[Errno 28] No space left on device"}}. OSBS build id: 0ad-stable-079e0-5'> 0 free 0 open 0 done 1 failed 73543190 buildContainer (noarch) failed $ fedpkg flatpak-build Created task: 73543836 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=73543836 Watching tasks (this may be safely interrupted)... 73543836 buildContainer (noarch): free 73543836 buildContainer (noarch): free -> open (buildvm-x86-02.iad2.fedoraproject.org) 73543836 buildContainer (noarch): open (buildvm-x86-02.iad2.fedoraproject.org) -> FAILED: Fault: <Fault 2001: 'Image build failed. Error in plugin orchestrate_build: {"x86_64": {"flatpak_create_oci": "[Errno 28] No space left on device"}}. OSBS build id: 0ad-stable-079e0-6'> 0 free 0 open 0 done 1 failed 73543836 buildContainer (noarch) failed
Metadata Update from @humaton: - Issue tagged with: low-trouble, medium-gain, ops
Metadata Update from @mohanboddu: - Issue untagged with: low-trouble - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-trouble
So, we can of course grow the disk... but it seems like we should be able to clean up some images?
/dev/vda5 vg-docker lvm2 a-- 58.90g 468.00m
REPOSITORY TAG IMAGE ID CREATED SIZE test latest 0a0baa296cee 4 days ago 177 MB <none> <none> b51daf0b1804 4 days ago 177 MB <none> <none> 9c5044229d52 4 days ago 172 MB <none> <none> 1bbbc3abe710 4 days ago 172 MB <none> <none> 5f5349d429c7 4 days ago 172 MB <none> <none> dd701b5dc41f 4 days ago 172 MB registry.fedoraproject.org/fedora 35 3c159d48f2a4 5 days ago 172 MB registry.fedoraproject.org/fedora 34 28317703decd 2 weeks ago 178 MB registry.fedoraproject.org/fedora latest 28317703decd 2 weeks ago 178 MB buildroot latest 70a86033f184 3 months ago 1.11 GB <none> <none> 873d9f2ad6b3 3 months ago 1.11 GB <none> <none> 9bca77042f26 3 months ago 1.11 GB <none> <none> cfcc485f145c 3 months ago 1.11 GB <none> <none> fed1d2f4c016 3 months ago 1.11 GB <none> <none> e00ebc8dce01 3 months ago 1.11 GB <none> <none> 22ad1f13f661 3 months ago 1.11 GB <none> <none> 744124faae21 3 months ago 1.11 GB <none> <none> d1c577c3635f 3 months ago 1.11 GB buildroot old e61b60c9f166 4 months ago 1.11 GB <none> <none> 1b73f3142036 4 months ago 1.11 GB <none> <none> b41ceb428098 4 months ago 1.11 GB <none> <none> fdbff1b87ce9 4 months ago 1.11 GB <none> <none> 5ff56cd51628 4 months ago 1.11 GB <none> <none> b4a23decbd43 4 months ago 1.11 GB <none> <none> d3f63e21d1c2 4 months ago 1.11 GB <none> <none> f07537254f0f 4 months ago 1.11 GB <none> <none> 0d114dcc6594 4 months ago 1.11 GB registry.fedoraproject.org/fedora 33 79fd58dc7611 9 months ago 175 MB docker.io/fedora latest a368cbcfa678 13 months ago 182 MB buildroot backup 8597d5152533 13 months ago 1.08 GB <none> <none> 09de5bafc1d8 13 months ago 1.08 GB <none> <none> 226b826e582b 13 months ago 1.08 GB <none> <none> e23d0a35195f 13 months ago 1.08 GB <none> <none> 9544ec797bc5 13 months ago 1.08 GB <none> <none> 0100d80970d4 13 months ago 1.08 GB <none> <none> 0600983745f0 13 months ago 1.08 GB <none> <none> 274f177e75e2 13 months ago 1.08 GB <none> <none> 62ff02f922cc 13 months ago 1.08 GB registry.redhat.io/openshift3/ose-node v3.11 1ea6ff15c000 14 months ago 1.21 GB registry.redhat.io/openshift3/ose-pod v3.11 6acdd3267a6f 14 months ago 254 MB registry.redhat.io/openshift3/ose-docker-registry v3.11 a3fef146e51e 14 months ago 305 MB registry.fedoraproject.org/fedora 31 a7a37f74ff86 15 months ago 192 MB registry.redhat.io/openshift3/ose-pod v3.11.216 da5e7389d54b 15 months ago 253 MB
I am not sure what all the `<none>' images are but I think that should be safe to delete :)
I think running docker rmi $(docker images -f dangling=true -q) should take care of it
docker rmi $(docker images -f dangling=true -q)
I thought so too, but:
Error response from daemon: conflict: unable to delete 62ff02f922cc (cannot be forced) - image has dependent child images
docker images -f dangling=true -q returns nothing at all.
Is there anything else we can try? I don't know much about docker, but how about just killing all of its image cache? I assume it should re-download the images it needs from the registry?
ok. I tried removing the old buildroot image and perhaps that freed enough space? Can you give it a try now?
Still runs out of space :(
$ fedpkg flatpak-build Created task: 74513659 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=74513659 Watching tasks (this may be safely interrupted)... 74513659 buildContainer (noarch): free 74513659 buildContainer (noarch): free -> open (buildvm-x86-02.iad2.fedoraproject.org) 74513659 buildContainer (noarch): open (buildvm-x86-02.iad2.fedoraproject.org) -> FAILED: Fault: <Fault 2001: 'Image build failed. Error in plugin orchestrate_build: {"x86_64": {"flatpak_create_oci": "[Errno 28] No space left on device"}}. OSBS build id: 0ad-stable-079e0-11'> 0 free 0 open 0 done 1 failed 74513659 buildContainer (noarch) failed
ok. I added (I hope correctly) 120GB to both nodes.
Can you try again now?
Still out of space :(
I did 3 build attempts. First one failed due to an unrelated dep issue in flatpak-runtime, which I've fixed, and then the last two failed due to out of space again.
$ fedpkg flatpak-build Created task: 74543579 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=74543579 Watching tasks (this may be safely interrupted)... 74543579 buildContainer (noarch): free 74543579 buildContainer (noarch): free -> open (buildvm-x86-02.iad2.fedoraproject.org) 74543579 buildContainer (noarch): open (buildvm-x86-02.iad2.fedoraproject.org) -> FAILED: Fault: <Fault 2001: 'Image build failed. Error in plugin orchestrate_build: {"x86_64": {"docker_api": "The command \'/bin/sh -c cat /tmp/atomic-reactor-includepkgs >> /etc/dnf/dnf.conf && INSTALLDIR=/var/tmp/flatpak-build && DNF=\' dnf -y --nogpgcheck --disablerepo=* --enablerepo=atomic-reactor-koji-plugin-* --enablerepo=atomic-reactor-module-* \' && $DNF --installroot=$INSTALLDIR-init install system-release && mkdir -p $INSTALLDIR/etc/ && cp $INSTALLDIR-init/etc/os-release $INSTALLDIR/etc/os-release && $DNF --installroot=$INSTALLDIR module enable 0ad:stable flatpak-common:f34 flatpak-runtime:f34 && $DNF --installroot=$INSTALLDIR install 0ad flatpak-runtime-config\' returned a non-zero code: 1"}, "aarch64": {"docker_api": "The command \'/bin/sh -c cat /tmp/atomic-reactor-includepkgs >> /etc/dnf/dnf.conf && INSTALLDIR=/var/tmp/flatpak-build && DNF=\' dnf -y --nogpgcheck --disablerepo=* --enablerepo=atomic-reactor-koji-plugin-* --enablerepo=atomic-reactor-module-* \' && $DNF --installroot=$INSTALLDIR-init install system-release && mkdir -p $INSTALLDIR/etc/ && cp $INSTALLDIR-init/etc/os-release $INSTALLDIR/etc/os-release && $DNF --installroot=$INSTALLDIR module enable 0ad:stable flatpak-common:f34 flatpak-runtime:f34 && $DNF --installroot=$INSTALLDIR install 0ad flatpak-runtime-config\' returned a non-zero code: 1"}}. OSBS build id: 0ad-stable-079e0-12'> 0 free 0 open 0 done 1 failed 74543579 buildContainer (noarch) failed $ fedpkg flatpak-build Created task: 74549013 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=74549013 Watching tasks (this may be safely interrupted)... 74549013 buildContainer (noarch): free 74549013 buildContainer (noarch): free -> open (buildvm-x86-01.iad2.fedoraproject.org) 74549013 buildContainer (noarch): open (buildvm-x86-01.iad2.fedoraproject.org) -> FAILED: Fault: <Fault 2001: 'Image build failed. Error in plugin orchestrate_build: {"x86_64": {"flatpak_create_oci": "[Errno 28] No space left on device"}}. OSBS build id: 0ad-stable-079e0-13'> 0 free 0 open 0 done 1 failed 74549013 buildContainer (noarch) failed $ fedpkg flatpak-build Created task: 74549182 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=74549182 Watching tasks (this may be safely interrupted)... 74549182 buildContainer (noarch): free 74549182 buildContainer (noarch): free -> open (buildvm-x86-04.iad2.fedoraproject.org) 74549182 buildContainer (noarch): open (buildvm-x86-04.iad2.fedoraproject.org) -> FAILED: Fault: <Fault 2001: 'Image build failed. Error in plugin orchestrate_build: {"x86_64": {"flatpak_create_oci": "[Errno 28] No space left on device"}}. OSBS build id: 0ad-stable-079e0-14'> 0 free 0 open 0 done 1 failed 74549182 buildContainer (noarch) failed
So weird. ;(
{"log":"Commit: 8d3f2389d27147a308a8f81073607cfc6e9fe07a5ea65084abf6baab7e84f70f\n","stream":"stdout","time":"20 21-08-26T10:25:51.957095319Z"} {"log":"Metadata Total: 337\n","stream":"stdout","time":"2021-08-26T10:25:51.957125275Z"} {"log":"Metadata Written: 160\n","stream":"stdout","time":"2021-08-26T10:25:51.957128635Z"} {"log":"Content Total: 367\n","stream":"stdout","time":"2021-08-26T10:25:51.95713087Z"} {"log":"Content Written: 355\n","stream":"stdout","time":"2021-08-26T10:25:51.957133081Z"} {"log":"Content Bytes Written: 3351166328 (3.4 GB)\n","stream":"stdout","time":"2021-08-26T10:25:51.957135143Z"} {"log":"2021-08-26 10:28:55,618 platform:- - atomic_reactor.plugin - DEBUG - Traceback (most recent call last):\ n","stream":"stderr","time":"2021-08-26T10:28:55.619820091Z"} {"log":" File \"/usr/lib/python3.9/site-packages/atomic_reactor/plugin.py\", line 268, in run\n","stream":"stde rr","time":"2021-08-26T10:28:55.619947657Z"} {"log":" plugin_response = plugin_instance.run()\n","stream":"stderr","time":"2021-08-26T10:28:55.619952525Z" } {"log":" File \"/usr/lib/python3.9/site-packages/atomic_reactor/plugins/prepub_flatpak_create_oci.py\", line 116, in run\n","stream":"stderr","time":"2021-08-26T10:28:55.619955077Z"} {"log":" ref_name, outfile, tarred_outfile = self.builder.build_container(tarred_filesystem)\n","stream":"stderr","time":"2021-08-26T10:28:55.619957771Z"} {"log":" File \"/usr/lib/python3.9/site-packages/flatpak_module_tools/flatpak_builder.py\", line 945, in build_container\n","stream":"stderr","time":"2021-08-26T10:28:55.619960205Z"} {"log":" tf.add(os.path.join(outfile, f), f)\n","stream":"stderr","time":"2021-08-26T10:28:55.619962901Z"} {"log":" File \"/usr/lib64/python3.9/tarfile.py\", line 1983, in add\n","stream":"stderr","time":"2021-08-26T10:28:55.619965046Z"} {"log":" self.add(os.path.join(name, f), os.path.join(arcname, f),\n","stream":"stderr","time":"2021-08-26T10:28:55.619967281Z"} {"log":" File \"/usr/lib64/python3.9/tarfile.py\", line 1983, in add\n","stream":"stderr","time":"2021-08-26T10:28:55.619972543Z"} {"log":" self.add(os.path.join(name, f), os.path.join(arcname, f),\n","stream":"stderr","time":"2021-08-26T10:28:55.619974759Z"} {"log":" File \"/usr/lib64/python3.9/tarfile.py\", line 1977, in add\n","stream":"stderr","time":"2021-08-26T10:28:55.619977124Z"} {"log":" self.addfile(tarinfo, f)\n","stream":"stderr","time":"2021-08-26T10:28:55.619979663Z"} {"log":" File \"/usr/lib64/python3.9/tarfile.py\", line 2005, in addfile\n","stream":"stderr","time":"2021-08-26T10:28:55.619981967Z"} {"log":" copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)\n","stream":"stderr","time":"2021-08-26T10:28:55.619984651Z"} {"log":" File \"/usr/lib64/python3.9/tarfile.py\", line 250, in copyfileobj\n","stream":"stderr","time":"2021-08-26T10:28:55.619986983Z"} {"log":" dst.write(buf)\n","stream":"stderr","time":"2021-08-26T10:28:55.619989411Z"} {"log":"OSError: [Errno 28] No space left on device\n","stream":"stderr","time":"2021-08-26T10:28:55.619991476Z"}
It's like it finishes then errors... it's only the x86_64 one, the aarch64 one finishes fine.
docker info says:
Data Space Used: 25.67 GB Data Space Total: 190.1 GB Data Space Available: 164.4 GB Metadata Space Used: 9.867 MB Metadata Space Total: 67.11 MB Metadata Space Available: 57.24 MB Thin Pool Minimum Free Space: 19 GB
I do see both of those failed on osbs-node01. ;( I've cordoned it off, can you try again and see if they complete on osbs-node02?
Now it's error 401:
$ fedpkg flatpak-build Created task: 74588101 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=74588101 Watching tasks (this may be safely interrupted)... 74588101 buildContainer (noarch): free 74588101 buildContainer (noarch): free -> FAILED: Fault: <Fault 2001: 'Image build failed. Error in plugin resolve_module_compose: 401 Client Error: Unauthorized for url: https://odcs.fedoraproject.org/api/1/composes/. OSBS build id: 0ad-stable-079e0-18'> 0 free 0 open 0 done 1 failed 74588101 buildContainer (noarch) failed
I retried several times just to make sure and it's hitting the same error every time.
If I look at other builds that worked, it looks like it's trying to write to /tmp when it's running out of space. Is that on the same file system or is it a memory-backed file system?
I'm running the playbook over the cluster to see if it's just something misconfigured.
/tmp is on the main fs:
Filesystem Size Used Avail Use% Mounted on /dev/mapper/GuestVolGroup00-root 59G 5.5G 54G 10% /
I don't think that helped, but you can try some more builds to confirm. :(
perhaps @cverna or @mobrien or @dkirwan can see whats going on here?
I tried a few more builds of two different flatpaks and I'm just getting 401 errors now.
Thanks for the help in any case!
ok, got this figured out.
First the odcs token for osbs was expired (thus the 401's...) so I extended it's expiration date.
Next the out of space problem was because docker defaults to 10GB for containers and this build went over that.
So, I edited /etc/sysconfig/docker-storage on osbs-node01/osbs-node02 and added: "--storage-opt dm.basesize=20G" restarted docker and then rebuilt the build container with the playbook. That got things building. :)
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.