How are you using bootstrap caps? We're not supporting it at all for now (it is more a future RFE).

My koji builders are EL7, but I'm building Fedora and EPEL8 packages.

I'm just setting config_opts['use_bootstrap_container'] = True in site-defaults.

With Mock v2 using bootstrap containers by default this is probably a bit more interesting....

Yep, looks we will not evade this bullet :-)

We are hitting this issue in production. Over the past week we've filled up our drive three times. Twice after growing it to 250G.

Is there any production workarounds until this issue is resolved?

@praiskup Is there some mock option, which deletes the bootstrap data?

I was able to work-around it by adding: config_opts['use_bootstrap'] = False
to my /etc/mock/site-defaults.cfg.

Only command-line option, mock --scrub bootstrap.

I mean, bootstrap is sort of cache - when cleanup is requested, it takes non-trivial
time to reinitialize it in the next mock command. So it should be cleaned only when
really needed. Since it wastes a lot of space for you, I suppose you are using
something like --uniqueext? There's an upstream question which I'm not sure about ATM.

@praiskup - separate bootstrap dir is created for every build in koji (as separete buildroot per task is created), so space is easily exhausted.

And one more question: Can I expect, that boostrap dir is always /mock_dir/buildroot-bootstrap if /mock_dir/buildroot is buildroot directory?

separete buildroot per task is created

How do you clean the normal buildroot then? Using --scrub=all would work for you
(cleans everything, including bootstrap, and dnf caches).

Can I expect, that boostrap dir is always /mock_dir/buildroot-bootstrap if /mock_dir/buildroot is buildroot directory?

yes

@praiskup There is more complicated policy for that. If build is successful, then after two minutes we delete contents of rootdir. After another 24 hours we delete rootdir itself + mock config. There are some issues with hanging processes and interaction with rpmlib. For failed buildroots there is more complicated policy to have them long enough for debugging. So, it is being done manually outside of mock.

Anyway, at least final step can be done via mock scrub=all + deleting config.

Related code starts here: https://pagure.io/koji/blob/master/f/koji/daemon.py#_727

There are some issues with hanging processes and interaction with rpmlib.

This will be fixed in mock v2.2, merged just
today.
Well, hanging processes were killed by mock for some time - but only those
processes that were started by rpmbuild process (not those started by
packages' scriptlets from rpm --root -i process).

For failed buildroots there is more complicated policy

Ok, in this case you indeed want --scrub bootstrap.

Nevermind, if you miss some functionality/tweak/fix that would help brewkoji here, fill RFE.

PR #2212

@tkopecek I'm curious if we can rely on mock's clean/scrub behavior totally in koji

Yes, but not for the version in epel6. When we drop rhel6 support we can drop also this complicated cleaning code.

Commit 9e4148e6 fixes this issue

Commit 2b930de2 fixes this issue

a trivial follow-up fix: #2372

Commit a04f6413 fixes this issue

Commit 54f047fc fixes this issue

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/1988

Please continue any further discussion there.