We have been seeing this on f34-updates* pushes too.

@dustymabe @walters any ideas here? It looks like /dev/fuse is not in the chroot, but nothing should have changed with mock here...

I checked some of the usual bits, no new bubblewrap or mock release recently, and infra has been in freeze due to F-34 release, it started in rawhide a week or 10 days ago so I'm guessing it's something in f34-updates

Hm, may be fallout from https://github.com/coreos/rpm-ostree/pull/2692 and would probably also be fixed by https://github.com/coreos/rpm-ostree/pull/2772

But... I don't understand how it could work before as we currently depend on fuse. Well...oh right, we aren't using unified core in pungi. OK so that second PR should fix it, I will look at cherry picking.

But... I don't understand how it could work before as we currently depend on fuse. Well...oh right, we aren't using unified core in pungi. OK so that second PR should fix it, I will look at cherry picking.

Comments like "Well...oh right, we aren't using unified core in pungi." aren't particularly useful Colin, literally the first time I saw anything about "unified core" was when I was debugging this.

rawhide/f34 builds running (https://koji.fedoraproject.org/koji/taskinfo?taskID=66812575 for f34)

For more information about unified-core see https://github.com/coreos/rpm-ostree/issues/729

This seems to be hitting f33 as well? Does that line up, or could it mean it's something else?

https://kojipkgs.fedoraproject.org/compose/updates/Fedora-33-updates-20210428.1/logs/x86_64/Everything/ostree-1/create-ostree-repo.log

Right, will backport to f33 too. But...did it get fixed?

Argh right I guess I need to make a bodhi update too because AIUI we use the rpm-ostree in the target (i.e. f34-updates) to build too? Ah OK peter made https://bodhi.fedoraproject.org/updates/FEDORA-2021-d742866eb2

And so that should be used for the next updates-testing run except it looks like https://kojipkgs.fedoraproject.org/compose/updates/Fedora-34-updates-testing-20210428.0/logs/x86_64/Everything/ostree-1/create-ostree-repo.log happened before that.

I haven't found a recent rawhide run.

I think a buildroot override in bodhi will get it used by both updates-testing and updates.

Yeah, rawhide got stuck day before yesterday. I just fired a new compose of it a few min ago...

I can fire a f34-updates-testing to test this? Sound good?

For reference the build override didn't appear to:
https://koji.fedoraproject.org/koji/taskinfo?taskID=66867579

OK, it got farther; let's try https://github.com/coreos/rpm-ostree/pull/2790

Failing that though I may just add some code to forcibly create /dev/fuse and load the module.

(Bigger picture of course there's a whole thing here around unifying build tools and processes; for coroes-assembler we ended up just sticking rpm-ostree in a VM via supermin to boil down our requirements to "host can run standard containers but with /dev/kvm" and no nested containers or privileges. Perhaps we can try to replicate that in koji/mock but it'd be much better to align these builds to use if not coreos-assembler, something like it in approach)

From the releng irc channel:

[12:47:01] <pbrobinson> nirik: don't suppose there's a compose overrides we could put that F-34 NVR in so I can test if it fixes the issue for IoT?
[12:49:15] <+nirik> pbrobinson: yes, I think the composes inherit overrides from buildroot overrides... just make a buildroot override in bodhi?
[12:50:17] <pbrobinson> nirik: OK, I had already put it in there, I thought composes may have been uncoupled from standard overrides because they were getting pulled into RCs and causing issues but I'm out of the loop there TBH
[12:52:15] <+nirik> I could be wrong on them pulling overrides, but I think thats the case.
[13:11:10] <pbrobinson> nirik: for reference it doesn't appear to AFAICT, maybe bodhi does it differently though
[13:11:32] <+nirik> :(
[13:31:05] <+mboddu> pbrobinson: Whats the build?
[13:31:31] <pbrobinson> mboddu: rpm-ostree-2021.4-2.fc34
[13:36:14] <+mboddu> pbrobinson: Its in the compose - https://kojipkgs.fedoraproject.org/compose/iot/Fedora-IoT-34-20210428.0/compose/IoT/source/tree/Packages/r/ and also in the buildroot (f34-build) which is what you are using for runroot tag in your pungi config
[13:36:26] <+mboddu> So, whats the problem?
[13:37:32] <pbrobinson> mboddu: TBH no idea, could you add that into the ticket so Colin can reply?

So, whats the problem here?

We need another build, working on it. We can talk about tagging/overrides after that.

We need another build, working on it. We can talk about tagging/overrides after that.

ack, I guess override and tagging to f34-iot should fix the compose issue.

OK build coming for rawhide/34 (https://koji.fedoraproject.org/koji/taskinfo?taskID=66876196 ) and 33 that should work.

I thought it should show up in the updates-testing run at least at https://kojipkgs.fedoraproject.org/compose/updates/Fedora-34-updates-testing-20210429.0/logs/x86_64/Everything/ostree-1/
but it seemed to get the old one.

I tried making a buildroot override:
https://bodhi.fedoraproject.org/overrides/rpm-ostree-2021.4-3.fc34

Let's try another run for at least updates-testing?

f34-updates-testing push started.
https://kojipkgs.fedoraproject.org/compose/updates/Fedora-34-updates-testing-20210429.0/

https://kojipkgs.fedoraproject.org/compose/updates/Fedora-34-updates-testing-20210429.1/logs/x86_64/Everything/ostree-1/create-ostree-repo.log
worked. Will https://bodhi.fedoraproject.org/updates/FEDORA-2021-d742866eb2 automatically be used for the next updates (non-testing) push or does it need to go stable first, i.e. is it two days or one? Or can we just kick off an updates push now?

It will definitely use it if it's stable...
I sure think it will use it if it's an override. The runroot task in koji runs with f34-build...

So, I think we can do a f34-updates push now.

Is there a f33 update?

https://bodhi.fedoraproject.org/updates/FEDORA-2021-b724063bdc for 33 and made a buildroot override too.

Cool. Started a f34-updates. If that goes, I'll push f33 (both) and hopefully we will be back on track. Thanks for the quick fixes @walters !

f34-updates finished fine.

f33 pushing now.

Finished fine. :)

I think we are all back on track now... many thanks for the assist.

Feel free to re-open this is there's anything more for us to do.