After some investigation we are restoring content from a snapshot.

We have also disabled the updates-sync and ostree-pruner. These are the only 2 scripts that touch on that repo.

This may be related to the bodhi-backend01 upgrade/reinstall earlier today, but it's unclear what actually might have caused this.

I don't think this is the updates-sync script. It only does ostree pull-local and ostree summary --update, neither of which should trigger a prune.

As to the ostree pruner, AFAICT it's not actually fully online yet. It seems like the pod just sleeps for now, and @dustymabe 's PR to add it is still pending (https://github.com/coreos/fedora-coreos-releng-automation/pull/79).

So, I suspect there might've been something else going on here. Based on the fact whole subdirs from the objects/ dir are gone, I'd say something not ostree aware.

If need be for coreos we can pretty easily get the objects from our cached ostree tarballs. For example, https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210426.3.0/x86_64/fedora-coreos-33.20210426.3.0-ostree.x86_64.tar is a tarball of an archive mode repo of the latest stable update. So, something like this:

$ mkdir repo
$ cd repo
$ curl -L https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210426.3.0/x86_64/fedora-coreos-33.20210426.3.0-ostree.x86_64.tar | tar xf -
$ ostree --repo=/path/to/prod/repo pull-local . 

AFAIK IoT and Silverblue do not have this, but in the end forcing a new build (with e.g. --force-nocache, I think there's a way to do that with pungi) will add things back and clients will be able to update to it fine (history doesn't matter for updates).

Seems to be resolved at least for SB.

The only change I can think of is that recently I re-deployed the coreos-ostree-importer because i updated the container image to f34 and changed the branch names for the upstream repo to main.

The importer itself doesn't prune anything, though. So I would be surprised if that was the source, unless there is a serious bug in either F34 or newer rpm-ostree that is causing data to get deleted.

So, status update:
We have completed our restore from snapshot. The repo should be fine again as of 4-6 hours ago.
We have manually run the updates-sync script and it worked fine with no unexpected output or errors.
I have now re-enabled updates pushes, should go out here soon.
Once updates are all pushed out, I am going to start a 'ostree fsck' on the prod repo

We are still at a loss as to root cause here. I can't think of anything that would just delete part of the ostree repo and nothing else. ;(

@humaton / @asaleh can you think of anything at all that could have caused this during the bodhi-backend01 re-install? I mean, the ftpsync user had the wrong uid, but that would just cause it to get permission denied I would think, no way it could delete anything. Very puzzling. ;(

So, the ostree fsck took weeks, but it did finish without errors. So I guess we are all ok now...

Thanks for checking this!