I've been running something like the above for loop weekly. There was something merged into the koji upstream code base, but I never followed it enough to know if it got into a release and deployed. I guess we know our answer now.

We are running 1.35.3 now, and the fix was supposed to be in there I thought?

so we should stop running distrepos manually and the next time one disappears we investigate with you?

How about stopping doing them manually... then the next time you see latest disappear let me know the exact time/date... I can then look up logs and add them here and we can hopefully figure out if it's a fedora instance issue or a upstream one. ;) Then you can regen it to unblock, just let me know when the time it happens is so I can look at logs.

Here are the current checks that Koji does through before marking a repo expired.

1. If the repo is current, i.e. it represents the current state of the corresponding tag, then the repo will remain in the ready state.
2. If the age of the repo is less than the configured lifetime, then the repo will remain in the ready state. (For dist repos, this is the dist_repo_lifetime setting. Fedora sets this to 7 days, which is same as default).
3. If the repo is the latest for its tag, then it will remain in the ready state. I.e. if there is no newer repo of the same type (for dist repos, only only other dist repos are considered).

As long as a repo remains in the ready state, it will not be deleted. When repos go into the expired state, they are marked for deletion and will normally be deleted after the expired_repo_lifetime setting (or the deprecated deleted_repo_lifetime alias setting). Fedora has this set to 3 days, which is shorter than the upstream default, but still a reasonable safety net.

Further reading -- https://docs.pagure.org/koji/repo_generation/#repository-lifecycle

~~Your subject is "kojiria gc'ed latest repos-dist of f42-coreos-continuous", however the link you provide shows this is not actually the case. The latest dist repos for this tag are still present.~~

At this time, the most recently deleted repo for this tag is this one -- https://koji.fedoraproject.org/koji/repoinfo?repoID=6575011

This dist repo was generated May 14, two weeks ago. The three remaining repos are all newer.

A little more digging.

The kojira instance in Koji was updated to 1.35.3 on May 22. The most recently deleted repo for this tag was marked expired on May 21, one day before, using the older 1.35.2 code. At that time, it appears to have been the latest dist repo for the tag. The new code would not have done this.

The protections we added do not apply to expired repos, which are already considered marked for deletion. Going forward, you should see the results of the changes in koji-1.35.3, but any repos marked expired before the update will be deleted after the lifetime setting.

Is there a command we can run to see the expired_repo_lifetime and dist_repo_lifetime settings for our repos ?

Is there a command we can run to see the expired_repo_lifetime and dist_repo_lifetime settings for our repos ?

No. These are settings for the kojira daemon. They apply to all repos.

When I looked earlier today, the Fedora kojira.conf had

dist_repo_lifetime = 604800
deleted_repo_lifetime = 259200

So, 7 days and 3 days respectively.

If the repo is current, i.e. it represents the current state of the corresponding tag, then the repo will remain in the ready state.

I assume this means that the latest dist-repo for a given tag should always be in the ready state, unless the tag has been modified for whatever reason?

I assume this means that the latest dist-repo for a given tag should always be in the ready state, unless the tag has been modified for whatever reason?

If the repo is latest, it will not exit the ready state from the actions of Kojira.

If the repo was already expired by the old code, as in the case above, then it will be deleted after a timeout whether it is latest or not.

Anyway, afaict this is a transitional issue stemming from repos that were expired using the old code. Since we are more than 3 days past the update, any repos expired that way should already be deleted.

From this point we can expect the latest repo (dist and regular) for any tag to remain indefinitely.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/4401

Please continue any further discussion there.