fedora_nightlies

#8 remove old images
Closed: Fixed by adamwill. Opened by kparal.

Closed: Fixed
Reopen Issue

When looking at https://www.happyassassin.net/nightlies.html , there are some old images that should be pruned, because they are no longer composed. Let's avoid people downloading them by mistake, or just making the proper images hard to find. I found these to be obsolete:

Everything boot
ppc64   Fedora-Rawhide-20180810.n.0 
i386    Fedora-Rawhide-20190719.n.1
Workstation dvd-ostree
x86_64  Fedora-Rawhide-20170907.n.0
Workstation live
i386    Fedora-Rawhide-20190712.n.0
Workstation boot
aarch64     Fedora-Rawhide-20190808.n.0
armhfp  Fedora-Rawhide-20190808.n.0 
i386    Fedora-Rawhide-20190712.n.0
x86_64  Fedora-Rawhide-20190808.n.0
Server qcow2
ppc64le     Fedora-Modular-Rawhide-20171003.n.0     
aarch64     Fedora-Modular-Rawhide-20171003.n.0     
x86_64  Fedora-Modular-Rawhide-20171003.n.0 
Server raw-xz
ppc64le     Fedora-Modular-Rawhide-20171003.n.0 
x86_64  Fedora-Modular-Rawhide-20171003.n.0 
Server dvd
ppc64   Fedora-Rawhide-20180810.n.0     
i386    Fedora-Rawhide-20190712.n.0
Server boot
ppc64   Fedora-Rawhide-20180810.n.0 
i386    Fedora-Rawhide-20190712.n.0
Cloud_Base qcow2
ppc64   Fedora-Rawhide-20180810.n.0     
i386    Fedora-Rawhide-20160322.n.0 
Cloud_Base raw-xz
ppc64   Fedora-Rawhide-20180810.n.0     
i386    Fedora-Rawhide-20160322.n.0 
Cloud_Base vagrant-libvirt
i386    Fedora-Rawhide-20160322.n.0 
Cloud_Base vagrant-virtualbox
i386    Fedora-Rawhide-20160322.n.0 
Container_Minimal_Base docker
armhfp  Fedora-Rawhide-20190312.n.1     
ppc64   Fedora-Rawhide-20170903.n.0     
Docker_Base docker
ppc64le     Fedora-Rawhide-20171108.n.0     
aarch64     Fedora-Rawhide-20171108.n.0     
armhfp  Fedora-Rawhide-20171108.n.0     
ppc64   Fedora-Rawhide-20170903.n.0     
x86_64  Fedora-Rawhide-20171108.n.0 
Atomic qcow2
ppc64le     Fedora-Rawhide-20171114.n.0     
aarch64     Fedora-Rawhide-20180309.n.0     
x86_64  Fedora-Rawhide-20180309.n.0 
Atomic raw-xz
ppc64le     Fedora-Rawhide-20171114.n.0     
aarch64     Fedora-Rawhide-20180309.n.0     
x86_64  Fedora-Rawhide-20180309.n.0 
Cloud boot
ppc64le     Fedora-Rawhide-20180310.n.0     
aarch64     Fedora-Rawhide-20180310.n.0     
ppc64   Fedora-Rawhide-20180310.n.0     
i386    Fedora-Rawhide-20180220.n.0     
x86_64  Fedora-Rawhide-20180310.n.0     
s390x   Fedora-Rawhide-20180310.n.0 
Atomic dvd-ostree
ppc64le     Fedora-Rawhide-20180301.n.0     
aarch64     Fedora-Rawhide-20180301.n.0     
x86_64  Fedora-Rawhide-20180301.n.0 
Atomic vagrant-libvirt
x86_64  Fedora-Rawhide-20180309.n.0 
Atomic vagrant-virtualbox
x86_64  Fedora-Rawhide-20180309.n.0 
Container_Base docker
armhfp  Fedora-Rawhide-20190312.n.1 
KDE live
i386    Fedora-Rawhide-20190712.n.0
Xfce live
i386    Fedora-Rawhide-20190711.n.1 
SoaS live
i386    Fedora-Rawhide-20190712.n.0 
Mate live
i386    Fedora-Rawhide-20190712.n.0 
Cinnamon live
i386    Fedora-Rawhide-20190712.n.0 
LXDE live
i386    Fedora-Rawhide-20190711.n.1
Container_Minimal docker
ppc64le     Fedora-Modular-Rawhide-20171002.n.1     
aarch64     Fedora-Modular-Rawhide-20171002.n.1     
armhfp  Fedora-Modular-Rawhide-20171002.n.1     
x86_64  Fedora-Modular-Rawhide-20171002.n.1 
WorkstationOstree dvd-ostree
x86_64  Fedora-Rawhide-20180301.n.0
Modular boot
ppc64le     Fedora-Rawhide-20180314.n.2     
aarch64     Fedora-Rawhide-20180314.n.2     
ppc64   Fedora-Rawhide-20180314.n.2     
i386    Fedora-Rawhide-20180314.n.2     
x86_64  Fedora-Rawhide-20180314.n.2     
s390x   Fedora-Rawhide-20180314.n.2 
Scientific vagrant-libvirt
i386    Fedora-Rawhide-20180723.n.0     
x86_64  Fedora-Rawhide-20190725.n.0 
Scientific vagrant-virtualbox
i386    Fedora-Rawhide-20180723.n.0     
x86_64  Fedora-Rawhide-20190725.n.0 
Scientific_KDE live
i386    Fedora-Rawhide-20180726.n.2     
x86_64  Fedora-Rawhide-20190725.n.0 
AtomicWorkstation dvd-ostree
x86_64  Fedora-Rawhide-20180810.n.1 
Robotics live
i386    Fedora-Rawhide-20180919.n.0 
AtomicHost qcow2
ppc64le     Fedora-Rawhide-20181027.n.0     
aarch64     Fedora-Rawhide-20190203.n.0     
x86_64  Fedora-Rawhide-20190214.n.0
AtomicHost raw-xz
ppc64le     Fedora-Rawhide-20181027.n.0     
aarch64     Fedora-Rawhide-20190203.n.0     
x86_64  Fedora-Rawhide-20190214.n.0 
AtomicHost vagrant-libvirt
x86_64  Fedora-Rawhide-20190214.n.0
Jam_KDE live
i386    Fedora-Rawhide-20181203.n.0 
AtomicHost vagrant-virtualbox
x86_64  Fedora-Rawhide-20190214.n.0 
AtomicHost dvd-ostree
ppc64le     Fedora-Rawhide-20190217.n.0     
aarch64     Fedora-Rawhide-20190217.n.0     
x86_64  Fedora-Rawhide-20190217.n.0 
Python_Classroom live
i386    Fedora-Rawhide-20190504.n.0
Astronomy_KDE live
i386    Fedora-Rawhide-20190529.n.0 
Security live
i386    Fedora-Rawhide-20190529.n.0 
Python_Classroom vagrant-libvirt
i386    Fedora-Rawhide-20190709.n.1 
Python_Classroom vagrant-virtualbox
i386    Fedora-Rawhide-20190709.n.1 
Design_suite live
i386    Fedora-Rawhide-20190712.n.0 
Games live
i386    Fedora-Rawhide-20190712.n.0 
LXQt live
i386    Fedora-Rawhide-20190712.n.0

because of how nightlies works, this is slightly tricky to do, which is why I haven't done it till now. there is an 'easy' way to do it, which is to just wipe the data and start over, but then we lose all the 'last known goods' that are valid. We also lose all actual images from the table entirely until they have built successfully at least one time.

It's impossible or very difficult to do this 100% correctly, because there's no easy way to distinguish between "image is obsolete" and "image just hasn't built successfully for ages" - there's nothing in the productmd metadata that indicates the latter, you have to go parse the pungi-fedora config to figure out what it's currently intended to try and build, and even then you don't know if a given image has been taken out of the config temporarily or permanently. So we'd just have to go with a dumb age-based cutoff, or something: dump image if it hasn't built successfully for X weeks. Either that or just write a temp script to do it based on human knowledge.
Edited by adamwill

Age-based cutoff seems the best possible option to me. All those images contain a timestamp, so it should be fairly easy to drop everything older than 3/6 months. The images aren't available anyway, because Koji prunes them, so no harm done.

If you think that's too much work, I believe that wiping all data is still a better approach than keeping the rust accumulate. We could wipe all data twice per year, each time a new Fedora is release (at that point, we know which composes are last known good - the GA ones :wink: ). We've passed that moment a long time ago, but I'd still wipe all data even now - better sooner than later.

I went ahead and implemented a 180 day cutoff. I've put it into production on happyassassin and restarted the consumer, so whenever the data is next refreshed that should kick in and the live data should be pruned.

Metadata Update from @adamwill:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Thanks a lot, the old images are gone now.

np. i also realized test results were missing because I hadn't updated the server with the new openQA python client that the latest FreeIPA code needs, so I fixed that too...test results should start being used again soon.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/quality/fedora-nightlies/issues/8

Please continue any further discussion there.
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.