I believe I almost have a proper fix to this issue, I'll know by the end of the week (possibly end of the day)

I can do one better, the fix here is now in production, I've tested the f30 armhfp/aarch64 container builds that failed yesterday and they succeed today. I'll keep an eye on it but do let me know of any updates.

I had a look at the f29 container this morning, the build successfully used an aarch64 box but it failed and the screenshot does not help much.

https://koji.fedoraproject.org/koji/taskinfo?taskID=33113838

The F31 builds for both base/base-minimal were successful though:

https://koji.fedoraproject.org/koji/taskinfo?taskID=33115461
https://koji.fedoraproject.org/koji/taskinfo?taskID=33115457

And we now actually have screenshots so we're a huge step forward :humbsup:

I believe the 4.20 kernel needs a patch that's in 5.0, I'll try and work out if that's the case and get it into 29/28 if so later today.

Just an update here. @pbrobinson is going to look into the current f28/f29 failures, but he's out on PTO next week, so it will be the week after.

@pbrobinson any news ? We still did not have a valid f30 container compose because of the armhfp failures.

So the current situation is that fedora:latest still points to fedora:29 and not fedora:30

I'm investigating as time allows. I'm not, or at least should not be, the only person that can assist in debugging the container failures, the arm stuff works in exactly the same way as x86

TBH I ll be happy to help investigating but I have no idea where to start. Do you have any pointers ?

koji uses a tool called imagefactory (which in turn uses oz), the packages to install are "imagefactory imagefactory-plugins imagefactory-plugins-TinMan imagefactory-plugins-Docker imagefactory-plugins-RHEVM"

sample commands for x86_64/aarch64/armhfp are:
imagefactory --debug base_image --parameter offline_icicle false tdl-x86_64.xml --file-parameter install_script koji-f30-build-33023264-base.ks
imagefactory --debug base_image --parameter generate_icicle false tdl-aarch64.xml --file-parameter install_script koji-f30-build-33023260-aarch64.ks
imagefactory --debug base_image --parameter generate_icicle false tdl-armhfp.xml --file-parameter install_script koji-f30-build-33023261-armhfp.ks

the tdl-<arch>.xml and koji.ks files can be grabbed from the latest koji tasks.

From running the command with armhfp I get the following stack trace.

```
2019-05-05 14:34:04,963 ERROR imgfac.Builder.Builder thread(0881c299) Message: UEFI firmware is not installed!
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/imgfac/Builder.py", line 132, in _build_image_from_template
self.os_plugin.create_base_image(self, template, parameters)
File "/usr/lib/python2.7/site-packages/imagefactory_plugins/TinMan/TinMan.py", line 346, in create_base_image
libvirt_xml = self.guest.install(self.app_config["timeout"])
File "/usr/lib/python2.7/site-packages/oz/RedHat.py", line 717, in install
self.virtio_channel_name)
File "/usr/lib/python2.7/site-packages/oz/Guest.py", line 1598, in _do_install
virtio_channel_name)
File "/usr/lib/python2.7/site-packages/oz/Guest.py", line 488, in _generate_xml
loader, nvram = oz.ozutil.find_uefi_firmware(self.tdl.arch)
File "/usr/lib/python2.7/site-packages/oz/ozutil.py", line 1074, in find_uefi_firmware
raise Exception("UEFI firmware is not installed!")
Exception: UEFI firmware is not installed!

Image build FAILED with error: UEFI firmware is not installed!```

Full log is there --> https://paste.fedoraproject.org/paste/6VlhAWtgzK6UioljrQe8WA

Oh I wonder if we should be looking for armhfp here too https://github.com/clalancette/oz/commit/289bf3681d4baaba7a4d005eed456c65b5446e27#diff-b84bb63aecd989c49b280cceba6cf224

Edit : Actually not the value seems correct from the xml file

Edit : for the record I needed to install edk2-arm-20190308stable-1.fc30.noarch

Oh I wonder if we should be looking for armhfp here too https://github.com/clalancette

No, armhfp isn't what the kernel reports

Edit : Actually not the value seems correct from the xml file
Edit : for the record I needed to install edk2-arm-20190308stable-1.fc30.noarch

Correct, and edk2-aarch64 for aarch64

Ok so I managed to reproduce the error, but I get the same output from the build and link to a screenshot. Unfortunately the screenshot just have the following "Guest disabled display".

Is there an easy way to have the display working ?

I suppose this https://bugzilla.redhat.com/show_bug.cgi?id=1705676 is related to the no display issue.

So I have not been able to make much progress on this. Should we make armv7 failable during the compose so that we can at least release the other arches ?

fedora:latest still points to Fedora 29 since we did not have a Fedora 30 compose in a while :-(

I suppose this https://bugzilla.redhat.com/show_bug.cgi?id=1705676 is related to the no display issue.

This is an issue about graphical display.

Right now there are 2 possible archives for each architecture.
Fedora-Container-Base-.tar.gz
Fedora-Container-Minimal-Base-.tar.gz

Is the issue for both Base and Minimal-Base?
If it is only related to "Base", can we skip "Base", running only "Minimal-Base"?

I suppose this https://bugzilla.redhat.com/show_bug.cgi?id=1705676 is related to the no display issue.

This is an issue about graphical display.
Right now there are 2 possible archives for each architecture.
Fedora-Container-Base-.tar.gz
Fedora-Container-Minimal-Base-.tar.gz
Is the issue for both Base and Minimal-Base?
If it is only related to "Base", can we skip "Base", running only "Minimal-Base"?

No both Base and Minimal-Base are failing

So I have not been able to make much progress on this. Should we make armv7 failable during the compose so that we can at least release the other arches ?
fedora:latest still points to Fedora 29 since we did not have a Fedora 30 compose in a while :-(

What need to be done to release other arches ?

Right now f30 s390x container image is only "Minimal-Base". It does not have "Base".
https://dl.fedoraproject.org/pub/fedora-secondary/releases/30/Container/s390x/images/
I like to see the "Base" will be prepared too after this issue will be fixed.

So armhfp container builds are now fixed, I finally got around to sorting this out, sorry for the delay.