I've checked the provision config and it looks fine. From the guest log it looks there's just 1GB:

Memory: 909300K/1048048K available

@astepano, could you please have a look what else might be wrong? Thanks.

I've replied on https://src.fedoraproject.org/rpms/lorax/pull-request/3 I see the test running on a VM with 4G, but the test failing because it couldn't allocate memory.

Once thing that might be causing the issue is the VM is created as having 17 CPUs, while the container is limited to just 1 CPU.

https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/blob/master/Jenkinsfile#L157

I'll submit a patch to the pipeline to fix this.

I've fixed the CPU count on the pipeline side, the test still fails, but I don't see the xz: (stdin): Cannot allocate memory issue any more.

https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pipeline/1846/artifact/package-tests/logs/

Thanks, that appears to have solved that one.
The missing hfsplus module is still a problem (and once that's fixed I'll look closer at the single.repo error.

@bcl if the test needs hfsplus kernel module that is available on kernel-modules package and then load it, these tasks should be done as part of the test case.

Are you sure? I haven't had to manually load a filesystem kernel module in many years. And on my local VM tests this all 'just works'. Something else is going on.

I'd say this is not a CI pipeline issue, if this module was suppose to be loaded by default on Fedora release, then I think it should be open a bug for it. As this module is available under extra kernel module package, I'd say it is intentional to not be loaded by default.

If is not by default, it should be loaded as part of the test case.

For your case, basically need to add kernel-modules to required_packages
https://src.fedoraproject.org/rpms/lorax/pull-request/3#_7__14

And something like modprobe hfsplus || exit in the run_tests.sh script
https://src.fedoraproject.org/rpms/lorax/pull-request/3#_6__11

I disagree. kernel-modules should always be present. And I'm pretty sure if it were this would be working, so could you take a look at the CI system install and figure out why it isn't? As far as I can tell it is required by kernel so for it not to be there some extra, unusual, steps must have been taken.

the CI system uses a base qcow2 image produced by pungi. If kernel-modules should be installed by default it is a bug on Fedora release that builds the qcow2 without installing it.

In this base qcow2 image, there is only kernel-core installed and it provides some modules with it.

Example of qcow2 image used by the CI system:
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190816.n.0/compose/Cloud/x86_64/images/Fedora-Cloud-Base-Rawhide-20190816.n.0.x86_64.qcow2

I guess this needs somebody from Fedora Cloud SIG ....

@bcl
kernel-modules should always be present.

I'm not so sure. If that were true then they would have never broken it into a subpackage.

@mvadkert
I guess this needs somebody from Fedora Cloud SIG ....

The kernel-modules package was removed from the cloud base image a long time ago. The rationale (AFAIU) is that it doesn't need all the modules in kernel-modules because of the limited target use case (i.e. virtualized environments). In the past when we had this problem and needed a specific module we worked with the kernel team and got it moved into the kernel-core package. Here is the commit from that interaction: https://src.fedoraproject.org/rpms/kernel/c/ccd2942fe56205c72da92b24beaa874ea1d948c5?branch=master

Ah, so the tests are not running in the environment that a normal user would expect. I think that's a mistake -- the vast majority of Fedora's users are not using a cloud environment.
Maybe there should be some way to select what base image to use for the tests to avoid problems like this?

Ah, so the tests are not running in the environment that a normal user would expect. I think that's a mistake -- the vast majority of Fedora's users are not using a cloud environment.
Maybe there should be some way to select what base image to use for the tests to avoid problems like this?

We can use any images, for which qcow2 image is generated (as we use that qcow2 to boot the VM). I am not sure if we have qcow2 images available for Everything for example ...

One option is to just add a step in the test that installs the kernel-modules package for the kernel that's installed?

One option is to just add a step in the test that installs the kernel-modules package for the kernel that's installed?

Yes, but it seems to me that defeats the purpose of testing. We shouldn't have to take special steps to setup the environment.

I still believe this just needs documentation (which image are we using) and proper ways how you can reproduce the environment when testing on your box.

We shouldn't have to take special steps to setup the environment.

I would disagree with that, looking at all other CI systems, setup of the environment is a step the user needs to specify.

I still believe this just needs documentation (which image are we using) and proper ways how you can reproduce the environment when testing on your box.

With the cloud base image it's pretty easy because they can just use vagrant to bring it up.

I would disagree with that, looking at all other CI systems, setup of the environment is a step the user needs to specify.

I agree. Tests typically have deps and have a "setup" step.

Anything else that can be done on this ticket or it can be closed?

Possibly? I tried to confirm adding kernel-modules works but the CI test isn't getting triggered, just the simple-koji-ci test.

That this issue, which is still not resolved :(
https://pagure.io/fedora-infrastructure/issue/8160

@bcl can you try to rerun if this is still an issue? Seems they rolled back some infra upgrade which should resolve the problem ...

I am going to close the issue here as the memory error is resolved.

Let's move the discussion around kernel-modules required for hfsplus into https://pagure.io/fedora-ci/general/issue/62

And I filed a separate issue on a generic question - which image we should use for the base test environment: https://pagure.io/fedora-ci/general/issue/73 Please comment there.