I'm pretty sure it is not the upload that is causing the memory issue. These particular tracebacks are happening in fast_incremental_upload(), which doesn't every send a ton of data in one go. It uploads in fairly small blocks actually (64k).

Both of these tasks are running into the same hub-side issue at the same time, so it is definitely a hub problem. If you look at the hub logs on koji01 from that time, you will see over a hundred memory errors from about that time. Most of them are not from uploads. Most of them are from calls that need to allocate more than the typical amount of memory (e.g. listBuilds, queryHistory).

I note that you do have RLIMIT_AS configured. Your setting seems kind of high. It looks like it's set to about 1/5 of total system memory. Hypothetically, a run of very large queries could exhaust system memory and cause other calls to fail if they need to allocate more.

That said, we have code in koji to log when a call makes memory use jump. The largest I see reported that day is 422M (and that's not near the rash of memory errors). I don't see any sudden change in these warnings leading up to the issue, so the problem might not be memory usage from kojihub.

Is it possible that this is happening at the virtualization level? Perhaps because of overcommit settings?

Absent further data, I don't see much we can do about this from the Koji side. Maybe dust off ideas from #955? But none of that will help if the issue is coming from the system level or below.

I don't think I see any oom messages from the kernel on koji01, and I don't see any memory usages warnings from koji that get anywhere close to your rlimit setting.

Maybe add some system logging for process memory sizes?

Yeah, I am not sure what is to blame here then.

We do have collectd... and this is interesting:

https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=koji01.iad2.fedoraproject.org&hostname=koji02.iad2.fedoraproject.org&plugin=vmem&timespan=604800&action=show_selection&ok_button=OK

and

https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=koji01.iad2.fedoraproject.org&hostname=koji02.iad2.fedoraproject.org&plugin=memory&timespan=604800&action=show_selection&ok_button=OK

shows that things got a lot noisier yesterday morning? But not to the point I would think it would cause memory errors.

I guess perhaps I should update/reboot them (they haven't been updated in a while. They are on f41, but with the 6.11.x kernel.

If there's nothing koji can do here and it's a hub os issue, feel free to close this

It looks like an system level issue from here. Will close, but let me know if new data appears

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/4300

Please continue any further discussion there.