fedora-ci / general

#232 Fedora CI storage - poor IOPS?
Closed by mvadkert. Opened by msrb.

Closed
Reopen Issue

Jenkins is pretty demanding when it comes to storage performance.

Fedora CI master runs on 200GiB of EBS/gp2 storage.

This might be a problem as it is likely giving us only ~700 IOPS (100 + 200 * 3). Which is an incredibly low number. Any modern SSD is capable of ~100k IOPS (notice the "k" after 100).

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html#EBSVolumeTypes_gp2

The instance feels sluggish and even small load (70-100 builds running, couple hundreds more queued) can bring it down.
We should migrate to something faster.

Metadata Update from @msrb:
- Issue tagged with: jenkins

We are actually seeing issues also with some test workloads, mostly those having fun with selinux, which seems to be quite IO bound.

Seems using io1/io2 could help something ... will pick up your brains sometime about the topic

Metadata Update from @msrb:
- Issue priority set to: High (was: Medium)

Is this about workers or Jenkins-master?

This is about master. Jenkins is pretty dummy as it synchronizes logs from workers to master. This normally shouldn't be a bottleneck in normal circumstances (for the expected load), but if the disk is slow, it can cause issues.

However, we recently switched from waiting for testing farm results in a loop (polling the API and logging), to webhooks where we simply wait for testing farm to call us back once results are available. This greatly reduces the amount of logs we generate and thus even this slower disk seems to be perfectly fine.
Edited by msrb

Since we switched to webhooks, I can reduce priority here.

Metadata Update from @msrb:
- Issue priority set to: Medium (was: High)

I will close this. Seems there are no more problems like this. Testing Farm runs on i3.2xlarge, which has good IO ops. All Fedora CI tests migrated to tmt, so they use Testing Farm.

Metadata Update from @mvadkert:
- Issue status updated to: Closed (was: Open)
Metadata
None

jenkins

None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.