#436 RFE: Smart component build scheduling
Closed 6 years ago Opened 7 years ago by psabata.

I ran some numbers and I think it would make sense to be smarter about component build scheduling, building the largest components first and filling the available queues (koji biulders / local threads) with the small ones.

I recorded the local build times for all the Base Runtime components and compared the best v. the worst case scheduling scenarios for various number of build queues. The average time saved was around 20% (nearly 40% in the very best case). Of course the results depend on the packages you're building, the number of components, the number of queues and your hardware specs but it's always going to be non-zero.

A few more notes:

  • you could gather your own build time statistics or you could get this information from koji; look at the past, say, three successful builds of the component and get the average build time
  • smart scheduling should be done per build order group
  • it should also be done after compoent re-use is evaluated, counting the re-used components as taking zero or close to zero time
  • this will affect both local and koji builds but will be more noticable locally

The sample Base Runtime component local build times and the script to calculate the best vs. the worst case differences:

Note this is a build time optimization thing and generally a low priority.

Koji has a getAverageBuildDuration(package_name) method in the API. It's used to estimate the completion time of the current build, and to dynamically assign a "weight" to tasks. Might be useful here.

I checked the numbers from our yesterday's local bootstrap build:

  • Total time spent (worst case): 43561s
  • Total time spent (best case): 25466s
  • Difference: 18095s (42%)

That's for the bootstrap package set, built in 72 threads (our typical configuration). Now that's more interesting :)

Metadata Update from @mprahl:
- Issue assigned to mprahl

6 years ago

Metadata Update from @mprahl:
- Issue status updated to: Closed (was: Open)

6 years ago

Log in to comment on this ticket.

Metadata