#22 OCI generation is a zlib benchmark
Closed a year ago by otaylor. Opened a year ago by otaylor.

To go from the install tree to the final OCI, we:

  1. Tar with gzip compression the tree during the export process
  2. Uncompress the tarfile when importing into the ostree repository
  3. Compress the files again during that import since the ostree repository has mode archive
  4. Uncompress the files to create the layer tarfile
  5. Compress the layer tarfile

So, that's 3x compress, 2x compress. Then we compress the layers again to get the final oci.tar.gz. Which (though it does very little actually compression) can take a long time.

More or less simple things we could do here:

  1. Hope a faster version of zlib becomes standard in Fedora. zlib-ng as a compat replacement for zlib. Dropping in zlib-ng for zlib reduced a build of the Fedora flatpak-runtime on my laptop from 9m13s to 6m49s.
  2. Use zstd or a faster level of gzip for the tar that is imported into ostree
  3. Configure the ostree repository as 'bare-user' or keep it as 'archive' but configure the compression level from the default 6 to something faster.
  4. For the last step Make a .tar.zstd image rather than a .tar.gz - this is recognized by skopeo for the oci-archive:, and zstd is fast at detecting "nothing to do"

With hacked up implementation of the above:

what wall clock
current 9m13s
1. (zlib-ng) alone 6m49s
2-4 alone 6m10s
All of 1-4 3m56s

The results are suprisingly additive - I had expected that with zlib-ng, the effects of 2-4 would be minimal, but that's not the case.

Final note - for 4. - writing an 'oci.tar' is another variant - and it's even a bit faster than '.oci.tar.zstd'. skopeo recognizes that equally well to 'oci.tar.gz' and 'oci.tar.zstd'.

This is all done now. Once https://github.com/flatpak/flatpak/pull/5540 lands in Flatpak it would be good to take advantage of that to make the layer compression smaller and faster.

Metadata Update from @otaylor:
- Issue status updated to: Closed (was: Open)

a year ago

Log in to comment on this ticket.

Metadata