I think there's general agreement on the direction here. There's a few parallel conversations going on around specifics (cc @kyle @jvreeland) so I'd probably wait to see where those land before committing implementation-wise. About btrfs-progs, it shouldn't be an issue to get it in EPEL (cc @ngompa), though I'd love to see it promoted to base at some point down the road.

Is btrfs-progs particularly useful in EPEL with a kernel that lacks support for it?

Notes from Dojo:
- there's already a LTS kernel build of some kind under altarch/x86_64 -- need to find out details and engage with them
- signing for secure boot on CBS using the official centos key isn't feasible with the current infra

@hughesjr During Dojo somebody mentioned that you are the one building LTS kernels for CentOS 7, do you think you could participate in the next SIG meeting (next Wednesday, 18th at 18:00 UTC)?

started a repo for our kernel at https://pagure.io/centos-sig-hyperscale/linux

Don't forget to add the SIG group as admins to the repo (that's not automatic)

ahh didn't realize that, thanks! Should be updated now.

One note that I figured out today. Running sha512hmac during the kernel build process actually fails if built in mock (with systemd-nspawn, not sure about normal chroot)…

libkcapi - Error: Netlink error: sendmsg failed
libkcapi - Error: Netlink error: sendmsg failed
libkcapi - Error: NETLINK_CRYPTO: cannot obtain cipher information for hmac(sha512) (is required crypto_user.c patch missing? see documentation)
Allocation of hmac(sha512) cipher failed (ret=-111)

To solve this problem, I had to update koji builder to 5.10 itself.

Another thing I found out today that during the build of the kernel-tools, they use kernel-headers from the system which actually fails as they are quite outdated - although solution here was quite easy ⇒ build kernel without tools and then rebuild it again with.

In Fedora they are split into the separate source packages (kernel-headers, kernel, kernel-tools) but they use some very weird packaging so that I can't find how to get same sources myself with newer kernel tree :)

The kernel spec has the ability to build all these from one thing, but Fedora's extra source package spec files are independently managed. Honestly, I don't think it was a good strategy for them to split it up like that, but it is what it is.

Not as complete as I'd hoped but I have a working first pass at a kernel and I'm sure other people have input. It's on the c8s-sig-hyperscale branch in the centos rpms/kernel. It's 5.10.31 tested on real hardware and a VM based on the fedora 33 packaging. It's working fine though it was reported on irc with 5.10.31 that there were issues with modules, it happened to me but I rebuilt my vm and cleaed the mock buidroot and everything seems to be working fine since then.

Taking a look at the initial requirements:

Take latest stable kernel as a base
5.10 seems solid

Create x86-64 only config with modules that are relevant for servers and disable support for things from 80's

There are configs for other architectures included but they're not used. The config that the kernel is using right now is mostly from fedora with some obvious things removed like HAM. I'm sure I missed a ton of things that could be updated that I missed or didn't know what they were.

Do not forget to enable btrfs support there (and I guess we will have to build btrfs-progs, although not sure if we want to add that in EPEL or maintain it as part of this SIG - either is fine with me)

btrfs support is enabled.

Make sure we support multiple configs (Generic, Facebook, Twitter, whatnot) in some simple manner - I guess we even should build those in CentOS koji with some suffix so that anybody out there could simply install facebook variant for instance
Implement automated testing (be that CKI or any other thing)

In the systemd package there's a variable check to compile as a facebook package. I figure we can do the same thing here similar to how there's '.rhel' '.fedora', I added '.centos-sig-hyperscale' I figure '.twitter' and '.facebook' would keep most things the same and let us update from the continuing fedora development and still have customization where needed.

Don't know how to check if we're compiling for hyperscale so i just replaced rhel with centos-sig-hyperscale for now.

Signing?

didn't look into this at all.

I figure we will be downloading sources from the pagure linux git repo so we probably won't be using the packaging patch functionality as much in the future. I'd like to do something for automated updates and build to experimental like we're doing with systemd as well. Something I didn't notice until i pushed to the centos git repo to test the build in cbs is that all of a sudden the kabi stuff matters. I've never paid attention to it since we don't use it internally. But maybe as a SIG we should take more care?

In the systemd package there's a variable check to compile as a facebook package. I figure we can do the same thing here similar to how there's '.rhel' '.fedora', I added '.centos-sig-hyperscale' I figure '.twitter' and '.facebook' would keep most things the same and let us update from the continuing fedora development and still have customization where needed.

Yeah, we can get a tag for Twitter. See https://pagure.io/centos-infra/issue/212 for how to request one. Just make sure you ask for a custom disttag so we can tell the Twitter packages apart (e.g. for FB we use %dist .hs+fb.el8, for Twitter you could do %dist .hs+tw.el8 or similar.

Don't know how to check if we're compiling for hyperscale so i just replaced rhel with centos-sig-hyperscale for now.

We could ask to get a hyperscale macro added to all the tags to do this reliably, but I'm not sure we need to. As long as the package includes %{dist} in the Release, it will come out earmarked as a Hyperscale package when built on CBS, so it might already be ok.

.hs+twtr.el8 is probably the right suffix for Twitter builds.

I'm personally hoping to avoid having to company specific builds, but if I need to, then I'll probably get .hs+datto.el8 on a build tag.

https://pagure.io/centos-infra/issue/307 is tracking signing for secure boot

Summarizing the Dojo discussion with @ngompa:

• you should watch https://www.youtube.com/watch?v=3J6BjXjc_hQ
• we need to figure out if c9s will settle on 5.12 or 5.13
• we should consider rebasing to 5.12, especially if that's the kernel that c9s is going to use
• we should consider contributing (some of) our kernel work to https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9 down the road

That video was informative, thanks for the link. Looks like 5.13 made it into 9-pre. I rebased the sig kernel packaging onto 5.12 and replaced the kernel configs I was using before with some generated in an ark tree. I pushed my ark branch to our linux repo as well. I think more CONFIGS ended up enabled in these than the previous ones but it should have LIVEPATCHING enabled now as well. The ark branch should integrate well with the centos-stream-9 repo when we have a better idea what our kernel development/release workflow looks like. For now though it should make it easier to pull in changes from newer version that fedora has already dealt with.

Ark branch: https://pagure.io/centos-sig-hyperscale/linux/tree/centos-sig-hyperscale
Kernel Builds: https://cbs.centos.org/koji/taskinfo?taskID=2489545

Tagged kernel-5.12.4 to hyperscale8s-packages-experimental-release

So VDO is broken unless we build kmod-kvdo for this kernel. :cry:

CentOS Stream 9 transitioned to Linux 5.14-rc last week, and given the timeline (RHEL 9 Beta will release around the same time Fedora Linux 35 does), it looks like RHEL 9 will lock onto Linux 5.14.

We should probably start planning what we need added on top of the 5.14 kernel to start contributing to the RHEL kernel. That means we need to get in touch with @dzickus and @prarit to figure out how this would work.

Based on the conversations I had with @dcavalca a couple weeks ago, it looks like our plan is to align with the latest RHEL kernel for Hyperscale. That means the following:

• We will transition to tracking the CentOS Stream 9 kernel source tree once 5.14 stable lands.
• Our kernel source tree repository will follow a new branch convention for Hyperscale kernels: centos-hyperscale-<kernel-evr> where we branch from a tag from the CS9 kernel source tree. As an example, we'd create the centos-hyperscale-v5.14.0-0.rc3.29 branch from the kernel-5.14.0-0.rc3.29.el9 tag in the CS9 kernel source tree. From there, we can stack our own changes on top that we will stage to send to the CS9 Git tree. We will periodically rebase to newer EVRs as our changes make it through into RHEL mainline.
• When the CentOS Stream 10 kernel source tree is created and the final kernel version has been selected for that, we will rebase the Hyperscale kernel to that one and repeat the process again. I expect that we'll have transitioned fully over to CS9 by that point as the plan is to switch from targeting CS8 to CS9 as soon as reasonably possible. And when CS10 becomes available, we'll do the same then too. And so on. Basically, Hyperscale intends to track the latest stable CentOS Stream release for the majority of its development.

This gives us several distinct advantages:

1. The work we're doing will benefit the entire Red Hat ecosystem and give an opportunity for Red Hat to leverage community contributions to help support new features for RHEL itself as they ramp up to support it internally.
2. It keeps us somewhat aligned with the ecosystem in a way that may allow us to retain some kernel API compatibility with RHEL itself, which can help avoid some userspace wonkiness. Our work can then also be used by the CentOS Kmods SIG for providing kmod packages for the standard RHEL kernel too.
3. We are not trapping ourselves into maintaining the same kernel version for 10 years. This effectively says that we're maintaining a kernel version for 3 years, which is close to what Facebook and Twitter already do internally.
4. We benefit from the work Red Hat does on maintaining subsystems we have no ability to work on. This is particularly beneficial for the Hyperscale Workstation variants.
5. This makes the Hyperscale kernel appealing as a base for the CentOS Plus and altarch kernels.
6. Downstreams who need extra functionality in CentOS can consider leveraging and contributing to Hyperscale as a pipeline to getting the functionality they need supported in the ecosystem.

I expect that we'll begin this transition once the final 5.14 release lands in CentOS Stream 9, as that will be the point we know we can start stacking changes on top and contributing to their tree.

The 5.14 kernel has now been forked from ARK to the c9s kernel tree: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/commits/kernel-5.14.0-1.el9

We're now tracking stuff we want to backport to the c9s kernel in #72

Sorry for being quiet for some time…

1. Does anybody have any hint where/how to push my own configuration (Kconfig) flavor? I'd like to request some new tag like hyperscale-gooddata and I'd like to have quite minimalistic kernel build there…
2. As we are building 5.14+, we should backport few patches for "crash" 1 2, otherwise it is not usable

@ignatenkobrain If you need a variant CBS tag, you can request that at https://pagure.io/centos-infra

For Facebook, they've got the following setup:

• Build tag: hyperscale8s-packages-facebook-el8-build
• Release tags: hyperscale8s-packages-facebook-{candidate,testing,release}

All Hyperscale tags are x86_64 and aarch64. For GoodData, you'll want to have a GoodData-specific macro (%gooddata?) and set the DistTag to something like .hs+gdc.el8.

As for pushing your Kconfig flavor, we can get our tree branched here next week so that you can push it somewhere.

For kernel backports, you'll want to contribute here: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9

I'm effectively working on this now, so...

I think the only remaining item is working with @jvreeland to get my kernel builds also building for EL8. Once we have that done, I think we can close this ticket.

This is basically done now that CentOS Stream 8 has aged out.