Why does the update have to contain two differently-versioned builds of the same package? That seems...bizarre.

Historically, rubygem-json was independent package while later, it was bundled into Ruby. Nevertheless, it still has independent release cycle from Ruby. Therefore sometimes, the independent package is updated out of the cycle. E.g. if you take a look at Ruby in F37, it bundles rubygem-json 2.6.1, while the independent package is already bumped to 2.6.3.

There is more such packages in Ruby, but also elsewhere (in Perl, there are some AFAIK).

The issue is that CI is trying to install all RPMs from the Bodhi update before running the tests.

If you use tmt for components with conflicting subpackages, you can instruct it to exclude some of the subpackages:
https://tmt.readthedocs.io/en/stable/spec/plans.html#install

But, ideally, CI should just stop installing everything :)

I kind of understand, that CI downloads everything, but it should probably create repo and use the repo as a source, not independent packages. I think CI is currently overly cautious, but that creates another issues.

"Create a repo and use that" is what openQA does, but it's not entirely clear what you're suggesting. Yes, CI tries to install all packages from the update: it does this intentionally, it's considered to be a test - at least, that's my understanding. The expectation is that all packages in an update should be installable (or should have explicit declared conflicts). Trying to install them all together is a test to make sure this is true. If CI creates a repo, what does it do with that repo? Does it have to extract just the name from every package in the update and try to install all the names? Does it then have to add a post-process step to make sure the packages that got installed are actually the ones from the update and not just ones from the base repo (openQA does this, and it's quite complicated to keep track of)?

It's all very well to say "create a repo and use that", but it doesn't seem to be quite so straightforward in fact.

"Create a repo and use that" is what openQA does

Ok, good!

but it's not entirely clear what you're suggesting. Yes, CI tries to install all packages from the update: it does this intentionally, it's considered to be a test - at least, that's my understanding. The expectation is that all packages in an update should be installable (or should have explicit declared conflicts).

We are talking here about two versions of single package. It is never expected that they would have explicit conflict. One is simply update of the other. I am quite sure dnf install rubygem-json will do the right thing. It won't try to install both versions. That is the user scenario we should care about.

Trying to install them all together is a test to make sure this is true. If CI creates a repo, what does it do with that repo? Does it have to extract just the name from every package in the update and try to install all the names?

Right, that was my idea.

Does it then have to add a post-process step to make sure the packages that got installed are actually the ones from the update and not just ones from the base repo (openQA does this, and it's quite complicated to keep track of)?

Well, probably. Or possibly just the package with highest NVR is selected instead of just the names.

It's all very well to say "create a repo and use that", but it doesn't seem to be quite so straightforward in fact.

It admittedly is more complicated. But still, there is nothing wrong with the update in the question. The only problem is that the CI cannot handle it.