releng

#11947 Nothing provides python39-pip-wheel/python39-setuptools-wheel in epel8-next
Closed: Fixed by zlopez. Opened by churchyard.

Closed: Fixed
Reopen Issue
  • Describe the issue

When I build a package with BuildRequires: python39-devel in epel8 and epel8-next, epel8 works but epel8-next does not. It says:

Error: 
 Problem: package python39-devel-3.9.16-1.module_el8.8.0+1243+5f5a1e61.aarch64 from build requires python39-libs(aarch-64) = 3.9.16-1.module_el8.8.0+1243+5f5a1e61, but none of the providers can be installed
  - package python39-devel-3.9.16-1.module_el8.8.0+1243+5f5a1e61.aarch64 from build requires libpython3.9.so.1.0()(64bit), but none of the providers can be installed
  - conflicting requests
  - nothing provides python39-pip-wheel needed by python39-libs-3.9.16-1.module_el8.8.0+1243+5f5a1e61.aarch64 from build
  - nothing provides python39-setuptools-wheel needed by python39-libs-3.9.16-1.module_el8.8.0+1243+5f5a1e61.aarch64 from build

Example scratchbuilds from the same commit:

  • https://koji.fedoraproject.org/koji/taskinfo?taskID=113485253 epel8 -- good
  • https://koji.fedoraproject.org/koji/taskinfo?taskID=113485319 epel8-next -- bad

It seems epel8-next buildroot is configured weirdly with regards to modularity. Unfortunately, python39 in RHEL 8 and c8s is a module. Coudl you please make the modular python39-*-wheel packages available in epel8-next Koji buildroot?

  • When do you need this? I already need it now, but it's not urgent

  • When is this no longer needed or useful? when epel8-next goes EOL

  • If we cannot complete your request, what is the impact? I cannot sync python3-rpm package with the changes done in c8s. I cannot add python3.12-rpm to EPEL 8 Next without removing python39-rpm.

Thank you.

Metadata Update from @phsmoura:
- Issue tagged with: high-trouble, low-gain, ops

I'm not sure how easy/possible this is going to be to sort out.

epel8-next syncs centos stream 8, then runs a script on the modular repo (grobisplitter) that 'demodularizes' them so they can be used to build against.

I suspect the problem here might be some module updated in one place and never in another or some old module still in the repo causing problems or something.

Needs someone to examine the repo and try and figure things out. ;(

Perhaps @tdawson or @carlgeorge might be interested in looking?

Can somebody please help me?

This is now blocking @nphilipp who cannot install python3.11-rpm to their c8s mock in order to do some rpmautospec things.

I’m unblocked now (using rhel+epel for the time being).
Edited by nphilipp

Thanks. Either way, users of c8s+epel-next have the same problem. I need to rebuild the package. Please help.

The error message says that it wants to install "python39-devel-3.9.16-1.module_el8.8.0+1243+5f5a1e61.aarch64". That package comes from python39:3.9:8080020230117181848:aed85c85:aarch64 CentOS module.

That module build is indeed the latest in that stream as listed in CentOS repository.

Then the error continues "nothing provides python39-pip-wheel".

Assuming python39-pip-wheel should belong to the same stream, then l can only confirm that python39:3.9:8080020230117181848:aed85c85:aarch64 module does not contain that package. Hence the DNF error is correct.

python39-pip-wheel-0:20.2.4-7.module_el8.7.0+1213+291b6551.noarch package is provided by an older python39:3.9:8080020221205125727:aed85c85:aarch64 module build.

So the question is whether grobisplitter considers older builds of the same stream. It should.

However, I have to note, that not delivering the package in the latest module build is very unusual. Either python39:3.9 maintainer stopped supporting and building that package intentionally, or it is a bug in packaging.

Either python39:3.9 maintainer stopped supporting and building that package intentionally, or it is a bug in packaging.

There was no intentional change like that.

The packages are listed in https://gitlab.com/redhat/centos-stream/modules/python39/-/blob/stream-python39-3.9-rhel-8.10.0/python39.yaml?ref_type=heads#L73

Locally, in x86_64 mock, I get:

$ mock -r centos-stream-8-x86_64 install python39-devel
...
  Installing       : python39-setuptools-wheel-50.3.2-5.module_el8+695+192a31a9.noarch
  Installing       : python39-pip-wheel-20.2.4-9.module_el8+963+a9c12c76.noarch
  Installing       : python39-libs-3.9.18-3.module_el8+830+f2c150f9.x86_64
  Installing       : python39-3.9.18-3.module_el8+830+f2c150f9.x86_64
  Installing       : python39-setuptools-50.3.2-5.module_el8+695+192a31a9.noarch
  Installing       : python3-rpm-generators-5-8.el8.noarch
  Installing       : python39-devel-3.9.18-3.module_el8+830+f2c150f9.x86_64
...

I see the build of python39 package in python39 module is: https://kojihub.stream.rdu2.redhat.com/koji/buildinfo?buildID=29680 (python39-3.9.16-1.module_el8.8.0+1243+5f5a1e61) so I wanted to take a look at what happened during that modular build but I cannot find it.

But https://mbs-ui.apps.stream.rdu2.redhat.com/mbs-ui/module/1243 gives me HTTP/500 and the newest build there has id 997.

I see the latest build of python39 module is https://mbs-ui.apps.stream.rdu2.redhat.com/mbs-ui/module/963 (python39-3.9-820240214182535.17377f89) and python39 package in that build is inherited from modular build https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=3357358 (python39-3.9.18-3.module_el8+830+f2c150f9). That exactly matches what Miro sees in his mock.

I'm confused about the build id because one of the first modular builds of python39 module in RHEL 8.4.0 has id 9303 (much higher than 1243) and the newest modular builds in CS have id < 1000. I have no idea where the build 1243 is from.

 
$ mock -r centos-stream-8-x86_64 --dnf-cmd repoquery python39-devel
...
python39-devel-0:3.9.13-1.module_el8.7.0+1178+0ba51308.x86_64
python39-devel-0:3.9.14-2.module_el8.8.0+1230+a6d278ed.x86_64
python39-devel-0:3.9.16-1.module_el8.8.0+1243+5f5a1e61.x86_64
python39-devel-0:3.9.17-1.module_el8+547+65817b2b.x86_64
python39-devel-0:3.9.17-2.module_el8+639+3fcd10de.x86_64
python39-devel-0:3.9.18-3.module_el8+830+f2c150f9.x86_64
python39-devel-0:3.9.2-1.module_el8.5.0+738+dc19af12.x86_64
python39-devel-0:3.9.2-2.module_el8.5.0+766+33a59395.x86_64
python39-devel-0:3.9.6-1.module_el8.5.0+872+ab54e8e5.x86_64
python39-devel-0:3.9.6-2.module_el8.5.0+897+68c4c210.x86_64
python39-devel-0:3.9.7-1.module_el8.6.0+930+10acc06f.x86_64

The c8s repos have all of them. My dnf/mock installs the latest one.

But the epel8-next koji repo only has one. Why this one?

$ mock -r centos-stream+epel-next-8-x86_64 --disablerepo='*' --enablerepo=local --dnf-cmd repoquery python39-devel
python39-devel-0:3.9.16-1.module_el8.8.0+1243+5f5a1e61.x86_64

This seems like a bug in the thing that takes c8s content and makes it a Koji repo.

Is it possible that the reason this build "wins" is the highest ID, 1243?

Similarily:

$ mock -r centos-stream+epel-next-8-x86_64 --dnf-cmd repoquery python39-pip-wheel
python39-pip-wheel-0:20.2.4-3.module_el8.5.0+738+dc19af12.noarch
python39-pip-wheel-0:20.2.4-4.module_el8.5.0+832+b0e10cdc.noarch
python39-pip-wheel-0:20.2.4-5.module_el8.5.0+859+e98e3670.noarch
python39-pip-wheel-0:20.2.4-6.module_el8.5.0+897+68c4c210.noarch
python39-pip-wheel-0:20.2.4-6.module_el8.6.0+930+10acc06f.noarch
python39-pip-wheel-0:20.2.4-7.module_el8+518+084be518.noarch
python39-pip-wheel-0:20.2.4-7.module_el8.6.0+961+ca697fb5.noarch
python39-pip-wheel-0:20.2.4-7.module_el8.7.0+1213+291b6551.noarch
python39-pip-wheel-0:20.2.4-8.module_el8+639+3fcd10de.noarch
python39-pip-wheel-0:20.2.4-9.module_el8+963+a9c12c76.noarch
$ mock -r centos-stream+epel-next-8-x86_64 --disablerepo='*' --enablerepo=local --dnf-cmd repoquery python39-pip-wheel
(nothing)

How is the c8s content selected for the epel8-next Koji repo? If you give me some pointers for what code does that, I can have a look.

It's done here:

https://pagure.io/fedora-infra/ansible/blob/main/f/roles/grobisplitter

First, I've done:

dnf reposync --repoid=c8s-appstream --download-metadata

The packages were downloaded:

$ ls -1 c8s-appstream/Packages/python39-*-wheel-*
c8s-appstream/Packages/python39-pip-wheel-20.2.4-3.module_el8.5.0+738+dc19af12.noarch.rpm
c8s-appstream/Packages/python39-pip-wheel-20.2.4-4.module_el8.5.0+832+b0e10cdc.noarch.rpm
c8s-appstream/Packages/python39-pip-wheel-20.2.4-5.module_el8.5.0+859+e98e3670.noarch.rpm
c8s-appstream/Packages/python39-pip-wheel-20.2.4-6.module_el8.5.0+897+68c4c210.noarch.rpm
c8s-appstream/Packages/python39-pip-wheel-20.2.4-6.module_el8.6.0+930+10acc06f.noarch.rpm
c8s-appstream/Packages/python39-pip-wheel-20.2.4-7.module_el8+518+084be518.noarch.rpm
c8s-appstream/Packages/python39-pip-wheel-20.2.4-7.module_el8.6.0+961+ca697fb5.noarch.rpm
c8s-appstream/Packages/python39-pip-wheel-20.2.4-7.module_el8.7.0+1213+291b6551.noarch.rpm
c8s-appstream/Packages/python39-pip-wheel-20.2.4-8.module_el8+639+3fcd10de.noarch.rpm
c8s-appstream/Packages/python39-pip-wheel-20.2.4-9.module_el8+963+a9c12c76.noarch.rpm
c8s-appstream/Packages/python39-setuptools-wheel-50.3.2-3.module_el8.5.0+738+dc19af12.noarch.rpm
c8s-appstream/Packages/python39-setuptools-wheel-50.3.2-4.module_el8+518+084be518.noarch.rpm
c8s-appstream/Packages/python39-setuptools-wheel-50.3.2-4.module_el8+639+3fcd10de.noarch.rpm
c8s-appstream/Packages/python39-setuptools-wheel-50.3.2-4.module_el8.5.0+897+68c4c210.noarch.rpm
c8s-appstream/Packages/python39-setuptools-wheel-50.3.2-4.module_el8.6.0+930+10acc06f.noarch.rpm
c8s-appstream/Packages/python39-setuptools-wheel-50.3.2-5.module_el8+695+192a31a9.noarch.rpm
c8s-appstream/Packages/python39-wheel-wheel-0.35.1-3.module_el8.5.0+738+dc19af12.noarch.rpm
c8s-appstream/Packages/python39-wheel-wheel-0.35.1-4.module_el8+518+084be518.noarch.rpm
c8s-appstream/Packages/python39-wheel-wheel-0.35.1-4.module_el8+639+3fcd10de.noarch.rpm
c8s-appstream/Packages/python39-wheel-wheel-0.35.1-4.module_el8.5.0+897+68c4c210.noarch.rpm
c8s-appstream/Packages/python39-wheel-wheel-0.35.1-4.module_el8.6.0+930+10acc06f.noarch.rpm

Then, I run splitter.py like this (line extracted from cents8-split.sh):

$ python splitter.py --action hardlink --target CS-8-002 ~/tmp/c8s-appstream/ --only-defaults

And as a result, the packages are missing:

$ find CS-8-002/ -name 'python39*'
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-3.9.16-1.module_el8.8.0+1243+5f5a1e61.x86_64.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-PyMySQL-0.10.1-2.module_el8.4.0+680+7b309a77.noarch.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-cffi-1.14.3-2.module_el8.7.0+1178+0ba51308.x86_64.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-chardet-3.0.4-19.module_el8.4.0+680+7b309a77.noarch.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-cryptography-3.3.1-2.module_el8.4.0+680+7b309a77.x86_64.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-devel-3.9.16-1.module_el8.8.0+1243+5f5a1e61.x86_64.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-idle-3.9.16-1.module_el8.8.0+1243+5f5a1e61.x86_64.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-idna-2.10-3.module_el8.4.0+680+7b309a77.noarch.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-libs-3.9.16-1.module_el8.8.0+1243+5f5a1e61.x86_64.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-lxml-4.6.5-1.module_el8.6.0+1078+3991166b.x86_64.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-mod_wsgi-4.7.1-5.module_el8.7.0+1213+291b6551.x86_64.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-numpy-1.19.4-3.module_el8.6.0+930+10acc06f.x86_64.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-numpy-doc-1.19.4-3.module_el8.6.0+930+10acc06f.noarch.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-numpy-f2py-1.19.4-3.module_el8.6.0+930+10acc06f.x86_64.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-pyyaml-5.4.1-1.module_el8.6.0+930+10acc06f.x86_64.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-rpm-macros-3.9.16-1.module_el8.8.0+1243+5f5a1e61.noarch.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-scipy-1.5.4-3.module_el8.4.0+680+7b309a77.x86_64.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-test-3.9.16-1.module_el8.8.0+1243+5f5a1e61.x86_64.rpm
CS-8-002/python39:3.9:8080020230117181848:aed85c85:x86_64/python39-tkinter-3.9.16-1.module_el8.8.0+1243+5f5a1e61.x86_64.rpm

So the problem is likely either in splitter.py or in the repository metadata used to extract the packages. Will dig further.

I see the problem. The found modules are:

python39:3.9:8080020230117181848:aed85c85:x86_64
python39:3.9:8080020221205125727:aed85c85:x86_64
python39:3.9:8070020220920211220:be1f0497:x86_64
python39:3.9:8070020220908144956:be1f0497:x86_64
python39:3.9:8070020220705182401:be1f0497:x86_64
python39:3.9:8070020220624145249:be1f0497:x86_64
python39:3.9:8060020220127154911:d3811607:x86_64
python39:3.9:8060020211016004914:d3811607:x86_64
python39:3.9:8060020210921015444:d3811607:x86_64
python39:3.9:8050020210825170628:d428a79b:x86_64
python39:3.9:8050020210802143738:d428a79b:x86_64
python39:3.9:8050020210709161752:d428a79b:x86_64
python39:3.9:8050020210611160013:d428a79b:x86_64
python39:3.9:8050020210506172806:d428a79b:x86_64
python39:3.9:8050020210422164837:d428a79b:x86_64
python39:3.9:8050020210330215900:d428a79b:x86_64
python39:3.9:820240214182535:17377f89:x86_64
python39:3.9:820240207141750:17377f89:x86_64
python39:3.9:820240117165712:17377f89:x86_64
python39:3.9:820231018065245:17377f89:x86_64
python39:3.9:820230810143808:17377f89:x86_64
python39:3.9:820230717121523:a8b5a7c1:x86_64

And the script assumes the greater the number at position 3 (e.g. 8080020230117181848) the newer the module.

Eg. when 8080020230117181848 is greater than 820240214182535 so the first line in our list is assumed to be the newest/latest one. This assumption by this script is wrong because at some point in c8s history, the number got reset and 820240214182535 is the latest one despite being lower in value than 8080020230117181848.

The wrong assumption is codified in https://pagure.io/fedora-infra/ansible/blob/63a8c4a3/f/roles/grobisplitter/files/splitter.py#_281 by @smooge (I guess). When the code was written, the assumption was correct.

I have no idea how to correctly find the newest thing.

I was able to produce accurate results for pytohn39 with this hack:

diff --git a/roles/grobisplitter/files/splitter.py b/roles/grobisplitter/files/splitter.py
index 9b40d0e5d2..da11528cca 100755
--- a/roles/grobisplitter/files/splitter.py
+++ b/roles/grobisplitter/files/splitter.py
@@ -282,7 +282,7 @@ def get_default_modules(directory):
                             # don't as it is a sorted list but we
                             # could have multiple contexts which would
                             # change things.
-                            if ( int(ts1) > int(ts2) ):
+                            if ( int(ts1) > int(ts2) or len(str(ts1)) < len(str(ts2)) ):
                                 # print ("%s > %s newer for %s", ts1,ts2,ourname)
                                 tempdict[tmp_name] = ourname
                         else:

It assumes that when the ID has fewer digits, it is newer. That assumption is probably not generally applicable.

$ find CS-8-002/ -name 'python39*'
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-3.9.18-3.module_el8+830+f2c150f9.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-PyMySQL-0.10.1-2.module_el8+639+3fcd10de.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-cffi-1.14.3-2.module_el8+639+3fcd10de.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-chardet-3.0.4-19.module_el8+639+3fcd10de.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-cryptography-3.3.1-3.module_el8+908+e4fa621d.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-devel-3.9.18-3.module_el8+830+f2c150f9.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-idle-3.9.18-3.module_el8+830+f2c150f9.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-idna-2.10-3.module_el8+639+3fcd10de.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-libs-3.9.18-3.module_el8+830+f2c150f9.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-lxml-4.6.5-1.module_el8+639+3fcd10de.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-mod_wsgi-4.7.1-7.module_el8+639+3fcd10de.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-numpy-1.19.4-3.module_el8+639+3fcd10de.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-numpy-doc-1.19.4-3.module_el8+639+3fcd10de.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-numpy-f2py-1.19.4-3.module_el8+639+3fcd10de.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-pip-20.2.4-9.module_el8+963+a9c12c76.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-pip-wheel-20.2.4-9.module_el8+963+a9c12c76.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-ply-3.11-10.module_el8+639+3fcd10de.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-psutil-5.8.0-4.module_el8+639+3fcd10de.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-psycopg2-2.8.6-3.module_el8+824+8c59dd11.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-psycopg2-doc-2.8.6-3.module_el8+824+8c59dd11.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-psycopg2-tests-2.8.6-3.module_el8+824+8c59dd11.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-pycparser-2.20-3.module_el8+639+3fcd10de.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-pysocks-1.7.1-4.module_el8+639+3fcd10de.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-pyyaml-5.4.1-1.module_el8+639+3fcd10de.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-requests-2.25.0-3.module_el8+639+3fcd10de.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-rpm-macros-3.9.18-3.module_el8+830+f2c150f9.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-scipy-1.5.4-5.module_el8+639+3fcd10de.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-setuptools-50.3.2-5.module_el8+695+192a31a9.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-setuptools-wheel-50.3.2-5.module_el8+695+192a31a9.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-six-1.15.0-3.module_el8+639+3fcd10de.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-test-3.9.18-3.module_el8+830+f2c150f9.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-tkinter-3.9.18-3.module_el8+830+f2c150f9.x86_64.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-toml-0.10.1-5.module_el8+639+3fcd10de.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-urllib3-1.25.10-5.module_el8+732+e8fbb6ce.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-wheel-0.35.1-4.module_el8+639+3fcd10de.noarch.rpm
CS-8-002/python39:3.9:820240214182535:17377f89:x86_64/python39-wheel-wheel-0.35.1-4.module_el8+639+3fcd10de.noarch.rpm

As a result of this problem, I assume that epel8-next Koji repo contains old modular packages from back when c8s was built after RHEL 8.8. Any default modular stream built in c8s after that is likely not in the repository.

In the meantime, i also asked for the incomplete module to be removed from the c8s repository in https://issues.redhat.com/browse/CS-2025

Good detective work.

I could push the hacky change after freeze is over (wed morning). Or do you think we should just wait for the incomplete module removal and not try and hack our side?

The incomplete module removal should fix my immediate problem (nothing provides python39-pip-wheel/python39-setuptools-wheel) by replacing the module with an older but hopefully complete one (8080020221205125727) in the epel8-next Koji repo. I have verified that by doing this:

diff --git a/roles/grobisplitter/files/splitter.py b/roles/grobisplitter/files/splitter.py
index 9b40d0e5d2..ba5de0f93d 100755
--- a/roles/grobisplitter/files/splitter.py
+++ b/roles/grobisplitter/files/splitter.py
@@ -282,7 +282,7 @@ def get_default_modules(directory):
                             # don't as it is a sorted list but we
                             # could have multiple contexts which would
                             # change things.
-                            if ( int(ts1) > int(ts2) ):
+                            if ( int(ts1) > int(ts2) or ts2 == "8080020230117181848" ):
                                 # print ("%s > %s newer for %s", ts1,ts2,ourname)
                                 tempdict[tmp_name] = ourname
                         else:

I get the missing packages:

$ find CS-8-002/ -name 'python39*wheel*'
CS-8-002/python39:3.9:8080020221205125727:aed85c85:x86_64/python39-pip-wheel-20.2.4-7.module_el8.7.0+1213+291b6551.noarch.rpm
CS-8-002/python39:3.9:8080020221205125727:aed85c85:x86_64/python39-setuptools-wheel-50.3.2-4.module_el8.6.0+930+10acc06f.noarch.rpm
CS-8-002/python39:3.9:8080020221205125727:aed85c85:x86_64/python39-wheel-0.35.1-4.module_el8.6.0+930+10acc06f.noarch.rpm
CS-8-002/python39:3.9:8080020221205125727:aed85c85:x86_64/python39-wheel-wheel-0.35.1-4.module_el8.6.0+930+10acc06f.noarch.rpm

However, the real problem won't be fixed, the epel8-next repository contains old modular packages, nott he latest ones.

Do not push my hacky change -- it was only tested with the python39 module. We need a canonical way to figure out which module is the latest and I have no idea how to tell when the numerical IDs do not sort reasonably.

I could get all the numerical IDs of all the c8s modules and figure out a pattern.

I have verified that my hacky len(str(ts1)) < len(str(ts2)) check works for all c8s-appstream modules. I have not checked other repositories and I have not checked RHELs or c9s.

I can also not check it all as I do not have enough disk space for that.

RHEL 8.0 was released with short versions, while later RHEL builds use long versions. I fear your hack will break selecting the latest build from RHEL repositories.

Agreed with @ppisar . The fix was put in to deal with the fact that RHEL-8.0 MBS output modules with a different convention and the various other methods which had been tried did not work. We went over various other methods and this was the easiest at that time.

We talked about this at the 2023-03-27 EPEL steering meeting and looked at a couple of solutions which all would not work like "rm the bad modules from the rsync mirror and lock rsync NOT to sync the old one". Kevin remembered when we tried this the first time and it broke because some part of the other repository tooling needs to see them all there in order to report that it can do its job.

At the moment, I think we need to work with @ppisar or other developers who know dnf/modularity a bit to copy over how modularity itself KNOWS which module upgrades which module. We then use that logic in grobisplitter.

How DNF constructs an upgrade path is documented at https://docs.fedoraproject.org/en-US/modularity/core-concepts/upgrade-paths/. In case of EPEL8, you need to follow Dynamic context section. Static-context modules are not supported in RHEL 8.

The script is not considering an upgrade path. It is trying to figure out which modular build is the latest/greatest/newest. I've read the linked documentation and I am afraid I lack the necessary knowledge to comprehend it. It speaks of lineages formed by DNF and frankly, makes zero sense to me, sorry.

Given the following 2 NSVCAs:

  • python39:3.9:820240214182535:17377f89:x86_64
  • python39:3.9:8080020230117181848:aed85c85:x86_64

How do I determine which one is the "latest/greatest/newest"? I suspect that there is no way to tell just based on those values, correct?
Edited by churchyard

https://docs.fedoraproject.org/en-US/modularity/core-concepts/nsvca/#_version explicitly says:

Versions are just updates of a given stream. Technically, version is a number generated by the build system. Higher number always wins. This means that version does not identify a major/minor version of the software but the update/commit to a particular stream.

Emphasis mine. If this is true, then the grobisplitter script is correct and the modules in c8s repository are... not correct?

Yes. Shrinking the version format by CentOS Stream 8 MBS in the middle of C8S life cycle is the cause. CentOS infra should revert the format and rebuild the affected modules.

For some reason people here are trying to come up with workarounds instead of fixing the cause. If the same issue happened with plain RPM NEVRs nobody would doubt what to do.

The script is not considering an upgrade path. It is trying to figure out which modular build is the latest/greatest/newest. I've read the linked documentation and I am afraid I lack the necessary knowledge to comprehend it. It speaks of lineages formed by DNF and frankly, makes zero sense to me, sorry.

That's because an upgrade path is dealing in selecting contexts. The selection depends on which context was installed. An upgrade is a function of not only what you have in a repository, but also of what is installed before the upgrade.

Given the following 2 NSVCAs:

  • python39:3.9:820240214182535:17377f89:x86_64
  • python39:3.9:8080020230117181848:aed85c85:x86_64

How do I determine which one is the "latest/greatest/newest"?

The latter If the build has satisfied run-time dependencies.

I suspect that there is no way to tell just based on those values, correct?

As you find out in your later post, you do not need to do an upgrade. You do an installation. Installation is easier: You sort builds by a version and pick the highest one which has satisfied modular dependencies (that involves default streams of all modules). If you find multiple installable builds of the same version, you pick any of the contexts.

I've reported this problem to CentOS Stream

https://issues.redhat.com/browse/CS-2044

However, considering the usual turnout to deal with modularity issues in CentOS Stream, this is unlikely to be fixed before the EOL.

EPEL releng folks, would you accept a dirty workaround for the script that adds a specific conditional for the python39 module?

I suppose... there's really not a lot of time left for epel8-next, but if it's really wanted...

https://pagure.io/fedora-infra/ansible/pull-request/1931

Commit fc460139 fixes this issue

 
DEBUG util.py:463:  Dependencies resolved.
DEBUG util.py:463:  ================================================================================
DEBUG util.py:463:   Package                   Arch   Version                           Repo   Size
DEBUG util.py:463:  ================================================================================
DEBUG util.py:463:  Installing:
DEBUG util.py:463:   python39-devel            x86_64 3.9.18-3.module_el8+830+f2c150f9  build 266 k
DEBUG util.py:463:  Installing dependencies:
DEBUG util.py:463:   python3-rpm-generators    noarch 5-8.el8                           build  25 k
DEBUG util.py:463:   python39                  x86_64 3.9.18-3.module_el8+830+f2c150f9  build  34 k
DEBUG util.py:463:   python39-libs             x86_64 3.9.18-3.module_el8+830+f2c150f9  build 8.8 M
DEBUG util.py:463:   python39-pip-wheel        noarch 20.2.4-9.module_el8+963+a9c12c76  build 1.1 M
DEBUG util.py:463:   python39-rpm-macros       noarch 3.9.18-3.module_el8+830+f2c150f9  build  16 k
DEBUG util.py:463:   python39-setuptools       noarch 50.3.2-5.module_el8+695+192a31a9  build 969 k
DEBUG util.py:463:   python39-setuptools-wheel noarch 50.3.2-5.module_el8+695+192a31a9  build 497 k
DEBUG util.py:463:  Transaction Summary
DEBUG util.py:463:  ================================================================================

Thank you all.
Metadata
None

high-trouble low-gain ops

None
None
None
Related Pull Requests
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.