I looked at this a bit friday, but I couldn't see why it wasn't working. ;(

Perhaps @lenkaseg could look tomorrow (I'm away on PTO tomorrow) and confirm they are getting added to the retired packages json.

Taking a look...
I'm a bit confused, I wanted to to see some logs, but I cannot see the toddler koji_block_retired on openshift, neither under pods nor under jobs.. where can I see it?

Found it with help of @humaton :100:
Checking logs...

Perhaps @lenkaseg could look tomorrow (I'm away on PTO tomorrow) and confirm they are getting added to the retired packages json.

These packages are present in the json file with retired packages, the problem is not there.

yeah, so we have both that and the block_retired.py releng script call at the beginning of the rawhide compose (which uses pdc) and neither one seems to be blocking things. ;(

It is probably the toddler that is supposed to do the koji blocking.

Ok, the problem is that the koji command in the toddler blocking the packages is complaining about koji not being logged in. Fixing and adding an error catcher for such cases.

Thanks for digging into this one lenka!

Is this supposed to be fixed?

There have been successful rawhide composes since "fixing and adding an error catcher" two days ago, and retired packages are still not getting blocked in koji.

No, it's not fixed yet. We tracked it down I think to a permissions issue somewhere... (the toddler not using the right keytab or something).

But it's not fixed yet.

I'm working on it, making the toddler use a keytab for the koji session.

Thanks for looking into it, @kevin and @lenkaseg. In the meantime, can a releng person please process the following retirements manually?

php-doctrine-common3
php-doctrine-orm
python-googleapis-common-protos
python-pytest-grpc
rsibreak

They are all retired in distgit but remain in Koji. It would be really great to get this sorted before branching. Thanks.

What exactly needs to be done in order to manually process the retirements? So they are already orphaned in dist-git and have the dead.package file, good.

Now they need to be blocked in Koji and then have their EOL adjusted by running the adjust-eol.py script? Is anything else required?

cc: @jnsamyak @kevin

Thanks for looking into it, @kevin and @lenkaseg. In the meantime, can a releng person please process the following retirements manually?

php-doctrine-common3 php-doctrine-orm python-googleapis-common-protos python-pytest-grpc rsibreak

They are all retired in distgit but remain in Koji. It would be really great to get this sorted before branching. Thanks.

I don't have the correct permissions, but I passed it to the folks that do.

What exactly needs to be done in order to manually process the retirements? So they are already orphaned in dist-git and have the dead.package file, good.

Now they need to be blocked in Koji and then have their EOL adjusted by running the adjust-eol.py script? Is anything else required?

cc: @jnsamyak @kevin

I suppose. Could you do it please? I see they're still not blocked.

Okay so I have blocked these for rawhide in Koji, please let me know if these need to be done for others as well:

php-doctrine-common3 | blocked for f41
php-doctrine-orm | blocked for f41
python-googleapis-common-protos | blocked for f41
python-pytest-grpc | blocked for f41
rsibreak| blocked for f41

We have three main playing characters in terms of retirement/unretirements - dist-git, koji and PDC (hopefully gonna go soon).

What exactly needs to be done in order to manually process the retirements? So they are already orphaned in dist-git and have the dead.package file, good.

Yep in theory, the first things should be - the commit should be reverted back from dist-git

Now they need to be blocked in Koji and then have their EOL adjusted by running the adjust-eol.py script? Is anything else required?

In koji these should be blocked for the respective tag, and the active should be set to false when it comes to PDC, that can be done using the adjust-eol script

But it's mostly preferred to not to do it manually but let the script do the job but yeah

If it is possible to manually block from koji, could you please also do that for other packages that were retired since the automatic processing broke, not only for those five?

We could do that, I need to fetch a list of all the packages that were retired but not blocked in Koji; I did have a fix for the script somewhere in my system and will process the manual blocking!

I did manually block following packages on rawhide branch:

Blocking anaconda-liveinst on branch f41
Blocking apostrophe on branch f41
Blocking clang16 on branch f41
Blocking fmt10 on branch f41
Blocking genwqe-tools on branch f41
Blocking golang-github-avast-retry on branch f41
Blocking golang-github-cpuguy83-md2man-epel on branch f41
Blocking golang-github-quic-qtls-go1-20 on branch f41
Blocking isis on branch f41
Blocking libmetalink on branch f41
Blocking libplacebo5 on branch f41
Blocking libxcrypt-epel on branch f41
Blocking lld16 on branch f41
Blocking llvm16 on branch f41
Blocking lorax-templates-rhel on branch f41
Blocking mariadb on branch f41
Blocking mozjs91 on branch f41
Blocking ncurses-epel on branch f41
Blocking nodejs-epel on branch f41
Blocking openjpeg2 on branch f41
Blocking openssl-ibmpkcs11 on branch f41
Blocking parsertl14 on branch f41
Blocking python3.11-requests_ntlm-epel on branch f41
Blocking python39-pyelftools-epel on branch f41
Blocking python39-zstd-epel on branch f41
Blocking python-googleapis-common-protos on branch f41
Blocking python-jsonschema-epel on branch f41
Blocking python-pynose on branch f41
Blocking qqc2-desktop-style on branch f41
Blocking redis on branch f41
Blocking ruff-lsp on branch f41
Blocking rust-cairo-sys-rs0.15 on branch f41
Blocking rust-gdk-pixbuf-sys0.15 on branch f41
Blocking rust-gio0.15 on branch f41
Blocking rust-libcramjam0.2 on branch f41
Blocking rust-pango-sys0.15 on branch f41
Blocking rust-portable-atomic0.3 on branch f41
Blocking rust-routinator-ui on branch f41
Blocking spirv-llvm-translator17 on branch f41
Blocking webextension-token-signing on branch f41

Thank you! :blue_heart:

Please see #12219 wrt lorax-templates-rhel.

What's the current status here? Do the recent commits to the new koji_block_retired toddler fix the issue?

Now when PDC seems to be retired, is it OK to block retired packages in Koji manually after retiring them, or is additional action needed?

Please manually block

• golang-helm-3
• polkit-gnome

Please manually block

• godep

as well. Thank you!

koji block-pkg f41 golang-helm-3 polkit-gnome godep done

libxcrypt-epel is an ELN Extras package, please koji add-pkg --owner salimma eln libxcrypt-epel to unblock it

Could you please manually block golang-github-vultr-govultr-2? I've just finished retiring it.

I also retired some more packages:

rust-ouroboros0.17
rust-pem1
rust-test-case1
rust-gio-sys0.15
rust-glib0.15

I just retired golang-oras-1, golang-github-masterminds-squirrel and golang-github-gosuri-uitable

I retired golang-github-yashtewari-glob-intersection and golang-github-sacloud-libsacloud.

Just retired crypto, decnumber and softfloat.

I just processed the remaining orphaned-for-more-than-six-weeks packages. Those need to be blocked also:

• python-astunparse
• rust-async-mutex
• rust-awc
• rust-cargo-manifest
• rust-fn-error-context
• rust-infer
• rust-inlinable_string
• rust-intervaltree
• rust-libxml
• rust-log-mdc
• rust-snake_case
• rust-tiger
• rust-ubyte

I have retired six more packages:

rust-directories4
rust-glib-macros0.15
rust-gobject-sys0.15
rust-lscolors0.15
rust-ouroboros_macro0.17
rust-uutils_term_grid0.3

More golang packages retired:

golang-github-apache-arrow
golang-github-apache-arrow12
golang-github-oracle-oci-sdk
golang-k8s-klog
golang-github-apparentlymart-textseg-13

Update:

This issue should be solved by merging/fixing these:
https://pagure.io/releng/issue/12246
https://pagure.io/fedora-infra/ansible/pull-request/2198
https://pagure.io/fedora-infra/toddlers/pull-request/241

Additional retired packages:

• golang-github-adalogics-fuzz-headers
• golang-github-aquarapid-vaultlib
• golang-github-buger-jsonparser
• golang-github-martini-contrib-auth
• golang-github-martini-contrib-gzip
• golang-github-martini-contrib-render
• golang-github-monochromegane-gitignore
• golang-github-opentracing-contrib-grpc
• golang-github-pires-proxyproto
• golang-github-planetscale-pargzip
• golang-github-planetscale-tengo
• golang-github-sjmudd-stopwatch
• golang-github-z-division-zookeeper
• golang-gopkg-ldap-2

Thanks @lenkaseg !

I've blocked the ones mentioned here, we can land those pr's here soon...

golang-github-apache-arrow blocked
golang-github-apache-arrow12 blocked
golang-github-oracle-oci-sdk blocked
golang-k8s-klog blocked
golang-github-apparentlymart-textseg-13 blocked
golang-github-adalogics-fuzz-headers blocked
golang-github-aquarapid-vaultlib blocked
golang-github-buger-jsonparser blocked
golang-github-martini-contrib-auth blocked
golang-github-martini-contrib-gzip blocked
golang-github-martini-contrib-render blocked
golang-github-monochromegane-gitignore blocked
golang-github-opentracing-contrib-grpc blocked
golang-github-pires-proxyproto blocked
golang-github-planetscale-pargzip blocked
golang-github-planetscale-tengo blocked
golang-github-sjmudd-stopwatch blocked
golang-github-z-division-zookeeper blocked
golang-gopkg-ldap-2 blocked

Please block also python-r128gain (in rawhide and f41), which I retired recently and still isn't blocked.

I have retired three more packages in both rawhide and f41:

rust-glib-sys0.15
rust-nu-ansi-term0.49
rust-rust-embed6

Blocked those.

Hopefully the automation will be fixed here very soon.

rust-glib-sys0.15 blocked
rust-nu-ansi-term0.49 blocked
rust-rust-embed6 blocked
python-r128gain blocked

I have retired rust-rust-embed-impl6 in rawhide and f41 today.

I just processed the orphaned for 6+ weeks packages. Please handle those as well.

bes
owncloud-client
python-tmuxp

EDIT: The packages should be retired from both rawhide and f41.

$ koji block-pkg f42 bes owncloud-client python-tmuxp
$ koji block-pkg f41 bes owncloud-client python-tmuxp

Done.

I have retired rust-rust-embed-impl6 in rawhide and f41 today.

blocked.

We are hoping that this is now fixed.

If anyone here could check after their next retirement that it was properly blocked that would be great.

I just retired rubygem-activemodel-serializers-xml in rawhide and f41 (another 6+ weeks package). Let's see if it works.

I just retired rubygem-activemodel-serializers-xml in rawhide and f41 (another 6+ weeks package). Let's see if it works.

It apparently did not. It still does not show as blocked.

$ koji list-pkgs --show-blocked --tag f42 --quiet --package rubygem-activemodel-serializers-xml
rubygem-activemodel-serializers-xml f42                                      orphan
$ koji list-pkgs --show-blocked --tag f41 --quiet --package rubygem-activemodel-serializers-xml
rubygem-activemodel-serializers-xml f41                                      orphan  

Please handle these manually in both rawhide and f41:

• python-EvoPreprocess
• rubygem-activemodel-serializers-xml

I have retired two more packages in rawhide and f41:

rust-rust-embed-utils7
rust-globwalk0.8

Please ensure they're blocked in koji.

Okay these are all blocked for rawhide and f41

python-EvoPreprocess    f41                                      orphan          [BLOCKED]
python-EvoPreprocess    f42                                      orphan          [BLOCKED]
rubygem-activemodel-serializers-xml f41                                      orphan          [BLOCKED]
rubygem-activemodel-serializers-xml f42                                      orphan          [BLOCKED]
rust-rust-embed-utils7  f41                                      decathorpe      [BLOCKED]
rust-rust-embed-utils7  f42                                      decathorpe      [BLOCKED]
rust-globwalk0.8        f41                                      decathorpe      [BLOCKED]
rust-globwalk0.8        f42                                      decathorpe      [BLOCKED]

So ... I retired three packages yesterday, within seconds of each other, but they're all in very weird (and different!) states now:

• rust-femme: retired in rawhide, f41, and epel9, blocked in no tag
• rust-kv-log-macro: retired in rawhide, f41, and epel9, only blocked in f42 tag
• rust-serial_test0.6: retired in rawhide and f41, only blocked in f41 tag

Looking at it.

Meanwhile I saw some of your packages retired on epel10 did not get blocked because of the new tagging system. There is a fix in progress: https://pagure.io/fedora-infra/toddlers/pull-request/257

some of your packages retired on epel10

I have not yet retired any packages on epel10?

some of your packages retired on epel10

I have not yet retired any packages on epel10?

I saw this message: https://apps.fedoraproject.org/datagrepper/v2/id?id=79d391d3-7f0c-4d04-afe2-37b3e1133352&is_raw=true&size=extra-large

But it has a timestamp from the year 2021: "date": "2021-11-19T12:12:46+01:00", weird, I wonder why it was sent again...

That package was retired and then un-retired in rawhide, and I merged rawhide -> epel10. But the package is not supposed to be retired in epel10.

So ... I retired three packages yesterday, within seconds of each other, but they're all in very weird (and different!) states now:

• rust-femme: retired in rawhide, f41, and epel9, blocked in no tag
• rust-kv-log-macro: retired in rawhide, f41, and epel9, only blocked in f42 tag
• rust-serial_test0.6: retired in rawhide and f41, only blocked in f41 tag

I see the problem. There are two actually, first one is fixed by restarting the pod, second one should be fixed by the aforementioned PR (fixes epel blocking).

That package was retired and then un-retired in rawhide, and I merged rawhide -> epel10. But the package is not supposed to be retired in epel10.

Ok, we identified the problem. When creating new branch from an old branch the messages are sent to message bus with all the commits. And in case the package has an addition of a dead.package in it's history, the toddler gets triggered and tries to block it in koji for the new branch.
We're working on a solution.

Which could be either limit the messages sent when new branch is created, adding a field to messages that would mark new branch creation or not blocking packages that have delta > 1 day between header["sent-at"] and commit["date"].

@ondruch retired rpm-local-generator-support 6 days ago on rawhide:

https://src.fedoraproject.org/rpms/rpm-local-generator-support/c/fe440ba74982f374517b8f812d858bdea71c1fc8?branch=rawhide

The package is still not blocked.

So ... I retired three packages yesterday, within seconds of each other, but they're all in very weird (and different!) states now:

• rust-femme: retired in rawhide, f41, and epel9, blocked in no tag
• rust-kv-log-macro: retired in rawhide, f41, and epel9, only blocked in f42 tag
• rust-serial_test0.6: retired in rawhide and f41, only blocked in f41 tag

I see the problem. There are two actually, first one is fixed by restarting the pod, second one should be fixed by the aforementioned PR (fixes epel blocking).

I don't think this was fixed by rebooting the pod. It's been two days now, and the status is still the same (broken) one I posted above.

I don't think this was fixed by rebooting the pod. It's been two days now, and the status is still the same (broken) one I posted above.

I mean the pod was fixed, but the packages have to blocked manually. I don't have permissions to do that, but I can bring it to attention of somebody who can.

Blocked on rawhide:
rpm-local-generator-support
rust-femme
rust-kv-log-macro
rust-serial_test0.6

Blocked on epel9:
rust-femme
rust-kv-log-macro

Blocked on f41:
rust-femme
rust-kv-log-macro
rust-serial_test0.6

It looks like the new service that's supposed to block packages in koji upon retirement is stuck and / or not processing correctly today.

I retired the following packages in rawhide and f41 today, but they're not getting blocked in koji:

rust-gstreamer-editing-services
rust-gstreamer-player
rust-tower-test
rust-tower-util
rust-benfred-read-process-memory
rust-gstreamer-editing-services0.22
rust-gstreamer-player0.22

I see the koji.AuthError: Invalid session or bad credentials error again...

The 7 packages in my last comment have now been blocked, thanks!

.

As far as I can tell, this has been fixed for all branches of Fedora, so thank you for that!

However, it looks like epel9 / epel10 retirements are now not getting processed at all, for example these three packages that I retired a few days ago are not getting blocked in koji at all:

• rust-pbkdf2_0.11
• rust-quick-xml0.34
• rust-password-hash0.4

Looking at it!
The change in the toddler responsible for blocking packages retired in epel was merged and deployed only 4 days ago. If the packages were retired before that, they will have to be blocked manually. I cannot do it myself, but I can contact somebody who can.

The script that will detect the retired packages that were not blocked in koji for some reason is well underway, PR here: https://pagure.io/fedora-infra/toddlers/pull-request/261
Once deployed, it should run daily and hopefully all retired packages will be blocked then.

Thanks - I can confirm that these three packages are now blocked in epel9 and epel10.0 tags. :100:

With that resolved ... if anything new pops up, it's likely a new issue and not related to the original one that prompted this ticket. So I think it can be closed.

+1, thanks for your work here @lenkaseg!