Thanks for reporting, we are tracking it downstream here:

https://issues.redhat.com/browse/TFT-3610

Updates in:

https://status.testing-farm.io/issues/2025-08-18-fedora-ci-jenkins-is-down/

Hmmm, it still doesn't look good, even after TFT-3610 declared solved. Yesterday I restarted ([citest]) the merge requests. Look at https://src.fedoraproject.org/rpms/gcc/pull-request/66, for example: restarted, the scratch build completed in Koji, but no tests running. Looks like CI doesn't know about the build.

@vkadlcik yeah, that looks like another problem, not related to TFT-3610 at all.

@vkadlcik so I looked at the history of PRs there:

https://src.fedoraproject.org/rpms/gcc/pull-requests

And this is not really a new issue, I see these things happen also in 2023 and 2024 :(

It's an issue with the current design of the jenkins CI. We are making a builder wait on koji scratch builds which often times-out. We can increase the time-out, but then we are hitting the issue that other jobs are piling up in the build queue.

Also, with zuul jobs we have a similar issue. I suspect with packit's fedora-ci we do not have this issue though since it uses the fedora messaging system directly and does not have to listen to a cli. We probably could redesign the jenkins builds to do something similar, but it seems like a lot of work.

@vkadlcik what do you think about opting-in to packit's fedora-ci? I think the best hammer for these is to assist packit and move it forward in its deployment.

Regarding packit, I'm not the GCC maintainer, so I don't really have an answer. We can ask @jakub when he returns from his vacation, next week I think.

Is there a way for me to nudge Fedora CI to notice the scratch build is in https://src.fedoraproject.org/rpms/gcc/pull-request/66 and continue with the rest of testing?

Regarding packit, I'm not the GCC maintainer, so I don't really have an answer. We can ask @jakub when he returns from his vacation, next week I think.

You can technically onboard on behalf of someone else

Is there a way for me to nudge Fedora CI to notice the scratch build is in https://src.fedoraproject.org/rpms/gcc/pull-request/66 and continue with the rest of testing?

Not really. It's an issue with the designs here. The simple hammer is to just increase the timeout. Fixing it properly is a challenge and requires quite a lot of Jenkins knowledge and infra around it that I cannot hack my way around. Someone like @msrb is more likely to know if we can make a proper fix for Jenkins or if it is even worth it.

I understand it doesn't a have a proper solution. My question was about a workaround applicable to a stuck PR, some secret comment to drop there, whatever, just to move the processing further. I guess there's nothing like that...

I understand it doesn't a have a proper solution. My question was about a workaround applicable to a stuck PR, some secret comment to drop there, whatever, just to move the processing further. I guess there's nothing like that...

@vkadlcik if the build was built, you can trigger manually this job with the task ID:

https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/ (functional tests)
https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/rpminspect-pipeline/job/master/

I wanted to try it for F43, but that is still being onboarded:
https://gitlab.com/testing-farm/infrastructure/-/merge_requests/1052

not completely sure if the results will get here, but you should be able to at least have results ...

Thanks Miro, that's what I was looking for!

I wanted to file this to fedora-infra, but found a gcc build from a year ago that took 25 hours, so I guess we will need to find a way to deal with these slow builds on our side :(

i.e. it is not a regression or a change in the fedora infra