releng / archive-repo-manager

#35 fedora_messaging consumer stalls periodically
Opened by dustymabe. Modified

Open
Close Issue

The messages usually look something like:

2026-05-15 16:01:07,665 WARNING fedora_messaging.twisted.consumer - The channel was closed by the server, you may have to increase the consumer_timeout configuration on RabbitMQ and/or decrease qos.prefetch_count in the fedora-messaging configuration file. Consuming will resume, starting with the last processed message again.
2026-05-15 16:01:07,726 WARNING pika.channel - Received remote Channel.Close (404): "NOT_FOUND - no queue 'amq.gen-wz4FyIH_rAYnEBNSVyFgdg' in vhost '/public_pubsub'" on <Channel number=2 OPEN conn=<pika.adapters.twisted_connection._TwistedConnectionAdapter object at 0xffffa42670e0>>

and then the consumer just doesn't progress any more.

Full output example below. See where it didn't do anything between May 15th and 18th (when it was restarted). 

May 15 16:01:06 archive-repo-manager archive-repo-manager[286116]: sending incremental file list
May 15 16:01:07 archive-repo-manager archive-repo-manager[286116]: deleting c1b3a3ff25100f7183eab3698699e1f6120d7370cc97ef6359c997556489b943-filelists.xml.zck
May 15 16:01:07 archive-repo-manager archive-repo-manager[286116]: deleting bbdbd2e860812f6a92f4daa9472efd085924a489d1f3845295dc36ca8e4f1aa2-primary.xml.zck
May 15 16:01:07 archive-repo-manager archive-repo-manager[286116]: deleting 9125f5e348a7c3908c89785c2246630af60cebb71b3fce11bb6d50826b686a43-primary.xml.zst
May 15 16:01:07 archive-repo-manager archive-repo-manager[286116]: deleting 871f6f4a3c7da2c27ee98e4ab36d6f9eb80ebfa0be8ad96bc14dc9ce88b2e187-other.xml.zck
May 15 16:01:07 archive-repo-manager archive-repo-manager[286116]: deleting 5a1142541e033e413e1d6510c9e43b7516d140bb3ab286d4dce1b2b85a026bad-filelists.xml.zst
May 15 16:01:07 archive-repo-manager archive-repo-manager[286116]: deleting 176fa05c9582c8cce542de69f533730df88ef042017151e621d7022bd80892b9-other.xml.zst
May 15 16:01:07 archive-repo-manager archive-repo-manager[286116]: 
May 15 16:01:07 archive-repo-manager archive-repo-manager[286116]: sent 617 bytes  received 516 bytes  755.33 bytes/sec
May 15 16:01:07 archive-repo-manager archive-repo-manager[286116]: total size is 400.42M  speedup is 353,417.81
May 15 16:01:07 archive-repo-manager archive-repo-manager[286116]: 2026-05-15 16:01:07,665 WARNING fedora_messaging.twisted.consumer - The channel was closed by the server, you may have to increase the consumer_timeout configuration on RabbitMQ and/or decrease qos.prefetch_count in the fedora-messaging configuration file. Consuming will resume, starting with the last processed message again.
May 15 16:01:07 archive-repo-manager archive-repo-manager[286116]: 2026-05-15 16:01:07,726 WARNING pika.channel - Received remote Channel.Close (404): "NOT_FOUND - no queue 'amq.gen-wz4FyIH_rAYnEBNSVyFgdg' in vhost '/public_pubsub'" on <Channel number=2 OPEN conn=<pika.adapters.twisted_connection._TwistedConnectionAdapter object at 0xffffa42670e0>>
May 18 19:51:35 archive-repo-manager systemd[1139]: Stopping archive-repo-manager.service - Container for archive-repo-manager...
May 18 19:51:45 archive-repo-manager archive-repo-manager[1169237]: time="2026-05-18T19:51:45Z" level=warning msg="StopSignal SIGTERM failed to stop container archive-repo-manager in 10 seconds, resorting to SIGKILL"
May 18 19:51:45 archive-repo-manager podman[1169237]: 2026-05-18 19:51:45.911334364 +0000 UTC m=+10.423262251 container died cbad864e5bdb21759657b16ba277ec7f335df4554ba2deaebdbf9e412fc1aad8 (image=localhost/archive-repo-manager:latest, name=archive-repo-manager, org.opencontainers.image.vendor=Fedora Project, PODMAN_SYSTEMD_UNIT=archive-repo-manager.service, org.opencontainers.image.name=fedora, org.opencontainers.image.version=43, vendor=Fedora Project, license=MIT, o>
May 18 19:51:46 archive-repo-manager podman[1169237]: 2026-05-18 19:51:46.045923883 +0000 UTC m=+10.557851762 container remove cbad864e5bdb21759657b16ba277ec7f335df4554ba2deaebdbf9e412fc1aad8 (image=localhost/archive-repo-manager:latest, name=archive-repo-manager, org.opencontainers.image.name=fedora, name=fedora, org.opencontainers.image.license=MIT, org.opencontainers.image.licenses=MIT, org.opencontainers.image.url=https://fedoraproject.org/, org.opencontainers.imag>
May 18 19:51:46 archive-repo-manager archive-repo-manager[1169237]: archive-repo-manager
May 18 19:51:46 archive-repo-manager systemd[1139]: archive-repo-manager.service: Main process exited, code=exited, status=137/n/a
May 18 19:51:46 archive-repo-manager systemd[1139]: archive-repo-manager.service: Failed with result 'exit-code'.
May 18 19:51:46 archive-repo-manager systemd[1139]: Stopped archive-repo-manager.service - Container for archive-repo-manager.
May 18 19:51:46 archive-repo-manager systemd[1139]: archive-repo-manager.service: Consumed 17h 48min 55.518s CPU time over 4d 23h 39min 3.099s wall clock time, 6.7G memory peak, 666.7M memory swap peak.
May 18 19:51:46 archive-repo-manager systemd[1139]: Starting archive-repo-manager.service - Container for archive-repo-manager...
May 18 19:51:46 archive-repo-manager podman[1169283]: 2026-05-18 19:51:46.164624713 +0000 UTC m=+0.045076672 container create d2594cd4e19e716c80664e96d9491f74cff9fb4c74f0f1fef73b8c6f7b7384f8 (image=localhost/archive-repo-manager:latest, name=archive-repo-manager, org.opencontainers.image.version=43, vendor=Fedora Project, org.opencontainers.image.license=MIT, org.opencontainers.image.title=fedora, org.opencontainers.image.name=fedora, org.opencontainers.image.url=https>
May 18 19:51:46 archive-repo-manager podman[1169283]: 2026-05-18 19:51:46.217140825 +0000 UTC m=+0.097592800 container init d2594cd4e19e716c80664e96d9491f74cff9fb4c74f0f1fef73b8c6f7b7384f8 (image=localhost/archive-repo-manager:latest, name=archive-repo-manager, io.buildah.version=1.43.0, org.opencontainers.image.version=43, name=fedora, org.opencontainers.image.license=MIT, org.opencontainers.image.name=fedora, org.opencontainers.image.vendor=Fedora Project, PODMAN_SYS>
May 18 19:51:46 archive-repo-manager podman[1169283]: 2026-05-18 19:51:46.227847735 +0000 UTC m=+0.108299727 container start d2594cd4e19e716c80664e96d9491f74cff9fb4c74f0f1fef73b8c6f7b7384f8 (image=localhost/archive-repo-manager:latest, name=archive-repo-manager, version=43, name=fedora, org.opencontainers.image.license=MIT, org.opencontainers.image.name=fedora, org.opencontainers.image.url=https://fedoraproject.org/, vendor=Fedora Project, org.opencontainers.image.titl>
May 18 19:51:46 archive-repo-manager systemd[1139]: Started archive-repo-manager.service - Container for archive-repo-manager.
May 18 19:51:46 archive-repo-manager archive-repo-manager[1169283]: d2594cd4e19e716c80664e96d9491f74cff9fb4c74f0f1fef73b8c6f7b7384f8
May 18 19:51:46 archive-repo-manager podman[1169283]: 2026-05-18 19:51:46.142737721 +0000 UTC m=+0.023189696 image pull e8d20d75e75d031dc4f563fa21bb169f846ea00ea3bc84a5086acff3ec151835 localhost/archive-repo-manager:latest
May 18 19:51:46 archive-repo-manager archive-repo-manager[1169303]: 2026-05-18 19:51:46,759 INFO builtins - Initial startup. Running once for each target release.
May 18 19:51:46 archive-repo-manager archive-repo-manager[1169303]: 2026-05-18 19:51:46,759 INFO builtins - Running command: ['/usr/local/bin/run-archive-repo-update', '42']
May 18 19:51:47 archive-repo-manager archive-repo-manager[1169303]: Previous datestamp is 1778830530 (Fri May 15 07:35:30 UTC 2026)
May 18 19:51:47 archive-repo-manager archive-repo-manager[1169303]: Current datestamp is 1779133907 (Mon May 18 19:51:47 UTC 2026)
May 18 19:51:47 archive-repo-manager archive-repo-manager[1169303]: Checking for new packages since 1778830530 til now (1779133907).
May 18 19:51:59 archive-repo-manager archive-repo-manager[1169303]: Downloading new builds for aarch64.
May 18 19:52:04 archive-repo-manager archive-repo-manager[1169303]: Downloading [1/1]: awf-gtk2-4.1.0-1.fc42.aarch64.rpm
May 18 19:52:08 archive-repo-manager archive-repo-manager[1169303]: Downloading [1/1]: awf-gtk3-4.1.0-1.fc42.aarch64.rpm

Cross linking to https://github.com/fedora-infra/fedora-messaging/issues/574
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.