Issue #11560: countme log gathering script improvment/cleanup - fedora-infrastructure

fedora-infrastructure

#11560 countme log gathering script improvment/cleanup

Closed: Fixed 4 months ago by kevin. Opened 7 months ago by kevin.

There is a cron job on log01 that gathers httpd logs from various machines for countme and other purposes.

It's got some errors and could use some improvement.

My personal desire for cron jobs is that they do not output anything if everything worked, and only output errors/anything when something went wrong.
That way if you see the email you know to look, right now I have no idea if it's working mostly, or not at all.

Last nights run:

/etc/cron.daily/sync-http-logs-and-merge.sh:

=== START OF RUN ===
Started sync from proxy03.vpn.fedoraproject.org.
Started sync from proxy01.iad2.fedoraproject.org.
Started sync from proxy05.vpn.fedoraproject.org.
Started sync from proxy10.iad2.fedoraproject.org.
Started sync from proxy09.vpn.fedoraproject.org.
... host proxy01.iad2.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy03.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy01.iad2.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy03.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy05.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy05.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy09.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy09.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy10.iad2.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy10.iad2.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy01.iad2.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy10.iad2.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy01.iad2.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy10.iad2.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy01.iad2.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy10.iad2.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy01.iad2.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy10.iad2.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy01.iad2.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy10.iad2.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy01.iad2.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy10.iad2.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy01.iad2.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
... host proxy10.iad2.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy01.iad2.fedoraproject.org.
Finished sync from proxy10.iad2.fedoraproject.org.
Started sync from proxy02.vpn.fedoraproject.org.
Started sync from proxy11.vpn.fedoraproject.org.
... host proxy02.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy02.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy11.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy11.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy03.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy03.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy03.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy03.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy03.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy03.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy03.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy03.vpn.fedoraproject.org.
Started sync from proxy04.vpn.fedoraproject.org.
... host proxy04.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy04.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy11.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy11.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy11.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy11.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy11.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy11.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy11.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy11.vpn.fedoraproject.org.
Started sync from proxy12.vpn.fedoraproject.org.
... host proxy12.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy12.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy04.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy04.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy04.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy04.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy04.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy04.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy04.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy04.vpn.fedoraproject.org.
Started sync from proxy30.vpn.fedoraproject.org.
... host proxy30.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy30.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy02.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy02.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy02.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy12.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy12.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy12.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy12.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy02.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy02.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy02.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy12.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy12.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy12.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy12.vpn.fedoraproject.org.
Started sync from proxy14.vpn.fedoraproject.org.
... host proxy14.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy14.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy02.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy02.vpn.fedoraproject.org.
Started sync from proxy32.vpn.fedoraproject.org.
... host proxy32.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy32.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy05.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy05.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy05.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy05.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy05.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy05.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy05.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy05.vpn.fedoraproject.org.
Started sync from proxy06.vpn.fedoraproject.org.
... host proxy06.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy06.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy14.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy14.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy14.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy14.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy14.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy14.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy14.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy14.vpn.fedoraproject.org.
Started sync from proxy34.vpn.fedoraproject.org.
... host proxy34.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy34.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy34.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy34.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy34.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy34.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy34.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy34.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy34.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy34.vpn.fedoraproject.org.
Started sync from proxy35.vpn.fedoraproject.org.
... host proxy35.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy35.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy32.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy32.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy32.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy32.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy32.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy32.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy32.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy32.vpn.fedoraproject.org.
Started sync from proxy33.vpn.fedoraproject.org.
... host proxy33.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy33.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy35.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy35.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy35.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy30.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy30.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy30.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy35.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy35.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy35.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy35.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
... host proxy33.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy33.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
Finished sync from proxy35.vpn.fedoraproject.org.
... host proxy33.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
Started sync from proxy36.vpn.fedoraproject.org.
... host proxy36.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy36.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy30.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy30.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy30.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy33.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy33.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy33.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy30.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy30.vpn.fedoraproject.org.
Started sync from proxy31.vpn.fedoraproject.org.
... host proxy31.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy31.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
rsync: [sender] link_stat "/httpd/*20231004*" (in log) failed: No such file or directory (2)
... host proxy33.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy33.vpn.fedoraproject.org.
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
Started sync from proxy38.vpn.fedoraproject.org.
... host proxy38.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy38.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy31.vpn.fedoraproject.org, log date 2023-10-04, attempt 2
rsync: [sender] link_stat "/httpd/*20231004*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-10-04, attempt 3
rsync: [sender] link_stat "/httpd/*20231004*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy31.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy31.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy06.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy06.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy06.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
rsync: [sender] link_stat "/httpd/*20231003*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-10-03, attempt 2
... host proxy06.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy06.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy06.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy06.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy06.vpn.fedoraproject.org.
Started sync from proxy40.vpn.fedoraproject.org.
... host proxy40.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy40.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
rsync: [sender] link_stat "/httpd/*20231003*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-10-03, attempt 3
... host proxy36.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy36.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy36.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
rsync: [sender] link_stat "/httpd/*20231003*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy31.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy31.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy38.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy38.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy38.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy36.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy36.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy36.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
rsync: [sender] link_stat "/httpd/*20231002*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy40.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy31.vpn.fedoraproject.org, log date 2023-10-02, attempt 2
... host proxy40.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy40.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy36.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy36.vpn.fedoraproject.org.
Started sync from proxy37.vpn.fedoraproject.org.
... host proxy37.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy37.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy40.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy40.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy40.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy38.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
rsync: [sender] link_stat "/httpd/*20231002*" (in log) failed: No such file or directory (2)
... host proxy38.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy38.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-10-02, attempt 3
... host proxy40.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy40.vpn.fedoraproject.org.
Started sync from proxy101.iad2.fedoraproject.org.
... host proxy101.iad2.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy101.iad2.fedoraproject.org, log date 2023-10-04, attempt 1
rsync: [sender] link_stat "/httpd/*20231002*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy31.vpn.fedoraproject.org.
Started sync from proxy110.iad2.fedoraproject.org.
... host proxy110.iad2.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy110.iad2.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy101.iad2.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy101.iad2.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy101.iad2.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy101.iad2.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy101.iad2.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy101.iad2.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy101.iad2.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy101.iad2.fedoraproject.org.
Started sync from people02.vpn.fedoraproject.org.
... host proxy09.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host people02.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host people02.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy110.iad2.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy38.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
... host proxy09.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy110.iad2.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
Finished sync from proxy38.vpn.fedoraproject.org.
... host proxy09.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy110.iad2.fedoraproject.org, log date 2023-10-03, attempt 1
Started sync from proxy39.vpn.fedoraproject.org.
... host proxy110.iad2.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy39.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host proxy39.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host proxy110.iad2.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy110.iad2.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy110.iad2.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy110.iad2.fedoraproject.org.
Started sync from value01.iad2.fedoraproject.org.
... host value01.iad2.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host value01.iad2.fedoraproject.org, log date 2023-10-04, attempt 1
... host value01.iad2.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host value01.iad2.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host value01.iad2.fedoraproject.org, log date 2023-10-03, attempt 1
... host value01.iad2.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host value01.iad2.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host value01.iad2.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy09.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy09.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host value01.iad2.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
... host proxy09.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host people02.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host people02.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host people02.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
Finished sync from value01.iad2.fedoraproject.org.
Started sync from dl01.iad2.fedoraproject.org.
... host dl01.iad2.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host dl01.iad2.fedoraproject.org, log date 2023-10-04, attempt 1
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host dl01.iad2.fedoraproject.org, log date 2023-10-04, attempt 2
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host dl01.iad2.fedoraproject.org, log date 2023-10-04, attempt 3
... host people02.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host dl01.iad2.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host dl01.iad2.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host dl01.iad2.fedoraproject.org, log date 2023-10-03, attempt 1
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host people02.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host people02.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host dl01.iad2.fedoraproject.org, log date 2023-10-03, attempt 2
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host people02.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
... host dl01.iad2.fedoraproject.org, log date 2023-10-03, attempt 3
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host dl01.iad2.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
Finished sync from people02.vpn.fedoraproject.org.
Started sync from noc01.iad2.fedoraproject.org.
... host dl01.iad2.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host dl01.iad2.fedoraproject.org, log date 2023-10-02, attempt 1
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host noc01.iad2.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host noc01.iad2.fedoraproject.org, log date 2023-10-04, attempt 1
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
... host noc01.iad2.fedoraproject.org, log date 2023-10-04, attempt 2
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
... host dl01.iad2.fedoraproject.org, log date 2023-10-02, attempt 2
... host noc01.iad2.fedoraproject.org, log date 2023-10-04, attempt 3
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
... host noc01.iad2.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host noc01.iad2.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host noc01.iad2.fedoraproject.org, log date 2023-10-03, attempt 1
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host noc01.iad2.fedoraproject.org, log date 2023-10-03, attempt 2
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
... host noc01.iad2.fedoraproject.org, log date 2023-10-03, attempt 3
... host proxy09.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
... host dl01.iad2.fedoraproject.org, log date 2023-10-02, attempt 3
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
... host noc01.iad2.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
Finished sync from proxy09.vpn.fedoraproject.org.
... host noc01.iad2.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host noc01.iad2.fedoraproject.org, log date 2023-10-02, attempt 1
Started sync from proxy09.vpn.fedoraproject.org.
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host proxy09.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host dl01.iad2.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from dl01.iad2.fedoraproject.org.
... host proxy09.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
Started sync from dl02.iad2.fedoraproject.org.
... host dl02.iad2.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host dl02.iad2.fedoraproject.org, log date 2023-10-04, attempt 1
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host dl02.iad2.fedoraproject.org, log date 2023-10-04, attempt 2
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host dl02.iad2.fedoraproject.org, log date 2023-10-04, attempt 3
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host dl02.iad2.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host dl02.iad2.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host dl02.iad2.fedoraproject.org, log date 2023-10-03, attempt 1
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host dl02.iad2.fedoraproject.org, log date 2023-10-03, attempt 2
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host dl02.iad2.fedoraproject.org, log date 2023-10-03, attempt 3
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host dl02.iad2.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host dl02.iad2.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host dl02.iad2.fedoraproject.org, log date 2023-10-02, attempt 1
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host dl02.iad2.fedoraproject.org, log date 2023-10-02, attempt 2
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host dl02.iad2.fedoraproject.org, log date 2023-10-02, attempt 3
@ERROR: max connections (20) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host dl02.iad2.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
Finished sync from dl02.iad2.fedoraproject.org.
Started sync from dl03.iad2.fedoraproject.org.
... host dl03.iad2.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host dl03.iad2.fedoraproject.org, log date 2023-10-04, attempt 1
... host noc01.iad2.fedoraproject.org, log date 2023-10-02, attempt 2
... host proxy09.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy09.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host dl03.iad2.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy09.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host dl03.iad2.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host dl03.iad2.fedoraproject.org, log date 2023-10-03, attempt 1
... host dl03.iad2.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host dl03.iad2.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host dl03.iad2.fedoraproject.org, log date 2023-10-02, attempt 1
... host dl03.iad2.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from dl03.iad2.fedoraproject.org.
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
Started sync from dl04.iad2.fedoraproject.org.
... host dl04.iad2.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host dl04.iad2.fedoraproject.org, log date 2023-10-04, attempt 1
... host noc01.iad2.fedoraproject.org, log date 2023-10-02, attempt 3
... host dl04.iad2.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host dl04.iad2.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host dl04.iad2.fedoraproject.org, log date 2023-10-03, attempt 1
... host dl04.iad2.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy39.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host dl04.iad2.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host dl04.iad2.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy39.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy09.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy39.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy09.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy09.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host dl04.iad2.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from dl04.iad2.fedoraproject.org.
Started sync from dl05.iad2.fedoraproject.org.
... host dl05.iad2.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host dl05.iad2.fedoraproject.org, log date 2023-10-04, attempt 1
... host dl05.iad2.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host dl05.iad2.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host dl05.iad2.fedoraproject.org, log date 2023-10-03, attempt 1
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
... host noc01.iad2.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from noc01.iad2.fedoraproject.org.
Started sync from sundries01.iad2.fedoraproject.org.
... host sundries01.iad2.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host sundries01.iad2.fedoraproject.org, log date 2023-10-04, attempt 1
... host dl05.iad2.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host sundries01.iad2.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host dl05.iad2.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host dl05.iad2.fedoraproject.org, log date 2023-10-02, attempt 1
... host sundries01.iad2.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host sundries01.iad2.fedoraproject.org, log date 2023-10-03, attempt 1
... host dl05.iad2.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from dl05.iad2.fedoraproject.org.
Started sync from download-ib01.vpn.fedoraproject.org.
... host sundries01.iad2.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-04, seeding dated logfiles from undated
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-04, attempt 1
... host sundries01.iad2.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host sundries01.iad2.fedoraproject.org, log date 2023-10-02, attempt 1
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-04, attempt 2
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-04, attempt 3
... host sundries01.iad2.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from sundries01.iad2.fedoraproject.org.
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host proxy09.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-03, attempt 2
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-03, attempt 3
Finished sync from proxy09.vpn.fedoraproject.org.
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-02, attempt 2
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-02, attempt 3
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from download-ib01.vpn.fedoraproject.org.
... host proxy39.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy39.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy39.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy37.vpn.fedoraproject.org, log date 2023-10-04, linking back undated logfiles from dated
... host proxy37.vpn.fedoraproject.org, log date 2023-10-03, seeding dated logfiles from undated
... host proxy37.vpn.fedoraproject.org, log date 2023-10-03, attempt 1
... host proxy39.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy39.vpn.fedoraproject.org.
... host proxy37.vpn.fedoraproject.org, log date 2023-10-03, linking back undated logfiles from dated
... host proxy37.vpn.fedoraproject.org, log date 2023-10-02, seeding dated logfiles from undated
... host proxy37.vpn.fedoraproject.org, log date 2023-10-02, attempt 1
... host proxy37.vpn.fedoraproject.org, log date 2023-10-02, linking back undated logfiles from dated
Finished sync from proxy37.vpn.fedoraproject.org.
=== FINISH OF RUN ===

seddik commented 7 months ago

At the first glance we can catch the subprocess output and analyse this string ..
What do you think ?

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: dev, low-gain, medium-trouble

7 months ago

Metadata Update from @james:
- Issue assigned to james

7 months ago

kevin commented 7 months ago

So, I finally had a chance to look at this again.

There's 3 proxies that are failing... proxy31, proxy38 and proxy09

Seems logrotate isn't working right on proxy31 and the 'current' logs were just piling up since forever. ie, mirrors.fedoraproject.org-access.log on proxy31 was ~30GB.
I ran logrotate -d and it seemed normal/correct, so I ran logrotate -v and it rotated things and is compressing them now. Nothing in the journal about failures. Very odd.

I am not sure whats going on with proxy09 or proxy38. The files do seem to be there. I wonder if it's taking a while to compress them and thus isn't done when the sync script runs?

@seddik I think we should first adjust the script so it only outputs errors and also so it tells us what we need to debug it... ie, something like:

proxy31.vpn.fedoraproject.org: failed to find any '20231023' files
proxy123.vpn.fedoraproject.org: connection failed connecting

etc. If that makes sense?

kevin commented 7 months ago

Here's the current output btw:

kevin commented 6 months ago

Hum. Just saw that there's a Cron countme@log01 /usr/local/bin/countme-update.sh cron giving:

Traceback (most recent call last):
  File "/usr/bin/fedora-messaging", line 11, in <module>
    load_entry_point('fedora-messaging==3.0.0', 'console_scripts', 'fedora-messaging')()
  File "/usr/lib/python3.6/site-packages/click/core.py", line 721, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 1065, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.6/site-packages/click/core.py", line 894, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/fedora_messaging/cli.py", line 372, in publish
    api.publish(msg, exchange)
  File "/usr/lib/python3.6/site-packages/fedora_messaging/api.py", line 316, in publish
    eventual_result.wait(timeout=timeout)
  File "/usr/lib/python3.6/site-packages/crochet/_eventloop.py", line 241, in wait
    result.raiseException()
  File "/usr/lib64/python3.6/site-packages/twisted/python/failure.py", line 488, in raiseException
    raise self.value.with_traceback(self.tb)
fedora_messaging.exceptions.ConnectionException
Traceback (most recent call last):
  File "/usr/bin/fedora-messaging", line 11, in <module>
    load_entry_point('fedora-messaging==3.0.0', 'console_scripts', 'fedora-messaging')()
  File "/usr/lib/python3.6/site-packages/click/core.py", line 721, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 1065, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.6/site-packages/click/core.py", line 894, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/fedora_messaging/cli.py", line 372, in publish
    api.publish(msg, exchange)
  File "/usr/lib/python3.6/site-packages/fedora_messaging/api.py", line 316, in publish
    eventual_result.wait(timeout=timeout)
  File "/usr/lib/python3.6/site-packages/crochet/_eventloop.py", line 241, in wait
    result.raiseException()
  File "/usr/lib64/python3.6/site-packages/twisted/python/failure.py", line 488, in raiseException
    raise self.value.with_traceback(self.tb)
fedora_messaging.exceptions.ConnectionException

kevin commented 6 months ago

This started happening on 2023-10-27

seddik commented 6 months ago

@kevin please could you try to run this version ? https://pagure.io/fedora-infra/ansible/pull-request/1634

seddik commented 6 months ago

Hum. Just saw that there's a Cron countme@log01 /usr/local/bin/countme-update.sh cron giving:

Traceback (most recent call last): File "/usr/bin/fedora-messaging", line 11, in <module> load_entry_point('fedora-messaging==3.0.0', 'console_scripts', 'fedora-messaging')() File "/usr/lib/python3.6/site-packages/click/core.py", line 721, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3.6/site-packages/click/core.py", line 696, in main rv = self.invoke(ctx) File "/usr/lib/python3.6/site-packages/click/core.py", line 1065, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3.6/site-packages/click/core.py", line 894, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3.6/site-packages/click/core.py", line 534, in invoke return callback(*args, **kwargs) File "/usr/lib/python3.6/site-packages/fedora_messaging/cli.py", line 372, in publish api.publish(msg, exchange) File "/usr/lib/python3.6/site-packages/fedora_messaging/api.py", line 316, in publish eventual_result.wait(timeout=timeout) File "/usr/lib/python3.6/site-packages/crochet/_eventloop.py", line 241, in wait result.raiseException() File "/usr/lib64/python3.6/site-packages/twisted/python/failure.py", line 488, in raiseException raise self.value.with_traceback(self.tb) fedora_messaging.exceptions.ConnectionException Traceback (most recent call last): File "/usr/bin/fedora-messaging", line 11, in <module> load_entry_point('fedora-messaging==3.0.0', 'console_scripts', 'fedora-messaging')() File "/usr/lib/python3.6/site-packages/click/core.py", line 721, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3.6/site-packages/click/core.py", line 696, in main rv = self.invoke(ctx) File "/usr/lib/python3.6/site-packages/click/core.py", line 1065, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3.6/site-packages/click/core.py", line 894, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3.6/site-packages/click/core.py", line 534, in invoke return callback(*args, **kwargs) File "/usr/lib/python3.6/site-packages/fedora_messaging/cli.py", line 372, in publish api.publish(msg, exchange) File "/usr/lib/python3.6/site-packages/fedora_messaging/api.py", line 316, in publish eventual_result.wait(timeout=timeout) File "/usr/lib/python3.6/site-packages/crochet/_eventloop.py", line 241, in wait result.raiseException() File "/usr/lib64/python3.6/site-packages/twisted/python/failure.py", line 488, in raiseException raise self.value.with_traceback(self.tb) fedora_messaging.exceptions.ConnectionException

Hum strange ..
It seems to be a problem with fedora-messaging ?

kevin commented 6 months ago

Yeah, forgot to update this ticket, it was an expired cert. We got it all fixed a while back. ;)

seddik commented 6 months ago

@kevin , you can see below the output of the script.

[root@log01 ~][PROD-IAD2]# /usr/local/bin/sync-http-logs.py --syslog --syslog-verbosity=2
=== START OF RUN ===
rsync: [sender] link_stat "/httpd/*20231115*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
rsync: [sender] link_stat "/httpd/*20231115*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
rsync: [sender] link_stat "/httpd/*20231115*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
rsync: failed to connect to noc01.iad2.fedoraproject.org (10.3.163.10): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]
=== FINISH OF RUN ===

Script crashed only error logs :)
Otherwise, we should work on these error..

With @zlopez , we tried to test connection to noc01.iad2.fedoraproject.org , but seems to be a pb with DNS resolution :(

output of the ping

ping noc01.iad2.fedoraproject.org
PING noc01.iad2.fedoraproject.org (10.3.163.10) 56(84) bytes of data.
64 bytes from dhcp.iad2.fedoraproject.org (10.3.163.10): icmp_seq=1 ttl=64 time=0.326 ms

seddik commented 6 months ago

According to our discussion, @kevin suggested to add new custom rule to allow rsync from log01 to noc01.

seddik commented 6 months ago

@kevin I created PR here => https://pagure.io/fedora-infra/ansible/pull-request/1648
Let me know if something missed ;)

Edited 6 months ago by seddik

Metadata Update from @patrikp:
- Issue priority set to: None (was: Waiting on Assignee)

6 months ago

kevin commented 6 months ago

The cron is now outputing:

/etc/cron.daily/sync-http-logs-and-merge.sh:

=== START OF RUN ===
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
=== FINISH OF RUN ===

and the 'condense-mirrorlogs.sh' script is outputting:

sort: cannot read: /mnt/fedora_stats/data/mirrors/2023/11/worked-22: No such file or directory
sort: open failed: /mnt/fedora_stats/data/mirrors/2023/11/worked-22: No such file or directory
sort: cannot read: /mnt/fedora_stats/data/mirrors/2023/11/worked-22: No such file or directory
sort: cannot read: /mnt/fedora_stats/data/mirrors/2023/11/worked-22: No such file or directory

The 'worked' number changes from day to day, but its those same 4 lines...

smooge commented 6 months ago

rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
@ERROR: Unknown module 'log'

that seems to say it is trying to get the section from the rsync on the proxies for the log group which doesn't exist anymore..

and because of that..

sort: cannot read: /mnt/fedora_stats/data/mirrors/2023/11/worked-22: No such file or directory
sort: open failed: /mnt/fedora_stats/data/mirrors/2023/11/worked-22: No such file or directory
sort: cannot read: /mnt/fedora_stats/data/mirrors/2023/11/worked-22: No such file or directory
sort: cannot read: /mnt/fedora_stats/data/mirrors/2023/11/worked-22: No such file or directory

fails because there are no logs being gotten from the proxies

kevin commented 6 months ago

These are not proxies... looking at the old logs (like the first comment), I think they are:

Started sync from download-ib01.vpn.fedoraproject.org.
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-15, seeding dated logfiles from undated
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-15, attempt 1
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-15, attempt 2
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-15, attempt 3
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-15, linking back undated logfiles from dated
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-14, seeding dated logfiles from undated
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-14, attempt 1
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-14, attempt 2
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-14, attempt 3
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-14, linking back undated logfiles from dated
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-13, seeding dated logfiles from undated
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-13, attempt 1
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-13, attempt 2
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]
... host download-ib01.vpn.fedoraproject.org, log date 2023-11-13, attempt 3
@ERROR: Unknown module 'log'
rsync error: error starting client-server protocol (code 5) at main.c(1668) [Receiver=3.1.3]

So, download-ib01 and download-cc-rdu01... need to add log entry to their rsync and/or adjust firewalls or rsync perms.

smooge commented 6 months ago

OK the second set of errors should only happen if the old python and worked scripts aren't working. Basically those would try to parse the combined logs for proxies and then for a specific 'day' put in a worked-?? of that file. If those are no longer working or running then there may need to be some other cleanup

seddik commented 6 months ago

Why not try to reproduce the bug manually ? on these machines

kevin commented 6 months ago

I added log01.vpn.fedoraproject.org's ip to rsyncd.conf on those download servers. That should take care of those errors. I guess we will see tomorrow.

We still need to drop the:

=== START OF RUN ===
=== FINISH OF RUN ===

output.

I am not sure how to test the sort errors. Something in countme...

CC: @james @nphilipp @lenkaseg

seddik commented 6 months ago

@kevin ok , have just commented the output message as you suggested.
PR here => https://pagure.io/fedora-infra/ansible/pull-request/1671

kevin commented 4 months ago

So, this has been outputting:

rsync: [sender] link_stat "/httpd/*20240107*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
rsync: [sender] link_stat "/httpd/*20240107*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
rsync: [sender] link_stat "/httpd/*20240107*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
rsync: [sender] link_stat "/httpd/*20240106*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
rsync: [sender] link_stat "/httpd/*20240106*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
rsync: [sender] link_stat "/httpd/*20240106*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
rsync: [sender] link_stat "/httpd/*20240105*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
rsync: [sender] link_stat "/httpd/*20240105*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
rsync: [sender] link_stat "/httpd/*20240105*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]

Thats because proxy31 was not rotating it's logs right. ;( I fixed that.

I guess one slight improvement we could still do is to have it report the host on errors like above... but not a great big deal.

So, I will close this now, feel free to reopen if I missed anything.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

4 months ago

Metadata

Assignee

james

Tags

Blocking

None

Depending on

None

Priority

None

Boards 1

dev Status: Backlog

Attachments 1

1698175974.R15924078883793305176.logain2S

Attached 7 months ago View Comment

Related Pull Requests

#1671 Merged 6 months ago

fedora-infrastructure

Source Code

#11560 countme log gathering script improvment/cleanup Closed: Fixed 4 months ago by kevin. Opened 7 months ago by kevin.

Metadata

low-gain medium-trouble dev

Boards 1

Attachments 1

Related Pull Requests

#11560 countme log gathering script improvment/cleanup

Closed: Fixed 4 months ago by kevin. Opened 7 months ago by kevin.