% pwd /mnt/fedora_stats/combined-http/2023/05 % ls -1hs 0*/mirrors.fedoraproject.org-access.log 9.4G 01/mirrors.fedoraproject.org-access.log 9.9G 02/mirrors.fedoraproject.org-access.log 2.0G 03/mirrors.fedoraproject.org-access.log 2.0G 04/mirrors.fedoraproject.org-access.log 2.1G 05/mirrors.fedoraproject.org-access.log 2.0G 06/mirrors.fedoraproject.org-access.log 1.8G 07/mirrors.fedoraproject.org-access.log 1.9G 08/mirrors.fedoraproject.org-access.log 2.0G 09/mirrors.fedoraproject.org-access.log
...everything seems to be fixed from May 10th onwards.
So we have a few different issues which have different priority:
Anything not caught within 7 days is usually removed from the proxies by logrotate. The servers run out of general disk space quickly otherwise. The oldest log currently is the 20230516 logs so all the missing days are gone. I would say that means 1, 2 and 3 are off the table.
Getting 5 and 4 worked out are what is left to be done.
Ok, I guess that means we can drop the priority way down. Although I suspect Matt is now much more anxious about 5 happening soon.
Metadata Update from @kevin: - Issue priority set to: Waiting on Assignee
I noticed some errors today in the cron job:
Started sync from proxy31.vpn.fedoraproject.org. ... host proxy31.vpn.fedoraproject.org, log date 2023-06-01, seeding dated logfiles from undated ... host proxy31.vpn.fedoraproject.org, log date 2023-06-01, attempt 1 rsync: [sender] link_stat "/httpd/*20230601*" (in log) failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3] ... host proxy31.vpn.fedoraproject.org, log date 2023-06-01, attempt 2 rsync: [sender] link_stat "/httpd/*20230601*" (in log) failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3] ... host proxy31.vpn.fedoraproject.org, log date 2023-06-01, attempt 3 rsync: [sender] link_stat "/httpd/*20230601*" (in log) failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3] ... host proxy31.vpn.fedoraproject.org, log date 2023-06-01, linking back undated logfiles from dated ... host proxy31.vpn.fedoraproject.org, log date 2023-05-31, seeding dated logfiles from undated ... host proxy31.vpn.fedoraproject.org, log date 2023-05-31, attempt 1 rsync: [sender] link_stat "/httpd/*20230531*" (in log) failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3] ... host proxy31.vpn.fedoraproject.org, log date 2023-05-31, attempt 2 ... host proxy32.vpn.fedoraproject.org, log date 2023-06-01, linking back undated logfiles from dated ... host proxy32.vpn.fedoraproject.org, log date 2023-05-31, seeding dated logfiles from undated ... host proxy32.vpn.fedoraproject.org, log date 2023-05-31, attempt 1 rsync: [sender] link_stat "/httpd/*20230531*" (in log) failed: No such file or directory (2) ... host proxy32.vpn.fedoraproject.org, log date 2023-05-31, linking back undated logfiles from dated rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3] ... host proxy32.vpn.fedoraproject.org, log date 2023-05-30, seeding dated logfiles from undated ... host proxy31.vpn.fedoraproject.org, log date 2023-05-31, attempt 3 ... host proxy32.vpn.fedoraproject.org, log date 2023-05-30, attempt 1 ... host proxy32.vpn.fedoraproject.org, log date 2023-05-30, linking back undated logfiles from dated rsync: [sender] link_stat "/httpd/*20230531*" (in log) failed: No such file or directory (2) Finished sync from proxy32.vpn.fedoraproject.org. Started sync from proxy33.vpn.fedoraproject.org. ... host proxy33.vpn.fedoraproject.org, log date 2023-06-01, seeding dated logfiles from undated ... host proxy33.vpn.fedoraproject.org, log date 2023-06-01, attempt 1 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3] ... host proxy31.vpn.fedoraproject.org, log date 2023-05-31, linking back undated logfiles from dated ... host proxy31.vpn.fedoraproject.org, log date 2023-05-30, seeding dated logfiles from undated ... host proxy31.vpn.fedoraproject.org, log date 2023-05-30, attempt 1 rsync: [sender] link_stat "/httpd/*20230530*" (in log) failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3] ... host proxy31.vpn.fedoraproject.org, log date 2023-05-30, attempt 2 ... host proxy33.vpn.fedoraproject.org, log date 2023-06-01, linking back undated logfiles from dated ... host proxy33.vpn.fedoraproject.org, log date 2023-05-31, seeding dated logfiles from undated ... host proxy33.vpn.fedoraproject.org, log date 2023-05-31, attempt 1 rsync: [sender] link_stat "/httpd/*20230530*" (in log) failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3] ... host proxy31.vpn.fedoraproject.org, log date 2023-05-30, attempt 3 rsync: [sender] link_stat "/httpd/*20230530*" (in log) failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3] ... host proxy31.vpn.fedoraproject.org, log date 2023-05-30, linking back undated logfiles from dated Finished sync from proxy31.vpn.fedoraproject.org.
May 25 00:00:04 proxy31.fedoraproject.org systemd[1]: Starting logrotate.service - Rotate log files... May 25 00:00:04 proxy31.fedoraproject.org audit[2954978]: AVC avc: denied { read } for pid=2954978 comm="logrotate" name="httpd" dev="nvme1n1p1" ino=5111810 scontext=system_u:system_r:logrotate_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=0
rsyslog was in a weird state, but not sure why it got that denial there. We should probibly investigate this asap on monday at the latest so we can save the data...
Metadata Update from @zlopez: - Issue tagged with: Needs investigation
ok. Is there anything we need to still do here? Or should we just move on?
Let's close it as this didn't happened again and it's unlikely we can see what happened then or why.
Metadata Update from @zlopez: - Issue close_status updated to: Insufficient data - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.