#11331 Syncing of mirror log files to log01 broke around May 5th-10th, bad DNF countme data
Closed: Insufficient data a year ago by zlopez. Opened a year ago by james.

% pwd
/mnt/fedora_stats/combined-http/2023/05
% ls -1hs 0*/mirrors.fedoraproject.org-access.log
9.4G 01/mirrors.fedoraproject.org-access.log
9.9G 02/mirrors.fedoraproject.org-access.log
2.0G 03/mirrors.fedoraproject.org-access.log
2.0G 04/mirrors.fedoraproject.org-access.log
2.1G 05/mirrors.fedoraproject.org-access.log
2.0G 06/mirrors.fedoraproject.org-access.log
1.8G 07/mirrors.fedoraproject.org-access.log
1.9G 08/mirrors.fedoraproject.org-access.log
2.0G 09/mirrors.fedoraproject.org-access.log

...everything seems to be fixed from May 10th onwards.

So we have a few different issues which have different priority:

  1. Need to make sure the data is kept around at the source, so we don't lose it (urgent/high).
  2. Need to generate new combined logs (medium-ish but within a couple of weeks)
  3. Need to regenerate the countme data for that period (medium-ish, but within a couple of weeks)
  4. Need to work out what happened (low)
  5. Need to work out some way infra. can be notified if this happens again (medium - but doesn't need to be soon -- also I can probably volunteer for some of this).

Anything not caught within 7 days is usually removed from the proxies by logrotate. The servers run out of general disk space quickly otherwise. The oldest log currently is the 20230516 logs so all the missing days are gone. I would say that means 1, 2 and 3 are off the table.

Getting 5 and 4 worked out are what is left to be done.

Ok, I guess that means we can drop the priority way down. Although I suspect Matt is now much more anxious about 5 happening soon.

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee

a year ago

I noticed some errors today in the cron job:

Started sync from proxy31.vpn.fedoraproject.org.
... host proxy31.vpn.fedoraproject.org, log date 2023-06-01, seeding dated logfiles from undated
... host proxy31.vpn.fedoraproject.org, log date 2023-06-01, attempt 1
rsync: [sender] link_stat "/httpd/*20230601*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-06-01, attempt 2
rsync: [sender] link_stat "/httpd/*20230601*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-06-01, attempt 3
rsync: [sender] link_stat "/httpd/*20230601*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-06-01, linking back undated logfiles from dated
... host proxy31.vpn.fedoraproject.org, log date 2023-05-31, seeding dated logfiles from undated
... host proxy31.vpn.fedoraproject.org, log date 2023-05-31, attempt 1
rsync: [sender] link_stat "/httpd/*20230531*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-05-31, attempt 2
... host proxy32.vpn.fedoraproject.org, log date 2023-06-01, linking back undated logfiles from dated
... host proxy32.vpn.fedoraproject.org, log date 2023-05-31, seeding dated logfiles from undated
... host proxy32.vpn.fedoraproject.org, log date 2023-05-31, attempt 1
rsync: [sender] link_stat "/httpd/*20230531*" (in log) failed: No such file or directory (2)
... host proxy32.vpn.fedoraproject.org, log date 2023-05-31, linking back undated logfiles from dated
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy32.vpn.fedoraproject.org, log date 2023-05-30, seeding dated logfiles from undated
... host proxy31.vpn.fedoraproject.org, log date 2023-05-31, attempt 3
... host proxy32.vpn.fedoraproject.org, log date 2023-05-30, attempt 1
... host proxy32.vpn.fedoraproject.org, log date 2023-05-30, linking back undated logfiles from dated
rsync: [sender] link_stat "/httpd/*20230531*" (in log) failed: No such file or directory (2)
Finished sync from proxy32.vpn.fedoraproject.org.
Started sync from proxy33.vpn.fedoraproject.org.
... host proxy33.vpn.fedoraproject.org, log date 2023-06-01, seeding dated logfiles from undated
... host proxy33.vpn.fedoraproject.org, log date 2023-06-01, attempt 1
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-05-31, linking back undated logfiles from dated
... host proxy31.vpn.fedoraproject.org, log date 2023-05-30, seeding dated logfiles from undated
... host proxy31.vpn.fedoraproject.org, log date 2023-05-30, attempt 1
rsync: [sender] link_stat "/httpd/*20230530*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-05-30, attempt 2
... host proxy33.vpn.fedoraproject.org, log date 2023-06-01, linking back undated logfiles from dated
... host proxy33.vpn.fedoraproject.org, log date 2023-05-31, seeding dated logfiles from undated
... host proxy33.vpn.fedoraproject.org, log date 2023-05-31, attempt 1
rsync: [sender] link_stat "/httpd/*20230530*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-05-30, attempt 3
rsync: [sender] link_stat "/httpd/*20230530*" (in log) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [Receiver=3.1.3]
... host proxy31.vpn.fedoraproject.org, log date 2023-05-30, linking back undated logfiles from dated
Finished sync from proxy31.vpn.fedoraproject.org.
May 25 00:00:04 proxy31.fedoraproject.org systemd[1]: Starting logrotate.service - Rotate log files...
May 25 00:00:04 proxy31.fedoraproject.org audit[2954978]: AVC avc:  denied  { read } for  pid=2954978 comm="logrotate" name="httpd" dev="nvme1n1p1" ino=5111810 scontext=system_u:system_r:logrotate_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=0

rsyslog was in a weird state, but not sure why it got that denial there. We should probibly investigate this asap on monday at the latest so we can save the data...

Metadata Update from @zlopez:
- Issue tagged with: Needs investigation

a year ago

ok. Is there anything we need to still do here? Or should we just move on?

Let's close it as this didn't happened again and it's unlikely we can see what happened then or why.

Metadata Update from @zlopez:
- Issue close_status updated to: Insufficient data
- Issue status updated to: Closed (was: Open)

a year ago

Log in to comment on this ticket.

Metadata