fedora-infrastructure

#11305 Some docs.fedoraproject.org mirror sites don't update sind April 25, 2023
Closed: Fixed with Explanation by kevin. Opened by pboy.

Closed: Fixed with Explanation
Reopen Issue

Describe what you would like us to do:

The problem is that from my homelab as well from the datacenter of some of my servers I get mostly outdated version of docs.fedoraproject.org (and subpages) with build date 2023-04-25 (constantly), not the current build (which is created every hour, every day).

Their seems to be a difference berween IPv6 and IPv4 with IPv6 outdated most of the time and IPv4 mostly up-to-date. But that is merely a feeling. The system always uses IPv6 when available.

With IPv4 my home lab switches between several IPv4 addresses, with IPv6 it takes always the same.

Some examples:

test 01: 07 May 2023 16:02 CEST IPv4, content outdated

testfedora % curl -v -4 https://docs.fedoraproject.org/en_US/
* Trying 85.236.55.6...
* TCP_NODELAY set
* Connected to docs.fedoraproject.org (85.236.55.6) port 443 (#0)
...
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 404
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-frame-options: SAMEORIGIN
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< referrer-policy: same-origin
< last-modified: Tue, 25 Apr 2023 23:52:58 GMT <--- Outdated
< etag: "4fde-5fa31d2aca991"
< accept-ranges: bytes
< content-length: 20446
< vary: Accept-Encoding
< apptime: D=253
< x-fedora-proxyserver: proxy02.fedoraproject.org
< x-fedora-requestid: ZFevh1xUMrZCUZJ5vQWD-AAQ0RY
< content-type: text/html
< date: Sun, 07 May 2023 14:02:47 GMT
< server: Apache
< 

<p class="text-center text-xs text-fp-gray-dark">Last build: 2023-04-25 23:50:09 UTC </p>.   <--- OUTDATED


* Connection #0 to host docs.fedoraproject.org left intact
* Closing connection 0

traceroute to 85.236.55.6 (85.236.55.6), 64 hops max, 52 byte packets
1 fritz.box (192.168.158.1) 0.811 ms 0.508 ms 0.443 ms
2 p3e9bf31d.dip0.t-ipconnect.de (62.155.243.29) 8.561 ms 9.716 ms 8.243 ms
3 m-ec4-i.m.de.net.dtag.de (217.0.202.170) 22.864 ms
m-ec4-i.m.de.net.dtag.de (217.0.202.174) 23.116 ms
m-ec4-i.m.de.net.dtag.de (217.0.202.182) 23.190 ms
4 87.190.232.82 (87.190.232.82) 24.347 ms 24.637 ms
87.190.232.83 (87.190.232.83) 41.180 ms
5 c02-vl904.muc01.net.internetx.com (85.236.32.146) 24.220 ms
c01-vl903.muc01.net.internetx.com (85.236.32.142) 25.745 ms 25.842 ms
6 6-55-236-85.rev.customer-net.de (85.236.55.6) 22.636 ms !Z 22.477 ms !Z 22.623 ms !Z

test 02. Sun, 07 May 2023 16:10 CEST. IPv4, Content outdated

  • Trying 152.19.134.142...
  • TCP_NODELAY set
  • Connected to docs.fedoraproject.org (152.19.134.142) port 443 (#0)
    ...
  • Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
    ...
    < last-modified: Tue, 25 Apr 2023 23:52:58 GMT <--- outdated
    ...
    < server: Apache
    <

...

Last build: 2023-04-25 23:50:09 UTC

. <---- content outdated
...

* Connection #0 to host docs.fedoraproject.org left intact
* Closing connection 0

traceroute to 152.19.134.142 (152.19.134.142), 64 hops max, 52 byte packets
1 fritz.box (192.168.158.1) 0.850 ms 0.474 ms 0.412 ms
2 p3e9bf31d.dip0.t-ipconnect.de (62.155.243.29) 8.327 ms 8.168 ms 8.397 ms
3 f-eh1-i.f.de.net.dtag.de (62.154.4.165) 14.835 ms 15.085 ms 15.207 ms
4 * * *
5 4.69.219.146 (4.69.219.146) 118.578 ms 117.351 ms 117.511 ms
6 mcnc.edge6.atlanta2.level3.net (4.15.180.74) 123.813 ms 123.743 ms 123.366 ms
7 ws-gw-to-hntvl-gw.ncren.net (128.109.9.22) 123.519 ms 123.686 ms 123.326 ms
8 uncphillips-to-ws-gw.ncren.net (128.109.1.90) 128.311 ms 128.014 ms 127.859 ms
9 core-p-v1213.net.unc.edu (152.2.255.66) 126.267 ms 126.419 ms 126.594 ms
10 core-m-v1528.net.unc.edu (152.2.255.166) 126.817 ms 126.400 ms 126.634 ms
11 vm3.fedora.ibiblio.org (152.19.134.142) 124.155 ms !Z 124.692 ms !Z 124.317 ms !Z

Test 03: Mon, 08 May 2023 08:58 CEST, Content up-to-date

% curl -v -4 https://docs.fedoraproject.org/en_US/
* Trying 38.145.60.20...
* TCP_NODELAY set
* Connected to docs.fedoraproject.org (38.145.60.20) port 443 (#0)
...
* Using Stream ID: 1 (easy handle 0x7f85a9810800)

GET /en_US/ HTTP/2
Host: docs.fedoraproject.org
User-Agent: curl/7.64.1
Accept: /

  • Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
    < HTTP/2 404
    < strict-transport-security: max-age=31536000; includeSubDomains; preload
    < x-frame-options: SAMEORIGIN
    < x-xss-protection: 1; mode=block
    < x-content-type-options: nosniff
    < referrer-policy: same-origin
    < last-modified: Mon, 08 May 2023 05:53:00 GMT. <---- Looks fine!
    < etag: "4fde-5fb28405ea0ac"
    < accept-ranges: bytes
    < content-length: 20446
    < vary: Accept-Encoding
    < apptime: D=387
    < x-fedora-proxyserver: proxy01.iad2.fedoraproject.org
    < x-fedora-requestid: ZFidhm0DHqj4CmQwFyg1ggADDgM
    < content-type: text/html
    < date: Mon, 08 May 2023 06:58:14 GMT
    < server: Apache
    < 
<p class="text-center text-xs text-fp-gray-dark">Last build: 2023-05-08 05:50:09 UTC </p>.   <---- content uptodate



* Connection #0 to host docs.fedoraproject.org left intact

Test 04: Mon, 08 May 2023 09:06:03 CEST, Content outdated (5 Minutes later, another IP)

% curl -v -4 https://docs.fedoraproject.org/en_US/
* Trying 152.19.134.142...
* TCP_NODELAY set
* Connected to docs.fedoraproject.org (152.19.134.142) port 443 (#0)
...
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd3ca810800)

GET /en_US/ HTTP/2
Host: docs.fedoraproject.org
User-Agent: curl/7.64.1
Accept: /

  • Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
    < HTTP/2 404
    < strict-transport-security: max-age=31536000; includeSubDomains; preload
    < x-frame-options: SAMEORIGIN
    < x-xss-protection: 1; mode=block
    < x-content-type-options: nosniff
    < referrer-policy: same-origin
    < last-modified: Tue, 25 Apr 2023 23:52:58 GMT. <---- Looks bad
    < etag: "4fde-5fa31d2aca991"
    < accept-ranges: bytes
    < content-length: 20446
    < vary: Accept-Encoding
    < apptime: D=349
    < x-fedora-proxyserver: proxy04.fedoraproject.org
    < x-fedora-requestid: ZFifWw2spbtUND1UuVQRRwATjAw
    < content-type: text/html
    < date: Mon, 08 May 2023 07:06:03 GMT
    < server: Apache
    <
...

Last build: 2023-04-25 23:50:09 UTC

. <----- content outdated ...
  • Connection #0 to host docs.fedoraproject.org left intact
  • Closing connection 0

% traceroute 152.19.134.142
traceroute to 152.19.134.142 (152.19.134.142), 64 hops max, 52 byte packets
1 fritz.box (192.168.158.1) 0.739 ms 0.480 ms 0.433 ms
2 p3e9bf31d.dip0.t-ipconnect.de (62.155.243.29) 8.487 ms 8.853 ms 8.658 ms
3 f-eh1-i.f.de.net.dtag.de (62.154.4.165) 15.177 ms 15.633 ms 15.507 ms
4 * * *
5 4.69.219.146 (4.69.219.146) 118.327 ms 117.743 ms 118.678 ms
6 mcnc.edge6.atlanta2.level3.net (4.15.180.74) 123.968 ms 123.521 ms 123.454 ms
7 ws-gw-to-hntvl-gw.ncren.net (128.109.9.22) 123.194 ms 124.018 ms 123.792 ms
8 uncphillips-to-ws-gw.ncren.net (128.109.1.90) 128.086 ms 128.552 ms 128.349 ms
9 core-p-v1213.net.unc.edu (152.2.255.66) 126.304 ms 126.293 ms 126.153 ms
10 core-m-v1528.net.unc.edu (152.2.255.166) 126.975 ms 127.123 ms 126.721 ms
11 vm3.fedora.ibiblio.org (152.19.134.142) 124.214 ms !Z 127.584 ms !Z 125.291 ms !Z

Test 05: Mon, 08 May 2023 09:06:03 CEST, IPv6, home lab, Content outdated

% curl -v https://docs.fedoraproject.org/en-US/fedora-server/
* Trying 2001:4178:2:1269::fed2...
* TCP_NODELAY set
* Connected to docs.fedoraproject.org (2001:4178:2:1269::fed2) port 443 (#0)
...
< last-modified: Tue, 25 Apr 2023 23:50:57 GMT. <---- Outdated
... 

<p class="text-center text-xs text-fp-gray-dark">Last build: 2023-04-25 23:50:09 UTC  | Last content update: 2022-12-17 </p>


* Connection #0 to host docs.fedoraproject.org left intact
* Closing connection 0
traceroute6 2001:4178:2:1269::fed2
traceroute6 to 2001:4178:2:1269::fed2 (2001:4178:2:1269::fed2) from 2003:ca:7f1e:8600:5014:b0a1:f02c:aa10, 64 hops max, 12 byte packets
1 fritz.box 0.926 ms 0.522 ms 0.579 ms
2 2003:0:8500:e000::1 11.302 ms 11.733 ms 11.964 ms
3 * * *
4 2003:0:1808:a::2 27.322 ms 27.445 ms
2003:0:1808:a::3 24.336 ms
5 2001:4178:1::112 58.979 ms
2001:4178:1::11a 55.473 ms
2001:4178:1::112 59.006 ms
6 2001:4178:2:1269::fed2 23.621 ms !P 23.433 ms !P 25.464 ms !P

Test 06: 11 May 2023 03:27 CEST, IPv6, Content uptodate

curl -v https://docs.fedoraproject.org/en-US/fedora-server/
* Trying 2a05:d01c:c6a:cc01:269:da52:9ae1:43e6:443...
* Connected to docs.fedoraproject.org (2a05:d01c:c6a:cc01:269:da52:9ae1:43e6) port 443 (#0)
< last-modified: Thu, 11 May 2023 00:50:57 GMT

Last build: 2023-05-11 00:50:09 UTC | Last content update: 2023-05-01


* Connection #0 to host docs.fedoraproject.org left intact

traceroute 2a05:d01c:c6a:cc01:269:da52:9ae1:43e6
traceroute to 2a05:d01c:c6a:cc01:269:da52:9ae1:43e6 (2a05:d01c:c6a:cc01:269:da52:9ae1:43e6), 30 hops max, 80 byte packets
1 ex9k1.dc3.fsn1.hetzner.com (2a01:4f8::a:12:a) 0.346 ms 0.307 ms 0.382 ms
2 core23.fsn1.hetzner.com (2a01:4f8:0:3::281) 0.578 ms 0.557 ms 0.598 ms
3 core0.fra.hetzner.com (2a01:4f8:0:3::4c2) 4.915 ms core1.fra.hetzner.com (2a01:4f8:0:3::4b6) 4.897 ms core0.fra.hetzner.com (2a01:4f8:0:3::4c2) 4.881 ms
4 2a01:4f8:0:e0f0::10e (2a01:4f8:0:e0f0::10e) 5.230 ms 2620:107:4008:39d::1 (2620:107:4008:39d::1) 5.396 ms 2a01:4f8:0:e0f0::10e (2a01:4f8:0:e0f0::10e) 5.200 ms
5 2a01:578:0:8003::81 (2a01:578:0:8003::81) 6.038 ms 2a01:578:0:8003::85 (2a01:578:0:8003::85) 5.659 ms 5.635 ms
6 2a01:578:0:12::33 (2a01:578:0:12::33) 18.479 ms 2a01:578:0:12::44 (2a01:578:0:12::44) 18.702 ms 2a01:578:0:12::28 (2a01:578:0:12::28) 17.697 ms
7 2a01:578:0:12::7c (2a01:578:0:12::7c) 21.165 ms 2a01:578:0:12::79 (2a01:578:0:12::79) 30.683 ms 2a01:578:0:12::74 (2a01:578:0:12::74) 22.368 ms
8 2a01:578:0:12::2a (2a01:578:0:12::2a) 18.941 ms 2a01:578:0:12::37 (2a01:578:0:12::37) 19.242 ms 2a01:578:0:12::2a (2a01:578:0:12::2a) 17.541 ms
9 2a01:578:0:12::7e (2a01:578:0:12::7e) 21.145 ms 2a01:578:0:12::7f (2a01:578:0:12::7f) 20.179 ms 2a01:578:0:12::7a (2a01:578:0:12::7a) 23.274 ms
10 2a01:578:0:12::58 (2a01:578:0:12::58) 23.661 ms 2a01:578:0:12::59 (2a01:578:0:12::59) 18.320 ms 19.605 ms
11 2a01:578:0:12::3 (2a01:578:0:12::3) 22.212 ms 20.318 ms 20.292 ms
12 2a01:578:0:4200:8000:0:6441:a40 (2a01:578:0:4200:8000:0:6441:a40) 22.387 ms 2a01:578:0:4200:8000:0:6441:ac0 (2a01:578:0:4200:8000:0:6441:ac0) 20.545 ms 2a01:578:0:4200:8000:0:6441:8a0 (2a01:578:0:4200:8000:0:6441:8a0) 22.387 ms
13 2a01:578:0:4200:8000:0:6442:4a0 (2a01:578:0:4200:8000:0:6442:4a0) 30.422 ms 2a01:578:0:4200:8000:0:6442:460 (2a01:578:0:4200:8000:0:6442:460) 31.162 ms 31.146 ms
14 2a01:578:0:4200:8000:0:6442:7ee (2a01:578:0:4200:8000:0:6442:7ee) 20.942 ms 2a01:578:0:4200:8000:0:6442:7c8 (2a01:578:0:4200:8000:0:6442:7c8) 18.954 ms 2a01:578:0:4200:8000:0:6442:642 (2a01:578:0:4200:8000:0:6442:642) 21.806 ms
15 2a01:578:0:4200:8000:0:6442:ba1 (2a01:578:0:4200:8000:0:6442:ba1) 21.523 ms 2a01:578:0:4200:8000:0:6442:a8d (2a01:578:0:4200:8000:0:6442:a8d) 78.390 ms 2a01:578:0:4200:8000:0:6442:b8d (2a01:578:0:4200:8000:0:6442:b8d) 95.494 ms
16 2a01:578:0:4200:8000:0:6442:8d1 (2a01:578:0:4200:8000:0:6442:8d1) 23.301 ms 23.285 ms 2a01:578:0:4200:8000:0:6442:8f9 (2a01:578:0:4200:8000:0:6442:8f9) 20.360 ms
17 * * *
18 2a05:d01c:c6a:cc01:269:da52:9ae1:43e6 (2a05:d01c:c6a:cc01:269:da52:9ae1:43e6) 19.643 ms !X 19.621 ms !X 19.011 ms !X

When do you need this to be done by? (YYYY/MM/DD)

asap

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: high-gain, medium-trouble, ops

Metadata Update from @kevin:
- Issue assigned to kevin

ok. So, I found that 3 proxies were 'stuck' syncing.

They were running rsync and taking up 100% cpu and just never finishing.

strace showed some kind of weird loop:

newfstatat(AT_FDCWD, "pl/_/css/search.css", {st_mode=S_IFREG|0666, st_size=2988, ...}, AT_SYML
INK_NOFOLLOW) = 0

I killed all those and am re-running the sync to see if it's a one off thing or something wrong on those hosts.

ok, they all completed.

So, I think this is fixed and it was some network oddity on the 26th.

Please re-open if you still see anything here.

Thanks for the report!

Metadata Update from @kevin:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)
Metadata

high-gain medium-trouble ops

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.