Issue #12812: dist-git lookaside cache files served with wrong HTTP headers - fedora-infrastructure

fedora-infrastructure

#12812 dist-git lookaside cache files served with wrong HTTP headers

Closed: Migrated to Fedora Forge Feb 2, 2026 by ryanlerch. Opened Sep 21, 2025 by decathorpe.

It looks like .tar.gz / .crate files that are served by the lookaside cache have the following http headers set:

content-type: application/x-tar
content-encoding: x-gzip

This causes some clients (those without workarounds) to uncompress .tar.gz files on-the-fly while downloading, which results in plain tarfiles being stored on disk (but with .tar.gz file extensions).

It looks like fedpkg sources already works despite that, similar to how spectool -g also handles this case (due to a workaround that's present in its download handling code that suppresses on-the-fly decompression of gzipped files).

An indicator for when this breaks is checksum mismatches, like the one that happened in https://koji.fedoraproject.org/koji/taskinfo?taskID=137346319

Also see https://discussion.fedoraproject.org/t/verifying-the-authenticity-of-files-uploaded-to-the-lookaside-cache/134196 for a discussion thread where the "clients uncompress stuff on-the-fly even if you really don't want that to happen" was also relevant.

In general, it looks like the lookaside cache should not guess the file's MIME type or encoding, but always set content-type: application/octet-stream (which is the default for unknown / binary data): https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/MIME_types/Common_types and send responses with no content-encoding header present at all (since the data is supposed to be used as-is, without decoding first)?

The headers that are sent currently (content-type: application/x-tar, content-encoding: x-gzip) basically tell clients to "decompress this with gzip to get the .tar file you ordered", which is wrong and explains why clients that don't specifically override this behaviour get un-gzipped plain "tar" files (with ".tar.gz" file endings) or un-gzipped ".crate" files (which are just .tar.gz in disguise).

Additionally, for an as of yet unknown reason, anubis-enabled src.fedoraproject.org seems to have caused fedpkg sources from fedpkg-minimal to break with this - or a very similar - failure mode, causing checksum mismatches during koji builds. I don't see a difference in the HTTP headers that would explain that difference in behaviour though. :(

kevin commented Sep 21, 2025

To note: I have disabled anubis for src.fedoraproject.org for now, but we should figure out a workaround so we can re-enable it.

decathorpe commented Sep 22, 2025

It should be relatively easy to work around this issue in clients like fedpkg-minimal - spectool does two things for this case:

set Accept-Encoding: identity for GET requests
set decode_content=False on the stream

It appears that both are needed to cover all cases (i.e. don't break downloading plain-text files and don't decompress gzipped files sent by confused servers).

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: high-gain, medium-trouble, ops

Sep 22, 2025

kevin commented Sep 22, 2025

Yeah, although with that we don't fix it for external users? or do we... does 'fedpkg' (non minimal) do the right thing?

decathorpe commented Sep 22, 2025

It appears that non-minimal fedpkg sources downloads the files correctly, but I'm not sure what's the difference - fedpkg uses pycurl and fedpkg minimal uses the curl CLI, and it looks like they use the same settings (i.e. don't appear to set any headers at all, other than Pragma:). So, I'm quite confused. :)

kevin commented Sep 23, 2025

so... can you duplcate this with src.stg.fedoraproject.org?

(we may need to upload/sync some lookaside files there)

if so, I can reenable anubis there and I can try some fixes on the server end...

music commented Sep 24, 2025

ben@bean:~/fedora/rust-sig/rust-jod-thread$ fedpkg -d -v sources
Creating repo object from /home/ben/fedora/rust-sig/rust-jod-thread
Initiating a koji session to https://koji.fedoraproject.org/kojihub
Downloading jod-thread-1.0.0.crate from https://src.fedoraproject.org/repo/pkgs
Full url: https://src.fedoraproject.org/repo/pkgs/rpms/rust-jod-thread/jod-thread-1.0.0.crate/sha512/14d0851a0a7d8d805a81313e6ec60a778267acd83f600d259dffbf63fe3f7ebd6a8d98d3ed49a1cb271ff024fac2c35acc1b287d5fb91f4bbe52bb3df3f2b4b1/jod-thread-1.0.0.crate
######################################################################## 100.0%
$ curl -L https://src.fedoraproject.org/repo/pkgs/rpms/rust-jod-thread/jod-thread-1.0.0.crate/sha512/14d0851a0a7d8d805a81313e6ec60a778267acd83f600d259dffbf63fe3f7ebd6a8d98d3ed49a1cb271ff024fac2c35acc1b287d5fb91f4bbe52bb3df3f2b4b1/jod-thread-1.0.0.crate | tar -tz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6407  100  6407    0     0  18165      0 --:--:-- --:--:-- --:--:-- 18150
jod-thread-1.0.0/.cargo_vcs_info.json
jod-thread-1.0.0/.gitignore
jod-thread-1.0.0/Cargo.lock
jod-thread-1.0.0/Cargo.toml
jod-thread-1.0.0/Cargo.toml.orig
jod-thread-1.0.0/LICENSE-APACHE
jod-thread-1.0.0/LICENSE-MIT
jod-thread-1.0.0/README.md
jod-thread-1.0.0/src/lib.rs
$ curl -L https://src.stg.fedoraproject.org/repo/pkgs/rpms/rust-jod-thread/jod-thread-1.0.0.crate/sha512/14d0851a0a7d8d805a81313e6ec60a778267acd83f600d259dffbf63fe3f7ebd6a8d98d3ed49a1cb271ff024fac2c35acc1b287d5fb91f4bbe52bb3df3f2b4b1/jod-thread-1.0.0.crate
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

Is that what you meant to try?

decathorpe commented Sep 24, 2025

I can't seem to find any packages that have uploaded sources to the stg lookaside cache. Do you have an example?

kevin commented Sep 24, 2025

The intent was that stg would have a read only copy of the prod lookaside.

It did, but there was a messed up link to it. ;)

Can you try again now?

decathorpe commented Sep 24, 2025

$ curl -I https://src.stg.fedoraproject.org/repo/pkgs/rpms/rust-jod-thread/jod-thread-1.0.0.crate/sha512/14d0851a0a7d8d805a81313e6ec60a778267acd83f600d259dffbf63fe3f7ebd6a8d98d3ed49a1cb271ff024fac2c35acc1b287d5fb91f4bbe52bb3df3f2b4b1/jod-thread-1.0.0.crate
(...)
content-type: application/x-tar
content-encoding: x-gzip
(...)

Which are the same headers as for the non-stg variant.

Running wget for both URLs (with and without .stg domain) gives me plain uncompressed .tar archives:

$ file jod-thread-1.0.0.crate 
jod-thread-1.0.0.crate: POSIX tar archive (GNU)

... which is exactly what the HTTP headers sent by the server tell wget to do.

Coincidentally, curl ing the same URLs gives me:

$ file jod-thread-1.0.0.crate
jod-thread-1.0.0.crate: gzip compressed data, was "jod-thread-1.0.0.crate", max compression, original size modulo 2^32 25088

So behaviour of what's saved to disk seems to depend on the tool that's used. FWIW I don't think "curl" is doing the correct thing here. :)

kevin commented Oct 10, 2025

So I am a bit confused here.

stg is behaving the same was a prod? Or it's not?

The reason I wanted to check that is so that I could try and change things in staging to find a 'fix' on the server side and then we could deploy it on prod.

I just noticed that I had disabled anubis in staging src too. I am turning it back on so we can see it it has the behavior...

decathorpe commented Oct 10, 2025

So I am a bit confused here.

stg is behaving the same was a prod? Or it's not?

As far as I can tell, they're giving HTTP responses with identical headers, yes - but if you disabled anubis on stg.src.fp.o too that is kind of expected :)

I'll try again later.

kevin commented Oct 10, 2025

anubis is now enabled on src.stg again.

decathorpe commented Oct 10, 2025

I tried again, looks like curl is still giving me (mostly) the same HTTP headers for both:

$ curl -I https://src.fedoraproject.org/repo/pkgs/rpms/rust-jod-thread/jod-thread-1.0.0.crate/sha512/14d0851a0a7d8d805a81313e6ec60a778267acd83f600d259dffbf63fe3f7ebd6a8d98d3ed49a1cb271ff024fac2c35acc1b287d5fb91f4bbe52bb3df3f2b4b1/jod-thread-1.0.0.crate | sort 
accept-ranges: bytes
apptime: D=64700
content-encoding: x-gzip
content-length: 6407
content-type: application/x-tar
date: Fri, 10 Oct 2025 18:21:42 GMT
etag: "1907-63f20ec0c5b00"
HTTP/2 200 
last-modified: Fri, 19 Sep 2025 05:41:32 GMT
referrer-policy: same-origin
server: Apache
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
x-fedora-appserver: pkgs01.rdu3.fedoraproject.org
x-fedora-proxyserver: proxy10.rdu3.fedoraproject.org
x-fedora-requestid: aOlOthUO8dOW_ab7KBDV2QAI0xQ
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block

$ curl -I https://src.stg.fedoraproject.org/repo/pkgs/rpms/rust-jod-thread/jod-thread-1.0.0.crate/sha512/14d0851a0a7d8d805a81313e6ec60a778267acd83f600d259dffbf63fe3f7ebd6a8d98d3ed49a1cb271ff024fac2c35acc1b287d5fb91f4bbe52bb3df3f2b4b1/jod-thread-1.0.0.crate | sort 
accept-ranges: bytes
apptime: D=73227
content-encoding: x-gzip
content-length: 6407
content-type: application/x-tar
date: Fri, 10 Oct 2025 18:21:46 GMT
etag: "1907-63f20ec0c5b00"
HTTP/2 200 
last-modified: Fri, 19 Sep 2025 05:41:32 GMT
referrer-policy: same-origin
server: Apache
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
x-fedora-appserver: pkgs01.stg.rdu3.fedoraproject.org
x-fedora-proxyserver: proxy02.stg.rdu3.fedoraproject.org
x-fedora-requestid: aOlOug3vXHWevdWTx-OcoQAAENE
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block

@music can you try reproducing the issue again? because I can't seem to see any differences here.

(The headers are still wrong IMO - they should tell clients to use the data as-is, not that it needs to be gzip compressed on receipt - but they're not different between stg and prod as far as I can tell.)

kevin commented Oct 14, 2025

Looks like someone else hit this in #12842

I have switched prod anubis off on src for now.

kevin commented Oct 15, 2025

Rough and ditry pr... I applied it in staging already, and it changes the content-type... but it still ungzips it.

https://pagure.io/fedora-infra/ansible/pull-request/2904

kevin commented Oct 15, 2025

ok. I had to put it back... things were just too bad without anubis. ;(

That said, the internal proxies that are used by builders don't have it enabled, so I think real builds should work fine.

music commented Oct 15, 2025

Rough and ditry pr... I applied it in staging already, and it changes the content-type... but it still ungzips it.

https://pagure.io/fedora-infra/ansible/pull-request/2904

Hmm, I wouldn’t expect changing just the Content-Type to affect the Content-Encoding (unless it happened to interact with a pre-existing MIME-based rule for content encoding), so at least that result is explicable.

I tried fedpkg sources for rust-jod-thread, and it’s failing again:

$ fedpkg sources
Downloading jod-thread-1.0.0.crate from https://src.fedoraproject.org/repo/pkgs
Could not execute sources: jod-thread-1.0.0.crate failed checksum

I tried fedpkg scratch-build and that failed, too:

DEBUG util.py:461:  jod-thread-1.0.0.crate: FAILED
DEBUG util.py:459:  sha512sum: WARNING: 1 computed checksum did NOT match

https://koji.fedoraproject.org/koji/taskinfo?taskID=138160053

That said, the internal proxies that are used by builders don't have it enabled, so I think real builds should work fine.

I think a (dist-git based, not --srpm) scratch build should be the same as a real build, right? If so, then I expect real builds to fail as well.

music commented Oct 15, 2025

I haven’t taken the time to dig through the Ansible configurations and understand how they actually fit together, but is the lookaside cache governed by the mod_deflate configuration in https://pagure.io/fedora-infra/ansible/blob/main/f/roles/fedora-web/main/files/deflate.conf ? Might something like this work?

<Directory /srv/cache/lookaside>
    SetOutputFilter NONE
    […]
</Directory>

I’m not sure if you would need to mess around with the Vary header as well or not, or if something like

<Directory /srv/cache/lookaside>
    SetEnv no-gzip
    SetEnv dont-vary
    […]
</Directory>

would be processed in the right order with respect to the mod_deflate configuration to have the desired effect.

james commented Oct 15, 2025

I think the "content-encoding: x-gzip" part is coming from the proxies. But while it's not useful for lookaside data, clients should deal with that.

music commented Oct 15, 2025

I think the "content-encoding: x-gzip" part is coming from the proxies. But while it's not useful for lookaside data, clients should deal with that.

The clients do deal with it, by removing the declared content-encoding to obtain what the server is claiming should be the actual payload. The problem is that the content-encoding header is lying. For it to be accurate, the HTTP response would have to be gzipped (again, on top of the gzip encoding that happens to be the outermost layer in the .crate file format).

decathorpe commented Oct 15, 2025

OOF this is now hitting me too ...

Is there anything we can do, at least as a temporary workaround? Maybe patch fedpkg-minimal (or whatever downloads the sources in the buildSRPMfromSCM task in koji) to avoid decompressing stuff on-the-fly?

decathorpe commented Oct 15, 2025

It appears that adding -H "Accept-Encoding: identity" to the curl CLI call in fedpkg-minimal works around this issue. I'll file a PR with that ...

decathorpe commented Oct 15, 2025

https://src.fedoraproject.org/rpms/fedpkg-minimal/pull-request/2

kevin commented Oct 15, 2025

I think thats fine, but it's going to take a while to get that to where it can help. I was hoping we could do something on the server side to get things working faster. ;(

Sigh, I found that dns was wrong and internal stuff was using proxy01/10 instead of proxy101/110.
I have changed it over now, please retry your scratch build from scm and I hope it will work.

decathorpe commented Oct 15, 2025

Ha, it's always DNS :laughing: I can confirm that a build that failed earlier this evening due to a checksum issue now passed. :balloon:

asn commented Oct 16, 2025

sccache also started building for me

james commented Oct 20, 2025

Okay, I'm going to close this as it looks like it was all DNS :(

Metadata Update from @james:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Oct 20, 2025

music commented Oct 20, 2025

Okay, I'm going to close this as it looks like it was all DNS :(

I wouldn’t consider this fixed. This is no longer breaking koji builds from dist-git, because the DNS issue causing koji to use the external proxies was fixed. However, things are still utterly broken for anything that accesses the lookaside cache externally, including COPR builds from dist-git (e.g. https://copr.fedorainfracloud.org/coprs/music/tungstenite/build/9704757/) and things like:

rust-jod-thread$ fedpkg sources
Downloading jod-thread-1.0.0.crate from https://src.fedoraproject.org/repo/pkgs
Could not execute sources: jod-thread-1.0.0.crate failed checksum

Trying to find, patch, and redeploy every piece of code that accesses the lookaside patch to set Accept-Encoding: identity à la https://src.fedoraproject.org/rpms/fedpkg-minimal/pull-request/2 is a plausible workaround, but the root problem is that the server and/or proxy are confused and are lying about what they are serving by tacking on a Content-Encoding: gzip header but not actually adding another layer of gzip compression.

kevin commented Oct 20, 2025

Yeah. We should still try and fix the server end if we can, or failing that at least fix fedpkg/fedpkg-minimal.

Metadata Update from @kevin:
- Issue status updated to: Open (was: Closed)

Oct 20, 2025

decathorpe commented Oct 22, 2025

Okay, I'm going to close this as it looks like it was all DNS :(

I wouldn’t consider this fixed. This is no longer breaking koji builds from dist-git, because the DNS issue causing koji to use the external proxies was fixed. However, things are still utterly broken for anything that accesses the lookaside cache externally, including COPR builds from dist-git (e.g. https://copr.fedorainfracloud.org/coprs/music/tungstenite/build/9704757/) and things like:

```
rust-jod-thread$ fedpkg sources
Downloading jod-thread-1.0.0.crate from https://src.fedoraproject.org/repo/pkgs

Could not execute sources: jod-thread-1.0.0.crate failed checksum
```

Ah, so fedpkg sources really is affected too? That explains why I couldn't find a reason why it shouldn't be :(

Does anybody want to submit a PR to fedpkg equivalent to the one I sent for fedpkg-minimal, or should I do that?

kevin commented Nov 6, 2025

ok, I updated anubis today... can you see if it changed anything? There's a bunch of changes, but it's hard to tell if any would affect this issue.

music commented Nov 6, 2025

ben@bean:~/fedora/rust-sig/rust-jod-thread$ fedpkg sources
Downloading jod-thread-1.0.0.crate from https://src.fedoraproject.org/repo/pkgs
Could not execute sources: jod-thread-1.0.0.crate failed checksum

Looks the same to me.

decathorpe commented Nov 8, 2025

Filed a PR for rpkg too: https://pagure.io/rpkg/pull-request/758 , PTAL.

music commented Nov 21, 2025

I just encountered this problem in a real Koji build again.

https://koji.fedoraproject.org/koji/taskinfo?taskID=139122958

music commented Nov 21, 2025

I just encountered this problem in a real Koji build again.

Just to clarify the impact, the fact that this is happening in koji builds from dist-git again means that a large number of Rust packages will fail to build. No external workaround exists, and much Rust packaging work will be impossible until some kind of fix is implemented.

kevin commented Nov 21, 2025

ok, I dug into this and I think I see what the problem is.

I did make changes to varnish to cache src, and I think it's doing the same thing anubis did...

I'll look at a commit to fix this.

kevin commented Nov 21, 2025

ok, please try now. I tested and it seems to be working in that testing.

music commented Nov 21, 2025

ok, please try now. I tested and it seems to be working in that testing.

Yes, it looks like it’s working again!

https://koji.fedoraproject.org/koji/taskinfo?taskID=139139537

Thank you for investigating.

kevin commented Nov 22, 2025

Sorry for the trouble... ;( I knew varnish would break something there, but it's really helping the scraper load...

music commented Nov 22, 2025

Sorry for the trouble... ;( I knew varnish would break something there, but it's really helping the scraper load...

We certainly need all the help we can get! Thanks for looking at it. I’m glad it wasn’t too difficult to get things back to the status quo.

kevin commented Jan 20, 2026

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/infra/tickets/issues/12812

Please continue any further discussion there.

Metadata Update from @ryanlerch:
- Issue close_status updated to: Migrated to Fedora Forge
- Issue status updated to: Closed (was: Open)

Feb 2, 2026

Metadata

Assignee

None

Tags

Blocking

None

Depending on

None

Priority

Normal

Related Pull Requests

#2904

fedora-infrastructure

#12812 dist-git lookaside cache files served with wrong HTTP headers Closed: Migrated to Fedora Forge Feb 2, 2026 by ryanlerch. Opened Sep 21, 2025 by decathorpe.

Metadata

high-gain medium-trouble ops

Related Pull Requests

#12812 dist-git lookaside cache files served with wrong HTTP headers

Closed: Migrated to Fedora Forge Feb 2, 2026 by ryanlerch. Opened Sep 21, 2025 by decathorpe.