Fix Cloudflare stripping Content-Length on HEAD requests for text/plain

[elmiomar]

Problem

After Cloudflare was added in front of data.nist.gov, HEAD requests for .txt files are missing the Content-Length header. Clients that rely on this header to determine file size get no value.

curl -I https://data.nist.gov/od/ds/mds2-4022/README.txt

This returns 200 OK but Content-Length is missing

Other file types (e.g. .csv) seem to be unaffected.

curl -I https://data.nist.gov/od/ds/mds2-4022/Fig2c.csv

Query Results

Method	File	Content-Type	Content-Length	Result
HEAD	README.txt	text/plain	MISSING	curl -I returns but no Content-Length; curl -X HEAD hangs
HEAD	Fig2c.csv	text/csv	338	works
GET	README.txt	(redirect)	0	302 to S3, works

Note: curl -I (the proper HEAD) returns immediately because it knows not to expect a body. curl -X HEAD overrides the method but curl still internally waits for a body, without Content-Length it doesn't know to stop, so it hangs until the connection times out.

Root Cause

Based on a Cloudflare community discussion where users reported the same behavior, my guess is that Cloudflare compresses text/plain responses on the fly. When it does, it strips Content-Length because the compressed size is unknown at the time headers are sent. For GET requests this is fine, Cloudflare uses Transfer-Encoding: chunked instead. For HEAD requests (no body), there is nothing to chunk, so Content-Length just disappears.

A Cloudflare representative said:

"We're compressing it on the fly and don't know what the length will be for the object when we return the headers."

And then said:

"As soon as I add a Cache-Control: no-transform header to the file on the Origin, I do receive a content-length header."

So based on that, the no-transform directive should come from the origin response, and setting it via Cloudflare Transform Rules or at the CDN layer has no effect.

Fix

I added Cache-Control: no-transform to the origin response via Spring Security's header configuration in CacheSecurityConfig.java. This disables Spring Security's default Cache-Control header and replaces it with one that includes no-transform, which will tell Cloudflare not to compress the response.

Why in CacheSecurityConfig and not in the controller

Another option is to add the header manually in the HEAD handler (DatasetAccessController.downloadFileInfo()). But placing it in the security config:

• Applies to all responses globally, not just HEAD, which could avoid similar issues with other endpoints that are sitting behind Cloudflare (thinking rclone)
• Avoids defining the header manually in multiple controller methods
• Is the only place where security-related headers are managed

Duplicate Cache-Control headers

The response currently shows two Cache-Control headers:

cache-control: no-cache, no-store, max-age=0, must-revalidate
cache-control: no-store, no-cache, no-transform, must-revalidate, proxy-revalidate, max-age=0

The nginx reverse proxy does not add any Cache-Control for the /od/ds/ block, so it is not affect the cache-control directives.

The first comes from Spring Security's defaults (no no-transform), and I am assuming the second comes from Cloudflare itself, but Cloudflare only checks the origin's headers when deciding whether to compress, not its own additions. This fix replaces the first line with a version that includes no-transform, so the origin explicitly signals no compression.

Changes

• CacheSecurityConfig.java: I disabled Spring Security's default Cache-Control and added a custom one with no-transform
• DatasetAccessControllerTest.java: I added test assertions that HEAD responses include Content-Length and Cache-Control: no-transform

Test

All DatasetAccessControllerTest tests pass, including the new assertions.

Verification

This needs to be deployed to production, and run the curl commands again:

curl -I https://data.nist.gov/od/ds/mds2-4022/README.txt
curl -I https://data.nist.gov/od/ds/mds2-4022/Fig2c.csv

Without access to the Cloudflare config, I can't confirm exactly how it is configured or what is stripping the header. This fix is a best-effort attempt based on community discussions. If Content-Length is still missing after deploy, we should check the Cloudflare config.

[RayPlante]

We've since determined that the Cache-Control header was not ultimately the cause of the problem that motivated this change, this PR works as intended, and it makes sense to make the header fully complete. Will merge.