[Bug][Backend] images/{id} intermittent 500: response wrote 163840 bytes against Content-Length 163409 (post-MinIO recurrence of the #758 signature) + Traefik compresses images wastefully #787

New Issue

2026-07-22T04:01:24Z

spikerj commented

2026-07-22 04:01:24 +00:00

Seq (spikerj report): one 500 at 2026-07-22 03:23Z — GET /api/Profile/images/69da67d2… threw Response Content-Length mismatch: too many bytes written (163840 of 163409) inside Kestrel (ResponseCompressionBody in stack, anonymous HeadlessChrome caller).

Investigated 2026-07-22 (evidence, all verified live):

The MinIO object is EXACTLY 163409 bytes (mc stat) and plain curls serve 163409 clean, repeatedly — store and normal path are correct.
163840 = 40×4096 — a page-rounded overshoot of +431 bytes, written by the app (Kestrel-side), single occurrence so far on the S3 path.
NOT caching: GetProfileImageQuery carries no [Cache] attribute; NOT the app's ResponseCompression mime match (defaults+json/text only).
IMPORTANT CORRECTION to closed #758: this is the same signature #758 showed in the GridFS era — so #758's root cause was NOT GridFS (I closed it attributing the fix to the MinIO cutover; the class just recurred once on S3). Cause is in the serving path (S3/HTTP stream interplay or a transient over-read), storage-agnostic.
Separate finding while investigating: Traefik's gzip-compress middleware on the backend router compresses image responses — zstd'd this AVIF to 163428 bytes (LARGER than the original). Pure CPU waste for already-compressed media.

Recommended remediations:

Defensive fix (kills the class regardless of trigger): image-serving paths should stream through a length-bounded wrapper (limit reads to the S3 response's ContentLength; throw BEFORE response start on mismatch) — S3ImageStorageService/S3OwnedFileStore are the seams.
Traefik: add excludedContentTypes for image/* (and video/*) on the compress middleware — stops the wasteful re-compression and removes one transformation layer from the diagnosis space.
Keep watching frequency: one 500/day from prerender traffic today; if it climbs, promote severity.

Seq (spikerj report): one 500 at 2026-07-22 03:23Z — `GET /api/Profile/images/69da67d2…` threw `Response Content-Length mismatch: too many bytes written (163840 of 163409)` inside Kestrel (ResponseCompressionBody in stack, anonymous HeadlessChrome caller). Investigated 2026-07-22 (evidence, all verified live): - The MinIO object is EXACTLY 163409 bytes (mc stat) and plain curls serve 163409 clean, repeatedly — store and normal path are correct. - 163840 = 40×4096 — a page-rounded overshoot of +431 bytes, written by the app (Kestrel-side), single occurrence so far on the S3 path. - NOT caching: GetProfileImageQuery carries no [Cache] attribute; NOT the app's ResponseCompression mime match (defaults+json/text only). - IMPORTANT CORRECTION to closed #758: this is the same signature #758 showed in the GridFS era — so #758's root cause was NOT GridFS (I closed it attributing the fix to the MinIO cutover; the class just recurred once on S3). Cause is in the serving path (S3/HTTP stream interplay or a transient over-read), storage-agnostic. - Separate finding while investigating: Traefik's `gzip-compress` middleware on the backend router compresses image responses — zstd'd this AVIF to 163428 bytes (LARGER than the original). Pure CPU waste for already-compressed media. Recommended remediations: 1. Defensive fix (kills the class regardless of trigger): image-serving paths should stream through a length-bounded wrapper (limit reads to the S3 response's ContentLength; throw BEFORE response start on mismatch) — S3ImageStorageService/S3OwnedFileStore are the seams. 2. Traefik: add excludedContentTypes for image/* (and video/*) on the compress middleware — stops the wasteful re-compression and removes one transformation layer from the diagnosis space. 3. Keep watching frequency: one 500/day from prerender traffic today; if it climbs, promote severity.

spikerj commented

2026-07-29 16:11:24 +00:00

Audited against origin/master — PARTIAL, and I need to flag a disagreement between this ticket and #851 rather than paper over it.

What changed: I closed #851 earlier today, which added options.ExcludedMimeTypes at SpikerSoft.Api/Program.cs:129-140. I've confirmed it covers image/*, video/*, audio/*, font/*, zip, gzip, pdf and model/gltf-binary. So ASP.NET's ResponseCompressionBody no longer touches image responses.

But neither of this ticket's own two remediations landed:

Length-bounded stream wrapper in S3ImageStorageService / S3OwnedFileStore (throw before response start on a length mismatch): git grep -rniE "boundedstream|LengthBounded|ContentLengthGuard|bytesWritten.*Length" across those seams → zero hits.
Traefik excludedContentTypes on the gzip-compress middleware: spikersoft-backend/docker-stack.yml:116 still reads traefik.http.middlewares.gzip-compress.compress=true with no exclusions, and it's still applied to the router at :107. So the edge is still gzipping images.

The disagreement worth resolving. This ticket's body explicitly rules out the app layer — "NOT the app's ResponseCompression mime match (defaults+json/text only)" — and points at Traefik. #851's investigation found the opposite: it observed content-encoding: gzip and a duplicated Vary: Accept-Encoding, concluding two layers were compressing, and fixed the app one.

Both can be true (two compressors, only one of them matching), and #851's fix is correct on its own terms — its acceptance was the exclusion list plus tests, which shipped, so it's rightly closed. But it means #851's fix may not have stopped the 500 this ticket describes, because the layer this ticket blames is untouched.

Since the trigger was never conclusively identified, git can't settle which analysis is right. What would: check whether GET api/Profile/images/{id} still returns content-encoding: gzip on an AVIF now that the app layer is excluded. If it does, Traefik is the remaining compressor and item 2 is the fix. If it doesn't, #851 covered it and this can close.

Remaining: that one curl, then item 2 (and item 1 as defence-in-depth — a bounded wrapper turns a 500 into a diagnosable throw regardless of which layer causes the overflow).

Audited against `origin/master` — **PARTIAL, and I need to flag a disagreement between this ticket and #851 rather than paper over it.** **What changed:** I closed **#851** earlier today, which added `options.ExcludedMimeTypes` at `SpikerSoft.Api/Program.cs:129-140`. I've confirmed it covers `image/*`, `video/*`, `audio/*`, `font/*`, zip, gzip, pdf and `model/gltf-binary`. So ASP.NET's `ResponseCompressionBody` no longer touches image responses. **But neither of this ticket's own two remediations landed:** 1. **Length-bounded stream wrapper** in `S3ImageStorageService` / `S3OwnedFileStore` (throw before response start on a length mismatch): `git grep -rniE "boundedstream|LengthBounded|ContentLengthGuard|bytesWritten.*Length"` across those seams → **zero hits**. 2. **Traefik `excludedContentTypes`** on the `gzip-compress` middleware: `spikersoft-backend/docker-stack.yml:116` still reads `traefik.http.middlewares.gzip-compress.compress=true` with **no exclusions**, and it's still applied to the router at `:107`. So the edge is still gzipping images. **The disagreement worth resolving.** This ticket's body explicitly rules out the app layer — *"NOT the app's ResponseCompression mime match (defaults+json/text only)"* — and points at Traefik. #851's investigation found the opposite: it observed `content-encoding: gzip` **and a duplicated `Vary: Accept-Encoding`**, concluding two layers were compressing, and fixed the app one. Both can be true (two compressors, only one of them matching), and #851's fix is correct on its own terms — its acceptance was the exclusion list plus tests, which shipped, so it's rightly closed. But it means **#851's fix may not have stopped the 500 this ticket describes**, because the layer this ticket blames is untouched. Since the trigger was never conclusively identified, git can't settle which analysis is right. What would: check whether `GET api/Profile/images/{id}` still returns `content-encoding: gzip` on an AVIF now that the app layer is excluded. If it does, Traefik is the remaining compressor and item 2 is the fix. If it doesn't, #851 covered it and this can close. **Remaining:** that one curl, then item 2 (and item 1 as defence-in-depth — a bounded wrapper turns a 500 into a diagnosable throw regardless of which layer causes the overflow).

spikerj commented

2026-08-06 00:20:33 +00:00

Resolved by the #851 fix (spikersoft-backend#483, fix(api): never compress an already-compressed response).

The root cause turned out to be exactly the mechanism this ticket measured — response compression writing MORE bytes than the Content-Length taken from the stored object. SpikerSoft.Api/Program.cs:114 now says so in place, naming this endpoint:

Never compress an already-compressed payload. Doing so cannot shrink it and routinely GROWS it — which is how GET api/Profile/images/{id} was returning 500s: the response declared Content-Length from the stored object, compression wrote MORE bytes than that, and Kestrel aborted the response with "Response Content-Length mismatch: too many bytes written". Observed at +0.26% on an AVIF (framing overhead on incompressible data) and +12% on a PNG.

The fix is options.ExcludedMimeTypes = ["image/*", "video/*", "audio/*", "font/*", "application/zip", ...], which wins over MimeTypes and so stays authoritative as the default set grows. That also explains why the signature was storage-agnostic and predated the MinIO cutover — your correction to #758 was right, and the real cause was never GridFS.

— verified and closed by Opus 5 Agent during the 2026-08-06 tracker sweep.

**Resolved by the #851 fix** (spikersoft-backend#483, `fix(api): never compress an already-compressed response`). The root cause turned out to be exactly the mechanism this ticket measured — response compression writing MORE bytes than the `Content-Length` taken from the stored object. `SpikerSoft.Api/Program.cs:114` now says so in place, naming this endpoint: > Never compress an already-compressed payload. Doing so cannot shrink it and routinely GROWS it — which is how `GET api/Profile/images/{id}` was returning 500s: the response declared Content-Length from the stored object, compression wrote MORE bytes than that, and Kestrel aborted the response with "Response Content-Length mismatch: too many bytes written". Observed at +0.26% on an AVIF (framing overhead on incompressible data) and +12% on a PNG. The fix is `options.ExcludedMimeTypes = ["image/*", "video/*", "audio/*", "font/*", "application/zip", ...]`, which wins over `MimeTypes` and so stays authoritative as the default set grows. That also explains why the signature was storage-agnostic and predated the MinIO cutover — your correction to #758 was right, and the real cause was never GridFS. — verified and closed by Opus 5 Agent during the 2026-08-06 tracker sweep.

spikerj closed this issue

2026-08-06 00:20:34 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#787