[Data integrity] One-off FormatException: Mongo response field 'Tags' arrived as 'Taws' (single bit flip) in GetTrailCoverageQuery on SERVER #1198

Open
opened 2026-10-10 04:17:21 +00:00 by spikerj · 0 comments
Owner

What happened

2026-10-10 03:02:09Z, one GET api/Trails/coverage request failed:

System.FormatException: Element 'Taws' does not match any field or property of class <>f__AnonymousType29`2[System.String,System.String]
  at MongoDB.Bson.Serialization.BsonClassMapSerializer`1.DeserializeClass …

GetTrailCoverageQueryHandler projects new { t.Region, t.Tags }. The failing element name Taws is Tags with one bit flipped (g 0x67 → w 0x77). Checks:

  • The trails collection (1,208,787 docs) has exactly 16 top-level keys in every document; no document contains a Taws key (full $objectToArray scan).
  • Seq has no other "does not match any field or property" error in 14 days; this element name appears once (one request, four log lines).
  • SERVER (runs both the API container and mongo-cluster_mongo-router) shows no MCE/EDAC/hardware-error lines in dmesg or kern.log, but the box has no ECC so a RAM fault would not be logged.

So one BSON response was corrupted in flight or in memory between mongos and the API's deserializer. This one was a read, so nothing was persisted wrong; the same fault on a write path would store a corrupt key silently.

Ask

  • Treat as a watch item: if a second element-mismatch with a nonsense key appears (Seq filter @Exception like '%does not match any field or property%'), schedule a memtest on SERVER and/or move mongo-router off it.
  • Optional: a nightly job that scans the hot collections for unknown top-level keys would catch a persisted flip early.

Found by the 2026-10-10 04:10Z swarm recon.

## What happened 2026-10-10 03:02:09Z, one `GET api/Trails/coverage` request failed: ``` System.FormatException: Element 'Taws' does not match any field or property of class <>f__AnonymousType29`2[System.String,System.String] at MongoDB.Bson.Serialization.BsonClassMapSerializer`1.DeserializeClass … ``` `GetTrailCoverageQueryHandler` projects `new { t.Region, t.Tags }`. The failing element name `Taws` is `Tags` with one bit flipped (`g` 0x67 → `w` 0x77). Checks: - The `trails` collection (1,208,787 docs) has exactly 16 top-level keys in every document; **no document contains a `Taws` key** (full `$objectToArray` scan). - Seq has **no other** "does not match any field or property" error in 14 days; this element name appears once (one request, four log lines). - SERVER (runs both the API container and `mongo-cluster_mongo-router`) shows no MCE/EDAC/hardware-error lines in dmesg or kern.log, but the box has no ECC so a RAM fault would not be logged. So one BSON response was corrupted in flight or in memory between mongos and the API's deserializer. This one was a read, so nothing was persisted wrong; the same fault on a write path would store a corrupt key silently. ## Ask - Treat as a watch item: if a second element-mismatch with a nonsense key appears (Seq filter `@Exception like '%does not match any field or property%'`), schedule a memtest on SERVER and/or move `mongo-router` off it. - Optional: a nightly job that scans the hot collections for unknown top-level keys would catch a persisted flip early. Found by the 2026-10-10 04:10Z swarm recon.
Sign in to join this conversation.