[Safety][Compliance] Add CSAM hash-detection + NCMEC reporting on user-uploaded images (the one residual beyond existing NSFW/malware/moderation controls) #402

New Issue

2026-07-05T18:51:12Z

spikerj commented

2026-07-05 18:51:12 +00:00

Surfaced by the architecture review and narrowed by direct source verification — most of the review's "child-safety gaps" were refuted; this is the single genuine residual.

Safety controls that ALREADY exist (verified in code — for context, so this ticket stays scoped)

Prompt moderation on generation: IPromptModerationService.ModerateAsync(prompt) at SubmitArtAssetCommandHandler.cs:137 + RestartFromStageCommandHandler.cs:73, fail-closed (PromptModerationResult.Rejected).
Output image NSFW gate (#349): ArtifactSafetyGate — every AI-generated image is NSFW-checked before storage, enabled + fail-closed by default (ArtPipeConfig.cs:103).
Upload malware scan: SecurityScanner + ClamAV (SecurityScanService.cs), size caps, quarantine.
Gallery moderation (#352): flagged-asset queue, staff review, quotas/usage meters.
GDPR erasure + portability: DeleteAccountCommand (cascade to child accounts + Keycloak user + student git repo) and ExportAccountDataQuery (data-export ZIP), ServiceCollectionExtensions.cs:614.

The gap

A student can upload a source image to the art pipeline (SubmitArtAssetCommand.SourceImageGridFsId). That upload is malware-scanned (ClamAV) and quarantined, and generated outputs are NSFW-gated — but there is no known-CSAM hash-matching on the uploaded image itself (searched: photodna|csam|thorn|hash-match → none). NSFW-classification ≠ CSAM detection: NSFW models score adult content probabilistically; CSAM detection is perceptual-hash matching against NCMEC/industry hash sets and is a distinct control.

Why it matters

A US platform that hosts imagery uploaded by/around minors has statutory obligations under 18 U.S.C. §2258A: on actual knowledge of apparent CSAM, report to NCMEC (CyberTipline) and preserve. Relying on an NSFW classifier neither reliably detects known CSAM nor satisfies the reporting duty.

Recommendation

Integrate a CSAM-detection service on the upload path — Microsoft PhotoDNA or Thorn Safer (perceptual-hash match against NCMEC hash sets), running in the UploadCoordinator/SecurityScanner flow before the image is usable, fail-closed like #349.
Add an NCMEC CyberTipline reporting pipeline + evidence-preservation + staff escalation for positive matches (this is an event-handler-shaped workstream: match → SecurityMonitor/incident → report + quarantine-hold).
Legal review of the retention/preservation obligations for flagged content.

Type: backend/security/compliance. Priority: high (legal + child-safety), but scoped — this is an additive control on top of an already-strong safety stack, not a rebuild. Relates to #349, #352, [#401].

Surfaced by the architecture review and **narrowed by direct source verification** — most of the review's "child-safety gaps" were refuted; this is the single genuine residual. ## Safety controls that ALREADY exist (verified in code — for context, so this ticket stays scoped) - **Prompt moderation** on generation: `IPromptModerationService.ModerateAsync(prompt)` at `SubmitArtAssetCommandHandler.cs:137` + `RestartFromStageCommandHandler.cs:73`, fail-closed (`PromptModerationResult.Rejected`). - **Output image NSFW gate (#349)**: `ArtifactSafetyGate` — every AI-generated image is NSFW-checked before storage, **enabled + fail-closed by default** (`ArtPipeConfig.cs:103`). - **Upload malware scan**: `SecurityScanner` + **ClamAV** (`SecurityScanService.cs`), size caps, quarantine. - **Gallery moderation** (#352): flagged-asset queue, staff review, quotas/usage meters. - **GDPR erasure + portability**: `DeleteAccountCommand` (cascade to child accounts + Keycloak user + student git repo) and `ExportAccountDataQuery` (data-export ZIP), `ServiceCollectionExtensions.cs:614`. ## The gap A student can upload a **source image** to the art pipeline (`SubmitArtAssetCommand.SourceImageGridFsId`). That upload is malware-scanned (ClamAV) and quarantined, and *generated* outputs are NSFW-gated — but there is **no known-CSAM hash-matching** on the uploaded image itself (searched: `photodna|csam|thorn|hash-match` → none). NSFW-classification ≠ CSAM detection: NSFW models score adult content probabilistically; CSAM detection is perceptual-hash matching against NCMEC/industry hash sets and is a distinct control. ## Why it matters A US platform that hosts imagery uploaded by/around minors has statutory obligations under **18 U.S.C. §2258A**: on *actual knowledge* of apparent CSAM, report to **NCMEC** (CyberTipline) and preserve. Relying on an NSFW classifier neither reliably detects known CSAM nor satisfies the reporting duty. ## Recommendation 1. Integrate a CSAM-detection service on the upload path — **Microsoft PhotoDNA** or **Thorn Safer** (perceptual-hash match against NCMEC hash sets), running in the `UploadCoordinator`/`SecurityScanner` flow before the image is usable, fail-closed like #349. 2. Add an **NCMEC CyberTipline reporting** pipeline + evidence-preservation + staff escalation for positive matches (this is an event-handler-shaped workstream: match → `SecurityMonitor`/incident → report + quarantine-hold). 3. Legal review of the retention/preservation obligations for flagged content. **Type:** backend/security/compliance. **Priority:** high (legal + child-safety), but **scoped** — this is an additive control on top of an already-strong safety stack, not a rebuild. Relates to #349, #352, [#401].

spikerj referenced this issue

2026-07-05 20:24:37 +00:00

[Safety][Compliance][Go-live] Make fail-open safety gates fail closed (SMS/TCPA, ClamAV, passport PII) #408

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#402