spikerj/spikersoft-issues

0 Fork 0

[Safety][Compliance] Add CSAM hash-detection + NCMEC reporting on user-uploaded images (the one residual beyond existing NSFW/malware/moderation controls) #402

New Issue

Closed

opened 2026-07-05 18:51:12 +00:00 by spikerj · 1 comment

spikerj commented

2026-07-05 18:51:12 +00:00

Owner

Surfaced by the architecture review and narrowed by direct source verification — most of the review's "child-safety gaps" were refuted; this is the single genuine residual.

Safety controls that ALREADY exist (verified in code — for context, so this ticket stays scoped)

Prompt moderation on generation: IPromptModerationService.ModerateAsync(prompt) at SubmitArtAssetCommandHandler.cs:137 + RestartFromStageCommandHandler.cs:73, fail-closed (PromptModerationResult.Rejected).
Output image NSFW gate (#349): ArtifactSafetyGate — every AI-generated image is NSFW-checked before storage, enabled + fail-closed by default (ArtPipeConfig.cs:103).
Upload malware scan: SecurityScanner + ClamAV (SecurityScanService.cs), size caps, quarantine.
Gallery moderation (#352): flagged-asset queue, staff review, quotas/usage meters.
GDPR erasure + portability: DeleteAccountCommand (cascade to child accounts + Keycloak user + student git repo) and ExportAccountDataQuery (data-export ZIP), ServiceCollectionExtensions.cs:614.

The gap

A student can upload a source image to the art pipeline (SubmitArtAssetCommand.SourceImageGridFsId). That upload is malware-scanned (ClamAV) and quarantined, and generated outputs are NSFW-gated — but there is no known-CSAM hash-matching on the uploaded image itself (searched: photodna|csam|thorn|hash-match → none). NSFW-classification ≠ CSAM detection: NSFW models score adult content probabilistically; CSAM detection is perceptual-hash matching against NCMEC/industry hash sets and is a distinct control.

Why it matters

A US platform that hosts imagery uploaded by/around minors has statutory obligations under 18 U.S.C. §2258A: on actual knowledge of apparent CSAM, report to NCMEC (CyberTipline) and preserve. Relying on an NSFW classifier neither reliably detects known CSAM nor satisfies the reporting duty.

Recommendation

Integrate a CSAM-detection service on the upload path — Microsoft PhotoDNA or Thorn Safer (perceptual-hash match against NCMEC hash sets), running in the UploadCoordinator/SecurityScanner flow before the image is usable, fail-closed like #349.
Add an NCMEC CyberTipline reporting pipeline + evidence-preservation + staff escalation for positive matches (this is an event-handler-shaped workstream: match → SecurityMonitor/incident → report + quarantine-hold).
Legal review of the retention/preservation obligations for flagged content.

Type: backend/security/compliance. Priority: high (legal + child-safety), but scoped — this is an additive control on top of an already-strong safety stack, not a rebuild. Relates to #349, #352, [#401].

Surfaced by the architecture review and **narrowed by direct source verification** — most of the review's "child-safety gaps" were refuted; this is the single genuine residual. ## Safety controls that ALREADY exist (verified in code — for context, so this ticket stays scoped) - **Prompt moderation** on generation: `IPromptModerationService.ModerateAsync(prompt)` at `SubmitArtAssetCommandHandler.cs:137` + `RestartFromStageCommandHandler.cs:73`, fail-closed (`PromptModerationResult.Rejected`). - **Output image NSFW gate (#349)**: `ArtifactSafetyGate` — every AI-generated image is NSFW-checked before storage, **enabled + fail-closed by default** (`ArtPipeConfig.cs:103`). - **Upload malware scan**: `SecurityScanner` + **ClamAV** (`SecurityScanService.cs`), size caps, quarantine. - **Gallery moderation** (#352): flagged-asset queue, staff review, quotas/usage meters. - **GDPR erasure + portability**: `DeleteAccountCommand` (cascade to child accounts + Keycloak user + student git repo) and `ExportAccountDataQuery` (data-export ZIP), `ServiceCollectionExtensions.cs:614`. ## The gap A student can upload a **source image** to the art pipeline (`SubmitArtAssetCommand.SourceImageGridFsId`). That upload is malware-scanned (ClamAV) and quarantined, and *generated* outputs are NSFW-gated — but there is **no known-CSAM hash-matching** on the uploaded image itself (searched: `photodna|csam|thorn|hash-match` → none). NSFW-classification ≠ CSAM detection: NSFW models score adult content probabilistically; CSAM detection is perceptual-hash matching against NCMEC/industry hash sets and is a distinct control. ## Why it matters A US platform that hosts imagery uploaded by/around minors has statutory obligations under **18 U.S.C. §2258A**: on *actual knowledge* of apparent CSAM, report to **NCMEC** (CyberTipline) and preserve. Relying on an NSFW classifier neither reliably detects known CSAM nor satisfies the reporting duty. ## Recommendation 1. Integrate a CSAM-detection service on the upload path — **Microsoft PhotoDNA** or **Thorn Safer** (perceptual-hash match against NCMEC hash sets), running in the `UploadCoordinator`/`SecurityScanner` flow before the image is usable, fail-closed like #349. 2. Add an **NCMEC CyberTipline reporting** pipeline + evidence-preservation + staff escalation for positive matches (this is an event-handler-shaped workstream: match → `SecurityMonitor`/incident → report + quarantine-hold). 3. Legal review of the retention/preservation obligations for flagged content. **Type:** backend/security/compliance. **Priority:** high (legal + child-safety), but **scoped** — this is an additive control on top of an already-strong safety stack, not a rebuild. Relates to #349, #352, [#401].

spikerj referenced this issue

2026-07-05 20:24:37 +00:00

[Safety][Compliance][Go-live] Make fail-open safety gates fail closed (SMS/TCPA, ClamAV, passport PII) #408

spikerj commented

2026-08-07 13:50:43 +00:00

Author

Owner

Migrated to spikerj/spikersoft-backend#564 as part of the umbrella-tracker breakup.

Verified 2026-08-07 against spikersoft-backend@98102023: not started. git grep -rniE "photodna|csam|ncmec|hash-match|cybertip" returns zero matches — no perceptual-hash matching on the upload path, no CyberTipline reporting pipeline. The adjacent controls this ticket deliberately scopes around are all still in place (prompt moderation, ArtifactSafetyGate, ClamAV SecurityScanner, gallery moderation, GDPR erasure/export), and live api.spikersoft.com/healthz shows the art.model.safetycheck.tasks queue running, so the new control would slot alongside an existing safety stage.

Status: not started. Worth flagging that the long pole is procurement — PhotoDNA/Thorn both need an account and a legal agreement before any handler can be written.

Closing here. Work now lives in the repo that holds the fix, so fixes #564 in a PR will auto-close it on merge. The umbrella tracker keeps cross-repo epics only.

— Opus 5 Agent

Migrated to **spikerj/spikersoft-backend#564** as part of the umbrella-tracker breakup. Verified 2026-08-07 against `spikersoft-backend@98102023`: **not started.** `git grep -rniE "photodna|csam|ncmec|hash-match|cybertip"` returns **zero matches** — no perceptual-hash matching on the upload path, no CyberTipline reporting pipeline. The adjacent controls this ticket deliberately scopes around are all still in place (prompt moderation, `ArtifactSafetyGate`, ClamAV `SecurityScanner`, gallery moderation, GDPR erasure/export), and live `api.spikersoft.com/healthz` shows the `art.model.safetycheck.tasks` queue running, so the new control would slot alongside an existing safety stage. Status: not started. Worth flagging that the long pole is procurement — PhotoDNA/Thorn both need an account and a legal agreement before any handler can be written. Closing here. Work now lives in the repo that holds the fix, so `fixes #564` in a PR will auto-close it on merge. The umbrella tracker keeps cross-repo epics only. — Opus 5 Agent

spikerj closed this issue

2026-08-07 13:50:43 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#402