spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[ArtStudio P6] Safety, quotas & fairness — prompt moderation, NSFW gating policy, per-student GPU-minute quotas, queue fairness #352

New Issue
Closed
opened 2026-07-04 05:11:46 +00:00 by spikerj · 1 comment
spikerj commented 2026-07-04 05:11:46 +00:00
Owner
Copy Link
Copy Source

Sub-ticket of epic #346. Depends on #347; hardens #348–#350. Gate for opening the studio to all students — until this lands, access stays staff + pilot cohort.

What: the guardrails that make a GPU-backed generative feature safe and fair on a minors platform with one 24 GB card.

Scope:

  • Prompt-side moderation in SubmitArtAssetCommand / motion-prompt validation (FluentValidation via the existing ValidationBehavior + a moderation service): blocklist + classifier before any job is queued; rejected prompts get a kid-appropriate message and are logged for staff (not silently dropped)
  • Output-side NSFW gating (introduced per-stage in #349) promoted to a shared worker service with one policy: flag → stage fails → asset flagged → staff review queue; add staff override to release false positives
  • Per-student quotas: GPU-minutes and asset-count per day/week (config), enforced at submit time with a clear "come back tomorrow / ask your teacher" UX; staff exempt; usage surfaced on the asset library ("you've used 12 of 30 minutes this week")
  • Queue fairness: one active pipeline per student (additional submissions park as drafts), FIFO with staff priority lane; queue-position events already surfaced in P5's UI
  • Rate limiting on submit/upload endpoints via the existing RedisRateLimitMiddleware conventions (return 429 — see #341)
  • Audit trail: who generated what, prompts, moderation verdicts — queryable by staff (Mongo, staff-only query + controller action)
  • Abuse drills documented as integration tests: blocked prompt, NSFW output, quota exhaustion, queue flooding

Note: decide the moderation classifier here (local model on the GPU host vs a small CPU model vs external API) — a minors platform probably wants belt-and-suspenders: blocklist + classifier. Keycloak realm roles (student/staff/admin, mapped in Authentication.cs::MapRealmRolesToClaims) drive exemptions.

Acceptance: the abuse drills pass; a student cannot exceed quota, jump the queue, see a flagged image, or get raw model errors; staff can review flagged assets and release false positives; every generation is attributable in the audit log.

Sub-ticket of epic #346. Depends on #347; hardens #348–#350. **Gate for opening the studio to all students** — until this lands, access stays staff + pilot cohort. **What:** the guardrails that make a GPU-backed generative feature safe and fair on a minors platform with one 24 GB card. **Scope:** - [ ] **Prompt-side moderation** in `SubmitArtAssetCommand` / motion-prompt validation (FluentValidation via the existing `ValidationBehavior` + a moderation service): blocklist + classifier before any job is queued; rejected prompts get a kid-appropriate message and are logged for staff (not silently dropped) - [ ] Output-side NSFW gating (introduced per-stage in #349) promoted to a shared worker service with one policy: flag → stage fails → asset flagged → staff review queue; add staff override to release false positives - [ ] **Per-student quotas**: GPU-minutes and asset-count per day/week (config), enforced at submit time with a clear "come back tomorrow / ask your teacher" UX; staff exempt; usage surfaced on the asset library ("you've used 12 of 30 minutes this week") - [ ] **Queue fairness**: one active pipeline per student (additional submissions park as drafts), FIFO with staff priority lane; queue-position events already surfaced in P5's UI - [ ] Rate limiting on submit/upload endpoints via the existing `RedisRateLimitMiddleware` conventions (return 429 — see #341) - [ ] Audit trail: who generated what, prompts, moderation verdicts — queryable by staff (Mongo, staff-only query + controller action) - [ ] Abuse drills documented as integration tests: blocked prompt, NSFW output, quota exhaustion, queue flooding **Note:** decide the moderation classifier here (local model on the GPU host vs a small CPU model vs external API) — a minors platform probably wants belt-and-suspenders: blocklist + classifier. Keycloak realm roles (`student`/`staff`/`admin`, mapped in `Authentication.cs::MapRealmRolesToClaims`) drive exemptions. **Acceptance:** the abuse drills pass; a student cannot exceed quota, jump the queue, see a flagged image, or get raw model errors; staff can review flagged assets and release false positives; every generation is attributable in the audit log.
spikerj added the enhancement label 2026-07-04 05:11:46 +00:00
spikerj commented 2026-07-04 21:14:44 +00:00
Author
Owner
Copy Link
Copy Source

Both halves merged to master: backend PR #70 (blocklist prompt moderation with kid-safe rejections + audit, per-student quotas — one active pipeline / 10 per day / 30 GPU-min per week — usage endpoint, staff review queue with dismiss/release wired to restart-from-stage, full audit trail, rate limits) and angular PR #118 (usage meter in the studio shell, staff review queue at /admin/art-review with blurred-by-default quarantine previews, inline kid-safe composer errors). Output-side NSFW gating landed earlier via #67/#69 (#349/#362). Staff/admin exempt throughout.

The gate for opening the studio beyond staff/pilot is now in place. Remaining nuances tracked elsewhere: ML prompt classifier (interface ready, documented), queue-position events (#364 — needs GpuCoordinator emit), owner display-name lookup in the review queue (small follow-up if wanted). Closing.

Both halves merged to `master`: backend PR #70 (blocklist prompt moderation with kid-safe rejections + audit, per-student quotas — one active pipeline / 10 per day / 30 GPU-min per week — usage endpoint, staff review queue with dismiss/release wired to restart-from-stage, full audit trail, rate limits) and angular PR #118 (usage meter in the studio shell, staff review queue at `/admin/art-review` with blurred-by-default quarantine previews, inline kid-safe composer errors). Output-side NSFW gating landed earlier via #67/#69 (#349/#362). Staff/admin exempt throughout. The gate for opening the studio beyond staff/pilot is now in place. Remaining nuances tracked elsewhere: ML prompt classifier (interface ready, documented), queue-position events (#364 — needs GpuCoordinator emit), owner display-name lookup in the review queue (small follow-up if wanted). Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#352