spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Infra][MinIO] Migrate lesson-videos + LessonVideoProcessor to S3 (lesson-videos bucket, epic #413) #523

New Issue
Closed
opened 2026-07-13 00:40:39 +00:00 by spikerj · 3 comments
spikerj commented 2026-07-13 00:40:39 +00:00
Owner
Copy Link
Copy Source

Parent epic: #413. Per-service MinIO migration (recipe: docs/minio-storage-migration.md).

Scope: lesson-videos bucket — /mnt/fusionio/spikersoft/lesson-videos (+ uploads/lesson-videos staging prefix in the uploads batch).

Current state: SpikerSoft.EventHandlers.LessonVideoProcessor is filesystem-only. Key files: Services/FfmpegVideoTranscoder.cs (ffmpeg reads/writes local paths — transcode via local temp files: download from S3 to temp, transcode, upload result), LessonVideoMoveConsumer.cs, LessonVideoScanConsumer.cs, LessonVideoTranscodeConsumer.cs. Also SpikerSoft.Business/Services/LessonVideoStagingService.cs (staging writer) and API static serving of /app/lesson-videos at /lesson-videos (covered by the API media-serving ticket).

Steps: bucket + scoped key → backend PR (S3 behind Storage:UseS3; ffmpeg works on temp copies) → mc mirror backfill → dual-run → cut over stack file, relax placement.

Parent epic: #413. Per-service MinIO migration (recipe: `docs/minio-storage-migration.md`). **Scope:** `lesson-videos` bucket — `/mnt/fusionio/spikersoft/lesson-videos` (+ `uploads/lesson-videos` staging prefix in the uploads batch). **Current state:** `SpikerSoft.EventHandlers.LessonVideoProcessor` is **filesystem-only**. Key files: `Services/FfmpegVideoTranscoder.cs` (ffmpeg reads/writes local paths — transcode via local temp files: download from S3 to temp, transcode, upload result), `LessonVideoMoveConsumer.cs`, `LessonVideoScanConsumer.cs`, `LessonVideoTranscodeConsumer.cs`. Also `SpikerSoft.Business/Services/LessonVideoStagingService.cs` (staging writer) and API static serving of `/app/lesson-videos` at `/lesson-videos` (covered by the API media-serving ticket). **Steps:** bucket + scoped key → backend PR (S3 behind `Storage:UseS3`; ffmpeg works on temp copies) → `mc mirror` backfill → dual-run → cut over stack file, relax placement.
spikerj commented 2026-07-13 20:13:09 +00:00
Author
Owner
Copy Link
Copy Source

Backend PR #262: LessonVideoMoveConsumer dual-writes moved video+thumbnail to the lesson-videos bucket (same relative keys served today → #528/#531 cutover = prefix swap). Transcoder deliberately unchanged (local paths) — S3-temp transcode is cutover-phase work with the #529 staging batch. This was the LAST per-tree migration under #413: every bucket in the map now has a dual-writing producer merged or in review.

Backend PR #262: LessonVideoMoveConsumer dual-writes moved video+thumbnail to the lesson-videos bucket (same relative keys served today → #528/#531 cutover = prefix swap). Transcoder deliberately unchanged (local paths) — S3-temp transcode is cutover-phase work with the #529 staging batch. This was the LAST per-tree migration under #413: every bucket in the map now has a dual-writing producer merged or in review.
spikerj commented 2026-07-13 20:32:40 +00:00
Author
Owner
Copy Link
Copy Source

Code MERGED (backend #262). Remaining = ops via #529 (lesson-videos bucket + videos-svc key + VIDEOS_S3_SECRET_KEY — backend #264/infra #60 — backfill, phase-2 bind drop). S3-temp transcode deferred to cutover per ticket.

Code MERGED (backend #262). Remaining = ops via #529 (lesson-videos bucket + videos-svc key + VIDEOS_S3_SECRET_KEY — backend #264/infra #60 — backfill, phase-2 bind drop). S3-temp transcode deferred to cutover per ticket.
spikerj commented 2026-07-18 17:59:15 +00:00
Author
Owner
Copy Link
Copy Source

Epic #413 accuracy pass (2026-07-18) — closing: migration scope complete; residuals are #529 lines.

Verified against current masters:

  • Code merged: backend #262 (LessonVideoMoveConsumer dual-writes video + thumbnail to lesson-videos, keys match served URLs) — on backend master
  • Stack ready: spikersoft-lesson-video-processor/docker-stack.yml carries Storage__UseS3=true + videos-svc
  • The deploy blocker is gone: #581 (VIDEOS_S3_SECRET_KEY never created) was fixed 2026-07-15 via backend #303 — the workflow now fetches the key from OpenBao (secret/ci/backend/minio/videos), so the deploy guard passes
  • lesson-videos bucket exists (created in the 2026-07-14 phase 0); it is empty because there are no lesson videos anywhere, disk included — correct, not a gap
  • The related serving bug #599 (root-relative URLs hitting nginx) is also closed

Residuals, both already #529's:

  1. one-time confirmation the processor actually deployed post-#581 and writes the bucket when the first real video flows (phase 1 soak line)
  2. S3-temp transcode (deliberately deferred to cutover per this ticket's own scoping) — phase 2

Residual tracked in: #529.

**Epic #413 accuracy pass (2026-07-18) — closing: migration scope complete; residuals are #529 lines.** Verified against current masters: - Code merged: backend #262 (`LessonVideoMoveConsumer` dual-writes video + thumbnail to `lesson-videos`, keys match served URLs) — on backend master - Stack ready: `spikersoft-lesson-video-processor/docker-stack.yml` carries `Storage__UseS3=true` + `videos-svc` - The deploy blocker is gone: #581 (VIDEOS_S3_SECRET_KEY never created) was fixed 2026-07-15 via backend #303 — the workflow now fetches the key from OpenBao (`secret/ci/backend/minio/videos`), so the deploy guard passes - `lesson-videos` bucket exists (created in the 2026-07-14 phase 0); it is empty **because there are no lesson videos anywhere, disk included** — correct, not a gap - The related serving bug #599 (root-relative URLs hitting nginx) is also closed Residuals, both already #529's: 1. one-time confirmation the processor actually deployed post-#581 and writes the bucket when the first real video flows (phase 1 soak line) 2. S3-temp transcode (deliberately deferred to cutover per this ticket's own scoping) — phase 2 Residual tracked in: #529.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#523