[Bug][CI][Deploy] lesson-video-processor has NEVER deployed — the VIDEOS_S3_SECRET_KEY Actions secret was never created, so its own deploy guard rejects every run #581

New Issue

2026-07-14T19:57:14Z

spikerj commented

2026-07-14 19:57:14 +00:00

QA Team — found while verifying CI on backend PR #280 (2026-07-14).

spikersoft-lesson-video-processor does not exist in the swarm and never has. The README audit flagged this as an open question ("not yet deployed, deployed under an unmatched name, or gated like Scheduler #540?"). This is the answer, and it is not a mystery — the pipeline has been rejecting its own deploy on every single run.

Root cause

.gitea/workflows/spikersoft-lesson-video-processor.yml guards the deploy on a non-empty secret (lines 112-115):

script: |
  if [ -z "$VIDEOS_S3_SECRET_KEY" ]; then
    echo "VIDEOS_S3_SECRET_KEY is empty — refusing to deploy (check the repo Actions secret)."
    exit 1
  fi
  cd /mnt/infrastructure/spikersoft-lesson-video-processor
  sudo VIDEOS_S3_SECRET_KEY="$VIDEOS_S3_SECRET_KEY" \
    docker stack deploy --with-registry-auth -c docker-stack.yml spikersoft-lesson-video-processor

The secret is empty. Straight from the runner log (job 24483, run #5655, 2026-07-14 19:45:07Z):

expression '${{ secrets.VIDEOS_S3_SECRET_KEY }}' rewritten to 'format('{0}', secrets.VIDEOS_S3_SECRET_KEY)'
evaluating expression 'format('{0}', secrets.VIDEOS_S3_SECRET_KEY)'
expression 'format('{0}', secrets.VIDEOS_S3_SECRET_KEY)' evaluated to '%!t(string=)'
...
  ❌  Failure - Main Run entrypoint.sh
exitcode '1': failure
  ❌  Failure - Main Deploy to production

%!t(string=) is Go formatting an empty string — the repo Actions secret VIDEOS_S3_SECRET_KEY does not exist. The guard does exactly what it was written to do, and has presumably been doing it since the guard was added.

What this means

The images are fine. Both build_and_publish legs (amd64 + arm64v8) succeed, and docker manifest create / docker manifest push succeed — the log shows Created manifest list git.spikersoft.com/spikerj/spikersoft-lesson-video-processor:latest. Only the deploy step fails.
So the service has simply never been created. docker service ls | grep lesson returns nothing.
create_manifest shows red on every PR that touches this project, which is exactly the kind of permanently-red leg that trains people to ignore CI. It went red on a docs-only PR.

This is the same failure shape as #540 (Scheduler was never wired into CI and nobody noticed it wasn't running). Same lesson: a service that never deploys produces no errors, because it produces nothing at all.

Fix

Create the VIDEOS_S3_SECRET_KEY repo Actions secret — the scoped MinIO key for the lesson-videos dual-write (per the workflow comment: #523, cutover batch #529).
Re-run the workflow and confirm spikersoft-lesson-video-processor actually appears in docker service ls.
Then settle the open question this blocks: who serves /lesson-videos in production today — nginx static hosting, the API's /lesson-videos static route, or both? LessonVideoMirror.cs claims both, and that cannot be verified while the worker has never run (#528/#531 cutover).

The create_manifest deploy-gate pattern is shared across the backend multi-arch workflows, so it is worth checking whether any other service is silently failing the same guard on a missing secret. scheduler (#540) and this one make two.
Found during backend PR #280 (README audit); the audit's open question about this service's deployment status can now be answered from this ticket.

**QA Team** — found while verifying CI on backend PR #280 (2026-07-14). `spikersoft-lesson-video-processor` **does not exist in the swarm** and never has. The README audit flagged this as an open question ("not yet deployed, deployed under an unmatched name, or gated like Scheduler #540?"). This is the answer, and it is not a mystery — the pipeline has been rejecting its own deploy on every single run. ## Root cause `.gitea/workflows/spikersoft-lesson-video-processor.yml` guards the deploy on a non-empty secret (lines 112-115): ```yaml script: | if [ -z "$VIDEOS_S3_SECRET_KEY" ]; then echo "VIDEOS_S3_SECRET_KEY is empty — refusing to deploy (check the repo Actions secret)." exit 1 fi cd /mnt/infrastructure/spikersoft-lesson-video-processor sudo VIDEOS_S3_SECRET_KEY="$VIDEOS_S3_SECRET_KEY" \ docker stack deploy --with-registry-auth -c docker-stack.yml spikersoft-lesson-video-processor ``` The secret is **empty**. Straight from the runner log (job 24483, run #5655, 2026-07-14 19:45:07Z): ``` expression '${{ secrets.VIDEOS_S3_SECRET_KEY }}' rewritten to 'format('{0}', secrets.VIDEOS_S3_SECRET_KEY)' evaluating expression 'format('{0}', secrets.VIDEOS_S3_SECRET_KEY)' expression 'format('{0}', secrets.VIDEOS_S3_SECRET_KEY)' evaluated to '%!t(string=)' ... ❌ Failure - Main Run entrypoint.sh exitcode '1': failure ❌ Failure - Main Deploy to production ``` `%!t(string=)` is Go formatting an **empty string** — the repo Actions secret `VIDEOS_S3_SECRET_KEY` does not exist. The guard does exactly what it was written to do, and has presumably been doing it since the guard was added. ## What this means - **The images are fine.** Both `build_and_publish` legs (amd64 + arm64v8) succeed, and `docker manifest create` / `docker manifest push` succeed — the log shows `Created manifest list git.spikersoft.com/spikerj/spikersoft-lesson-video-processor:latest`. Only the deploy step fails. - **So the service has simply never been created.** `docker service ls | grep lesson` returns nothing. - **`create_manifest` shows red on every PR that touches this project**, which is exactly the kind of permanently-red leg that trains people to ignore CI. It went red on a **docs-only** PR. This is the same failure shape as **#540** (Scheduler was never wired into CI and nobody noticed it wasn't running). Same lesson: a service that never deploys produces no errors, because it produces nothing at all. ## Fix 1. Create the `VIDEOS_S3_SECRET_KEY` repo Actions secret — the scoped MinIO key for the lesson-videos dual-write (per the workflow comment: #523, cutover batch #529). 2. Re-run the workflow and confirm `spikersoft-lesson-video-processor` actually appears in `docker service ls`. 3. Then settle the **open question this blocks**: who serves `/lesson-videos` in production today — nginx static hosting, the API's `/lesson-videos` static route, or both? `LessonVideoMirror.cs` claims both, and that cannot be verified while the worker has never run (#528/#531 cutover). ## Related - The `create_manifest` deploy-gate pattern is shared across the backend multi-arch workflows, so it is worth checking whether any **other** service is silently failing the same guard on a missing secret. `scheduler` (#540) and this one make two. - Found during backend PR #280 (README audit); the audit's open question about this service's deployment status can now be answered from this ticket.

spikerj referenced this issue

2026-07-14 20:05:57 +00:00

[Bug][CI] 28 of 30 path-filtered workflows still don't re-run when their own workflow file changes — #538 fixed only artpipe #583

spikerj referenced this issue

2026-07-14 20:11:45 +00:00

[Bug][CI][Deploy] 28 of 29 deploy jobs never 'git pull' /mnt/infrastructure — they deploy STALE stack files and report success #584

spikerj referenced this issue

2026-07-14 20:38:46 +00:00

[Bug][CI][Deploy] cancel-in-progress on master kills DEPLOY jobs — back-to-back merges cancel each other and fixes silently never ship #590

spikerj referenced this issue

2026-07-14 21:55:54 +00:00

[Backend][MinIO] Replace API static-file media serving with S3-backed delivery (epic #413) #528

spikerj referenced this issue

2026-07-14 21:57:07 +00:00

[Bug][Angular][Backend] Lesson-video and geography-media URLs are root-relative, so the browser asks nginx (learn.spikersoft.com) — which serves index.html with a 200 into the <video> element, never the media #599

spikerj referenced this issue

2026-07-14 22:39:00 +00:00

[Backend][MinIO] Replace API static-file media serving with S3-backed delivery (epic #413) #528

spikerj commented

2026-07-15 18:02:09 +00:00

Resolved in spikersoft-backend PR #303 (batch 3a, merged to master — commit 13d6ffb3). spikersoft-lesson-video-processor.yml now fetches VIDEOS_S3_SECRET_KEY from OpenBao (secret/ci/backend/minio/videos, a verified-populated path) via the bao-secrets action, instead of the VIDEOS_S3_SECRET_KEY Actions secret that was never created. The deploy guard now sees a real key, so the pipeline can deploy for the first time. Closing.

Resolved in spikersoft-backend PR #303 (batch 3a, merged to `master` — commit `13d6ffb3`). `spikersoft-lesson-video-processor.yml` now fetches `VIDEOS_S3_SECRET_KEY` from OpenBao (`secret/ci/backend/minio/videos`, a verified-populated path) via the `bao-secrets` action, instead of the `VIDEOS_S3_SECRET_KEY` Actions secret that was never created. The deploy guard now sees a real key, so the pipeline can deploy for the first time. Closing.

spikerj closed this issue

2026-07-15 18:02:09 +00:00

spikerj referenced this issue

2026-07-15 18:02:30 +00:00

[Security][CI] OpenBao Phase 1: Gitea Actions secrets → AppRole fetch; rotate the committed runner token (epic #543) #545

spikerj referenced this issue

2026-07-15 18:04:38 +00:00

[Bug][Infra][CI] SpikerSoft.EventHandlers.Scheduler is built nowhere and deployed nowhere — scheduled tasks (GeoIpUpdate incl.) never run #540

spikerj referenced this issue

2026-07-16 05:14:12 +00:00

[Bug][Angular][Backend] Lesson-video and geography-media URLs are root-relative, so the browser asks nginx (learn.spikersoft.com) — which serves index.html with a 200 into the <video> element, never the media #599

spikerj referenced this issue