spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Prod][GPU] quiz-generation is green, consuming, and CANNOT generate a quiz — deployed with an empty Storage__SecretKey and no model bind (follow-on to #553 / infra #81) #592

New Issue
Closed
opened 2026-07-14 20:48:10 +00:00 by spikerj · 1 comment
spikerj commented 2026-07-14 20:48:10 +00:00
Owner
Copy Link
Copy Source

Filed by: QA Team

Summary

spikersoft-quiz-generation is deployed, reports 1/1 healthy, passes its health check, and is actively consuming generation.process — but it is structurally incapable of generating a single quiz. It has no model weights, no local fallback, and no working credential to fetch them with.

This is a follow-on to #553 (closed) / infra #81 (merged + deployed 2026-07-14 20:11 UTC). #81 correctly moved the worker off SERVER's 8 GB card onto the 4090 — but the deploy that carried it went out with an empty secret.

This is worse than a crash-loop, because a crash-loop is visible. This service looks perfectly green.

Evidence (live, 4090 node)

docker service inspect spikersoft-quiz-generation_quiz-generation:

Setting Value Verdict
placement constraint node.labels.artpipe-gpu == true ✅ #553 fixed — running on 4090
Storage__UseS3 true ✅
Storage__ServiceUrl https://minio.spikersoft.com ✅
Storage__AccessKey metadata-svc ✅
Storage__SecretKey empty string ❌ the bug
mounts volume:…quiz-generation-models -> /tmp/spikersoft-ai-models ⚠️ the old /mnt/fusionio/spikersoft/ai:/app/ai:ro bind is gone

The models volume is empty — created at the deploy timestamp, never written:

$ docker run --rm -v spikersoft-quiz-generation_quiz-generation-models:/m alpine ls -la /m
drwxr-xr-x 2 root root 4096 Jul 14 20:11 .
drwxr-xr-x 1 root root 4096 Jul 14 20:45 ..
   (4.0K total — nothing in it)

For contrast, the two sibling services that use the same metadata-svc identity both have a populated 40-character secret:

spikersoft-file-movement       AccessKey=metadata-svc  SecretKey=<SET, 40 chars>
spikersoft-metadata-extractor  AccessKey=metadata-svc  SecretKey=<SET, 40 chars>
spikersoft-quiz-generation     AccessKey=metadata-svc  SecretKey=<EMPTY>          <--

So the credential is valid and already present in the swarm. Only quiz-generation was deployed without it.

Why it looks healthy

The model is loaded lazily, on first request — not at startup. Container logs from the 20:11 task show a clean, fully-successful boot:

[20:11:57 INF] ✅ RabbitMQ connection established. Exchange: quiz.generation, Queue: generation.process, DLQ enabled: True
[20:11:57 INF] Quiz Generation Processor started successfully, listening on queue: generation.process

Nothing has asked it for a quiz yet, so nothing has failed yet. The moment a real request arrives it will try to self-provision its GGUF from the ai-models bucket (#527), get rejected by MinIO for an empty secret, find no /app/ai bind to fall back on, fail, retry ×3, and land in generation.process.dlq — which already holds 5 real user quizzes (#589).

Root cause

docker stack deploy does not honor Compose's ${VAR:?err} syntax. It substitutes an unset variable with the empty string and exits 0. Infra #81 introduced ${METADATA_S3_SECRET_KEY} into the stack file and removed the /app/ai bind in the same change, so an unset var went from "harmless" to "fatal, silently".

The guard for exactly this — backend PR #281, which makes the deploy job exit 1 on an empty secret — merged after this deploy had already gone out, so it never got the chance to block it.

Fix

Immediate (unbreak prod). Copy the known-good secret from a sibling without ever echoing it:

docker service update \
  --env-add "Storage__SecretKey=$(docker service inspect \
      spikersoft-metadata-extractor_spikersoft-metadata-extractor \
      --format '{{range .Spec.TaskTemplate.ContainerSpec.Env}}{{println .}}{{end}}' \
    | grep '^Storage__SecretKey=' | cut -d= -f2-)" \
  spikersoft-quiz-generation_quiz-generation

Durable. Set the METADATA_S3_SECRET_KEY Actions secret on spikerj/spikersoft-backend to metadata-svc's MinIO secret key, then re-run the Quiz Generation workflow. With #281 merged, the deploy job now refuses to proceed on an empty secret instead of shipping a broken service.

Verify — the volume should stop being empty:

docker run --rm -v spikersoft-quiz-generation_quiz-generation-models:/m alpine du -sh /m   # expect ~13.5 GB, not 4.0K

Related

  • #553 (closed) — the placement bug #81 fixed. Placement is genuinely correct now.
  • #582 (open) — tracked "#553's fix never reached production". The placement half is now resolved; this ticket is what replaced it.
  • #589 (open) — the DLQ that this will silently feed.
  • #527 — the self-provision-from-ai-models-bucket design that the empty secret defeats.
  • backend #281 (merged) — the guard that would have caught this.
**Filed by: QA Team** ## Summary `spikersoft-quiz-generation` is deployed, reports **1/1 healthy**, passes its health check, and is **actively consuming `generation.process`** — but it is **structurally incapable of generating a single quiz**. It has no model weights, no local fallback, and no working credential to fetch them with. This is a follow-on to **#553** (closed) / infra **#81** (merged + deployed 2026-07-14 20:11 UTC). #81 correctly moved the worker off SERVER's 8 GB card onto the 4090 — but the deploy that carried it went out with an empty secret. This is **worse than a crash-loop**, because a crash-loop is visible. This service looks perfectly green. ## Evidence (live, 4090 node) `docker service inspect spikersoft-quiz-generation_quiz-generation`: | Setting | Value | Verdict | |---|---|---| | placement constraint | `node.labels.artpipe-gpu == true` | ✅ #553 fixed — running on 4090 | | `Storage__UseS3` | `true` | ✅ | | `Storage__ServiceUrl` | `https://minio.spikersoft.com` | ✅ | | `Storage__AccessKey` | `metadata-svc` | ✅ | | **`Storage__SecretKey`** | **empty string** | ❌ **the bug** | | mounts | `volume:…quiz-generation-models -> /tmp/spikersoft-ai-models` | ⚠️ the old `/mnt/fusionio/spikersoft/ai:/app/ai:ro` bind is **gone** | The models volume is **empty** — created at the deploy timestamp, never written: ``` $ docker run --rm -v spikersoft-quiz-generation_quiz-generation-models:/m alpine ls -la /m drwxr-xr-x 2 root root 4096 Jul 14 20:11 . drwxr-xr-x 1 root root 4096 Jul 14 20:45 .. (4.0K total — nothing in it) ``` For contrast, the two sibling services that use the **same** `metadata-svc` identity both have a populated 40-character secret: ``` spikersoft-file-movement AccessKey=metadata-svc SecretKey=<SET, 40 chars> spikersoft-metadata-extractor AccessKey=metadata-svc SecretKey=<SET, 40 chars> spikersoft-quiz-generation AccessKey=metadata-svc SecretKey=<EMPTY> <-- ``` So the credential is valid and already present in the swarm. Only quiz-generation was deployed without it. ## Why it looks healthy The model is loaded **lazily, on first request** — not at startup. Container logs from the 20:11 task show a clean, fully-successful boot: ``` [20:11:57 INF] ✅ RabbitMQ connection established. Exchange: quiz.generation, Queue: generation.process, DLQ enabled: True [20:11:57 INF] Quiz Generation Processor started successfully, listening on queue: generation.process ``` Nothing has asked it for a quiz yet, so nothing has failed **yet**. The moment a real request arrives it will try to self-provision its GGUF from the `ai-models` bucket (#527), get rejected by MinIO for an empty secret, find no `/app/ai` bind to fall back on, fail, retry ×3, and **land in `generation.process.dlq`** — which already holds 5 real user quizzes (#589). ## Root cause `docker stack deploy` does **not** honor Compose's `${VAR:?err}` syntax. It substitutes an unset variable with the **empty string and exits 0**. Infra #81 introduced `${METADATA_S3_SECRET_KEY}` into the stack file *and* removed the `/app/ai` bind in the same change, so an unset var went from "harmless" to "fatal, silently". The guard for exactly this — backend PR **#281**, which makes the deploy job `exit 1` on an empty secret — **merged after this deploy had already gone out**, so it never got the chance to block it. ## Fix **Immediate (unbreak prod).** Copy the known-good secret from a sibling without ever echoing it: ```bash docker service update \ --env-add "Storage__SecretKey=$(docker service inspect \ spikersoft-metadata-extractor_spikersoft-metadata-extractor \ --format '{{range .Spec.TaskTemplate.ContainerSpec.Env}}{{println .}}{{end}}' \ | grep '^Storage__SecretKey=' | cut -d= -f2-)" \ spikersoft-quiz-generation_quiz-generation ``` **Durable.** Set the `METADATA_S3_SECRET_KEY` Actions secret on `spikerj/spikersoft-backend` to metadata-svc's MinIO secret key, then re-run the **Quiz Generation** workflow. With #281 merged, the deploy job now refuses to proceed on an empty secret instead of shipping a broken service. **Verify** — the volume should stop being empty: ```bash docker run --rm -v spikersoft-quiz-generation_quiz-generation-models:/m alpine du -sh /m # expect ~13.5 GB, not 4.0K ``` ## Related - **#553** (closed) — the placement bug #81 fixed. Placement is genuinely correct now. - **#582** (open) — tracked "#553's fix never reached production". The placement half is now resolved; this ticket is what replaced it. - **#589** (open) — the DLQ that this will silently feed. - **#527** — the self-provision-from-`ai-models`-bucket design that the empty secret defeats. - backend **#281** (merged) — the guard that would have caught this.
spikerj commented 2026-07-17 02:07:00 +00:00
Author
Owner
Copy Link
Copy Source

Verified resolved on the live swarm (2026-07-17):

  • spikersoft-quiz-generation now runs with Storage__UseS3=true and a non-empty Storage__SecretKey (48 chars, Bao-sourced via the deploy workflow) — the empty-credential condition this ticket describes is gone.
  • Zero errors in the last 12h of service logs.
  • Positive proof of generation: book-generated-quizzes holds 17 quizzes, the most recent created 2026-07-16T15:03Z — after this ticket was filed (07-14 20:48Z) — so the worker demonstrably fetches its model from the ai-models bucket and completes generations end-to-end.

The structural incapability (no weights, no credential, no fallback) is resolved by the MinIO model path (#527: backend #261 ObjectFileCache/ModelPathResolver + infra stack flips) with the working key injected at deploy time. Closing.

Verified resolved on the live swarm (2026-07-17): - `spikersoft-quiz-generation` now runs with `Storage__UseS3=true` and a **non-empty** `Storage__SecretKey` (48 chars, Bao-sourced via the deploy workflow) — the empty-credential condition this ticket describes is gone. - Zero errors in the last 12h of service logs. - **Positive proof of generation**: `book-generated-quizzes` holds 17 quizzes, the most recent created 2026-07-16T15:03Z — after this ticket was filed (07-14 20:48Z) — so the worker demonstrably fetches its model from the `ai-models` bucket and completes generations end-to-end. The structural incapability (no weights, no credential, no fallback) is resolved by the MinIO model path (#527: backend #261 ObjectFileCache/ModelPathResolver + infra stack flips) with the working key injected at deploy time. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#592