[Bug][Prod][Data-loss] 10 messages are permanently dead in DLQs with ZERO consumers — 5 of them are real user quizzes killed by #553, now replayable #589

New Issue

2026-07-14T20:34:46Z

spikerj commented

2026-07-14 20:34:46 +00:00

QA Team — sweep 2026-07-14 ~21:45Z. Found via the RabbitMQ management API. Filed against closed #553.

The dead-letter queues are a graveyard, not a buffer. Nothing consumes them, nothing retries them, and the config promises an auto-retry that does not exist in code. Right now they hold real, permanently-lost user work.

What is actually dead

generation.process.dlq    5 msgs   0 consumers
keycloak.events.dlq       4 msgs   0 consumers
embeddings.process.dlq    1 msg    0 consumers

Every single one died the same way: exhausted *.retry.3 and then reason=expired. Not a poison-message rejection — they aged out of the retry ladder and fell into a queue nobody reads.

The 5 quizzes are real user content (peeked, requeued — nothing consumed)

QuizId 6a531899…  "Sample PDF"                                            user: spikerj
QuizId 6a531656…  "Untitled"                                              user: spikerj
QuizId 6a542e3f…  "Untitled"                                              user: spikerj
QuizId 6a53ce21…  "Untitled"                                              user: spikerj
QuizId 6a55ad64…  "Machine Learning Foundations: Volume I: Supervised…"   user: spikerj

These are #553 casualties. Every one is a quiz that failed while quiz-generation was pinned to SERVER's 8 GB card and could not load a 13.49 GB dual-model footprint. The books were uploaded, scanned, moved, described, embedded — the whole pipeline ran and paid for itself — and then the quiz died at the last stage and was silently binned.

From the user's point of view: they uploaded a book, and the quiz never appeared. No error, no retry, nothing.

The others

embeddings.process.dlq — one book (bookId 6a531656…, requested 2026-07-12) whose embeddings never generated. Same book as one of the dead quizzes, so that upload is doubly broken.
keycloak.events.dlq — 4 EventClientNotificationMqMsg events from the keycloak-to-rabbit SPI, expired out of keycloak.events.retry.3. These are audit-trail events that no longer exist anywhere. The keycloak-events collection is our audit log; it now has holes, and nothing will tell us which.

Why they will never come back

Every DLQ has consumers=0. Nothing is listening. Ever.
RabbitMQ:DeadLetterQueue declares AutoRetryTransientErrors and AutoRetryDelayHours — and the README audit already established that no code reads or implements them. The config promises a reprocessor that was never written. Anyone reading the config would reasonably assume these messages are being retried. They are not.
There is no alert on DLQ depth, so this sat unnoticed.

The good news, and it is time-sensitive

#553 is now fixed and deployed (quiz-generation moved to the 4090 at 20:12Z, verified — see #582). The exact condition that killed these 5 quizzes no longer exists.

They are replayable right now. Shovelling generation.process.dlq back onto generation.process should regenerate all five successfully — the pipeline work upstream is already done and stored, only the final quiz step needs to re-run. Same for the single embeddings.process.dlq message.

This is a rare case where the fix and the recovery are both available at the same time. If we leave it, the user's five books stay quiz-less forever and nobody ever finds out.

Suggested, in order:

Replay generation.process.dlq (5) and embeddings.process.dlq (1) now that #553 is deployed. Confirm the quizzes appear.
Decide what to do about the 4 lost Keycloak audit events (probably unrecoverable — worth knowing they are gone).
Then fix the systemic hole: either implement the auto-retry the config already advertises, or delete those keys so they stop lying — and add a DLQ-depth alert, because a silent graveyard is the worst possible failure mode for data loss.

Also spotted

art.asset.stage.concept.requests has 1 message and 0 consumers — a concept-generation request with nothing to serve it, because spikersoft-artpipe-modeling is down (#519). Same shape of problem: work accepted, queued, and abandoned.

Related: closed #553 (the cause of the 5 quiz deaths), #582 (the fix, now deployed), #519 (artpipe-modeling down).

**QA Team** — sweep 2026-07-14 ~21:45Z. Found via the RabbitMQ management API. Filed against **closed #553**. The dead-letter queues are a **graveyard, not a buffer**. Nothing consumes them, nothing retries them, and the config promises an auto-retry that does not exist in code. Right now they hold real, permanently-lost user work. ## What is actually dead ``` generation.process.dlq 5 msgs 0 consumers keycloak.events.dlq 4 msgs 0 consumers embeddings.process.dlq 1 msg 0 consumers ``` Every single one died the same way: exhausted `*.retry.3` and then **`reason=expired`**. Not a poison-message rejection — they aged out of the retry ladder and fell into a queue nobody reads. ### The 5 quizzes are real user content (peeked, requeued — nothing consumed) ``` QuizId 6a531899… "Sample PDF" user: spikerj QuizId 6a531656… "Untitled" user: spikerj QuizId 6a542e3f… "Untitled" user: spikerj QuizId 6a53ce21… "Untitled" user: spikerj QuizId 6a55ad64… "Machine Learning Foundations: Volume I: Supervised…" user: spikerj ``` These are **#553 casualties**. Every one is a quiz that failed while quiz-generation was pinned to SERVER's 8 GB card and could not load a 13.49 GB dual-model footprint. The books were uploaded, scanned, moved, described, embedded — the whole pipeline ran and paid for itself — and then the quiz died at the last stage and was silently binned. From the user's point of view: **they uploaded a book, and the quiz never appeared. No error, no retry, nothing.** ### The others - **`embeddings.process.dlq`** — one book (`bookId 6a531656…`, requested **2026-07-12**) whose embeddings never generated. Same book as one of the dead quizzes, so that upload is doubly broken. - **`keycloak.events.dlq`** — 4 `EventClientNotificationMqMsg` events from the keycloak-to-rabbit SPI, expired out of `keycloak.events.retry.3`. These are **audit-trail events that no longer exist anywhere**. The `keycloak-events` collection is our audit log; it now has holes, and nothing will tell us which. ## Why they will never come back 1. **Every DLQ has `consumers=0`.** Nothing is listening. Ever. 2. **`RabbitMQ:DeadLetterQueue` declares `AutoRetryTransientErrors` and `AutoRetryDelayHours`** — and the README audit already established that **no code reads or implements them**. The config promises a reprocessor that was never written. Anyone reading the config would reasonably assume these messages are being retried. They are not. 3. There is no alert on DLQ depth, so this sat unnoticed. ## The good news, and it is time-sensitive **#553 is now fixed and deployed** (quiz-generation moved to the 4090 at 20:12Z, verified — see #582). The exact condition that killed these 5 quizzes no longer exists. **They are replayable right now.** Shovelling `generation.process.dlq` back onto `generation.process` should regenerate all five successfully — the pipeline work upstream is already done and stored, only the final quiz step needs to re-run. Same for the single `embeddings.process.dlq` message. This is a rare case where the fix and the recovery are both available at the same time. If we leave it, the user's five books stay quiz-less forever and nobody ever finds out. Suggested, in order: 1. Replay `generation.process.dlq` (5) and `embeddings.process.dlq` (1) now that #553 is deployed. Confirm the quizzes appear. 2. Decide what to do about the 4 lost Keycloak audit events (probably unrecoverable — worth knowing they are gone). 3. Then fix the systemic hole: either implement the auto-retry the config already advertises, or delete those keys so they stop lying — **and add a DLQ-depth alert**, because a silent graveyard is the worst possible failure mode for data loss. ## Also spotted `art.asset.stage.concept.requests` has **1 message and 0 consumers** — a concept-generation request with nothing to serve it, because `spikersoft-artpipe-modeling` is down (**#519**). Same shape of problem: work accepted, queued, and abandoned. Related: closed **#553** (the cause of the 5 quiz deaths), **#582** (the fix, now deployed), **#519** (artpipe-modeling down).

spikerj referenced this issue

2026-07-14 20:48:10 +00:00

[Bug][Prod][GPU] quiz-generation is green, consuming, and CANNOT generate a quiz — deployed with an empty Storage__SecretKey and no model bind (follow-on to #553 / infra #81) #592

spikerj referenced this issue

2026-07-14 21:15:33 +00:00

[Bug][Prod][ArtPipe] Concept-art is dead because ArtStudio:StageModelMap is EMPTY — API routes to the legacy in-process stage queue whose consumer can't boot, while the 5 resident model containers sit idle on queues nobody publishes to #595

spikerj referenced this issue

2026-07-27 15:18:36 +00:00

EPIC: Sponsor family card redesign — image-backed destination tiles, achievements strip, parent-gated age #868

spikerj referenced this issue

2026-07-29 15:31:49 +00:00

Family achievements strip on the card (ui-award-chips) + skills.service route fixes #872

spikerj commented

2026-07-29 15:59:56 +00:00

Audited against origin/master — NOT DONE. Neither the replay nor the systemic fix exists. Flagging the time-sensitive part first.

The systemic hole is still open, and master documents it as open. SpikerSoft.Common/README.md:20 — "There is no DLQ auto-reprocessor… stays there until someone drains it by hand"; README.md:66 — "AutoRetryTransientErrors / AutoRetryDelayHours are configured everywhere and read nowhere." Those inert keys are still copy-pasted across ~15 appsettings.json files (e.g. SpikerSoft.Api/appsettings.json:369-370). That's the same shape as several other findings this week: config that looks like a feature and is wired to nothing.

One acceptance item is partially present: DLQ-depth alerting exists as a health check — SpikerSoft.Api/Infrastructure/HealthChecks/RabbitMQ_HealthCheck.cs:205,247 compares dlqDepth against dlqConfig.AlertThreshold (DlqConfiguration.cs:46). Whether that check actually pages anyone isn't determinable from git, and given #756 (ops alerting silently dead — every alert email fails SMTP AUTH 535) it probably doesn't.

Time-sensitive, and worth acting on before anything else here: the 5 generation.process.dlq + 1 embeddings.process.dlq messages this ticket describes as "real user quizzes, now replayable" have no record of ever being replayed, and no record they still exist. RabbitMQ DLQs are not archival — and note the 2026-07-29 power outage (#895) destroyed the broker's mnesia DB, taking all queued-but-unprocessed messages with it. These messages were most likely lost in that outage. Someone should check whether they're still there before planning a replay; if they're gone, the user-facing decision (do those 5 people get their quizzes regenerated?) is the actual remaining work.

Remaining:

Determine whether the 10 messages survived #895; if not, decide how to make those users whole.
Decide on the 4 lost Keycloak audit events.
Implement the DLQ reprocessor, or delete the config keys that claim it exists — leaving them is worse than either.

Audited against `origin/master` — **NOT DONE.** Neither the replay nor the systemic fix exists. Flagging the time-sensitive part first. **No replay tooling of any kind.** `git grep -iE 'dlq.?replay|replay.?dlq|DeadLetterReplay|drain.?dlq|Shovel'` across all three repos returns nothing; searching `*.sh|*.py|*.mjs|*.js` for shovel/DLQ patterns hits only unrelated Python handler files. **The systemic hole is still open, and master documents it as open.** `SpikerSoft.Common/README.md:20` — *"There is no DLQ auto-reprocessor… stays there until someone drains it by hand"*; `README.md:66` — *"`AutoRetryTransientErrors` / `AutoRetryDelayHours` are configured everywhere and read nowhere."* Those inert keys are still copy-pasted across ~15 `appsettings.json` files (e.g. `SpikerSoft.Api/appsettings.json:369-370`). That's the same shape as several other findings this week: config that looks like a feature and is wired to nothing. **One acceptance item is partially present:** DLQ-depth alerting exists as a health check — `SpikerSoft.Api/Infrastructure/HealthChecks/RabbitMQ_HealthCheck.cs:205,247` compares `dlqDepth` against `dlqConfig.AlertThreshold` (`DlqConfiguration.cs:46`). Whether that check actually pages anyone isn't determinable from git, and given **#756** (ops alerting silently dead — every alert email fails SMTP AUTH 535) it probably doesn't. **Time-sensitive, and worth acting on before anything else here:** the 5 `generation.process.dlq` + 1 `embeddings.process.dlq` messages this ticket describes as "real user quizzes, now replayable" have **no record of ever being replayed, and no record they still exist**. RabbitMQ DLQs are not archival — and note the 2026-07-29 power outage (#895) destroyed the broker's mnesia DB, taking all queued-but-unprocessed messages with it. **These messages were most likely lost in that outage.** Someone should check whether they're still there before planning a replay; if they're gone, the user-facing decision (do those 5 people get their quizzes regenerated?) is the actual remaining work. **Remaining:** 1. Determine whether the 10 messages survived #895; if not, decide how to make those users whole. 2. Decide on the 4 lost Keycloak audit events. 3. Implement the DLQ reprocessor, or delete the config keys that claim it exists — leaving them is worse than either.

spikerj referenced this issue

2026-07-29 16:02:56 +00:00

[Critical][Ops][Notifications] Ops alerting is silently dead — every system-remediation-alert email fails SMTP AUTH 535; critical disk/OOM alerts never delivered #756

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#589