spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

[Bug][Backend][Data] BookPages inflation — quiz saw 6 pages on a 3-page book; one book linked 570 pages (staleness-retry double-processing suspected) #509

New Issue
Closed
opened 2026-07-12 18:13:05 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-12 18:13:05 +00:00
Owner
Copy Link
Copy Source

Split from #507 (the takeable-empty-quiz + page-window fixes are PRs #231 + the ComputePageWindow PR). What remains is a data-level anomaly:

Evidence (2026-07-12, Seq)

  • 17:25: quiz for a 3-page sample PDF iterated page 4/6..6/6 not found in database — its quiz/book believed 6 pages.
  • 03:55: Updated 570 pages with BookId 6a531016… — a 37-page upload linked 570 BookPage rows.
  • The same sample PDF (identical SHA256 3dc20419…) was uploaded several times; multiple staleness retries re-drove stages for the same uploads during the outage windows.

Why the obvious suspect ISN'T it

BookManagementService links pages with p.Isbn == isbn && p.BookId == ObjectId.Empty — properly scoped, and system ISBNs are per-extraction-unique. So cross-ISBN theft shouldn't be possible… unless:

  1. Metadata re-extraction on staleness retry inserts a fresh page set per attempt (new ISBN each run) and a later CreateBookAndQuiz consumed an accumulated backlog — needs checking whether metadata.extracted → CreateBookAndQuiz ran multiple times per upload (duplicate 'Created quiz record' lines WERE observed for the same minute).
  2. EF-Mongo translation of the compound filter behaving unexpectedly (verify with a live query).

Needs (Mongo inspection — can't be done from logs alone)

  • book-pages grouped by BookId/Isbn: duplicates per PageNumber? orphans with BookId == 000…?
  • books/book-generated-quizzes: duplicate records per upload (same SourceFileName/SHA, same minute)?
  • Then the code fix follows the data: likely (a) idempotency key on metadata extraction per uploadId (skip if pages already exist for this upload), and/or (b) CreateBookAndQuiz de-dup guard per uploadId.

Impact

Inflated TotalPages skews quiz windows, page progress %, and storage; orphan/duplicate page rows accumulate silently on every retry storm.

Related: #507, #493 (retry storms during tonight's outages made this visible).

Split from #507 (the takeable-empty-quiz + page-window fixes are PRs #231 + the ComputePageWindow PR). What remains is a **data-level anomaly**: ## Evidence (2026-07-12, Seq) - 17:25: quiz for a 3-page sample PDF iterated `page 4/6..6/6 not found in database` — its quiz/book believed **6 pages**. - 03:55: `Updated 570 pages with BookId 6a531016…` — a 37-page upload linked **570** BookPage rows. - The same sample PDF (identical SHA256 3dc20419…) was uploaded several times; multiple staleness retries re-drove stages for the same uploads during the outage windows. ## Why the obvious suspect ISN'T it BookManagementService links pages with `p.Isbn == isbn && p.BookId == ObjectId.Empty` — properly scoped, and system ISBNs are per-extraction-unique. So cross-ISBN theft shouldn't be possible… unless: 1. **Metadata re-extraction on staleness retry** inserts a fresh page set per attempt (new ISBN each run) and a later CreateBookAndQuiz consumed an accumulated backlog — needs checking whether metadata.extracted → CreateBookAndQuiz ran multiple times per upload (duplicate 'Created quiz record' lines WERE observed for the same minute). 2. EF-Mongo translation of the compound filter behaving unexpectedly (verify with a live query). ## Needs (Mongo inspection — can't be done from logs alone) - `book-pages` grouped by BookId/Isbn: duplicates per PageNumber? orphans with BookId == 000…? - `books`/`book-generated-quizzes`: duplicate records per upload (same SourceFileName/SHA, same minute)? - Then the code fix follows the data: likely (a) idempotency key on metadata extraction per uploadId (skip if pages already exist for this upload), and/or (b) CreateBookAndQuiz de-dup guard per uploadId. ## Impact Inflated TotalPages skews quiz windows, page progress %, and storage; orphan/duplicate page rows accumulate silently on every retry storm. Related: #507, #493 (retry storms during tonight's outages made this visible).
spikerj commented 2026-07-12 18:21:10 +00:00
Author
Owner
Copy Link
Copy Source

Root-caused WITH production data and fixed — PR #234. Forensics (read-only via mongo-router): the bad book holds pages [1,1,2,2,3,3] under TWO ISBNs (both uploads of the same PDF); the 570-page case is exonerated (single ISBN — genuinely a 570-page manual); zero orphans fleet-wide. Mechanism: SHA dedup returned the existing book but book.created couldn't express it, so the orchestrator linked the duplicate upload's fresh page set onto the existing book. Fix: WasDuplicate flows through BookCreatedResult→event→orchestrator, which now deletes the redundant extraction and completes the workflow honestly ('This book already exists — using the existing copy'). Backward-compatible on the wire. 3 tests. Remaining after merge: optionally clean the one bad test book (or let the #499 purge take it).

Root-caused WITH production data and fixed — **PR #234**. Forensics (read-only via mongo-router): the bad book holds pages [1,1,2,2,3,3] under TWO ISBNs (both uploads of the same PDF); the 570-page case is exonerated (single ISBN — genuinely a 570-page manual); zero orphans fleet-wide. Mechanism: SHA dedup returned the existing book but book.created couldn't express it, so the orchestrator linked the duplicate upload's fresh page set onto the existing book. Fix: WasDuplicate flows through BookCreatedResult→event→orchestrator, which now deletes the redundant extraction and completes the workflow honestly ('This book already exists — using the existing copy'). Backward-compatible on the wire. 3 tests. Remaining after merge: optionally clean the one bad test book (or let the #499 purge take it).
spikerj commented 2026-07-12 18:48:11 +00:00
Author
Owner
Copy Link
Copy Source

Fixed and closing: PR #234 (merged, auto-deployed) — WasDuplicate flows through BookCreatedResult → book.created → orchestrator, which deletes the redundant extraction and completes the duplicate workflow honestly instead of merging a second page set onto the existing book. Root cause was proven with production data (pages [1,1,2,2,3,3] under two ISBNs on one book; the 570-page case exonerated as a genuinely large manual). The one contaminated test book was since deleted; the #499 purge sweeps its records at the 30-day mark.

Fixed and closing: PR #234 (merged, auto-deployed) — WasDuplicate flows through BookCreatedResult → book.created → orchestrator, which deletes the redundant extraction and completes the duplicate workflow honestly instead of merging a second page set onto the existing book. Root cause was proven with production data (pages [1,1,2,2,3,3] under two ISBNs on one book; the 570-page case exonerated as a genuinely large manual). The one contaminated test book was since deleted; the #499 purge sweeps its records at the 30-day mark.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#509