[Bug][Backend] Book deletion leaves all artifacts behind — on disk AND in the MinIO ebooks bucket (no purge path exists) #499

New Issue

2026-07-12T06:08:29Z

spikerj commented

2026-07-12 06:08:29 +00:00

Surfaced by Joey testing the #493 MinIO flow: after deleting uploaded books, the mirrored books/<ISBN>/ trees remained in the ebooks bucket.

Finding — this predates MinIO

DeleteBookCommandHandler is a pure soft delete: sets DeletedAt/DeletedBy on the book record. Nothing ever removes:

the filesystem tree /mnt/fusionio/spikersoft/ebooks/books/<ISBN>/ (covers, pages, images, the moved PDF itself),
the BookPage/BookPageImage/quiz records' blobs,
(new) the mirrored ebooks bucket prefix.

No purge/retention job exists anywhere for books (only trails have a purge command). The bucket is in exact parity with the disk — the debt was just invisible before.

Why fix

Storage growth on both media + privacy: a user-deleted book's full content persists indefinitely.

Options (needs a product pick)

Retention purge (recommended): background job hard-purges books where DeletedAt < now - N days — Mongo children, disk tree, and bucket prefix (metadata-svc already has DeleteObject on ebooks/*). Keeps an undelete window; matches the soft-delete intent.
Immediate hard cleanup on delete: simplest, but forfeits recoverability the soft delete implies.
Document as intended (rejected — privacy).

Interim cleanup done: the orphaned test tree (books/SPIKER4684262/, 151 MiB / 75 objects) removed from the bucket; the matching disk trees under /mnt/fusionio/spikersoft/ebooks/books/ still need an rm -rf by ops.

Related: #493 (mirror seam), #413.

Surfaced by Joey testing the #493 MinIO flow: after deleting uploaded books, the mirrored `books/<ISBN>/` trees remained in the `ebooks` bucket. ## Finding — this predates MinIO `DeleteBookCommandHandler` is a pure **soft delete**: sets `DeletedAt`/`DeletedBy` on the book record. Nothing ever removes: - the filesystem tree `/mnt/fusionio/spikersoft/ebooks/books/<ISBN>/` (covers, pages, images, **the moved PDF itself**), - the BookPage/BookPageImage/quiz records' blobs, - (new) the mirrored `ebooks` bucket prefix. No purge/retention job exists anywhere for books (only trails have a purge command). The bucket is in exact parity with the disk — the debt was just invisible before. ## Why fix Storage growth on both media + **privacy**: a user-deleted book's full content persists indefinitely. ## Options (needs a product pick) 1. **Retention purge (recommended):** background job hard-purges books where `DeletedAt < now - N days` — Mongo children, disk tree, and bucket prefix (`metadata-svc` already has DeleteObject on `ebooks/*`). Keeps an undelete window; matches the soft-delete intent. 2. **Immediate hard cleanup on delete:** simplest, but forfeits recoverability the soft delete implies. 3. Document as intended (rejected — privacy). Interim cleanup done: the orphaned test tree (`books/SPIKER4684262/`, 151 MiB / 75 objects) removed from the bucket; the matching disk trees under `/mnt/fusionio/spikersoft/ebooks/books/` still need an `rm -rf` by ops. Related: #493 (mirror seam), #413.

spikerj commented

2026-07-12 06:21:26 +00:00

Option 1 implemented (Joey's pick — retention purge, 30-day undelete window):

spikersoft-backend #227 — DeletedBookPurgeService in file-movement (the file-lifecycle owner with the /app/ebooks rw bind): daily sweep, hard-purges books soft-deleted >30d (Retention:PurgeSoftDeletedBooks:*, max 50/cycle). Crash-safe ordering: artifacts (disk tree + ebooks bucket prefix — the destructive inverse of the #493 mirror) → Mongo children (book-pages / book-page-images / book-generated-quizzes; note Quiz.BookId is a string) → the book record LAST. Book.Path is treated as untrusted data: anything that doesn't map cleanly under /app/ebooks throws instead of reaching the recursive delete. 13 tests.
spikersoft-infrastructure #39 — Storage__UseS3=true + metadata-svc key on the file-movement stack (already has DeleteObject on ebooks/*; no new credential).

Interim cleanup from tonight already done: both orphaned test trees removed from the bucket (0 objects); disk twins under /mnt/fusionio/spikersoft/ebooks/books/{SPIKER4684262,SPIKER1844867} still need an ops rm (or just wait 30 days — the purge will take them once merged, since their book records are soft-deleted).

After both merge + /mnt/infrastructure pull: #227's merge auto-deploys file-movement; the service logs 'DeletedBookPurgeService active: purging books soft-deleted more than 30 days ago' on boot. Closing once that log line shows in Seq.

**Option 1 implemented** (Joey's pick — retention purge, 30-day undelete window): - **spikersoft-backend #227** — `DeletedBookPurgeService` in file-movement (the file-lifecycle owner with the `/app/ebooks` rw bind): daily sweep, hard-purges books soft-deleted >30d (`Retention:PurgeSoftDeletedBooks:*`, max 50/cycle). Crash-safe ordering: artifacts (disk tree + `ebooks` bucket prefix — the destructive inverse of the #493 mirror) → Mongo children (book-pages / book-page-images / book-generated-quizzes; note Quiz.BookId is a string) → the book record LAST. `Book.Path` is treated as untrusted data: anything that doesn't map cleanly under `/app/ebooks` throws instead of reaching the recursive delete. 13 tests. - **spikersoft-infrastructure #39** — `Storage__UseS3=true` + metadata-svc key on the file-movement stack (already has DeleteObject on `ebooks/*`; no new credential). Interim cleanup from tonight already done: both orphaned test trees removed from the bucket (0 objects); disk twins under `/mnt/fusionio/spikersoft/ebooks/books/{SPIKER4684262,SPIKER1844867}` still need an ops rm (or just wait 30 days — the purge will take them once merged, since their book records are soft-deleted). After both merge + `/mnt/infrastructure` pull: #227's merge auto-deploys file-movement; the service logs 'DeletedBookPurgeService active: purging books soft-deleted more than 30 days ago' on boot. Closing once that log line shows in Seq.

spikerj commented

2026-07-12 06:30:38 +00:00

Verified deployed: Seq shows DeletedBookPurgeService active: purging books soft-deleted more than 30 days ago, every 1.00:00:00 on the rolled file-movement (backend #227 + infra #39, both merged). Tonight's soft-deleted test books will be hard-purged (disk + bucket + Mongo) when they pass the 30-day window. Closing.

Verified deployed: Seq shows `DeletedBookPurgeService active: purging books soft-deleted more than 30 days ago, every 1.00:00:00` on the rolled file-movement (backend #227 + infra #39, both merged). Tonight's soft-deleted test books will be hard-purged (disk + bucket + Mongo) when they pass the 30-day window. Closing.

spikerj closed this issue

2026-07-12 06:30:38 +00:00

spikerj referenced this issue

2026-07-12 08:06:53 +00:00

[Bug][Infra][CRITICAL] SERVER node crash-looping — kernel 'BUG: Bad page state in process dockerd' (memory corruption / likely bad RAM); flaps all SERVER-pinned services #503

spikerj referenced this issue

2026-07-12 08:09:32 +00:00

[Bug][Infra] MinIO doesn't recover after a transient /mnt/fusionio drive blip — stuck '0 drives provided', 'mc ready local' healthcheck misses it, needs manual restart #504

spikerj referenced this issue

2026-07-12 18:21:10 +00:00

[Bug][Backend][Data] BookPages inflation — quiz saw 6 pages on a 3-page book; one book linked 570 pages (staleness-retry double-processing suspected) #509

spikerj referenced this issue