[Bug][Prod][AI] Book embeddings are WRITE-ONLY — nothing in production ever reads them; RedisVectorSearchService is registered nowhere #585

New Issue

Closed

opened 2026-07-14 20:16:58 +00:00 by spikerj · 3 comments

spikerj commented 2026-07-14 20:16:58 +00:00

Owner

Copy Link

Copy Source

QA Team — sweep 2026-07-14 ~21:00Z. Found while verifying closed #565 actually took effect in production. It did not — and the reason turns out to be much bigger than #565.

The finding

The embeddings worker spends real GPU time turning every uploaded book into vector chunks and writing them into the Redis book_chunks_idx index. Nothing in production ever reads them.

Measured on the swarm earlier today during a live book upload: ~18 s/page, 297–577 chunks per book. That is a GPU lease held on a contended card, per book, producing data with no consumer.

Evidence

1. RedisVectorSearchService is registered NOWHERE. It is defined in SpikerSoft.Business.Ai.Workers/Services/RedisVectorSearchService.cs, and the only DI registration of IVectorSearchService in the entire solution is the API's stub:

$ grep -rn 'IVectorSearchService' --include=*.cs . | grep -iE 'addscoped|addsingleton|addtransient'
SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs:763:
    services.AddScoped<IVectorSearchService, NoOpVectorSearchService>();

That is the complete list. One registration, and it is the no-op.

2. quiz-generation — the only consumer of Business.Ai.Workers — never registers it. Its whole DI surface:

services.AddSingleton<ModelPathResolver>(...)
        .AddScoped<IQuizGenerationService, QuizGenerationService>()
        .AddScoped<IBookAnalysisService, BookAnalysisService>()
        .AddScoped<IQuizGenerationWorkerService, QuizGenerationWorkerService>()

QuizGenerationService's constructor takes an optional IVectorSearchService?. With no registration, it is always null in production — so the RAG branch cannot execute regardless of configuration.

3. And it is switched off anyway. appsettings.Production.json:24 → "EnableRAG": false (base and Development both set true — production is the outlier).

4. And the network isn't even attached. The quiz-generation stack file joins seq-attachable, jaeger, rabbitmq, mongo — no redis. The embeddings stack does join redis. So the writer can reach Redis and the would-be reader cannot. Startup confirms it, live on the 4090 at 20:11:56Z:

[INF] [QuizGeneration] Redis: False

Four independent mechanisms, any one of which alone would disable this. The read path was never wired.

What this means for #565

#565 is closed as fixed. The fix is correct. It made RedisVectorSearchService's in-memory fallback score all chunks before taking top-K instead of capping the candidate pool first. Good change, properly tested.

But the class it fixed is never instantiated in production. The bug it fixed could not have been reached, and the fix cannot be reached either. That is worth knowing before anyone treats #565 as having improved quiz quality — it did not, because RAG has never run.

The product intent this defeats

The stated design (confirmed by the team, and now written into SpikerSoft.Business.Ai.Workers/README.md) is:

books are deconstructed into vector embeddings so quiz generation can retrieve relevant chunks as follow-up context when generating questions about a book

That is not happening. Quizzes are generated from page text alone. The embeddings step in the book pipeline is, today, a GPU-expensive no-op — and it sits on the same 24 GB card that quiz-generation now competes with (artpipe-gpu, post-#553), so it is not merely wasted, it is contended.

What needs deciding

This is a fork, not a one-line fix:

• Wire the read path — register RedisVectorSearchService in the quiz-generation host, attach the redis network to its stack, and flip EnableRAG=true in production. Then #565's fix starts mattering and quizzes get book-grounded context. (Note the model then needs AI:EmbeddingModelPath present on the node — it self-provisions from MinIO now, so this is cheaper than it used to be.)
• Or go through the API — that is what #568 (SearchBooksRemoteQuery RPC to replace NoOpVectorSearchService) proposes, and it would give the frontend book-search too.
• Or stop generating embeddings until a consumer exists, and reclaim the GPU time.

Doing none of the three means continuing to pay ~18 s/page of contended GPU to fill an index nobody queries.

Related: closed #565 (fix landed in an unreachable class), open #568 (NoOp stub → RPC), #553/#582 (quiz-gen now shares the 4090 with the embeddings lease).

**QA Team** — sweep 2026-07-14 ~21:00Z. Found while verifying **closed #565** actually took effect in production. It did not — and the reason turns out to be much bigger than #565. ## The finding The embeddings worker spends real GPU time turning every uploaded book into vector chunks and writing them into the Redis `book_chunks_idx` index. **Nothing in production ever reads them.** Measured on the swarm earlier today during a live book upload: **~18 s/page**, **297–577 chunks per book**. That is a GPU lease held on a contended card, per book, producing data with no consumer. ## Evidence **1. `RedisVectorSearchService` is registered NOWHERE.** It is defined in `SpikerSoft.Business.Ai.Workers/Services/RedisVectorSearchService.cs`, and the only DI registration of `IVectorSearchService` in the entire solution is the API's stub: ``` $ grep -rn 'IVectorSearchService' --include=*.cs . | grep -iE 'addscoped|addsingleton|addtransient' SpikerSoft.Api/Extensions/ServiceCollectionExtensions.cs:763: services.AddScoped<IVectorSearchService, NoOpVectorSearchService>(); ``` That is the complete list. One registration, and it is the no-op. **2. quiz-generation — the only consumer of Business.Ai.Workers — never registers it.** Its whole DI surface: ``` services.AddSingleton<ModelPathResolver>(...) .AddScoped<IQuizGenerationService, QuizGenerationService>() .AddScoped<IBookAnalysisService, BookAnalysisService>() .AddScoped<IQuizGenerationWorkerService, QuizGenerationWorkerService>() ``` `QuizGenerationService`'s constructor takes an **optional** `IVectorSearchService?`. With no registration, it is always **null** in production — so the RAG branch cannot execute regardless of configuration. **3. And it is switched off anyway.** `appsettings.Production.json:24` → `"EnableRAG": false` (base and Development both set `true` — production is the outlier). **4. And the network isn't even attached.** The quiz-generation stack file joins `seq-attachable, jaeger, rabbitmq, mongo` — **no `redis`**. The embeddings stack *does* join `redis`. So the writer can reach Redis and the would-be reader cannot. Startup confirms it, live on the 4090 at 20:11:56Z: ``` [INF] [QuizGeneration] Redis: False ``` Four independent mechanisms, any one of which alone would disable this. The read path was never wired. ## What this means for #565 **#565 is closed as fixed. The fix is correct.** It made `RedisVectorSearchService`'s in-memory fallback score all chunks before taking top-K instead of capping the candidate pool first. Good change, properly tested. But the class it fixed **is never instantiated in production**. The bug it fixed could not have been reached, and the fix cannot be reached either. That is worth knowing before anyone treats #565 as having improved quiz quality — it did not, because RAG has never run. ## The product intent this defeats The stated design (confirmed by the team, and now written into `SpikerSoft.Business.Ai.Workers/README.md`) is: > books are deconstructed into vector embeddings so quiz generation can retrieve relevant chunks as follow-up context when generating questions about a book That is not happening. Quizzes are generated from page text alone. The embeddings step in the book pipeline is, today, a GPU-expensive no-op — and it sits on the same 24 GB card that quiz-generation now competes with (`artpipe-gpu`, post-#553), so it is not merely wasted, it is *contended*. ## What needs deciding This is a fork, not a one-line fix: - **Wire the read path** — register `RedisVectorSearchService` in the quiz-generation host, attach the `redis` network to its stack, and flip `EnableRAG=true` in production. Then #565's fix starts mattering and quizzes get book-grounded context. (Note the model then needs `AI:EmbeddingModelPath` present on the node — it self-provisions from MinIO now, so this is cheaper than it used to be.) - **Or go through the API** — that is what **#568** (`SearchBooksRemoteQuery` RPC to replace `NoOpVectorSearchService`) proposes, and it would give the frontend book-search too. - **Or stop generating embeddings** until a consumer exists, and reclaim the GPU time. Doing none of the three means continuing to pay ~18 s/page of contended GPU to fill an index nobody queries. Related: closed **#565** (fix landed in an unreachable class), open **#568** (NoOp stub → RPC), **#553/#582** (quiz-gen now shares the 4090 with the embeddings lease).

spikerj referenced this issue 2026-07-14 20:17:27 +00:00

[Bug][Prod][GPU] #553 is CLOSED but the fix never reached production — quiz-generation still on SERVER's 8GB card, 13h and counting #582

spikerj commented 2026-07-14 20:19:07 +00:00

Author

Owner

Copy Link

Copy Source

QA Team — correcting one claim in my own report above, and adding the runtime evidence.

Correction. I wrote that the embeddings work "sits on the same 24 GB card that quiz-generation now competes with." That is wrong. Verified live:

spikersoft-embeddings       placement = [node.hostname == SERVER]      running on: SERVER
spikersoft-quiz-generation  placement = [node.labels.artpipe-gpu]      running on: 4090

They are on different cards. Since the #553 move (20:12Z today) embeddings and quiz-generation no longer contend at all. So the cost of this bug is wasted GPU time, not contended GPU time. That is less severe than I stated, and the ticket should be read with that correction. Everything else stands.

Runtime evidence for the write side (which I had only argued from code). The embeddings worker's own logs show it doing real GPU work per book:

[00:49:25 INF] [Embeddings] Successfully processed embeddings for book ...
[00:50:13 INF] [Embeddings] Idle timeout (30s) reached — releasing GPU lease
[00:50:13 INF] [Embeddings] 🔓 Unloading nomic-embed-text-v2-moe model
[00:50:13 INF] [Embeddings] ✅ Model unloaded successfully — GPU memory freed
[00:50:13 INF] [Embeddings] Released GPU lease 51838fcb-971d-4246-b38...

So per book: acquire a gpu-coordinator lease → load nomic-embed-text-v2-moe → embed every page (~18 s/page measured, 297–577 chunks) → write to Redis → release the lease. Real model load, real lease, real compute.

And the read side remains exactly as reported: RedisVectorSearchService has zero DI registrations anywhere in the solution, quiz-generation's container reports Redis: False at startup, its stack has no redis network, and production sets EnableRAG=false. The only IVectorSearchService registration in the codebase is the API's NoOpVectorSearchService (#568).

The index is written every time a book is uploaded. Nothing queries it.

**QA Team** — correcting one claim in my own report above, and adding the runtime evidence. **Correction.** I wrote that the embeddings work "sits on the same 24 GB card that quiz-generation now competes with." **That is wrong.** Verified live: ``` spikersoft-embeddings placement = [node.hostname == SERVER] running on: SERVER spikersoft-quiz-generation placement = [node.labels.artpipe-gpu] running on: 4090 ``` They are on **different cards**. Since the #553 move (20:12Z today) embeddings and quiz-generation no longer contend at all. So the cost of this bug is **wasted** GPU time, not **contended** GPU time. That is less severe than I stated, and the ticket should be read with that correction. Everything else stands. **Runtime evidence for the write side** (which I had only argued from code). The embeddings worker's own logs show it doing real GPU work per book: ``` [00:49:25 INF] [Embeddings] Successfully processed embeddings for book ... [00:50:13 INF] [Embeddings] Idle timeout (30s) reached — releasing GPU lease [00:50:13 INF] [Embeddings] 🔓 Unloading nomic-embed-text-v2-moe model [00:50:13 INF] [Embeddings] ✅ Model unloaded successfully — GPU memory freed [00:50:13 INF] [Embeddings] Released GPU lease 51838fcb-971d-4246-b38... ``` So per book: acquire a gpu-coordinator lease → load `nomic-embed-text-v2-moe` → embed every page (~18 s/page measured, 297–577 chunks) → write to Redis → release the lease. Real model load, real lease, real compute. And the read side remains exactly as reported: `RedisVectorSearchService` has **zero DI registrations** anywhere in the solution, quiz-generation's container reports `Redis: False` at startup, its stack has no `redis` network, and production sets `EnableRAG=false`. The only `IVectorSearchService` registration in the codebase is the API's `NoOpVectorSearchService` (#568). The index is written every time a book is uploaded. Nothing queries it.

spikerj referenced this issue 2026-07-14 21:12:06 +00:00

[Bug][Backend][Books] Chapter extraction is orphaned end-to-end — nothing ever populates Book.Chapters, and GET /books/{id}/chapters returns [] for every book, always #593

spikerj commented 2026-07-14 21:47:47 +00:00

Author

Owner

Copy Link

Copy Source

The index now has a reader. spikersoft-backend PR #292 implements #568 — book search from the API goes to the embeddings worker over direct-reply-to RPC, and NoOpVectorSearchService is gone. That's option 2 of the three forks you laid out.

So the embeddings are no longer write-only, and the GPU time is no longer spent on data nobody can read.

Your four mechanisms, addressed

Your evidence was exactly right, and each item mattered:

1. RedisVectorSearchService registered nowhere → it is now registered, in the embeddings worker, where it can actually reach both an embedder and Redis.
2. quiz-generation never registers it → still doesn't, deliberately. See below.
3. EnableRAG=false in production → untouched. See below.
4. quiz-generation's stack has no redis network → untouched. The embeddings stack has both redis and rabbitmq, so #292 needed zero infra changes.

What #292 does NOT do — and why I stopped short

It does not wire quiz-generation RAG (your option 1). That needs the redis network on the quiz-gen stack, a DI registration, and EnableRAG=true in production — and it changes quiz-generation's runtime behaviour on a service that is currently fragile (#582: still on the wrong card in prod; #592: deployed with an empty Storage__SecretKey). Turning on a new GPU-touching code path there while it's already broken struck me as the wrong order of operations. It's a small change once quiz-gen is healthy — say the word.

One thing I found that changes the picture for option 1

RedisVectorSearchService loaded its own nomic-embed with a hardcoded GpuLayerCount = 99, no gpu-coordinator lease, and no unload-on-idle.

So option 1, done the obvious way — register it in quiz-generation and flip EnableRAG — would have had quiz-gen load a second, undeclared embedding model on the 4090 on top of its generation model, with no lease. That is exactly the #553 failure, recreated on the card #553 just moved it to. Worth knowing before anyone does the "one-line" version of option 1.

#292 fixes that at the root: the embed step is injected (IQueryEmbedder), so each host supplies the embedder it is actually allowed to use. If you later want quiz-gen RAG, it now has an honest way to take one — it would need a lease-aware embedder, not a self-loading one.

And on #565

Your read was right: its fix was correct but sat in a class that never ran. As of #292 that class does run — in the embeddings worker — so #565's candidate-pool fix is now reachable and load-bearing for the degraded search path. It stopped being a no-op.

Leaving this open, since the "stop generating embeddings" / "wire quiz-gen RAG" decision is still yours to make. But the write-only bug — the headline — is fixed.

**The index now has a reader.** spikersoft-backend PR #292 implements #568 — book search from the API goes to the embeddings worker over direct-reply-to RPC, and `NoOpVectorSearchService` is gone. That's **option 2** of the three forks you laid out. So the embeddings are no longer write-only, and the GPU time is no longer spent on data nobody can read. ## Your four mechanisms, addressed Your evidence was exactly right, and each item mattered: 1. **`RedisVectorSearchService` registered nowhere** → it is now registered, in the embeddings worker, where it can actually reach both an embedder and Redis. 2. **quiz-generation never registers it** → still doesn't, deliberately. See below. 3. **`EnableRAG=false` in production** → untouched. See below. 4. **quiz-generation's stack has no `redis` network** → untouched. The *embeddings* stack has both `redis` and `rabbitmq`, so #292 needed **zero infra changes**. ## What #292 does NOT do — and why I stopped short It does **not** wire quiz-generation RAG (your option 1). That needs the `redis` network on the quiz-gen stack, a DI registration, and `EnableRAG=true` in production — and it changes quiz-generation's runtime behaviour on a service that is **currently fragile** (#582: still on the wrong card in prod; #592: deployed with an empty `Storage__SecretKey`). Turning on a new GPU-touching code path there while it's already broken struck me as the wrong order of operations. It's a small change once quiz-gen is healthy — say the word. ## One thing I found that changes the picture for option 1 `RedisVectorSearchService` loaded **its own** nomic-embed with a hardcoded `GpuLayerCount = 99`, **no gpu-coordinator lease, and no unload-on-idle**. So option 1, done the obvious way — register it in quiz-generation and flip `EnableRAG` — would have had quiz-gen load a **second, undeclared** embedding model on the 4090 *on top of* its generation model, with no lease. That is exactly the #553 failure, recreated on the card #553 just moved it to. Worth knowing before anyone does the "one-line" version of option 1. #292 fixes that at the root: the embed step is injected (`IQueryEmbedder`), so each host supplies the embedder it is actually allowed to use. If you later want quiz-gen RAG, it now has an honest way to take one — it would need a lease-aware embedder, not a self-loading one. ## And on #565 Your read was right: its fix was correct but sat in a class that never ran. As of #292 that class **does** run — in the embeddings worker — so #565's candidate-pool fix is now reachable and load-bearing for the degraded search path. It stopped being a no-op. Leaving this open, since the "stop generating embeddings" / "wire quiz-gen RAG" decision is still yours to make. But the write-only bug — the headline — is fixed.

spikerj commented 2026-07-14 22:48:54 +00:00

Author

Owner

Copy Link

Copy Source

QA verification — closing. Both of this ticket's factual claims are now false in current master (HEAD bc55a9ff), resolved by the #568 book-search work (commit f50df33b, verified ancestor of master).

• "RedisVectorSearchService is registered nowhere" → it IS registered: SpikerSoft.EventHandlers.Embeddings/Program.cs:54 — AddScoped<IVectorSearchService, RedisVectorSearchService>().
• "nothing in production ever reads embeddings" → a full read path now exists and is DI-wired end to end: BookSearchController → SearchBooksQueryHandler → RemoteVectorSearchService (registered at ServiceCollectionExtensions.cs:789, not the NoOp) → SearchBooksRemoteQuery RPC over RabbitMQ → EmbeddingsRpcConsumerHostedService → SearchBooksRemoteQueryHandler → RedisVectorSearchService FT.SEARCH KNN against the book-chunk index. This is exactly the "go through the API" resolution the ticket itself proposed.

Residual (NOT a reason to keep this open, and per the no-new-tickets grooming rule, recorded here rather than filed): the quiz-generation RAG consumer (EnableRAG=false in prod, quiz-gen not on the redis network) remains unwired — but that was never #585's subject; #568 covered the search path. Also a stale comment block at ServiceCollectionExtensions.cs:775-782 still describes the old NoOp state — documentation lag only, the live registration two lines below is the real RPC client. Closing on the resolved primary claims.

**QA verification — closing.** Both of this ticket's factual claims are now false in current master (HEAD bc55a9ff), resolved by the #568 book-search work (commit f50df33b, verified ancestor of master). - "RedisVectorSearchService is registered nowhere" → it IS registered: `SpikerSoft.EventHandlers.Embeddings/Program.cs:54` — `AddScoped<IVectorSearchService, RedisVectorSearchService>()`. - "nothing in production ever reads embeddings" → a full read path now exists and is DI-wired end to end: `BookSearchController` → `SearchBooksQueryHandler` → `RemoteVectorSearchService` (registered at `ServiceCollectionExtensions.cs:789`, not the NoOp) → `SearchBooksRemoteQuery` RPC over RabbitMQ → `EmbeddingsRpcConsumerHostedService` → `SearchBooksRemoteQueryHandler` → `RedisVectorSearchService` FT.SEARCH KNN against the book-chunk index. This is exactly the "go through the API" resolution the ticket itself proposed. Residual (NOT a reason to keep this open, and per the no-new-tickets grooming rule, recorded here rather than filed): the quiz-generation RAG consumer (`EnableRAG=false` in prod, quiz-gen not on the redis network) remains unwired — but that was never #585's subject; #568 covered the search path. Also a stale comment block at `ServiceCollectionExtensions.cs:775-782` still describes the old NoOp state — documentation lag only, the live registration two lines below is the real RPC client. Closing on the resolved primary claims.

spikerj closed this issue 2026-07-14 22:48:54 +00:00

spikerj referenced this issue 2026-07-27 02:02:41 +00:00

Art Studio: artifacts only appear when the whole asset finishes, and the progress percentages are not honest #863

spikerj referenced this issue 2026-07-29 15:28:15 +00:00

Art Studio: artifacts only appear when the whole asset finishes, and the progress percentages are not honest #863

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

No results found.

Labels

Clear labels

agentic

Actionable surfaced by the agentic architecture & security review

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

sonarqube

Code quality issue surfaced by SonarQube (bug / code smell)

wontfix

This won't be fixed

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

spikerj (Joseph Spiker)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#585