Photo gallery: Florence-2 AI auto-tagging pipeline step (auto-tags + caption, suppressible, full backfill) #894

New Issue

Closed

opened 2026-07-29 05:48:36 +00:00 by spikerj · 2 comments

spikerj commented 2026-07-29 05:48:36 +00:00

Owner

Copy Link

Copy Source

After a photograph reaches Ready, run Florence-2 (microsoft/Florence-2-base) on its 2048px preview to produce AI tags + a caption. Depends on #893 (tags feature).

Design (user-approved):

• AI tags stored SEPARATELY: Photograph.AutoTags + RejectedAutoTags (suppress-on-delete is durable across re-runs; invariant disjoint, so effective tags = Tags ∪ AutoTags), AutoCaption (doubles as alt text), AutoTagModelVersion/AutoTaggedAt idempotency markers.
• New GPU lane copying the QR-art dispatch pattern: contracts photograph.autotag.lifecycle exchange, PhotographAutoTagConsumer in ArtPipeProcessor (prefetch 1, always-ack, per-job GPU lease 2048MB pinned RequiredNodeId=SERVER — per-job so an idle tagger holds 0 VRAM; coordinator Priority is a no-op so low priority = own queue), input = photograph-previews bucket download (Florence can't read ORF).
• artpipe: models/Florence2 manifest (transformers pinned exactly; weight_patterns must include *.py for trust_remote_code under HF_HUB_OFFLINE=1) + florence2.py backend running <MORE_DETAILED_CAPTION> + , deterministic derive_tags (≤15 tags), tags.json sidecar as output_path (executor requires an output file).
• Publisher hook in PhotographOrchestrator.HandleMovedAsync (single Status=Ready site), gated PhotoAutoTag__Enabled; publisher flips only after the consumer stack is live.
• infra: spikersoft-artpipe-model-florence2 stack (per-job shape, SERVER pin) + CI arms in model-env-images.yml and spikersoft-artpipe-processor.yml.
• Backfill everything: tools/BackfillPhotographAutoTags publishes one command per Ready photo (ModelVersion skip = idempotent, doubles as retry).
• Angular: AI chips styled (outlined + sparkle) in the lightbox editor, delete → POST photos/{id}/auto-tags/reject; gallery filter needs no client change (server-side union).

IPTC keyword seeding (#893) remains separate and unaffected.

After a photograph reaches Ready, run Florence-2 (microsoft/Florence-2-base) on its 2048px preview to produce AI tags + a caption. Depends on #893 (tags feature). Design (user-approved): - AI tags stored SEPARATELY: Photograph.AutoTags + RejectedAutoTags (suppress-on-delete is durable across re-runs; invariant disjoint, so effective tags = Tags ∪ AutoTags), AutoCaption (doubles as alt text), AutoTagModelVersion/AutoTaggedAt idempotency markers. - New GPU lane copying the QR-art dispatch pattern: contracts photograph.autotag.lifecycle exchange, PhotographAutoTagConsumer in ArtPipeProcessor (prefetch 1, always-ack, per-job GPU lease 2048MB pinned RequiredNodeId=SERVER — per-job so an idle tagger holds 0 VRAM; coordinator Priority is a no-op so low priority = own queue), input = photograph-previews bucket download (Florence can't read ORF). - artpipe: models/Florence2 manifest (transformers pinned exactly; weight_patterns must include *.py for trust_remote_code under HF_HUB_OFFLINE=1) + florence2.py backend running <MORE_DETAILED_CAPTION> + <OD>, deterministic derive_tags (≤15 tags), tags.json sidecar as output_path (executor requires an output file). - Publisher hook in PhotographOrchestrator.HandleMovedAsync (single Status=Ready site), gated PhotoAutoTag__Enabled; publisher flips only after the consumer stack is live. - infra: spikersoft-artpipe-model-florence2 stack (per-job shape, SERVER pin) + CI arms in model-env-images.yml and spikersoft-artpipe-processor.yml. - Backfill everything: tools/BackfillPhotographAutoTags publishes one command per Ready photo (ModelVersion skip = idempotent, doubles as retry). - Angular: AI chips styled (outlined + sparkle) in the lightbox editor, delete → POST photos/{id}/auto-tags/reject; gallery filter needs no client change (server-side union). IPTC keyword seeding (#893) remains separate and unaffected.

spikerj commented 2026-07-29 15:27:00 +00:00

Author

Owner

Copy Link

Copy Source

Audited against origin/master (backend/angular) and origin/main (artpipe). 4 of 5 workstreams landed — staying open for the Angular half. Notes updated so the remaining work is unambiguous.

Landed

Backend — PR #508 (766cb8e4): all five markers on Photograph.cs:131,138,142,150,153 (AutoTags, RejectedAutoTags, AutoCaption, AutoTagModelVersion, AutoTaggedAt). Importantly SpikerDbContext.cs:1675-1686 maps AutoTags/RejectedAutoTags as plain BSON arrays with only a ValueComparer — it did not re-introduce the JSON converter #893 just removed, so the server-side union filter actually translates. Disjoint invariant enforced idempotently in RejectPhotographAutoTagCommandHandler.cs:31-36. Consumer PhotographAutoTagConsumer.cs: prefetch 1 (:99), always-ack (:116), per-job lease with RequiredNodeId (:222-228) released in finally (:285-287), ModelVersion skip (:185). Publisher gated at PhotographOrchestrator.cs:151 on PhotoAutoTag:Enabled (default false, correct for staged rollout). Backfill at tools/BackfillPhotographAutoTags/Program.cs.

artpipe — PR #47 (2053880, origin/main): models/Florence2/artpipe.json with transformers==4.49.0 pinned and weight_patterns including "*.py"; src/artpipe/models/florence2.py running <MORE_DETAILED_CAPTION> (:201) + <OD> (:207), DEFAULT_MAX_TAGS = 15 (:43), deterministic derive_tags (:58), tags.json sidecar (:223,245); tests in tests/test_florence2_tags.py.

infrastructure — PR #155 (1f08d2b): spikersoft-artpipe-model-florence2/docker-stack-gpu.yml (per-job shape, PhotoAutoTag__Enabled=true, SERVER placement, replicas: 0 staged), README row, and the PhotoAutoTag__Enabled=false gate on spikersoft-photograph-processor/docker-stack.yml:35.

Remaining

1. The entire Angular workstream is absent. git grep -in "autotag\|auto_tag\|auto-tag\|autoCaption" origin/master over spikersoft-angular returns zero hits; git log origin/master --grep='894' is empty. Specifically missing:

• autoTags / autoCaption / rejectedAutoTags on libraries/domain/photograph/src/lib/photograph.model.ts (:64 has only tags: string[]). The API already serves these fields (PhotographyController.cs:517-518,580,583), so the client currently discards them silently.
• AI chip styling (outlined + sparkle) in the lightbox editor — photo-lightbox.component.ts:130 reads only photo.tags.
• The delete→reject call. POST photos/{id}/auto-tags/reject exists (PhotographyController.cs:378) with no caller; photograph.service.ts has only setTags/getTagUniverse (:72,:77).
• i18n keys for the AI-tag UI in devTools/en.json + es.json.

Net effect: once the stack goes live, AI tags will be invisible and unrejectable in the UI. (The gallery filter genuinely needs no client change — that part is server-side and done.)

2. florence2 missing from DEFAULT_IMAGES in spikersoft-artpipe-processor.yml. :188 ends "... textto3d qrmonster", and the workflow_dispatch input description (:33-44) doesn't list it. The build loop is generic, so workflow_dispatch images=florence2 works today — but artpipe-model-florence2:latest, the tag the stack pulls, is currently only producible by manual dispatch and never by a push. The infra stack header schedules this as rollout step 6; flagging it as a real item rather than pure sequencing because README.md:309 notes this is exactly the class of bug that killed triposr (#839).

To close: ship the Angular auto-tag UI (model fields, AI chip styling, reject call, en+es i18n), and add florence2 to DEFAULT_IMAGES in both spikersoft-artpipe-processor.yml and artpipe's model-env-images.yml:220 when the staged rollout reaches step 6.

Audited against `origin/master` (backend/angular) and `origin/main` (artpipe). **4 of 5 workstreams landed — staying open for the Angular half.** Notes updated so the remaining work is unambiguous. ### Landed **Backend** — PR #508 (`766cb8e4`): all five markers on `Photograph.cs:131,138,142,150,153` (`AutoTags`, `RejectedAutoTags`, `AutoCaption`, `AutoTagModelVersion`, `AutoTaggedAt`). Importantly `SpikerDbContext.cs:1675-1686` maps `AutoTags`/`RejectedAutoTags` as plain BSON arrays with only a `ValueComparer` — it did **not** re-introduce the JSON converter #893 just removed, so the server-side union filter actually translates. Disjoint invariant enforced idempotently in `RejectPhotographAutoTagCommandHandler.cs:31-36`. Consumer `PhotographAutoTagConsumer.cs`: prefetch 1 (`:99`), always-ack (`:116`), per-job lease with `RequiredNodeId` (`:222-228`) released in `finally` (`:285-287`), ModelVersion skip (`:185`). Publisher gated at `PhotographOrchestrator.cs:151` on `PhotoAutoTag:Enabled` (default `false`, correct for staged rollout). Backfill at `tools/BackfillPhotographAutoTags/Program.cs`. **artpipe** — PR #47 (`2053880`, `origin/main`): `models/Florence2/artpipe.json` with `transformers==4.49.0` pinned and `weight_patterns` including `"*.py"`; `src/artpipe/models/florence2.py` running `<MORE_DETAILED_CAPTION>` (`:201`) + `<OD>` (`:207`), `DEFAULT_MAX_TAGS = 15` (`:43`), deterministic `derive_tags` (`:58`), `tags.json` sidecar (`:223,245`); tests in `tests/test_florence2_tags.py`. **infrastructure** — PR #155 (`1f08d2b`): `spikersoft-artpipe-model-florence2/docker-stack-gpu.yml` (per-job shape, `PhotoAutoTag__Enabled=true`, SERVER placement, `replicas: 0` staged), README row, and the `PhotoAutoTag__Enabled=false` gate on `spikersoft-photograph-processor/docker-stack.yml:35`. ### Remaining **1. The entire Angular workstream is absent.** `git grep -in "autotag\|auto_tag\|auto-tag\|autoCaption" origin/master` over spikersoft-angular returns **zero hits**; `git log origin/master --grep='894'` is empty. Specifically missing: - `autoTags` / `autoCaption` / `rejectedAutoTags` on `libraries/domain/photograph/src/lib/photograph.model.ts` (`:64` has only `tags: string[]`). The API already serves these fields (`PhotographyController.cs:517-518,580,583`), so the client currently **discards them silently**. - AI chip styling (outlined + sparkle) in the lightbox editor — `photo-lightbox.component.ts:130` reads only `photo.tags`. - The delete→reject call. `POST photos/{id}/auto-tags/reject` exists (`PhotographyController.cs:378`) with **no caller**; `photograph.service.ts` has only `setTags`/`getTagUniverse` (`:72,:77`). - i18n keys for the AI-tag UI in `devTools/en.json` + `es.json`. Net effect: once the stack goes live, AI tags will be invisible and unrejectable in the UI. (The gallery *filter* genuinely needs no client change — that part is server-side and done.) **2. `florence2` missing from `DEFAULT_IMAGES` in `spikersoft-artpipe-processor.yml`.** `:188` ends `"... textto3d qrmonster"`, and the `workflow_dispatch` input description (`:33-44`) doesn't list it. The build loop is generic, so `workflow_dispatch images=florence2` works today — but `artpipe-model-florence2:latest`, the tag the stack pulls, is currently only producible by manual dispatch and never by a push. The infra stack header schedules this as rollout step 6; flagging it as a real item rather than pure sequencing because `README.md:309` notes this is exactly the class of bug that killed triposr (#839). **To close:** ship the Angular auto-tag UI (model fields, AI chip styling, reject call, en+es i18n), and add `florence2` to `DEFAULT_IMAGES` in both `spikersoft-artpipe-processor.yml` and artpipe's `model-env-images.yml:220` when the staged rollout reaches step 6.

spikerj referenced this issue 2026-07-29 16:56:07 +00:00

ArtPipeProcessor deploy: /mnt/infrastructure pull fails auth (stale checkout deployed, only a WARNING) + one stack's 'update out of sequence' aborts all later deploys #841

spikerj referenced this issue 2026-07-29 16:56:19 +00:00

[Feature][Calendar] Create-for-child from the parent calendar — owner selector in the event modal (follow-up to #788) #790

spikerj referenced this issue 2026-07-29 18:28:10 +00:00

[ArtStudio] Model-resident workers — one always-loaded model per service instance, task queues per model, coordinator-arbitrated, scale by replicas #368

spikerj commented 2026-07-29 22:33:46 +00:00

Author

Owner

Copy Link

Copy Source

Closing — rollout complete end to end (2026-07-29):

• Code: backend#508 (consumer + reject endpoint + backfill tool), artpipe#47 (Florence-2 model backend), backend#509 (stack deploy), backend#510 (corrective PascalCase casing migration), artpipe#48 (florence2 in ModelEnvImages DEFAULT_IMAGES), infra#156 (PhotoAutoTag__Enabled=true IaC sync) — all merged.
• Data: #510 script run against prod — all 71 photographs converted (Tags/AutoTags/RejectedAutoTags proper arrays, junk camelCase removed, residuals 0/0/0).
• Backfill: BackfillPhotographAutoTags queued 67/71 (4 without a usable derivative); florence2 consumer processed all 67 with GPU lease per photo, 0 errors — every photo has AutoTagModelVersion: florence2-base-v1, ~5 tags + caption. 47 distinct captions/67; duplicates are burst neighbors, verified not a processing bug.
• Gate: publisher gate live on photograph-processor since 6h+ and now consistent with IaC, so future stack deploys keep it enabled.

EXIF keyword seeding (#893 sibling) confirmed a permanent no-op — the ORFs carry no embedded keywords.

Closing — rollout complete end to end (2026-07-29): - **Code**: backend#508 (consumer + reject endpoint + backfill tool), artpipe#47 (Florence-2 model backend), backend#509 (stack deploy), backend#510 (corrective PascalCase casing migration), artpipe#48 (florence2 in ModelEnvImages DEFAULT_IMAGES), infra#156 (`PhotoAutoTag__Enabled=true` IaC sync) — all merged. - **Data**: #510 script run against prod — all 71 photographs converted (`Tags`/`AutoTags`/`RejectedAutoTags` proper arrays, junk camelCase removed, residuals 0/0/0). - **Backfill**: `BackfillPhotographAutoTags` queued 67/71 (4 without a usable derivative); florence2 consumer processed all 67 with GPU lease per photo, 0 errors — every photo has `AutoTagModelVersion: florence2-base-v1`, ~5 tags + caption. 47 distinct captions/67; duplicates are burst neighbors, verified not a processing bug. - **Gate**: publisher gate live on photograph-processor since 6h+ and now consistent with IaC, so future stack deploys keep it enabled. EXIF keyword seeding (#893 sibling) confirmed a permanent no-op — the ORFs carry no embedded keywords.

spikerj closed this issue 2026-07-29 22:33:47 +00:00

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

No results found.

Labels

Clear labels

agentic

Actionable surfaced by the agentic architecture & security review

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

sonarqube

Code quality issue surfaced by SonarQube (bug / code smell)

wontfix

This won't be fixed

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

spikerj (Joseph Spiker)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#894