spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

Research: Origin Private File System (OPFS) as selective offline pin/outbox beside MinIO #865

New Issue
Open
opened 2026-07-27 02:38:32 +00:00 by spikerj · 0 comments
spikerj commented 2026-07-27 02:38:32 +00:00
Owner
Copy Link
Copy Source

Summary

Research spike: evaluate whether the Origin Private File System (OPFS) is worth utilizing beside (not instead of) our MinIO/S3 storage for offline + online use — to the maximum extent that is practical.

Verdict: OPFS is worth pursuing as a selective client pin / upload-outbox layer. It is not worth auto-mirroring every object that lands in MinIO.

Uploads remain API-mediated today (browser → multipart → API → staging/workers → MinIO). Presigned URLs are GET-only for serving. OPFS would never bypass ClamAV, ownership checks, or parental controls.

Context (current architecture)

Server truth = MinIO

Relevant buckets / prefixes (user-facing + pipeline):

Bucket / prefix Content
uploads (+ blogs/, lesson-videos/, code/, media/) Staging + geography media
ebooks Books (auth'd; not on public media fallback)
blogs Blog pictures
lesson-videos Lesson video (HTTP Range)
dlls Decompile uploads
assets 3D / Trellis-style assets
photos Camera photographs (still partly disk-bound)
vault-files Info-vault
photograph-previews Gallery preview derivatives
art-asset-artifacts / art-asset-quarantine Art Studio
profile-images Avatars
sensitive-documents Passports etc. (server AES-GCM ciphertext)
quarantine, ai-models, gitea-packages, reference-data Pipeline / infra — not client offline material

Shared seam: SpikerSoft.Storage (IObjectStore, keyed stores, GetPresignedUrl, Range reads). Media proxy: S3MediaFallbackMiddleware for /blog-pictures, /geography-media, /lesson-videos.

Client storage we already have (no OPFS yet)

Mechanism Where Purpose
IndexedDB spikersoft-books-offline book-offline-store.service.ts PDF blobs + EPUB pages/TOC/resources; navigator.storage.persist()
Cache API art-studio-game-mesh-v1 art-asset-mesh-blob-cache.ts GLB meshes; ~512 MB LRU soft cap
Cache API spikersoft-offline-preload-v1 offline-cache.service.ts WASM / Pyodide / .NET / Clang preload
Angular SW (ngsw) ngsw-config.json App shell + some API dataGroups
IndexedDB trails, Marks customers, tool-storage Structured / small caches

Gap: large binary performance (PDF / GLB / TIFF / video) and offline write staging before MinIO sync.

Size / limit reminders

  • Global multipart ~100 MB (route overrides higher)
  • Lesson videos ~200 MB; Art Studio TIFF / variant uploads up to ~500 MB
  • Art GLBs ~50–115 MB; camera RAW ~64 MB; DLLs ≤50 MB
  • Decompile trees/source live in Redis/memory — not MinIO

What OPFS is (and isn't)

Is Isn't
Origin-scoped sandboxed FS; no picker/permission User-visible disk / Downloads
Fast byte I/O; sync access handles in workers; random seek A MinIO replacement or multi-device sync fabric
Good for large binary blobs Indexed queries (keep manifests in IndexedDB)
Shares quota with IndexedDB + Cache API Guaranteed persistent (need navigator.storage.persist(); Safari private = off / limited)

Baseline across Chromium / Firefox / Safari since ~2023. Still subject to eviction and shared origin quota.

Recommended layering:

MinIO (source of truth)
        ↕ auth'd API / workers
Angular
  ├── IndexedDB  → manifests, ACL snapshots, decompile trees, pin index
  ├── OPFS       → large pinned blobs + upload outbox
  ├── Cache API  → HTTP-immutable / small media / WASM (keep)
  └── ngsw       → app shell

Key on stable object id / hash, not short-TTL presigned URL query strings (same lesson as art #784 / #804).

Use cases & fit (within reason)

Content Offline value OPFS? Notes
Ebooks (user + parent→child) Very high Yes — upgrade Move PDF (and maybe EPUB resource blobs) off IndexedDB; keep manifest + sha256hash in IDB
Art Studio GLBs / concept art High Yes — upgrade Better than Cache API for 50–115 MB immutable meshes
Camera photos / stack TIFFs High (pros) Yes (selective) Pin albums / sessions; don't keep every RAW forever
Photo previews High Optional Cache API is enough
Lesson videos High, hard Yes if pin-to-watch Progressive download into OPFS; don't auto-cache libraries
Vault files High Yes (opt-in + caution) Consider client encrypt-at-rest for sensitive vault items
Blog media Medium Selective Images → Cache API; large blog video → OPFS pin
DLLs + decompile Medium (dev) Partial DLL blob → OPFS ok; tree/source → IndexedDB
Geography media Medium Selective Same pin pattern
Profile images Low No SW / Cache
Sensitive documents Risky Default no Server already AES-GCM; offline plaintext expands blast radius
Quarantine / ai-models / gitea / staging pipeline None for browser No Server-only
Upload outbox (before MinIO) High UX Yes — strongest Offline-first creation, not just consumption

Explicit non-goals

  • Auto-mirror every MinIO object into OPFS
  • Browser PUT/POST directly to MinIO (bypass API / ClamAV)
  • Caching unapproved / quarantined art or photo safety-flagged assets
  • Multi-device sync via OPFS alone (MinIO remains the hub)

Suggested pattern: Upload Outbox

Strongest product ROI is often offline write staging, not download mirroring.

User picks / captures file (online or offline)
  → write bytes to OPFS outbox/{localId}
  → write IndexedDB row: { localId, targetDomain, meta, status: queued|uploading|acked|failed, contentHash }
  → when online: Background Sync / app resume
      → multipart upload to existing API endpoints
      → on success: mark acked, delete OPFS outbox entry (or keep pin if also "available offline")
      → on conflict / auth fail: surface retry UI; never invent server keys client-side

Outbox candidates (first wave)

  1. Photograph upload (gallery / RAW / Touchup / Stacked) — large, frequent, painful on flaky mobile
  2. Art Studio variant / submit — TIFF / image uploads
  3. Vault file upload — explicit offline capture for learning vault
  4. Book upload (parent / teacher) — optional; rate limits still apply once online

Outbox rules

  • Outbox is not a MinIO bypass: same auth, scan, ownership, parental gates apply at sync time
  • Failures after partial upload must be idempotent (content-hash / client upload id)
  • Purge outbox on logout / account switch (multi-tenant)
  • Quota: outbox budget separate from "pinned downloads" budget
  • No Background Sync yet today — start with resume-on-online + optional SW sync later

Suggested pattern: Pin / Available Offline (downloads)

Product-shaped, opt-in (plus smart defaults):

  • Explicit "Available offline" / pin
  • Smart defaults: current book, assigned lesson video, open art loadout
  • Manifest in IndexedDB: { objectKey or domainId, sha256/etag, sizeBytes, pinnedAt, userId }
  • Bytes in OPFS under a stable path (ebooks/{bookId}/file, art/{assetId}/{artifactId}, …)
  • Stale when server hash ≠ local hash → re-download
  • Parental revocation: child's pins must not outlive ApprovedBookIds — reconcile on next online sync

Hard constraints to design for

  1. Shared quota — books + meshes + videos fight one pool; per-domain budgets + LRU (mesh cache already ~512 MB)
  2. Auth offline — cached entitlement snapshot + hash (books already do sha256hash)
  3. Parental controls — revocable pins for child libraries
  4. Multi-device — OPFS is per browser profile; MinIO syncs
  5. Sensitive — never auto-pin sensitive-documents; vault only opt-in
  6. Video — pin one quality; full-file (or progressive) download before offline play; Range online stays as today
  7. Presign TTL — cache by object key / hash, not URL

Suggested architecture / spike

Introduce a small shared client seam, e.g. LocalObjectStore:

  • OPFS for blob bodies
  • IndexedDB for manifests / pin index / outbox queue
  • Same conceptual key shape as server object identity (bucket/key or domain-stable ids)

Smallest honest prototype (ordered)

  1. Migrate book PDF offline from IndexedDB → OPFS (keep EPUB page model or move resources later)
  2. Photograph upload outbox (OPFS + IDB queue → existing photo upload API)
  3. Optionally move art mesh cache from Cache API → OPFS (measure load time / memory)

Success metrics for the spike:

  • Write/read speed vs IndexedDB for ~50–200 MB blobs
  • Quota pressure on typical student / ProArt devices
  • Safari persistence behavior with navigator.storage.persist()
  • Logout / user-switch purge correctness
  • No regression to online-only flows when OPFS unavailable (graceful degrade)

Priority recommendation (if we green-light)

Priority Work Why
P0 research spike LocalObjectStore + book PDF migration + photo outbox Answers "worth it?" with numbers
P1 Art mesh OPFS cache Clear pain (50–115 MB re-transfer)
P1 Lesson video "download for later" Product-shaped offline learning
P2 Vault opt-in pin (+ optional client encryption) High value, higher threat model
Later Blog large video pins, DLL blob cache Nice-to-have
Skip by default sensitive-documents, quarantine, ai-models, profile pics, blanket mirror Wrong threat model or low value

Related existing work

Ask

Treat this as an enhancement / research ticket:

  1. Decide whether to schedule the P0 spike (LocalObjectStore + book PDF + photo outbox).
  2. Keep MinIO as sole source of truth; OPFS is a device-local tier only.
  3. Do not auto-mirror all MinIO content — pin + outbox only.
## Summary Research spike: evaluate whether the **Origin Private File System (OPFS)** is worth utilizing **beside** (not instead of) our MinIO/S3 storage for offline + online use — to the maximum extent that is practical. **Verdict:** OPFS is worth pursuing as a **selective client pin / upload-outbox layer**. It is **not** worth auto-mirroring every object that lands in MinIO. Uploads remain API-mediated today (browser → multipart → API → staging/workers → MinIO). Presigned URLs are GET-only for serving. OPFS would never bypass ClamAV, ownership checks, or parental controls. --- ## Context (current architecture) ### Server truth = MinIO Relevant buckets / prefixes (user-facing + pipeline): | Bucket / prefix | Content | |---|---| | `uploads` (+ `blogs/`, `lesson-videos/`, `code/`, `media/`) | Staging + geography media | | `ebooks` | Books (auth'd; not on public media fallback) | | `blogs` | Blog pictures | | `lesson-videos` | Lesson video (HTTP Range) | | `dlls` | Decompile uploads | | `assets` | 3D / Trellis-style assets | | `photos` | Camera photographs (still partly disk-bound) | | `vault-files` | Info-vault | | `photograph-previews` | Gallery preview derivatives | | `art-asset-artifacts` / `art-asset-quarantine` | Art Studio | | `profile-images` | Avatars | | `sensitive-documents` | Passports etc. (server AES-GCM ciphertext) | | `quarantine`, `ai-models`, `gitea-packages`, `reference-data` | Pipeline / infra — **not** client offline material | Shared seam: `SpikerSoft.Storage` (`IObjectStore`, keyed stores, `GetPresignedUrl`, Range reads). Media proxy: `S3MediaFallbackMiddleware` for `/blog-pictures`, `/geography-media`, `/lesson-videos`. ### Client storage we already have (no OPFS yet) | Mechanism | Where | Purpose | |---|---|---| | IndexedDB `spikersoft-books-offline` | `book-offline-store.service.ts` | PDF blobs + EPUB pages/TOC/resources; `navigator.storage.persist()` | | Cache API `art-studio-game-mesh-v1` | `art-asset-mesh-blob-cache.ts` | GLB meshes; ~512 MB LRU soft cap | | Cache API `spikersoft-offline-preload-v1` | `offline-cache.service.ts` | WASM / Pyodide / .NET / Clang preload | | Angular SW (`ngsw`) | `ngsw-config.json` | App shell + some API dataGroups | | IndexedDB | trails, Marks customers, tool-storage | Structured / small caches | **Gap:** large binary performance (PDF / GLB / TIFF / video) and **offline write staging** before MinIO sync. ### Size / limit reminders - Global multipart ~100 MB (route overrides higher) - Lesson videos ~200 MB; Art Studio TIFF / variant uploads up to ~500 MB - Art GLBs ~50–115 MB; camera RAW ~64 MB; DLLs ≤50 MB - Decompile **trees/source** live in Redis/memory — **not** MinIO --- ## What OPFS is (and isn't) | Is | Isn't | |---|---| | Origin-scoped sandboxed FS; no picker/permission | User-visible disk / Downloads | | Fast byte I/O; sync access handles in workers; random seek | A MinIO replacement or multi-device sync fabric | | Good for large binary blobs | Indexed queries (keep manifests in IndexedDB) | | Shares quota with IndexedDB + Cache API | Guaranteed persistent (need `navigator.storage.persist()`; Safari private = off / limited) | Baseline across Chromium / Firefox / Safari since ~2023. Still subject to eviction and shared origin quota. Recommended layering: ```text MinIO (source of truth) ↕ auth'd API / workers Angular ├── IndexedDB → manifests, ACL snapshots, decompile trees, pin index ├── OPFS → large pinned blobs + upload outbox ├── Cache API → HTTP-immutable / small media / WASM (keep) └── ngsw → app shell ``` **Key on stable object id / hash**, not short-TTL presigned URL query strings (same lesson as art #784 / #804). --- ## Use cases & fit (within reason) | Content | Offline value | OPFS? | Notes | |---|---|---|---| | **Ebooks (user + parent→child)** | Very high | **Yes — upgrade** | Move PDF (and maybe EPUB resource blobs) off IndexedDB; keep manifest + `sha256hash` in IDB | | **Art Studio GLBs / concept art** | High | **Yes — upgrade** | Better than Cache API for 50–115 MB immutable meshes | | **Camera photos / stack TIFFs** | High (pros) | **Yes (selective)** | Pin albums / sessions; don't keep every RAW forever | | **Photo previews** | High | Optional | Cache API is enough | | **Lesson videos** | High, hard | **Yes if pin-to-watch** | Progressive download into OPFS; don't auto-cache libraries | | **Vault files** | High | **Yes (opt-in + caution)** | Consider client encrypt-at-rest for sensitive vault items | | **Blog media** | Medium | Selective | Images → Cache API; large blog video → OPFS pin | | **DLLs + decompile** | Medium (dev) | Partial | DLL blob → OPFS ok; **tree/source → IndexedDB** | | **Geography media** | Medium | Selective | Same pin pattern | | **Profile images** | Low | No | SW / Cache | | **Sensitive documents** | Risky | **Default no** | Server already AES-GCM; offline plaintext expands blast radius | | **Quarantine / ai-models / gitea / staging pipeline** | None for browser | **No** | Server-only | | **Upload outbox (before MinIO)** | High UX | **Yes — strongest** | Offline-first creation, not just consumption | ### Explicit non-goals - Auto-mirror every MinIO object into OPFS - Browser PUT/POST directly to MinIO (bypass API / ClamAV) - Caching unapproved / quarantined art or photo safety-flagged assets - Multi-device sync via OPFS alone (MinIO remains the hub) --- ## Suggested pattern: Upload Outbox Strongest product ROI is often **offline write staging**, not download mirroring. ```text User picks / captures file (online or offline) → write bytes to OPFS outbox/{localId} → write IndexedDB row: { localId, targetDomain, meta, status: queued|uploading|acked|failed, contentHash } → when online: Background Sync / app resume → multipart upload to existing API endpoints → on success: mark acked, delete OPFS outbox entry (or keep pin if also "available offline") → on conflict / auth fail: surface retry UI; never invent server keys client-side ``` ### Outbox candidates (first wave) 1. **Photograph upload** (gallery / RAW / Touchup / Stacked) — large, frequent, painful on flaky mobile 2. **Art Studio variant / submit** — TIFF / image uploads 3. **Vault file upload** — explicit offline capture for learning vault 4. **Book upload** (parent / teacher) — optional; rate limits still apply once online ### Outbox rules - Outbox is **not** a MinIO bypass: same auth, scan, ownership, parental gates apply at sync time - Failures after partial upload must be idempotent (content-hash / client upload id) - Purge outbox on logout / account switch (multi-tenant) - Quota: outbox budget separate from "pinned downloads" budget - No Background Sync yet today — start with resume-on-online + optional SW sync later --- ## Suggested pattern: Pin / Available Offline (downloads) Product-shaped, opt-in (plus smart defaults): - Explicit **"Available offline"** / pin - Smart defaults: current book, assigned lesson video, open art loadout - Manifest in IndexedDB: `{ objectKey or domainId, sha256/etag, sizeBytes, pinnedAt, userId }` - Bytes in OPFS under a stable path (`ebooks/{bookId}/file`, `art/{assetId}/{artifactId}`, …) - Stale when server hash ≠ local hash → re-download - Parental revocation: child's pins must not outlive `ApprovedBookIds` — reconcile on next online sync ### Hard constraints to design for 1. **Shared quota** — books + meshes + videos fight one pool; per-domain budgets + LRU (mesh cache already ~512 MB) 2. **Auth offline** — cached entitlement snapshot + hash (books already do `sha256hash`) 3. **Parental controls** — revocable pins for child libraries 4. **Multi-device** — OPFS is per browser profile; MinIO syncs 5. **Sensitive** — never auto-pin `sensitive-documents`; vault only opt-in 6. **Video** — pin one quality; full-file (or progressive) download before offline play; Range online stays as today 7. **Presign TTL** — cache by object key / hash, not URL --- ## Suggested architecture / spike Introduce a small shared client seam, e.g. `LocalObjectStore`: - OPFS for blob bodies - IndexedDB for manifests / pin index / outbox queue - Same conceptual key shape as server object identity (`bucket/key` or domain-stable ids) ### Smallest honest prototype (ordered) 1. **Migrate book PDF offline** from IndexedDB → OPFS (keep EPUB page model or move resources later) 2. **Photograph upload outbox** (OPFS + IDB queue → existing photo upload API) 3. **Optionally move art mesh cache** from Cache API → OPFS (measure load time / memory) Success metrics for the spike: - Write/read speed vs IndexedDB for ~50–200 MB blobs - Quota pressure on typical student / ProArt devices - Safari persistence behavior with `navigator.storage.persist()` - Logout / user-switch purge correctness - No regression to online-only flows when OPFS unavailable (graceful degrade) --- ## Priority recommendation (if we green-light) | Priority | Work | Why | |---|---|---| | P0 research spike | `LocalObjectStore` + book PDF migration + photo outbox | Answers "worth it?" with numbers | | P1 | Art mesh OPFS cache | Clear pain (50–115 MB re-transfer) | | P1 | Lesson video "download for later" | Product-shaped offline learning | | P2 | Vault opt-in pin (+ optional client encryption) | High value, higher threat model | | Later | Blog large video pins, DLL blob cache | Nice-to-have | | Skip by default | `sensitive-documents`, quarantine, ai-models, profile pics, blanket mirror | Wrong threat model or low value | --- ## Related existing work - Offline books: `spikersoft-angular` `book-offline-*` services - Art mesh Cache API: `art-asset-mesh-blob-cache.ts` - WASM offline preload: `offline-cache.service.ts` - MinIO epic / dual-run: epic #413, `docs/minio-storage-migration.md` - Presigned art media caching patterns: #784 / #804 - Parental book approval: `ApprovedBookIds` / `SetChildBookApproval` --- ## Ask Treat this as an **enhancement / research** ticket: 1. Decide whether to schedule the P0 spike (`LocalObjectStore` + book PDF + photo outbox). 2. Keep MinIO as sole source of truth; OPFS is a device-local tier only. 3. Do **not** auto-mirror all MinIO content — pin + outbox only.
spikerj added the enhancement label 2026-07-27 02:38:32 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#865