spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

Gitea LFS storage → MinIO (keep 751MB of large assets off laptop-server host disk) #801

New Issue
Closed
opened 2026-07-22 19:11:35 +00:00 by spikerj · 2 comments
spikerj commented 2026-07-22 19:11:35 +00:00
Owner
Copy Link
Copy Source

Follow-on from angular #544 (moved 751MB of binaries to Git LFS) + #800 (history purge). Mirror the existing packages→MinIO setup for LFS so the blobs live in MinIO, not on the laptop-server host FS.

Done (safe, reversible prep — 2026-07-22)

  • Bucket gitea-lfs created in MinIO.
  • Policy gitea-lfs-rw created (Get/Put/Delete on gitea-lfs/*, List on the bucket) and attached to user gitea-svc alongside the existing gitea-packages-rw (packages policy untouched). Verified: gitea-svc can RW gitea-lfs and still lists gitea-packages.
  • Infra PR adds [storage.lfs]→minio env to gitea/docker-stack.yml (durable config).

Remaining (run on laptop-server / with GITEA_S3_SECRET_KEY exported)

  1. Copy existing LFS objects (non-destructive; gitea keeps serving from local disk): 
    docker exec $(docker ps -qf name=gitea_gitea) gitea migrate-storage \
  --type lfs --storage minio --minio-endpoint minio:9000 \
  --minio-access-key-id gitea-svc --minio-secret-access-key "$GITEA_S3_SECRET_KEY" \
  --minio-bucket gitea-lfs --minio-location us-east-1 --minio-bucket-lookup path
  2. Verify the object count/size landed: mc du <alias>/gitea-lfs (~matches on-disk LFS).
  3. Cut over — merge the infra PR and redeploy, OR scoped live update: 
    docker service update gitea_gitea \
  --env-add GITEA__storage_0X2E_lfs__STORAGE_TYPE=minio \
  --env-add GITEA__storage_0X2E_lfs__MINIO_ENDPOINT=minio:9000 \
  --env-add GITEA__storage_0X2E_lfs__MINIO_ACCESS_KEY_ID=gitea-svc \
  --env-add GITEA__storage_0X2E_lfs__MINIO_SECRET_ACCESS_KEY="$GITEA_S3_SECRET_KEY" \
  --env-add GITEA__storage_0X2E_lfs__MINIO_BUCKET=gitea-lfs \
  --env-add GITEA__storage_0X2E_lfs__MINIO_LOCATION=us-east-1 \
  --env-add GITEA__storage_0X2E_lfs__MINIO_USE_SSL=false \
  --env-add GITEA__storage_0X2E_lfs__MINIO_BUCKET_LOOKUP=path
    (brief rolling restart of just gitea)
  4. Verify served-from-MinIO: GET /api/v1/repos/spikerj/spikersoft-angular/media/projects/spikersoft/src/assets/v86/images/windows95.img → real bytes; a git lfs pull works.
  5. Only after confidence: delete the local /data/lfs (or /data/git/lfs) tree to reclaim host disk.

Rollback

docker service update gitea_gitea --env-rm GITEA__storage_0X2E_lfs__STORAGE_TYPE ... (or redeploy the pre-change stack). Local LFS files are retained until step 5, so reverting is instant.

Why the migration wasn't auto-run from 4090: the safe path is docker exec into the on-node gitea container (inherits exact config, can't mutate the shared app.ini). That must run on laptop-server; this session has no shell there, and spinning up a fresh gitea container to do it risks the shared /etc/gitea/app.ini.

Follow-on from angular #544 (moved 751MB of binaries to Git LFS) + #800 (history purge). Mirror the existing packages→MinIO setup for LFS so the blobs live in MinIO, not on the laptop-server host FS. ## Done (safe, reversible prep — 2026-07-22) - Bucket `gitea-lfs` created in MinIO. - Policy `gitea-lfs-rw` created (Get/Put/Delete on `gitea-lfs/*`, List on the bucket) and **attached to user `gitea-svc` alongside the existing `gitea-packages-rw`** (packages policy untouched). Verified: gitea-svc can RW `gitea-lfs` and still lists `gitea-packages`. - Infra PR adds `[storage.lfs]`→minio env to `gitea/docker-stack.yml` (durable config). ## Remaining (run on laptop-server / with GITEA_S3_SECRET_KEY exported) 1. **Copy existing LFS objects** (non-destructive; gitea keeps serving from local disk): ``` docker exec $(docker ps -qf name=gitea_gitea) gitea migrate-storage \ --type lfs --storage minio --minio-endpoint minio:9000 \ --minio-access-key-id gitea-svc --minio-secret-access-key "$GITEA_S3_SECRET_KEY" \ --minio-bucket gitea-lfs --minio-location us-east-1 --minio-bucket-lookup path ``` 2. **Verify** the object count/size landed: `mc du <alias>/gitea-lfs` (~matches on-disk LFS). 3. **Cut over** — merge the infra PR and redeploy, OR scoped live update: ``` docker service update gitea_gitea \ --env-add GITEA__storage_0X2E_lfs__STORAGE_TYPE=minio \ --env-add GITEA__storage_0X2E_lfs__MINIO_ENDPOINT=minio:9000 \ --env-add GITEA__storage_0X2E_lfs__MINIO_ACCESS_KEY_ID=gitea-svc \ --env-add GITEA__storage_0X2E_lfs__MINIO_SECRET_ACCESS_KEY="$GITEA_S3_SECRET_KEY" \ --env-add GITEA__storage_0X2E_lfs__MINIO_BUCKET=gitea-lfs \ --env-add GITEA__storage_0X2E_lfs__MINIO_LOCATION=us-east-1 \ --env-add GITEA__storage_0X2E_lfs__MINIO_USE_SSL=false \ --env-add GITEA__storage_0X2E_lfs__MINIO_BUCKET_LOOKUP=path ``` (brief rolling restart of just gitea) 4. **Verify served-from-MinIO**: `GET /api/v1/repos/spikerj/spikersoft-angular/media/projects/spikersoft/src/assets/v86/images/windows95.img` → real bytes; a `git lfs pull` works. 5. Only after confidence: delete the local `/data/lfs` (or `/data/git/lfs`) tree to reclaim host disk. ## Rollback `docker service update gitea_gitea --env-rm GITEA__storage_0X2E_lfs__STORAGE_TYPE ...` (or redeploy the pre-change stack). Local LFS files are retained until step 5, so reverting is instant. **Why the migration wasn't auto-run from 4090:** the safe path is `docker exec` into the on-node gitea container (inherits exact config, can't mutate the shared app.ini). That must run on laptop-server; this session has no shell there, and spinning up a fresh gitea container to do it risks the shared `/etc/gitea/app.ini`.
spikerj commented 2026-07-22 20:24:10 +00:00
Author
Owner
Copy Link
Copy Source

Done — LFS is serving from MinIO (2026-07-22)

Cutover complete and verified: git lfs pull returns real bytes for the whole set (windows95.img 242MB, duckdb 38MB, pyodide 8.6MB, intro.mp3 16MB, …), media endpoint 200.

The one gotcha (for future reference)

gitea migrate-storage writes objects at the bucket root, but Gitea's minio LFS storage reads under a lfs/ base-path by default (same way packages uses packages/) — and it ignores an empty MINIO_BASE_PATH override. So after the migrate + cutover, every object 404'd. Fix was to move all objects under the lfs/ prefix:

mc mv --recursive <alias>/gitea-lfs/<hex-prefix>/ <alias>/gitea-lfs/lfs/<hex-prefix>/   # for each root prefix

After that, no config change was needed (Gitea's default base path already points at lfs/). Bucket now: 63 objects / 744MiB under gitea-lfs/lfs/.

Note for the runbook: the correct order is either (a) migrate-storage with an explicit --minio-base-path lfs, or (b) migrate to root then move objects under lfs/ (what we did). Don't set MINIO_BASE_PATH= empty expecting root — Gitea ignores it.

State

  • Live gitea_gitea service has the 8 [storage.lfs]→minio env vars (matches infra PR #139).
  • Infra PR #139 should be merged so a future docker stack deploy stays consistent (until then it works via the persisted app.ini).
  • Local /data/git/lfs on laptop-server still holds the objects (migrate only copied) — safe fallback; delete later to reclaim host disk once confident.

Keeping open until #139 merges + local LFS dir deleted.

## Done — LFS is serving from MinIO (2026-07-22) Cutover complete and verified: `git lfs pull` returns real bytes for the whole set (windows95.img 242MB, duckdb 38MB, pyodide 8.6MB, intro.mp3 16MB, …), media endpoint 200. ### The one gotcha (for future reference) `gitea migrate-storage` writes objects at the bucket **root**, but Gitea's minio LFS storage reads under a **`lfs/` base-path** by default (same way packages uses `packages/`) — and it **ignores an empty `MINIO_BASE_PATH` override**. So after the migrate + cutover, every object 404'd. Fix was to move all objects under the `lfs/` prefix: ``` mc mv --recursive <alias>/gitea-lfs/<hex-prefix>/ <alias>/gitea-lfs/lfs/<hex-prefix>/ # for each root prefix ``` After that, no config change was needed (Gitea's default base path already points at `lfs/`). Bucket now: 63 objects / 744MiB under `gitea-lfs/lfs/`. **Note for the runbook:** the correct order is either (a) `migrate-storage` with an explicit `--minio-base-path lfs`, or (b) migrate to root then move objects under `lfs/` (what we did). Don't set `MINIO_BASE_PATH=` empty expecting root — Gitea ignores it. ### State - Live `gitea_gitea` service has the 8 `[storage.lfs]`→minio env vars (matches infra PR #139). - Infra PR #139 should be **merged** so a future `docker stack deploy` stays consistent (until then it works via the persisted app.ini). - Local `/data/git/lfs` on laptop-server still holds the objects (migrate only copied) — safe fallback; delete later to reclaim host disk once confident. Keeping open until #139 merges + local LFS dir deleted.
spikerj commented 2026-07-22 21:06:54 +00:00
Author
Owner
Copy Link
Copy Source

Closing — LFS→MinIO migration is complete and verified live: objects served from gitea-lfs/lfs/, git lfs pull returns real bytes across the set, and it survived the Gitea 1.27.0 upgrade. Infra PR #139 (durable stack config) is merged.

Only trailing item is the optional host-disk reclaim: delete /data/git/lfs on laptop-server once you've had a few days of green CI/dev clones (it's the current safe fallback). Not worth keeping a ticket open for — recorded in the project memory. Closing.

Closing — LFS→MinIO migration is complete and verified live: objects served from `gitea-lfs/lfs/`, `git lfs pull` returns real bytes across the set, and it survived the Gitea 1.27.0 upgrade. Infra PR #139 (durable stack config) is merged. Only trailing item is the **optional** host-disk reclaim: delete `/data/git/lfs` on laptop-server once you've had a few days of green CI/dev clones (it's the current safe fallback). Not worth keeping a ticket open for — recorded in the project memory. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#801