spikerj/spikersoft-issues
1
0
Fork 0
Code Issues 196 Pull Requests Actions Packages Projects Releases Wiki Activity

spikersoft-angular: purge big binaries from git history (repo-size tier 3 — coordinated rewrite) #800

New Issue
Closed
opened 2026-07-22 17:28:38 +00:00 by spikerj · 1 comment
spikerj commented 2026-07-22 17:28:38 +00:00
Owner
Copy Link
Copy Source

Context

spikersoft-angular/.git is ~595 MB (backend: 80 MB). Tier 1+2 (angular PR #544) moved the 65 live binaries (751 MB) to Git LFS and made CI clones partial — that stops the growth but history still carries every old blob, including ~250+ MB of files deleted long ago (MacOS753_.img 100 MB, WIN95.json 75 MB, msdos622.img 64 MB, DCImage.img 50 MB…). Only a history rewrite reclaims that.

This rewrites every SHA and force-pushes. It needs a quiet window.

Preconditions

  • PR #544 merged + one full CI cycle green (build/e2e/publish with LFS)
  • Zero open PRs (rewrite orphans their base SHAs)
  • Announce: all clones must be re-cloned afterward; local worktrees recreated
  • git clone --mirror backup taken and stashed (keep ≥30 days)
  • git-filter-repo + git-lfs installed on the operator machine

Procedure

# 1. Fresh working clone (NOT the mirror backup)
git clone https://git.spikersoft.com/spikerj/spikersoft-angular.git rewrite && cd rewrite

# 2. Convert the kept binary paths to LFS across ALL history
#    (historical checkouts stay buildable — every rev gets pointers)
git lfs migrate import --everything \
  --include="*.img,*.iso,*.wasm,*.parquet,*.mp3,*.traineddata.gz,projects/spikersoft/src/assets/v86/**/*.bin"

# 3. Strip the dead monsters (deleted at HEAD, pure history baggage).
#    After step 2 the kept files are tiny pointers, so a size cut is safe:
git filter-repo --strip-blobs-bigger-than 10M --force

# 4. Verify BEFORE pushing
git count-objects -v -H          # expect size-pack well under 100 MB
git lfs ls-files | wc -l         # expect ~65+
git checkout master && ls -la projects/spikersoft/src/assets/v86/images/  # real files via LFS

# 5. Force-push all refs (temporarily lift master branch protection if enabled)
git remote add origin https://git.spikersoft.com/spikerj/spikersoft-angular.git  # filter-repo removes it
git push origin --force --all
git push origin --force --tags

Post-op

  • Gitea admin → Garbage collect the repository (note: Gitea's internal refs/pull/* for old PRs may pin some old objects server-side; clone size is what matters — fresh clones only fetch branch/tag refs)
  • CI: trigger a master run, confirm green end-to-end
  • Everyone re-clones (do NOT pull into an old clone — divergent histories)
  • Fresh-clone size check: expect < 100 MB + LFS on demand
  • SonarQube: old analysis revision SHAs become unknown — harmless, next scan re-baselines

Related hardening (optional, same effort family)

  • Move Gitea LFS storage to MinIO (app.ini [lfs] → STORAGE_TYPE = minio) — server config + restart via a SERVER-pinned swarm job; keeps the 751 MB (and growth) off the Gitea host disk
  • Enable Gitea's periodic git gc cron for all repos

Tier 1+2: spikersoft-angular PR #544. Diagnosis details in that PR's description.

## Context `spikersoft-angular/.git` is ~595 MB (backend: 80 MB). Tier 1+2 (angular PR #544) moved the 65 live binaries (751 MB) to Git LFS and made CI clones partial — that **stops the growth** but history still carries every old blob, including ~250+ MB of files deleted long ago (`MacOS753_.img` 100 MB, `WIN95.json` 75 MB, `msdos622.img` 64 MB, `DCImage.img` 50 MB…). Only a history rewrite reclaims that. **This rewrites every SHA and force-pushes. It needs a quiet window.** ## Preconditions - [ ] PR #544 merged + one full CI cycle green (build/e2e/publish with LFS) - [ ] Zero open PRs (rewrite orphans their base SHAs) - [ ] Announce: all clones must be re-cloned afterward; local worktrees recreated - [ ] `git clone --mirror` backup taken and stashed (keep ≥30 days) - [ ] `git-filter-repo` + `git-lfs` installed on the operator machine ## Procedure ```bash # 1. Fresh working clone (NOT the mirror backup) git clone https://git.spikersoft.com/spikerj/spikersoft-angular.git rewrite && cd rewrite # 2. Convert the kept binary paths to LFS across ALL history # (historical checkouts stay buildable — every rev gets pointers) git lfs migrate import --everything \ --include="*.img,*.iso,*.wasm,*.parquet,*.mp3,*.traineddata.gz,projects/spikersoft/src/assets/v86/**/*.bin" # 3. Strip the dead monsters (deleted at HEAD, pure history baggage). # After step 2 the kept files are tiny pointers, so a size cut is safe: git filter-repo --strip-blobs-bigger-than 10M --force # 4. Verify BEFORE pushing git count-objects -v -H # expect size-pack well under 100 MB git lfs ls-files | wc -l # expect ~65+ git checkout master && ls -la projects/spikersoft/src/assets/v86/images/ # real files via LFS # 5. Force-push all refs (temporarily lift master branch protection if enabled) git remote add origin https://git.spikersoft.com/spikerj/spikersoft-angular.git # filter-repo removes it git push origin --force --all git push origin --force --tags ``` ## Post-op - [ ] Gitea admin → Garbage collect the repository (note: Gitea's internal `refs/pull/*` for old PRs may pin some old objects server-side; clone size is what matters — fresh clones only fetch branch/tag refs) - [ ] CI: trigger a master run, confirm green end-to-end - [ ] Everyone re-clones (do NOT pull into an old clone — divergent histories) - [ ] Fresh-clone size check: expect < 100 MB + LFS on demand - [ ] SonarQube: old analysis `revision` SHAs become unknown — harmless, next scan re-baselines ## Related hardening (optional, same effort family) - Move Gitea LFS storage to MinIO (`app.ini` `[lfs]` → `STORAGE_TYPE = minio`) — server config + restart via a SERVER-pinned swarm job; keeps the 751 MB (and growth) off the Gitea host disk - Enable Gitea's periodic `git gc` cron for all repos Tier 1+2: spikersoft-angular PR #544. Diagnosis details in that PR's description.
spikerj commented 2026-07-22 18:56:56 +00:00
Author
Owner
Copy Link
Copy Source

Done — executed 2026-07-22

Result: fresh-clone .git 595 MB → 142 MB (76% reduction), HEAD 100% intact, LFS assets materialize normally.

What ran

  • Chose --strip-blobs-bigger-than 5M (not the migrate-then-strip in the original plan): HEAD's largest non-LFS file is chess green.jpg at 4.58 MB and the next thing up is 5.49 MB (a .mp3 already an LFS pointer at HEAD) — a clean cutline, so a plain size strip removes every binary monster while leaving HEAD untouched and never uploading dead files to LFS.
  • Mirror backup taken first (555 MB, retained ~30 days on the 4090).
  • git filter-repo --strip-blobs-bigger-than 5M over all 10 branches + 105 tags → 0 blobs >5M remain in history; MacOS753_.img/WIN95.json/msdos622.img etc. purged.
  • Verified master unchanged since backup (no lost work), lifted master force-push protection, push --force --all --tags, restored protection.
  • Triggered git_gc_repos + gc_lfs.

Verified

  • Fresh clone .git = 142 MB; duckdb-mvp.wasm pulls 37 MB of real bytes via LFS; green.jpg (kept, non-LFS) intact.
  • CI on rewritten master 1324c8bf: compute_version ✅, test-and-lint ✅ (partial clone works), build (LFS checkout) green on #544's run.

Notes / residual

  • All SHAs changed — anyone with a clone must re-clone (do not pull across the rewrite). The 9 non-master branches + all tags were force-updated to new SHAs.
  • Server-side disk may not fully drop until Gitea expires refs/pull/* (they still pin old objects); fresh-clone size — what devs/CI pull — is fixed at 142 MB regardless.
  • Didn't go below 100 MB on purpose: the 2–5 MB tail contains live HEAD files (chess textures, tesseract .wasm.js glue, .wavs) that a lower threshold would delete. Not worth the risk for ~40 MB.

Closing. Tiers 1+2 shipped in angular #544; optional follow-ups (Gitea LFS→MinIO, gc cron) noted there.

## Done — executed 2026-07-22 **Result: fresh-clone `.git` 595 MB → 142 MB** (76% reduction), HEAD 100% intact, LFS assets materialize normally. ### What ran - Chose `--strip-blobs-bigger-than 5M` (not the migrate-then-strip in the original plan): HEAD's largest non-LFS file is chess `green.jpg` at 4.58 MB and the next thing up is 5.49 MB (a `.mp3` already an LFS pointer at HEAD) — a clean cutline, so a plain size strip removes every binary monster while leaving HEAD untouched and never uploading dead files to LFS. - Mirror backup taken first (555 MB, retained ~30 days on the 4090). - `git filter-repo --strip-blobs-bigger-than 5M` over all 10 branches + 105 tags → 0 blobs >5M remain in history; `MacOS753_.img`/`WIN95.json`/`msdos622.img` etc. purged. - Verified master unchanged since backup (no lost work), lifted master force-push protection, `push --force --all --tags`, restored protection. - Triggered `git_gc_repos` + `gc_lfs`. ### Verified - Fresh clone `.git` = **142 MB**; `duckdb-mvp.wasm` pulls 37 MB of real bytes via LFS; `green.jpg` (kept, non-LFS) intact. - CI on rewritten master `1324c8bf`: `compute_version` ✅, `test-and-lint` ✅ (partial clone works), `build` (LFS checkout) green on #544's run. ### Notes / residual - **All SHAs changed** — anyone with a clone must re-clone (do not pull across the rewrite). The 9 non-master branches + all tags were force-updated to new SHAs. - Server-side disk may not fully drop until Gitea expires `refs/pull/*` (they still pin old objects); **fresh-clone size — what devs/CI pull — is fixed at 142 MB regardless**. - Didn't go below 100 MB on purpose: the 2–5 MB tail contains live HEAD files (chess textures, tesseract `.wasm.js` glue, `.wav`s) that a lower threshold would delete. Not worth the risk for ~40 MB. Closing. Tiers 1+2 shipped in angular #544; optional follow-ups (Gitea LFS→MinIO, gc cron) noted there.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agentic
Actionable surfaced by the agentic architecture & security review
 bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 sonarqube
Code quality issue surfaced by SonarQube (bug / code smell)
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
spikerj (Joseph Spiker)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: spikerj/spikersoft-issues#800