Git History Secret-Scrub Runbook

Last updated: 2026-06-13 Owner: Engineering / Security Status: Operator runbook (reusable for any future committed-secret incident) Related: SECURITY_REMEDIATION.md · SECRETS_MANAGEMENT.md · SECURITY_ARCHITECTURE.md

This is the step-by-step procedure for removing a secret that was committed to git history, not just from the current files, but from every past commit so it can't be fished out of the history. It is written to be followed carefully, one step at a time, by an operator. The current known incident (the leaked GitHub PAT + Discord webhook, tracker items O1/O2) is used as the worked example throughout, but the procedure generalizes to any future leak.

💡 Want a guided run? Ask Claude Code to co-pilot: it will have you run each step and paste the output, and it will confirm before the one irreversible step (git push --force). Recommended for the first time you do this.

---

⚠️ Read this before you touch anything

Three hard truths about history rewriting. Internalize them or you'll either waste effort or break the repo for your collaborators.

1. Rotation is the real fix. The scrub is secondary. Rewriting history removes the dead copy of a secret. It does not make a live secret safe, anyone who already cloned or forked the repo still has it, and GitHub may serve cached copies for a while. Always rotate (revoke + reissue) the secret first. The order is: rotate → then scrub → then delete the .gitleaksignore lines. A scrub without rotation is security theater.

2. This rewrites every commit SHA from the first leak forward, across all branches. Everyone with a clone (including your other local sessions and every open feature branch) must re-clone or hard-reset afterward, or their next push will resurrect the old history. Do this on a quiet tree: no other sessions mid-edit, ideally when open PRs are at a lull.

3. Your real working repo is never touched until the final push. This runbook operates on a throwaway mirror clone in /tmp. If anything looks wrong before Step 8, you just rm -rf the clone and nothing has changed. The point of no return is the git push --force in Step 8. (You can rewrite your existing repo in place instead, see Appendix B, but you give up this undo button and it disrupts any parallel sessions on the repo.)

---

Prerequisites

• Java (for BFG), java -version. (Already installed on the dev box: OpenJDK 21.)
• Admin access to the GitHub repo, you'll temporarily lift branch protection.
• (Optional, for verification) gitleaks, to re-scan the cleaned history.
• Tool of choice: BFG Repo-Cleaner (simplest for "replace these strings / delete these files"). git filter-repo is the modern alternative, see Appendix A.

---

Step 0, Identify exactly what leaked (without printing it)

You want the file paths and a regex pattern for each secret, never the literal value pasted into a terminal you're recording. For this repo, .gitleaksignore already enumerates the known findings:

cd /path/to/lantern_app
cat .gitleaksignore     # lists: <commitSHA>:<filepath>:<ruleID>:<line>

Confirm the footprint with the pickaxe (shows commits, never the value):

git log --all -S 'ghp_'                          --oneline | wc -l   # classic GitHub PAT
git log --all -S 'discord.com/api/webhooks'      --oneline | wc -l   # Discord webhook URL

Known leaks in this repo (the worked example):

Secret	Where	Pattern to redact
Classic GitHub PAT (ghp_…)	functions/.runtimeconfig.json, functions/.env.lantern-app-dev, docs/.../INTEGRATION_FIXES.md	regex:ghp_[A-Za-z0-9]{36}
Discord webhook URL	across ~14 commits	regex:https://discord(app)?\.com/api/webhooks/[0-9]+/[\w.-]+

The discord-client-id entries in .gitleaksignore are public IDs, not secrets, leave them.

---

Step 1, Rotate the secret FIRST (the real kill switch)

Before scrubbing, kill the live secret so the copies already out there become useless:

• GitHub PAT: GitHub → Settings → Developer settings → Personal access tokens → revoke the old token, issue a new one, update wherever it's consumed (Secret Manager / CI secrets, see SECRETS_MANAGEMENT.md).
• Discord webhook: Server Settings → Integrations → Webhooks → delete + recreate, update the consumer.

(If you are deliberately deferring rotation, e.g. until prod-prep, that's a risk decision: treat the token as compromised until you do. The scrub below still works, it just doesn't neutralize copies others already hold.)

---

Step 2, Throwaway mirror clone + fetch BFG

A --mirror clone is a bare copy of all refs (every branch + tag), which is what we need to rewrite everything at once.

cd /tmp
git clone --mirror https://github.com/cattreedev/lantern_app.git
curl -L -o bfg.jar https://repo1.maven.org/maven2/com/madgag/bfg/1.14.0/bfg-1.14.0.jar

You now have /tmp/lantern_app.git (bare mirror) and /tmp/bfg.jar.

---

Step 3, Write the redaction patterns

One regex per line. BFG replaces each match with ***REMOVED***. Using regex means you never paste the literal secret.

cd /tmp
cat > replacements.txt <<'EOF'
regex:ghp_[A-Za-z0-9]{36}
regex:https://discord(app)?\.com/api/webhooks/[0-9]+/[\w.-]+
EOF

For a future leak, add a line per new secret shape (e.g. regex:AKIA[0-9A-Z]{16} for an AWS key). If you can't express it as a clean regex, you can use a literal line (literal:the-exact-string), but prefer regex to avoid handling the value.

---

Step 4, Run BFG (redact content + delete leaked files)

--replace-text rewrites the secret content wherever it appears; --delete-files removes whole files by name (handy for .env files that should never have existed in history).

cd /tmp
java -jar bfg.jar --replace-text replacements.txt \
     --delete-files '{.env.lantern-app-dev,.runtimeconfig.json}' \
     lantern_app.git

Note: BFG intentionally protects the current commit (HEAD of the default branch), it won't rewrite your latest tree. That's fine here because the working tree is already redacted. If your current files still contained the secret, you'd fix and commit that normally first.

---

Step 5, Expire the old objects

BFG rewrites history but leaves the old objects reachable via reflog until you expire + garbage-collect them. This step is what actually makes the old blobs unrecoverable in the clone.

cd /tmp/lantern_app.git
git reflog expire --expire=now --all
git gc --prune=now --aggressive

---

Step 6, Verify locally BEFORE pushing (point of no return is next)

Confirm the secret is gone from the rewritten history while it's still just a throwaway clone:

cd /tmp/lantern_app.git
# Quick smoke check, these should now print 0 (or only ***REMOVED*** hits):
git log --all -S 'ghp_'                     --oneline | wc -l
git log --all -S 'discord.com/api/webhooks' --oneline | wc -l

# Authoritative re-scan (if gitleaks is installed):
gitleaks detect --source . --no-banner

If anything still shows the secret, fix replacements.txt and redo Steps 4–6. Nothing has reached GitHub yet, you can also just rm -rf /tmp/lantern_app.git and start over.

---

Step 7, Temporarily lift branch protection

GitHub rejects force-pushes to protected branches. In GitHub → Settings → Branches, temporarily disable (or allow force-push on) the protected branches, for this repo that's dev and main. Note what you changed so you can restore it in Step 9.

---

Step 8, Force-push the rewritten history ⚠️ (irreversible)

This is the point of no return. A mirror clone pushes all refs; --force overwrites the diverged (rewritten) history on the remote.

cd /tmp/lantern_app.git
git push --force

---

Step 9, Re-enable branch protection

Immediately restore the protection settings you changed in Step 7. Don't leave dev/main unprotected.

---

Step 10, Post-scrub cleanup

1. Tell every collaborator + close your other sessions, then re-clone fresh. Old clones will try to resurrect the old commits on their next push. The safe move for everyone is a fresh git clone (or git fetch + git reset --hard origin/<branch> on a clean tree).
2. Remove the now-stale .gitleaksignore lines for the scrubbed secrets and commit that (the findings no longer exist, so the suppressions are misleading).
3. GitHub caches old commits. Rewritten commits can remain reachable by direct SHA URL until GitHub runs its own GC. For guaranteed removal, contact GitHub Support and ask them to GC the repository (and purge cached views).
4. Forks still have the old history. If the repo has forks, the secret lives on in them, another reason rotation (Step 1) is the real control.

---

Verification checklist

• [ ] Secret rotated (revoked + reissued) and consumers updated
• [ ] git log --all -S '<pattern>' returns 0 on the pushed remote (re-clone fresh and check)
• [ ] gitleaks detect clean (or only ***REMOVED*** placeholders)
• [ ] .gitleaksignore lines for the scrubbed secret removed + committed
• [ ] Branch protection restored on dev / main
• [ ] Collaborators notified to re-clone; other local sessions refreshed
• [ ] (If forks/critical) GitHub Support asked to GC

---

Appendix A, git filter-repo alternative

Modern, faster, the tool the git project recommends. Same outcome, different invocation:

pip3 install --user git-filter-repo        # one-time; ensure ~/.local/bin is on PATH
cd /tmp
git clone https://github.com/cattreedev/lantern_app.git scrub-clone
cd scrub-clone
git filter-repo \
  --replace-text /tmp/replacements.txt \
  --invert-paths --path functions/.env.lantern-app-dev --path functions/.runtimeconfig.json
# filter-repo removes the 'origin' remote on purpose; re-add then force-push:
git remote add origin https://github.com/cattreedev/lantern_app.git
git push --force --all && git push --force --tags

filter-repo's replace-text file uses the same regex: / literal: syntax as BFG's replacements.txt, so you can reuse the file from Step 3.

---

Appendix B, In-place rewrite (no clone)

You don't have to use a throwaway clone, git filter-repo can rewrite your existing local repo directly. The clone is a safety convention, not a requirement; in-place is valid, it just removes your undo button.

# Rewrites your EXISTING repo (all local branches) + updates the working tree to match
cd /path/to/lantern_app
pip3 install --user git-filter-repo        # one-time; ensure ~/.local/bin is on PATH

git filter-repo --force \
  --replace-text /tmp/replacements.txt \
  --invert-paths --path functions/.env.lantern-app-dev --path functions/.runtimeconfig.json

# filter-repo deletes 'origin' on purpose (so you can't push without thinking). Re-add + force:
git remote add origin https://github.com/cattreedev/lantern_app.git
git push --force --all && git push --force --tags

--force is required because filter-repo refuses to touch a repo that isn't a fresh clone. It also expires the reflog + repacks internally, so you skip Step 5.

When in-place is fine: a solo repo, clean working tree, nothing else in flight.

Why the clone is the default in this runbook (the tradeoff):

• Not reversible. A botched in-place rewrite mangles your actual repo; undoing a history rewrite is fiddly even with filter-repo's backup. With a clone you just rm -rf and retry.
• Disrupts parallel sessions. In-place changes every commit SHA out from under any other session/branch working on this repo, they end up on orphaned commits.
• Needs a clean tree. Uncommitted changes / stashes complicate the in-place run.

Either way you still force-push and everyone still re-clones (Step 10), the clone only protects your copy during the rewrite; it doesn't change the collaborator impact.

---

Why this procedure (design notes)

• Mirror clone, not your working repo, so the destructive operation is fully reversible until the final push, and your day-to-day repo is never at risk.
• Regex patterns, not literals, so the secret is never re-typed into a shell history, CI log, or this very document.
• Rotate-first ordering, because history rewriting is the cleanup, not the fix; the live secret is neutralized by rotation alone.
• Verify-before-push, the only irreversible, collaborator-affecting step is isolated and gated behind a local verification.