Remove files from Git history using git-filter-repo
Marco Franssen
7 min read•1279 words
Loading...
Blog.
Loading...
Marco Franssen
Many of you have probably committed a file to your repository that shouldn't have been there in the first place: credentials, for example, or a huge file that made cloning painfully slow. There are plenty of guides to removing these files entirely, often involving git filter-branch or bfg sorcery. In this post, I'll show you how to do it with git-filter-repo, which makes the process much simpler.
Recently, I had to rewrite a repository whose huge video files were slowing down the CI pipeline. Those videos were removed in later commits, but they still existed in Git history and made the repository large and slow to clone. In the past, I've also committed credentials that I had to remove. Trust me, we've all been there. I started with the git filter-branch approach I'd used for years, then noticed the following message.
$ git filter-branch --index-filter 'git rm --cached --ignore-unmatch my-huge-video.mp4' HEAD
WARNING: git-filter-branch has a glut of gotchas generating mangled history
rewrites. Hit Ctrl-C before proceeding to abort, then use an
alternative filtering tool such as 'git filter-repo'
(https://github.com/newren/git-filter-repo/) instead. See the
filter-branch manual page for more details; to squelch this warning,
set FILTER_BRANCH_SQUELCH_WARNING=1.At the time of writing, the Git CLI gives us this warning, but there isn't much written about
git-filter-repoyet. The documentation on GitHub still refers tofilter-branch. That's a good reason for me to write about this tool and raise awareness.
After reading about git-filter-repo, I found that it makes straightforward history rewrites, such as removing a file entirely, much easier. You get a simpler command, and you don't need to run tools like BFG for the final cleanup. For other use cases, you can still use the more advanced features of git filter-branch, but for simple rewrites, git-filter-repo is the recommended approach in this walkthrough.
So how do we get started? First, we need to install git-filter-repo. Follow the documentation for your package manager. Here's how I did this on my MacBook using Homebrew.
brew install git-filter-repoWith the tool installed, I restarted the process of removing the files entirely from the repository. Before showing that specific step, though, I want to walk you through the process from the beginning.
I'm not responsible for mistakes or data loss when you perform this process yourself. Becoming a master at any skill takes the full effort of your heart, mind, and soul.
We start by cloning the repository with the --mirror option, which brings all branch information and other refs into our local copy.
Cloning with
--mirrordoes not give you a working directory.
$ git clone --mirror git@github.com:marcofranssen/my-repo-to-be-rewritten.git
Cloning into bare repository 'my-repo-to-be-rewritten'...
remote: Enumerating objects: 211, done.
remote: Counting objects: 100% (211/211), done.
remote: Compressing objects: 100% (108/108), done.
remote: Total 211 (delta 98), reused 205 (delta 95), pack-reused 0
Receiving objects: 100% (211/211), 50.09 MiB | 5.21 MiB/s, done.
Resolving deltas: 100% (98/98), done.Next, I want to find the files deleted in the past, as I don't remember all of them. The following command shows which files were removed in previous commits.
$ git log --diff-filter=D --summary
commit 22a5d9613f51821b08d54e1d96cdae7c7b15652a
Author: Marco Franssen <marco.franssen@gmail.com>
Date: Tue Jul 28 13:24:24 2020 +0100
Removed processed video data from repo and changed code to write outside of repo
delete mode 100644 data/my-video.mp4
delete mode 100644 data/my-other-video.mp4
commit 910a4e00df2044ad77feed05c1d1627490fa02b1
Author: Marco Franssen <marco.franssen@gmail.com>
Date: Wed Feb 26 15:09:31 2020 +0100
Renaming video-processor.py to video_processor.py
delete mode 100644 src/video-processor.py
commit ba73d5ecb5b18137f90e3eb186bb244a9368d7a5
Author: Marco Franssen <marco.franssen@gmail.com>
Date: Thu Mar 7 10:09:31 2020 +0100
Removed service credentials
delete mode 100644 .envFrom the output above, we can see that I removed a .env file that apparently contained credentials, as well as some processed videos. A file was also renamed, but that's fine. Now that we know which files to remove, we can use git filter-repo.
$ git filter-repo --invert-paths --path .env
Parsed 197 commits
New history written in 0.11 seconds; now repacking/cleaning...
Repacking your repo and cleaning out old unneeded objects
Enumerating objects: 210, done.
Counting objects: 100% (210/210), done.
Delta compression using up to 12 threads
Compressing objects: 100% (127/127), done.
Writing objects: 100% (210/210), done.
Building bitmaps: 100% (48/48), done.
Total 210 (delta 98), reused 144 (delta 75), pack-reused 0
Completely finished after 0.64 seconds.
$ git filter-repo --invert-paths --path data/my-video.mp4
Parsed 197 commits
New history written in 0.09 seconds; now repacking/cleaning...
Repacking your repo and cleaning out old unneeded objects
Enumerating objects: 210, done.
Counting objects: 100% (210/210), done.
Delta compression using up to 12 threads
Compressing objects: 100% (127/127), done.
Writing objects: 100% (210/210), done.
Building bitmaps: 100% (48/48), done.
Total 210 (delta 98), reused 144 (delta 75), pack-reused 0
Completely finished after 0.64 seconds.
$ git filter-repo --invert-paths --path data/my-other-video.mp4
Parsed 197 commits
New history written in 0.10 seconds; now repacking/cleaning...
Repacking your repo and cleaning out old unneeded objects
Enumerating objects: 210, done.
Counting objects: 100% (210/210), done.
Delta compression using up to 12 threads
Compressing objects: 100% (127/127), done.
Writing objects: 100% (210/210), done.
Building bitmaps: 100% (48/48), done.
Total 210 (delta 98), reused 144 (delta 75), pack-reused 0
Completely finished after 0.64 seconds.We've now rewritten the repository and removed the files we accidentally committed. Two steps remain: push the rewritten history and tell your teammates to make a fresh clone.
Yes, this time it will be fast because the big files are gone!
git push --no-verify --mirror
Enumerating objects: 199, done.
Writing objects: 100% (199/199), 266.73 KiB | 3.03 MiB/s, done.
Total 199 (delta 0), reused 0 (delta 0), pack-reused 199
remote: Resolving deltas: 100% (96/96), done.
To github.com:marcofranssen/my-repo-to-be-rewritten.git
+ 0b09ac1...3c843c0 feature/ci -> feature/ci (forced update)
+ 175089f...7fa3dfb master -> master (forced update)
* [new reference] refs/replace/0362424174271cda267832b0ef3e78dfeb59ec72 -> refs/replace/0362424174271cda267832b0ef3e78dfeb59ec72
* [new reference] refs/replace/22a5d9613f51821b08d54e1d96cdae7c7b15652a -> refs/replace/22a5d9613f51821b08d54e1d96cdae7c7b15652a
* [new reference] refs/replace/e09dd373762b51f361d0033326982db62e3a524f -> refs/replace/e09dd373762b51f361d0033326982db62e3a524f
* [new reference] refs/replace/f1831654ceb3cad707d9d3f579a6aac14144e139 -> refs/replace/f1831654ceb3cad707d9d3f579a6aac14144e139
! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref)
! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref)
! [remote rejected] refs/pull/2/merge -> refs/pull/2/merge (deny updating a hidden ref)
! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
error: failed to push some refs to 'github.com:marcofranssen/my-repo-to-be-rewritten.git'You can ignore the "deny updating a hidden ref" messages shown here; they relate to GitHub pull requests. If you check the repository on GitHub, you'll see the new history. Cloning should be much faster now that the huge files have been removed from that history.
To make things easier, I've also added an alias to my .gitconfig that lets me find deleted files in Git history with a shorter command that's easier to remember.
git config --global alias.deleted "log --diff-filter=D --summary"This allows me to type git deleted as opposed to git log --diff-filter=D --summary.
You can find my entire .gitconfig, including a whole bunch of aliases, in my dotfiles GitHub repository. These aliases save me a lot of typing and speed up my development workflow.
Thanks for reading my post. Please spread the word about this tool so others can find a less error-prone way to clean up their Git history.
Marco Franssen
Build enum-like types in Go with iota, generate String methods with stringer, and marshal and unmarshal named values as JSON strings.
Marco Franssen
Why I'm moving my blog from Hexo to Next.js, how I'm migrating Markdown posts and React components, and what's ready in the new blog preview.
Marco Franssen
Configure a React app in the Nginx 1.19 Docker image with environment variables and templates, including API proxy settings for different deployments.
Marco Franssen
Build a two-node Elasticsearch cluster with Docker Compose, load-balance requests through Traefik, and add Cerebro as a web-based admin interface.