Merge multiple Gradle projects while preserving Git history
Marco Franssen
9 min read•1766 words
Loading...
Blog.
Loading...
Marco Franssen
Two years after starting with a microservices architecture, we ran into team productivity issues. That made me think about what we could improve from both an architecture and a development perspective. We'd put the various components of our microservices in separate Git repositories, and over time it became harder for developers to work across them. I decided to simplify the workflow by merging the repositories for one functional bounded context into a single repository, while preserving their Git history.
I've finally found time to document the process for my own reference and to share it with the community. If you're thinking, "I'm not using Gradle", and are about to stop reading, don't! Only the final part is Gradle-specific. You can reuse most of the process for other kinds of projects.
To keep the impact on the development team as low as possible, I wrote a Bash migration script. It let me complete the migration in a few minutes and perform dry runs to check that everything still worked. By "everything", I mean:
Let's start with a folder structure similar to the one we had. Each folder contained a Git repository with a Gradle project, a Jenkinsfile, and other files for CI and CD.
$ tree code-folder
code-folder
├── domain-a-api
├── domain-a-commands
├── domain-a-consumer
├── domain-a-events
├── domain-b-api
├── domain-b-commands
├── domain-b-consumer
└── domain-b-events
8 directories, 0 filesFor the rest of this article, I'll focus on domain-a. The goal is to end up with the following structure.
$ tree -a code-folder/domain-a
code-folder/domain-a
├── .git
├── Jenkinsfile
├── api
│ └── src
├── build.gradle
├── commands
│ └── src
├── consumer
│ └── src
└── events
└── src
9 directories, 2 filesThe first step is to move the files in each existing repository into a subfolder. I wrote a Bash function to reuse across the projects I needed to migrate.
function move_into_subfolder () {
local subfolder=$1
if [ ! -z "$2" ] ; then
subfolder=$2
fi
pushd $1
mkdir $subfolder
git mv `ls -1 | grep -v $subfolder` $subfolder
git mv .gitignore $subfolder
git commit -a -m "Moving files into subfolder $subfolder"
# navigate back to previous folder
popd
}For example, calling move_into_subfolder "domain-a-api" "api" moves the files from domain-a-api into domain-a-api/api. The function also commits the move in the existing repository.
Since I was migrating several repositories, I needed to make sure I merged their latest develop branches. I wrote another Bash function to clone or update those repositories.
function clone_or_pull_develop() {
if [ ! -d "${1}" ] ; then
git clone -b develop git@github.com:marcofranssen/${1}.git
else
pushd ${1}
git reset HEAD~1 --hard
git pull
popd
fi
}This function clones the repository or pulls the latest develop changes. For an existing clone, it removes the previous move-into-subfolder commit so I can repeat the migration during dry runs.
We now have a way to prepare the repositories for merging without losing their commit history. For the merge itself, I wrote another reusable Bash function.
function merge_old_repo_commits () {
git remote add $1 ../$1
git fetch $1
git merge --no-edit --allow-unrelated-histories $1/develop
git remote rm $1
}First, we'll prepare a new Git repository. Then we can call the function to merge the existing repositories into it.
rm -rf domain-a #So we can easily restart from scratch
mkdir -p domain-a
cd domain-a
git init
git remote add origin git@github.com:marcofranssen/domain-a.git
merge_old_repo_commits "commands"
merge_old_repo_commits "events"
merge_old_repo_commits "consumer"
merge_old_repo_commits "api"We now have all four repositories' commit histories merged into a single repository. Next, we'll add a few commits to turn it into a working multi-module Gradle project. This is where you'll probably need to make your own project-specific changes. I'll show you what I did as a reference.
First, I added a README.md at the project root linking to the README.md files in the subprojects.
touch README.md
echo '# Domain A' > README.md
echo '' >> README.md
echo '## Api' >> README.md
echo '' >> README.md
echo '[README.md](api/README.md)' >> README.md
echo '## Consumer' >> README.md
echo '' >> README.md
echo '[README.md](consumer/README.md)' >> README.md
echo '## Commands' >> README.md
echo '' >> README.md
echo '[README.md](commands/README.md)' >> README.md
echo '## Events' >> README.md
echo '' >> README.md
echo '[README.md](events/README.md)' >> README.md
git add README.md
git commit -m "Add readme to root of project, linking to the module README.md files"Then I added a manually merged .gitignore file at the root and removed the original files.
cp ../.gitignore .
git add .gitignore
git rm api/.gitignore consumer/.gitignore events/.gitignore commands/.gitignore
git commit -m "Add .gitignore to root of project and remove .gitignore files from subfolders"I did the same for the Gradle configuration, so I could manage the build from the root of the new project.
gradle init
cp ../gradle* ../settings.gradle .
git add .
git update-index --chmod=+x gradlew
git commit -m "Initialized gradle project"
cp ../build.gradle .
echo "include 'api', 'consumer', 'events', 'commands'" >> settings.gradle
git add build.gradle settings.gradle
git rm -r api/build.gradle api/settings.gradle api/gradle*
git rm -r consumer/build.gradle consumer/settings.gradle consumer/gradle*
git rm -r events/build.gradle events/settings.gradle events/gradle*
git rm -r commands/build.gradle commands/settings.gradle commands/gradle*
git commit -m "Add new gradle configuration and remove the old ones"Finally, I added a new Docker Compose setup and removed the old files. I also added an updated Jenkinsfile, which I'd merged manually.
cp ../docker-compose.yaml .
git add docker-compose.yaml
git rm api/docker-compose.yml consumer/docker-compose.yml
git commit -m "Add docker-compose file in the root of project replacing the specific module ones"
cp ../ApiDockerfile api/Dockerfile
cp ../ConsumerDockerfile consumer/Dockerfile
git add api/ consumer/
git commit -m "Updated Dockerfiles"
# Put in place new Jenkins config
cp ../Jenkinsfile .
git add Jenkinsfile
git rm api/Jenkinsfile
git rm consumer/Jenkinsfile
git rm events/Jenkinsfile
git rm commands/Jenkinsfile
git commit -m "Add new Jenkinsfile and remove the old ones"As you can imagine, I ran the script from scratch a few times to fix bugs and mistakes. In the end, I checked the result by running the Gradle build and pushing to the repository to trigger the Jenkins build.
./gradlew buildOnce everything passed, I arranged a ten-minute code freeze with the affected team. We grabbed some coffee, ran the migration, and continued working in the new repository. After the first migration, the others were easier because most of our projects used a similar setup.
Here's the complete process in one script, along with the folder layout I used to run it.
$ tree -a my-merge-folder
my-merge-folder
├── .gitignore
├── ApiDockerfile
├── ConsumerDockerfile
├── Jenkinsfile
├── build.gradle
├── docker-compose.yml
├── gradle.proeprties
├── merge-repos.sh
└── settings.gradle
0 directories, 9 filesI've omitted the project-specific files I merged manually, but the full script is below. You'll need to supply those files for your own project.
#!/bin/bash
function move_into_subfolder () {
local subfolder=$1
if [ ! -z "$2" ] ; then
subfolder=$2
fi
pushd domain-a-$1
mkdir $subfolder
# git mv !($1) $1 # this bashism doesn't seem to work on windows git and is replaced by following 2 lines
git mv `ls -1 | grep -v $subfolder` $subfolder
git mv .gitignore $subfolder
git commit -a -m "Moving files into subfolder $subfolder"
# navigate back to previous folder
popd
}
function merge_old_repo_commits () {
git remote add $1 ../$1
git fetch $1
git merge --no-edit --allow-unrelated-histories $1/develop
git remote rm $1
}
function clone_or_pull_develop() {
if [ ! -d "${1}" ] ; then
git clone -b develop git@github.com:marcofranssen/${1}.git
else
pushd ${1}
git reset HEAD~1 --hard
git pull
popd
fi
}
# Pull the latest develop branches and move the files in a subfolder as preparation for the merge
clone_or_pull_develop "domain-a-api"
move_into_subfolder "domain-a-api" "api"
clone_or_pull_develop "domain-a-commands"
move_into_subfolder "domain-a-commands" "commands"
clone_or_pull_develop "domain-a-consumer"
move_into_subfolder "domain-a-consumer" "consumer"
clone_or_pull_develop "domain-a-events"
move_into_subfolder "domain-a-events" "events"
# Prepare a new git repo to execute the merge
rm -rf domain-a
mkdir -p domain-a
cd domain-a
git init
git remote add origin git@github.com:marcofranssen/domain-a.git
merge_old_repo_commits "commands"
merge_old_repo_commits "events"
merge_old_repo_commits "consumer"
merge_old_repo_commits "api"
#######################################################
# Below you would like to customize to your own needs #
#######################################################
# Add a new README.md to the root of the project.
touch README.md
echo '# Domain A' > README.md
echo '' >> README.md
echo '## Api' >> README.md
echo '' >> README.md
echo '[README.md](api/README.md)' >> README.md
echo '## Consumer' >> README.md
echo '' >> README.md
echo '[README.md](consumer/README.md)' >> README.md
echo '## Commands' >> README.md
echo '' >> README.md
echo '[README.md](commands/README.md)' >> README.md
echo '## Events' >> README.md
echo '' >> README.md
echo '[README.md](events/README.md)' >> README.md
git add README.md
git commit -m "Add readme to root of project, linking to the module README.md files"
# Put in place the manually merged .gitignore file and remove the old ones
cp ../.gitignore .
git add .gitignore
git rm api/.gitignore consumer/.gitignore event/.gitignore command/.gitignore
git commit -m "Add .gitignore to root of project and remove .gitignore files from subfolders"
# Put in place the manually merged gradle file and remove the old ones
gradle init
cp ../gradle* ../settings.gradle .
git add .
git update-index --chmod=+x gradlew
git commit -m "Initialize gradle project"
cp ../build.gradle .
echo "include 'api', 'consumer', 'events', 'commands'" >> settings.gradle
git add build.gradle settings.gradle
git rm -r api/build.gradle api/settings.gradle api/gradle*
git rm -r consumer/build.gradle consumer/settings.gradle consumer/gradle*
git rm -r events/build.gradle events/settings.gradle events/gradle*
git rm -r commands/build.gradle commands/settings.gradle commands/gradle*
git commit -m "Add new gradle configuration and remove the old ones"
# Add a new enhanced docker composer for better dev experience
cp ../docker-compose.yaml .
git add docker-compose.yaml
git rm api/docker-compose.yml consumer/docker-compose.yml
git commit -m "Add docker-compose file in the root of project replacing the specific module ones"
cp ../ApiDockerfile api/Dockerfile
cp ../ConsumerDockerfile consumer/Dockerfile
git add api/ consumer/
git commit -m "Update Dockerfiles"
# Put in place new Jenkins config
cp ../Jenkinsfile .
git add Jenkinsfile
git rm api/Jenkinsfile
git rm consumer/Jenkinsfile
git rm events/Jenkinsfile
git rm commands/Jenkinsfile
git commit -m "Add new Jenkinsfile and remove the old ones"
./gradlew build
git push -f -u origin master
git co -b develop
git push -f -u origin developA post on Medium.com by a former colleague inspired me. Thanks, Fred! :) Please share this article if you liked it. As always, I'd love your feedback in the comments below.
Marco Franssen
Set up your first Go project on Windows, learn basic syntax, write and run tests, and cross-compile a simple application for Linux and Raspberry Pi.
Marco Franssen
Find your Ethernet driver, add it to a Windows 10 PE image with DISM, and start Windows setup from a network share on a laptop without an optical drive.
Marco Franssen
Explore responsive and adaptive web design, progressive web apps and how I improved my blog with a static site, caching, manifests and service workers.
Marco Franssen
Follow my Raspberry Pi upgrade from Raspbian Jessie to Stretch, including backups, firmware updates, APT sources, Bluetooth audio changes, and cleanup.