Merge multiple gradle projects including git history
Marco Franssen /
10 min read • 1952 words
As we started 2 years ago with a micro-service architecture we faced some issues in the productivity of the teams a while ago. This made me think what we could do about that from an architecture and development point of view. As we used different Git repos for the various components of our micro services in the long run it became harder and harder for developers to work with those components. At that point in time I decided to simplify the development workflow to merge the different Git repos of one functional bounded context into one Git repository.
Finally I found some time to document this process in the form of a blogpost for my own reference, but also to share this with the community. In case you are thinking, I'm not using Gradle and are at the point to quite reading, please don't! Only last bit goes Gradle specific. The first 75% you can reuse for whatever kind of project you are running.
To keep impact for development team low to non-existing I wrote a migration script in bash to be able to do the migration within a couple of minutes and do some dry runs to verify if everything was still working. By everything I mean the following parts:
Do the projects still compile?
Do the unit and integration test still succeed?
Does the continuous integration still work on Jenkins?
Does the continuous deployment still work on Kubernetes?
Lets start with a folder structure example which represents an similar situation we faced. In every folder we had a Git repository containing the Gradle project including a Jenkinsfile and some other common files for CI and CD purposes.
I'll zoom in for the remainder of this article on domain-a. The goal is to end up with a structure like the following.
The first step is to move all files in the existing repositories into a subfolder. For that I wrote a bash function to be reused in all the projects I had to migrate.
This function can be used as following, move_into_subfolder "domain-a-api" "api", which will result in all files within the folder domain-a-api to be moved into domain-a-api/api. The move of files will also be committed in the existing repository.
As I was doing this for many repos and I had to make sure I was merging the latest develop branches of these repositories I also wrote a small bash function to clone or pull the latest develop branches from the repositories I was about to merge.
This function clones get's the latest changes from the develop branch and removes the last move-into-subfolder commit so you can run the function over and over again for testing if all comes allong nicely.
Now we have a way to have all the repositories prepared to be merged as one repository without loosing the commit history. Also for this action I wrote a small bash function to be easily reused on the various repositories I wanted to merge.
To start we will have to prepare a new git repository first and then I can call my new bash function to merge the repositories in one.
The result is we got all the 4 separate repository commit histories merged together in this single repository. Last but not least we will do a few additional commits on top of the merged repositories to make it a fully working multi-module gradle repository. This is probably where you would to go your own route to do your project specific modifications. As a reference I will show you below what I did.
First off all I made a new commit adding a new README.md file in the root of the project which links to the README.md files in my subprojects.
Then I made a commit where I add a new .gitignore file which I manualy merged and remove the original .gitignore files.
Same I did for my Gradle files so I have one file to manage the Gradle build from the root of the new project folder.
Last but not least I also put in place new docker-compose setup and remove the old files. And put in place my updated Jenkinsfile which I also merged manually.
As you can imagine I had to run my script of few times from scratch to fix the bugs and mistakes. So in the end I tested all by running the gradle build and pushing it to my repo to trigger the Jenkins build.
Once all succeeded I planned with the impacted team a 10 minute code freeze (grabbed some coffee) and then we continued our work on the new repository.
Once the first one finished the other ones where more easy as most our projects use a similar setup.
Here the whole summary in one script. Including my folder layout from where I executed this script.
I omit all my project specific files that I merged manually, but will provide you with the full script below. It is up to you to put all together.
I got inspired by an nice post on Medium.com from an ex colleague of mine. Thanks Fred! :) Please share this article if you liked it and as always I would love your feedback in the comments below.
You have disabled cookies. To leave me a comment please allow cookies at functionality level.
A couple of months ago I started to do some coding in Go a.k.a Golang. Not only because they have an awesome logo ;-). My main reason was because I wanted to have something running as bare metal as possible on my Raspberry Pi and I wanted to have it available for different platforms to be easy to install. Some other reasons are the ease of creating async code by using Go in front of your methods and the unique approach of channels to sync between go routines (threads). I have been reading a lot…
Very recently I have been trying to reinstall my Laptop using my WinPE approach as it didn't have a optical drive anymore. However my problem was that the WinPE image I created was lacking the network driver for my laptop. So then I recreated a WinPE USB drive using the Windows 10 ADK, hoping it would include the required driver. However unlucky me I still had no network when I booted by new laptop using my new Windows 10 PE USB drive. Therefore I had to add the network driver for my laptop to t…
In the last couple of years web applications technologies and frameworks went through a fast paced transformation and evolution. For all these evolutions there was coined a marketing term which (by coincidence) all end on …ive. So lets coin another one. In this article I'm going to explain you the basic concepts of all these principles which combined allow you to build an impressive web application.
Responsive Web Design
It started all back in (from top of my head) late 2010, with the idea of…
Very recently I have upgraded my Raspberry 3 to the new Raspbian OS, code named "Stretch". Due to some security issues in the chipset of the Raspberry Pi 3 and Raspberry zero, I decided to upgrade mine to Raspbian Stretch, which resolves these security issues.
Before you get yourself into any trouble make sure to have a backup of any important data and also please note I can't be responsible for any data loss. In this guide I also assume you know how to connect to your raspberry using ssh and h…