As someone who frequently works with both GitHub and GitLab repositories, I often find the need to get an overview of various projects that exist within an organization or group. At my workplace, we use GitLab’s group- and subgroup functionality to organize our team-specific and customer-specific structures.
Recently, I needed to clone all 27 repositories associated with one of our microservice-projects (where else would you find so many repositories?). Doing this the traditional way by hand would require me to copy the URL for each repository and run “git clone \$URL” on the command line. This is a time-consuming and tedious process I want to avoid.
Fortunately,my habit for automating things lead me to discovering a tool called ghorg that simplifies this process.
With just a single command, I was able to clone entire GitHub organizations or GitLab (sub-)groups. Below is the command I used to do this:
ghorg clone customergroup/services --base-url=https://git.example.com --scm=gitlab --token=$TOKEN --preserve-dir -p .
This command clones all repositories in the group customergroup/services, connecting to our private GitLab instance, using the specified token $TOKEN to authenticate. The --preserve-dir option clones the repos in a directory structure that matches GitLab groups, so repository foo will be cloned into customergroup/services/foo.
ghorg clone customergroup/services --base-url=https://git.example.com --scm=gitlab --token=$TOKEN --preserve-dir -p .
+-+-+-+-+ +-+-+ +-+-+-+-+-+
|T|I|M|E| |T|O| |G|H|O|R|G|
+-+-+-+-+ +-+-+ +-+-+-+-+-+
*************************************
* SCM : gitlab
* Type : org
* Protocol : https
* Location : ./
* Concurrency : 25
* Base URL : https://git.example.com
* Preserve Dir : true
* Config Used : none
* Ghorg version : v1.9.3
*************************************
27 repos found in customergroup/services
Success cloning repo: https://git.example.com/customergroup/services/foo.git -> branch: main
Success cloning repo: https://git.example.com/customergroup/services/foo-audit-report.git -> branch: master
Success cloning repo: https://git.example.com/customergroup/services/foo-tools.git -> branch: main
Success cloning repo: https://git.example.com/customergroup/services/foo-base.git -> branch: main
Success cloning repo: https://git.example.com/customergroup/services/foo-pages.git -> branch: main
Success cloning repo: https://git.example.com/customergroup/services/foo-helm.git -> branch: master
Success cloning repo: https://git.example.com/customergroup/services/foo-application-base.git -> branch: master
Interestingly, ghorg is not limited to GitLab or GitHub; it also offers support for other repository providers such as Bitbucket.
But I’m also a great fan of the gitlab-cli and it has a similar feature, too! Albeit the Gitlab-cli supports only Gitlab, naturally.
The same batch cloning process can be performed through the gitlab-cli tool with this command:
GITLAB_HOST=git.example.com glab repo clone --group customergroup/services -p
The syntax deviates slightly from the one used in “ghorg”. With --group $GROUP you define the particular group you want to clone and with the -p option you preserve the group-structure.
> GITLAB_HOST=git.example.com glab repo clone --group customergroup/services -p
Cloning into 'customergroup/services/foo'...
remote: Enumerating objects: 59, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 59 (delta 0), reused 0 (delta 0), pack-reused 54
Receiving objects: 100% (59/59), 13.36 KiB | 1.03 MiB/s, done.
Subsequently, both tools, ghorg and gitlab-cli, produced the same directory structure for the cloned repositories when tested, thus proving equivalently efficient.
> tree -d -L 3
.
└── customergroup
└── services
├── foo
├── foo-audit-report
├── foo-tools
├── foo-base
├── foo-pages
├── foo-helm
└── foo-application-base
Many other tools offer similar batch cloning features, like myrepos, vcsh or gitlabber, but I haven’t explored these alternatives yet. If anyone has experience with one of these tools, tell me! I’d love to hear about it.