After some recently release PoC of Script Code we had the challenge to search through every repository and branch of our Gitlab Instance. For technical Background – there is no Elasticsearch or similar backend linked to this repository. Because the Security Issue was quite dangerous we need a short-mid term solution to achieve a result for now and later.
For the Short Term Resolution I set up a vanilla Linux Server with Access to Gitlab. After some research I found a script which handles to Download all Repositories on Gitlab ( Original here: https://github.com/angristan/gitlab-repo-dl , fixed Repo from me: https://github.com/packhack/gitlab-repo-dl ).
So I downloaded the script, installed via
dnf install jq -y
the requirements and started the Commands:
mkdir ~/git
export GITLAB_URL=https://your.instance.com
export GITLAB_TOKEN=your_token_from_gitlab_with_API_and_READ-Repository_rights
./gitlab-repo-dl all-repo-list > list.txt
./gitlab-repo-dl from-list list.txt git/
After this step which took some time (get ready for some coffee) the next challenge waited – how to search ALL branches for this specific piece of code?
Git offers git grep (check here: https://git-scm.com/docs/git-grep) but it actually search only in the current checked out branch. But usually you are faced with more than one main branch. So how can you check every branch which exists after some piece of code?
The command git branch -a lists all branches, remotely and locally. So you could combine with git grep “text” branchname a search through all branches. So now we need to find a way to handle all branches. First steps, remove the pre-characters for a list:
sh: git branch -a
develop
remotes/origin/HEAD -> origin/develop
remotes/origin/bugfix/link_to_comment
remotes/origin/bugfix/greensock-update
remotes/origin/bugfix/secondary_colors_adjustment
remotes/origin/bugfix/change-svg-image-paths
remotes/origin/bugfix/interactive_maps_aem_integration_fixes
remotes/origin/develop
So we have to remove the trailing spaces/characters to get to the Branch Name:
sh: cut –help
Usage: cut OPTION… [FILE]…
Print selected parts of lines from each FILE to standard output.
With no FILE, or when FILE is -, read standard input.
Mandatory arguments to long options are mandatory for short options too.
-b, --bytes=LIST select only these bytes
-c, --characters=LIST select only these characters
-d, --delimiter=DELIM use DELIM instead of TAB for field delimiter
-f, --fields=LIST select only these fields; also print any line
that contains no delimiter character, unless
the -s option is specified
-n with -b: don't split multibyte characters
--complement complement the set of selected bytes, characters
or fields
-s, --only-delimited do not print lines not containing delimiters
--output-delimiter=STRING use STRING as the output delimiter
the default is to use the input delimiter
-z, --zero-terminated line delimiter is NUL, not newline
--help display this help and exit
--version output version information and exit
Use one, and only one of -b, -c or -f. Each LIST is made up of one
range, or many ranges separated by commas. Selected input is written
in the same order that it is read, and is written exactly once.
Each range is one of:
N N'th byte, character or field, counted from 1
N- from N'th byte, character or field, to end of line
N-M from N'th to M'th (included) byte, character or field
-M from first to M'th (included) byte, character or field
GNU coreutils online help: https://www.gnu.org/software/coreutils/
Full documentation at: https://www.gnu.org/software/coreutils/cut
or available locally via: info '(coreutils) cut invocation'
The interesting option is the -c Command. So we count and get 3 Chars have to be removed.
sh: git branch -a | cut -c3-
develop
remotes/origin/HEAD -> origin/develop
remotes/origin/bugfix/link_to_comment
remotes/origin/bugfix/greensock-update
remotes/origin/bugfix/secondary_colors_adjustment
remotes/origin/bugfix/change-svg-image-paths
remotes/origin/bugfix/interactive_maps_aem_integration_fixes
remotes/origin/develop
Close but we have still the -> origin/develop at one branch. So we could split with a delimiter – the best would be a single space.
So from the help, -d is what we need.
For the Nature that cut can only split with one Parameter another pipe is needed:
sh: git branch -a | cut -c3- | cut -d' ' -f1
develop
remotes/origin/HEAD
remotes/origin/bugfix/link_to_comment
remotes/origin/bugfix/greensock-update
remotes/origin/bugfix/secondary_colors_adjustment
remotes/origin/bugfix/change-svg-image-paths
remotes/origin/bugfix/interactive_maps_aem_integration_fixes
remotes/origin/develop
Now a nice list of all branches is available. But what now? How can we give that to the grep command. I’m a fan of nice one-liners so a for loop is not my favourite thing to do. xargs is the tool to take out this list.
Let’s go:
sh: git branch -a | cut -c3- | cut -d' ' -f1 | xargs git grep "test"
remotes/origin/master:.gitignore:# testing
Success. Now we have one Repository but what happens to the other x Repos? They have to be scanned too.
From former actions it’s hard to find valid repositories and you have to fiddle through group and reponames, subgroup etc.
A nice way to find these settings is to search for .git within the folders like this:
sh: find git/ -iname .git
git/example-group/example-repo/.git
git/antoher-example/one-repo/.git
Several directories are found, but the git grep command won’t work in these directories. The util cut will be our friend again.
sh: find git/ -iname .git | cut -d. -f1
git/example-group/example-repo/
git/another-example/one-repo/
Looks better. The final step is now to combine everything in a fine one-line bash magic. Let’s go
sh: for i in find git/ -iname .git | cut -d. -f1; do cd ${i}; echo "Working on ${i}"; git branch -a | cut -c3- | cut -d' ' -f 1 | xargs git grep -i "test" | tee -a ~/output.txt; cd ~; done
Working on git/example-group/example-repo/
remotes/origin/master:.gitignore:# testing
Working on git/another-example/one-repo/
remotes/bugix/typo_fix:src/main.java:# function_test()
The output file will be helpful for later searching in the findings.
Don’t forget to let it run in a screen or tmux session if something fails.