The Git Iterator giit is a small tool for running commands on
branches and tags of a git repository.
It's original purpose was to allow Sphinx documentation to be easily generated for all available tags of a bunch of different repositories. However, should you find a different use for it - you should also be able to adapt it to other scenarios.
Table of Contents:
To use giit you define a giit.json file which contains the steps
you want giit to be able to run. Note, the giit.json file can
live in the root of the repository.
Let's say we want to generate the Sphinx documentation for a specific repository.
giit is a Python package so you can pip install it. If you just want to
try it out use a virtualenv or similar:
$ virtualenv giit $ source giit/bin/activate
Now install the giit package:
$ pip install giit
giit uses a giit.json file to describe the different steps:
{
"docs": {
"branches.regex.filters": [
"origin/master"
],
"scripts": [
"sphinx-build -b html . ${build_path}"
],
"cwd": "${source_path}/docs",
"requirements": "${source_path}/docs/requirements.txt"
}
}
Lets build the endian Sphinx documentation
(https://steinwurf-endian.netlify.app/latest/) by running giit:
giit docs https://github.com/steinwurf/endian.git --config_path ./giit.json
You should now seem something like:
Lets go: docs Building into: /tmp/giit/data/build/endian-30a816 Using git version: 2.34.1 Using git repository: https://github.com/steinwurf/endian.git Running: git clone into /tmp/giit/data/clones/endian-30a816 Using giit.json from path /tmp/giit_cwd/giit.json Tasks generated 1 Running task [1/1]: scope 'branch' name 'master' checkout 'origin/master' Python: sphinx-build -b html . /tmp/giit/data/build/endian-30a816
If you visit /tmp/giit/data/build/endian-30a816 with your web browser
you should be able to see the endian Sphinx documentation.
Since the content of the giit.json file fully determines the steps
taken by giit understanding how the giit.json file is found is
quite important.
The following outlines the rules:
Passing a path using
--config_pathor passing a branch using--config_branch.If no config path or branch is passed by the user and
giitis invoked with an URL (like in theendianexample).Example:
giit docs https://github.com/endian/endian.git
In this case
giitwill look at the root of the repository inorigin/masterbranch for agiit.json.If no config path or branch is passed by the user and
giitis invoked with a path:git docs ../path/to/repo
In this case
giitwill try to find agiit.jsonat../path/to/repo/giit.json.
As we saw in the endian example a single task is generated for building
the origin/master branch. We can generate more tasks by setting up more
filters.
As a quick note it is also possible to not specify any filters. In that case a single task for running the specified scripts will be gererated (with a limited context - see below).
To specify the different filters here are the available options:
This is a list of regular expressions that will be matched against the branch name. If the regular expression matches a task will be generated.
For example (in giit.json):
"branches.regex.filters": [
"origin/master",
"(\d+\.\d+.\d+)-LTS"
]
When invoking giit with a path to a repository e.g.:
giit docs ../path/repo. giit can be instructed to build the
remote tracking branch currently checkout out in ../path/repo.
This is useful in continuous integration systems.
For example (in giit.json):
"branches.source_branch": true
This is a list of regular expressions that will be matched against the tag name. If the regular expression matches a task will be generated.
For example (in giit.json):
"tags.regex.filters": [
"(\d+\.\d+.\d+)"
]
If a project uses sematic versioning the semver filter can be used.
For example (in giit.json):
"tags.semver.filters": [
">=0.1.1", "<0.3.0"
]
We use https://python-semanticversion.readthedocs.io/en/latest/ you can find more examples of requirement specifications there.
If a project uses "kind-of" semver, such as 1.20, you can set the
semver filter in relaxed mode and still use the filters.
For example (in giit.json):
"tags.semver.relaxed": true
The workingtree filter is useful for quickly iterating on stuff.
It is similar to the source_branch filter. In that if giit is
invoked with a path, then that path will be the workingtree this
allows you to run giit without commit'ing pushing changes.
For example (in giit.json):
"workingtree": true
If you pass no filter e.g. tags, branches or workingtree, giit
will generate a single task for just running the script.
Per default giit will run all tasks generated by the config file. By specifying
the task_filter argument you can filter which tasks are run.
Any tasks that doesn't match the given shell-style wildcard
expression will not be run.
The special characters used in shell-style wildcards are:
| Pattern | Meaning |
|---|---|
* |
matches everything |
? |
matches any single character |
[seq] |
matches any character in seq |
[!seq] |
matches any character not in seq |
In the endian example you may have noticed what we used the
${build_path} and ${source_path} in the json configuration.
These denote variables that will be substituted when running the tasks. The following variables are always available:
build_path: This points to the directory where the command is expected to output any artifacts produced by the command. It is up to thegiit.jsonauthor to ensure this happens.source_path: This is the path to where the current git repository is checked out.checkout: This is the checkout of that was used.name: This is a shorter version of checkout. E.g. for branches if the checkout isorigin/masterthe name will bemaster. Also if thecheckoutcontains/that may result in unwanted sub-directories. In thenamewe replace/with_. So if a branch is calledorigin/bug/543the name will bebug_543.scope: This can be one of three values. Eithertag,branchorworkingtree.
Note, only the ${build_path} variable is available when running without
any filters.
Here we will use the ${name} variable to output documentation
for the different tags to different folders:
{
"docs": {
"branches.regex.filters": [
"origin/master"
],
"tags.semver.filters": [
">=1.20"
],
"tags.semver.relaxed": true,
"scripts": [
"sphinx-build -b html . ${build_path}/${name}"
],
"python_path": "${source_path}/src",
"cwd": "${source_path}/docs",
"requirements": "${source_path}/docs/requirements.txt"
}
}
In some cases we want to define our own variables according to some simple rules.
This is done either using the variables attribute in the json or by using
the --variable [name] [value] command line argument.
User variables are define using the following syntax:
scope:remote_branch:variable_name
Where scope and remote_branch are optional.
This can be used to customize e.g. the output of a command. Consider the following example:
{
"docs": {
...
"scripts": [
"sphinx-build -b html . ${output_path}"
],
"variables": {
"branch:origin/master:output_path": "${build_path}/docs/latest",
"branch:output_path": "${build_path}/sphinx/${name}",
"tag:output_path": "${build_path}/docs/${name}",
"workingtree:output_path": "${build_path}/workingtree/sphinx"
}
}
}
When calling giit docs ... we use the user defined output_path
variable.
Let walk though the different values output_path can take.
- If scope is
branchand the branch isorigin/masterthenoutput_pathwill be${build_path}/docs/latest. - For all other branches
output_pathwill be${build_path}/sphinx/${name}where${name}will be the branch name. - For the tags
output_pathwill be${build_path}/docs/${name}where name is the tag value e.g.1.0.0etc. - Finally if we are in the
workingtreescope theoutput_pathvariable will be${build_path}/workingtree/sphinx
Lets see how this could look (build_path is /tmp/project):
Tag 1.0.0 -----------> /tmp/project/docs/1.0.0 Tag 2.0.0 -----------> /tmp/project/docs/2.0.0 Tag 2.1.0 -----------> /tmp/project/docs/2.1.0 Tag 3.0.0 -----------> /tmp/project/docs/3.0.0 Branch master -------> /tmp/project/docs/latest Branch trying_new ---> /tmp/project/sphinx/trying_new Branch new_idea -----> /tmp/project/sphinx/new_idea Workingtree ---------> /tmp/project/workingtree
In some cases you may want to have optional variables. These can be specified
in a similar way as with non optional variables, the only difference is that you
need to use the £ character instead of the $ character.
If the variable doesn't exists it simply be removed.
If you want to use either $ or £ as characters in the giit configuration
file, you need to escape them.
This is done using $$ or ££ respectively.
The giit tool takes two mandatory arguments and a number of options:
giit STEP REPOSITORY [--options]
Selects the step in the giit.json file to run.
The URL or path to the git repository.
Sets the build path (i.e. where the output artifacts/data) will be generated/
built. This argument is available in the giit.json as the ${build_path}
variable.
This path is where the giit tool will store configurations, virtualenvs
clones created while running the tool. It also serves as a cache, to speed up
builds.
Specifies the a branch where the giit.json file will be take from.
Sets the path to where the giit.json file.
Extends the variables set for each step.
Allows the verbosity level of the tool to be increased generating more debug information on the command line.
This step is always defined, in addition to the steps defined in
the giit.json file. The clean step just remove the
build_path.