Reconfigure from gitlab.com

Gitlab has many features worth exploring by configuring mainly the gitlab.rb file of gitlab and the config.toml file of gitlab-runner. Reconfiguring my gitlab instance and runner from gitlab.com helped me:

1. for each set of configuration parameters I consider using, now I can review and document my intention behind them and any manual steps needed, so opening an MR helped
2. keep track of what didn't work, how it failed and why in order to avoid repeat it, so closed unmerged MRs and their build job logs helped too!

Setup gitlab

1. One local ubuntu 18.04 VM named gitlab.prv with an omnibus installation of gitlab. Omnibus is the recommended installation method too.
2. Domain (a dynamic one set up in my router), router forwarding ports 80 and 443 to the same ports of gitlab.prv and have let's encrypt of gitlab installer generate SSL to work from browser but also when registering a runner. Otherwise, installing gitlab would be more complicated.

Configure gitlab

It is more safe to create a shell runner than to expose the ssh port of your gitlab and use the default shared runners of gitlab.com.

At gitlab.prv assuming your user is XXX, run the following. The first command prevents this issue on debian based OS and needs sudo -E to pass the env var to systemd later on.

export GITLAB_RUNNER_DISABLE_SKEL=true
wget https://s3.amazonaws.com/gitlab-runner-downloads/master/binaries/gitlab-runner-linux-amd64
chmod +x gitlab-runner-linux-amd64
mv gitlab-runner-linux-amd64 gr
./gr run

Then in another terminal run ./gr register set as coordinator https://gitlab.com, for the token check runners section at /-/settings/ci_cd page of your project at gitlab.com that holds the gitlab.rb file in the playbook.yml and the following .gitlab-ci.yml and as executor set shell.

reconfigure:
  script:
  - sudo cp gitlab.rb /etc/gitlab/gitlab.rb
  - sudo gitlab-ctl reconfigure

Then kill gr and run it as a service:

sudo -E ./gr install --user XXX --syslog --config /home/XXX/.gitlab-runner/config.toml
sudo -E ./gr start

Setup runner

The rest of this post is overengineering but it was fun :)

Reconfiguring an executor sometimes requires restarting the runner so the runner that runs the provisioner is a docker executor running ansible in a separate local VM:

1. One local ubuntu 18.04 VM named runner.prv with amd64 binary of gitlab-runner. You don't want your gitlab to be slow or go down in case your runner experience performance issues, so having it on a separate VM is recommended by gitlab too.
2. One local ubuntu 18.04 VM named ansible.lan with docker and ssh access to the runner VM to run ansible. That could be gitlab.prv by the way as it won't be heavily used.

A few oneoff configurations worth mentioning:

1. At runner.prv sudo EDITOR=vi visudo and add NOPASSWD: ALL
2. At runner.prv mkdir ~/.ssh
3. Login with ssh from ansible.lan to runner.prv once to add it to known_hosts
4. At ansible.lan install docker through snap so that it is set up with overlay2 and start on boot.

In case you wonder why I use ansible only for the runner, running reconfigure (which triggers chef) as an ansible task was not fun as it broke the build logs in gitlab.com.

Configure runner

Install the gitlab runner at ansible.lan as before but register it with type docker. Then, assuming your user is XXX, at ~/.gitlab-runner/config.toml set volumes in docker section as ["/home/XXX/.ansible/roles:/root/.ansible/roles", "/home/XXX/.ssh:/root/.ssh:ro"].

To tail logs of the systemd service: journalctl -u gitlab-runner -f

To run the runner locally assuming the repository is cloned at ~/runner even with changes that are not committed: ./../gr exec docker srv-and-runner --docker-image "spy86/ansible" --docker-volumes "${PWD}:/work:ro" --docker-volumes "/home/XXX/.ansible/roles:/root/.ansible/roles" --docker-volumes "/home/XXX/.ssh:/root/.ssh:ro"

To run it at ansible.lan: ansible-playbook playbook.yml

Files

.gitlab-ci.yml

variables:
  PYTHONUNBUFFERED: "1"
  CI_DEBUG_TRACE: "true"
  ANSIBLE_CONFIG: "ansible.cfg"

srv-and-runner:
  script:
  - ansible-playbook playbook.yml

playbook.yml. Note that once you register your runner to your gitlab instance, you have to copy your config.toml manually here

--------
- hosts: runner.prv
  name: runner
  become: yes

  tasks:
  - name: Setup
    blockinfile:
      path: /etc/gitlab-runner/config.toml
      block: |
        concurrent = 1
        check_interval = 0
        [session_server]
          session_timeout = 1900

  - name: Restart
    shell: "gitlab-runner restart"

ansible.cfg

[defaults]
inventory=inventory.ini
no_target_syslog = True
host_key_checking = False
retry_files_enabled = False
PY_COLORS = 1
ANSIBLE_FORCE_COLOR = 1

inventory.ini

[all]
runner.prv

[all:vars]
ansible_python_interpreter="/usr/bin/python3"
ansible_ssh_user="XXX"

Your turn

Now you can try that by making 2 private projects in gitlab.com, one to configure your gitlab instance and another one to configure your runner. The first one can refer to files like prometheus rules so it can configure more things if necessary.

Did you find a typo? Submit an MR and gitlab will update this post.