We can get autoscaling runners hosting the Gitlab fork from docker-machine, MinIo (for the shared cache) and the hetzner-gitlab-runner image in a server.
September 10, 2019 (~5 minutes)
Because my different projects use a lot of pipelines for tests and deployments, I wanted to use the different possibilities of the cloud to reduce the load on my GitLab server and instead run these short-lived containers on other virtual machines.
In this post, I will explain the different steps needed to install a GitLab Runner and the necessary adapter for Docker Machine in a container that can be deployed on your server, configure the runner to launch new Hetzner Cloud servers and deploy a shared cache to centralize the cache storage between theses machines. The configuration presented here is based on the official GitLab container configured and launched through Docker Compose.
First, on the Hetzner Cloud Console, create a new project and an API key for this project. You should use this project only for the automatically created runners. The key can be created under Access -> API Tokens -> New in the project page on the Hetzner Cloud Console.
Then, on my docker-compose.yml that contains the GitLab service, I added a runner service that will build a custom container with the Docker Machine driver needed to talk to the Hetzner API.
version: '2'
services:
runner:
restart: on-failure
build: ./hetzner-gitlab-runner
volumes:
- /srv/gitlab-ce/runner:/etc/gitlab-runner
gitlab:
[...]
Change the volume storing the configuration with something that is matching the volumes of your GitLab service container. Here, I will store the configuration in the /srv/gitlab-ce/runner/config.toml configuration file.
We then need to add the instructions to build our custom hetzner-gitlab-runner container image.
For this custom container, we will create a subfolder hetzner-gitlab-runner containing the Dockerfile responsible for building the container image. I took this route to facilitate matching the versions of your GitLab and GitLab runner installation.
The hetzner-gitlab-runner/Dockerfile is as follow. It uses a builder to download the 2.0.1 release of the Docker Machine driver and then copies that into the official gitlab-runner container image. You can change the gitlab-runner version here to match your deployed GitLab version.
FROM alpine:3.10 as builder
RUN apk add --no-cache --virtual .fetch-deps curl tar
WORKDIR /build
RUN curl -sLo hetzner.tar.gz https://github.com/JonasProgrammer/docker-machine-driver-hetzner/releases/download/2.0.1/docker-machine-driver-hetzner_2.0.1_linux_amd64.tar.gz
RUN tar xf hetzner.tar.gz && chmod +x docker-machine-driver-hetzner
FROM gitlab/gitlab-runner:v12.3.0
COPY --from=builder /build/docker-machine-driver-hetzner /usr/bin
In the /srv/gitlab-ce/runner/config.toml configuration, configure everything needed for Docker Machine to create new runner virtual machines.
concurrent = 2
check_interval = 0
[[runners]]
name = "docker-runner"
url = "https://git.your.domain/"
token = "your-gitlab-token"
executor = "docker+machine"
[runners.docker]
tls_verify = false
image = "maven:3-jdk-8"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
[runners.machine]
IdleCount = 0
IdleTime = 1800
MachineDriver = "hetzner"
MachineName = "gitlab-runner-%s"
MachineOptions = [
"hetzner-api-token=your-hetzner-api-key",
"hetzner-image=debian-10",
"hetzner-server-type=cx11",
"hetzner-server-location=fsn1",
]
In this file, you will have to insert the Hetzner Cloud API created in the first step and the GitLab URL and registration token that can be found in the Runner administration page on your GitLab instance.
Once these three changes are made, you can test these changes on GitLab, you should see a new server being created on the Cloud Console, the runner logs should show that a runner is executing the pipeline. For now, you should get a warning in the logs that the cache will not be downloaded from a shared cache server. This is normal as we have not yet configured the cache.
To store the cache from the different runners, we will deploy a new Minio service on our Docker Compose configuration. This will allow each cache to be stored on the main server and be accessible to each runner machine.
On my server, I deployed it behind Apache to secure the access with HTTPS, same as with the other containers running on it. You will need to adapt that configuration to your system.
First, generate two random chains of characters that will act as access and secret keys.
In the Docker Compose file, we will add a new Minio service with a volume to persist the cache data and the environement variables defining our keys :
version: '2'
services:
runner:
[...]
minio:
restart: on-failure
image: minio/minio:RELEASE.2019-09-26T19-42-35Z
ports:
- "19000:9000"
environment:
MINIO_ACCESS_KEY: 3eA4pp73u8rtsydZ
MINIO_SECRET_KEY: DRSweWE3GDmNygTa
command: server /data
volumes:
- /srv/gitlab-ce/minio-data:/data
gitlab:
[...]
As before, change the path to your volume so that the data storage matches with what you used for your GitLab install. Adapt also the exported port. Here I used 19000 because that’s my convention, use whatever matches your install.
For reference, the configuration for Apache is the following. You will need to adapt for your domain, your SSL key and certificate and the port defined in your Docker Compose file.
<VirtualHost *:443>
ServerName s3.your.domain
SSLEngine On
ProxyPreserveHost On
ProxyRequests Off
ProxyVia Block
ProxyPass / http://localhost:19000/
ProxyPassReverse / http://localhost:19000/
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/your.domain/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/your.domain/privkey.pem
</VirtualHost>
Once Apache is configured, you should be able to access s3.your.domain with your browser and login using the access and secret keys defined before.
In the Minio web interface, create a new runner bucket by clicking on the orange plus on the bottom right of the screen. This will be needed in the next steps.
In the previous section about the GitLab Runner config.toml configuration, we left the [runners.cache] section empty. We will now configure the Runner to use the newly created Minio S3 storage as a shared cache.
[runners.cache]
Type = "s3"
Path = "cache"
Shared = false
[runners.cache.s3]
ServerAddress = "s3.your.domain"
AccessKey = "3eA4pp73u8rtsydZ"
SecretKey = "DRSweWE3GDmNygTa"
BucketName = "runner"
Insecure = false
Don’t forget to update the AccessKey, the SecretKey and your Address to match what was configured in the previous section.
Once the configuration is updated, you can stop the gitlab_runner_1 container and restart it using Docker Compose to force the GitLab Runner to refresh its configuration.
Launching a new pipeline, you should now have a cache available. You can verify that the runner uploads it after the run by checking the content of the bucket after a successful execution of a job.
Copyright © Thomas Schwery, No rights reserved (CC0).