We can get autoscaling runners hosting the Gitlab fork from
MinIo (for the shared cache) and the
hetzner-gitlab-runner image in a server.
September 10, 2019 (~5 minutes)
Because my different projects use a lot of pipelines for tests and deployments, I wanted to use the different possibilities of the cloud to reduce the load on my GitLab server and instead run these short-lived containers on other virtual machines.
In this post, I will explain the different steps needed to install a GitLab Runner and the necessary adapter for Docker Machine in a container that can be deployed on your server, configure the runner to launch new Hetzner Cloud servers and deploy a shared cache to centralize the cache storage between theses machines. The configuration presented here is based on the official GitLab container configured and launched through Docker Compose.
First, on the Hetzner Cloud Console, create a new project and an API key for this project. You should use this project only for the automatically created runners. The key can be created under Access -> API Tokens -> New in the project page on the Hetzner Cloud Console.
Then, on my
docker-compose.yml that contains the GitLab service, I added a
runner service that will build a custom container with the Docker Machine driver needed to talk to the Hetzner API.
version: '2' services: runner: restart: on-failure build: ./hetzner-gitlab-runner volumes: - /srv/gitlab-ce/runner:/etc/gitlab-runner gitlab: [...]
Change the volume storing the configuration with something that is matching the volumes of your GitLab service container. Here, I will store the configuration in the
/srv/gitlab-ce/runner/config.toml configuration file.
We then need to add the instructions to build our custom hetzner-gitlab-runner container image.
For this custom container, we will create a subfolder
hetzner-gitlab-runner containing the
Dockerfile responsible for building the container image. I took this route to facilitate matching the versions of your GitLab and GitLab runner installation.
hetzner-gitlab-runner/Dockerfile is as follow. It uses a builder to download the 2.0.1 release of the Docker Machine driver and then copies that into the official gitlab-runner container image. You can change the gitlab-runner version here to match your deployed GitLab version.
FROM alpine:3.10 as builder RUN apk add --no-cache --virtual .fetch-deps curl tar WORKDIR /build RUN curl -sLo hetzner.tar.gz https://github.com/JonasProgrammer/docker-machine-driver-hetzner/releases/download/2.0.1/docker-machine-driver-hetzner_2.0.1_linux_amd64.tar.gz RUN tar xf hetzner.tar.gz && chmod +x docker-machine-driver-hetzner FROM gitlab/gitlab-runner:v12.3.0 COPY --from=builder /build/docker-machine-driver-hetzner /usr/bin
/srv/gitlab-ce/runner/config.toml configuration, configure everything needed for Docker Machine to create new runner virtual machines.
concurrent = 2 check_interval = 0 [[runners]] name = "docker-runner" url = "https://git.your.domain/" token = "your-gitlab-token" executor = "docker+machine" [runners.docker] tls_verify = false image = "maven:3-jdk-8" privileged = false disable_cache = false volumes = ["/cache"] shm_size = 0 [runners.cache] [runners.machine] IdleCount = 0 IdleTime = 1800 MachineDriver = "hetzner" MachineName = "gitlab-runner-%s" MachineOptions = [ "hetzner-api-token=your-hetzner-api-key", "hetzner-image=debian-10", "hetzner-server-type=cx11", "hetzner-server-location=fsn1", ]
In this file, you will have to insert the Hetzner Cloud API created in the first step and the GitLab URL and registration token that can be found in the Runner administration page on your GitLab instance.
Once these three changes are made, you can test these changes on GitLab, you should see a new server being created on the Cloud Console, the runner logs should show that a runner is executing the pipeline. For now, you should get a warning in the logs that the cache will not be downloaded from a shared cache server. This is normal as we have not yet configured the cache.
To store the cache from the different runners, we will deploy a new Minio service on our Docker Compose configuration. This will allow each cache to be stored on the main server and be accessible to each runner machine.
On my server, I deployed it behind Apache to secure the access with HTTPS, same as with the other containers running on it. You will need to adapt that configuration to your system.
First, generate two random chains of characters that will act as access and secret keys.
In the Docker Compose file, we will add a new Minio service with a volume to persist the cache data and the environement variables defining our keys :
version: '2' services: runner: [...] minio: restart: on-failure image: minio/minio:RELEASE.2019-09-26T19-42-35Z ports: - "19000:9000" environment: MINIO_ACCESS_KEY: 3eA4pp73u8rtsydZ MINIO_SECRET_KEY: DRSweWE3GDmNygTa command: server /data volumes: - /srv/gitlab-ce/minio-data:/data gitlab: [...]
As before, change the path to your volume so that the data storage matches with what you used for your GitLab install. Adapt also the exported port. Here I used 19000 because that’s my convention, use whatever matches your install.
For reference, the configuration for Apache is the following. You will need to adapt for your domain, your SSL key and certificate and the port defined in your Docker Compose file.
<VirtualHost *:443> ServerName s3.your.domain SSLEngine On ProxyPreserveHost On ProxyRequests Off ProxyVia Block ProxyPass / http://localhost:19000/ ProxyPassReverse / http://localhost:19000/ Include /etc/letsencrypt/options-ssl-apache.conf SSLCertificateFile /etc/letsencrypt/live/your.domain/fullchain.pem SSLCertificateKeyFile /etc/letsencrypt/live/your.domain/privkey.pem </VirtualHost>
Once Apache is configured, you should be able to access s3.your.domain with your browser and login using the access and secret keys defined before.
In the Minio web interface, create a new
runner bucket by clicking on the orange plus on the bottom right of the screen. This will be needed in the next steps.
In the previous section about the GitLab Runner
config.toml configuration, we left the
[runners.cache] section empty. We will now configure the Runner to use the newly created Minio S3 storage as a shared cache.
[runners.cache] Type = "s3" Path = "cache" Shared = false [runners.cache.s3] ServerAddress = "s3.your.domain" AccessKey = "3eA4pp73u8rtsydZ" SecretKey = "DRSweWE3GDmNygTa" BucketName = "runner" Insecure = false
Don’t forget to update the AccessKey, the SecretKey and your Address to match what was configured in the previous section.
Once the configuration is updated, you can stop the
gitlab_runner_1 container and restart it using Docker Compose to force the GitLab Runner to refresh its configuration.
Launching a new pipeline, you should now have a cache available. You can verify that the runner uploads it after the run by checking the content of the bucket after a successful execution of a job.
Copyright © Thomas Schwery, No rights reserved (CC0).