/assets/img/2020-03-10_gitlab-runner_gcp-console-640.png

Autoscaling GitLab Runner Instances on Google Cloud Platform

Contents

Date
09.03.2021
Reading time
7 Minutes
Comments
0 comments
Tags

tl;dr: Migrating GitLab CI jobs to Google Cloud Platform is possible with little effort due to the good support provided by GitLab and relieves the load off your hardware.
This can be worthwhile even for small projects or private GitLab instances without generating major costs.

Contents

Currently, most of my CI jobs run either with GitHub Actions1 or on a dedicated Jenkins2 server.

Since Jenkins requires a lot of resources, I am currently trying to move some of the jobs to external servers. I already use GitLab3, so I want to utilize GitLab CI/CD4 in a public cloud for this.

Most of the jobs only run for a relatively short time, but require some power during that time, so a cloud provider that charges by the minute would be a good choice.

The goal of the setup is to have a lightweight GitLab Runner5 running on the Gitlab hardware, which takes CI/CD jobs, creates VMs for them in the Google Cloud, and terminates them after execution. If no jobs are currently running, no costs are incurred for computing power in this way.

My choice fell on Google Cloud Platform6 and Google Cloud Storage7 (used for caching), the necessary steps for a successful setup are discussed here.

Google Cloud Platform Project

First, we create a project, a service user, and a storage bucket for the cache in Google Cloud Platform.

If you haven’t registered with Google Cloud Platform yet, you get $300 budget for the first year. Only once this budget is used up will there be any real costs.

After the successful registration / login, we can now create a new project with a speaking name.

/assets/img/2020-03-10_gitlab-runner_gcp-project_06-640.png
/assets/img/2020-03-10_gitlab-runner_gcp-project_06-640.png

Within the project we now create a new service account under IAM & Admin > Service Accounts. This account needs the following permissions to start instances later and to access the cache:

  • Compute Instance Admin (v1).
  • Service Account User
  • Storage Object Admin

Depending on how many projects are running in your GCP, the permissions can be further refined or accepted as is for now.

The access data for the ServiceAccount can then be created under KEYS > Add Key > Create new key and downloaded as JSON.

/assets/img/2020-03-10_gitlab-runner_gcp-project_04-640.png
/assets/img/2020-03-10_gitlab-runner_gcp-project_04-640.png
Finally, we create a Google Storage Bucket, here it is advantageous to select the region identical to the one configured later in the GitLab Runner.

Now that the configuration in GCP is complete, we install the GitLab Runner and configure it accordingly.

GitLab Runner - Docker Machine

The VMs started in the GCP must therefore themselves support Docker. Ideally, one uses an appropriate image for this, Google recommended CoreOS8 for a long time, but this is deprecated and should be replaced by Container-Optimized OS9.

Unfortunately, the Docker Machine Version10 used by GitLab Runner is also no longer actively maintained, so there is an issue here with Google’s Container-Optimized OS images. A workaround is to use a “normal” Linux image like debian-10 and then install Docker in it.

But the better solution is to replace Docker machine with the GitLab fork11, which also copes with Container-Optimized OS.

Installation

The GitLab Runner can either be installed directly on the system or started via Docker.

For simplicity, we’ll use Docker here and build a customized image that includes the fork version of Docker-Machine. The configuration for the GitLab Runner lies in the ./config-gcp folder. We need to copy the GCP client_secret.json file created earlier here too.

docker-compose.yml:

version: "3.8"
services:
  gitlab-runner-gcp:    
    build: .
    volumes:
      - "./config-gcp:/etc/gitlab-runner"
    environment:      
      - "GOOGLE_APPLICATION_CREDENTIALS=/etc/gitlab-runner/client_secret.json"

Dockerfile:

FROM gitlab/gitlab-runner:latest
RUN wget -q https://gitlab-docker-machine-downloads.s3.amazonaws.com/v0.16.2-gitlab.11/docker-machine-Linux-x86_64 -O /usr/bin/docker-machine && \
    chmod +x /usr/bin/docker-machine

We can now start the GitLab Runner with docker-compose up, don’t worry if it complains about invalid configurations, we’ll come to this next.

Creating gitlab-runner-test_gitlab-runner-gcp_1 ... done
Attaching to gitlab-runner-test_gitlab-runner-gcp_1
gitlab-runner-gcp_1  | Runtime platform                                    arch=amd64 os=linux pid=8 revision=2ebc4dc4 version=13.9.0
gitlab-runner-gcp_1  | Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0
gitlab-runner-gcp_1  | Running in system-mode.
gitlab-runner-gcp_1  |
gitlab-runner-gcp_1  | Configuration loaded                                builds=0
gitlab-runner-gcp_1  | listen_address not defined, metrics & debug endpoints disabled  builds=0
gitlab-runner-gcp_1  | [session_server].listen_address not defined, session endpoints disabled  builds=0
gitlab-runner-gcp_1  | ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory  builds=0
gitlab-runner-gcp_1  | ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory  builds=0

Register Runner

Each GitLab Runner must register with the GitLab instance. For this purpose there is the gitlab-runner register command. We can find the required credentials in the GitLab instance under Admin Area > Overview > Runners.

root@docker:/opt/gitlab-runner# docker exec -it gitlab-runner-test_gitlab-runner-gcp_1 gitlab-runner register
Runtime platform                          arch=amd64 os=linux pid=19 revision=2ebc4dc4 version=13.9.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
https://YOUR_GITLAB_INSTANCE/
Enter the registration token:
GITLAB_RUNNER_REGISTRATION_TOKEN
Enter a description for the runner:
[4847c6296d8f]: gitlab-runner-test
Enter tags for the runner (comma-separated):                                                                                                                                         
OPTIONAL_TAGS  
Registering runner... succeeded                     runner=XYZABCDE    
Enter an executor: custom, docker, parallels, virtualbox, docker+machine, docker-ssh, shell, ssh, docker-ssh+machine, kubernetes:
docker+machine
Enter the default Docker image (for example, ruby:2.6):
alpine:latest                                                                                                                                    
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

GitLab Runner now creates config-gcp/config.toml file based on the input, which we need to customize next.

Configuration - General

# Spawn up to 5 machines
concurrent = 5
check_interval = 0

[session_server]
  session_timeout = 1800
  
[[runners]]
  name = "gitlab-runner-gce"
  url = "https://YOUR_GITLAB_INSTANCE/"
  token = "SOME_TOKEN"
  executor = "docker+machine"
  limit = 5
  
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

Configuration - Google Cloud Platform

First you need to decide, which machine types you want to create in GCP, in most cases one of the E2 instance types should fit perfect. I use e2-highcpu-4 which has 4 vCPUs and 4GB Memory and is quite cheap12.

I also recommend enabling the google-preemptible flag in your config. This reduces the costs for an instance dramatically (usually the prementible instance costs ~33% of the normal instance) with the disadvantage, that the instance might be terminated by Google. In this (rather rare) case the job fails, and you need to retry it.

You can have a look at the complete documentation on the GitLab Runner configuration13 and adjust the settings for your scenario.

Make sure to set the google-zone to the same zone as your Storage Bucket, otherwise you might increase the latency.

  [runners.machine]
    IdleCount = 0
    IdleTime = 30
    MachineDriver = "google"
    MachineName = "auto-scale-runner-%s"
    MachineOptions = [
      "google-project=GOOGLE_CLOUD_PROJECT",
      # Depending on your requirements, choose another instance
      "google-machine-type=e2-highcpu-4",
      # When running the forked docker-machine, you should use cos-stable
      "google-machine-image=cos-cloud/global/images/family/cos-stable",
      # Otherwise you can use debian-10
#      "google-machine-image=debian-cloud/global/images/family/debian-10",
      "google-preemptible=true",
      "google-zone=europe-north1-a",
      "engine-registry-mirror=https://mirror.gcr.io"
    ]

Configuration - Google Storage Cache

Since a new container starts for each job, tools like npm have to re-download all dependencies each time. To reduce this time, project dependencies can be cached14. This cache is then downloaded, unpacked, before the job is run and saved updated after a success.

Among other things, GitLab Runner also supports Google Cloud Storage as a cache hoster, so the configuration is very straightforward15.

  [runners.cache]
    Type = "gcs"
    Path = "gitlab-runner"
    Shared = false

    [runners.cache.gcs]
      BucketName = "YOUR_BUCKET_NAME"
      CredentialsFile = "/etc/gitlab-runner/client_secret.json"

Configuration - Summary

The complete config.toml should now look similar to this:

# Spawn up to 5 machines
concurrent = 5
check_interval = 0

[session_server]
  session_timeout = 1800
  
[[runners]]
  name = "gitlab-runner-gce"
  url = "https://YOUR_GITLAB_INSTANCE/"
  token = "SOME_TOKEN"
  executor = "docker+machine"
  limit = 5
  
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

  [runners.cache]
    Type = "gcs"
    Path = "gitlab-runner"
    Shared = false

    [runners.cache.gcs]
      BucketName = "YOUR_BUCKET_NAME"
      CredentialsFile = "/etc/gitlab-runner/client_secret.json"
      
  [runners.machine]
    IdleCount = 0
    IdleTime = 30
    MachineDriver = "google"
    MachineName = "auto-scale-runner-%s"
    MachineOptions = [
      "google-project=GOOGLE_CLOUD_PROJECT",
      # Depending on your requirements, choose another instance
      "google-machine-type=e2-highcpu-2",
      # When running the forked docker-machine, you should use cos-stable
      "google-machine-image=cos-cloud/global/images/family/cos-stable",
      "google-preemptible=true",  
      "google-zone=europe-north1-a",
      "engine-registry-mirror=https://mirror.gcr.io"
    ]

In Action

Now that we have configured both the GitLab Runner and GCP, new CI pipeline jobs are now processed by Google instances.

Here it is worth checking again that the instances have the correct settings. Later changes to the config.toml are taken up without restarting the GitLab Runner, so it is quite easy to try out larger / smaller machine types.

After finishing the first job of a project, you should see some content in your cache bucket and the following logs in subsequent jobs:

Restoring cache
Checking cache for develop...
Downloading cache.zip from https://storage.googleapis.com/....
Successfully extracted cache

Footnotes

  1. GitHub Actions ↩︎

  2. jenkins.io ↩︎

  3. GitLab ↩︎

  4. GitLab CI/CD ↩︎

  5. GitLab Runner ↩︎

  6. Google Cloud Platform ↩︎

  7. Google Cloud Storage ↩︎

  8. CoreOS ↩︎

  9. Container-Optimized OS ↩︎

  10. Docker Machine ↩︎

  11. GitLab Docker Machine Fork ↩︎

  12. Google Cloud Platform - VM instances pricing ↩︎

  13. GitLab Runner - Autoscale Config ↩︎

  14. GitLab CI/CD - Caching ↩︎

  15. GitLab Runner - Caching Config ↩︎

Gallery

/assets/img/2020-03-10_gitlab-runner_gcp-project_01-640.png
Create project
/assets/img/2020-03-10_gitlab-runner_gcp-project_02-640.png
/assets/img/2020-03-10_gitlab-runner_gcp-project_03-640.png
Create Service Account
/assets/img/2020-03-10_gitlab-runner_gcp-project_04-640.png
Create Service Account - Permissions
/assets/img/2020-03-10_gitlab-runner_gcp-project_07-640.png
Export Service Account Key
/assets/img/2020-03-10_gitlab-runner_gcp-project_05-640.png
/assets/img/2020-03-10_gitlab-runner_gcp-project_06-640.png
Create Storage Bucket
/assets/img/2020-03-10_gitlab-runner_admin-640.png
/assets/img/2020-03-10_gitlab-runner_pipelines-640.png
/assets/img/2020-03-10_gitlab-runner_job-640.png
/assets/img/2020-03-10_gitlab-runner_gcp-console-640.png

Tags

Comments

Related