Current Gitlab Runner setup (T287279)

We're currently relying on WMCS VPSs for shared runner capacity. There is a project named gitlab-runners in which to provision new instances, and a profile to help provision Docker based runners on those instances. Note that a standalone puppetmaster in the same project stores the runner registration token under /etc/puppet/secret, and Puppet autosigning is turned off to protect the token value.

Usage of Shared Runners

Shared Runners are CI workers which are available to every project by default. So CI jobs should be executed by the WMCS Shared Runners without any additional configuration. To force a run on Shared Runners use the tag wmcs. So all jobs which need Shared Runners should specify this tag:

wmcs-build-job:
  stage: cloud-build
  image: docker-registry.wikimedia.org/bookwom
  script:
    - echo "Compiling the code..."
    - echo "Compile complete."
  tags:
    - wmcs # run job on wmcs Shared Runners only

Setting up a new shared runner (for administrators)

To set up a new shared runner, following these steps.

1. Create a new WMCS VPS instance.
   1. Log in to https://horizon.wikimedia.org and navigate to the gitlab-runners project.
   2. Launch a new Debian bookworm instance, following the runner-{nnnn} naming convention.
   3. Choose the g3.cores8.ram24.disk20.ephemeral90.4xiops flavor
2. Wait until the new instance has fully provisioned and you can successfully ssh to the running instance using your authorized key and sudo as root. (This typically takes a few minutes.)
3. Check if there is an existing profile::gitlab::runner::token on the standalone puppetmaster in /etc/puppet/private/hieradata. If not, generate a new token and add it there.
4. Run puppet on the new instance, this might fail and needs a manual approval of the certificate
   1. Do the little SSL dance that is required of instances that use a standalone puppetmaster.
      1. On the new runner (runner-{nnnn}.gitlab-runners.eqiad1.wikimedia.cloud).
         1. Run sudo rm -rf /var/lib/puppet/ssl to remove the existing SSL certs used by the default puppetmaster.
         2. Run sudo -i puppet agent --test --verbose --server gitlab-runners-puppetserver-01.gitlab-runners.eqiad1.wikimedia.cloud to have the puppet client generate a new SSL cert.
      2. On gitlab-runners-puppetserver-01.gitlab-runners.eqiad1.wikimedia.cloud sign the new instance's SSL cert.
         1. Run sudo -i puppetserver ca list --all and find the new instance in the list.
         2. Run sudo -i puppetserver ca sign --certname runner-{nnnn}.gitlab-runners.eqiad1.wikimedia.cloud to sign the client cert.
5. Run puppet again to make sure the runners uses the project puppet master
6. Add role::gitlab_runner to the instance's Puppet Classes under the Puppet Configuration tab.
7. Run sudo -i puppet agent --test --verbose --server gitlab-runners-puppetserver-01.gitlab-runners.eqiad1.wikimedia.cloud on the runner to ensure it has fully provisioned the role::gitlab_runner role.
8. Verify that the runner has successfully registered with our GitLab instance by viewing the runner list.