GitLab/Replica
gitlab-replica-a.wikimedia.org and gitlab-replica-b.wikimedia.org are copies of production GitLab. Currently gitlab1003 and gitlab1004 are the replicas of gitlab2002.
The purpose of the GitLab replica is:
- to have a almost identical machine with production Gitlab to test changes
- to verify the installation path developed by S&F for gitlab1001
- to have a host for failover in case production GitLab breaks (WIP)
- to have parity with gerrit-replica
The purpose is not to share the load between both GitLab instances.
Manual sync of replica
To manually sync of production GitLab with the replica the latest backups have to be moved from production GitLab to the replica. You can either use the existing rsync configuration or cumin. In most cases the most recent backup should be present under /etc/gitlab/config_backup and /srv/gitlab-backup and you can skip this step.
Transfer of backup to replica
For rsync, run the following commands on the production host to sync the latest backup to the replica:
gitlab2002:~# sudo systemctl start rsync-config-backup-gitlab1003.wikimedia.org.service
gitlab2002:~# sudo systemctl start rsync-data-backup-gitlab1003.wikimedia.org.service
Restore of backup on replica
All restore logic for the replica is handled by a custom shell script which is present on the replica. So the easiest option to restore is to run the script on the replica:
sudo systemctl start backup-restore.service
# or
sudo /srv/gitlab-backup/gitlab-restore.sh
It is also possible to do the restore manually if needed. The GitLab restore guide should be used. The exception here is that we have to keep the gitlab.rb from the replica when restoring the config backup. Before restore, make sure to copy the old gitlab.rb before following the guide.
sudo cp /etc/gitlab/gitlab.rb /etc/gitlab/gitlab.rb.bak
Before restore, make sure the backup can be found by GitLab.
ls /srv/gitlab-backup/
After the restore is finished, the settings have to be reapplied to the replica using gitlab-settings:
./settings --instance https://gitlab-replica-a.wikimedia.org --settings-file settings-replica.yaml update
Automated sync of replica
Automated sync and restore between the replica and production GitLab is active. For automated sync of the backups a systemd timer with rsync is configured on production GitLab and rsyncd is running on the replica. The job runs every 24h and transfers the most recent data and config backup to the replica.[1] The actual restore of the backup on the replica is handled by custom shell script and a systemd timer running this script. The restore runs every 24 hours.
The most recent backup can be found in: /etc/gitlab/config_backup and /srv/gitlab-backup.
To restore a custom backup make sure to use the correct timestamp when using the gitlab-backup restore BACKUP=<name> command and follow the GitLab restore guide.
Usage
The usage of gitlab-replicas (replica-a or replica-b) should be mostly internal for testing, troubleshooting and in case of an emergency. Thus login with SSO needs the groups ops, wmf or nda.