Stage Gitlab Mirror

Automatic backup staging

Gitlab secrets (stored under /etc/gitlab/gitlab-secrets.json) only need to be transferred once!

Thus, they are not part of the Ansible automatic backup. Make sure to perform the initial backup staging manually according to the manual below

Run the ansible playbook provided in the gitlab-stage repo to automatically stage backups. This will copy files on your target host, so be very careful:

ansible-playbook -v --ask-vault-pass -i inventories stage-backups.yml

Manual backup staging

Stop Gitlab instance

Create mountpoints owned by root (only needed once)

sudo su
mkdir -p /mnt/backup/gitlab

Disable periodic background jobs: Admin > Monitoring > Background jobs > Cron > Disable All

Stop all gitlab services

sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket save && sudo gitlab-ctl stop && sudo gitlab-ctl start postgresql && sudo gitlab-ctl start gitaly

Disable services in /etc/gitlab/gitlab.rb to avoid restarts:

# Remove existing definitions
sudo sed -i "/gitlab_pages['enable']/d" /etc/gitlab/gitlab.rb
sudo sed -i "/puma['enable']/d" /etc/gitlab/gitlab.rb

# Append 'false' to deactivate
echo "gitlab_pages['enable'] = false" | sudo tee -a /etc/gitlab/gitlab.rb
echo "puma['enable'] = false"        | sudo tee -a /etc/gitlab/gitlab.rb

Reconfigure gitlab

sudo gitlab-ctl reconfigure

Stop redis and verify everything is stoped

sudo gitlab-ctl stop redis
sudo gitlab-ctl status

Only gitaly and postgres should be running If other services are not stopped, try to re-run gitlab-ctl stop and check again

Stage BORG Backup

Mount BORG

Mount BORG gitlab backup to mountpoint. You will need the BORG passphrase, which is either stored in the PasswordSafe (LRZ) or maybe obtained from the current EKUT gitlab admin!

borg mount /dss/general-backup/<CURRENT BACKUP DIR>/ /mnt/backup/gitlab

Based on the official gitlab docu

Copy files

Copy all relevant files and dirs manually using rsync:

cd /mnt/backup/gitlab/<LATEST BACKUP>/

# Copy data to NAS
sudo rsync -av var/opt/gitlab/git-data/repositories/ --progress /mnt/gitlab_nas/git-data/repositories/
sudo rsync -av var/opt/gitlab/gitlab-rails/shared/lfs-objects/ --progress /mnt/gitlab_nas/git-data/gitlab-rails/shared/lfs-objects/

# Copy secrets (only needed once)
sudo rsync -av etc/gitlab/gitlab-secrets.json --progress /etc/gitlab/

# Copy backup files
sudo rsync -av var/opt/gitlab/backups/ --progress /var/opt/gitlab/backups/

Unmount BORG

Unmount the BORG backup

cd
borg umount /mnt/backup/gitlab

Restore backup directory

Ensure ownership

Ensure proper ownership for all directories again:

sudo chown -R git:git /mnt/gitlab_nas/git-data
sudo chown git:git /var/opt/gitlab/gitaly/gitaly.pid
sudo chown -R git:git /var/opt/gitlab/gitlab-rails/uploads
sudo chmod 00700 /var/opt/gitlab/gitlab-rails/uploads/*
sudo chown git:git /var/opt/gitlab/.ssh/*
sudo chown git:git /var/opt/gitlab/backups/*

Restore backup

Start the redis service explicitly and all others

sudo gitlab-ctl start redis
sudo gitlab-ctl start
sudo gitlab-ctl reconfigure

Check if the latest backup files are present:

ls -l /var/opt/gitlab/backups/

Stop DB related services

sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq
# Verify
sudo gitlab-ctl status

Restore backup. Important: Your gitlab versions MUST MATCH! If not, this step will fail…

sudo gitlab-backup restore

Restart Gitlab (again)

Enable services in /etc/gitlab/gitlab.rb again:

# Remove existing definitions
sudo sed -i "/gitlab_pages['enable']/d" /etc/gitlab/gitlab.rb
sudo sed -i "/puma['enable']/d" /etc/gitlab/gitlab.rb

# Append 'true' to activate
echo "gitlab_pages['enable'] = true" | sudo tee -a /etc/gitlab/gitlab.rb
echo "puma['enable'] = true"        | sudo tee -a /etc/gitlab/gitlab.rb

Reconfigure

sudo gitlab-ctl reconfigure

Set custom Sign-out/Home URLs restored from DB to new domain:

gitlab-rails runner "s = ApplicationSetting.current; s.home_page_url='https://gitlab.plantmicrobe.rdm.lab.lrz.de/'; s.after_sign_out_path='https://gitlab.plantmicrobe.rdm.lab.lrz.de/'; s.save!"

Re-start gitlab again (previously stoped services)

sudo gitlab-ctl start

Optional steps

Sanity checks

Check overall sanity

sudo gitlab-rake gitlab:check SANITIZE=true

Check if decryption works

sudo gitlab-rake gitlab:doctor:secrets
sudo gitlab-rake gitlab:artifacts:check
sudo gitlab-rake gitlab:lfs:check
sudo gitlab-rake gitlab:uploads:check

Re-enable cronjobs: Admin > Monitoring > Background jobs > Cron > Enable All

One final reconfigure

sudo gitlab-ctl reconfigure