docker-gitlab: Restore fails when gitaly isn't running

Tying together #1655 and #1744.

I have a backup created in docker-gitlab, Gitlab v11.5.3, which I then restored on a separate VM using the instructions per the readme.

This fails on the “Restoring Repositories” step with:

rake aborted!
GRPC::Unavailable: 14:Connect Failed
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:181:in `attach_status_results_and_complete_call'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:377:in `request_response'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:178:in `block in request_response'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:177:in `request_response'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:170:in `block (3 levels) in rpc_stub_class'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:136:in `call'
/home/git/gitlab/lib/gitlab/gitaly_client/storage_service.rb:19:in `delete_all_repositories'
/home/git/gitlab/lib/backup/repository.rb:45:in `block in prepare_directories'
/home/git/gitlab/lib/backup/repository.rb:44:in `each'
/home/git/gitlab/lib/backup/repository.rb:44:in `prepare_directories'
/home/git/gitlab/lib/backup/repository.rb:77:in `restore'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:87:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:60:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
Tasks: TOP => gitlab:backup:repo:restore
(See full trace by running task with --trace)

I did some digging, and the issue here is that gitaly isn’t running. I was able to get the backup to restore by, while in the container that I was executing the restore with, getting gitaly up and running before that step was executed. After that, everything worked fine.

I’d write a PR for the readme, but I’m not sure exactly how you’d like folks to approach doing a restore; my solution (and the others in the linked tickets) are pretty hacky. If you have some guidance on approach, I’d be happy to take a shot at a fix.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 17 (1 by maintainers)

Most upvoted comments

Having now backed up and restored a couple times, here are refined instructions on how to run backup and restore. Do not stop your usual GitLab container before doing this.

docker exec -it gitlab bash
supervisorctl stop all active services except gitaly
Still from inside the container, run bundle exec rake gitlab:backup:create or bundle exec rake gitlab:backup:restore, as needed
When done, turn back on all deactivated services with supervisorctl start

CommanderTso on Feb 13, 2019

Hey @CommanderTso ,

This issue is problem because gitaly is changed as a core requirement. This means it needs to be run when executing rake tasks relating to git because gitaly does this git commands.

The current way you describe is a workaround with this image. But it would work and I think I would also do it so.

A better approach would be to separate gitaly from this gitlab container and start a separate gitaly container so it’s easier to handle from the docker side. But this would be a breaking change. I think we could takle this with 12.X .

solidnerd on Jan 10, 2019