gitea: On Prem Azure Devops Migration fails

Hi, Any plans to support migration from on prem Azure DevOps (git) to gitea?

I have tried and it just fails - possibly because authentication in azure devops is not using basic authentication but something else?

It just logs:

May 19 18:15:27 git gitea[687]: #033[36m2022/05/19 18:15:27 #033[0m#033[32m...ervices/task/task.go:56:#033[32mhandle()#033[0m #033[1;31m[E]#033[0m Run task failed: #033[1mAuthentication failed: Clone: exit status 128 - fatal: Authentication failed for 'http://devops.root.dom/ROOT/common/_git/Helm/'

I am 100% certain that the username/password combo is correct, since I use it on a daily basis - and also can access the repo from another service - using the exact same combo.

I also have this in my app.ini

[migrations]
ALLOW_LOCALNETWORKS = true

My guess is that it is probably because I have a “special” character in my password and when I turn on debug logging I can see the url it tries to clone - and if I copy/paste that URL and do a manual git clone from the prompt - I also get the same:

fatal: Authentication failed for 'http://devops.root.dom/ROOT/common/_git/Helm/'

If I do a

git clone http://root%5Cbbs@devops.root.dom/ROOT/common/_git/Helm

And manually type in the password - it just works.

I have no idea if its some kind of escaping that happens that messes up the http request inside git - or if its simply wrongly escaped from gitea’s side.

Do I just have to accept this?

According to “people” on the internet, then it seems to have been escaped correctly according to https://en.wikipedia.org/wiki/Percent-encoding - but no dice - I have used other systems in the past that did not like my password.

To give a hint about what character that messes it up - its # - which I know is bad in a url - but I refuse to change my password just because some systems do not like it.

wget on the other hand just accepts the encoded url - so I am beginning to think its a bug in git.

Edit:

Just tested on a windows machine and git clone just works with the encoded url (git version 2.33) - on the gitea server (Rocky Linux) - git version 2.27.0 it fails - so either its a bug that is fixed in 2.33 - or an OS issue that is causing this.

Edit2: Updated git to version 2.31 on the server - and its the same issue. So I am thinking its a git on linux issue.

_Originally posted by @bjornbouetsmith in https://github.com/go-gitea/gitea/issues/8689#issuecomment-1131956921_

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

Got it working. Basically the on prem Devops needs the Authorization header for cloning. https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=Linux#use-a-pat

I will not create a PR for that because this should be implemented as a generic “add custom headers to git requests” instead of just the Authorization header. A possible implementation needs to support the migration/clone, pull/push mirror, LFS client pull/push, … and everything I forgot. It may not be worth the effort just to enable migrations from on prem Devops as no other service needs this.

These are the code changes I made:

diff --git a/modules/git/repo.go b/modules/git/repo.go
index 8ba3ae4fd..f601b7300 100644
--- a/modules/git/repo.go
+++ b/modules/git/repo.go
@@ -98,16 +98,17 @@ func (repo *Repository) IsEmpty() (bool, error) {
 
 // CloneRepoOptions options when clone a repository
 type CloneRepoOptions struct {
-       Timeout       time.Duration
-       Mirror        bool
-       Bare          bool
-       Quiet         bool
-       Branch        string
-       Shared        bool
-       NoCheckout    bool
-       Depth         int
-       Filter        string
-       SkipTLSVerify bool
+       Timeout             time.Duration
+       Mirror              bool
+       Bare                bool
+       Quiet               bool
+       Branch              string
+       Shared              bool
+       NoCheckout          bool
+       Depth               int
+       Filter              string
+       SkipTLSVerify       bool
+       AuthorizationHeader string
 }
 
 // Clone clones original repository to target path.
@@ -126,6 +127,9 @@ func CloneWithArgs(ctx context.Context, args []CmdArg, from, to string, opts Clo
        if opts.SkipTLSVerify {
                cmd.AddArguments("-c", "http.sslVerify=false")
        }
+       if opts.AuthorizationHeader != "" {
+               cmd.AddArguments("-c").AddDynamicArguments("http.extraHeader=" + opts.AuthorizationHeader)
+       }
        if opts.Mirror {
                cmd.AddArguments("--mirror")
        }
diff --git a/modules/migration/options.go b/modules/migration/options.go
index 1e92a1b0b..944d83bc1 100644
--- a/modules/migration/options.go
+++ b/modules/migration/options.go
@@ -11,13 +11,15 @@ import "code.gitea.io/gitea/modules/structs"
 // this is for internal usage by migrations module and func who interact with it
 type MigrateOptions struct {
        // required: true
-       CloneAddr             string `json:"clone_addr" binding:"Required"`
-       CloneAddrEncrypted    string `json:"clone_addr_encrypted,omitempty"`
-       AuthUsername          string `json:"auth_username"`
-       AuthPassword          string `json:"-"`
-       AuthPasswordEncrypted string `json:"auth_password_encrypted,omitempty"`
-       AuthToken             string `json:"-"`
-       AuthTokenEncrypted    string `json:"auth_token_encrypted,omitempty"`
+       CloneAddr              string `json:"clone_addr" binding:"Required"`
+       CloneAddrEncrypted     string `json:"clone_addr_encrypted,omitempty"`
+       AuthUsername           string `json:"auth_username"`
+       AuthPassword           string `json:"-"`
+       AuthPasswordEncrypted  string `json:"auth_password_encrypted,omitempty"`
+       AuthToken              string `json:"-"`
+       AuthTokenEncrypted     string `json:"auth_token_encrypted,omitempty"`
+       UseAuthorizationHeader bool   `json:"use_authorization_header"`
+       AuthorizationHeader    string `json:"-"`
        // required: true
        UID int `json:"uid" binding:"Required"`
        // required: true
diff --git a/modules/repository/repo.go b/modules/repository/repo.go
index de6de3bda..1e69acb67 100644
--- a/modules/repository/repo.go
+++ b/modules/repository/repo.go
@@ -74,10 +74,11 @@ func MigrateRepositoryGitData(ctx context.Context, u *user_model.User,
        }
 
        if err = git.Clone(ctx, opts.CloneAddr, repoPath, git.CloneRepoOptions{
-               Mirror:        true,
-               Quiet:         true,
-               Timeout:       migrateTimeout,
-               SkipTLSVerify: setting.Migrations.SkipTLSVerify,
+               Mirror:              true,
+               Quiet:               true,
+               Timeout:             migrateTimeout,
+               SkipTLSVerify:       setting.Migrations.SkipTLSVerify,
+               AuthorizationHeader: opts.AuthorizationHeader,
        }); err != nil {
                return repo, fmt.Errorf("Clone: %w", err)
        }
@@ -95,11 +96,12 @@ func MigrateRepositoryGitData(ctx context.Context, u *user_model.User,
                        }
 
                        if err := git.Clone(ctx, wikiRemotePath, wikiPath, git.CloneRepoOptions{
-                               Mirror:        true,
-                               Quiet:         true,
-                               Timeout:       migrateTimeout,
-                               Branch:        "master",
-                               SkipTLSVerify: setting.Migrations.SkipTLSVerify,
+                               Mirror:              true,
+                               Quiet:               true,
+                               Timeout:             migrateTimeout,
+                               Branch:              "master",
+                               SkipTLSVerify:       setting.Migrations.SkipTLSVerify,
+                               AuthorizationHeader: opts.AuthorizationHeader,
                        }); err != nil {
                                log.Warn("Clone wiki: %v", err)
                                if err := util.RemoveAll(wikiPath); err != nil {
diff --git a/services/migrations/gitea_uploader.go b/services/migrations/gitea_uploader.go
index 8a7533b3d..5de25dfb6 100644
--- a/services/migrations/gitea_uploader.go
+++ b/services/migrations/gitea_uploader.go
@@ -7,6 +7,7 @@ package migrations
 
 import (
        "context"
+       "encoding/base64"
        "fmt"
        "io"
        "os"
@@ -118,19 +119,24 @@ func (g *GiteaLocalUploader) CreateRepo(repo *base.Repository, opts base.Migrate
        r.DefaultBranch = repo.DefaultBranch
        r.Description = repo.Description
 
+       if opts.UseAuthorizationHeader {
+               opts.AuthorizationHeader = fmt.Sprintf("Authorization: Basic %s", base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s:%s", opts.AuthUsername, opts.AuthPassword))))
+       }
+
        r, err = repo_module.MigrateRepositoryGitData(g.ctx, owner, r, base.MigrateOptions{
-               RepoName:       g.repoName,
-               Description:    repo.Description,
-               OriginalURL:    repo.OriginalURL,
-               GitServiceType: opts.GitServiceType,
-               Mirror:         repo.IsMirror,
-               LFS:            opts.LFS,
-               LFSEndpoint:    opts.LFSEndpoint,
-               CloneAddr:      repo.CloneURL, // SECURITY: we will assume that this has already been checked
-               Private:        repo.IsPrivate,
-               Wiki:           opts.Wiki,
-               Releases:       opts.Releases, // if didn't get releases, then sync them from tags
-               MirrorInterval: opts.MirrorInterval,
+               RepoName:            g.repoName,
+               Description:         repo.Description,
+               OriginalURL:         repo.OriginalURL,
+               GitServiceType:      opts.GitServiceType,
+               Mirror:              repo.IsMirror,
+               LFS:                 opts.LFS,
+               LFSEndpoint:         opts.LFSEndpoint,
+               CloneAddr:           repo.CloneURL, // SECURITY: we will assume that this has already been checked
+               Private:             repo.IsPrivate,
+               Wiki:                opts.Wiki,
+               Releases:            opts.Releases, // if didn't get releases, then sync them from tags
+               MirrorInterval:      opts.MirrorInterval,
+               AuthorizationHeader: opts.AuthorizationHeader,
        }, NewMigrationHTTPTransport())
 
        g.sameApp = strings.HasPrefix(repo.OriginalURL, setting.AppURL)
+1
KN4CK3R on Oct 30, 2022
Read more comments on GitHub
← gitea: New git security policy `safe.directory` (eg: 1.16.6 docker image) makes pushing fails
gitea: Open Pull-Requests view with > 500 organizations need more than 1 minute →