composer: GitDownloader fails on composer.json update, when there is packages added as git repositories

[exec] *** [err :: internal.xxx.com] [RuntimeException]
     [exec] *** [err :: internal.xxx.com] Failed to execute git remote set-url composer 'ssh://xxx@xxx.xxx.com/home/repositories/xxx.git' && git fetch composer && git fetch --tags composer
     [exec] *** [err :: internal.xxx.com] 
     [exec] *** [err :: internal.xxx.com] Error reading response length from authentication socket.
     [exec] *** [err :: internal.xxx.com] Permission denied, please try again.
     [exec] *** [err :: internal.xxx.com] Permission denied, please try again.
     [exec] *** [err :: internal.xxx.com] Permission denied (publickey,password).
     [exec] *** [err :: internal.xxx.com] fatal: The remote end hung up unexpectedly

Looking at code i see only one option that there is no composer remote setup yet, and he trying to set url for that…?

its in composer.phar/src/Composer/Downloader/GitDownloader.php:74