lambci: Errors with ssh/git-ssh: PRIV_END: seteuid: Operation not permitted

Came here on a google. Had this same issue, but I’m using containers to deploy my lambdas. So, I just need to have the patched version of ssh in my container image. Bare minimum example:

FROM public.ecr.aws/lambda/nodejs:18 as ssh_builder
RUN yum update -y && \
    yum groupinstall -y "Development Tools" && \
    yum install -y wget openssl-devel && \
    wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-7.4p1.tar.gz && \
    tar -xzf openssh-7.4p1.tar.gz && \
    cd openssh-7.4p1 && \
    sed -i -e 's/fatal/printf/' sshconnect.h && \
    sed -i -e 's/OpenSSH_7.4/OpenSSHForLambda_7.4/' version.h && \
    ./configure && \
    make && make install

FROM public.ecr.aws/lambda/nodejs:18
COPY --from=ssh_builder /usr/local/bin/ssh /usr/local/bin/ssh
# Here is where you would set up the rest of your image

So here’s a full working example – this works directly from the Lambda console (with a Node.js 4.3 function):

var fs = require('fs')
var execSync = require('child_process').execSync

exports.handler = function(event, context, cb) {
  execSync('curl https://s3.amazonaws.com/lambci/pkg/lambda-git-test.tgz | tar -C /tmp -xz')

  fs.writeFileSync('/tmp/known_hosts', knownHosts())
  fs.writeFileSync('/tmp/id_rsa', privateKey(), {mode: 0o600})

  var env = {
    PATH: `/tmp/usr/bin:${process.env.PATH}`,
    LD_LIBRARY_PATH: `/tmp/usr/lib64:${process.env.LD_LIBRARY_PATH}`,
    GIT_TEMPLATE_DIR: '/tmp/usr/share/git-core/templates',
    GIT_EXEC_PATH: '/tmp/usr/libexec/git-core',
    GIT_SSH_COMMAND: 'ssh -i /tmp/id_rsa -o UserKnownHostsFile=/tmp/known_hosts',
  }

  execSync('rm -rf /tmp/aws4')

  console.log(execSync('git clone git@github.com:mhart/aws4.git /tmp/aws4', {env: env, encoding: 'utf8'}))

  console.log(execSync('ls -l /tmp/aws4', {encoding: 'utf8'}))

  cb()
}

function knownHosts() {
  return `github.com ssh-dss AAAAB3NzaC1kc3MAAACBANGFW2P9xlGU3zWrymJgI/lKo//ZW2WfVtmbsUZJ5uyKArtlQOT2+WRhcg4979aFxgKdcsqAYW3/LS1T2km3jYW/vr4Uzn+dXWODVk5VlUiZ1HFOHf6s6ITcZvjvdbp6ZbpM+DuJT7Bw+h5Fx8Qt8I16oCZYmAPJRtu46o9C2zk1AAAAFQC4gdFGcSbp5Gr0Wd5Ay/jtcldMewAAAIATTgn4sY4Nem/FQE+XJlyUQptPWMem5fwOcWtSXiTKaaN0lkk2p2snz+EJvAGXGq9dTSWHyLJSM2W6ZdQDqWJ1k+cL8CARAqL+UMwF84CR0m3hj+wtVGD/J4G5kW2DBAf4/bqzP4469lT+dF2FRQ2L9JKXrCWcnhMtJUvua8dvnwAAAIB6C4nQfAA7x8oLta6tT+oCk2WQcydNsyugE8vLrHlogoWEicla6cWPk7oXSspbzUcfkjN3Qa6e74PhRkc7JdSdAlFzU3m7LMkXo1MHgkqNX8glxWNVqBSc0YRdbFdTkL0C6gtpklilhvuHQCdbgB3LBAikcRkDp+FCVkUgPC/7Rw==
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
`
}

// Include your own private key here
function privateKey() {
  return `-----BEGIN RSA PRIVATE KEY-----
AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCP
...
HkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEY
-----END RSA PRIVATE KEY-----
`
}

I’ll leave lambda-git-test.tgz up on S3 for the foreseeable future – but if I don’t discover any issues with it, then I’ll replace the existing git bundle in LambCI – so LambCI users shouldn’t have any further issues with ssh repos.

Alright, some good news- running ssh (and subsequently git) in a Lambda environment is possibly albeit a little hacky.

A quick summary:

• recompile ssh client to not use seteuid() (surprisingly easy. I can provide more info if requested)
• upload ssh binary and supporting libraries
  • libcrypto.so
  • libssl.so
• upload git binary and needed supporting binaries (typically found in /usr/lib/git-core [at least on Arch]) (You may need different binaries. These are just the git functions I used.)
  • git clone
  • git-commit
  • git-diff
  • git-fetch-pack
  • git-index-pack
  • git-push
• set PATH variable
  • needs to get to ssh, git, and all git binaries
• set LD_LIBRARY_PATH variable
  • needs to include .so files for ssh

@jdchmiel one of my requirements was that I had to use ssh keys. Thanks for the tip though.

@mhart I wasn’t aware of these layers. Just yesterday I was wondering if anything like this existed out there. Thanks for sharing!

@sjackman Thank you! I was finally able to build it from source

@mhart Thanks for the detailed follow up. It works for me too! Here’s my Lambda function that uses git, ssh and ruby. https://github.com/sjackman/linuxbrew-lambda Here’s the tarballs of binaries that I used:

• git https://raw.githubusercontent.com/lambci/lambci/v0.9.14/vendor/git-2.4.3.tar
• ruby http://d6r77u77i8pq3.cloudfront.net/releases/traveling-ruby-20150715-2.2.2-linux-x86_64.tar.gz
• ssh https://github.com/sjackman/linuxbrew-lambda/raw/master/bin/ssh