serverless-python-requirements: Unable to get good binding path

For anyone coming from the internet looking to get GDAL working with Serverless, you should know that I went through hell and back trying to get it to work messing around with Docker containers and using @dev360’s outline as a rough guide.

However, there’s a much easier way now that AWS Lambda supports layers. Simply use one of the prebuilt layers made by https://github.com/developmentseed/geolambda at runtime to provide all of GDAL’s dependencies. For your dev environment, specify one of the images from that project that contain Python. The plugin will use the image to create a container to compile the module bindings within at build time, and serverless deploy will link the preexisting layer to your lambda so that the bindings can seamlessly load the GDAL library binaries from the layer’s filesystem at runtime. It Just Works™️.

Here’s what my serverless.yml looks like:

provider:
  name: aws
  runtime: python3.6  # Must match the Python version used by the image you select below
  region: us-east-1  # Must match the layer since layers are scoped to regions

functions:
  gdal:
    handler: gdal.main
    layers:
      - arn:aws:lambda:us-east-1:552188055668:layer:geolambda:1

custom:
  pythonRequirements:
    dockerizePip: true
    dockerImage: developmentseed/geolambda:1.1.0-python36

I don’t know if this would be worth anything to you guys, but I found a workaround that was a little bit elaborate and it works in my particular case (django + postgis, gdal).

1. Set dockerizePip: true
2. create a docker file off lambci/lambda:build-python3.6 and reference that image in your serverless pythonRequirements config via the dockerImage attribute. To find the name, do a docker-compose build and look up the name with docker images.
3. Add the build steps for GDAL on Amazon Linux to your Dockerfile; similar to here. This gets the gdal/geos/proj C deps built and creates .so files that will be available under /tmp/app/local/lib that are built for the correct architecture.
4. This was my hack: Add a vendor: ./serverless-vendor setting to the pythonRequirements section in serverless.yml … this will copy your local folder called serverless-vendor into the sitepackages directory so its available at runtime and thus can be linked to.
5. docker-compose run --rm [your docker image name] bash into your docker image and copy the /tmp/app/local/lib/ files into the ./serverless-vendor/[your name] folder we referenced earlier.
6. In my django settings, I was able to point geodjango to the GDAL .so file by setting the GDAL_LIBRARY_PATH in Django’s settings.py. I’d wager theres some similar way to point gdal to the so file - it would be a matter of looking at what geo django does under the hood with this variable.

Also worth noting - read carefully:

• I’m a noob with serverless and was suprised to see it upload the file which was 65mb.
• It was annoying getting the path to the serverless-vendor folder - I ended up importing some pip dep and going .. from there to find the folder that had the libgdal.so file.
• CAVEAT: I haven’t been able to test this 100% because I haven’t been able to get a postgis server up and running yet, but Django was able to get past the GDAL load step that was holding me back before, so I figured it must have worked. I know the files get to the lambda function which should be good enough.

Hope this is useful to anyone. Experience with serverless+python so far hasn’t been for the faint of heart 😃

Yeah. One issue after another. Looking into this option to see. https://github.com/joshtkehoe/lambda-python-gdal