typeorm: [serverless] Connection Reuse is broken in a Lambda environment

well that was painless. ugly as all hell but for the other people running into this, a temp fix (so far) seems to be:

lib/typeorm-monkeypatch

import { Connection, EntitySchema, EntityMetadata } from 'typeorm'

// this is a copypasta of the existing typeorm Connection method
// with one line changed
// @ts-ignore
Connection.prototype.findMetadata = function (target: Function | EntitySchema<any> | string): EntityMetadata | undefined {
  console.log('monkeypatched function')
  return this.entityMetadatas.find(metadata => {
    // @ts-ignore
    if (metadata.target.name === target.name) { // in latest typeorm it is metadata.target === target
      console.log('found target===target')
      return true;
    }
    if (target instanceof EntitySchema) {
      console.log('found name===name')
      return metadata.name === target.options.name;
    }
    if (typeof target === "string") {
      if (target.indexOf(".") !== -1) {
        return metadata.tablePath === target;
      } else {
        return metadata.name === target || metadata.tableName === target;
      }
    }

    return false;
  });
}

and then i require that file before i do anything with typeorm. thank you again @Wintereise, this was killing our velocity

Here’s my patch against 0.2.18. For ManyToMany, metadata.target.name is undefined. This should work.

index 851ffba..0f61d12 100644
--- a/node_modules/typeorm/connection/Connection.js
+++ b/node_modules/typeorm/connection/Connection.js
@@ -465,8 +465,12 @@ var Connection = /** @class */ (function () {
      */
     Connection.prototype.findMetadata = function (target) {
         return this.entityMetadatas.find(function (metadata) {
-            if (metadata.target === target)
+            if (metadata.target.name && metadata.target.name === target.name) {
                 return true;
+            }
+            if (metadata.target === target) {
+                return true;
+            }
             if (target instanceof __1.EntitySchema) {
                 return metadata.name === target.options.name;
             }

@gl2748 Here’s a patch instead, to be applied against typeorm@0.2.12

diff --git a/node_modules/typeorm/connection/Connection.js b/node_modules/typeorm/connection/Connection.js
index 66dad98..4d885fc 100644
--- a/node_modules/typeorm/connection/Connection.js
+++ b/node_modules/typeorm/connection/Connection.js
@@ -443,7 +443,7 @@ var Connection = /** @class */ (function () {
      */
     Connection.prototype.findMetadata = function (target) {
         return this.entityMetadatas.find(function (metadata) {
-            if (metadata.target === target)
+            if (metadata.target.name === target.name)
                 return true;
             if (target instanceof __1.EntitySchema) {
                 return metadata.name === target.options.name;

Obvious caveats are that you now need to guard against reusing the same class name (and turn off any and all post compilation transforms (a la webpack minify / what not) that might cause that.)

I auto-apply this using https://github.com/ds300/patch-package using npm’s postinstall hook.

Update to sls offline 6 and you won’t even need that. Just use worker threads or child processes. That deals with the shared singleton scope problem quite well.

On Thu, Apr 30, 2020, 11:51 PM rifflock notifications@github.com wrote:

–skipCacheInvalidation fixed this issue for us as well running version 0.2.24

@Wintereise https://github.com/Wintereise, I think you’re right that this can be closed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/typeorm/typeorm/issues/3427#issuecomment-622006492, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHGGEQIC3GTGRSFT6TR3DDRPG3D3ANCNFSM4GQJF5YA .

I too would like to see this issue addressed. This also seems to be happening on Firebase.

@pleerock Is there confirmation on including this fix / a more elegant approach for this one. I’m starting to move large amounts of my nest/typeorm work over to serverless and I can’t even contemplate life without typeorm ahaha

Any news on this? This is what I did to bypass it…

// src/app.module.ts
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { AdsModule } from './ads/ads.module';
import { ConfigModule } from 'nestjs-config';
import * as path from 'path';
import { TypeOrmConfigService } from './config/database';

@Module({
  imports: [
    ConfigModule.load(path.resolve(__dirname, 'config', '**', '!(*.d).{ts,js}')),
    TypeOrmModule.forRootAsync({
      useClass: TypeOrmConfigService,
    }),
  ],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

// src/config/database.ts
import { Injectable } from '@nestjs/common';
import { TypeOrmModuleOptions, TypeOrmOptionsFactory } from '@nestjs/typeorm';
import { ConnectionManager, getConnectionManager } from 'typeorm';
import * as dotenv from 'dotenv';
import * as fs from 'fs';

@Injectable()
export class TypeOrmConfigService implements TypeOrmOptionsFactory {
  async createTypeOrmOptions(): Promise<TypeOrmModuleOptions> {
    const connectionManager: ConnectionManager = getConnectionManager();
    let options: any;

    if (connectionManager.has('default')) {
      options = connectionManager.get('default').options;
      await connectionManager.get('default').close();
    } else {
      const env: any = dotenv.parse(fs.readFileSync(`.env`));
      options = {
        type: env.TYPEORM_CONNECTION,
        host: env.TYPEORM_HOST,
        username: env.TYPEORM_USERNAME,
        password: env.TYPEORM_PASSWORD,
        database: env.TYPEORM_DATABASE,
        schema: env.TYPEORM_SCHEMA,
        port: env.TYPEORM_PORT,
        logging: true,
        entities: [__dirname + '/../entities/**.entity{.ts,.js}'],
        synchronize: false,
      };
    }

    return options;
  }
}

This solution below from @pueue (from https://github.com/typeorm/typeorm/issues/2598) fixed this for me.

I’m now able to successfully re-use open connections in Lambda functions (both serverless-offline and “live”).

I no longer get the RepositoryNotFoundError due to entities not reloading properly from cached connections.

export const connectDatabase = async (callback: Function = () => {}) => {
  try {
    const connectionManager = getConnectionManager()
    let connection: Connection;

    if (connectionManager.has('default')) {
      console.log('use a connection already connected')
      connection = injectConnectionOptions(connectionManager.get(), connectionOptions)
    } else {
      connection = connectionManager.create(connectionOptions)
      await connection.connect()
    }

    await callback();
    return connection;
  } catch (e) {
    console.log(e);
    throw e;
  }
};

export const injectConnectionOptions = (connection: Connection, connectionOptions: ConnectionOptions) => {
  /**
   * from Connection constructor()
   */

  // @ts-ignore
  connection.options = connectionOptions
  // @ts-ignore
  connection.manager = connection.createEntityManager();
  // @ts-ignore
  connection.namingStrategy = connection.options.namingStrategy || new DefaultNamingStrategy();
  // @ts-ignore
  connection.relationLoader = new RelationLoader(connection);
  // @ts-ignore
  connection.relationIdLoader = new RelationIdLoader(connection);

  /**
   * from Connection connect()
   */
  // @ts-ignore
  connection.buildMetadatas();

  return connection;
}

A long time has passed, and I finally had some time to actually look into this. It appears that this isn’t really a problem in the AWS runtime at all, merely when using sls-offline to locally debug.

The reason (and fix) for that are mentioned in #2623 (which this issue is a duplicate of).

tl;dr: If you’re using sls-offline, please invoke it with --skipCacheInvalidation and none of the patches mentioned in this issue will be required. It’s always been fine in the actual runtime (AWS) itself.

What basically happens is that the singleton instance of TypeORM gets messed with when using the shared code runner from sls-offline.

@pleerock If more people confirm this is effective, I think you can close this issue.

watching this issue with bated breath, trying some of the fixes suggested here.

same use case: connection pooling shits the bed when we use it in serverless offline. TY @Wintereise for all the research and troubleshooting!

We’re also having the same problem with serverless offline and typeorm 0.2.18.

It actually seems like the --skipCacheInvalidation might be causing issues with the hot reload. I’ve noticed since adding it that the reload isn’t running the changed code. Perhaps I spoke too soon, because in order to run the changed code I have to kill the process and run it again.

Well known, yes. It doesn’t cause issues, it flat out kills it. Between that and v6 lacking the feature completely, we don’t have a working HMR solution at the moment.

Backend minification makes little sense, my recommendation is to just turn it off. I have it off too. @sdebaun

ok i isolated my problem to some typeorm + webpack interaction hell, it may only be peripherally related to the original serverless connection reuse. i have something that looks like its working in my environment. i looks like the root of some of these issues lies in how typeorm stores and searches for metadata for a given entity instance.

when you tell typeorm about entities via the decorator and base class its registering metadata in a list

when you later try to do things with instances of one of those entities typeorm tries to find the matching metadata that its tracking

problem: no matter how you use @Entity or BaseEntity (even if you supply a name property) when typeorm tries to match the target instance it tries to match on target.name which is the name of the function, in this case the class name

in most use cases, that would be ‘Profile’ (for my Profile entity) which would then let typeorm find the Profile metadata and work its magic

with webpack: the minification does shenanigans like:

let d = class extends BaseEntity

so the ‘name’ property for every. single. fucking. entity. is ‘d’

fixes:

1. turn off minification in webpack with

module.exports = {
  ...
  optimization: {
    minimize: false,
  },
  ...
};

2. if we really need minification

2.1 add a static property to all our @Entity classes

static entityName = ‘Profile’

and monkeypatch typeorm findMetadata to use that field

if (metadata.target.entityName === target.entityName)

@Wintereise I’ll have to go through the discussion when I get off work, but I did end up using the fix / workaround suggested in #2598 by the OP.

Here’s what it boiled down to (formatting slightly adjusted for readability):

import {
    createConnection,
    Connection,
    getConnection,
    getConnectionManager,
    ConnectionOptions,
    DefaultNamingStrategy,
} from "typeorm";
import { User } from "./entity/User";
import { Listing } from "./entity/Listing";
import { RelationLoader } from "typeorm/query-builder/RelationLoader";
import { RelationIdLoader } from "typeorm/query-builder/RelationIdLoader";

const CONNECTION_OPTIONS : ConnectionOptions = {
    type: process.env.TYPEORM_CONNECTION,
    host: process.env.TYPEORM_HOST,
    port: process.env.TYPEORM_PORT,
    username: process.env.TYPEORM_USERNAME,
    password: process.env.TYPEORM_PASSWORD,
    database: process.env.TYPEORM_DATABASE,
    dropSchema: false,
    synchronize: false,
    logging: false,
    entities: [ User, Listing ],
}

/**
 * Establishes and returns a connection to the database server. If an existing
 * connection is found, the connection is reused.
 *
 * @see https://github.com/typeorm/typeorm/issues/2598#issue-345445322
 * @export
 * @returns {Promise<Connection>}
 */
export async function getDatabaseConnection() : Promise<Connection> {
    try {
        console.log("Establishing connection...");
        const connectionManager = getConnectionManager();
        let connection: Connection;

        if (connectionManager.has("default")) {
            console.log("Reusing existion connection...");
            connection = injectConnectionOptions(
                connectionManager.get(),
                CONNECTION_OPTIONS,
            );
        } else {
            connection = connectionManager.create(CONNECTION_OPTIONS);
            await connection.connect();
        }

        console.log("Connection established");
        return connection;
    } catch (e) {
        console.error(e);
        throw e;
    }
};

/**
 * Injects missing / outdated connection options into an existing database
 * connection.
 *
 * @see https://github.com/typeorm/typeorm/issues/2598#issue-345445322
 * @param {Connection} connection
 * @param {ConnectionOptions} CONNECTION_OPTIONS
 * @returns {Connection}
 */
function injectConnectionOptions(
    connection: Connection,
    CONNECTION_OPTIONS: ConnectionOptions,
) : Connection {
    // @ts-ignore
    connection.options = CONNECTION_OPTIONS
    // @ts-ignore
    connection.manager = connection.createEntityManager();
    // @ts-ignore
    connection.namingStrategy = connection.options.namingStrategy ||
        new DefaultNamingStrategy();
    // @ts-ignore
    connection.relationLoader = new RelationLoader(connection);
    // @ts-ignore
    connection.relationIdLoader = new RelationIdLoader(connection);
    // @ts-ignore
    connection.buildMetadatas();

    return connection;
}

The above code is located in a Database.ts file in the root of the project source directory. It can then be used in the following fashion:

import { getDatabaseConnection } from "./Database";

export async function getStuff(event: APIGatewayEvent, context: Context, callback: Callback) {
    const connection = await getDatabaseConnection();
    // Do stuff with the connection
}

Needed it for a proof of concept project and it did its job.

I have the development environment set up in Vagrant and the project itself with Serverless so I can make something reproducible out of it, I just need to find the time.

@Wintereise Thanks for the heads up! I stumbled upon that particular issue by spamming endpoints with curl GET requests via a shell script. For the purposes of my proof of concept project it didn’t matter much, but it’s undoubtedly an issue for any serious work.

Please note, that in new version of serverless-offline --skipCacheInvalidation does not exist use --allowCache instead

For those of us using the monkey patch (a la https://github.com/typeorm/typeorm/issues/3427#issuecomment-504134702 and MAYBE the PR at https://github.com/typeorm/typeorm/pull/4804 assuming it’s a verbatim copy), be advised that cascade with ManyToMany does NOT function past the first load (in sls-offline).

Reverting and using the option above will fix this.

Hi All – I’m using AWS Lambda functions and AWS RDS Aurora Postgres-Serverless which sleeps the database after n-minutes of inactivity. The DB takes 20-40 seconds to come back online. Despite all the various workarounds people have posted across different issues/threads, none work reliably. TypeORM’s connection model fails in various ways without an easy avenue to a workable solution.

I love TypeORM and truly, truly, truly want to use it with the new Serverless stacks, but this connection issue is preventing it.

1. Has anybody else found a reliable workaround to reuse connections via Lambda functions that have not been purged by the node Execution Environment, and Aurora which sleeps and warm-starts the database?
2. If so, can we get a PR into a standard release (soon) as serverless projects are already quite mainstream.

Any help is greatly appreciated.

Aurora’s sleep -> rewake model is problematic for many issues, but that isn’t the issue being discussed here.

You’ll have more luck using the Aurora data-api driver TypeORM has now, but pre-warming and keepalives (of the aurora sls instance) are still a requirement.