prisma: Error opening a TLS connection - Race condition

Bug description

During some deployment (we have never experienced it locally) we receive the following error

Error: Error opening a TLS connection: error:24067044:random number generator:rand_pool_add:internal error:crypto/rand/rand_lib.c:733: (self signed certificate in certificate chain)
    at startFn (/app/node_modules/@prisma/client/core/runtime/library.js:100:2563)
    at Proxy.onModuleInit (/app/dist/apps/aurora/webpack:/libs/data-access/server/prisma-core/src/lib/base-core-prisma.service.ts:34:9)
    at async Promise.all (index 1)
    at callModuleInitHook (/app/node_modules/@nestjs/core/hooks/on-module-init.hook.js:43:5)
    at NestApplication.callInitHook (/app/node_modules/@nestjs/core/nest-application-context.js:210:13)
    at NestApplication.init (/app/node_modules/@nestjs/core/nest-application.js:97:9)
    at NestApplication.listen (/app/node_modules/@nestjs/core/nest-application.js:155:33)

// base-core-prisma.service.ts

@Injectable()
export abstract class BaseCorePrismaService<T extends Prisma.PrismaClientOptions = Prisma.PrismaClientOptions>
    extends PrismaClient
    implements OnModuleInit, OnModuleDestroy
{
    protected constructor(
        protected readonly config: BaseCorePrismaConfig,
        protected readonly optionsArg?: Prisma.Subset<T, Prisma.PrismaClientOptions>
    ) {
        super(optionsArg);
    }

    async onModuleInit(): Promise<void> {
        await this.$connect(); // Row 34

        if (!this.config.readReplica) {
            // Enable soft deletes
            this.$use(SoftDeleteMiddleware);
        }
    }

    async onModuleDestroy(): Promise<void> {
        await this.$disconnect();
    }

    async enableShutdownHooks(app: INestApplication): Promise<void> {
        this.$on('beforeExit', async () => {
            await app.close();
        });
    }
}

This is an abstract class in our NestJS APIs that our db model wrapper extend so we have shared configuration between the class that connects to our Postgres DB and the class that connects to our MongoDB (thanks to prisma having a unified setup this makes it really easy! thanks!).

This error only happens sometimes (pretty rarely actually, as I thought it was fixed a month ago since we have not seen it since).

Our connection strings are using &TrustServerCertificate=true&sslmode=require and the server is deployed in a Docker container based on FROM node:18.12.1-bullseye to Elastic beanstalk on t4g.micro instances (Arm-based AWS Graviton2 processors), this does not seem to happen on the x86 elastic beanstalk environments (at least not that we have scene yet). the binary targets we have are binaryTargets = ["native", "linux-musl-openssl-3.0.x", "linux-arm64-openssl-1.1.x", "debian-openssl-1.1.x"]

How to reproduce

Reproduction is VERY tricky as it only happens rarely, only on ARM instances, and only when deployed to Elastic beanstalk. Open to ideas here I would love to be able to repro it.

Expected behavior

No issues with self signed certs and connecting.

Prisma information

>3000 lines, if a prisma engineer needs specifics feel free to let me know and I can provide what you need.

Removed some stuff for clarity, the Mongo service is using the base service in a similar way to the CorePrismaService

@Injectable()
export abstract class BaseCorePrismaService<T extends Prisma.PrismaClientOptions = Prisma.PrismaClientOptions>
    extends PrismaClient
    implements OnModuleInit, OnModuleDestroy
{
    protected constructor(
        protected readonly config: BaseCorePrismaConfig,
        protected readonly optionsArg?: Prisma.Subset<T, Prisma.PrismaClientOptions>
    ) {
        super(optionsArg);
    }

    async onModuleInit(): Promise<void> {
        await this.$connect();

        if (!this.config.readReplica) {
            // Enable soft deletes
            this.$use(SoftDeleteMiddleware);
        }
    }

    async onModuleDestroy(): Promise<void> {
        await this.$disconnect();
    }

    async enableShutdownHooks(app: INestApplication): Promise<void> {
        this.$on('beforeExit', async () => {
            await app.close();
        });
    }
}


@Injectable()
export class CorePrismaService extends BaseCorePrismaService {
    constructor(@Inject(CORE_PRISMA_CONFIGURATION) configuration: CorePrismaConfiguration) {
        super(
            {},
            {
                log: configuration.debug ? ['query'] : undefined,
                datasources: {
                    db: configuration.primary,
                },
            }
        );
    }
}

Environment & setup

AWS Linux t4g.micro Arm-based AWS Graviton2 processors Docker node:18.12.1-bullseye AWS RDS Postgres & MongoDB Atlas

Prisma Version

4.10.0

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 3
  • Comments: 26 (7 by maintainers)

Most upvoted comments

@darvelo that sounds very plausible, I’m 90% sure it indeed has the same root cause. It might be challenging to try to reproduce this exact error on my side so it would be very helpful if someone who encountered it could test 5.3.0-integration-arm-openssl-5-2-0.1 and see if it fixes the issue.

@aqrln Could this issue possibly also have the same root cause as #18510? They both seem to be OpenSSL issues on arm64.

I switched my CI and container setup all over to x86 and haven’t seen this issue again.

Any information that helps us to understand what is happening, optimally a reliable reproduction.

I will see what I can do.

as I thought it was fixed a month ago since we have not seen it since

Does that imply the error also happened in older versions before 4.10.0 already? Important for us to pinpoint if this could be something we recently introduced - or not.

Yes, it also happened in 4.9.x, I upgraded to 4.10.0 in case that helped.