prisma: Schema generation freezed after bitbucket OS upgrade (FlatCarOS upgrade 2605.12.0 to 2765.2.2 (kernel move from 5.4.92 directly to 5.10.25))

Bug description

Bitbucket have upgraded CI pipeline platform https://support.atlassian.com/bitbucket-cloud/docs/infrastructure-changes-in-bitbucket-pipelines/

Since 4 may all ours builds, including previous in success, stay freeze on this step

npm install prisma@2.22.1 @prisma/client@2.22.1

Logs with DEBUG=“prisma:*” here

> prisma@2.22.1 preinstall /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/prisma
> node scripts/preinstall-entry.js
> sq-native@1.0.7 install /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/sq-native
> node ./install.js
> prisma@2.22.1 install /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/prisma
> node scripts/install-entry.js
> core-js@3.7.0 postinstall /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/core-js
> node -e "try{require('./postinstall')}catch(e){}"
> @apollo/protobufjs@1.0.5 postinstall /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/@apollo/protobufjs
> node scripts/postinstall
> @prisma/engines@2.22.0-21.60cc71d884972ab4e897f0277c4b84383dddaf6c postinstall /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/@prisma/engines
> node download/index.js
2021-05-11T15:35:31.966Z prisma:download file /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/@prisma/engines/query-engine-debian-openssl-1.1.x does not exist and must be downloaded
2021-05-11T15:35:31.967Z prisma:download file /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/@prisma/engines/prisma-fmt-debian-openssl-1.1.x does not exist and must be downloaded
2021-05-11T15:35:31.967Z prisma:download file /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/@prisma/engines/migration-engine-debian-openssl-1.1.x does not exist and must be downloaded
2021-05-11T15:35:31.968Z prisma:download file /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/@prisma/engines/introspection-engine-debian-openssl-1.1.x does not exist and must be downloaded
2021-05-11T15:35:31.969Z prisma:download Downloading https://binaries.prisma.sh/all_commits/60cc71d884972ab4e897f0277c4b84383dddaf6c/debian-openssl-1.1.x/query-engine.gz to /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/@prisma/engines/query-engine-debian-openssl-1.1.x
2021-05-11T15:35:31.986Z prisma:download Downloading https://binaries.prisma.sh/all_commits/60cc71d884972ab4e897f0277c4b84383dddaf6c/debian-openssl-1.1.x/migration-engine.gz to /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/@prisma/engines/migration-engine-debian-openssl-1.1.x
2021-05-11T15:35:31.987Z prisma:download Downloading https://binaries.prisma.sh/all_commits/60cc71d884972ab4e897f0277c4b84383dddaf6c/debian-openssl-1.1.x/introspection-engine.gz to /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/@prisma/engines/introspection-engine-debian-openssl-1.1.x
2021-05-11T15:35:31.988Z prisma:download Downloading https://binaries.prisma.sh/all_commits/60cc71d884972ab4e897f0277c4b84383dddaf6c/debian-openssl-1.1.x/prisma-fmt.gz to /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/@prisma/engines/prisma-fmt-debian-openssl-1.1.x
> @prisma/client@2.22.1 postinstall /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/@prisma/client
> node scripts/postinstall.js
2021-05-11T15:35:33.839Z prisma:loadEnv project root found at /opt/atlassian/pipelines/agent/build/packages/api.promtime/package.json
2021-05-11T15:35:33.854Z prisma:tryLoadEnv Environment variables not found at null
2021-05-11T15:35:33.854Z prisma:tryLoadEnv Environment variables not found at undefined
2021-05-11T15:35:33.854Z prisma:tryLoadEnv No Environment variables loaded
2021-05-11T15:35:33.855Z prisma:engines using NAPI: false
2021-05-11T15:35:33.856Z prisma:engines binaries to download query-engine, migration-engine, introspection-engine, prisma-fmt
Prisma schema loaded from schema.prisma
2021-05-11T15:35:34.798Z prisma:download copying /root/.cache/prisma/master/60cc71d884972ab4e897f0277c4b84383dddaf6c/debian-openssl-1.1.x/query-engine to /opt/atlassian/pipelines/agent/build/packages/api.promtime/node_modules/prisma/query-engine-debian-openssl-1.1.x

How to reproduce

Use a bitbucket pipeline on new infra S Kubernetes FlatcarOS to v2765.2.2 run npm install prisma with a schema file available

Expected behavior

Schema generation on premium or business infra

Prisma information

// learn more about it in the docs: https://pris.ly/d/prisma-schema

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

generator client {
  provider = "prisma-client-js"
}

model Admin {
  id        String    @id @default(uuid())
  userId    String    @unique                                       // ...via foreigne key.
  createdAt DateTime  @default(now())
  updatedAt DateTime  @default(now()) @updatedAt
}

Environment & setup

OS: Debian docker cypress/base:14.16.0
Database: Postgres docker postgres:12.5-alpine
Node.js version: 14.16.0
Prisma version: 2.22.1

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 3
Comments: 15 (8 by maintainers)

Commits related to this issue

fix(download): Workaround for docker hanging on file copy Implement workaround suggested in https://github.com/prisma/prisma/issues/7037#issuecomment-846370629 — committed to prisma/prisma by janpio 3 years ago

Most upvoted comments

Someone could try this integration release prisma@2.24.0-integration-docker-workaround.1 for the CLI, based on https://github.com/prisma/prisma/pull/7222 that I quickly hacked together based on @leomao10 's comment: https://github.com/prisma/prisma/issues/7037#issuecomment-846370629 Let me know if this fixes the problem and we could merge this workaround into our next release to unblock all of you.

janpio on May 28, 2021

You can actually see more details from here regarding the ruby related issue: https://github.com/docker/for-linux/issues/1015

In this comment, @ZimbiX managed to figure out the main issue is from copy_file_range syscall.

So I did a bit of digging and found that fs.copyFilewill get affected, so I made an POC (which is the nodeJS copy of @pedantic-git 's reproduce-copystream)

And you will see the script hang indefinitely in docker the OS kernal is higher than 5.6

Given that in #5304, @JacobMuchow found that it hangs in:

prisma:download copying /root/.cache/prisma/master/e421996c87d5f3c8f7eeadd502d4ad402c89464d/debian-openssl-1.1.x/query-engine to /app/node_modules/prisma/query-engine-debian-openssl-1.1.x +0ms

which is calling fs.copyFile

I think it is very likely that it is the same cause as docker/for-linux#1015

For the impact, it seems to affect all nodeJS versions that is using libuv > 1.38.0, with kernal version > 5.6.

leomao10 on May 21, 2021

I am not sure, maybe purely using fs.read/fs.write?

But there is a temporary walk mentioned in docker/for-linux#1015 as well. You can use fs.utime to touch the file you gonna copy, then the issue should be go away.

Something like this:

  await copyFile('/mnt/example.png', '/tmp/example.png');
  await utime('/tmp/example.png', new Date(), new Date());
  await copyFile('/tmp/example.png', '/tmp/test.png');

leomao10 on May 22, 2021

Encounter the same problem when using mac’s latest docker desktop app, https://github.com/prisma/prisma/pull/7222 fixed it!

w121211 on May 28, 2021

yes. it does work for our customers in pipelines again because of the rollback. We’ve seen other yarn hanging issues that we didn’t directly track to prism, some issues in ruby based applications that in specific situations copies zero sized files instead of the original content and some docker build issues that we managed to workaround by configuring overlayfs.

mkleint on May 18, 2021

Hi, Bitbucket Pipelines developer here. Just here to ping you the update we have done on our infrastructure:

FlatCarOS upgrade 2605.12.0 to 2765.2.2 (kernel move from 5.4.92 directly to 5.10.25)

please let me know if you need more details.

mkleint on May 12, 2021

Just for your follow @janpio we have found a quick workaround. 1 - Download all the binaries on an other working OS 2 - Generate the Client locally and commit it with all binaries (query, migration, format,…) 3 - Use env var to targets binaries

We also have test NAPI with previous steps and works fine.

lcalvy on May 12, 2021

Thanks for the report @Icalvy.

We had similar reports for Docker based installations (see https://github.com/prisma/prisma/issues/5304) and this now also seems to sometimes affect hosting providers. I also just saw a similar report for Bitbucket Pipelines specifically in our public Slack: https://prisma.slack.com/archives/CA491RJH0/p1620739486417100 (signup via slack.prisma.io).

Obviously we want to look into this and hopefully provide a fix or - earlier - a workaround.

Do you by chance know enough about FlatcarOS to help us set up a simple, local reproduction maybe? What would I need to do to get started with https://www.atlassian.com/de/software/bitbucket/features/pipelines?

janpio on May 11, 2021