prisma: Allow skipping sha checksum

I would be more than happy to implement this and submit a PR if I can get some assurance that such a change would be accepted.

Here’s my situation: the environment I deploy my application to does not have internet access for security reasons. Typically, when we use a library that includes platform-specific binaries like prisma does, we setup a repository in artifactory and that basically behaves like a caching proxy for the binary file server.

In this case, we then configured the PRISMA_ENGINES_MIRROR environment variable so that the files would be downloaded from artifactory instead of binaries.prisma.sh.

Unfortunately in this case, you’re validating the sha256 checksum not only of the gzipped file, but also of the un-gzipped file. Artifactory creates it’s own checksums, so when it pulls in, for example, prisma-fmt.gz, it creates a sha256 checksum and makes it accessible as prisma-fmt.gz.sha256. This is all fine so far, but the prisma download scripts also attempt to download prisma-fmt.sha256, but there’s no prisma-fmt file (no extension) on the remote file server, so artifactory never generates a sha256 file for it.

The end result is that when fetching the files from artifactory, it fails to download prisma-fmt.sha256, which then causes the unzipped checksum validation to fail.

As far as I can tell, there’s no way to fix this in artifactory, so it’d be really nice if I could just disable the validation check in the download script.

Again, I’d be more than happy to submit a PR that introduces a new environment variable for disabling the checksum validation. Perhaps PRISMA_IGNORE_CHECKSUM? With this enabled, it would warn about checksum issues but would not fail the build. Would the prisma team be open to this change?

I don’t think we have a workaround at the moment. Even if you download the binaries and use env vars to tell prisma where they are, it still fails because it tries to download the checksum files.

I don’t think checking in node modules will work unless you’re deploying to the same platform you develop on.

I’m currently working on a fix (as described above) but it will likely be a week or so before I’m able to submit it as a pull request.

Not a dumb question at all @thw0rted. Our dependency on OpenSSL makes it impossible to use the built-in os specific package support of Npm and Co, and our binaryTargets feature makes it more complex as well (as it allows configuring the installation of multiple packages, including an auto detected native one, but also only of engine that are not for your current platform). But we are thinking about this a lot, and hope to make progress in that direction soon. Npm would be really neat as a distribution mechanism for our engines.

Not a dumb question, that is definitely an option we could consider. The complexity is that in @prisma/engines-version we actually do not have that information present (if you are interested), so would need to gather it first… but still doable. Just not a priority right now as we assume that adding the env var right now would at least unblock everyone until we have time to really rethink this functionality.

In case it helps anybody we did have success with grabbing all the files (in our case, I believe 4x binaries plus 4x binary checksums, and 4x .gz checksums), moving them all into our private / offline Nexus, and setting PRISMA_ENGINES_MIRROR=https://our_nexus_user:the_password@our.nexus.server/repo-path. It’s not pretty but it gets the job done for now.

This might be a dumb question, but is it totally impractical to distribute the native binaries via npm? I know everybody is moving away from node-gyp but is there nothing to take its place? It would be great if we could just depend on @prisma/query-engine-debian-openssl-1.1.x instead of having it download the binary at runtime. That would also neatly handle the checksum issue.

I am facing this exact issue with corporate servers lacking internet connectivity and artifactory is the only place to store external binaries. Agree with the above solution which is a life saver. However, at this point what is a temporary solution? checking in node_modules and do a npm rebuild instead of npm i in jenkins build?

The only item I’m not sure about is number 2 above. If it were me using those environment variables, I would expect it never to even attempt to validate the checksum, since I’d be providing either a custom binary or a pre-downloaded binary.

That is absolutely correct, and I have hopefully misunderstood something when I wrote that list.

To your suggestion, that sounds valid but somehow the “specificity” of the env var makes me … cautious. But I can indeed not see anything bad with it. I think this would be worth trying to implement!

(PRISMA_ENGINES_CHECKSUM_IGNORE_MISSING would be my preferred naming by the way, as that would allow an alternative PRISMA_ENGINES_CHECKSUM_SKIP if we ever need that for the other use case.)

Sounds good to me. Here’s a proposed solution that (I think) resolves all 3 of the issues you listed above:

New env var: PRISMA_IGNORE_MISSING_CHECKSUM

When this environment property contains a truthy value, it will still attempt to download the checksum files.

If the checksum file download fails (either due to no internet or 404 error) it will log a warning and skip the checksum validation, allowing the build to continue.

If the checksum download succeeds but the checksum doesn’t match the downloaded binary, the postinstall script will fail just like it it does today.

In my case (artifactory with missing unzipped checksum file), it will fail to download the unzipped checksum file but it will still validate that the zipped checksum file matches the sha256 file from the server.

The only item I’m not sure about is number 2 above. If it were me using those environment variables, I would expect it never to even attempt to validate the checksum, since I’d be providing either a custom binary or a pre-downloaded binary.

Thoughts?

@janpio The prisma installation process runs inside a Jenkins pipeline that is setup by the enterprise policy. Therefore, I can’t give networking details (I’m also unfamiliar with these).

I suspect that you are right, and that the binaries should fail to download as well, I guess previously, I just installed with npm --ignore-scripts i therefore instead of failing in the installation, it failed later while trying to pull the db schema.

Here is the error from the pipeline, when I install Prisma (without ignoring scripts).

17:40:14  Error: request to https://binaries.prisma.sh/all_commits/b371888aaf8f51357c7457d836b86d12da91658b/debian-openssl-1.1.x/introspection-engine.gz.sha256 failed, reason: connect ECONNREFUSED 108.156.107.46:443

Ok, so what is the recommendation here? Just copy the binaries into our cloud storage, and the mirror env?