berry: [Bug] Cannot run script without installing dependencies even if the script is trying to `install`

This seems like a crazy degradation in dev experience. We have a react native project with multiple different target platforms and multiple different installation scripts depending on precisely what you’re trying to do. i.e. an iOS engineer would run a different yarn install alias that would include ios dependencies and an android engineer would not need (and wouldn’t even be capable of it if they’re developing on Windows). So a generic postinstall script does not work for us.

+1 for reverting this behaviour.

I understand this might be a little frustrating, but our position is simple: we must fulfill the contract, or fail early. For package.json, it means that your dependencies must be in the $PATH when you execute a script. If we can’t fulfill this contract, then we must fail early.

Yarn will fail even if I am trying to execute yarn install alias, or if I try to run some node script.js script, and the reason “why?” is because it is not aware what I am actually trying to run. It just assumes that any script from package.json requires full dependency tree to be installed. Since it can’t really understand what I am trying to run in my script, I do not feel that this is a good thing to take away control from user on whether they can run certain script or not. If yarn knew better, that’s another story, but it doesn’t, hence I, as a user, should have full control over it IMO, and if I run some script and it fails because of missing dependencies in $PATH or whatnot, well, this is my fault then, but I am willing to take that responsibility. Right now it just causes frustration, because I know that I can run some script, but yarn refuses to, because it thinks it knows better, and there is no good way around as well, like there is no yarn sudo-run.

With that being said, the best workaround I can suggest is to implement your proxy connection by configuring a hook. We don’t have a generic beforeAllInstalled (PR welcome!), but the validateProject hook should be interchangeable for this purpose.

While this is indeed sort of a solution, I do not think this is a good one. First of all, I would need to give up a clear and convenient way of using some scripts via package.json "scripts": in favor of some really specific way (hooks) to do it in a specific package manager (yarn@2 and higher). This fact alone IMO is enough for user to not want to use a hooks solution. It’s not some fancy thing I want my package manager to do, it is not some project specific thing that I would like to make a plugin for, it is just running scripts from package.json. Another point, is that right now there is now way through yarn API to detect that it will bail with an “install error” (the only hook that runs before the said error is setupScriptEnvironment), so to achieve desirable behavior (i.e. run script without needing to install first), you would need to add tricky code that will use some pseudo-evidence that yarn will bail with an error (like lack of .yarn-state.yml for example, or generating a new one). And even if I am wrong and just haven’t found it in a docs, or if it will be added later, it forces you to add some complex solution for what was before a trivial task (from a user standpoint). Every time I update yarn, I would need to go though changelog very carefully because public API for classes like Project or Configuration IMO more likely to change, than let’s say enableGlobalCache flag, there is a high chance of making an error when adding or maintaining such a custom plugin and furthermore it adds unwanted complexity to user’s project infrastructure. And while it is acceptable to have those cons when developing such solution for a project-specific problem, I do not think this is one of them.

Running into this issue as well. The policy in our project is that all necessary CLI commands must be documented by putting them into a package.json script and this includes npm install and should also include yarn install. It’s quite unexpected that the install script must be run using npm instead of yarn, this causes quite a bit of friction during onboarding.

@wjmao88 and future yarn’ers: I wrote a little plugin that generates an empty state file if none is present upon execution of scripts before yarn install. The empty state file removes the error upon startup and thus allows for customized commands to run. I know it is probably not ideal (state restoration might be impacted by this?), but in my case it can serve as a workaround just fine. You can specify which commands you want the state file generation to be triggered upon in the ALLOWED_SCRIPTS array.

.yarn/override-state/plugin-override-state.js

// Edit this to allow commands
const ALLOWED_SCRIPTS=["install:ui", "install:api"];

const NODE_MODULES="node_modules";
const STATE_FILE=".yarn-state.yml";

module.exports = {
  name: `plugin-override-state`,
  factory: require => ({
    hooks: {
      async setupScriptEnvironment(project, scriptEnv) {
        if (scriptEnv != null && ALLOWED_SCRIPTS.find(script => script == scriptEnv.npm_lifecycle_event) != null) {
          if (project.configuration.get(`nodeLinker`) === `node-modules`) {
            const stateFile = [project.cwd, NODE_MODULES, STATE_FILE].join("/");
            const fs = require("fs");
            
            if (!fs.existsSync(stateFile)) {
              console.log("Detected command allowed in stateless environment and state file is missing.");
              console.log("Generating empty state file...");

              fs.mkdirSync(require('path').dirname(stateFile), { recursive: true });
              fs.appendFileSync(stateFile, "# Autogenerated\n", {"flag":"w+"});
              fs.appendFileSync(stateFile, "__metadata:\n");
              fs.appendFileSync(stateFile, "  version: 1\n");
              fs.appendFileSync(stateFile, "  nmMode: classic\n");
            }
          }
        }
      }
    },
  })
};

Also add this to your plugins: section in your .yarnrc.yml:

plugins:
  - .yarn/override-state/plugin-override-state.js

I understand this might be a little frustrating, but our position is simple: we must fulfill the contract, or fail early. For package.json, it means that your dependencies must be in the $PATH when you execute a script. If we can’t fulfill this contract, then we must fail early.

With that being said, the best workaround I can suggest is to implement your proxy connection by configuring a hook. We don’t have a generic beforeAllInstalled (PR welcome!), but the validateProject hook should be interchangeable for this purpose.

I encountered this error when I upgraded to yarn@3.2.0 from 3.0.0. My use case was to use the workspace-tools to generate github releases for each package in the monorepo in CI:

yarn workspaces foreach exec "sh $PWD/scripts/release.sh"

This has stopped working now since I didn’t install node modules before running this shell script. Adding an unnecessary yarn install to my pipeline adds another 30s to the build (even with cache).

It makes sense to throw this error for “run” commands since those are coming from package.json and it is reasonable to expect node_modules/packages to be present for them, but for “exec” where the aim is to “Execute a shell script” it shouldn’t be thrown.

I think this issue makes it difficult to use yarn as a “project manager” and forces us to download packages where it’s not needed. I hope it can be reconsidered.

I don’t think this issue should be closed, having scripts in package.json that do not require a prior install is a perfectly valid construct.

For example, we use this to initialize proxy connections to our private registry. This is now broken after upgrading to yarn v2. Running install will also fail in this case, because the proxy connection is not there yet, thus retrieving some of the packages will fail.

This change in behavior now requires us to create separate shell scripts, which is against the intention of scripts defined in package.json: i.e. one central place to bundle all project-related commands.

Because of this behaviour, I am unable to bootstrap my project cleanly by using yarn scripts right now.

My project involves compiling rust source code to wasm by using wasm-pack. The wasm-pack generated packages is part of the yarn workspace, and is a dependency of other modules in that workspace. Unfortunately, because of how wasm-pack works, the package.json manifest files are not present before the package is built. With that, yarn installation fails, as it complains about missing dependencies. Unfortunately, my script yarn wasm, which is supposed to generate those, cannot be run until yarn detects a valid install. Which cannot happen because there are no built packages… so I can’t bootstrap my project.

I also tried a preinstall script to run that wasm build process, but yarn doesn’t run preinstall before attemtping an install, as the name would suggest. That seems to be just a completely broken feature right now.

My current solution is to use npm run wasm before first installation, or invoke the necessary wasm-pack build manually. Unfortunately, there is no nice way to do that automatically right now. I would love the preinstall script to do what it says on the tin.

That is still a huge problem in Yarn v4. I hope it can be reconsidered in the future.

I know that, and I’m fine with that if there are other ways to automate an repository setup that needs to be done before first install. Though to be honest, why is it that way? It seems that the only reason for this behaviour is that the scipts cannot be run before install in general, which this whole issue is about. And in my opinion it’s a mis-feature.

Same issue here which causes a painful workaround on our side.