nx: NX_* Environment Variables not working on prod builds in our test/staging deployments from gitlab, but only sometimes

Current Behavior

We use environment variables to point to different aws resources in our dev, test, and prod environments.

Locally, we use a .env file, and prefix all of the environment variables with NX_, like we are supposed to, and it all works fine.

In a Dev environment deployment that we set up that is built in a gitlab pipeline, and deployed to an S3 bucket, it also works fine with the env variables we set up in our gitlab CI/CD settings that are scoped to the dev environments.

However, in the exact same kind of pipeline for our Test (Staging) environment, it seems as if the NX build just simply won’t recognize our environmental variables from our gitlab CI/CD settings that we have set there, but sometimes it does. We’ve tried adding the --skip-nx-cache to see if that was messing it up or not. Lately it seems like if we commit and push directly to our main branch, the build will include our env variables just fine, but if we make a merge request, and have a merge commit that results in the pipeline, none of our env variables get used and they are all undefined.

Here are our gitlab-ci steps for both of those pipelines. In the pipeline, we echo some of the variables that have been set as a sanity check to make sure gitlab itself is not misconfigured during the pipeline.

# store node_modules in a cache
cache:
  untracked: true
  key: '*******_portals_node_cache'
  paths:
    - node_modules/

# build apps dev
build_ap_dev:
  stage: build
  dependencies: []
  environment:
    name: ap_dev
    action: prepare
  only:
    - develop
    - /(^\d)*-feature-.*/
    - /(^\d)*-bugfix-.*/
  script:
    - echo "cognito user pool - $NX_COGNITO_USER_POOLS_ID" # successfully echos here, so gitlab variable not a problem
    - yarn build-ap --prod --skip-nx-cache
    - echo "Build Successful"
  artifacts:
    expire_in: 4 hour
    paths:
      - dist/apps/agent-portal-frontend

build_ap_test:
  stage: build
  dependencies: []
  environment:
    name: ap_test
    action: prepare
  only:
    - main
    - /(^\d)*-hotfix-.*/
  script:
    - echo "cognito user pool - $NX_COGNITO_USER_POOLS_ID" # successfully echos here, so gitlab variable not a problem
    - yarn build-ap --prod --skip-nx-cache
    - echo "Build Successful"
  artifacts:
    expire_in: 4 hour
    paths:
      - dist/apps/agent-portal-frontend

package.json build scripts are kept simple, the pipeline adds the flags we need above

Screenshot-20210827123053-497x78

Other things from googling around, we’ve tried making sure our environment.prod.ts is like this:

export const environment = {
  production: false,
};

Might this be related to some kind of Nx caching issue that gets stored in our node_modules cache for our pipelines?

Expected Behavior

Nx environment variables should work in all environments (dev, test, prod) when all of them have the same build command.

yarn build-appname --prod

Steps to Reproduce

  1. Set up environment variables on gitlab CI/CD that are scoped to specific environments
  2. Develop locally with a .env file in the root of the monorepo with variables that are prefixed with NX_*
  3. Make Merge Request to develop branch and merge - DEV environment build works fine.
  4. Make Merge Request from develop to main branch and merge - TEST environment build is missing env variables.
  5. Panic and directly commit some console.logs of env variables on the main branch and run a new pipeline in TEST env - It sometimes magically works, but sometimes not.

Failure Logs

When we download the artifacts from the build to look to see if the variables got used by Nx, we don’t see them in a grep, is this normal?

Screenshot-20210827120917-1190x1013

Environment

env for my local machine, but problem happens in our pipelines that run on a node:15.14 docker image

nx report

>  NX  Report complete - copy this into the issue template

  Node : 15.14.0
  OS   : linux x64
  yarn : 1.22.5
  
  nx : Not Found
  @nrwl/angular : Not Found
  @nrwl/cli : 12.6.4
  @nrwl/cypress : 12.6.4
  @nrwl/devkit : 12.6.4
  @nrwl/eslint-plugin-nx : 12.6.4
  @nrwl/express : Not Found
  @nrwl/jest : 12.6.4
  @nrwl/linter : 12.6.4
  @nrwl/nest : Not Found
  @nrwl/next : Not Found
  @nrwl/node : Not Found
  @nrwl/nx-cloud : Not Found
  @nrwl/react : 12.6.4
  @nrwl/schematics : Not Found
  @nrwl/tao : 12.6.4
  @nrwl/web : 12.6.4
  @nrwl/workspace : 12.6.4
  @nrwl/storybook : 12.7.2
  @nrwl/gatsby : Not Found
  typescript : 4.3.5

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 7
  • Comments: 39 (3 by maintainers)

Most upvoted comments

Nx why have you forsaken us?

I was able to get this working using runtimeCacheInputs something like:

"tasksRunnerOptions": {
    "default": {
      "runner": "@nrwl/workspace/tasks-runners/default",
      "options": {
        "cacheableOperations": [
          "build",
          "lint",
          "test",
          "e2e"
        ],
        "runtimeCacheInputs": [
          "echo $NX_ENV_VARIABLE",
          "echo $NX_ENV_VARIABLE_1",
          "echo $NX_ENV_VARIABLE_2",
          "echo $NX_ENV_VARIABLE_3"
        ]
      }
    }
  },

Maybe this might help someone else

This is ridiculous, how has this not been resolved by now

Any help with this? We still cannot seem to read the env vars on production.

Can confirm that this is a problem for builds. I haven’t tested it on Vercel, but running the built version of my app locally shows that the environment variables aren’t being passed in.

We have a very similar issue @KevinVandy, the env variables are cached between builds when going through a complete Git flow (push branch for PR review -> merge to dev -> PR dev to main -> merge to main).

If we push directly from the local main branch from the CLI to the remote main, the values are properly set.

This have a big impact for us because we control the meta robots index/follow through env variables, where we allow some content pages indexation only on production. At the moment, every time we push to main after going through a full Git flow, we have to do an extra & empty “magic” push directly from the main local branch (described above) to get the good values…

We are using Vercel for the deployments and we thought it was an issue on their end with the builds cache, but it might be more complex than that, based on several discussions I could find (including this one):

We are using Nx 12.7.2, we did not try to upgrade to the latest yet

@nurbashanghai - That depends. If you’re having Kevin Vandy’s problem, I can’t help you. However, if you’re having the same issue I was having (where the env vars don’t populate for Cypress tests), you are in luck my friend!

I worked around this issue by manually bootstrapping the environment variables already available in my project, using the env-cmd NPM package. Here’s the updated package.json command I use to launch my end-to-end tests: env-cmd -f ./apps/myapp/.env nx e2e myapp-e2e

Eventually, I imagine the Nx team will find and fix this bug. The good news is that regardless of if/when/how they do that, this fix should remain working regardless.

I’m not certain if it’s an identical issue, but it sure seems similar: My project, running Nx v13.0.2 presently, has zero problems whatsoever with environment issues when started or built normally. However, when I run my Cypress e2e tests, they all disappear completely. This makes very little sense to me, since I assumed the same Webpack config was being used to build both versions out… but when I check my variables while running the watcher used to execute Cypress tests, all I see are these:

{
  "NODE_ENV": "development",
  "NX_CLI_SET": "true",
  "NX_INVOKED_BY_RUNNER": "true",
  "NX_WORKSPACE_ROOT": "<dir>/parfait-nx",
  "NX_TASK_TARGET_PROJECT": "parfait-e2e",
  "NX_TASK_HASH": "<hash>"
}

There are around a dozen environment variables (usually read from my .env file) missing from that readout.

Hi, can this be prioritized please? Currently environment variables are not being read and it is a big issue

Hope these code help to you, it works for me

next.config.js

const webpack = require('webpack');

/**
 * @type {import('@nrwl/next/plugins/with-nx').WithNxOptions}
 **/
const nextConfig = {
  webpack(config) {
    // add bellow code
    if (process.env.NODE_ENV === 'production') {
      config.plugins.push(
        new webpack.DefinePlugin({
          'process.env': JSON.stringify(process.env),
        })
      );
    }

    return config;
  },
};

Same problem here deploying to Vercel. I have multiple CRA & Nextjs apps in the workspace.

@charleskoehl your problem is propably related to not havin ENV variables defined as an inputs in your nx.json. See: https://github.com/nrwl/nx/issues/6891#issuecomment-1449084647

NX environment variables for a project are ignored for me only in the CI (github actions) with v15.9.2 (locally - all builds and tests run fine).

For your webpack.config{.prod}.js add the following line if you are referencing environment variables (the error will be something great like The "path" argument must be of type string. Received undefined with no further output)

const path = require('path');
require('dotenv').config({ path: path.resolve(__dirname, '.env') });

If your application accesses env vars via process.env['NX_{VARNAME}'] (via apps/my_app/src/environments/environment.ts for example); then for your Jest tests you’ll need:

jest.config.ts

export default {
   ...
   setupFilesAfterEnv: ['<rootDir>/jest.setup.ts'],
}

jest.setup.ts

if (process.env['NX_INVOKED_BY_RUNNER']) {
  config({ path: __dirname + '/.env' });
}

At last, I found a solution by using dotenv-webpack plugin and passing a .env file to it! I tried it with env-cmd too but it didn’t help while running nx build

I noticed updating from nx 12.6.3 to 12.8.0, dotenv got updated. And looking at the releases, it seems to be in your interest to at least do this update.