Hi @submarines-and, firstly thanks for the reproduction, reproductions like this are paramount to get issues solved.

So far it appears that this issue is only reproducible when differential loading is enabled, also this doesn’t happen when NG_BUILD_MAX_WORKERS is configured to 2.

NG_BUILD_MAX_WORKERS=2 ng build --configuration=production

@submarines-and - Awesome. This should really help us get to the bottom of it. I might not have time today but will try to reproduce locally before the weekend.

We have this error as well. It appears randomly and only in our CI pipeline. Only happens for lazy loaded routes, but also completely random which one it happens to.

It is as if the i18n-variable substitution does not happen during build. I have looked in the build logs but can’t see anything suspicious, although I am note sure what I am looking for.

See samples below:

Sample how the code looks when broken:

return e = $localize:@@admin-clear-user-data␟84b9e941cbfed42244d77ad3fe91c45102351270␟5698175515008817140:Clear user data, t = $localize:@@admin-clear-user-data-description-v2␟dbaf9468af65ea407ed21b2408d62a2d3e7e28ae␟7967954382951777750:Use this dashboard to clear all Hej! data for a specific user. While the user is pending, deletion, you can see it in a table below., n = $localize:@@label-privacy-dashboard-add-yourself␟66b6e520d2535ef081b09d0b5a6d0a2c045342c5␟8720367929285043226:Delete for yourself, i = $localize:@@label-queue-for-deletion␟0bd9e01e687e3b46bd102deecd9bee571b27be6c␟3854011180104367841:Queue users for deletion, a = $localize:@@admin-clear-user-pending-deletion␟b38f0673b58973cb6c435c9485893191f0656622␟81765222213306383:User pending deletion, r = $localize:@@label-email␟244aae9346da82b0922506c2d2581373a15641cc␟4768749765465246664:Email, [[1, “elevated”, “p-4”], [1, “mb-3”], e, [1, “text-muted”, “mb-3”], t, [1, “d-block”, “mb-3”, 3, “defaultValue”, “onValueChanged”], [1, “d-lg-flex”], [1, “mr-3”, 3, “disabled”, “click”], n, [3, “disabled”, “click”], i, [4, “ngIf”], [4, “ngIf”, “ngIfElse”], [“loading”, “”], [“class”, “w-100 mt-4”, 4, “ngIf”], [1, “w-100”, “mt-4”], [1, “pb-3”], a, r, [4, “ngFor”, “ngForOf”], [“class”, “border-top”, 4, “ngIf”], [1, “border-top”], [1, “pt-2”, “pb-2”]]

Sample how the same code chunk looks when working properly. return e = “Clear user data”, t = “Use this dashboard to clear all data for a specific user. While the user is pending, deletion, you can see it in a table below.”, n = “Delete for yourself”, i = “Queue users for deletion”, a = “User pending deletion”, r = “Email”, [[1, “elevated”, “p-4”], [1, “mb-3”], e, [1, “text-muted”, “mb-3”], t, [1, “d-block”, “mb-3”, 3, “defaultValue”, “onValueChanged”], [1, “d-lg-flex”], [1, “mr-3”, 3, “disabled”, “click”], n, [3, “disabled”, “click”], i, [4, “ngIf”], [4, “ngIf”, “ngIfElse”], [“loading”, “”], [“class”, “w-100 mt-4”, 4, “ngIf”], [1, “w-100”, “mt-4”], [1, “pb-3”], a, r, [4, “ngFor”, “ngForOf”], [“class”, “border-top”, 4, “ngIf”], [1, “border-top”], [1, “pt-2”, “pb-2”]]

@cwaespi - would you be able to share a reproduction that we can play with?

@petebacondarwin Unfortunately, I am not allowed to share our project. But I am trying to create a reproduction.

@clydin Just tested the new snapshot. 12 out of 12 builds were successful 🚀. Hope others have the same success.

If anyone has some additional time, could you please try out another snapshot build to see if it resolves your problem:

npm install git+https://github.com/angular/angular-devkit-build-angular-builds.git#181255d64435aefc41e899fbf8db7a93c808d8ee --save-dev

We will also phase out IE when 13 is released, most of our organisation has done so already but we have some countries with really old equipment that needs more time to upgrade. When the day comes I will celebrate!!

That makes sense for us. Damn corporate IT and their legacy browsers!!!

I have applied the env variable workaround to all our builds and so far they have been running fine (although they have not ran super many times yet. Build times seem unaffected (although I would still choose 3x longer builds if that fixed the issue).

I will report back if I see anything new with the workaround active. Thank you for the help so far!

@mmonteiroc Just tried to reproduce the problem locally but had no success. Everything indicates that the problem only occurs in our CI pipeline.

The curious thing is that locally and in the CI pipeline i use the same Dockerfile for the build. We use Gitlab AutoDevops for CI/CD.

Our Dockerfile

ARG CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX

# Builder
FROM ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}node:16-alpine as builder
WORKDIR /app
COPY package.json package-lock.json .npmrc /app/
RUN npm ci
COPY . /app
RUN mkdir /app/dist
RUN npm run build -- easy --configuration production

# Runtime
FROM ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}nginx:1.21-alpine

COPY ./.nginx/default.conf.template /etc/nginx/conf.d/default.conf.template
COPY --from=builder /app/dist/ /usr/share/nginx/html

EXPOSE 5000
CMD ["/bin/sh", "-c", "envsubst < /etc/nginx/conf.d/default.conf.template | sed -e 's/§/$/g' > /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"]

We also have this issue in our Nx-based app: every once in a while, the app builds successfully and most of the routes work, but just one “random” lazy loaded module fails to load with ReferenceError: $localize is not defined. Rebuilding the app fixes the issue, but it is really hard to catch (most of our e2e tests run on dev builds which don’t have this issue). I’m sure it didn’t happen before updating to v12 of angular/localize

@petebacondarwin reached out via email.