backstage: 🐛 Bug Report: Upgrade v1.8.3 to v1.9.1 causes white screen in Production deployment (You need to enable JavaScript to run this app)
📜 Description
Following upgrade from v1.8.3 to v1.9.1 our production docker deployment in aws ecs with HTTPS and port 443 for both FE and BE stops working - displays white screen with You need to enable JavaScript to run this app.
- There are no console or network errors
- It works perfectly in local yarn dev AND in local dockerised build with port 7007 for both BE and FE
- aws ecs logs contain no errors
We made no config changes. When we roll back to previous version, v1.8.3, it starts working again (after we revert some DB knex migration changes)
Prime suspect - (https://github.com/backstage/backstage/blob/master/docs/releases/v1.9.0-changelog.md#patch-changes-5) “Apps will now rewrite the app.baseUrl configuration to match the current location.origin” - this might have messed up the ports in our configuration - effectively disconnecting the Backend?
👍 Expected behavior
If it works in local yarn dev AND localised dockerised environment, it should also work in QA and Production aws deployments. Our QA and Prod deployments have same config, just different urls/deployments.
👎 Actual Behavior with Screenshots
- It works perfectly in local yarn dev AND in local dockerised build with port 7007 for both BE and FE In aws ecs docker deployments on ports 443:
- white screen is displayed
- There are no console or network errors
- aws ecs logs contain no errors
Prod log shows BE and FE spin up correctly - no errors:
The only thing logged in the production log is backend serving the js files, it doesn’t receive regular path requests:
👟 Reproduction steps
- Have version 1.8.3 with app.config.yamls as in Context section
- Follow the minimum upgrade instructions from https://backstage.github.io/upgrade-helper/?from=1.8.3&to=1.9.1 after running “yarn backstage-cli versions:bump” to v1.9.1
- Spin on local ie yarn dev - is working - ports 3000 FE 7007 BE
- Build a dockerfile and spin on local with docker-compose (emulates prod PG DB) - is working - ports 7007 for both FE and BE
- Deploy to aws ECS environment with both ports set to 443 - breaks - white screen - no errors or issues in the log
📃 Provide the context for the Bug.
app-config.yaml which combines with app.config.production.yaml
app:
baseUrl: ${BACKSTAGE_URL}:${BACKSTAGE_PORT}
permission:
enabled: true
backend:
baseUrl: ${BACKSTAGE_URL}:${BACKSTAGE_BACKEND_PORT}
listen:
port: ${BACKSTAGE_BACKEND_PORT}
auth:
keys:
- secret: ${AUTH_BITBUCKET_CLIENT_SECRET}
csp:
connect-src: ["'self'", 'http:', 'https:']
frame-src: ["'self'", 'http:', 'https:']
script-src: ["'self'", "'unsafe-eval'", "'unsafe-inline'", 'https://www.google-analytics.com/analytics.js', 'https://www.google-analytics.com/analytics_debug.js']
img-src:
# "'self'" and 'data' are from the backstage default but must be set since img-src is overriden
- "'self'"
- 'data:'
- "'self' *.google-analytics.com"
cors:
origin: ${BACKSTAGE_URL}:${BACKSTAGE_PORT}
methods: [GET, HEAD, PATCH, POST, PUT, DELETE]
credentials: true
app.config.production.yaml:
app:
baseUrl: ${BACKSTAGE_URL}:${BACKSTAGE_PORT}
backend:
baseUrl: ${BACKSTAGE_URL}:${BACKSTAGE_BACKEND_PORT}
listen:
port: ${BACKSTAGE_BACKEND_PORT}
🖥️ Your Environment
Disclosure, yes we have multiple package versions due to external plugins lagging in updates, if those were failing we’d expect same behaviour on local! Breaking backstage-cli info from v1.9.1:
node: v16.18.1
yarn: 1.22.17
cli: 0.22.0 (installed)
backstage: 1.9.1
Dependencies:
@backstage/app-defaults 1.0.10
@backstage/backend-app-api 0.2.4
@backstage/backend-common 0.17.0
@backstage/backend-plugin-api 0.2.0
@backstage/backend-tasks 0.4.0
@backstage/backend-test-utils 0.1.31
@backstage/catalog-client 1.2.0
@backstage/catalog-model 1.1.1, 1.1.4
@backstage/cli-common 0.1.11
@backstage/cli 0.22.0
@backstage/config-loader 1.1.7
@backstage/config 1.0.5
@backstage/core-app-api 1.3.0
@backstage/core-components 0.11.1, 0.11.2, 0.12.2
@backstage/core-plugin-api 1.0.6, 1.2.0
@backstage/dev-utils 1.0.10
@backstage/errors 1.1.4
@backstage/integration-aws-node 0.1.0
@backstage/integration-react 1.1.8
@backstage/integration 1.4.1
@backstage/plugin-analytics-module-ga 0.1.24
@backstage/plugin-api-docs 0.8.13
@backstage/plugin-app-backend 0.3.39
@backstage/plugin-auth-backend 0.17.2
@backstage/plugin-auth-node 0.2.8
@backstage/plugin-badges-backend 0.1.33
@backstage/plugin-badges 0.2.37
@backstage/plugin-bazaar-backend 0.2.2
@backstage/plugin-bazaar 0.2.2
@backstage/plugin-bitbucket-cloud-common 0.2.2
@backstage/plugin-catalog-backend-module-bitbucket-cloud 0.1.6
@backstage/plugin-catalog-backend 1.6.0
@backstage/plugin-catalog-common 1.0.9
@backstage/plugin-catalog-graph 0.2.25
@backstage/plugin-catalog-import 0.9.3
@backstage/plugin-catalog-node 1.3.0
@backstage/plugin-catalog-react 1.1.4, 1.2.3
@backstage/plugin-catalog 1.7.1
@backstage/plugin-events-node 0.2.0
@backstage/plugin-explore-common 0.0.1
@backstage/plugin-explore-react 0.0.24
@backstage/plugin-explore 0.3.44
@backstage/plugin-github-actions 0.5.13
@backstage/plugin-home 0.4.29
@backstage/plugin-jenkins-backend 0.1.29
@backstage/plugin-jenkins-common 0.1.11
@backstage/plugin-jenkins 0.7.12
@backstage/plugin-lighthouse 0.3.13
@backstage/plugin-org 0.6.3
@backstage/plugin-permission-backend 0.5.14
@backstage/plugin-permission-common 0.6.4, 0.7.2
@backstage/plugin-permission-node 0.7.2
@backstage/plugin-permission-react 0.4.8
@backstage/plugin-proxy-backend 0.2.33
@backstage/plugin-scaffolder-backend 1.9.0
@backstage/plugin-scaffolder-common 1.2.3
@backstage/plugin-scaffolder 1.9.1
@backstage/plugin-search-backend-module-elasticsearch 1.1.0
@backstage/plugin-search-backend-module-pg 0.5.0
@backstage/plugin-search-backend-node 1.1.0
@backstage/plugin-search-backend 1.2.0
@backstage/plugin-search-common 1.2.0
@backstage/plugin-search-react 1.3.1
@backstage/plugin-search 1.0.6
@backstage/plugin-sentry 0.4.6
@backstage/plugin-sonarqube-react 0.1.0
@backstage/plugin-sonarqube 0.6.1
@backstage/plugin-stack-overflow 0.1.9
@backstage/plugin-tech-radar 0.5.20
@backstage/plugin-techdocs-backend 1.5.0
@backstage/plugin-techdocs-module-addons-contrib 1.0.8
@backstage/plugin-techdocs-node 1.4.3
@backstage/plugin-techdocs-react 1.1.1
@backstage/plugin-techdocs 1.4.2
@backstage/plugin-user-settings 0.6.1
@backstage/release-manifests 0.0.8
@backstage/test-utils 1.2.3
@backstage/theme 0.2.16
@backstage/types 1.0.2
@backstage/version-bridge 1.0.3
Working v1.8.3 backstage-cli info:
node: v16.18.1
yarn: 1.22.17
cli: 0.22.0 (installed)
backstage: 1.8.3
Dependencies:
@backstage/app-defaults 1.0.8
@backstage/backend-app-api 0.2.3
@backstage/backend-common 0.16.0
@backstage/backend-plugin-api 0.1.4
@backstage/backend-tasks 0.3.7
@backstage/backend-test-utils 0.1.30
@backstage/catalog-client 1.1.2
@backstage/catalog-model 1.1.1, 1.1.3
@backstage/cli-common 0.1.10
@backstage/cli 0.21.1
@backstage/config-loader 1.1.6
@backstage/config 1.0.4
@backstage/core-app-api 1.2.0
@backstage/core-components 0.11.1, 0.11.2, 0.12.0
@backstage/core-plugin-api 1.0.6, 1.1.0
@backstage/dev-utils 1.0.8
@backstage/errors 1.1.3
@backstage/integration-react 1.1.6
@backstage/integration 1.4.0
@backstage/plugin-analytics-module-ga 0.1.22
@backstage/plugin-api-docs 0.8.11
@backstage/plugin-app-backend 0.3.38
@backstage/plugin-auth-backend 0.17.1
@backstage/plugin-auth-node 0.2.7
@backstage/plugin-badges-backend 0.1.32
@backstage/plugin-badges 0.2.35
@backstage/plugin-bazaar-backend 0.2.1
@backstage/plugin-bazaar 0.2.0
@backstage/plugin-bitbucket-cloud-common 0.2.1
@backstage/plugin-catalog-backend-module-bitbucket-cloud 0.1.5
@backstage/plugin-catalog-backend 1.5.1
@backstage/plugin-catalog-common 1.0.8
@backstage/plugin-catalog-graph 0.2.23
@backstage/plugin-catalog-import 0.9.1
@backstage/plugin-catalog-node 1.2.1
@backstage/plugin-catalog-react 1.1.4, 1.2.1
@backstage/plugin-catalog 1.6.1
@backstage/plugin-events-node 0.1.0
@backstage/plugin-explore-react 0.0.23
@backstage/plugin-explore 0.3.42
@backstage/plugin-github-actions 0.5.11
@backstage/plugin-home 0.4.27
@backstage/plugin-jenkins-backend 0.1.28
@backstage/plugin-jenkins-common 0.1.10
@backstage/plugin-jenkins 0.7.10
@backstage/plugin-lighthouse 0.3.11
@backstage/plugin-org 0.6.1
@backstage/plugin-permission-backend 0.5.13
@backstage/plugin-permission-common 0.6.4, 0.7.1
@backstage/plugin-permission-node 0.7.1
@backstage/plugin-permission-react 0.4.7
@backstage/plugin-proxy-backend 0.2.32
@backstage/plugin-scaffolder-backend 1.8.0
@backstage/plugin-scaffolder-common 1.2.2
@backstage/plugin-scaffolder 1.8.0
@backstage/plugin-search-backend-module-elasticsearch 1.0.4
@backstage/plugin-search-backend-module-pg 0.4.2
@backstage/plugin-search-backend-node 1.0.4
@backstage/plugin-search-backend 1.1.1
@backstage/plugin-search-common 1.1.1
@backstage/plugin-search-react 1.2.1
@backstage/plugin-search 1.0.4
@backstage/plugin-sentry 0.4.4
@backstage/plugin-sonarqube 0.5.0
@backstage/plugin-stack-overflow 0.1.7
@backstage/plugin-tech-radar 0.5.18
@backstage/plugin-techdocs-backend 1.4.1
@backstage/plugin-techdocs-module-addons-contrib 1.0.6
@backstage/plugin-techdocs-node 1.4.2
@backstage/plugin-techdocs-react 1.0.6
@backstage/plugin-techdocs 1.4.0
@backstage/plugin-user-settings 0.5.1
@backstage/release-manifests 0.0.7
@backstage/test-utils 1.2.2
@backstage/theme 0.2.16
@backstage/types 1.0.1
@backstage/version-bridge 1.0.2
👀 Have you spent some time to check if this bug has been raised before?
- I checked and didn’t find similar issue
🏢 Have you read the Code of Conduct?
- I have read the Code of Conduct
Are you willing to submit PR?
No, I don’t have time to work on this right now
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 18 (16 by maintainers)
I don’t understand why this problem is occurring, but I did find some information that helped me identify a workaround.
First, I observed that I don’t even need to upgrade from 1.8.0 to a higher backstage version to reproduce. Just doing a ‘clean’ to delete all node_modules and yarn.lock, then running
yarn installto get the latest versions of dependencies will reproduce the behavior.I tested a working Backstage deployment side-by-side in a browser with a failing deployment. Using React dev tools, I discovered that the
basenamevalue for some of the elements was different. In a failure scenario,basenamewas “/:443”, but it was simply “/” in a working scenario. Screenshots may help show the differences in the react DOM:Working scenario:
Failure scenario (although not shown in this screenshot, the tree in the left terminates at the
gtnode):In my
app-config.production.yamlfile, there are a few references where a 443 port would appear.I modified the config file to remove the
:${BACKSTAGE_PORT}references since the ${PROTOCOL} was already going to be HTTPS. An image rebuild and redeploy to ECS is now successful. I’ve also fully upgraded to 1.10.1 and it still successfully runs with the app-config.production.yaml file changes to remove:${BACKSTAGE_PORT}.An interesting observation is that running the image locally with the
:${BACKSTAGE_PORT}still included in the yaml file will run successfully. A key difference between running locally with this setting vs AWS ECS deployment is that I don’t have an SSL cert and am instead running withPROTOCOL=HTTPandBACKSTAGE_PORT=7007. It would be interesting to run the container locally with a self-signed certificate, but I haven’t had time to test that out.My
yarn backstage-cli infooutput after 1.10.1 upgrade in case it’s helpful: