backstage: Catalog 'Policy check failed' repeats without end

Expected Behavior

My App should not keep sending warning logs about Policy check failed on entities which do not exist anymore.

2022-07-23T02:33:48.665Z catalog warn Policy check failed; caused by Error: "metadata.namespace" is not valid; expected a string that is sequences of [a-z0-9] separated by [-], at most 63 characters in total but found "sdp.today". To learn more about catalog file format, visit: https://github.com/backstage/backstage/blob/master/docs/architecture-decisions/adr002-default-catalog-file-format.md type=plugin entity=component:sdp.today/ingress
2022-07-23T02:33:55.666Z catalog warn Policy check failed; caused by Error: "metadata.namespace" is not valid; expected a string that is sequences of [a-z0-9] separated by [-], at most 63 characters in total but found "sdp.today". To learn more about catalog file format, visit: https://github.com/backstage/backstage/blob/master/docs/architecture-decisions/adr002-default-catalog-file-format.md type=plugin entity=component:sdp.today/portal
2022-07-23T02:34:18.690Z catalog warn Policy check failed; caused by Error: "metadata.namespace" is not valid; expected a string that is sequences of [a-z0-9] separated by [-], at most 63 characters in total but found "sdp.today". To learn more about catalog file format, visit: https://github.com/backstage/backstage/blob/master/docs/architecture-decisions/adr002-default-catalog-file-format.md type=plugin entity=resource:sdp.today/gke
2022-07-23T02:34:29.706Z catalog warn Policy check failed; caused by Error: "metadata.namespace" is not valid; expected a string that is sequences of [a-z0-9] separated by [-], at most 63 characters in total but found "sdp.today". To learn more about catalog file format, visit: https://github.com/backstage/backstage/blob/master/docs/architecture-decisions/adr002-default-catalog-file-format.md type=plugin entity=resource:sdp.today/db
2022-07-23T02:34:30.583Z catalog warn Policy check failed; caused by Error: "metadata.namespace" is not valid; expected a string that is sequences of [a-z0-9] separated by [-], at most 63 characters in total but found "sdp.today". To learn more about catalog file format, visit: https://github.com/backstage/backstage/blob/master/docs/architecture-decisions/adr002-default-catalog-file-format.md type=plugin entity=system:sdp.today/sdp

Actual Behavior

My App keeps sending warning logs without end.

Steps to Reproduce

Set the namespace of an entity with bad format.
Delete the location entity of the entity in step 1.
Warning logs on Policy check failed never ends.

Context

Want to see a better surfacing and observability around processing loop errors as stated in the Errors section of The Life of an Entity.

Your Environment

Browser Information: Google Chrome 103.0.5060.114
Output of yarn backstage-cli info:

yarn run v1.22.19
$ /Users/tepisu/tmp/portal/node_modules/.bin/backstage-cli info
OS:   Darwin 21.5.0 - darwin/arm64
node: v16.15.1
yarn: 1.22.19
cli:  0.18.0 (installed)

Dependencies:
  @backstage/app-defaults                          1.0.4
  @backstage/backend-common                        0.14.1
  @backstage/backend-plugin-api                    0.1.0
  @backstage/backend-tasks                         0.3.3
  @backstage/catalog-client                        1.0.4
  @backstage/catalog-model                         1.1.0
  @backstage/cli-common                            0.1.9
  @backstage/cli                                   0.18.0
  @backstage/config-loader                         1.1.3
  @backstage/config                                1.0.1
  @backstage/core-app-api                          1.0.4
  @backstage/core-components                       0.10.0
  @backstage/core-plugin-api                       1.0.4
  @backstage/errors                                1.1.0
  @backstage/integration-react                     1.1.2
  @backstage/integration                           1.2.2
  @backstage/plugin-api-docs                       0.8.7
  @backstage/plugin-app-backend                    0.3.34
  @backstage/plugin-auth-backend                   0.15.0
  @backstage/plugin-auth-node                      0.2.3
  @backstage/plugin-catalog-backend-module-gitlab  0.1.5
  @backstage/plugin-catalog-backend                1.3.0
  @backstage/plugin-catalog-common                 1.0.4
  @backstage/plugin-catalog-graph                  0.2.19
  @backstage/plugin-catalog-import                 0.8.10
  @backstage/plugin-catalog-node                   1.0.0
  @backstage/plugin-catalog-react                  1.1.2
  @backstage/plugin-catalog                        1.4.0
  @backstage/plugin-github-actions                 0.5.7
  @backstage/plugin-kubernetes-backend             0.7.0
  @backstage/plugin-kubernetes-common              0.4.0
  @backstage/plugin-kubernetes                     0.7.0
  @backstage/plugin-org                            0.5.7
  @backstage/plugin-permission-backend             0.5.9
  @backstage/plugin-permission-common              0.6.3
  @backstage/plugin-permission-node                0.6.3
  @backstage/plugin-permission-react               0.4.3
  @backstage/plugin-proxy-backend                  0.2.28
  @backstage/plugin-scaffolder-backend             1.4.0
  @backstage/plugin-scaffolder-common              1.1.2
  @backstage/plugin-scaffolder                     1.4.0
  @backstage/plugin-search-backend-module-pg       0.3.5
  @backstage/plugin-search-backend-node            1.0.0
  @backstage/plugin-search-backend                 1.0.0
  @backstage/plugin-search-common                  1.0.0
  @backstage/plugin-search-react                   1.0.0
  @backstage/plugin-search                         1.0.0
  @backstage/plugin-tech-radar                     0.5.14
  @backstage/plugin-techdocs-backend               1.2.0
  @backstage/plugin-techdocs-module-addons-contrib 1.0.2
  @backstage/plugin-techdocs-node                  1.2.0
  @backstage/plugin-techdocs-react                 1.0.2
  @backstage/plugin-techdocs                       1.3.0
  @backstage/plugin-user-settings                  0.4.6
  @backstage/release-manifests                     0.0.5
  @backstage/test-utils                            1.1.2
  @backstage/theme                                 0.2.16
  @backstage/types                                 1.0.0
  @backstage/version-bridge                        1.0.1
✨  Done in 1.03s.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 18 (10 by maintainers)

Most upvoted comments

Having just investigated the refresh_state table it looks like this also impacts locations that have been deleted. Seems to me that when the refresh_state table is processed there should be a simple check to see if the entity_ref actually still exists in the catalog. If it doesn’t then perhaps the refresh_state entry could just be deleted?

gavinclarkeuk on Oct 14, 2022

After performing below commands on database backstage_plugin_catalog, the repeating error logs disappear.

select entity_ref from refresh_state where errors!='[]';

delete from refresh_state where errors!='[]';

Thank you for your help.

Closing this issue due to edge case.

tonyhsuclrous on Jul 26, 2022

We’ve been hitting this issue as well. I’m not convinced it is an edge case so think we should consider reopening. It seems this will happen if a bad catalog manifest is checked in, then repaired. We’re seeing examples where a component name has spaces in, then gets fixed. For these entities the location typically wouldn’t get removed, so I don’t think the issue will naturally resolve itself.

gavinclarkeuk on Oct 14, 2022

Perfect - glad this worked out for you, gonna tag in @freben in here anyways, as this could be a sign of something else not working as expected.

benjdlambert on Jul 26, 2022