backstage: 🐛 Bug Report: Github Discovery Catalogs via Wildcard config

📜 Description

Performance has been inconsistent between manual catalog upload via /catalog-import and Github Discovery feature with Wild Card entry.

I have a file with 48 OpenAPI catalogues injected into it. When performing import on the WebUI, it validated and imported successfully.

However, we have automated processes in the background which involves auto-gen catalog-info files and when Backstage background process performs catalog discovery, I get an error and thus failed to Add/Update the components.

I noticed the github URL path that its trying to call to retrieve the artifact is different from what is defined in the $text.

FYI: I’m currently running on 2 instances of Backstage in my environment.

👍 Expected behavior

Catalog added/updated successfully.

👎 Actual Behavior with Screenshots

Console log

"[2m2023-02-03T02:00:46.189Z[22m [34mcatalog[39m [33mwarn[39m Processor PlaceholderProcessor threw an error while preprocessing; caused by Error: Placeholder $text could not read location https://github.com/org/file.yaml, Error: Incorrect URL: https://github.com/org//tree/-/specs/file.yaml, Error: Invalid GitHub URL or file path [36mtype[39m=plugin [36mentity[39m=api:default/<api_name>"

👟 Reproduction steps

Have a yaml file containing over 40 catalogs. (Note: We have generated over 50 catalogs but noticed that it would result in timeout, so we capped it at 50)

Config

wildcardProviderId:
        organization: '<orgname>'
        catalogPath: '/backstage/**/*.yaml'

Sample Catalog

...
spec:
  system: <system_name>
  owner: <team_name>
  type: openapi
  lifecycle: experimental
  definition:
    $text: https://github.com/org/repo/blob/main/specs/file.yaml

Github Discovery

Configure Github discovery with wildcard matching.
Run backstage and let the GH discovery process run.

Via WebUI

Go to ‘catalog-import’ page.
Import the generated file.
Notice that validation is successful and ready to import

📃 Provide the context for the Bug.

No response

🖥️ Your Environment

OS: Darwin 22.3.0 - darwin/x64 node: v16.13.0 yarn: 1.22.19 cli: 0.20.0 (installed) backstage: 1.7.2

👀 Have you spent some time to check if this bug has been raised before?

I checked and didn’t find similar issue

🏢 Have you read the Code of Conduct?

I have read the Code of Conduct

Are you willing to submit PR?

None

About this issue

Original URL
State: closed
Created a year ago
Comments: 15 (3 by maintainers)

Most upvoted comments

@mscam - That’s great! I was curious about the issue and was about to suggest another possibility in the config that I experienced when my org had templates in different repos and we had to add another rule for Location as it was not importing the templates.

Good that you have resolved the issue.

darrenyung on Jun 9, 2023

definitely yes. I would try providing a Location entity that acts as an “index” for all service entities 💯 maybe it helps

mrs83 on Jun 9, 2023

@mscam - I’m not near my work pc atm but from memory, I’ve increased mine to 1 hour with frequency level at 1 hour and 10 minutes.

In terms of data, I have nearly triple the amount that you have (including APIs and Documentations). I haven’t gone into multi events for processing yet. IIRC, its the latest feature that is available for implementation.

UPDATE: If you manual import the catalog with success for location entity but the component is missing, I think there’s an actual error happening behind the scenes here? Are you able to find any logs related to the missing component? Apologies, AWS integration for Backstage isn’t my forte.

darrenyung on Jun 7, 2023