backstage: 🐛 Bug Report: Backstage 1.24. breaks scaffolder

📜 Description

I’ve upgraded backstage to version 1.24 and if I try to execute a template 401 unauthorized is displayed.

👍 Expected behavior

It should run the template as it was in previous versions.

👎 Actual Behavior with Screenshots

When clicking on create the scaffolder shows an error 401 Unauthorized. No Description or anything.

👟 Reproduction steps

Go to Scaffolder and run any template.

📃 Provide the context for the Bug.

I think this occurence of the bug comes from the update in the auth system of backstage. After debugging a little bit, I could not find the exact location where the unauthorized request is made. But adding the following code to permissions plugin fixes this error.

http.addAuthPolicy({
    path: "/",
    allow: "unauthenticated"
 });

I think the request must be made by the scaffolder-backend-plugin somewhere, as no errors are shown in the network tab.

🖥️ Your Environment

OS: Darwin 23.4.0 - darwin/arm64 node: v18.17.1 yarn: 3.7.0 cli: 0.26.0 (installed) backstage: 1.24.0

👀 Have you spent some time to check if this bug has been raised before?

I checked and didn’t find similar issue

🏢 Have you read the Code of Conduct?

I have read the Code of Conduct

Are you willing to submit PR?

No, but I’m happy to collaborate on a PR with someone else

About this issue

Original URL
State: closed
Created 3 months ago
Reactions: 5
Comments: 37 (19 by maintainers)

Most upvoted comments

Apologies - we’ve noticed that there was some issues with the previous patch releases that’s causing us some headaches with another patch release, so I think we’re going to hold off on both the -next release and the patch release until tomorrow just to make sure we’re in a good state. Sorry for any inconveniences.

benjdlambert on Mar 26, 2024

Great - expect some releases coming your way later today 🎉

benjdlambert on Mar 26, 2024

Works for me 👍

dariozachow on Mar 26, 2024

A new release is hot off the press. Gonna close this issue as it should be fixed in 1.25.0 🎉 Nice little bonus release just in time for Easter holidays. 😅

benjdlambert on Mar 27, 2024

My guess is that https://github.com/backstage/backstage/pull/23827 should fix this. We’re gonna be doing a -next release today, wondering if it makes sense to try with that -next release and then we can ship that as a patch.

Also going to throw together a quick yarn patch to try locally to see if the fix in the PR works which might be a bit quicker than waiting for the -next release.

benjdlambert on Mar 26, 2024

Some more context: we have migrated to the new backend system, including the new auth services (following these docs). We require an authenticated user everywhere, and do not allow guest access.

Might be able to debug myself a bit more later today

fstoerkle on Mar 25, 2024

We encounter the same error. After upgrading to 1.24 and moving to Backstage’s built-in authentication, software templates fail with the exact same error as reported above. There is no info in the logs, nor any more output in the UI.

We also use a custom policy, however I see in the network console that the /api/permission/authorize comes back with 200 and { "id": "...", "result": "ALLOW" }. Furthermore our policy only restricts specific actions (mostly related to catalog), but none related to the scaffolder. Any unhandled permission is allowed by default. Therefore I doubt that the permission handling is related to this issue.

It’s the same for all templates, it does not even start the first step of the template but immediately errors with the 401.

fstoerkle on Mar 22, 2024

@drodil, I think that’s because you’ll need to pass an Authorization header to actually see the event stream response. My guess is that once you pass that Authorization token you will see something like @fstoerkle

event: log
data: {"id":5,"taskId":"292f074f-866f-4a9f-b622-0bd74ae7c3df","body":{"message":"Starting up task with 2 steps"},"type":"log","createdAt":"2024-03-26T13:53:46.183Z"}

event: completion
data: {"id":6,"taskId":"292f074f-866f-4a9f-b622-0bd74ae7c3df","body":{"message":"Run completed with status: failed","error":{"name":"ResponseError","message":"Request failed with 401 Unauthorized"}},"type":"completion","createdAt":"2024-03-26T13:53:46.196Z"}

Can confirm 👍

Parsifal-M on Mar 26, 2024

@freben actually there is no 401 network calls, the message just appears out of nowhere. The tasks endpoint even returns the task as processing but it’s actually failed. No communication at all in the eventstream.

drodil on Mar 26, 2024

Occurs after clicking on the create button after filling out all input steps, right as the scaffolder tasks starts.

dariozachow on Mar 26, 2024

Upgraded to 1.24.2, issue still persists.

If it makes a difference, our permission policy has this structure:

async handle(
  request: PolicyQuery,
  user?: BackstageIdentityResponse,
): Promise<PolicyDecision> {
  if (!user) {
    // we require authentication, so we always need a valid user here
    return { result: AuthorizeResult.DENY };
  }

  if (isPermission(...)) {
    ...
    return { result: AuthorizeResult.DENY };
  }

  if (isPermission(...)) {
    ...
    return { result: AuthorizeResult.DENY };
  }

  return { result: AuthorizeResult.ALLOW };
}

Edit: we don’t import/use @backstage/plugin-permission-backend-module-allow-all-policy

fstoerkle on Mar 25, 2024

@freben thanks, just upgraded to 1.24.1, the issue still persists for us (as expected, as we’re not using the unprocessed module)

fstoerkle on Mar 22, 2024