bitbucket-branch-source-plugin: Bitbucket users being blocked on "Scan Organization Folder" and pipelines

Your checklist for this issue

• Jenkins version 2.245

• Plugin version 2.9.0

• Bitbucket cloud 7.2.0

• Bitbucket server and version

Description

Hello Since around 3 weeks ago we started having a strange behaviour on bitbucket checkout, the users started being blocked in the scanning organization selection with the following error:

ERROR: [Mon Jul 27 12:36:33 CEST 2020] Could not refresh actions for navigator com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator@248cc3ec
com.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: HTTP request error. Status: 401: .
HttpResponseProxy{HTTP/1.1 401  [Content-Type: application/json;charset=UTF-8, Date: Mon, 27 Jul 2020 10:36:33 GMT, vary: accept-encoding, WWW-Authenticate: Basic realm="Atlassian Bitbucket", X-AREQUESTID: @IC7BIZx756x5439124x0, X-ASEN: SEN-9644435, transfer-encoding: chunked, Connection: keep-alive] org.apache.http.client.entity.DecompressingEntity@635d13df}
	at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:862)
	at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getTeam(BitbucketServerAPIClient.java:734)
Caused: java.io.IOException: I/O error when accessing URL: /rest/api/1.0/projects/OW-APPS-SAMS
	at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getTeam(BitbucketServerAPIClient.java:739)
	at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator.retrieveActions(BitbucketSCMNavigator.java:539)
	at jenkins.scm.api.SCMNavigator.fetchActions(SCMNavigator.java:332)
	at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:440)
	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
	at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
	at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:970)
	at hudson.model.ResourceController.execute(ResourceController.java:97)

Sometimes the error that appears is related to the captcha selection. This means that the username/password were entered incorrectly too many times. Thing here is that if we manually access bitbucket, the same password that is configured on Jenkins works with no issue and we can unblock the user. This seems to be random and we are not sure why it started happening, we’ve already rolled back plugin and jenkins versions with no luck. We have also tried using the passwords in CASC files and using the AWS secrets manager and again the behaviour is the same. Asking the bitbucket team they showed us that the user/password were being received incorrectly, meaning that Jenkins/bitbucket plugin is sending the incorrect information.

Not sure what to look for next. Anyone have the same issue? Thanks