beam: [Bug]: Dependencies from private repositories unable to be seen

What happened?

Running a gcp dataflow, using the python sdk 2.44.0 I can no longer access my private repositories. It works on 2.43.0

My set up is as follows:

# These image repos have access to my private repos
FROM my.private.image-repo/python:3.8-slim-builder as builder
FROM my.private.image-repo/python:3.8-slim
COPY --from=apache/beam_python3.8_sdk:2.43.0 /opt/apache/beam /opt/apache/beam
# this virtual env has all the dependencies I need pre-installed on it, including ones from private repos
COPY --from=builder $VENV_PATH $VENV_PATH
ENTRYPOINT ["/opt/apache/beam/boot"]

This is my run command:

poetry run python -m projname.main \
  --project="$PROJECT_ID" \
  --runner=DataFlowRunner \
  --temp_location=gs://"$BUCKET_NAME"/temp \
  --region="$REGION" \
  --job_name="$JOB_NAME" \
  --setup_file=./setup.py \
  --subnetwork https://www.googleapis.com/compute/v1/projects/"$PROJECT_ID"/regions/"$REGION"/subnetworks/"$SUBNET" \
  --experiment=use_runner_v2 \
  --sdk_container_image=$IMAGE_NAME \
  --template_location=gs://"$BUCKET_NAME"/templates/"$JOB_NAME" \

Checking my dataflow worker logs it fails to see my private repos:

ERROR: Could not find a version that satisfies the requirement package-i-want<3.0.0,>=2.2.0 (from name-of-my-dataflow) (from versions: none)

I think this is the culprit PR: https://github.com/apache/beam/pull/23684/files#diff-cc1f3d7f808c692a6102847bec78809f2e4350c5ee34278100ce0f55d8c23d68R234

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner