beam: [Bug]: Dependencies from private repositories unable to be seen

What happened?

Running a gcp dataflow, using the python sdk 2.44.0 I can no longer access my private repositories. It works on 2.43.0

My set up is as follows:

# These image repos have access to my private repos
FROM my.private.image-repo/python:3.8-slim-builder as builder
FROM my.private.image-repo/python:3.8-slim
COPY --from=apache/beam_python3.8_sdk:2.43.0 /opt/apache/beam /opt/apache/beam
# this virtual env has all the dependencies I need pre-installed on it, including ones from private repos
COPY --from=builder $VENV_PATH $VENV_PATH
ENTRYPOINT ["/opt/apache/beam/boot"]

This is my run command:

poetry run python -m projname.main \
  --project="$PROJECT_ID" \
  --runner=DataFlowRunner \
  --temp_location=gs://"$BUCKET_NAME"/temp \
  --region="$REGION" \
  --job_name="$JOB_NAME" \
  --setup_file=./setup.py \
  --subnetwork https://www.googleapis.com/compute/v1/projects/"$PROJECT_ID"/regions/"$REGION"/subnetworks/"$SUBNET" \
  --experiment=use_runner_v2 \
  --sdk_container_image=$IMAGE_NAME \
  --template_location=gs://"$BUCKET_NAME"/templates/"$JOB_NAME" \

Checking my dataflow worker logs it fails to see my private repos:

ERROR: Could not find a version that satisfies the requirement package-i-want<3.0.0,>=2.2.0 (from name-of-my-dataflow) (from versions: none)

I think this is the culprit PR: https://github.com/apache/beam/pull/23684/files#diff-cc1f3d7f808c692a6102847bec78809f2e4350c5ee34278100ce0f55d8c23d68R234

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 19 (14 by maintainers)

Commits related to this issue

Most upvoted comments

Glad to hear you resolved the issue.