beam: [Bug]: Dependencies from private repositories unable to be seen
What happened?
Running a gcp dataflow, using the python sdk 2.44.0 I can no longer access my private repositories. It works on 2.43.0
My set up is as follows:
# These image repos have access to my private repos
FROM my.private.image-repo/python:3.8-slim-builder as builder
FROM my.private.image-repo/python:3.8-slim
COPY --from=apache/beam_python3.8_sdk:2.43.0 /opt/apache/beam /opt/apache/beam
# this virtual env has all the dependencies I need pre-installed on it, including ones from private repos
COPY --from=builder $VENV_PATH $VENV_PATH
ENTRYPOINT ["/opt/apache/beam/boot"]
This is my run command:
poetry run python -m projname.main \
--project="$PROJECT_ID" \
--runner=DataFlowRunner \
--temp_location=gs://"$BUCKET_NAME"/temp \
--region="$REGION" \
--job_name="$JOB_NAME" \
--setup_file=./setup.py \
--subnetwork https://www.googleapis.com/compute/v1/projects/"$PROJECT_ID"/regions/"$REGION"/subnetworks/"$SUBNET" \
--experiment=use_runner_v2 \
--sdk_container_image=$IMAGE_NAME \
--template_location=gs://"$BUCKET_NAME"/templates/"$JOB_NAME" \
Checking my dataflow worker logs it fails to see my private repos:
ERROR: Could not find a version that satisfies the requirement package-i-want<3.0.0,>=2.2.0 (from name-of-my-dataflow) (from versions: none)
I think this is the culprit PR: https://github.com/apache/beam/pull/23684/files#diff-cc1f3d7f808c692a6102847bec78809f2e4350c5ee34278100ce0f55d8c23d68R234
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 19 (14 by maintainers)
Commits related to this issue
- Document a breaking change that was raised in #25085 (#26471) — committed to apache/beam by tvalentyn a year ago
- Document a breaking change that was raised in #25085 (#26471) — committed to akvelon/beam by tvalentyn a year ago
Glad to hear you resolved the issue.