pipelines: [sdk] Containerized Python Component module not found error

There is a bug when building a containerized Python component that happens (at least) in the case when the longest path of the import graph ending at the component involves >2 modules.

Environment

KFP SDK 2.0.0-beta.6

Steps to reproduce

For example:

# component.py
from module_one import one
from kfp import dsl

@dsl.component
def comp(): ...

# module_one.py
from module_two import two
one = 1

# module_two.py
two = 2

Then: kfp component build .

You get a No module named error.

Expected result

Should build without an error.

Materials and Reference

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 18 (14 by maintainers)

Commits related to this issue

fix(sdk): fixes module not found error for containerized python components. Fixes #8385 (#9157) * Add module directory to sys.path * Add nested module imports unit test * Add release note to re... — committed to rd-pong/pipelines by Mithil467 a year ago
fix(sdk): fixes module not found error for containerized python components. Fixes #8385 (#9157) * Add module directory to sys.path * Add nested module imports unit test * Add release note to re... — committed to HumairAK/data-science-pipelines by Mithil467 a year ago

Most upvoted comments

Hey @connor-mccarthy , I understand the script nature - I also tend to run component as script to try (and I need to run it from project_dir for it to work). However, as the project grows and let’s say you want to have proper tests (under project_dir/test) and run them simply by pytest tests from project_dir, you will run into problem (using relative imports).

I might have another idea for a fix though 😃. What about adding components_directory (cli) argument from build function to the path? Like this:

def build(components_directory: str, component_filepattern: str, engine: str,
          kfp_package_path: Optional[str], overwrite_dockerfile: bool,
          build_image: bool, platform: str, push_image: bool):
    """Builds containers for KFP v2 Python-based components."""

    sys.path.append(components_directory)

    if build_image and engine != 'docker':

Then I can run kfp build wherever with argument pointing to project_dir and the absolute import should (tested locally) work. This would solve absolute imports, relative should not be affected. I cannot think of case when this would cause an issue, also shouldn’t be maintenance heavy. What do you think?

b4sus on May 16, 2023

This bug seems to have been introduced in kfp==1.8.12

pbalm on May 15, 2023

Thanks, @Davidnet. Some links:

connor-mccarthy on Jan 31, 2023