pants: python_requirement not working in a plugin

Describe the bug I am writing a custom pants plugin, and I fail to import 3rd party dependencies. Here is my setup:

python_sources(
  name="sources",
  sources=["**/*.py"],
  dependencies=[":requirements", ":pants_requirements"]
)

python_requirement(
  name="requirements",
  requirements=[
    "GitPython==3.1.12",
    "semver==2.13.0",
  ]
)

pants_requirements(
  name="pants_requirements",
)

Pants version 2.15.0

OS MacOS

Additional info I haven’t found any pants plugin that consumes 3rd party dependencies. Is that supported?

Thanks!

About this issue

Original URL
State: closed
Created a year ago
Comments: 20 (20 by maintainers)

Commits related to this issue

Support per backend requirements (#19406) Closes #19183 Before this change, the only way backends could use external dependencies is by adding them to the `[GLOBAL].plugins` section. This chan... — committed to pantsbuild/pants by idan-at a year ago

Most upvoted comments

Thanks! sure, having a configuration for the file is probably easy once we have the entire flow 😃

idan-at on Jul 4, 2023

Changes needed to support this feature:

Track plugin requirements from loaded backends in the BuildConfiguration.Builder.

I don’t think we need to propagate it into the BuildConfiguration itself, at least for now. (that would be for introspection/help output).
Change plugin/backend load sequence so we load backends first, then resolve and load plugins. Now we resolve (install) plugins first, then load plugins and backends as one operation.

This is a potentially breaking change, but can be handled by retrying to load failing backends after the plugins have been loaded.

Broken down into affected areas:

The BuildConfiguration.Builder would need a new _plugins: set[str] for plugins required by the loaded plugins, along with a register_plugins(plugins: Iterable[str]) to add plugins to this set and the matching registered_plugins() to access it.

https://github.com/pantsbuild/pants/blob/361a665a8b2ec471dec82907f63e0f9d40d7ccca/src/python/pants/build_graph/build_configuration.py#L126-L127

This would then be used from the load_backend() method:

https://github.com/pantsbuild/pants/blob/361a665a8b2ec471dec82907f63e0f9d40d7ccca/src/python/pants/init/extension_loader.py#L130-L131

With a

    plugins = invoke_entrypoint("plugins")
    if plugins:
        build_configuration.register_plugins(plugins)

Then the load sequence needs refactoring to load the backends first, get a hold of the BuildConfiguration.Builder instance to append the registered plugins from the backends with those the PluginResolver picked up from the global options when calling plugin_resolver.resolve(...) to get the working_set.

https://github.com/pantsbuild/pants/blob/361a665a8b2ec471dec82907f63e0f9d40d7ccca/src/python/pants/init/options_initializer.py#L35-L45

The PluginsRequest would likely need a new field to provide these additional plugins to install, appended to this list here:

https://github.com/pantsbuild/pants/blob/361a665a8b2ec471dec82907f63e0f9d40d7ccca/src/python/pants/init/plugin_resolver.py#L59

Don’t hesitate to ask for more details or clarifications, or in case I’ve overlooked something.

kaos on Jun 28, 2023