jsii: (aws_cdk): Py module error when using Bazel

So the workaround (at least in my minimal reproduction repo’s case) was a bit more complicated. Here were the changes I had to make to get this working:

https://github.com/psalvaggio-dl/bazel-cdk-repro/compare/master...aspect_rules

While this change set does provide a workaround, I don’t think it is a great solution for the following reasons:

1. Aspect’s rules are unofficial and are still in a pre-1.0 state
2. While this works in a toy example, I ran into other issues with Aspect’s ruleset in a more complicated repo. These mostly involved creating virtual environments via Bazel, which is a pretty common operation if you want your mono repo to work well with IDEs. As a result, I can’t adopt this workaround without breaking other things in my project.
3. This required replicating a list of packages in WORKSPACE.bazel (at least the ones you want to consume as wheels).
4. You now have 2 methods for depending on 3rd party libraries (one for wheels and one for non-wheels)
5. I have been able to use other namespace packages in my monorepo without running into this issue, which indicates that the issue is with CDK.

With respect to 5, I think this has given me a bit more insight into how this is supposed to look. I had another namespace package installed. When I look at the runfiles tree for my target, I see the following:

pip_<main_package>/
    __init__.py (empty)
   <main_package>/
     __init__.py
     ...
pip_<namespace_package>/
    __init__.py (empty)
    <main_package>/
      __init__.py
      <namespace_package>/
          ....

The contents of pip_<main_package>/<main_package>/__init__.py were as normal, but contained

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

to allow namespace packages and then the contents of pip_<namespace_package>/<main_package>/__init__.py were:

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

The two statements are I think identical, although they are written a bit different. If we compare this to the AWS CDK, the main package’s __init__.py is missing this line and the namespace packages’ __init__.py files at the aws_cdk level are missing entirely. I think replicating the behavior above might be the correct fix.

This issue also appears to be affecting versions prior to 2.51.0 if alpha modules are used. I can work around these issues by disabling testing on all code that uses alpha modules, but this is not ideal. I am still stuck on 2.50 because with 2.51, the issue propagates up to the main aws_cdk module, rendering all CDK code untestable.

Regarding the changes made by @psalvaggio-dl, it seems like the primary change for getting past the import failures are actually using the py_test offered by aspect_rules_py. I believe this is due to how they handle packages in their rules, which mitigates the namespace problem. In our case, it was a matter of adding:

http_archive(
    name = "aspect_rules_py",
    sha256 = "0cfd8d2492a1a63c3881abd759a54c4bb8a3555cb85971fb8de5cc8315546c42",
    strip_prefix = "rules_py-846674298d84663dcc109b6b20f5f8fc39aacfd4",
    url = "https://github.com/aspect-build/rules_py/archive/846674298d84663dcc109b6b20f5f8fc39aacfd4.tar.gz",
)

load("@aspect_rules_py//py:repositories.bzl", "rules_py_dependencies")

to WORKSPACE.bazel and changing the testing imports to:

#load("@rules_python//python:defs.bzl", "py_test")
load("@aspect_rules_py//py:defs.bzl", "py_test")

I still think this should be addressed , but I wanted to provide another workaround for anyone else stumbling into this problem.

Thanks, this might be a useful workaround for some people, but it is quite a major change to how Python is being built which many will not want to go with. I only mention this because I do not want an AWS developer who is not that familiar with Bazel to read your post and think “Ah there is an easy workaround, this does not need to be prioritised”, because that would not be right. I appreciate this is not what you were trying to say, and it is useful for the option you mention to be made more explicit.

+1 I have a customer facing this same issue and looking for guidance