tensorflow: tensorflow.keras IDE auto-completion fails

This has been a problem for a while now, see #53144. It would be really nice if this would be fixed. It can’t be that everyone who uses an IDE and code completion, which should be normal by now, has to hack around in his installation.

A side question - does everyone just use Notepad++? This has been a persistent issue with VSCode and Pycharm since at least 2017. I’m baffled that it’s not fixed, but I don’t understand it well enough to fix it myself.

I would happily just buy a different IDE if there’s one that supports code completion and doesn’t break on every new TF release.

As far as I know, this problem appeared from 2.6 and lasted until 2.9, if it is not solved, I am willing to use pytorch in the future.

We are working on a new keras import mechanism, which I expect would solve this issue along the way. Will post here when it is ready.

Related changes: https://github.com/keras-team/keras/blob/master/pip_build.py

Mimic my response from other thread in https://github.com/tensorflow/tensorflow/issues/53271#issuecomment-1239696497

=======================================

Thanks for the information.

Based on my rough knowledge for auto completion, I think the issue probably caused by difference between package/directory/module.

Most of the IDE will try to auto complete by the package tree structure, so if there is a directory with init under tensorflow call “keras”, then it will resolve correctly when you do “from tensorflow.keras.xxxx import yyy”. This was working before 2.7 since Keras was part of the Tensorflow PIP package.

Note that the “import tensorflow as tf; tf.keras.xxx.yyy” is resolved differently. tf.keras is firstly resolved into the logic in the tensorflow/init.py, and then the keras becomes the “keras.apis._v2.keras”, which resolved correctly to the keras PIP package.

Before TF 2.7, I think we did some manual mimic for the keras APIs to tensorflow/keras folder from tensorflow/python/keras/apis, so that the package level auto completion works, but that is not the case anymore after 2.7 since the keras is a standalone package.

As a quick test on my local, I manually created a symlink under venv/lib/python3.10/site-packages/tensorflow by ln -s ../keras/api/_v2/keras/ keras (assuming you have tensorflow and keras pip package installed properly), and then the IDE can resolve the package level auto completion correctly.

I am not sure if this fix can be applied to TF direclty, since the symlink is not well supported on windows, and might be in different location if the package is installed differently.

@Yuri-Su site-packages\tensorflow\keras is removed in tf 2.8 and tf 2.9

so you can’t import it like that.

The temporary solution can only be:

before: from tensorflow.keras.callbacks import EarlyStopping after: from tensorflow.python.keras.callbacks import EarlyStopping

or: import tensorflow as tf then tf.keras.~

@haifeng-jin a change in keras will not solve anything. The problem is in tensorflow and the problem is still there in the latest pypi release 2.13.0 as nothing changed in the lazy import in tensorflow.__init__.py

_keras_module = "keras.api._v2.keras"
_keras = _LazyLoader("keras", globals(), _keras_module)
_module_dir = _module_util.get_parent_dir_for_name(_keras_module)
if _module_dir:
  _current_module.__path__ = [_module_dir] + _current_module.__path__
setattr(_current_module, "keras", _keras)

and most IDEs thus can’t find tensorflow.keras in during code inspection. @bermeitinger-b which version are you referring to that would change this?

The problem will persist until either

• tensorflow does a normal from keras.api._v2 import keras (can anyone elaborate why this isn’t done?) or
• All IDEs adapt to understand that particular tensorflow specific import syntax.

I don’t find it realistic to require all dev tools and IDEs to adapt to that design choice. Imho it would be very beneficial for tensorflow in the future if the tensorflow dev team heeded the requests of hitherto loyal tensorflow users and responded to this somewhat annoying issue.

@Yuri-Su try this:

before:

# Explicitly import lazy-loaded modules to support autocompletion.
# pylint: disable=g-import-not-at-top
if _typing.TYPE_CHECKING:
  from tensorflow_estimator.python.estimator.api._v2 import estimator
# pylint: enable=g-import-not-at-top

after:

# Explicitly import lazy-loaded modules to support autocompletion.
# pylint: disable=g-import-not-at-top
if _typing.TYPE_CHECKING:
  from tensorflow_estimator.python.estimator.api._v2 import estimator
  from keras.api._v2 import keras
# pylint: enable=g-import-not-at-top

Thank you very much for your help, but still can’t solve the problem. Bye tensorflow, hello pytorch！！！

Just copy the folder keras from site-packages/tensorflow/python to site-packages/tensorflow. And the auto completion will start working perfectly

@alkatar21 Simply ‘pylance’ version 2023.2.10 resolved the issue.

I don’t know much because I’ve recently started studying, but I hope the problems that have occurred will be solved as soon as possible.

Thanks for your help. Have a great day 😃

This still seems to be an issue with TF 2.11.0 Python 3.9.4 and VSCode from tensorflow.keras import applications generates a warning that tensorflow.keras could not be resolved

Is it just not recommended to use this form of import now and rather use import tensorflow as tf; from tf.keras import ... ?

@Yuri-Su try this:

before:

# Explicitly import lazy-loaded modules to support autocompletion.
# pylint: disable=g-import-not-at-top
if _typing.TYPE_CHECKING:
  from tensorflow_estimator.python.estimator.api._v2 import estimator
# pylint: enable=g-import-not-at-top

after:

# Explicitly import lazy-loaded modules to support autocompletion.
# pylint: disable=g-import-not-at-top
if _typing.TYPE_CHECKING:
  from tensorflow_estimator.python.estimator.api._v2 import estimator
  from keras.api._v2 import keras
# pylint: enable=g-import-not-at-top

This issue has resurfaced in PyCharm with TF/Keras 2.15.

@GF-Huang This is the case because tf.keras is not yet resolved properly (resolves to LazyLoader which resulsts in Any) in 2.9.1. There is a commit in nightly (#54104) that will probably be in 2.10 that fixes this case.

let me install vscode and play with it. Not sure which part triggers this difference.

@qlzh727 This version actually fixes one bug, but various others not. I tested it in the current docker container:

>>> tf.__version__
'2.10.0-dev20220531'

With #54104 the bug originally reported in #53144 is fixed, that the autocompletion for tf.keras.<autocompletion> does not work.

However, it does not fix the problem described in https://github.com/tensorflow/tensorflow/issues/53144#issuecomment-1133027918. For example, it still does not work from tensorflow.keras import Model.

In the following a test protocol, which should help to solve this. It does not show all possible problems, but should help to solve all of them if implemented well (I used VSCode 1.67.2 with Pylance v2022.5.3 and the docker container for each version):

"""Testing tensorflow.keras autocompletion."""
# pyright: basic, reportMissingTypeStubs=false

''' Test 1: tf.keras.<autocompletion>'''
import tensorflow as tf

tf.keras.Model()
tf.keras.preprocessing.image_dataset_from_directory(None)

# Broken in 2.5.0, 2.7.0, 2.9.0 (resolves to (keras: LazyLoader) -> everything resolves to Any == No autocompletion)
# Works in 2.10.0-dev20220531

''' Test 2: from tf import keras -> keras.<autocompletion>'''
from tensorflow import keras

keras.Model()
keras.preprocessing.image_dataset_from_directory(None)

# Broken in 2.5.0, 2.7.0, 2.9.0 (resolves to (keras: LazyLoader) -> everything resolves to Any == No autocompletion)
# Broken in 2.10.0-dev20220531 (keras is resolved to tensorflow/__init__.py)

''' Test 3: from tensorflow.keras import Model'''
from tensorflow.keras import Model
from tensorflow.keras.layers import Dense

# Works in 2.5.0, 2.7.0
# Broken in 2.9.0, 2.10.0-dev20220531 (Import "tensorflow.keras" could not be resolved -> Model: Unknown)

In 2.7.0 the fix for all the issues was to add the following to tensorflow/__init__.py#L394-L400:

# Explicitly import lazy-loaded modules to support autocompletion.
# pylint: disable=g-import-not-at-top
if not _six.PY2:
  import typing as _typing
  if _typing.TYPE_CHECKING:
    from tensorflow_estimator.python.estimator.api._v2 import estimator
+   from keras.api._v2 import keras
# pylint: enable=g-import-not-at-top

For 2.10/nightly i do not know the best solution:

• I do not completly understand the problem for Test 2, but I think it would resolve to tensorflow/keras/__init__.py if the package still exists (at least it does in 2.7 with the fix).
• I think the reason for fail of Test 3 is that the tensorflow.keras package is no longer generated in 2.8 and onwards.

A quick additional question, if there would be a fix in a timely manner, but presumably we would still have to wait until the next release for this to be fixed and there would be no bugfix-release for this to work again?

@mshijie You tested the wrong tensorflow version. That it does not work there is known.

i made it autocompleting by changing import keras section line 397 in init.py tf 2.15 to:

# Lazy-load Keras v2/3.
import typing as _typing
if _typing.TYPE_CHECKING:
  from keras.api._v2 import keras
else:
  _tf_uses_legacy_keras = (
          _os.environ.get("TF_USE_LEGACY_KERAS", None) in ("true", "True", "1"))
  setattr(_current_module, "keras", _KerasLazyLoader(globals()))
  _module_dir = _module_util.get_parent_dir_for_name("keras._tf_keras.keras")
  _current_module.__path__ = [_module_dir] + _current_module.__path__
  if _tf_uses_legacy_keras:
    _module_dir = _module_util.get_parent_dir_for_name("tf_keras.api._v2.keras")
  else:
    _module_dir = _module_util.get_parent_dir_for_name("keras.api._v2.keras")
  _current_module.__path__ = [_module_dir] + _current_module.__path__

can someone please tell me if it’s going to crash

@OlegNovosad @weidler , Thanks for the reply! Unfortunately, I don’t think we can do much about the tf.keras import path.

On the Keras side, we have implemented a new mechanism for API exporting, which works for both Keras 2 (since TF 2.13) & 3 for code completion. It creates real directories and files under the import path instead of just adding attributes. So it should work with all code completion tools in IDEs.

A note for new users: Please import keras directly in order to use the code completion feature.

@weidler @OlegNovosad Are you using import keras or from tensorflow import keras? import keras should always work for autocompletion.

Thank you! Worked for me. I’m on windows 11, VS Code, python 3.10.8, tensorflow 2.10.0

Edit: I had a slightly different keras location mklink /D keras …\keras\api_v2\keras

Man, I can’t believe this is still an issue 😒

Symlink works fine on Windows, at least as of TF 2.9. Windows users - just open an elevated cmd prompt, go to your site-packages\tensorflow directory and type:

mklink /D keras …\keras\api_v2\keras

If for whatever reason you can’t make a symlink, a junction works just as well:

mklink /J keras C:\fullpath\site-packages\keras\api_v2\keras

@qlzh727 This issue is still happened on v2.10.0 https://github.com/tensorflow/tensorflow/issues/53271#issuecomment-1239103757