tensorflow: PyCharm doesn't resolve anything under tensorflow.keras

hi, @markorakita It’s a naming conflict issue. try modify the file site-packages/tensorflow/__init__.py near line 387.

before:

_keras_module = "keras.api._v2.keras"
keras = _LazyLoader("keras", globals(), _keras_module)
_module_dir = _module_util.get_parent_dir_for_name(_keras_module)
if _module_dir:
  _current_module.__path__ = [_module_dir] + _current_module.__path__
setattr(_current_module, "keras", keras)

after:

import typing as _typing
if _typing.TYPE_CHECKING:
  from keras.api._v2 import keras
else:
  _keras_module = "keras.api._v2.keras"
  keras = _LazyLoader("keras", globals(), _keras_module)
  _module_dir = _module_util.get_parent_dir_for_name(_keras_module)
  if _module_dir:
    _current_module.__path__ = [_module_dir] + _current_module.__path__
  setattr(_current_module, "keras", keras)

it work for me.

I’m using Emacs and I have same issue in TF2.8. For now, my solution is:

import typing

from tensorflow import keras

if typing.TYPE_CHECKING:
    from keras.api._v2 import keras

It works good for my environment.

Sorry I thought I replied to this.

You can write code and run it and it works, but you don’t have autocomplete nor you can go to declaration of any keras function. It’s like writing code in notepad 😃

The reason why this does not work has nothing to do with pycharm, but one think is that the tensorflow.keras package is no longer generated in 2.8 and onwards, which it used to be, as you can see from the header of all files from the package in 2.7.0:

# This file is MACHINE GENERATED! Do not edit.
# Generated by: tensorflow/python/tools/api/generator/create_python_api.py script.

Maybe @markorakita can adapt the title and maybe also the description?

Since there has been no activity from anyone from tensorflow since January 2022, the general question is how to get someone to pay attention to the issue? Could it be worth to open a new issue and give it upvotes from many people here? This probably affects everyone who is used to autocompletion/an IDE and doesn’t use stubs for tf from somewhere?

I don’t know what else has changed since then, but maybe generation just needs to be reactivated?

Edit: This does not solve the problem originally mentioned in this issue with tf.keras.<autocomplete>, but at least the autocompletion for from tensorflow.keras import <autocomplete>.

any updates on the issue? as mentioned now it is like coding in notepad

Any update on this? It’s been a minute and neither this issue nor #53271 have received any updates since Dec

As a working workaround, you can do the following: (no need to recompile).

Use the fix from https://github.com/tensorflow/tensorflow/issues/53144#issuecomment-985179600 (you need to modify it a bit for the current versions.)

Then, in your code:

import tensorflow as tf
keras = tf.keras
K = keras.backend # example

You have to “Invalidate Caches & Restart” in PyCharm and let it regenerate the indices which will take a few minutes.

With this, I have full autocomplete as it should be.

The same issue is still happening with 2.8.0, 2.8.1, 2.9.0, and any combination with Python/Conda versions 3.7, 3.8, 3.9, or 3.10.

Since this issue is a duplicate as https://github.com/tensorflow/tensorflow/issues/56231, I am going to close this one, so that everyone can track on same issue.

@markorakita Everything is stirred up, for the time being, only part of it can be solved first, and I hope that someone will solve the problem as soon as possible.

When a new 2.9.0-rc is being cut, and this story isn’t in the bug fixes section 🥲

@vanakema I already did that but no luck… Any other idea?

Oh I thought I saw someone else say that for whatever reason that style import doesn’t work with the fix. Do you get autocompletes when you write something like tf.keras.layers in your actual code?

To everyone that’s still couldn’t make it work. Try this, I just tweaked the popular solution on this thread:

modify the file site-packages/tensorflow/__init__.py near line 387 From:

_keras_module = "keras.api._v2.keras"
keras = _LazyLoader("keras", globals(), _keras_module)
_module_dir = _module_util.get_parent_dir_for_name(_keras_module)
if _module_dir:
  _current_module.__path__ = [_module_dir] + _current_module.__path__
setattr(_current_module, "keras", keras)

To:

Note that I used _keras instead of keras

import typing as _typing
if _typing.TYPE_CHECKING:
  from keras.api._v2 import keras
else:
  _keras_module = "keras.api._v2.keras"
  _keras = _LazyLoader("keras", globals(), _keras_module)
  _module_dir = _module_util.get_parent_dir_for_name(_keras_module)
  if _module_dir:
    _current_module.__path__ = [_module_dir] + _current_module.__path__
  setattr(_current_module, "keras", _keras)

And then make sure to copy folder /site-packages/keras/ into /site-packages/tensorflow/

After all of that, invalidate and restart your IDE

This works on my setup using Conda

It worked perfectly on PyCharm IDE with TensorFlow 2.13.0! Finally from tensorflow.keras.layers import Dense, Flatten, Conv2D is successfully recognized by IDE.

With a small change, instead of copy folder /site-packages/keras/ into /site-packages/tensorflow/, I use symbolic link ln -s <full-path-to-site-package>/site-packages/keras <full-path-to-site-package>/tensorflow/keras

To everyone that’s still couldn’t make it work. Try this, I just tweaked the popular solution on this thread:

modify the file site-packages/tensorflow/__init__.py near line 387 From:

_keras_module = "keras.api._v2.keras"
keras = _LazyLoader("keras", globals(), _keras_module)
_module_dir = _module_util.get_parent_dir_for_name(_keras_module)
if _module_dir:
  _current_module.__path__ = [_module_dir] + _current_module.__path__
setattr(_current_module, "keras", keras)

To:

Note that I used _keras instead of keras

import typing as _typing
if _typing.TYPE_CHECKING:
  from keras.api._v2 import keras
else:
  _keras_module = "keras.api._v2.keras"
  _keras = _LazyLoader("keras", globals(), _keras_module)
  _module_dir = _module_util.get_parent_dir_for_name(_keras_module)
  if _module_dir:
    _current_module.__path__ = [_module_dir] + _current_module.__path__
  setattr(_current_module, "keras", _keras)

And then make sure to copy folder /site-packages/keras/ into /site-packages/tensorflow/

After all of that, invalidate and restart your IDE

This works on my setup using Conda

This workaround with the .python. addition works to have autocompletion and syntax checking (in some instances but not all).

However, it does not help during runtime:

import tensorflow.python.keras as keras
l = keras.layers.LayerNormalization()
>>> AttributeError: module 'tensorflow.python.keras.layers' has no attribute 'LayerNormalization'

When not using .python.:

import tensorflow.keras as keras
l = keras.layers.LayerNormalization()
# works

It’s not a good workaround.

(TensorFlow 2.8.0)

I can confirm that this issue still exits in latest PyCharm (2021.3.3 professional edition), python 3.10, TF 2.8. The workaround from cpuimage, then invalidating caches and restarting, did not fix the issue.

Just copy the folder keras from site-packages/tensorflow/python to site-packages/tensorflow. And the auto completion will start working perfectly

Thank you for your idea! However, it only works for some. It looks like site-packages/tensorflow/python/keras is different from site-packages/keras.

After applying your method, I have the following results:

set_random_seed is not resolved when calling from the copied folder

Just copy the folder keras from site-packages/tensorflow/python to site-packages/tensorflow. And the auto completion will start working perfectly

Sorry for the very long wait.

This issue should be addressed by https://github.com/tensorflow/tensorflow/pull/54104, which is landed on tf-nightly (but didn’t included in tf 2.9). I was able to verify this locally with the latest tf-nightly pip package.

Could any of you test with latest tf-nightly on the IDE and see if it fix the issue? Thanks.

Can we run the generation manually as a workaround without having to compile TF from the source?

@winthropharvey @Tolure @nicolasperezdeo site-packages\tensorflow\keras is removed in tf 2.8.

The temporary solution can only be:

before: from tensorflow.keras.callbacks import EarlyStopping after: from tensorflow.python.keras.callbacks import EarlyStopping

I am also having the same issue. After updating the __init__.py however, Pycharm does not pick up the changes… I am using tensorflow-macos==2.8.0 in M1.

You’ll probably need to invalidate caches and restart. Your indexes are likely still from before the change. PyCharm index cache invalidations works differently with native interpreters it seems

I am also having the same issue. After updating the __init__.py however, Pycharm does not pick up the changes… I am using tensorflow-macos==2.8.0 in M1.