tensorflow: PyCharm doesn't resolve anything under tensorflow.keras

System information

  • OS Platform and Distribution: Windows 10
  • TensorFlow installed from: binary
  • TensorFlow version: 2.7.0
  • Python version: 3.9
  • Installed using: Conda
  • CUDA/cuDNN version: 11.2/8.1

Describe the problem I’ve upgraded tensorflow from 2.5 to 2.7 and now PyCharm doesn’t resolve anything under tensorflow.keras.

keras-no-autocompletion

Other modules of tensorflow work, it’s only keras that’s problematic. I believe this has something to do with the change in TF 2.6 where keras has been split into a separate PIP package.

People have also been reporting this problem to JetBrains (PyCharm developers): https://youtrack.jetbrains.com/issue/PY-50318

Provide the exact sequence of commands / steps that you executed before running into the problem

  1. Create the conda environment: ~ conda create --name tensorflow27
  2. Activate it: ~ conda activate tensorflow27
  3. Install Python: ~ conda install python=3.9
  4. Install TensorFlow: ~ pip install tensorflow
  5. Create new project in PyCharm
  6. Write this code:
import tensorflow as tf
tf.keras.preprocessing.image_dataset_from_directory()
  1. When you move mouse to image_dataset_from_directory function there will be no autocomplete.
  2. Writing tf.keras. yields no modules/functions

Any other info / logs /

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 44
  • Comments: 68 (6 by maintainers)

Most upvoted comments

hi, @markorakita It’s a naming conflict issue. try modify the file site-packages/tensorflow/__init__.py near line 387.

before:

_keras_module = "keras.api._v2.keras"
keras = _LazyLoader("keras", globals(), _keras_module)
_module_dir = _module_util.get_parent_dir_for_name(_keras_module)
if _module_dir:
  _current_module.__path__ = [_module_dir] + _current_module.__path__
setattr(_current_module, "keras", keras)

after:

import typing as _typing
if _typing.TYPE_CHECKING:
  from keras.api._v2 import keras
else:
  _keras_module = "keras.api._v2.keras"
  keras = _LazyLoader("keras", globals(), _keras_module)
  _module_dir = _module_util.get_parent_dir_for_name(_keras_module)
  if _module_dir:
    _current_module.__path__ = [_module_dir] + _current_module.__path__
  setattr(_current_module, "keras", keras)

it work for me.

I’m using Emacs and I have same issue in TF2.8. For now, my solution is:

import typing

from tensorflow import keras

if typing.TYPE_CHECKING:
    from keras.api._v2 import keras

It works good for my environment.

Sorry I thought I replied to this.

You can write code and run it and it works, but you don’t have autocomplete nor you can go to declaration of any keras function. It’s like writing code in notepad 😃

The reason why this does not work has nothing to do with pycharm, but one think is that the tensorflow.keras package is no longer generated in 2.8 and onwards, which it used to be, as you can see from the header of all files from the package in 2.7.0:

# This file is MACHINE GENERATED! Do not edit.
# Generated by: tensorflow/python/tools/api/generator/create_python_api.py script.

Maybe @markorakita can adapt the title and maybe also the description?

Since there has been no activity from anyone from tensorflow since January 2022, the general question is how to get someone to pay attention to the issue? Could it be worth to open a new issue and give it upvotes from many people here? This probably affects everyone who is used to autocompletion/an IDE and doesn’t use stubs for tf from somewhere?

I don’t know what else has changed since then, but maybe generation just needs to be reactivated?

Edit: This does not solve the problem originally mentioned in this issue with tf.keras.<autocomplete>, but at least the autocompletion for from tensorflow.keras import <autocomplete>.

any updates on the issue? as mentioned now it is like coding in notepad

Any update on this? It’s been a minute and neither this issue nor #53271 have received any updates since Dec

So far I haven’t seen anyone propose this solution, so I’m passing it on. If PyCharm does not suggest the code to you, but the code itself executes (i.e. if you run the code, e.g. tf.keras.layers.LayerNormalization, which the IDE does not suggest, executes correctly), you can install the package:

pip install types-tensorflow

which contains “stubs” compatible with the corresponding versions of TensorFlow. If you use TensorFlow 2.16.x, you can install, for example, types-tensorflow==2.16.0.20240606 (see which version of TensorFlow these definitions are for).

pip install types-tensorflow==2.16.0.20240606

PyCharm, after refreshing the indexes, should correctly suggest classes contained in TensorFlow, unfortunately it will not suggest documentation, but it always makes writing code easier.

As a working workaround, you can do the following: (no need to recompile).

Use the fix from https://github.com/tensorflow/tensorflow/issues/53144#issuecomment-985179600 (you need to modify it a bit for the current versions.)

Then, in your code:

import tensorflow as tf
keras = tf.keras
K = keras.backend # example

You have to “Invalidate Caches & Restart” in PyCharm and let it regenerate the indices which will take a few minutes.

With this, I have full autocomplete as it should be.

The same issue is still happening with 2.8.0, 2.8.1, 2.9.0, and any combination with Python/Conda versions 3.7, 3.8, 3.9, or 3.10.

Same problem in 2.18.0.

Hi,

as of Tensorflow 2.16, the preferred way of using keras is to import keras and use it directly:

import keras

inputs = keras.Input(...)

tf.keras is only provided for backwards compatibility.

My experience is that code completion for keras works in PyCharm with import keras.

@markorakita Everything is stirred up, for the time being, only part of it can be solved first, and I hope that someone will solve the problem as soon as possible.

Since this issue is a duplicate as https://github.com/tensorflow/tensorflow/issues/56231, I am going to close this one, so that everyone can track on same issue.

So far I haven’t seen anyone propose this solution, so I’m passing it on. If PyCharm does not suggest the code to you, but the code itself executes (i.e. if you run the code, e.g. tf.keras.layers.LayerNormalization, which the IDE does not suggest, executes correctly), you can install the package:

pip install types-tensorflow

which contains “stubs” compatible with the corresponding versions of TensorFlow. If you use TensorFlow 2.16.x, you can install, for example, types-tensorflow==2.16.0.20240606 (see which version of TensorFlow these definitions are for).

pip install types-tensorflow==2.16.0.20240606

PyCharm, after refreshing the indexes, should correctly suggest classes contained in TensorFlow, unfortunately it will not suggest documentation, but it always makes writing code easier.

The stubs are nice, but documentation is what I need most. I decided to downgrade to 2.12.1 to get documentation working again. Now code is linked correctly and documentation works, but GPU support is broken, because I use Cuda 12.3 which requires a newer tensorflow version.

To everyone that’s still couldn’t make it work. Try this, I just tweaked the popular solution on this thread:

modify the file site-packages/tensorflow/__init__.py near line 387 From:

_keras_module = "keras.api._v2.keras"
keras = _LazyLoader("keras", globals(), _keras_module)
_module_dir = _module_util.get_parent_dir_for_name(_keras_module)
if _module_dir:
  _current_module.__path__ = [_module_dir] + _current_module.__path__
setattr(_current_module, "keras", keras)

To:

Note that I used _keras instead of keras

import typing as _typing
if _typing.TYPE_CHECKING:
  from keras.api._v2 import keras
else:
  _keras_module = "keras.api._v2.keras"
  _keras = _LazyLoader("keras", globals(), _keras_module)
  _module_dir = _module_util.get_parent_dir_for_name(_keras_module)
  if _module_dir:
    _current_module.__path__ = [_module_dir] + _current_module.__path__
  setattr(_current_module, "keras", _keras)

And then make sure to copy folder /site-packages/keras/ into /site-packages/tensorflow/

After all of that, invalidate and restart your IDE

This works on my setup using Conda

When a new 2.9.0-rc is being cut, and this story isn’t in the bug fixes section 🥲

@vanakema I already did that but no luck… Any other idea?

image

Oh I thought I saw someone else say that for whatever reason that style import doesn’t work with the fix. Do you get autocompletes when you write something like tf.keras.layers in your actual code?

To everyone that’s still couldn’t make it work. Try this, I just tweaked the popular solution on this thread:

modify the file site-packages/tensorflow/__init__.py near line 387 From:

_keras_module = "keras.api._v2.keras"
keras = _LazyLoader("keras", globals(), _keras_module)
_module_dir = _module_util.get_parent_dir_for_name(_keras_module)
if _module_dir:
  _current_module.__path__ = [_module_dir] + _current_module.__path__
setattr(_current_module, "keras", keras)

To:

Note that I used _keras instead of keras

import typing as _typing
if _typing.TYPE_CHECKING:
  from keras.api._v2 import keras
else:
  _keras_module = "keras.api._v2.keras"
  _keras = _LazyLoader("keras", globals(), _keras_module)
  _module_dir = _module_util.get_parent_dir_for_name(_keras_module)
  if _module_dir:
    _current_module.__path__ = [_module_dir] + _current_module.__path__
  setattr(_current_module, "keras", _keras)

And then make sure to copy folder /site-packages/keras/ into /site-packages/tensorflow/

After all of that, invalidate and restart your IDE

This works on my setup using Conda

It worked perfectly on PyCharm IDE with TensorFlow 2.13.0! Finally from tensorflow.keras.layers import Dense, Flatten, Conv2D is successfully recognized by IDE.

With a small change, instead of copy folder /site-packages/keras/ into /site-packages/tensorflow/, I use symbolic link ln -s <full-path-to-site-package>/site-packages/keras <full-path-to-site-package>/tensorflow/keras

Just copy the folder keras from site-packages/tensorflow/python to site-packages/tensorflow. And the auto completion will start working perfectly

This workaround with the .python. addition works to have autocompletion and syntax checking (in some instances but not all).

However, it does not help during runtime:

import tensorflow.python.keras as keras
l = keras.layers.LayerNormalization()
>>> AttributeError: module 'tensorflow.python.keras.layers' has no attribute 'LayerNormalization'

When not using .python.:

import tensorflow.keras as keras
l = keras.layers.LayerNormalization()
# works

It’s not a good workaround.

(TensorFlow 2.8.0)

This worked for me as well

Works but that is a dirty way, and I think it need to be repeated after each update Should fixed on source side

hi, @markorakita It’s a naming conflict issue. try modify the file site-packages/tensorflow/__init__.py near line 387.

before:

_keras_module = "keras.api._v2.keras"
keras = _LazyLoader("keras", globals(), _keras_module)
_module_dir = _module_util.get_parent_dir_for_name(_keras_module)
if _module_dir:
  _current_module.__path__ = [_module_dir] + _current_module.__path__
setattr(_current_module, "keras", keras)

after:

import typing as _typing
if _typing.TYPE_CHECKING:
  from keras.api._v2 import keras
else:
  _keras_module = "keras.api._v2.keras"
  keras = _LazyLoader("keras", globals(), _keras_module)
  _module_dir = _module_util.get_parent_dir_for_name(_keras_module)
  if _module_dir:
    _current_module.__path__ = [_module_dir] + _current_module.__path__
  setattr(_current_module, "keras", keras)

it work for me.

I tried this wih Python 3.9 and tensorflow 2.11, though nothing changed!?

this issue still exist in PyCharm 2024.2 with TensorFlow 2.16.1

I can confirm that this issue still exits in latest PyCharm (2021.3.3 professional edition), python 3.10, TF 2.8. The workaround from cpuimage, then invalidating caches and restarting, did not fix the issue.

Hi,

as of Tensorflow 2.16, the preferred way of using keras is to import keras and use it directly:

import keras

inputs = keras.Input(…) tf.keras is only provided for backwards compatibility.

My experience is that code completion for keras works in PyCharm with import keras.

I don’t like having to resolve to importing directly from keras. Doing from tensorflow.keras.models import Sequential is much cleaner as it enforces the use of the tensorflow backend. Is there still no fix for this in March of 2025?

My experience is that code completion for keras works in PyCharm with import keras.

Working like a charm.

Also I am using this:

from keras.api.datasets import mnist
from keras.api.models import Sequential
from keras.api.layers import Dense, Flatten
from keras.api.utils import to_categorical

Image

Sorry for the very late reply. Both me and haifeng has moved to some new project for sometime. Let me cc @mattdangerw and @hertschuh here.

To everyone that’s still couldn’t make it work. Try this, I just tweaked the popular solution on this thread:

modify the file site-packages/tensorflow/__init__.py near line 387 From:

_keras_module = "keras.api._v2.keras"
keras = _LazyLoader("keras", globals(), _keras_module)
_module_dir = _module_util.get_parent_dir_for_name(_keras_module)
if _module_dir:
  _current_module.__path__ = [_module_dir] + _current_module.__path__
setattr(_current_module, "keras", keras)

To:

Note that I used _keras instead of keras

import typing as _typing
if _typing.TYPE_CHECKING:
  from keras.api._v2 import keras
else:
  _keras_module = "keras.api._v2.keras"
  _keras = _LazyLoader("keras", globals(), _keras_module)
  _module_dir = _module_util.get_parent_dir_for_name(_keras_module)
  if _module_dir:
    _current_module.__path__ = [_module_dir] + _current_module.__path__
  setattr(_current_module, "keras", _keras)

And then make sure to copy folder /site-packages/keras/ into /site-packages/tensorflow/

After all of that, invalidate and restart your IDE

This works on my setup using Conda

I am facing this super weird issue.

I am on version 2.16.1 First of my site-packages/tensorflow/__init__.py does not contain any such method. image And this is how the file looks like near line 387.

I still copied folder /site-packages/keras/ into /site-packages/tensorflow/ . But still no luck

Very interesting thing is that I can access the definition of __version__ in VS Code insiders but not on PyCharm. image image

At the same time I can go to Dense via from tensorflow.keras.layers import Dense on PyCharm, but not on VS Code insiders, also I am not able to access Input which also has the same parent package of Dense.

And ya, both of them are running on the same global interpreter 3.12.0

One thing I figured out was PyCharm was saving these packages at its own end. image Whereas, VS Code insiders was doing it from the /site-packages/tensorflow/keras. I was able to access tensorflow.keras after copying the folder, but not tensorflow.keras.layers.

Just copy the folder keras from site-packages/tensorflow/python to site-packages/tensorflow. And the auto completion will start working perfectly

Thank you for your idea! However, it only works for some. It looks like site-packages/tensorflow/python/keras is different from site-packages/keras.

After applying your method, I have the following results: image image

set_random_seed is not resolved when calling from the copied folder

Sorry for the very long wait.

This issue should be addressed by https://github.com/tensorflow/tensorflow/pull/54104, which is landed on tf-nightly (but didn’t included in tf 2.9). I was able to verify this locally with the latest tf-nightly pip package.

Could any of you test with latest tf-nightly on the IDE and see if it fix the issue? Thanks.

Can we run the generation manually as a workaround without having to compile TF from the source?

@winthropharvey @Tolure @nicolasperezdeo site-packages\tensorflow\keras is removed in tf 2.8.

The temporary solution can only be:

before: from tensorflow.keras.callbacks import EarlyStopping after: from tensorflow.python.keras.callbacks import EarlyStopping

I am also having the same issue. After updating the __init__.py however, Pycharm does not pick up the changes… I am using tensorflow-macos==2.8.0 in M1.

You’ll probably need to invalidate caches and restart. Your indexes are likely still from before the change. PyCharm index cache invalidations works differently with native interpreters it seems

I am also having the same issue. After updating the __init__.py however, Pycharm does not pick up the changes… I am using tensorflow-macos==2.8.0 in M1.