pipreqs: UnicodeDecodeError: 'utf-8' codec can't decode byte

$ pipreqs --encoding utf-8

I had a .venv directory with the usual hugh collection of python files. Something in the package black.py messed it up. A rm -rf .venv, and “pipreqs .” worked.

Thank you sir - this finally fixed my issue

Adding ignore parameters also work fine and you don’t have to remove virtual env!

pipreqs . --ignore ".env"

@kamalmeh, try this:

$ pipreqs --encoding=iso-8859-1

I had a .venv directory with the usual hugh collection of python files. Something in the package black.py messed it up. A rm -rf .venv, and “pipreqs .” worked.

This happens when certain modules/packages got files which are not encoded with the provided encoding (e.g. UTF-8 vs. UTF-8 with BOM). In my case IPython has a file called nonascii.py which was encoded with ISO-8859-5 (& got Cyrillic characters) and joblib got a file (test_func_inspect_special_encoding.py) which was encoded with Big5 (seems to be a Chinese encoding) and also had Chinese characters inside it, as well as srsly got a file called test_ujson.py which was encoded with UTF-8 BOM despite specifying UTF-8 in its header.

I solved my issue by commenting out the first two files and saving the last one in the correct encoding. To circumvent such error pipreqs maybe should try to determine the encoding of each file dynamically be either reading the encoding specified in the header of the file or somehow determine the encoding which most likely has been used.

As it was the whole environment folder that was causing the problem, what worked for me is:

pipreqs . --ignore <path_to_env_folder>

pipreqs . --ignore <path_to_env_folder> --force
it’s working @mervess

Hey.

I had a .venv directory with the usual hugh collection of python files. Something in the package black.py messed it up. A rm -rf .venv, and “pipreqs .” worked.

Thank you sir - this finally fixed my issue

Adding ignore parameters also work fine and you don’t have to remove virtual env!

pipreqs . --ignore ".env"

Thanks, this worked for me too. Just to add some context, this applies if you’ve labeled your virtual env and “env”. If its named “venv” then replace the “.env” with “.venv”

I had the similar problem. The way I how I fixed is by deleting all the temporary python files in the folder. rm -f .*py *.swp

This deletes all the temporary python scripts. pipreqs --encoding utf-8 path_to_folder/

Along the lines of beybars1 did but with less round-tripping:

replace contents = f.read() around line 115 in pipreqs.py with

try:
    contents = f.read()
except UnicodeDecodeError as ude:
    print(f'UnicodeDecodeError reading {file_name} (couldn\'t read contents). Skipping.')

Hello everyone!

As for me the issue is still present for some of you. Here is my “solution”:

As @phHartl mentioned, you can skip conflicting files (nonascii.py and test_func_inspect_special_encoding.py in my case) by doing this:

In the python file of your venv (lib64/python3.9/site-packages/pipreqs/pipreqs.py), make your code similar to this snippet (on time of writting this, line is 114)

with open(file_name, "r", encoding=encoding) as f:
     print(file_name)
     if file_name in ['...../nonascii.py', ...../'test_func_inspect_special_encoding.py']:
          continue
     contents = f.read() 

I agree it is not an elegant solution, but it works!

Hope, you find it useful.