tox: TypeError: write() argument must be str, not bytes

I’ve spent a few hours trying to figure this out, here is where I’m up to:

The issue is within the TextIOWrapper (stdout). "TypeError: write() argument must be str, not bytes does not mean that the argument you passed was actually bytes, sys.stdout is a buffered stream with .line_buffering enabled, which means something written to the line earlier could be the bytes, not the string you just submitted.

So, the issue is that something has sent bytes to the stdout stream that shouldn’t have.

My strong suspicion is that it’s related to the Popen/stdout/stderr buffer for the threads that Tox instantiates. Since Tox is designed to call Python 3 and Python 2 equally, it’s probably in a subprocess that’s the issue, not in the main tox application.

There are 2 places that concern me, one is- https://github.com/tox-dev/tox/blob/master/tox/session.py#L129-L146 which calls sys.stdout instead of using the stdout as overridden by the user.

The second and I think the culprit (because this is almost impossible to properly debug is https://github.com/tox-dev/tox/blob/master/tox/interpreters.py#L74-L77

Here is my tox.ini

[tox]
envlist = py27, py36, lint, pylint

[testenv]
setenv =
    PYTHONPATH = {toxinidir}:{toxinidir}/retox
commands = python setup.py test

If I call my process with the environments, py27 it’s fine, py36 is fine, py36,p27 is fine.

Whenever I include the environment ‘lint’ it crashes. Yes, I know this tox.ini doesn’t make any sense because I haven’t defined what the lint environment should run. It calls by default Python 3.6, because that’s what my virtualenv is configured with.

The error from Tox is on invocation

tox: ERROR: InvocationError: /Users/anthonyshaw/repo/retox-2/.tox/lint/bin/python setup.py test (see /Users/anthonyshaw/repo/retox-2/.tox/lint/log/lint-108.log)

Note, my test suite is supposed to fail, I’m checking that it fails properly.

======================================================================
ERROR: __main__ (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: __main__
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
  File "/Users/anthonyshaw/repo/retox-2/retox/__main__.py", line 11, in <module>
    from asciimatics.screen import Screen
  File "/Users/anthonyshaw/repo/retox-2/.eggs/asciimatics-1.9.0-py3.6.egg/asciimatics/screen.py", line 24, in <module>
    from asciimatics.event import KeyboardEvent, MouseEvent
ModuleNotFoundError: No module named 'asciimatics.event'


----------------------------------------------------------------------
Ran 5 tests in 0.001s

FAILED (errors=4)
Test failed: <unittest.runner.TextTestResult run=5 errors=4 failures=0>
error: Test failed: <unittest.runner.TextTestResult run=5 errors=4 failures=0>

So, in summary if you are getting this issue, try selecting only certain environments then try running your lint/pylint etc. environments under Python 2.7 virtualenv’s instead of 3+ @obestwalter

Just “for the record” or in case somebody else runs into the same isse: I’ve adjusted a small project to use Python 3 and running tox gave me the same error message. I’ve reproduced the issue with a fairly small setup:

foo.py:

def bar():
    import pdb; pdb.set_trace()
    return 42

test_foo.py:

def test_bar():
    from foo import bar
    assert bar() == 42

setup.py:

from setuptools import setup, find_packages

setup(name='foo',
    version='0.1',
    packages=find_packages(),
)

tox.ini:

[tox]
envlist = py36

[testenv]
usedevelop = true
deps =
    pytest
    pdbpp
commands =
    py.test {posargs}

Running tox gave me this:

~/tox-test (master)-> tox
py36 create: /Users/zeidler/tox-test/.tox/py36
py36 installdeps: pytest, pdbpp
py36 develop-inst: /Users/zeidler/tox-test
py36 installed: attrs==17.4.0,fancycompleter==0.8,-e git+git@github.com:witsch/tox-test.git@298376ac293d9a4e6b210dd1cd675963a5dc7ead#egg=foo,pdbpp==0.9.2,pluggy==0.6.0,py==1.5.2,Pygments==2.2.0,pytest==3.4.0,six==1.11.0,wmctrl==0.3
py36 runtests: PYTHONHASHSEED='871445808'
py36 runtests: commands[0] | py.test
======================================= test session starts ========================================
platform darwin -- Python 3.6.3, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
rootdir: /Users/zeidler/tox-test, inifile:
collected 1 item                                                                                   

test_foo.py 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
F                                                                                [100%]

============================================= FAILURES =============================================
_____________________________________________ test_bar _____________________________________________

    def test_bar():
        from foo import bar
>       assert bar() == 42

test_foo.py:5: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
foo.py:5: in bar
    return 42
foo.py:5: in bar
    return 42
../Development/python/parts/opt/lib/python3.6/bdb.py:48: in trace_dispatch
    return self.dispatch_line(frame)
../Development/python/parts/opt/lib/python3.6/bdb.py:66: in dispatch_line
    self.user_line(frame)
../Development/python/parts/opt/lib/python3.6/pdb.py:261: in user_line
    self.interaction(frame, None)
.tox/py36/lib/python3.6/site-packages/pdb.py:243: in interaction
    self.print_stack_entry(self.stack[self.curindex])
.tox/py36/lib/python3.6/site-packages/pdb.py:800: in print_stack_entry
    print('[%d] >' % frame_index, file=self.stdout, end=' ')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <encodings.utf_8.StreamWriter object at 0x106b70710>, object = '[31] >'

    def write(self, object):
    
        """ Writes the object's contents encoded to self.stream.
            """
        data, consumed = self.encode(object, self.errors)
>       self.stream.write(data)
E       TypeError: write() argument must be str, not bytes

.tox/py36/lib/python3.6/codecs.py:377: TypeError
===================================== 1 failed in 0.17 seconds =====================================
ERROR: InvocationError: '/Users/zeidler/tox-test/.tox/py36/bin/py.test'
_____________________________________________ summary ______________________________________________
ERROR:   py36: commands failed

Without the import pdb; pdb.set_trace() line things were fine. But of course at that point — i.e. after removing pretty much everything else — I already had suspect, and as it turns out, after removing pdbpp from the setup, I get my pdb prompt back… 😃

Now I wonder if we could or rather should try to detect the presence of pdbpp and issue a big warning that might save some other people from spending time on this (or merely wondering if Python 3.6’s pdb is broken? 😃)