UnicodeDecodeError at _run_module_as_main

[mcepl]

Running a test (using python-unittest2-1.1.0-4 on RHEL-7 with python-2.7.5-34) I end with this mysterious error. I don't even know where to start debugging the problem (there is no hint in the preceding text of any problem):

Mouse button 1 click at (341,67)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/unittest2/__main__.py", line 21, in <module>
    main_()
  File "/usr/lib/python2.7/site-packages/unittest2/__main__.py", line 18, in main_
    main(module=None)
  File "/usr/lib/python2.7/site-packages/unittest2/main.py", line 90, in __init__
    self.runTests()
  File "/usr/lib/python2.7/site-packages/unittest2/main.py", line 248, in runTests
    self.result = testRunner.run(self.test)
  File "/usr/lib/python2.7/site-packages/unittest2/runner.py", line 172, in run
    test(result)
  File "/usr/lib/python2.7/site-packages/unittest2/suite.py", line 87, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/site-packages/unittest2/suite.py", line 126, in run
    test(result)
  File "/usr/lib/python2.7/site-packages/unittest2/suite.py", line 87, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/site-packages/unittest2/suite.py", line 126, in run
    test(result)
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 673, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 644, in run
    result.stopTest(self)
  File "/usr/lib/python2.7/site-packages/unittest2/result.py", line 91, in stopTest
    self._stdout_buffer.seek(0)
  File "/usr/lib64/python2.7/StringIO.py", line 106, in seek
    self.buf += ''.join(self.buflist)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 25: ordinal not in range(128)

Any thoughts what should I do? Which debugging logs or something could help you to diagnose this issue, please?

[rbtcollins]

Well, that means your stdout had non-ascii written to it in the test; I suspect that that indicates a bug in the stdout capturing logic - disabling the stdout capturing should confirm this by avoiding the error.

[mcepl]

OK, with stdout capturing switched off I have a little bit shorter traceback, but still:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/unittest2/__main__.py", line 21, in <module>
    main_()
  File "/usr/lib/python2.7/site-packages/unittest2/__main__.py", line 18, in main_
    main(module=None)
  File "/usr/lib/python2.7/site-packages/unittest2/main.py", line 90, in __init__
    self.runTests()
  File "/usr/lib/python2.7/site-packages/unittest2/main.py", line 248, in runTests
    self.result = testRunner.run(self.test)
  File "/usr/lib/python2.7/site-packages/unittest2/runner.py", line 176, in run
    stopTestRun()
  File "/usr/lib/python2.7/site-packages/unittest2/runner.py", line 128, in stopTestRun
    self.printErrors()
  File "/usr/lib/python2.7/site-packages/unittest2/runner.py", line 116, in printErrors
    self.printErrorList('ERROR', self.errors)
  File "/usr/lib/python2.7/site-packages/unittest2/runner.py", line 124, in printErrorList
    self.stream.writeln("%s" % err)
  File "/usr/lib/python2.7/site-packages/unittest2/runner.py", line 32, in writeln
    self.write(arg)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2026' in position 413: ordinal not in range(128)

[mcepl]

Wouldn’t some check here https://hg.python.org/unittest2/file/tip/unittest2/result.py#l180 (or somewhere in the neighborhood; BTW, why is unittest2 not on GitHub?) making sure that all content of self.buffer is unicode (or str in py3k terminology) help?

[mcepl]

Hmm, I don't understand, I suppose. I tried to create this test case:

matej@mitmanek: unittest2 (test_fail *%)$ git diff master
diff --git a/unittest2/test/test_result.py b/unittest2/test/test_result.py
index a12e7b5..eeb873a 100644
--- a/unittest2/test/test_result.py
+++ b/unittest2/test/test_result.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 import sys
 import textwrap
 import traceback2 as traceback
@@ -261,6 +262,30 @@ class Test_TestResult(unittest2.TestCase):
         test_case, formatted_exc = result.errors[0]
         self.assertEqual('A tracebacklocals', formatted_exc)

+    def test_addError_locals_unicode(self):
+        class Foo(unittest.TestCase):
+            def test_1(self):
+                raise Exception(u'ошибка случилось')
+
+            def test_2(self):
+                raise Exception('ошибка случилось')
+
+        test_1 = Foo('test_1')
+        test_2 = Foo('test_2')
+        result = unittest.TestResult()
+        result.tb_locals = True
+
+        unittest.result.traceback = MockTraceback
+        self.addCleanup(restore_traceback)
+        result.startTestRun()
+        test_1.run(result)
+        test_2.run(result)
+        result.stopTestRun()
+
+        self.assertEqual(len(result.errors), 2)
+        test_case, formatted_exc = result.errors[0]
+        self.assertEqual('A tracebacklocals', formatted_exc)
+
     def test_addSubTest(self):
         log = []
         class Foo(unittest.TestCase):
matej@mitmanek: unittest2 (test_fail *%)$

And to my surprise it doesn't fail. Why?