black: "INTERNAL ERROR: Black produced different code on the second pass of the formatter"

Fixed by #2126.

One case that I believe refers to this bug: (non-relevant parts removed)

Using the workaround (running with --fast) worked for my case.

Mode(target_versions=set(), line_length=88, string_normalization=True, experimental_string_processing=False, is_pyi=False)
--- source
+++ first pass
-            if (
-                mission.get("status")
-                not in [
-                    "new",
-                    "test",
-                    "scheduled"
-                ]
-                and not mission.get("redeem")
-            ):
+            if mission.get("status") not in [
+                "new",
+                "test",
+                "scheduled",
+            ] and not mission.get("redeem"):
--- first pass
+++ second pass
@@ -7373,15 +7373,19 @@
-            if mission.get("status") not in [
-                "new",
-                "test",
-                "scheduled",
-            ] and not mission.get("redeem"):
+            if (
+                mission.get("status")
+                not in [
+                    "new",
+                    "test",
+                    "scheduled",
+                ]
+                and not mission.get("redeem")
+            ):

If anyone has time and interest in testing a potential fix for second-pass formatting instability, I’d welcome your feedback on https://github.com/psf/black/pull/1958.

As always, please read the changes and make sure you’re comfortable with them before checking them out and running them locally, as you should (ideally) for any other code. And feel free to ask in the pull request if anything is unclear. I can’t promise to give answers to everything but I’ll try.

I’ve used some of the examples from this thread during development and have had promising results so far, but in particular I’d be interested to hear about any counter-examples that continue to break.

Thanks @ichard26 - that’s almost true, but in the logs for that build, the fuzzer really did generate an input that reproduces #1629 (INTERNAL ERROR: Black produced different code on the second pass of the formatter.).

It’s slightly surprising, but is valid and relevant here, I think. #1913 seems to relate to the fuzzer exposing a separate libcst-related error.

Passing --fast to Black isn’t fully safe as Black can produce invalid code. --fast disables the unstable formatting check, the AST equivalence check, and the valid python code check. Black can accidentally destroy a valid Python program; here’s an example with --fast.

An example equivalent to something that happened naturally in some code (version 20.8b1):

Mode(target_versions=set(), line_length=80, string_normalization=True, experimental_string_processing=False, is_pyi=False)
--- source
+++ first pass
@@ -227,11 +227,15 @@
-    xx = funstuff(sthnsaotheusnthoeasnuthsnt, saotehusaontehusnotheusn, soantehusnoatheusnth) + [x, y, z, w]
+    xx = funstuff(
+        sthnsaotheusnthoeasnuthsnt,
+        saotehusaontehusnotheusn,
+        soantehusnoatheusnth,
+    ) + [x, y, z, w]
--- first pass
+++ second pass
@@ -227,15 +227,18 @@
-    xx = funstuff(
-        sthnsaotheusnthoeasnuthsnt,
-        saotehusaontehusnotheusn,
-        soantehusnoatheusnth,
-    ) + [x, y, z, w]
+    xx = (
+        funstuff(
+            sthnsaotheusnthoeasnuthsnt,
+            saotehusaontehusnotheusn,
+            soantehusnoatheusnth,
+        )
+        + [x, y, z, w]
+    )

I had thought that this issue was mostly focused on new and should_explode related instability bugs, not all black produced different code on the second pass of the formatter errors, but IDK and IDC now. Also, while in the beginning #1913 exposed a hypothesmith issue (libcst wasn’t related at all), the issue was reopened for the class A:\\\r\n# type: ignore\n pass\n issue.

@jayaddison that’s already tracked via #1913.

This is true, it is certainly not a fix. But in my case, for this one file, it produced semantically identical code after “stabilizing”, and let me move on with using Black normally.

On Sat, Dec 12, 2020, 10:54 AM Richard Si notifications@github.com wrote:

Passing --fast to Black isn’t fully safe as Black can produce invalid code. --fast disables the unstable formatting check, the AST equivalence check, and the valid python code check. Black can accidentally destroy a valid Python program; here’s an example with --fast https://black.now.sh/?version=stable&state=_Td6WFoAAATm1rRGAgAhARYAAAB0L-Wj4ACdAHBdAD2IimZxl1N_Wk2dwftOLUrsSjKoBlrvNB7l7P-6rF_SkDWmZI2C0c3VYgLdOTVDVENmN-rJDtUcvMGwCmSaHlEjw7vRCCi9s7cWv0jxpNf7wW495JR8Z-MnHa5iZsy3U0t4cVTyTK20DdL0z1kumG8A2kTgmCMfki8AAYwBngEAAG7VGsuxxGf7AgAAAAAEWVo= .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/psf/black/issues/1629#issuecomment-743799782, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEPRJZD7FGWH4XGRMUDUF3SUO35PANCNFSM4QL5QZSQ .

Wait, the --fast option just skips the determistic checking, though. Right?

@ThatXliner, it’s a bug. One that is already fixed in the development branch by commit https://github.com/psf/black/commit/1d2d7264ec7c448744b771910cc972da03b1cb80, now it’s just waiting to be included in the next release.

Black basically missed things. I think the style may be inconsistent… I prefer no whitespace before the colon but eh, it’s black

Not a problem for our repo, since this was caught by me missing a folder in our black-ignore. But if this is helpful for debugging this problem then that’d be great. The diff is quite large at around 400 lines so I put it in a gist. Also run black with --verbose and put it below.

diff https://gist.github.com/xylix/9b282167786bc7ba9447ab11db9a6c5b

stacktrace

(venv)  ~/C/robotframework-browser (install-browsers-in-site-packages)> black -v /Users/kerkko/Code/robotframework-browser/Browser/wrapper/node_modules/playwright/.local-browsers/webkit-1383/JavaScriptCore.framework/Versions/A/PrivateHeaders/generate_objc_backend_dispatcher_implementation.py
Using configuration from /Users/kerkko/Code/robotframework-browser/Browser/pyproject.toml.
Traceback (most recent call last):
  File "/Users/kerkko/Code/robotframework-browser/venv/lib/python3.9/site-packages/black/__init__.py", line 670, in reformat_one
    if changed is not Changed.CACHED and format_file_in_place(
  File "/Users/kerkko/Code/robotframework-browser/venv/lib/python3.9/site-packages/black/__init__.py", line 813, in format_file_in_place
    dst_contents = format_file_contents(src_contents, fast=fast, mode=mode)
  File "/Users/kerkko/Code/robotframework-browser/venv/lib/python3.9/site-packages/black/__init__.py", line 940, in format_file_contents
    assert_stable(src_contents, dst_contents, mode=mode)
  File "/Users/kerkko/Code/robotframework-browser/venv/lib/python3.9/site-packages/black/__init__.py", line 6170, in assert_stable
    raise AssertionError(
AssertionError: INTERNAL ERROR: Black produced different code on the second pass of the formatter.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: /var/folders/xz/6y6sty192sbg2m3bb623z6hw0000gn/T/blk_d46pcptu.log
error: cannot format /Users/kerkko/Code/robotframework-browser/Browser/wrapper/node_modules/playwright/.local-browsers/webkit-1383/JavaScriptCore.framework/Versions/A/PrivateHeaders/generate_objc_backend_dispatcher_implementation.py: INTERNAL ERROR: Black produced different code on the second pass of the formatter.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: /var/folders/xz/6y6sty192sbg2m3bb623z6hw0000gn/T/blk_d46pcptu.log
Oh no! 💥 💔 💥
1 file failed to reformat