python-language-server: flake8 and pyflakes plugins are not working on Atom at least on Linux (and maybe other OSes)

@ccordoba12 What I had to do is log pyls flake8_lint.py to file (stdout did not work for some reason, probably because atom messes with it)

logfile = open('pyls.log', 'a+b', buffering=0)

cmd.extend(args)
logfile.write(f'>>>> {args!r} :: {cmd!r}\n'.encode())
p = Popen(cmd, stdout=PIPE, stderr=PIPE)

Also started atom with logging as /usr/lib/atom/atom --executed-from="$(pwd)" --pid=$$ -f > atom.log 2>&1. It is supposed to have -l option, but it produces empty files, same for nohup.out

Below are attached the three log files

• terminal.log:

  [thd@aspire log]$ pip install --user jedi==0.17 parso==0.7
  Collecting jedi==0.17
    Using cached jedi-0.17.0-py2.py3-none-any.whl (1.1 MB)
  Collecting parso==0.7
    Using cached parso-0.7.0-py2.py3-none-any.whl (100 kB)
  Installing collected packages: parso, jedi
  Successfully installed jedi-0.17.0 parso-0.7.0
  [thd@aspire .local]$ pip install --user git+https://github.com/palantir/python-language-server
  Collecting git+https://github.com/palantir/python-language-server
    Cloning https://github.com/palantir/python-language-server to /tmp/pip-req-build-q42vs7en
    Running command git clone -q https://github.com/palantir/python-language-server /tmp/pip-req-build-q42vs7en
  Requirement already satisfied: jedi<0.18.0,>=0.17.0 in ./lib/python3.8/site-packages (from python-language-server==0.33.0+1.g9b5cb21) (0.17.0)
  Requirement already satisfied: python-jsonrpc-server>=0.3.2 in /usr/lib/python3.8/site-packages (from python-language-server==0.33.0+1.g9b5cb21) (0.3.4)
  Requirement already satisfied: pluggy in /usr/lib/python3.8/site-packages (from python-language-server==0.33.0+1.g9b5cb21) (0.13.1)
  Requirement already satisfied: ujson<=1.35 in /usr/lib/python3.8/site-packages (from python-language-server==0.33.0+1.g9b5cb21) (0.0.0)
  Requirement already satisfied: parso>=0.7.0 in ./lib/python3.8/site-packages (from jedi<0.18.0,>=0.17.0->python-language-server==0.33.0+1.g9b5cb21) (0.7.0)
  Installing collected packages: python-language-server
      Running setup.py install for python-language-server ... done
  Successfully installed python-language-server-0.33.0+1.g9b5cb21
  [thd@aspire log]$ 
  [thd@aspire log]$ /usr/lib/atom/atom --executed-from="$(pwd)" --pid=$$ -f > atom.log 2>&1
  

• atom.log:

  bash: cannot set terminal process group (-1): Inappropriate ioctl for device
  bash: no job control in this shell
  [2020-05-29 14:07:28.647] [ERROR] code-highlight - Error getting code highlights { TypeError: Cannot read property 'map' of null
      at Function.<anonymous> (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:32:31)
      at Generator.next (<anonymous>)
      at fulfilled (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:4:58) rawStack: [ CallSite {}, CallSite {}, CallSite {} ] }
  [2020-05-29 14:07:53.414] [ERROR] code-highlight - Error getting code highlights { TypeError: Cannot read property 'map' of null
      at Function.<anonymous> (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:32:31)
      at Generator.next (<anonymous>)
      at fulfilled (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:4:58) rawStack: [ CallSite {}, CallSite {}, CallSite {} ] }
  [2020-05-29 14:08:04.313] [ERROR] code-highlight - Error getting code highlights { TypeError: Cannot read property 'map' of null
      at Function.<anonymous> (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:32:31)
      at Generator.next (<anonymous>)
      at fulfilled (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:4:58) rawStack: [ CallSite {}, CallSite {}, CallSite {} ] }
  [2020-05-29 14:08:04.317] [ERROR] code-highlight - Error getting code highlights { TypeError: Cannot read property 'map' of null
      at Function.<anonymous> (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:32:31)
      at Generator.next (<anonymous>)
      at fulfilled (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:4:58) rawStack: [ CallSite {}, CallSite {}, CallSite {} ] }
  [2020-05-29 14:08:10.051] [ERROR] code-highlight - Error getting code highlights { TypeError: Cannot read property 'map' of null
      at Function.<anonymous> (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:32:31)
      at Generator.next (<anonymous>)
      at fulfilled (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:4:58) rawStack: [ CallSite {}, CallSite {}, CallSite {} ] }
  [2020-05-29 14:08:15.528] [ERROR] code-highlight - Error getting code highlights { TypeError: Cannot read property 'map' of null
      at Function.<anonymous> (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:32:31)
      at Generator.next (<anonymous>)
      at fulfilled (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:4:58) rawStack: [ CallSite {}, CallSite {}, CallSite {} ] }
  [2020-05-29 14:08:16.357] [ERROR] code-highlight - Error getting code highlights { TypeError: Cannot read property 'map' of null
      at Function.<anonymous> (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:32:31)
      at Generator.next (<anonymous>)
      at fulfilled (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:4:58) rawStack: [ CallSite {}, CallSite {}, CallSite {} ] }
  [2020-05-29 14:08:20.670] [ERROR] code-highlight - Error getting code highlights { TypeError: Cannot read property 'map' of null
      at Function.<anonymous> (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:32:31)
      at Generator.next (<anonymous>)
      at fulfilled (/home/thd/.atom/packages/ide-python/node_modules/atom-languageclient/build/lib/adapters/code-highlight-adapter.js:4:58) rawStack: [ CallSite {}, CallSite {}, CallSite {} ] }
  

• pyls.log

  >>>> ['/home/thd/Work/heliumcms/xrpc.py', '--exclude=.svn,CVS,.bzr,.hg,.git,__pycache__,.tox,.venv', '--filename=*.py', None, '--ignore=E121,;; continuation line under-indented for hanging indent\nE123,;; closing bracket does not match indentation of opening bracket’s line\nE126,;; continuation line over-indented for hanging indent\nE24,;; multiple spaces after ‘,’; tab after ‘,’; ...\nD101,;; Missing docstring in public class\nD102,;; Missing docstring in public method\nD103,;; Missing docstring in public function\nD105 \t;; Missing docstring in magic method', '--max-line-length=99', '--select=E,F,W,C90'] :: ['flake8']
  >>>> ['/home/thd/Work/heliumcms/xrpc.py', '--exclude=.svn,CVS,.bzr,.hg,.git,__pycache__,.tox,.venv', '--filename=*.py', None, '--ignore=E121,;; continuation line under-indented for hanging indent\nE123,;; closing bracket does not match indentation of opening bracket’s line\nE126,;; continuation line over-indented for hanging indent\nE24,;; multiple spaces after ‘,’; tab after ‘,’; ...\nD101,;; Missing docstring in public class\nD102,;; Missing docstring in public method\nD103,;; Missing docstring in public function\nD105 \t;; Missing docstring in magic method', '--max-line-length=99', '--select=E,F,W,C90'] :: ['flake8']
  >>>> ['/home/thd/Work/heliumcms/xrpc.py', '--exclude=.svn,CVS,.bzr,.hg,.git,__pycache__,.tox,.venv', '--filename=*.py', None, '--ignore=E121,;; continuation line under-indented for hanging indent\nE123,;; closing bracket does not match indentation of opening bracket’s line\nE126,;; continuation line over-indented for hanging indent\nE24,;; multiple spaces after ‘,’; tab after ‘,’; ...\nD101,;; Missing docstring in public class\nD102,;; Missing docstring in public method\nD103,;; Missing docstring in public function\nD105 \t;; Missing docstring in magic method', '--max-line-length=99', '--select=E,F,W,C90'] :: ['flake8']
  

Hope that helps

UPD: I see at least two errors in the previous arguments to flake8:

• Tried passing the same args to flake8 — it does not understand the ;;. These originate from my setup.cfg, which itself is a valid ini, understood by flake8:

  [flake8]
  ;; Print the total number of errors
  ;count = true
  
  ;; Print the source code generating the error/warning in question.
  ;show_source = true
  
  ;; Count the number of occurrences of each error/warning code and print a report
  ;statistics = true
  
  ;; Comma-separated list of glob patterns to exclude from checks
  exclude = .svn, CVS, .bzr, .hg, .git, __pycache__, .tox
  
  ;; Comma-separated list of codes to ignore
  ignore =
  	E121, 	;; continuation line under-indented for hanging indent
  	E123, 	;; closing bracket does not match indentation of opening bracket’s line
  	E126, 	;; continuation line over-indented for hanging indent
  	E24, 	;; multiple spaces after ‘,’; tab after ‘,’; ...
  	D101, 	;; Missing docstring in public class
  	D102, 	;; Missing docstring in public method
  	D103, 	;; Missing docstring in public function
  	D105 	;; Missing docstring in magic method
  
  ;; We don't follow PEP8 proposed length as nowadays screens are better
  max-line-length = 99
  
  ;; Check McCabe's code complexity using originally proposed value
  max-complexity = 10
  
  disable-noqa = False
  

• The second one is None in between '--filename=*.py' and '--ignore=E121. I don’t give a f where it comes from.

So, I tried to run without my setup.cfg having pure defaults. And this None is still there. Below are attached the new logs:

>>>> ['/home/thd/Work/heliumcms/xrpc.py', '--exclude=.svn,CVS,.bzr,.hg,.git,__pycache__,.tox,.eggs,*.egg,.venv', '--filename=*.py', None, '--ignore=E121,E123,E126,E226,E24,E704,W503,W504', '--max-line-length=79', '--select=E,F,W,C90'] :: ['flake8', '/home/thd/Work/heliumcms/xrpc.py', '--exclude=.svn,CVS,.bzr,.hg,.git,__pycache__,.tox,.eggs,*.egg,.venv', '--filename=*.py', None, '--ignore=E121,E123,E126,E226,E24,E704,W503,W504', '--max-line-length=79', '--select=E,F,W,C90']

And when I filter out this None, build the command and try running it manually, flake8 starts fine

Following the previous my investigation. A quick and dirty fix:

diff --git a/pyls/plugins/flake8_lint.py b/pyls/plugins/flake8_lint.py
index 9bd8ae2..a8be636 100644
--- a/pyls/plugins/flake8_lint.py
+++ b/pyls/plugins/flake8_lint.py
@@ -44,10 +44,14 @@ def pyls_lint(config, document):
     return parse_stdout(document, output)
 
 
+# a quick and dirty fix to deal with Atom
+_fix_ignores_re = re.compile(r'([^a-zA-Z0-9_,]*;.*(\W+||$))')
 def run_flake8(args):
     """Run flake8 with the provided arguments, logs errors
     from stderr if any.
     """
+    args = [(i if not i.startswith('--ignore=') else _fix_ignores_re.sub('', i))
+               for i in args if i is not None]
     log.debug("Calling flake8 with args: '%s'", args)
     try:
         cmd = ['flake8']

Tried it, everything works like a charm, both with my setup.cfg and without

flake8/pyflakes are activated because I activated it in the Atom ide-python plugin which uses pyls. We are pretty certain the problem isn’t with ide-python.

Other people have been using the flake8 plugin successfully with other editors.

Please follow the instructions in my https://github.com/palantir/python-language-server/issues/769#issuecomment-620200190 above to give us more info from your side.

@ccordoba12 may we ask, why are python-language-server builds only being tested for Windows?

They are tested on Linux and Windows. Please check any previous PR to see that that’s the case.

Then I found a problem, that is, the CI detection environment only checks Windows and not Linux or other platforms, which may cause problems on other platforms.

Maybe we can use PyTest or other function check suite to check the results of the data we use or output. This may first solve the problem whether it is from Python instead of Atom.

@hermidalc, the first error in the log you posted only has the first line:

Python (pyls) stderr Traceback (most recent call last):

That’s clearly an issue with Atom and with that is impossible to help you, sorry.

No, it isn’t. This log just messed up with Atom and JS stuff also, in this form it’s more readable.

sed -i '/^\(logger\.ts\)/!d' atom-console-pyls-flake8.log

I don’t know where the problem is nor what’s its cause unless I can see a proper log file. And that problem (that server errors are not logged correctly) is an Atom problem.

I meant the package in Atom that does the integration with the python-language-server.