oppia: Linter Checking file that does not exist

Describe the bug Linter fails on attempting to lint on a file that does not exist. Happens both locally on computer and remotely on circleci

On CircleCI


Unable to run the complete lint test, please check the following stack trace and fix the errors:
+--------------------------+
Traceback (most recent call last):
  File "scripts/concurrent_task_utils.py", line 61, in run
    self.output = self.func()
  File "scripts/linters/js_ts_linter.py", line 860, in perform_all_lint_checks
    self.parsed_js_and_ts_files = self._validate_and_parse_js_and_ts_files()
  File "scripts/linters/js_ts_linter.py", line 185, in _validate_and_parse_js_and_ts_files
    file_content = FILE_CACHE.read(compiled_js_filepath)
  File "/home/circleci/oppia/scripts/linters/pre_commit_linter.py", line 168, in read
    return self._get_data(filepath, mode)[0]
  File "/home/circleci/oppia/scripts/linters/pre_commit_linter.py", line 199, in _get_data
    with python_utils.open_file(filepath, mode) as f:
  File "python_utils.py", line 106, in open_file
    raise IOError('Unable to open file: %s' % filename)
IOError: Unable to open file: /home/circleci/oppia/tmpcompiledjs8IV6JQ/translation-language-select.component.js

Locally (it seems to happen on a different file)

Unable to run the complete lint test, please check the following stack trace and fix the errors:
+--------------------------+
Traceback (most recent call last):
  File "scripts/concurrent_task_utils.py", line 61, in run
    self.output = self.func()
  File "scripts/linters/js_ts_linter.py", line 862, in perform_all_lint_checks
    self.parsed_js_and_ts_files = self._validate_and_parse_js_and_ts_files()
  File "scripts/linters/js_ts_linter.py", line 182, in _validate_and_parse_js_and_ts_files
    file_content = FILE_CACHE.read(compiled_js_filepath)
  File "/Users/davidchenggoogle/opensource/oppia/scripts/linters/pre_commit_linter.py", line 167, in read
    return self._get_data(filepath, mode)[0]
  File "/Users/davidchenggoogle/opensource/oppia/scripts/linters/pre_commit_linter.py", line 198, in _get_data
    with python_utils.open_file(filepath, mode) as f:
  File "python_utils.py", line 106, in open_file
    raise IOError('Unable to open file: %s' % filename)
IOError: Unable to open file: /Users/davidchenggoogle/opensource/oppia/tmpcompiledjsRjCxNX/translation-language-select.component.spec.js

To Reproduce Steps to reproduce the behavior:

Go to PR #9443
Run linters

Observed behavior A clear and concise description of what you have observed.

Linter fails trying to access oppia/tmpcompiledjs8IV6JQ/translation-language-select.component.js

Expected behavior A clear and concise description of what you expected to happen.

Linter does not check file that does not exist

Desktop (please complete the following information; delete this section if the issue does not arise on desktop):

OS: MacOS 10.15.5

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 28 (26 by maintainers)

Commits related to this issue

Fix for #9458: Typescript linter fails on relative import (#9580) * fix for 9548 * review * Styling * Styling * Styling * Styling * docstr Co-authored-by: shavavo <shavavo@google... — committed to oppia/oppia by deleted user 4 years ago
Fix for #9458: Typescript linter fails on relative import (#9580) * fix for 9548 * review * Styling * Styling * Styling * Styling * docstr Co-authored-by: shavavo <shavavo@google... — committed to shavavo/oppia by deleted user 4 years ago

Most upvoted comments

Hi all,

Here are some benchmarks I ran, there are some interesting findings Note: Benchmarking on a 2.6 GHz 6-Core i7 MBP

Benchmark - Compiling all files

Command used: ./node_modules/typescript/bin/tsc -p tsconfig.json -outDir tmpcompiledjs Compiling all files at once took around 15 seconds for three separate trials.

Benchmark - Linter

Method: I isolated JsTsLintChecksManager._validate_and_parse_js_and_ts_files by:
- in JsTsLintChecksManager.perform_all_lint_checks, commenting out everything except self._validate_and_parse_js_and_ts_files(), and returning an empty array instead of all_messages to indicate success.
- Similarly, in ThirdPartyJsTsLintChecksManager.perform_all_lint_checks, comment out everything and return [] to skip checks
Command used: python -m scripts.linters.pre_commit_linter --path core/templates/services/

branch	trial 1	trial 2	trial 3	average
develop	244s	239s	238s	240.3s
#9580	50s	58s	48s	52s

Note: On branch develop, for the core/templates/services directory alone, esprima fails and we use _compile_ts_file 144 times, causing a massive slow down compared to compiling all at once.

Discussion

It appears #9580 is significantly faster than develop when it comes to running many files at once. However, much of the use case of pre_comit_linter is to only check files that have been modified, instead of running on a big directory of files. In this case, where we only need to check a few files, we should expect a maximum increase of 15s of runtime.

@ankita240796 @vojtechjelinek @seanlip @Hudda thoughts? Is this tradeoff worth it?

Let me know if there are any other benchmarks you would like to see. Perhaps knowing how many esprima fails equate to compiling all files would be useful.

shavavo on Jun 19, 2020