typescript-eslint: Make sure type-aware lint rules' test cases are type-error-free
Suggestion
We should add an extra type-checking step to rule-tester when testing type-aware lint rules. Right now, we have some test cases that don’t pass, most notably because they have undeclared variables (which become any). However we don’t guarantee linting behavior when there are type errors (because the result is likely any, which either created new lint errors or masks actual lint errors). Such type errors may be hard to catch in more complex situations.
Brad mentioned there may be a perf penalty, but it would be useful to experiment nonetheless.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 2
- Comments: 16 (16 by maintainers)
A few thoughts after iterating on https://github.com/typescript-eslint/typescript-eslint/pull/8351:
meta.docs.requiresTypeCheckingis not a fixed key (likedescriptionorurl), but a recommended name to telleslint-doc-generatorthat the rule uses typed linting. We will have to mention this in RuleTester docs. Perhaps we could also allow customization of whichdocskey should be used to decide whether to collect TS diagnostics or not?I think having it on by default is fine if the case uses type information. We can wrap the parser services to detect when a case accesses type info and flick the option on for those rules.
In that case I think being breaking is fine here — we’re highlighting a case where their test case could be plain wrong. Type errors fall to
anysilently and thus a case can pass silently and accidentally.But for non-type-aware cases I agree that it’s not correct to be on by default. I’d even say it’s plain wrong to be on at all for them. A non-type-aware rule only checks syntax so types are irrelevant and ensuring types are correct can get in the way of devx.
My thought too. We would check for type errors if the rule requires type checking. And each test case can have an option to disable type error checking.
That … could work somewhat. But it’s a little complex. We’d have to wrap versions of the services used in tests. Plus, rules don’t guarantee always calling to those APIs. There will likely be individual cases where they’re not called, but type errors should still be checked.
Maybe this can just always be on if
meta.docs.requiresTypeCheckingis enabled? That would be simpler.I’d think so, yeah.
For end-users it would probably be more convenient if this feature was opt-in. Since otherwise we may break a lot of existing tests.
I see that this feature can be enabled separately in each test case:
Or on each RuleTester instance:
Moreover, we can add the ability to configure default behavior globally for all RuleTester instances in some test “setup” file, like for
RuleTester.afterAlletc (https://typescript-eslint.io/packages/rule-tester#with-specific-frameworks)I agree that some rules don’t require all tests to be error free, some of them may be sensitive to TS errors only in some specific conditions to check some rule implementation details. But overall if we’re targeting TS error-free code, I think it worth to ensure this in all our tests.
Yes, ensuring that existing test cases are free of TS errors will require a lot of work, but later on when someone adds new tests, it won’t be so hard anymore. WDYT?
😭.
…although, hey, 1643 / 29272 is ~5.6%. I suppose that’s an ‘A’ grade…? 😂
I’m now wondering whether it should be opt in or opt out… We the typescript-eslint team are very good at doing this right. But I don’t know that we’d want to have that assumption hold by default for not-us folks. And even we’ve had slips in.
I like how #8351 has an
ignoreTsErrorsexplicitly set. That feels good to me.Having it key on specific error codes is a little trickier given that they aren’t stable version-to-version. That’ll likely cause pain down the line the next time TypeScript shuffles around error codes in a version. Which happens once in a while. And is a pity because in theory I do really like the idea of needing to specify which errors are allowed. 😞
Agreed - for any big change like this I’m a favor of splitting out
The issue is that there are a lot of tests where we don’t actually care if there is a type error because the code is testing a pure syntactic code path.
For example -
dot-notationonly cares about types specifically in the case of private member access. So all of those tests with errors are moot - whether or not they have errors they still accurately model the 99.9% case of not being member access on a class.So if we were to build this we would need a way to easily opt out, IMO. It’s not worth us writing type-correct test cases for every single case, IMO, as that would involve multi-line tests where a single-line would have been sufficient.
I sent a draft just to check how many tests contain semantic TS errors: https://github.com/typescript-eslint/typescript-eslint/pull/8351
There is really lot of them…
Some tests, such as
dot-notation.test.ts, contain errors in almost every case…https://github.com/typescript-eslint/typescript-eslint/pull/8351/files#diff-93cb24cbe290e1f30e97aaf0631ade9bd3751820d1f0bef8ecb4a0532243e2bd
So if we decide we want to fix all of them, that’s a huge git diff (maybe we can split it into several prs)
Also, I’m not sure how best to handle cases with fixers. I suppose we’d like to be able to write both types of test cases:
codeto contain a certain ts error. But theoutputshouldn’t contain ts errorscodeto contain a certain ts error. And theoutputshould contain ts errorAny thoughts on this? I just want to make sure we want to fix all the failed tests before I start fixing the 1643 cases 😄