micro: Error detecting file type by `signature`
Description of the problem or steps to reproduce
When opening the Pipfile.lock file it looks like this:
{
"_meta": {}
}
The type shown is unknown, but as specified in the signature of the file:
I did the same test with the file named index with the following content:
<!DOCTYPE html5>
From what I understand, both should be of type json and html5 given the signature of html5:
I didn’t get to test the other types of files that can be defined with signature but as both files should match with the regex I believe the same thing must be happening.
This does not happen in version 2.0.13. The file detection was changed with #2819.
Specifications
Commit hash: b518bda5 OS: Ubuntu 22.04.3 LTS Terminal: terminator
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 16 (8 by maintainers)
Although perhaps we should make
headerhave lower priority thanfilename, like we now already do withsignature(and unlike in the old implementation, where it was non-deterministic whetherfilenamematch orheadermatch would take precedence). Perhaps then we won’t even need to fix json?Ok, then we keep it short and
headerwill return.signatureis and can be used to solve multiple hits with the samefilenameandheaderwill step up in the moment nofilenamehit at all.Yep,
greptells that thejsondefinition is right now the only one with a very generic pattern.Last but not least the doc of
signaturecan then be extended/optimized too.I’ve read the discussion and I come to a conclusion that we should admit that replacing “headers” with “signatures” was a hasty decision. It is not just a matter of a proper doc update, it is a matter of breakage of useful functionality.
headerandsignatureare different things, with quite different purposes, both are useful, we should have added the latter without obstructing the former.If we have a file named
foowith the following content:with v2.0.13 it is recognized and highlighted as a shell script. With the newest master it is not.
To me, this is a regression, and a rather serious one.
The majority of micro’s syntax files have no
header/signaturepatterns. In those syntax files that do, those patterns are mainly shebangs (#!/bin/sh,#!/usr/bin/awketc) and things like<?xml ... ?>or<!DOCTYPE html5>, i.e. things that: 1. can be quite reliably used for detecting the filetype independently offilename, not just in addition to it. 2. should only occur in the first line of the file, not in the first 100 lines or so. In other words, the oldheadersemantics was exactly what was needed for those filetypes, while the newsignaturesemantics makes little sense for them.So I think we should restore the
headerfunctionality, restore its usage instead ofsignaturefor most filetypes, and keep usingsignatureprobably just for distinguishing C/C++/Objective-C headers.When it comes to false positives caused by
headerin the past, were #2894 and #3054 the only known cases of such? If so, in all cases json was guilty, so I think we should fix that simply by removing this"^{$"pattern fromjson.yaml, since this pattern is indeed too general and probably was causing more harm than good.No, as described in #3183 the
signaturecheck is currently used as last resort to sort out multiple detectedfilenames (more or less asVimdoes)See above.
It will iterate over the amount of
detectlimitlines to search for e.g. keywords and or patterns with which the file type can be recognized.As written, it’s currently used to define the chosen one of multiple
filenamematches, depending on the given/defined patterns.In case the pattern is insufficient or
filenamewill be changed without consideringsignatureonce again. That’s definitely something to keep in mind when we decide to add morefilenamepatterns into already existing definitions and don’t create new dedicated definitions.As of now, I agree that it most probably was a fault to keep them available (with the same pattern as before with
header) for everything else other than the C* headers. Because this caused unintended confusion. But as already written, there is still some room to use it as a absolute last fallback.Well, what shall I say/write? We figured out by discussion, since confusion came up due to the lack of proper documentation.
As already often said: This project is already larger than 2-3 maintainers can handle (which was the reason why I vote for you too). 😃 At least labeling should be possible for volunteers, but unfortunately it isn’t.
Help is appreciated with support to prepare the proper doc update. 😉
Placing an option that activates or deactivates the use of
signaturewould be trying to sweep the problem of detecting the file type from the file content down the table, in the same way as placing these bugs only in files that match with more than onefilename.Maybe warn that
signaturemay present errors and ask the user to report bugs found with this option activated in the option documentation, something like:But I still believe that
signatureshould be used to try to find the type of files of typeunknownby default, so signature will have a greater frequency of use than it is today, like https://github.com/zyedidia/micro/pull/3183#issuecomment-2011359501 that shows an example of use of the signature of thejsonfiles and in which scenarios this could happen unexpectedly.But the problem still persists but it will only happen if the file name matches more than one file type, the problem would be the signature patterns, they must be fixed, improved and added so that false positives can be corrected. The
signatureshould be used in case the file type isunknowntoo and that false positives are reported by whoever finds it and that the signatures that would avoid these errors are implemented.