micro: Error detecting file type by `signature`

Description of the problem or steps to reproduce

When opening the Pipfile.lock file it looks like this:

{
    "_meta": {}
}

The type shown is unknown, but as specified in the signature of the file:

https://github.com/zyedidia/micro/blob/433879046e18c79a6780080b233b04154582cbf7/runtime/syntax/json.yaml#L1-L5

I did the same test with the file named index with the following content:

<!DOCTYPE html5>

From what I understand, both should be of type json and html5 given the signature of html5:

https://github.com/zyedidia/micro/blob/433879046e18c79a6780080b233b04154582cbf7/runtime/syntax/html5.yaml#L1-L5

I didn’t get to test the other types of files that can be defined with signature but as both files should match with the regex I believe the same thing must be happening.

This does not happen in version 2.0.13. The file detection was changed with #2819.

Specifications

Commit hash: b518bda5 OS: Ubuntu 22.04.3 LTS Terminal: terminator

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (8 by maintainers)

Most upvoted comments

So I think we should restore the header functionality

Although perhaps we should make header have lower priority than filename, like we now already do with signature (and unlike in the old implementation, where it was non-deterministic whether filename match or header match would take precedence). Perhaps then we won’t even need to fix json?

To me, this is a regression, and a rather serious one.

Ok, then we keep it short and header will return. signature is and can be used to solve multiple hits with the same filename and header will step up in the moment no filename hit at all.

When it comes to false positives caused by header in the past, were https://github.com/zyedidia/micro/issues/2894 and https://github.com/zyedidia/micro/issues/3054 the only known cases of such?

Yep, grep tells that the json definition is right now the only one with a very generic pattern.

Last but not least the doc of signature can then be extended/optimized too.

I’ve read the discussion and I come to a conclusion that we should admit that replacing “headers” with “signatures” was a hasty decision. It is not just a matter of a proper doc update, it is a matter of breakage of useful functionality. header and signature are different things, with quite different purposes, both are useful, we should have added the latter without obstructing the former.

If we have a file named foo with the following content:

#!/bin/sh

echo "hello world"

with v2.0.13 it is recognized and highlighted as a shell script. With the newest master it is not.

To me, this is a regression, and a rather serious one.

The majority of micro’s syntax files have no header/signature patterns. In those syntax files that do, those patterns are mainly shebangs (#!/bin/sh, #!/usr/bin/awk etc) and things like <?xml ... ?> or <!DOCTYPE html5>, i.e. things that: 1. can be quite reliably used for detecting the filetype independently of filename, not just in addition to it. 2. should only occur in the first line of the file, not in the first 100 lines or so. In other words, the old header semantics was exactly what was needed for those filetypes, while the new signature semantics makes little sense for them.

So I think we should restore the header functionality, restore its usage instead of signature for most filetypes, and keep using signature probably just for distinguishing C/C++/Objective-C headers.

When it comes to false positives caused by header in the past, were #2894 and #3054 the only known cases of such? If so, in all cases json was guilty, so I think we should fix that simply by removing this "^{$" pattern from json.yaml, since this pattern is indeed too general and probably was causing more harm than good.

If I put the signature with “%YAML”, any file that has this content of this type?

No, as described in #3183 the signature check is currently used as last resort to sort out multiple detected filenames (more or less as Vim does)

What is this field for?

See above.

How is it used by the micro to carry out detection?

It will iterate over the amount of detectlimit lines to search for e.g. keywords and or patterns with which the file type can be recognized.

In which scenarios would I have which behaviors?

As written, it’s currently used to define the chosen one of multiple filename matches, depending on the given/defined patterns.

In what cases could there be an error?

In case the pattern is insufficient or filename will be changed without considering signature once again. That’s definitely something to keep in mind when we decide to add more filename patterns into already existing definitions and don’t create new dedicated definitions.

How to make proper use of it?

As of now, I agree that it most probably was a fault to keep them available (with the same pattern as before with header) for everything else other than the C* headers. Because this caused unintended confusion. But as already written, there is still some room to use it as a absolute last fallback.

How do I find this information without having to review PR’s and issues?

Well, what shall I say/write? We figured out by discussion, since confusion came up due to the lack of proper documentation.

As already often said: This project is already larger than 2-3 maintainers can handle (which was the reason why I vote for you too). 😃 At least labeling should be possible for volunteers, but unfortunately it isn’t.

Help is appreciated with support to prepare the proper doc update. 😉

Placing an option that activates or deactivates the use of signature would be trying to sweep the problem of detecting the file type from the file content down the table, in the same way as placing these bugs only in files that match with more than one filename.

Maybe warn that signature may present errors and ask the user to report bugs found with this option activated in the option documentation, something like:

* `detectsignature`:  If it is `true`, the `signature` field of the syntax files
   will be used to try to find the `filetype` when it is `unknown` or when
   more than one `filetype` is found (this function may end up generating false positives, if this happens please report the bug so we can improve its functionality).

	default value: `false`

But I still believe that signature should be used to try to find the type of files of type unknown by default, so signature will have a greater frequency of use than it is today, like https://github.com/zyedidia/micro/pull/3183#issuecomment-2011359501 that shows an example of use of the signature of the json files and in which scenarios this could happen unexpectedly.

But besides ^{$ is very unspecific so far and doesn’t really help the initial problem still persists.

But the problem still persists but it will only happen if the file name matches more than one file type, the problem would be the signature patterns, they must be fixed, improved and added so that false positives can be corrected. The signature should be used in case the file type is unknown too and that false positives are reported by whoever finds it and that the signatures that would avoid these errors are implemented.