eslint: messageId messages with placeholders fail during tests.

Tell us about your environment

• ESLint Version: 4.15+
• Node Version: 8.9.3
• npm Version: 5.5.1

What parser (default, Babel-ESLint, etc.) are you using? default

Please show your full configuration: N/A

What did you do? Please include the actual source code causing the issue, as well as the command that you used to run ESLint. Write a test where the message is a placeholder and the message is referred to via the new meta.messageId. Run the test runner against it with a messageId to match against.

module.exports = {
    meta: {
        messages: {
            avoidName: "Avoid using variables named '{{ name }}'"
        }
    },
    create(context) {
        return {
            Identifier(node) {
                if (node.name === 'foo') {
                    context.report({
                        node,
                        messageId: 'avoidName',
                        data: {
                            name: 'foo'
                        }
                    });
                }
            }
        };
    }
};

'use strict';
var rule = require('../../../lib/rules/no-foo');
var RuleTester = require('eslint').RuleTester;

var ruleTester = new RuleTester();
ruleTester.run('no-foo', rule, {
    valid: ['bar', 'baz'],
    invalid: [
        {
            code: 'foo',
            errors: [
                {
                    messageId: 'avoidName'
                }
            ]
        }
    ]
});

npm run test

What did you expect to happen? Tests pass.

What actually happened? Please include the actual, raw output from ESLint.

  1 failing

  1) no-foo invalid foo:

      AssertionError [ERR_ASSERTION]: 'Avoid using variables named \'foo\'' === 'Avoid using variables named \'{{ name }}\''
      + expected - actual

      -Avoid using variables named 'foo'
      +Avoid using variables named '{{ name }}'

      at assertMessageMatches (node_modules\eslint\lib\testers\rule-tester.js:442:24)
      at testInvalidTemplate (node_modules\eslint\lib\testers\rule-tester.js:517:29)
      at Context.RuleTester.it (node_modules\eslint\lib\testers\rule-tester.js:581:25)

So it looks like ruleTester is comparing the value of messageId’s pre-formatted template to the post-formatted runtime error.

It seems like the ideal behavior is:

1. If your test needs to get the post-formatted runtime report message, use message in the ruleTester.
2. If your test wants to know that a specific rule ID was used to generate the error message, use mesageId in the ruleTester.

My guess is that most use cases fall under the second category, but that the first category is worth supporting as well to give access to the direct output during testing if needed.