ava: TAP output is inconsistent with other producers

Description

The TAP output from AVA is inconsistent with other TAP producers in two ways.

Console logs within a test appear before the test title.
Console logs within a test are written to stderr.

Forgive me if either of these are a feature or reported before. There are many issues that may be related (example https://github.com/avajs/ava/issues/392) but I couldn’t find any that are exactly these issues.

Below I compare the output of ava’s TAP reporter to substack/tape. I see the same inconsistency when comparing to tapjs/node-tap.

Test Source

In ava: (using test.cb for comparison).

var test = require('ava');

test.cb('ava test', function (t) {
  console.log('inside ava test');
  t.pass();
  t.end();
});

test.cb('ava test #2', function (t) {
  console.log('inside ava test #2');
  t.pass();
  t.end();
});

test.cb('ava test #3', function (t) {
  console.log('inside ava test #3');
  t.pass();
  t.end();
});

outputs:

TAP version 13
inside ava test
# ava test
ok 1 - ava test
inside ava test #2
# ava test #2
ok 2 - ava test #2
inside ava test #3
# ava test #3
ok 3 - ava test #3

1..3
# tests 3
# pass 3
# fail 0

Notice that the “inside ava test” console logs appear before the test title. Additionally, the console logs are output to stderr which makes it difficult to work with TAP consumers (for example https://github.com/Hypercubed/tap-markdown).

in tape:

var test = require('tape');

test('tape test', function (t) {
  console.log('inside tape test');
  t.pass();
  t.end();
});

test('tape test #2', function (t) {
  console.log('inside tape test #2');
  t.pass();
  t.end();
});

test('tape test #3', function (t) {
  console.log('inside tape test #3');
  t.pass();
  t.end();
});

outputs:

TAP version 13
# tape test
inside tape test
ok 1 (unnamed assert)
# tape test #2
inside tape test #2
ok 2 (unnamed assert)
# tape test #3
inside tape test #3
ok 3 (unnamed assert)

1..3
# tests 3
# pass  3

# ok

Console logs are output after titles and to stdout.

Command-Line Arguments

For ava:

ava ./ava-test.js --tap --serial

For tape:

tape ./tape-test.js

Environment

Node.js v6.2.1
darwin 15.2.0
ava 0.15.2
npm 3.9.3

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 16 (9 by maintainers)

Most upvoted comments

@novemberborn The problem with TAP is that for the most part it’s largely unspecified. The spec only states that you need to declare the number of ok/not oks, and then you do the actual ok/not oks. That and support for #TODO and #SKIP is about it. The rest is convention people have built on top of TAP. TAP also specifies that anything a tap parser doesn’t understand should be ignored.

The entire thing seems to me like a confusing half-specified text protocol that people implement their own parsers for that are to a certain extent incompatible, which is bound to lead to problem downstream. IMHO, ava would be much better served by either allowing custom reporters (which have already been dismissed because TAP is apparently supposed to solve all of these problem), or invent a streaming JSON protocol or similar (something that has easy expandable data, and cannot be misinterpreted).

Simple translation from TAP for instance:

{"id":0,"file":"file.js","name":"test name","result":"success|skipped|todo|failed",......}
{"id":1,"file":"other.js",....}

The advantage with this, is that like TAP it’s streamed (one line per assert), but you get all the data inline. There is no speculation. No conventions. Sure, different keys might be used, but it should always be valid JSON. And none of the “anything not understood by a parser is ignored” bullshit that people abuse to tuck stacktraces in there.

And if we want it to be standard, than create a simple standard for it. Define required, optional, and expansion fields for each assert. Figure out how to deal with grouping, etc. Or find some standard that already solves this. Because as it stands, more than 50 percent of the TAP output the TAP parser produces, is not actually TAP. It’s just conventions somebody invented because they tried to shoehorn more data into TAP. It’s not standard, nor specified. At least as far as I’ve been able to find.

Now, enough of my rant. Regardless of whether we stay on TAP, do something else, or both, I think the reporter should have the option to emit each assert. So that if I wanted to I could format it like this:

myTest
> assert1
> assert2

Alxandr on May 29, 2017