testng: How to check the test case corresponding to the invocation-numbers value from testng-failed.xml?

do you mean the point 3 in that comment isn’t taken care of in your code sample?

Yes.

I’m wondering if there’s a simpler way to log all the test records corresponding to all the invocation-numbers. Since I see that the test failed is not the one retried, I’d like to check if the testng-failed.xml has correct invocation-numbers in the first place.

There does not exist anything today. Which was why I shared a sample that shows you how to do it.

Does it also mean the retry attempt with such a custom generated testng-failed.xml file as an input will not work since it does not have invocation-numbers (asking this since the sample generated testng suite xml does not have invocation-numbers)?

That was a file that I created manually for demonstration purposes. If you are generating the TestNG suite xml file wherein you would like to build the intelligence of a retry of the failed invocation counts, then you would need to ensure that you include these values as well. What I shared is not meant to be used as a “use as is solution” but something that you would need to work on and tweak. So please try it out and then raise any specific issues that you may have encountered.

If this has answered your question, please feel free to close this issue.

@juherr We are talking about across executions ( which means the state needs to be saved externally) since for every execution the size of the data set varies and so the invocation count changes as well.

If testNG can determine the size of the data provider in advance then maybe it would be able to pin itself against the exact invocation count. But since we also have a lazy loading data provider i am not sure how we would be able to determine the test data size in advance.

@akshayamaldhure

<include name="verifySomething" invocation-numbers="9"/>

This basically is telling you that the data provider’s 10th iteration (it starts from zero) would need to be run again because it failed before. This is TRUE only when the testng-failed.xml gets generated the first time. If you re-run this and if this fails again, then the value would become 0 because this time the data provider was run ONLY with 1 set of data and that also failed.

So I’m still not sure how to find out which exact test case it points to from my DataProvider setup. Any thoughts on how I could achieve the same?

Well this is going to be a tricky one because you would need to build some custom logic. Here’s one way of doing this.

1. Ensure that all the data records implement a custom interface that will help you always uniqueuely identify the record.
2. Use a data provider interceptor to filter out records.
3. build a custom reporter that will produce a file similar to testng-failed.xml but with an extra parameter that lists the keys that failed.

My interface

public interface Identifiable {

  long id();
}

public class Data implements Identifiable {

  private final String name;
  private final int rowNo;

  public Data(int rowNo, String name) {
    this.name = name;
    this.rowNo = rowNo;
  }

  @Override
  public long id() {
    return rowNo;
  }

  @Override
  public String toString() {
    return String.format("(%s, %d)", name, rowNo);
  }
}

import java.util.Arrays;
import java.util.List;
import org.testng.ITestContext;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(SimpleDataProviderInterceptor.class)
public class SampleTestCase {

  @Test(dataProvider = "dp")
  public void testMethod(Data data) {
    System.err.println(data);
  }

  @DataProvider(name = "dp")
  public Object[][] getData(ITestContext context) {
    return records().stream()
        .map(each -> new Object[] {each})
        .toArray(Object[][]::new);
  }

  private static List<Data> records() {
    return Arrays.asList(new Data(10, "a"), new Data(20, "b"));
  }
}

import java.util.Arrays;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.Optional;
import java.util.Spliterator;
import java.util.Spliterators;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;
import org.testng.IDataProviderInterceptor;
import org.testng.IDataProviderMethod;
import org.testng.ITestContext;
import org.testng.ITestNGMethod;

public class SimpleDataProviderInterceptor implements IDataProviderInterceptor {

  @Override
  public Iterator<Object[]> intercept(Iterator<Object[]> original,
      IDataProviderMethod dataProviderMethod, ITestNGMethod method, ITestContext context) {
    List<Integer> keys = keys(context);
    Stream<Object[]> stream = StreamSupport.stream(
        Spliterators.spliteratorUnknownSize(
            original,
            Spliterator.ORDERED), false);
    if (!keys.isEmpty()) {
      List<Object[]> list = stream
          .map(each -> (Data) each[0])
          .filter(each -> keys.contains((int) each.id()))
          .map(each -> new Object[]{each})
          .collect(Collectors.toList());
      if (list.isEmpty()) {
        throw new IllegalStateException("Data Provider filtering resulted in no records");
      }
      return list.iterator();
    }
    return original;
  }

  private static List<Integer> keys(ITestContext ctx) {
    String keys = Optional.ofNullable(ctx.getCurrentXmlTest().getParameter("dataprovider.keys"))
        .orElse("");
    if (keys.isEmpty()) {
      return Collections.emptyList();
    }
    return Arrays.stream(keys.split(","))
        .mapToInt(Integer::parseInt)
        .boxed()
        .collect(Collectors.toList());
  }
}

A testng suite xml file that you can generate should be like below

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="my_suite">
  <test thread-count="5" name="my_test" verbose="2">
    <parameter name="dataprovider.keys" value="10,20"/>
    <classes>
      <class name="com.rationaleemotions.issue2853.SampleTestCase"/>
    </classes>
  </test> <!-- my_test -->
</suite> <!-- my_suite -->