kotest: Confusing block execution with isInstancePerTest=true
I was trying to debug some test issues and was getting really confused following the code flow when isInstancePerTest
is set to true
. I really like isInstancePerTest
as it makes ensuring state is correct a lot easier, but it’s confusing how many times each block is executed. In this example test:
internal class ClassUnderTestTest : ShouldSpec() {
override fun isInstancePerTest(): Boolean = true
init {
"Scope 1" {
println(this.context.description().name)
"Scope 1a" {
println(this.context.description().name)
should("1aa") {
println(this.description().name)
1 shouldBe 1
}
"Scope 1ab" {
println(this.context.description().name)
should("1aba") {
println(this.description().name)
1 shouldBe 1
}
}
}
"Scope 1b" {
println(this.context.description().name)
should("1ba") {
println(this.description().name)
1 shouldBe 1
}
"Scope 1bb" {
println(this.context.description().name)
should("1bba") {
println(this.description().name)
1 shouldBe 1
}
}
}
"Scope 1c" {
println(this.context.description().name)
should("1ca") {
println(this.description().name)
1 shouldBe 1
}
}
}
}
}
With isInstancePerTest
set to false, it prints:
Scope 1
Scope 1a
should 1aa
Scope 1ab
should 1aba
Scope 1b
should 1ba
Scope 1bb
should 1bba
Scope 1c
should 1ca
Which seems expected. With isInstancePerTest
set to true, it prints:
Scope 1
Scope 1
Scope 1a
Scope 1
Scope 1a
should 1aa
Scope 1
Scope 1a
Scope 1ab
Scope 1
Scope 1a
Scope 1ab
should 1aba
Scope 1
Scope 1b
Scope 1
Scope 1b
should 1ba
Scope 1
Scope 1b
Scope 1bb
Scope 1
Scope 1b
Scope 1bb
should 1bba
Scope 1
Scope 1c
Scope 1
Scope 1c
should 1ca
Whereas I would have expected:
Scope 1
Scope 1a
should 1aa
Scope 1
Scope 1a
Scope 1ab
should 1aba
Scope 1
Scope 1b
should 1ba
Scope 1bb
should 1bba
Scope 1
Scope 1c
should 1ca
I’ve posted on here before about test execution order and I know it’s a tricky thing, but it’s pretty surprising to try and step through the code with the current way it executes using isInstancePerTest=true
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 22 (22 by maintainers)
Commits related to this issue
- Discovered nested tests now run before previously discovered tests #510 — committed to kotest/kotest by sksamuel 5 years ago
I’ve fixed this so will be in the next release candidate.
cool, thanks @sksamuel. Excited to upgrade to 3.2.