platformio-core: Native: PlatformIO reports test as "Failed" even when all passed

What kind of issue is this?

  • Question. This issue tracker is not the place for questions. If you want to ask how to do something, or to understand why something isn’t working the way you expect it to, use Community Forums or Premium Support

  • PlatformIO IDE. All issues related to PlatformIO IDE should be reported to appropriate repository: PlatformIO IDE for Atom or PlatformIO IDE for VSCode

  • Development Platform or Board. All issues (building, uploading, adding new boards, etc.) related to PlatformIO development platforms should be reported to appropriate repository related to your hardware https://github.com/topics/platformio-platform

  • Feature Request. Start by telling us what problem you’re trying to solve. Often a solution already exists! Don’t send pull requests to implement new features without first getting our support. Sometimes we leave features out on purpose to keep the project small.

  • PlatformIO Core. If you’ve found a bug, please provide an information below.

You can erase any parts of this template not applicable to your Issue.


Configuration

Operating system: Ubuntu 20.04.4 LTS

PlatformIO Version (platformio --version): 5.2.5

Description of problem

When running tests on native platform, PlatformIO marks some tests as “Failed” even when unity reports that all tests have been passed successfully and prints final message “OK”. This appears only on native platform. The interesting thing about it is if test-run order is shuffled then everything is ok. image image

Steps to Reproduce

  1. Clone repo: https://github.com/Di-Strix/PIOTestFailDemo
  2. Run test natively using pio test -e native
  3. Get “OK” from unity and “Fail” from PlatformIO
  4. Swap test-run order so that ‘test_injection_empty_file’ runs first and then ‘test_injection_url’
  5. Run test again using pio test -e native
  6. Get “OK” from unity and “Passed” from PlatformIO

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 25 (12 by maintainers)

Most upvoted comments

Finally I resolved this issue. That was completely my fault. Inside tests I create local server instances and provide them to MaterializeLayout. The problem is that MaterializeLayout is global and it is re-initialized before each test. The MaterializeLayout destructor tried to remove all handlers from the provided server but it was removed since test-function is completed. So destructor tried to access deleted local server instance. So there weren’t any issues with PlatformIO core.

Thanks for help

That was the bug in PlatformIO Core 5.0, see https://github.com/platformio/platformio-core/issues/2910

PlatformIO Core MUST resolve all dependencies.