PX4-Autopilot: Tailsitter SITL Test Failure
Describe the bug
[ 19.039|mavsdk_tests] [11:17:18|Debug] MAVLink: critical: Preflight Fail: vertical velocity unstable (system_impl.cpp:242)
[ 19.039|mavsdk_tests] [11:17:18|Debug] MAVLink: critical: Preflight Fail: Attitude failure (roll) (system_impl.cpp:242)
[ 19.039|mavsdk_tests] [11:17:18|Debug] MAVLink: info: Preflight Fail: No manual control input (system_impl.cpp:242)
[ 19.039|mavsdk_tests] Current speed factor: 1.99231 (set: 20)
[ 19.652|px4 ] WARN [health_and_arming_checks] Preflight Fail: vertical velocity unstable
[ 19.652|px4 ] WARN [health_and_arming_checks] Preflight Fail: height estimate error
[ 19.652|px4 ] WARN [health_and_arming_checks] Preflight Fail: Attitude failure (roll)
[ 19.652|px4 ] INFO [health_and_arming_checks] Preflight Fail: No manual control input
[ 19.880|mavsdk_tests] [11:17:19|Debug] MAVLink: critical: Preflight Fail: vertical velocity unstable (system_impl.cpp:242)
[ 19.880|mavsdk_tests] [11:17:19|Debug] MAVLink: critical: Preflight Fail: height estimate error (system_impl.cpp:242)
[ 19.880|mavsdk_tests] [11:17:19|Debug] MAVLink: critical: Preflight Fail: Attitude failure (roll) (system_impl.cpp:242)
[ 19.880|mavsdk_tests] [11:17:19|Debug] MAVLink: info: Preflight Fail: No manual control input (system_impl.cpp:242)
[ 19.880|mavsdk_tests] [11:17:19|Info ] Timeout, connected to vehicle but waiting for test for 20.1 seconds
[ 19.880|mavsdk_tests] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ 19.880|mavsdk_tests] mavsdk_tests is a Catch v2.13.8 host application.
[ 19.880|mavsdk_tests] Run with -? for options
[ 19.880|mavsdk_tests] -------------------------------------------------------------------------------
[ 19.880|mavsdk_tests] Fly forward in altitude control
[ 19.880|mavsdk_tests] -------------------------------------------------------------------------------
[ 19.880|mavsdk_tests] ../../../test/mavsdk_tests/test_multicopter_manual.cpp:50
[ 19.880|mavsdk_tests] ...............................................................................
[ 19.880|mavsdk_tests] ../../../test/mavsdk_tests/autopilot_tester.cpp:97: FAILED:
[ 19.880|mavsdk_tests] CHECK( poll_condition_with_timeout( [this]() { return _telemetry->health().is_armable; }, std::chrono::seconds(20)) )
[ 19.880|mavsdk_tests] with expansion:
[ 19.880|mavsdk_tests] false
[ 19.886|mavsdk_tests] [11:17:19|Info ] Waiting to get home position
[ 20.009|mavsdk_tests] [11:17:19|Warn ] command temporarily rejected (400). (mavlink_command_sender.cpp:205)
[ 20.010|mavsdk_tests] ../../../test/mavsdk_tests/autopilot_tester.cpp:169: FAILED:
[ 20.010|mavsdk_tests] REQUIRE( result == Action::Result::Success )
[ 20.010|mavsdk_tests] with expansion:
[ 20.010|mavsdk_tests] Command Denied == Success
[ 20.057|mavsdk_tests] ===============================================================================
Example failure run: https://github.com/PX4/PX4-Autopilot/actions/runs/4299642479/jobs/7495070876
The tailsitter SITL Test has been failing since around 2 weeks ago.
Diagnosis
It seems to be happening due to preflight failures (vertical velocity, height estimate, etc unstable).
I also noticed this in real drone using main branch (often the preflight failures would occur easily, with GPS sensor well positioned outside). So I am suspecting it probably is PX4 estimation issue.
Resource
- SITL Test Script: https://github.com/PX4/PX4-Autopilot/blob/e129534a58285d77be68b40705a6b82c761d8156/.github/workflows/sitl_tests.yml#L20
- SITL Test status: https://github.com/PX4/PX4-Autopilot/actions/workflows/sitl_tests.yml
- Latest Successful SITL Test runs: https://github.com/PX4/PX4-Autopilot/actions/workflows/sitl_tests.yml?query=is%3Asuccess
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 20 (20 by maintainers)
Commits related to this issue
- MAVSDK_Test_Runner: Place PX4 instance runner after Gazebo server runner - This was a nasty bug where starting PX4 instance first, then starting Gazebo server was causing PX4 instance' EKF to freak... — committed to junwoo091400/PX4-Autopilot by junwoo091400 a year ago
- Fix SITL Test failure: Place PX4 instance runner after Gazebo server runner (#21230) * MAVSDK_Test_Runner: Place PX4 instance runner after Gazebo server runner - This was a nasty bug where startin... — committed to PX4/PX4-Autopilot by junwoo091400 a year ago
@junwoo091400 Yes, I’m aware of that bug. It is caused by a bad initial attitude (the tailsitter is like a multirotor in hover mode, roll and pitch should be 0). I cannot reproduce it in replay, even with the CI logs (ekf2 replay was enabled to investigate that bug). It might be due to an initial sample that we don’t log but that is used by CI while running the test. The interesting thing is that it only occurs on the tailsitter CI and not every time (if you restart the test, it will most likely not fail again).