ray: [1.10.0 Release] `stress_test_many_tasks` regression

Context: https://github.com/ray-project/ray/pull/21512#discussion_r788310272

Result from 1.10.0:

"stage_3_creation_time": 0.06580400466918945,
"stage_3_time": 2932.907886981964,

This is a significant regression from 1.9:

"stage_3_creation_time": 0.05767488479614258, 
"stage_3_time": 1949.1530861854553

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 25 (23 by maintainers)

Most upvoted comments

The regression happens between 11/13 - 11/20

https://buildkite.com/ray-project/periodic-ci/builds/1621#524f9d0e-5d15-497f-962c-0d209ac19c43 https://buildkite.com/ray-project/periodic-ci/builds/1742#78b364bb-8d5a-476f-975f-3df71e222c77

Between 07add6f7f20fa17af8b539a57131cc97ebb8bbf8 and 88136fa495271a70629ac09ddbf3bc2c5c7e7574

I will take a look at this blocking issue!

I’ll do bisect to find the offending commit.