ray: [Bug] ray monitor reports negative usage

Search before asking

  • I searched the issues and found no similar issues.

Ray Component

Ray Core, Monitoring & Debugging

What happened + What you expected to happen

Started a script, left it run for a couple of hours, interrupted with ctrl+c, scripts exits, ray monitor shows negative usage:

Usage:
 -2.0/52.0 CPU (0.0 used of 2.0 reserved in placement groups)
 0.0/2.0 GPU
 0.0/1.0 accelerator_type:G
 0.0/1.0 accelerator_type:GT
 0.00/103.633 GiB memory
 0.00/48.406 GiB object_store_memory

Demands:
 (no resource demands)

Reproduction script

search_optuna = optuna.OptunaSearch()
    tune.run(
        tune.with_parameters(trainable, data=[df.index.values, df.bid.values, df.ask.values, df.decimals_price[0]]),
        name="DEMA_single_OM_process",
        search_alg=search_optuna,
        time_budget_s=3600*24*30,
        # resume=True,
        num_samples=-1,
        verbose=3,  
        config=conf,
        metric="score",
        mode="max",
        fail_fast=True,
        reuse_actors=True
    )

Anything else

ray 1.7.0

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (16 by maintainers)

Most upvoted comments

Btw, we will have a release for 1.8 very soon. We can revisit this if it still happens on 1.8