DeepSpeed: [BUG] "Setting ds_accelerator" uses `print` not `logger`
Describe the bug
When accelerate
tries to run any command while DeepSpeed
is available, we get "Setting ds_accelerator ..."
, which while not explicitly bad, is making a test fail due to a non-clean CLI print when non-deepspeed code is used (we have a check for is_deepspeed_available()
which is probably triggering this.
As a nice QOL, it would be good to have get_accelerator
(https://github.com/microsoft/DeepSpeed/blob/master/accelerator/real_accelerator.py#L102) and set_accelerator
(https://github.com/microsoft/DeepSpeed/blob/master/accelerator/real_accelerator.py#L109) use the already-existing logger in the framework if possible, so that we can disable these annoying prints when theyโre not needed ๐
To Reproduce Steps to reproduce the behavior:
pip install accelerate deepspeed -U
- Create a file with:
from accelerate.commands.tpu import tpu_command_parser, tpu_command_launcher
parser = tpu_command_parser()
args = parser.parse_args([
"--config_file", "tests/test_configs/latest.yaml",
"--install_accelerate",
"--debug"
])
tpu_command_launcher(args)
- Run
python {my_file_name.py}
- Should print:
Setting ds_accelerator to cuda (auto detect)
Running gcloud compute tpus tpu-vm ssh test-tpu --zone us-central1-a --command cd /usr/share; pip install accelerate -U; echo "hello world"; echo "this is a second command" --worker all
Expected behavior
A configurable option to silence these print statements by having them run through the logging system instead, perhaps as an info
or as a debug
.
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 21 (9 by maintainers)
@tjruwase I have update to 0.94. And There are still too much message โ[real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)โ printed in std io.
The bug still exists with version
0.10.2
@tjruwase nope, not with DeepSpeed yet (@pacman100 on that one with Accelerate?) itโs on a GPU runner, just our CLI picks up on it due to import (which was also a great way to know that something weird was happening ๐ )
No worries. Thanks for reporting the error. I will link a PR asap.
@tjruwase sadly donโt have the bandwidth/familiarity with the code right now to do so, would appreciate it if someone else could pick it up ๐