tensor_parallel: why raised cuda error?

i used tiiuae/falcon-40b

and want to doing full fine-tuning by lima instruction dataset

model = tp.tensor_parallel(model, sharded=True)

just use like this and i have 1) A100 80GB * 2 and another server 2) A100 80GB * 4 but when i running code on 1) device or 2) device raised error like this Model parameters were moved to incorrect devices, did call on model.cuda() or model.to(device)? If so, please avoid doing that

why?

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 18 (2 by maintainers)

Most upvoted comments

upper transformers v4.30.0, i think this error is accured