levanter: Sharding error when trying to serialize PEFT models

#622 fixes issue #609 for loading HF models into a sharded representation. But now when I try to serialize a PEFT model I’m getting a similar error as before:

ValueError: Received incompatible devices for jitted computation. Got argument x of _identity_fn with shape float32[32,8,4096] and device ids [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] on platform TPU and explicit output sharding with device ids [0, 1, 4, 5, 2, 3, 6, 7, 8, 9, 12, 13, 10, 11, 14, 15] on platform TPU

I tried the simple thing of removing the sharding annotation entirely and just running jax.array(input) but that yields the (I guess expected) error since I’m assuming the original sharding is spread across multiple machines:

RuntimeError: Fetching value for `jax.Array` that spans non-addressable devices is not possible. You can use `jax.experimental.multihost_utils.process_allgather` for this use case.

I haven’t yet reproduced as a test, but you can reproduce using the gsm8k example:

python infra/launch.py\ --foreground\ --tpu_name=tpu-0\
 --zone=us-west4-a\
 --\ 
 python examples/gsm8k-lora/gsm8k_lora.py\
 --config=examples/gsm8k-lora/gsm8k-llama2.yaml\
 --trainer.checkpointer.base_path=gs://wasabi-tpu-training/llama3-gsm8k/attempt-0\
 --hf_save_path=gs://wasabi-tpu-training/llama3-gsm8k/attempt-0\
 --data_cache_dir=gs://wasabi-tpu-training/gsm8k/data\
 --trainer.num_train_steps=10\
 --data_seed=0

The script needs one patch to load models correctly (there’s an error now if you try to load a model using a name the way the script used to). I’ll clean up the fixes and send them as a separate PR, but for now:

https://github.com/stanford-crfm/levanter/compare/main...rjpower:levanter:multi-lora?expand=1

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (8 by maintainers)

Most upvoted comments

Talking to @yashk2810, it seems like device_put is more capable than I thought (in particular it seems like it can do cross-host transfers)

import jax
import jax.numpy as jnp
import numpy as np
from jax.sharding import Mesh, PartitionSpec as P, NamedSharding
import jax.experimental.mesh_utils as mesh_utils

D = 4096

mesh_devices = mesh_utils.create_device_mesh((len(jax.devices()),))

smart_mesh = Mesh(mesh_devices, ('dp',))
# mesh = Mesh(jax.devices(), ('dp'))
host_mesh = Mesh(np.array(jax.devices()).reshape(jax.process_count(), -1), ('host', 'device'))

z = jnp.full((D, D), jax.process_index())

print(z.sharding)

smart_sharding = NamedSharding(smart_mesh, P('dp'))
host_sharding = NamedSharding(host_mesh, P('device'))

z2 = jax.jit(lambda: jnp.zeros((D, D)), out_shardings=smart_sharding)()

print(z2.sharding)

z3 = jax.jit(lambda: jnp.zeros((D, D)), out_shardings=host_sharding)()
print(z3.sharding)

# this is no good:
# Traceback (most recent call last):
#   File "/home/dlwh/test_device_put.py", line 30, in <module>
#     z4 = jax.jit(lambda x: x, out_shardings=host_sharding)(z2)
# ValueError: Received incompatible devices for jitted computation. Got argument x of <lambda> with shape float32[4096,4096] and device ids

# z4 = jax.jit(lambda x: x, out_shardings=host_sharding)(z2)
# print(z4.sharding)

z5 = jax.device_put(z2, host_sharding)

print(z5.sharding)