APPFL: ResNet Runtime Error

While running ResNet with 5% of LEAF FEMNIST sample dataset, the following runtime error occurred.

Traceback (most recent call last):
  File "./femnist.py", line 163, in <module>
    main()
  File "./femnist.py", line 156, in main
    rm.run_client(cfg, comm, model, loss_fn, num_clients, train_datasets)
  File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/appfl/run_mpi.py", line 270, in run_client
    local_states[cid] = client.update()
  File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/appfl/algorithm/client_optimizer.py", line 63, in update
    output = self.model(data)
  File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/torchvision/models/resnet.py", line 285, in forward
    return self._forward_impl(x)
  File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/torchvision/models/resnet.py", line 268, in _forward_impl
    x = self.conv1(x)
  File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 457, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 453, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[64, 1, 28, 28] to have 3 channels, but got 1 channels instead

This error occurs for both channel 1 and 3 parameters. The dataset works well with CNN.

The configuration for APPFL is below:

fed:
  type: federated
  servername: ServerFedAvg
  clientname: ClientOptim
  args:
    server_learning_rate: 0.01
    server_adapt_param: 0.001
    server_momentum_param_1: 0.9
    server_momentum_param_2: 0.99
    optim: Adam
    num_local_epochs: 1
    optim_args:
      lr: 0.001
    epsilon: false
    clip_value: false
    clip_norm: 1
device: cuda
num_clients: 1
num_epochs: 100
num_workers: 0
batch_training: true
train_data_batch_size: 64
train_data_shuffle: false
validation: true
test_data_batch_size: 64
test_data_shuffle: false
data_sanity: false
reproduce: true
pca_dir: ''
params_start: 0
params_end: 49
ncomponents: 40
use_tensorboard: true
load_model: false
load_model_dirname: ''
load_model_filename: ''
save_model: false
save_model_dirname: ''
save_model_filename: ''
checkpoints_interval: 2
save_model_state_dict: false
output_dirname: output
output_filename: result
logginginfo: {}
summary_file: ''
max_message_size: 104857600
operator:
  id: 1
server:
  id: 1
  host: localhost
  port: 50051
  use_tls: false
  api_key: null
client:
  id: 1

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 16 (6 by maintainers)

Most upvoted comments

Update:

  1. The cause of the OverflowError The cause of the error seems to be that the total data bytes from all MPI clients over the maximum boundary of the MPI gather function. This is because, a. The data points in the model’s state_dict from each client with ResNet are 11211666.
    b. For the float numbers, the total bytes from each client are 11211666 * 4 = 44846664. c. For the error condition, I used client number 197. Then, the gather function needs to capture 44846664 * 197 = 8,834,792,808. Which is similar to the error message “OverflowError: integer 8842650894 does not fit in ‘int’”

  2. The running condition In this case, the possible number of clients is 47, 2147483647(max integer number) / 44846664(the model’s state in a client) = ~47.xx. By the testing in my local and SWING cluster, I have the same result. The 47 clients setting runs well with ResNet The 48 clients setting generates the OverflowError: OverflowError: integer 2154542278 does not fit in ‘int’

So, it seems the maximum number of MPI clients should be decided according to the data length from the selected model’s state.

@yim0331 To handle small images, please see this: https://github.com/keras-team/keras/issues/11053. You should be able to write a similar code in pytorch: https://pytorch.org/docs/stable/generated/torch.nn.ZeroPad2d.html

Update:

  1. Figure out the problem while batch normalization
  1. Another error occurs while gathering the output from all clients. Traceback (most recent call last): File “./femnist.py”, line 161, in <module> main() File “./femnist.py”, line 154, in main rm.run_client(cfg, comm, model, loss_fn, args.num_clients, train_datasets) File “/gpfs/fs1/home/yim/workspace/APPFL-install/APPFL/src/appfl/run_mpi.py”, line 273, in run_client comm.gather(local_states, root=0) File “mpi4py/MPI/Comm.pyx”, line 1578, in mpi4py.MPI.Comm.gather File “mpi4py/MPI/msgpickle.pxi”, line 772, in mpi4py.MPI.PyMPI_gather File “mpi4py/MPI/msgpickle.pxi”, line 147, in mpi4py.MPI.pickle_dump File “mpi4py/MPI/msgbuffer.pxi”, line 50, in mpi4py.MPI.downcast OverflowError: integer 8842650894 does not fit in ‘int’

In-progress:

  • Verify the output from each client
  • Debugging the gathering problem.

Update: The error occurred by the channel mismatch between the model and preprocessed data. APPFL uses ResNet implementation which is “resnet18” module in Torchvision. The channel configuration in Tourchvision is fixed to 3: Line 197 at https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py. However, LEAF FEMINIST data is preprocessed by channel 1(grey).

In-progress: Testing with new preprocessed data with channel 3