APPFL: ResNet Runtime Error
While running ResNet with 5% of LEAF FEMNIST sample dataset, the following runtime error occurred.
Traceback (most recent call last):
File "./femnist.py", line 163, in <module>
main()
File "./femnist.py", line 156, in main
rm.run_client(cfg, comm, model, loss_fn, num_clients, train_datasets)
File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/appfl/run_mpi.py", line 270, in run_client
local_states[cid] = client.update()
File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/appfl/algorithm/client_optimizer.py", line 63, in update
output = self.model(data)
File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/torchvision/models/resnet.py", line 285, in forward
return self._forward_impl(x)
File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/torchvision/models/resnet.py", line 268, in _forward_impl
x = self.conv1(x)
File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 457, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/home/yim/.conda/envs/APPFL/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 453, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[64, 1, 28, 28] to have 3 channels, but got 1 channels instead
This error occurs for both channel 1 and 3 parameters. The dataset works well with CNN.
The configuration for APPFL is below:
fed:
type: federated
servername: ServerFedAvg
clientname: ClientOptim
args:
server_learning_rate: 0.01
server_adapt_param: 0.001
server_momentum_param_1: 0.9
server_momentum_param_2: 0.99
optim: Adam
num_local_epochs: 1
optim_args:
lr: 0.001
epsilon: false
clip_value: false
clip_norm: 1
device: cuda
num_clients: 1
num_epochs: 100
num_workers: 0
batch_training: true
train_data_batch_size: 64
train_data_shuffle: false
validation: true
test_data_batch_size: 64
test_data_shuffle: false
data_sanity: false
reproduce: true
pca_dir: ''
params_start: 0
params_end: 49
ncomponents: 40
use_tensorboard: true
load_model: false
load_model_dirname: ''
load_model_filename: ''
save_model: false
save_model_dirname: ''
save_model_filename: ''
checkpoints_interval: 2
save_model_state_dict: false
output_dirname: output
output_filename: result
logginginfo: {}
summary_file: ''
max_message_size: 104857600
operator:
id: 1
server:
id: 1
host: localhost
port: 50051
use_tls: false
api_key: null
client:
id: 1
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 16 (6 by maintainers)
Update:
The cause of the OverflowError The cause of the error seems to be that the total data bytes from all MPI clients over the maximum boundary of the MPI gather function. This is because, a. The data points in the model’s state_dict from each client with ResNet are 11211666.
b. For the float numbers, the total bytes from each client are 11211666 * 4 = 44846664. c. For the error condition, I used client number 197. Then, the gather function needs to capture 44846664 * 197 = 8,834,792,808. Which is similar to the error message “OverflowError: integer 8842650894 does not fit in ‘int’”
The running condition In this case, the possible number of clients is 47, 2147483647(max integer number) / 44846664(the model’s state in a client) = ~47.xx. By the testing in my local and SWING cluster, I have the same result. The 47 clients setting runs well with ResNet The 48 clients setting generates the OverflowError: OverflowError: integer 2154542278 does not fit in ‘int’
So, it seems the maximum number of MPI clients should be decided according to the data length from the selected model’s state.
@yim0331 To handle small images, please see this: https://github.com/keras-team/keras/issues/11053. You should be able to write a similar code in pytorch: https://pytorch.org/docs/stable/generated/torch.nn.ZeroPad2d.html
Update:
In-progress:
Update: The error occurred by the channel mismatch between the model and preprocessed data. APPFL uses ResNet implementation which is “resnet18” module in Torchvision. The channel configuration in Tourchvision is fixed to 3: Line 197 at https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py. However, LEAF FEMINIST data is preprocessed by channel 1(grey).
In-progress: Testing with new preprocessed data with channel 3