chainer: Exponential Shift does not effect some links after the links' lr are manually modified

• Conditions
  • Chainer version: 5.0.0b1
  • CuPy version: 5.0.0b1
  • OS/Platform: Ubuntu 16.04
  • CUDA/cuDNN version: 9.0

Problem

I tried to reproduce lr_mult in caffe, I multiply hyperparameter lr of some links. And I use ExponentialShift to multiply all lr in the model, but the modified link’s lr does not change. I tried with the code below, and conv1’s lr does not change and still remain to 0.03. I expected that conv1’s lr will be 0.15. Is this expected return for optimizer in Chainer?

import chainer


class ExampleModel(chainer.Chain):

    def __init__(self):
        super(ExampleModel, self).__init__()
        with self.init_scope():
            self.conv1 = chainer.links.Convolution2D(3, 10, 1, 1)
            self.conv2 = chainer.links.Convolution2D(10, 10, 1, 1)


def main():
    model = ExampleModel()
    optimizer = chainer.optimizers.MomentumSGD(lr=0.01)
    optimizer.setup(model)
    model.conv1.W.update_rule.hyperparam.lr *= 3
    print('Done: conv1 lr * 3')
    print('  conv1 lr: {}'.format(model.conv1.W.update_rule.hyperparam.lr))
    print('  conv2 lr: {}'.format(model.conv2.W.update_rule.hyperparam.lr))

    optimizer.lr *= 5
    print('Done: optimizer.lr * 5')
    print('  conv1 lr: {}'.format(model.conv1.W.update_rule.hyperparam.lr))
    print('  conv2 lr: {}'.format(model.conv2.W.update_rule.hyperparam.lr))


if __name__ == '__main__':
    main()

output

$ python example.py
Done: conv1 lr * 3
  conv1 lr: 0.03
  conv2 lr: 0.01
Done: optimizer.lr * 5
  conv1 lr: 0.03
  conv2 lr: 0.05