keras: How to implement a Residul block via keras?

def build_residual_block_conv(num_filters, name, input_shape, input_name=‘x’): “”" Rough sketch of building blocks of layers for residual learning. See http://arxiv.org/abs/1512.03385 for motivation. “”" block = Graph()

block.add_input(input_name, input_shape=input_shape)

h1 = Convolution2D(num_filters, 3, 3,activation='relu', border_mode='same')
block.add_node(h1, name=name+'h1', input=input_name)
block.add_node(Dropout(0.25), name=name+'d1', input=name+'h1')

h2 = Convolution2D(num_filters, 3, 3,activation='linear', border_mode='same')
block.add_node(h2, name=name+'h2', input=name+'d1')

block.add_output(name=name+'output', inputs=[name+'h1', name+'h2'], merge_mode='sum')

return block

About this issue

Original URL
State: closed
Created 9 years ago
Comments: 29 (12 by maintainers)

Most upvoted comments

I implemented a residual class based on the Regularizer class. Not sure if this is the correct way to do it or not, but it seems to work. Also slows down training to a crawl.

For the life of me this thing will not format right, so sorry for the awful formatting…

_CODE START_

from keras.layers.core import Layer from keras.regularizers import Regularizer

class ResidualRegularizer(Regularizer): def init(self): pass

def set_layer(self, layer):
    self.layer = layer

#When asked for the loss, just return 0 to prevent back-prop to the previous layers
def __call__(self, loss):
    return 0

def get_config(self):
    return {"name": self.__class__.__name__}

class Residual(Layer): “”" Layer that passes through its input unchanged, and applies no back propagation. It is simply a forward-propagated link intended for residual linking. “”" def init(self, **kwargs): super(Residual, self).init(**kwargs) residual_regularizer = ResidualRegularizer() residual_regularizer.set_layer(self) self.regularizers = [residual_regularizer]

def get_output(self, train=False):
    return self.get_input(train)

def get_config(self):
    config = {"name": self.__class__.__name__}
    base_config = super(Residual, self).get_config()
    return dict(list(base_config.items()) + list(config.items()))

_CODE END_

Then you implement it like… model.add_node(Dense(1536, activation=‘relu’), merge_mode=‘concat’, concat_axis=-1, name=‘class_dense1’, inputs=[‘flatten_embed’,‘flatten’]) model.add_node(Dense(1536, activation=‘relu’), name=‘class_dense2’, input=‘class_dense1’) model.add_node(Dense(1536, activation=‘relu’), name=‘class_dense3’, input=‘class_dense2’) model.add_node(Residual(), name=‘class_residual’, input=‘class_dense1’) model.add_node(Dense(vocab_size, activation=‘softmax’), name=‘class_softmax’, merge_mode=‘sum’, inputs=[‘class_residual’,‘class_dense3’])

Again sorry for the terrible formatting…

It’s not so much a residual layer as it is just a way to grab a previous layers outputs without back propagating. Then you can take that output and merge it, creating your ‘residual’.

Also note this method only works if all the layers are the same size.

courageon on Feb 17, 2016