Jump to content

Draft:Layer normalization

From Wikipedia, the free encyclopedia
  • Comment: One section is completely unsourced. Hitro talk 08:22, 21 September 2023 (UTC)

Layer normalization (also known as layer norm) helps make training of artificial neural networks faster and more stable. It is presented as an alternative to batch normalization. In layer norm, neural networks compute a mean and variance using all the summed inputs to the neuron in a single training step. This information is then used to normalize the summed input to that neuron for each and every training step. Layer normalization is commonly found in recurrent neural networks and in the Transformer (machine learning model).[1][2]

Introduction and Motivation

[edit]

Complex artificial neural networks trained with stochastic gradient descent often require multiple days of training and large amounts of computing power[citation needed]. One approach to reducing training time is to standardize the neuron inputs performed during feedforward to make the learning easier. Batch normalization achieves this by standardizing each neuron's input using the data's mean and standard deviation. However, batch normalization requires storing the mean and standard deviation for each layer. This is not possible with an RNN because RNNs typically vary in sequence length. Layer normalization mitigates this by directly producing normalization statistics from the layer inputs so that the statistics do not need to be stored between training cases.

Procedure

[edit]

The outputs of hidden layers in a deep neural network tend to be highly correlated with one another. This means the mean and variance of the summed layer inputs can be fixed as constants. The computation for layer norm statistics for all neurons in the same hidden layer is:[2][3]

Assume that is the number of neurons in the hidden layer.

The mean is:

The variance is:

References

[edit]
  1. ^ "Papers with Code - Layer Normalization Explained". paperswithcode.com. Retrieved 2023-08-29.
  2. ^ a b Lei Ba, Jimmy (21 Jul 2016). "Layer Normalization". arXiv:1607.06450 [stat.ML].
  3. ^ Xu, Jingjing (16 Nov 2019). "Understanding and Improving Layer Normalization". Advances in Neural Information Processing Systems. arXiv:1911.07013.