a hj:@sddlZddlmZmZddlZddlmZmZddlmZ m Z ddl m Z ddl mZddlmZgd ZGd d d eZGd d d eZeeeeefZGdddeZGdddeZGdddeZdS)N)OptionalUnion)SizeTensor) functionalinit) Parameter) CrossMapLRN2d)Module)LocalResponseNormr LayerNorm GroupNormRMSNormcsreZdZUdZgdZeed<eed<eed<eed<deeeed d fd d Ze e dddZ ddZ Z S)r aApplies local response normalization over an input signal. The input signal is composed of several input planes, where channels occupy the second dimension. Applies normalization across channels. .. math:: b_{c} = a_{c}\left(k + \frac{\alpha}{n} \sum_{c'=\max(0, c-n/2)}^{\min(N-1,c+n/2)}a_{c'}^2\right)^{-\beta} Args: size: amount of neighbouring channels used for normalization alpha: multiplicative factor. Default: 0.0001 beta: exponent. Default: 0.75 k: additive factor. Default: 1 Shape: - Input: :math:`(N, C, *)` - Output: :math:`(N, C, *)` (same shape as input) Examples:: >>> lrn = nn.LocalResponseNorm(2) >>> signal_2d = torch.randn(32, 5, 24, 24) >>> signal_4d = torch.randn(16, 5, 7, 7, 7, 7) >>> output_2d = lrn(signal_2d) >>> output_4d = lrn(signal_4d) )sizealphabetakrrrr-C6???Nrrrrreturncs&t||_||_||_||_dSNsuper__init__rrrrselfrrrr __class__L/var/www/auris/lib/python3.9/site-packages/torch/nn/modules/normalization.pyr5s  zLocalResponseNorm.__init__inputrcCst||j|j|j|jSr)FZlocal_response_normrrrrrr$r!r!r"forward>szLocalResponseNorm.forwardcCsdjfi|jSNz){size}, alpha={alpha}, beta={beta}, k={k}format__dict__rr!r!r" extra_reprAszLocalResponseNorm.extra_repr)rrr) __name__ __module__ __qualname____doc__ __constants__int__annotations__floatrrr'r- __classcell__r!r!rr"r s  r csleZdZUeed<eed<eed<eed<deeeedd fd d Zeed d dZe dddZ Z S)r rrrrrrr Nrcs&t||_||_||_||_dSrrrrr!r"rKs  zCrossMapLRN2d.__init__r#cCst||j|j|j|jSr)_cross_map_lrn2dapplyrrrrr&r!r!r"r'TszCrossMapLRN2d.forwardrcCsdjfi|jSr(r)r,r!r!r"r-WszCrossMapLRN2d.extra_repr)rrr ) r.r/r0r3r4r5rrr'strr-r6r!r!rr"r Es  r cseZdZUdZgdZeedfed<eed<e ed<de ee e d d fd d Z d d ddZ e e dddZed ddZZS)r a Applies Layer Normalization over a mini-batch of inputs. This layer implements the operation as described in the paper `Layer Normalization `__ .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated over the last `D` dimensions, where `D` is the dimension of :attr:`normalized_shape`. For example, if :attr:`normalized_shape` is ``(3, 5)`` (a 2-dimensional shape), the mean and standard-deviation are computed over the last 2 dimensions of the input (i.e. ``input.mean((-2, -1))``). :math:`\gamma` and :math:`\beta` are learnable affine transform parameters of :attr:`normalized_shape` if :attr:`elementwise_affine` is ``True``. The variance is calculated via the biased estimator, equivalent to `torch.var(input, unbiased=False)`. .. note:: Unlike Batch Normalization and Instance Normalization, which applies scalar scale and bias for each entire channel/plane with the :attr:`affine` option, Layer Normalization applies per-element scale and bias with :attr:`elementwise_affine`. This layer uses statistics computed from input data in both training and evaluation modes. Args: normalized_shape (int or list or torch.Size): input shape from an expected input of size .. math:: [* \times \text{normalized\_shape}[0] \times \text{normalized\_shape}[1] \times \ldots \times \text{normalized\_shape}[-1]] If a single integer is used, it is treated as a singleton list, and this module will normalize over the last dimension which is expected to be of that specific size. eps: a value added to the denominator for numerical stability. Default: 1e-5 elementwise_affine: a boolean value that when set to ``True``, this module has learnable per-element affine parameters initialized to ones (for weights) and zeros (for biases). Default: ``True``. bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`elementwise_affine` is ``True``). Default: ``True``. Attributes: weight: the learnable weights of the module of shape :math:`\text{normalized\_shape}` when :attr:`elementwise_affine` is set to ``True``. The values are initialized to 1. bias: the learnable bias of the module of shape :math:`\text{normalized\_shape}` when :attr:`elementwise_affine` is set to ``True``. The values are initialized to 0. Shape: - Input: :math:`(N, *)` - Output: :math:`(N, *)` (same shape as input) Examples:: >>> # NLP Example >>> batch, sentence_length, embedding_dim = 20, 5, 10 >>> embedding = torch.randn(batch, sentence_length, embedding_dim) >>> layer_norm = nn.LayerNorm(embedding_dim) >>> # Activate module >>> layer_norm(embedding) >>> >>> # Image Example >>> N, C, H, W = 20, 5, 10, 10 >>> input = torch.randn(N, C, H, W) >>> # Normalize over the last three dimensions (i.e. the channel and spatial dimensions) >>> # as shown in the image below >>> layer_norm = nn.LayerNorm([C, H, W]) >>> output = layer_norm(input) .. image:: ../_static/img/nn/layer_norm.jpg :scale: 50 % normalized_shapeepselementwise_affine.r<r=r>h㈵>TN)r<r=r>biasrcs||d}tt|tjr&|f}t||_||_||_|jrt t j |jfi||_ |r|t t j |jfi||_ q|ddn|dd|dd|dS)Ndevicedtyper@weight)rr isinstancenumbersIntegraltupler<r=r>rtorchemptyrDr@register_parameterreset_parameters)rr<r=r>r@rBrCfactory_kwargsrr!r"rs&      zLayerNorm.__init__r9cCs,|jr(t|j|jdur(t|jdSr)r>rones_rDr@zeros_r,r!r!r"rLs  zLayerNorm.reset_parametersr#cCst||j|j|j|jSr)r%Z layer_normr<rDr@r=r&r!r!r"r'szLayerNorm.forwardcCsdjfi|jS)NF{normalized_shape}, eps={eps}, elementwise_affine={elementwise_affine}r)r,r!r!r"r-szLayerNorm.extra_repr)r?TTNN)r.r/r0r1r2rHr3r4r5bool_shape_trrLrr'r:r-r6r!r!rr"r ^s( M!r cseZdZUdZgdZeed<eed<eed<eed<deeeed d fd d Z d d ddZ e e dddZ e d ddZZS)raApplies Group Normalization over a mini-batch of inputs. This layer implements the operation as described in the paper `Group Normalization `__ .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The input channels are separated into :attr:`num_groups` groups, each containing ``num_channels / num_groups`` channels. :attr:`num_channels` must be divisible by :attr:`num_groups`. The mean and standard-deviation are calculated separately over the each group. :math:`\gamma` and :math:`\beta` are learnable per-channel affine transform parameter vectors of size :attr:`num_channels` if :attr:`affine` is ``True``. The variance is calculated via the biased estimator, equivalent to `torch.var(input, unbiased=False)`. This layer uses statistics computed from input data in both training and evaluation modes. Args: num_groups (int): number of groups to separate the channels into num_channels (int): number of channels expected in input eps: a value added to the denominator for numerical stability. Default: 1e-5 affine: a boolean value that when set to ``True``, this module has learnable per-channel affine parameters initialized to ones (for weights) and zeros (for biases). Default: ``True``. Shape: - Input: :math:`(N, C, *)` where :math:`C=\text{num\_channels}` - Output: :math:`(N, C, *)` (same shape as input) Examples:: >>> input = torch.randn(20, 6, 10, 10) >>> # Separate 6 channels into 3 groups >>> m = nn.GroupNorm(3, 6) >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm) >>> m = nn.GroupNorm(6, 6) >>> # Put all 6 channels into a single group (equivalent with LayerNorm) >>> m = nn.GroupNorm(1, 6) >>> # Activating the module >>> output = m(input) ) num_groups num_channelsr=affinerSrTr=rUr?TN)rSrTr=rUrcs||d}t||dkr(td||_||_||_||_|jrxttj |fi||_ ttj |fi||_ n| dd| dd| dS)NrArz,num_channels must be divisible by num_groupsrDr@)rr ValueErrorrSrTr=rUrrIrJrDr@rKrL)rrSrTr=rUrBrCrMrr!r"rs     zGroupNorm.__init__r9cCs"|jrt|jt|jdSr)rUrrNrDrOr@r,r!r!r"rL3s zGroupNorm.reset_parametersr#cCst||j|j|j|jSr)r%Z group_normrSrDr@r=r&r!r!r"r'8szGroupNorm.forwardcCsdjfi|jS)Nz8{num_groups}, {num_channels}, eps={eps}, affine={affine}r)r,r!r!r"r-;szGroupNorm.extra_repr)r?TNN)r.r/r0r1r2r3r4r5rQrrLrr'r:r-r6r!r!rr"rs( -rcseZdZUdZgdZeedfed<ee ed<e ed<de ee e dd fd d Z dd d dZ ejejdddZed ddZZS)raApplies Root Mean Square Layer Normalization over a mini-batch of inputs. This layer implements the operation as described in the paper `Root Mean Square Layer Normalization `__ .. math:: y_i = \frac{x_i}{\mathrm{RMS}(x)} * \gamma_i, \quad \text{where} \quad \text{RMS}(x) = \sqrt{\epsilon + \frac{1}{n} \sum_{i=1}^{n} x_i^2} The RMS is taken over the last ``D`` dimensions, where ``D`` is the dimension of :attr:`normalized_shape`. For example, if :attr:`normalized_shape` is ``(3, 5)`` (a 2-dimensional shape), the RMS is computed over the last 2 dimensions of the input. Args: normalized_shape (int or list or torch.Size): input shape from an expected input of size .. math:: [* \times \text{normalized\_shape}[0] \times \text{normalized\_shape}[1] \times \ldots \times \text{normalized\_shape}[-1]] If a single integer is used, it is treated as a singleton list, and this module will normalize over the last dimension which is expected to be of that specific size. eps: a value added to the denominator for numerical stability. Default: ``torch.finfo(x.dtype).eps`` elementwise_affine: a boolean value that when set to ``True``, this module has learnable per-element affine parameters initialized to ones (for weights). Default: ``True``. Shape: - Input: :math:`(N, *)` - Output: :math:`(N, *)` (same shape as input) Examples:: >>> rms_norm = nn.RMSNorm([2, 3]) >>> input = torch.randn(2, 2, 3) >>> rms_norm(input) r;.r<r=r>NT)r<r=r>rcsv||d}tt|tjr&|f}t||_||_||_|jr^t t j |jfi||_ n | dd|dS)NrArD)rrrErFrGrHr<r=r>rrIrJrDrKrL)rr<r=r>rBrCrMrr!r"ros     zRMSNorm.__init__r9cCs|jrt|jdS)zS Resets parameters based on their initialization used in __init__. N)r>rrNrDr,r!r!r"rLszRMSNorm.reset_parameters)xrcCst||j|j|jS)z$ Runs forward pass. )r%Zrms_normr<rDr=)rrWr!r!r"r'szRMSNorm.forwardcCsdjfi|jS)z5 Extra information about the module. rPr)r,r!r!r"r-szRMSNorm.extra_repr)NTNN)r.r/r0r1r2rHr3r4rr5rQrRrrLrIrr'r:r-r6r!r!rr"rAs$ ( r)rFtypingrrrIrrZtorch.nnrr%rZtorch.nn.parameterrZ _functionsr r7moduler __all__r r3listrRr rrr!r!r!r"s   4]