a hd @s$dZddlZddlZddlmZmZmZmZm Z ddl m Z ddl Z ddl m Z gdZedZe dZed Zed ZdMe eeee je d d d ZdNe eeee je dddZdOe eeeeee je dddZe ee dddZe e dddZdPeee eefedddZdQe eeee je d dd ZdRe eeee je dd!d"ZdSe eeeeee je dd%d&Ze ee dd'd(Ze e dd)d*Z e e dd+d,Z!e e dd-d.Z"dTe ee d0d1d2Z#e e$eefdd3d4Z%dUe eee je d5d6d7Z&dVe eee je d5d8d9Z'e eed:d;d<Z(dWe eeeee je d?d@dAZ)dXe eeeee je d?dBdCZ*dYe eee je d5dDdEZ+dZe eeee je dGdHdIZ,eeefeeefdJdKdLZ-e-eZ.e-eZ/e-eZ0e-e"Z1e-e#Z2e-e&Z3e-e'Z4e-e)Z5e-e*Z6e-e+Z7e-e,Z8dS)[zHThis file contains utilities for initializing neural network parameters.N)CallableLiteralOptionalTypeVarUnion) ParamSpec)Tensor)calculate_gainuniform_normal_ trunc_normal_ constant_ones_zeros_eye_dirac_xavier_uniform_xavier_normal_kaiming_uniform_kaiming_normal_ orthogonal_sparse_uniformnormalconstanteyediracxavier_uniform xavier_normalkaiming_uniformkaiming_normal orthogonalsparse_R_P) linearconv1dconv2dconv3dconv_transpose1dconv_transpose2dconv_transpose3dsigmoidtanhrelu leaky_reluselu)fan_infan_out)tensorab generatorreturncCs<t |j|||dWdS1s.0YdSNr6)torchno_gradr r3r4r5r6r=;/var/www/auris/lib/python3.9/site-packages/torch/nn/init.py_no_grad_uniform_Ds r?)r3meanstdr6r7cCs<t |j|||dWdS1s.0YdSr8)r:r;r r3r@rAr6r=r=r>_no_grad_normal_Ks rC)r3r@rAr4r5r6r7c Csttddd}||d|ks0||d|kr>tjdddt||||}||||}|jd|dd|d|d|||t d | ||j ||d |WdS1s0YdS) N)xr7cSsdt|tddS)N?@)matherfsqrt)rDr=r=r>norm_cdf^sz(_no_grad_trunc_normal_..norm_cdfzjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect. stacklevelr9rF)minmax) floatwarningswarnr:r;r Zerfinv_mul_rGrIZadd_Zclamp_) r3r@rAr4r5r6rJlur=r=r>_no_grad_trunc_normal_Us     rW)r3valr7cCs6t||WdS1s(0YdSN)r:r;Zfill_r3rXr=r=r>_no_grad_fill_s r[)r3r7cCs4t|WdS1s&0YdSrY)r:r;zero_r3r=r=r>_no_grad_zero_s r^) nonlinearityparamr7cCsgd}||vs|dkrdS|dkr(dS|dkr:tdS|dkr|d urPd }n4t|tsdt|tsnt|trt|}ntd |d tdd|d S|dkrdStd|d S)aReturn the recommended gain value for the given nonlinearity function. The values are as follows: ================= ==================================================== nonlinearity gain ================= ==================================================== Linear / Identity :math:`1` Conv{1,2,3}D :math:`1` Sigmoid :math:`1` Tanh :math:`\frac{5}{3}` ReLU :math:`\sqrt{2}` Leaky Relu :math:`\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}` SELU :math:`\frac{3}{4}` ================= ==================================================== .. warning:: In order to implement `Self-Normalizing Neural Networks`_ , you should use ``nonlinearity='linear'`` instead of ``nonlinearity='selu'``. This gives the initial weights a variance of ``1 / N``, which is necessary to induce a stable fixed point in the forward pass. In contrast, the default gain for ``SELU`` sacrifices the normalization effect for more stable gradient flow in rectangular layers. Args: nonlinearity: the non-linear function (`nn.functional` name) param: optional parameter for the non-linear function Examples: >>> gain = nn.init.calculate_gain( ... "leaky_relu", 0.2 ... ) # leaky_relu with negative_slope=0.2 .. _Self-Normalizing Neural Networks: https://papers.nips.cc/paper/2017/hash/5d44ee6f2c3f71b73125876103c8f6c4-Abstract.html )r%r&r'r(r)r*r+r,rNr-g?r.rFr/N{Gz?znegative_slope z not a valid numberrKr0g?zUnsupported nonlinearity )rGrI isinstanceboolintrQ ValueError)r_r`Z linear_fnsZnegative_sloper=r=r>r s.&  r rEcCs4tj|r&tjjt|f||||dSt||||S)aFill the input Tensor with values drawn from the uniform distribution. :math:`\mathcal{U}(a, b)`. Args: tensor: an n-dimensional `torch.Tensor` a: the lower bound of the uniform distribution b: the upper bound of the uniform distribution generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.uniform_(w) r<)r: overrideshas_torch_function_variadichandle_torch_functionr r?r<r=r=r>r s  r cCs4tj|r&tjjt|f||||dSt||||S)aFill the input Tensor with values drawn from the normal distribution. :math:`\mathcal{N}(\text{mean}, \text{std}^2)`. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.normal_(w) rB)r:rgrhrir rCrBr=r=r>r s  r rFcCst||||||dS)aFill the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq \text{mean} \leq b`. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.trunc_normal_(w) r9)rW)r3r@rAr4r5r6r=r=r>r sr cCs,tj|r"tjjt|f||dSt||S)zFill the input Tensor with the value :math:`\text{val}`. Args: tensor: an n-dimensional `torch.Tensor` val: the value to fill the tensor with Examples: >>> w = torch.empty(3, 5) >>> nn.init.constant_(w, 0.3) rZ)r:rgrhrir r[rZr=r=r>r *s  r cCs t|dS)zFill the input Tensor with the scalar value `1`. Args: tensor: an n-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.ones_(w) rE)r[r]r=r=r>r<s rcCst|S)zFill the input Tensor with the scalar value `0`. Args: tensor: an n-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.zeros_(w) )r^r]r=r=r>rIs rcCsV|dkrtdt&tj|j||jdWdn1sH0Y|S)a=Fill the 2-dimensional input `Tensor` with the identity matrix. Preserves the identity of the inputs in `Linear` layers, where as many inputs are preserved as possible. Args: tensor: a 2-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.eye_(w) rK,Only tensors with 2 dimensions are supported)out requires_gradN) ndimensionrer:r;rshapermr]r=r=r>rVs  4rrN)r3groupsr7c Cs<|}|dvrtd|}|d|dkr8td|d|}t||d}t|t|D]}t|D]}|dkrd||||||ddf<qx|dkrd||||||dd|ddf<qxd||||||dd|dd|ddf<qxqlWd n1s.0Y|S) aFFill the {3, 4, 5}-dimensional input `Tensor` with the Dirac delta function. Preserves the identity of the inputs in `Convolutional` layers, where as many input channels are preserved as possible. In case of groups>1, each group of channels preserves identity Args: tensor: a {3, 4, 5}-dimensional `torch.Tensor` groups (int, optional): number of groups in the conv layer (default: 1) Examples: >>> w = torch.empty(3, 16, 5, 5) >>> nn.init.dirac_(w) >>> w = torch.empty(3, 24, 5, 5) >>> nn.init.dirac_(w, 3) )z5Only tensors with 3, 4, or 5 dimensions are supportedrz!dim 0 must be divisible by groupsrNrqrKrrN)rnresizerOr:r;r\range)r3rp dimensionssizesZout_chans_per_grpZmin_dimgdr=r=r>rksB    "        &rcCsp|}|dkrtd|d}|d}d}|dkrX|jddD] }||9}qJ||}||}||fS)NrKzNFan in and fan out can not be computed for tensor with fewer than 2 dimensionsrNr)Zdimrertro)r3rvZnum_input_fmapsZnum_output_fmapsZreceptive_field_sizesr1r2r=r=r>_calculate_fan_in_and_fan_outs    r{)r3gainr6r7cCsDt|\}}|tdt||}td|}t|| ||S)aFill the input `Tensor` with values using a Xavier uniform distribution. The method is described in `Understanding the difficulty of training deep feedforward neural networks` - Glorot, X. & Bengio, Y. (2010). The resulting tensor will have values sampled from :math:`\mathcal{U}(-a, a)` where .. math:: a = \text{gain} \times \sqrt{\frac{6}{\text{fan\_in} + \text{fan\_out}}} Also known as Glorot initialization. Args: tensor: an n-dimensional `torch.Tensor` gain: an optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain("relu")) Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.xavier_uniform_(w.T, ...)``. rF@)r{rGrIrQr?)r3r|r6r1r2rAr4r=r=r>rs" rcCs4t|\}}|tdt||}t|d||S)aFill the input `Tensor` with values using a Xavier normal distribution. The method is described in `Understanding the difficulty of training deep feedforward neural networks` - Glorot, X. & Bengio, Y. (2010). The resulting tensor will have values sampled from :math:`\mathcal{N}(0, \text{std}^2)` where .. math:: \text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan\_in} + \text{fan\_out}}} Also known as Glorot initialization. Args: tensor: an n-dimensional `torch.Tensor` gain: an optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.xavier_normal_(w) Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.xavier_normal_(w.T, ...)``. rFrf)r{rGrIrQrC)r3r|r6r1r2rAr=r=r>rs! r)r3moder7cCsH|}ddg}||vr,td|d|t|\}}|dkrD|S|S)Nr1r2zMode z" not supported, please use one of )lowerrer{)r3r~Z valid_modesr1r2r=r=r>_calculate_correct_fans  rr1r/)r3r4r~r_r6r7c Cstj|r(tjjt|f|||||dSd|jvr@td|St||}t ||}|t |}t d|}t "|j | ||dWdS1s0YdS)aFill the input `Tensor` with values using a Kaiming uniform distribution. The method is described in `Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification` - He, K. et al. (2015). The resulting tensor will have values sampled from :math:`\mathcal{U}(-\text{bound}, \text{bound})` where .. math:: \text{bound} = \text{gain} \times \sqrt{\frac{3}{\text{fan\_mode}}} Also known as He initialization. Args: tensor: an n-dimensional `torch.Tensor` a: the negative slope of the rectifier used after this layer (only used with ``'leaky_relu'``) mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'`` preserves the magnitude of the variance of the weights in the forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the backwards pass. nonlinearity: the non-linear function (`nn.functional` name), recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default). generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.kaiming_uniform_(w, mode="fan_in", nonlinearity="relu") Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.kaiming_uniform_(w.T, ...)``. )r3r4r~r_r6r,Initializing zero-element tensors is a no-opr}r9N)r:rgrhrirrorRrSrr rGrIr;r ) r3r4r~r_r6fanr|rAboundr=r=r>rs&+      rcCsvd|jvrtd|St||}t||}|t|}t |j d||dWdS1sh0YdS)aFill the input `Tensor` with values using a Kaiming normal distribution. The method is described in `Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification` - He, K. et al. (2015). The resulting tensor will have values sampled from :math:`\mathcal{N}(0, \text{std}^2)` where .. math:: \text{std} = \frac{\text{gain}}{\sqrt{\text{fan\_mode}}} Also known as He initialization. Args: tensor: an n-dimensional `torch.Tensor` a: the negative slope of the rectifier used after this layer (only used with ``'leaky_relu'``) mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'`` preserves the magnitude of the variance of the weights in the forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the backwards pass. nonlinearity: the non-linear function (`nn.functional` name), recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default). generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.kaiming_normal_(w, mode="fan_out", nonlinearity="relu") Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.kaiming_normal_(w.T, ...)``. rrr9N) rorRrSrr rGrIr:r;r )r3r4r~r_r6rr|rAr=r=r>rPs+     rc Cs|dkrtd|dkr$|S|d}||}|||fjdd|d}||krd|tj |\}}t |d}| } || 9}||kr|t *| ||||Wdn1s0Y|S)aFill the input `Tensor` with a (semi) orthogonal matrix. Described in `Exact solutions to the nonlinear dynamics of learning in deep linear neural networks` - Saxe, A. et al. (2013). The input tensor must have at least 2 dimensions, and for tensors with more than 2 dimensions the trailing dimensions are flattened. Args: tensor: an n-dimensional `torch.Tensor`, where :math:`n \geq 2` gain: optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_LAPACK) >>> w = torch.empty(3, 5) >>> nn.init.orthogonal_(w) rKz4Only tensors with 2 or more dimensions are supportedrrNr9N)rnreZnumelrtZ new_emptyr Zt_r:ZlinalgZqrZdiagsignr;Zview_asZcopy_rT) r3r|r6rowscolsZ flattenedqrryphr=r=r>rs&      (rra)r3sparsityrAr6r7c Cs|dkrtd|j\}}tt||}tP|jd||dt |D]&}t |}|d|} d|| |f<qRWdn1s0Y|S)aFill the 2D input `Tensor` as a sparse matrix. The non-zero elements will be drawn from the normal distribution :math:`\mathcal{N}(0, 0.01)`, as described in `Deep learning via Hessian-free optimization` - Martens, J. (2010). Args: tensor: an n-dimensional `torch.Tensor` sparsity: The fraction of elements in each column to be set to zero std: the standard deviation of the normal distribution used to generate the non-zero values generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.sparse_(w, sparsity=0.1) rKrkrr9N) rnrerordrGceilr:r;r ruZrandperm) r3rrAr6rrZ num_zerosZcol_idxZ row_indicesZ zero_indicesr=r=r>rs      ,r)methr7csTjddtjtjtdfdd }dddd|_|_|S) N)argskwargsr7cs,tjdddtdd|i|S)Nz `nn.init.z)` is now deprecated in favor of `nn.init.z`.rKrL)rRrS FutureWarning)rrrnew_nameZold_namer=r>deprecated_inits z(_make_deprecate..deprecated_initz z_(...) .. warning:: This method is now deprecated in favor of :func:`torch.nn.init.z"`. See :func:`~torch.nn.init.z` for details.)__name__r$rrr#__doc__)rrr=rr>_make_deprecates  r)N)N)N)N)rfrEN)rfrEN)rfrErjrFN)rN)rEN)rEN)rr1r/N)rr1r/N)rNN)raN)9rrGrRtypingrrrZ _OptionalrrZtyping_extensionsrr:r__all__r#r$Z_NonlinearityTypeZ_FanModerQ Generatorr?rCrWr[r^rdr r r r r rrrrtupler{rrrrrrrrrrrrrrrrr r!r"r=r=r=r>s2     , L     5 + '  C 7 6 '