a h+ @s`UddlZddlZddlZddlZddlmZmZmZddlmZddl Z ddl m Z ddl m Z m Z mZgZeeed<ee j eje j fZddZedeeeeee j d d d Zedeee j eeddddZedeeeeeee j dddZededdeeeeeee j dddZed eeeeddddZde_de_de_dS)!N)castOptionalUnion) deprecated)Tensor)_device_has_foreach_support"_group_tensors_by_device_and_dtype_has_foreach_support__all__csfdd}t||S)z This wrapper is needed to avoid a circular import when using @torch.no_grad on the exposed functions clip_grad_norm_ and clip_grad_value_ themselves. cs:t|i|WdS1s,0YdSN)torchZno_grad)argskwargsfuncF/var/www/auris/lib/python3.9/site-packages/torch/nn/utils/clip_grad.py_no_grad_wrapper"s z"_no_grad.._no_grad_wrapper) functoolsupdate_wrapper)rrrrr_no_grads  r@F)tensors norm_typeerror_if_nonfiniteforeachreturnc s"t|tjr|g}nt|}tt|dkr:tdS|djt|g}g}| D]t\\}}\\}}|dur~t ||s|rt |r| t |qZ|rtd|jdqZ| fdd|DqZtjtfdd|D} |rt| | rtd d | S) aCompute the norm of an iterable of tensors. The norm is computed over the norms of the individual tensors, as if the norms of the individual tensors were concatenated into a single vector. Args: tensors (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will be normalized norm_type (float): type of the used p-norm. Can be ``'inf'`` for infinity norm. error_if_nonfinite (bool): if True, an error is thrown if the total norm of :attr:`tensors` is ``nan``, ``inf``, or ``-inf``. Default: ``False`` foreach (bool): use the faster foreach-based implementation. If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` Returns: Total norm of the tensors (viewed as a single vector). rgN:foreach=True was passed, but can't use the foreach API on  tensorscsg|]}tj|qSr)r linalg vector_norm).0g)rrr `z#_get_total_norm..csg|]}|qSr)to)r!Znorm) first_devicerrr#dr$zThe total norm of order z for gradients from `parameters` is non-finite, so it cannot be clipped. To disable this error and scale the gradients by the non-finite norm anyway, set `error_if_nonfinite=False`) isinstancer rlistfloatlenZtensordeviceritemsr rextendZ _foreach_norm RuntimeErrortyperr stack logical_orisnanisinf) rrrrZgrouped_tensorsZnormsr+_Zdevice_tensors total_normr)r&rr_get_total_norm*sD      r6) parametersmax_normr5rrc Cst|tjr|g}dd|D}t|}t|dkr8dSt|g}||d}tj|dd}|D]z\\}} \\} } |durt| |s|rt |rt | | |qd|rt d|j d qd| |} | D]} | | qqddS) aScale the gradients of an iterable of parameters given a pre-calculated total norm and desired max norm. The gradients will be scaled by the following calculation .. math:: grad = grad * \frac{max\_norm}{total\_norm + 1e-6} Gradients are modified in-place. This function is equivalent to :func:`torch.nn.utils.clip_grad_norm_` with a pre-calculated total norm. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized max_norm (float): max norm of the gradients total_norm (Tensor): total norm of the gradients to use for clipping foreach (bool): use the faster foreach-based implementation. If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` Returns: None cSsg|]}|jdur|jqSr gradr!prrrr#r$z*_clip_grads_with_norm_..rNgư>g?)maxrr)r'r rr)r*rclampr,r rZ _foreach_mul_r%r.r/Zmul_) r7r8r5rgrads grouped_gradsZ clip_coefZclip_coef_clampedr+r4Z device_gradsZclip_coef_clamped_devicer"rrr_clip_grads_with_norm_qs0     rA)r7r8rrrrcCstt|tjr|g}n2t|tj}t|}|rFt|dkrFtjddddd|D}t ||||}t |||||S)aSClip the gradient norm of an iterable of parameters. The norm is computed over the norms of the individual gradients of all parameters, as if the norms of the individual gradients were concatenated into a single vector. Gradients are modified in-place. This function is equivalent to :func:`torch.nn.utils.get_total_norm` followed by :func:`torch.nn.utils.clip_grads_with_norm_` with the ``total_norm`` returned by ``get_total_norm``. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized max_norm (float): max norm of the gradients norm_type (float, optional): type of the used p-norm. Can be ``'inf'`` for infinity norm. Default: 2.0 error_if_nonfinite (bool, optional): if True, an error is thrown if the total norm of the gradients from :attr:`parameters` is ``nan``, ``inf``, or ``-inf``. Default: False foreach (bool, optional): use the faster foreach-based implementation. If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` Returns: Total norm of the parameter gradients (viewed as a single vector). rzD`parameters` is an empty generator, no gradient clipping will occur.) stacklevelcSsg|]}|jdur|jqSr r9r;rrrr#r$z#clip_grad_norm_..) r'r rtypes GeneratorTyper(r*warningswarnr6rA)r7r8rrrZ is_generatorr?r5rrrclip_grad_norm_s"  rHz_`torch.nn.utils.clip_grad_norm` is now deprecated in favor of `torch.nn.utils.clip_grad_norm_`.)categorycCst|||||S)zClip the gradient norm of an iterable of parameters. .. warning:: This method is now deprecated in favor of :func:`torch.nn.utils.clip_grad_norm_`. )rH)r7r8rrrrrrclip_grad_normsrJ)r7 clip_valuerrcCst|tjr|g}t|}dd|D}t|g}|D]\\}}\\}}|durjtttt||dsv|rt |rt ttt|| t ttt||q:|rt d|j dq:|D]}tt|j| |dqq:dS)aClip the gradients of an iterable of parameters at specified value. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized clip_value (float): maximum allowed value of the gradients. The gradients are clipped in the range :math:`\left[\text{-clip\_value}, \text{clip\_value}\right]` foreach (bool, optional): use the faster foreach-based implementation If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` cSsg|]}|jdur|jqSr r9r;rrrr#r$z$clip_grad_value_..N)r+rr)minr=)r'r rr)rr,r rr(rZ_foreach_clamp_min_Z_foreach_clamp_max_r.r/Zclamp_)r7rKrr?r@r+r4r:rrrclip_grad_value_s,   rMztorch.nn.utils)rFN)N)rFN)rFN)N)rrDtypingrFrrrZtyping_extensionsrr rZtorch.utils._foreach_utilsrrr r r(str__annotations__IterableZ_tensor_or_tensorsrr)boolr6rArH FutureWarningrJrM __module__rrrrs    F=2+