JThBd PSSKJrJr SSKJr SSKrSSKJs Jr SSKJ r SSK J r SSK Jr /SQrSSjrS r"S S \5r"S S \ 5rS SS.S\ S\S\\S\S\ 4 Sjjjr"SS\ 5rS!S\ S\S\4Sjjr"SS\ 5rS"S\ S\S\S\S\\S\ 4 Sjjrg)#)autoEnum)OptionalN)Tensor)Module) parametrize) orthogonal spectral_norm weight_normc:URS5URS5p2[R"X0RURS9nSU-[R "UR5R -n[R"URU-XAS9$)Ndtypedeviceg$@)atol) sizetorcheyerrfinfoepsallclosemH)QrnkIds W/var/www/auris/envauris/lib/python3.13/site-packages/torch/nn/utils/parametrizations.py_is_orthogonalrsk 66":qvvbzq 1GGAHH 5B (U[[)-- -C >>!$$(B 11c[R"U5up[RRX5nX1R SSS9R 5R S5-nU$)z{Assume that A is a tall matrix. Compute the Q factor s.t. A = QR (A may be complex) and diag(R) is real and non-negative. r rdim1dim2)rgeqrflinalghouseholder_productdiagonalsgn unsqueeze)AXtaurs r_make_orthogonalr.sU [[^FA ((0A" % ) ) + 5 5b 99A Hr c>\rSrSr\"5r\"5r\"5rSrg) _OrthMaps#N) __name__ __module__ __qualname____firstlineno__r matrix_expcayley householder__static_attributes__r2r rr0r0#sJ VF&Kr r0c^\rSrSr%\\S'SS.S\SS4U4SjjjrS \RS\R4S jr \RR5S \RS\R4S j5r S r U=r$) _Orthogonal)baseTuse_trivializationorthogonal_mapreturnNc>[TU]5 UR5(aU[R:Xa [ S5eUR UlX lU(aURSS5 gg)NzAThe householder parametrization does not support complex tensors.r>) super__init__ is_complexr0r9 ValueErrorshaperAregister_buffer)selfweightrAr@ __class__s rrE_Orthogonal.__init__,si      >Y5J5J#JS \\ ,    . r r,c URS5URS5p2X#:nU(aURnX2p2UR[R:XdUR[R :XGa=UR 5nX#:waI[R"XRX"U- 5R"/URSSQSPSP76/SS9nXR- nUR[R:Xa[R"U5nOUR[R :Xao[R"X%RURS9n[R R#[R$"XuSS9[R$"XuSS95nX#:wa WSSU24nOtUR SS 9nS S XU-R'SS9-- n[R R)XX5nXaR+SSS 9R-5R/S5-n[1US 5(aUR2W-nU(a WRnW$)Nr rdimrg)alphag?.)r(g@g?r"r>)rmTrAr0r7r8trilrcat new_zerosexpandrHrrrrr&solveaddsumr'r(intr*hasattrr>) rJr,rr transposedr+rrr-s rforward_Orthogonal.forwardGsvvbz166":1U Aq   9#7#7 7""i&6&66AvII A1u-44KaggcrlKBKKLRTDDA""i&:&::$$Q'$$ (8(88YYqALL&&IIb40%))B2Mvc2A2gJ#A{{r{223C 008AJJBRJ0446@@DDA 4  A A Ar rc"URUR:wa&[SURSURS35eUnURS5URS5pCX4:nU(aURnXCpC[ US5(dUR [ R:XdUR [ R:Xa [S5e[R"U5upgURSSS9R5 URSSS9US :H==S-ss'U(a UR$U$X4:Xa-[U5(d [U5nOuUR!5nOd[R""UR5SSX3U- 4-UR$UR&S .6n[R("X/SS 9n[U5nXl[R,"U5n U RSSS9R/S 5 U $) Nz0Expected a matrix or batch of matrices of shape z. Got a tensor of shape .r rr>ztIt is not possible to assign to the matrix exponential or the Cayley parametrizations when use_trivialization=False.r"grrOg)rHrGrrRr[rAr0r8r7NotImplementedErrorrr%r(sign_rr.clonerandnrrrTr> zeros_likefill_) rJrQ_initrr transposer+r-Nneg_Ids r right_inverse_Orthogonal.right_inversevs 77djj B4::,O))* 4  vvbz166":1E AqtV$$##y'7'77&&)*>*>>)T[[^FA JJBRJ ( . . 0 JJBRJ ( 4 : 4$144 +! +v%a(((+A AKKffhsmqa%j0IIqf"-$Q'I %%f-F OO"O - 3 3D 9Mr )r>rArH)r3r4r5r6r__annotations__r0rErr]autogradno_gradrkr: __classcell__rLs@rr<r<)s LHL/&// //6--%,,-^ ^^Eu||E EEr r<Tr?modulenamerAr@rBc[XS5n[U[5(d[SUSUS35eURS:a[SURS35eUc=UR S5UR S 5:XdUR 5(aS OS n[[US5nUc[S U35e[XEUS 9n[R"XUSS9 U$)aApply an orthogonal or unitary parametrization to a matrix or a batch of matrices. Letting :math:`\mathbb{K}` be :math:`\mathbb{R}` or :math:`\mathbb{C}`, the parametrized matrix :math:`Q \in \mathbb{K}^{m \times n}` is **orthogonal** as .. math:: \begin{align*} Q^{\text{H}}Q &= \mathrm{I}_n \mathrlap{\qquad \text{if }m \geq n}\\ QQ^{\text{H}} &= \mathrm{I}_m \mathrlap{\qquad \text{if }m < n} \end{align*} where :math:`Q^{\text{H}}` is the conjugate transpose when :math:`Q` is complex and the transpose when :math:`Q` is real-valued, and :math:`\mathrm{I}_n` is the `n`-dimensional identity matrix. In plain words, :math:`Q` will have orthonormal columns whenever :math:`m \geq n` and orthonormal rows otherwise. If the tensor has more than two dimensions, we consider it as a batch of matrices of shape `(..., m, n)`. The matrix :math:`Q` may be parametrized via three different ``orthogonal_map`` in terms of the original tensor: - ``"matrix_exp"``/``"cayley"``: the :func:`~torch.matrix_exp` :math:`Q = \exp(A)` and the `Cayley map`_ :math:`Q = (\mathrm{I}_n + A/2)(\mathrm{I}_n - A/2)^{-1}` are applied to a skew-symmetric :math:`A` to give an orthogonal matrix. - ``"householder"``: computes a product of Householder reflectors (:func:`~torch.linalg.householder_product`). ``"matrix_exp"``/``"cayley"`` often make the parametrized weight converge faster than ``"householder"``, but they are slower to compute for very thin or very wide matrices. If ``use_trivialization=True`` (default), the parametrization implements the "Dynamic Trivialization Framework", where an extra matrix :math:`B \in \mathbb{K}^{n \times n}` is stored under ``module.parametrizations.weight[0].base``. This helps the convergence of the parametrized layer at the expense of some extra memory use. See `Trivializations for Gradient-Based Optimization on Manifolds`_ . Initial value of :math:`Q`: If the original tensor is not parametrized and ``use_trivialization=True`` (default), the initial value of :math:`Q` is that of the original tensor if it is orthogonal (or unitary in the complex case) and it is orthogonalized via the QR decomposition otherwise (see :func:`torch.linalg.qr`). Same happens when it is not parametrized and ``orthogonal_map="householder"`` even when ``use_trivialization=False``. Otherwise, the initial value is the result of the composition of all the registered parametrizations applied to the original tensor. .. note:: This function is implemented using the parametrization functionality in :func:`~torch.nn.utils.parametrize.register_parametrization`. .. _`Cayley map`: https://en.wikipedia.org/wiki/Cayley_transform#Matrix_map .. _`Trivializations for Gradient-Based Optimization on Manifolds`: https://arxiv.org/abs/1909.09501 Args: module (nn.Module): module on which to register the parametrization. name (str, optional): name of the tensor to make orthogonal. Default: ``"weight"``. orthogonal_map (str, optional): One of the following: ``"matrix_exp"``, ``"cayley"``, ``"householder"``. Default: ``"matrix_exp"`` if the matrix is square or complex, ``"householder"`` otherwise. use_trivialization (bool, optional): whether to use the dynamic trivialization framework. Default: ``True``. Returns: The original module with an orthogonal parametrization registered to the specified weight Example:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_LAPACK) >>> orth_linear = orthogonal(nn.Linear(20, 40)) >>> orth_linear ParametrizedLinear( in_features=20, out_features=40, bias=True (parametrizations): ModuleDict( (weight): ParametrizationList( (0): _Orthogonal() ) ) ) >>> # xdoctest: +IGNORE_WANT >>> Q = orth_linear.weight >>> torch.dist(Q.T @ Q, torch.eye(20)) tensor(4.9332e-07) NModule '(' has no parameter or buffer with name ''z8Expected a matrix or batch of matrices. Got a tensor of z dimensions.r rr7r9zLorthogonal_map has to be one of "matrix_exp", "cayley", "householder". Got: r?Tunsafe) getattr isinstancerrGndimrrFr0r<rregister_parametrization)rrrsrAr@rK orth_enumorths rr r svV4 (F ff % %vhFtfA N  {{Q %{{m< 9  {{2&++b/1V5F5F5H5H    >48I "# %   v=O PD((tDI Mr cL^\rSrSrS S\\SS4U4SjjjrSrSrSr U=r $) _WeightNormi:rPrBNc8>[TU]5 UcSnXlg)Nr)rDrErP)rJrPrLs rrE_WeightNorm.__init__;s  ;Cr cD[R"X!UR5$N)r _weight_normrP)rJweight_gweight_vs rr]_WeightNorm.forwardDs!!(dhh??r cP[R"USUR5nUnX#4$)Nrx)rnorm_except_dimrP)rJrKrrs rrk_WeightNorm.right_inverseGs(((DHH=!!r rO)r) r3r4r5r6rrZrEr]rkr:rprqs@rrr:s8 c] @""r rrPcz^[U5n[R"UTUSS9 U4SjnURU5 U$)aApply weight normalization to a parameter in the given module. .. math:: \mathbf{w} = g \dfrac{\mathbf{v}}{\|\mathbf{v}\|} Weight normalization is a reparameterization that decouples the magnitude of a weight tensor from its direction. This replaces the parameter specified by :attr:`name` with two parameters: one specifying the magnitude and one specifying the direction. By default, with ``dim=0``, the norm is computed independently per output channel/plane. To compute a norm over the entire weight tensor, use ``dim=None``. See https://arxiv.org/abs/1602.07868 Args: module (Module): containing module name (str, optional): name of weight parameter dim (int, optional): dimension over which to compute the norm Returns: The original module with the weight norm hook Example:: >>> m = weight_norm(nn.Linear(20, 40), name='weight') >>> m ParametrizedLinear( in_features=20, out_features=40, bias=True (parametrizations): ModuleDict( (weight): ParametrizationList( (0): _WeightNorm() ) ) ) >>> m.parametrizations.weight.original0.size() torch.Size([40, 1]) >>> m.parametrizations.weight.original1.size() torch.Size([40, 20]) Tryc>UT S3nUT S3nXp;a=X;a7URU5n URU5n XUST S3'XUST S3'ggg)N_g_vzparametrizations.z .original0z .original1)pop) state_dictprefixlocal_metadatastrict missing_keysunexpected_keys error_msgsg_keyv_key original0 original1rss r_weight_norm_compat_hook-weight_norm.._weight_norm_compat_hook|s(4&#(4&#  5#6"u-I"u-IGP&!24& C DGP&!24& C D $7 r )rrr~"_register_load_state_dict_pre_hook)rrrsrPrrs ` rr r Ns@Vs#L((|DQQ" --.FG Mr c p^\rSrSrSS\R S\S\S\SS4 U4SjjjrS\R S\R 4S jr \RR5S \R S\SS4S j5r S\R S\R 4S jr S \R S\R 4SjrSrU=r$) _SpectralNormirKn_power_iterationsrPrrBNc >[T U]5 URnX5:dX5*:a[SUSUS- SUS35eUS::a[ SU35eUS:aUOX5-UlX@lUS:aX lURU5nUR5upxURU5RSS5n URU5RSS5n URS[R"U SUR S 95 URS [R"U SUR S 95 URUS 5 gg) Nz5Dimension out of range (expected to be in range of [-z, z ] but got )rzGExpected n_power_iterations to be positive, but got n_power_iterations=_urPrr)rDrEr} IndexErrorrGrPrr_reshape_weight_to_matrixr new_emptynormal_rIF normalize _power_method) rJrKrrPrr} weight_mathwuvrLs rrE_SpectralNorm.__init__sO {{ ;#+F"TAXJjQ8   "**<)=? (3  !8&8 #77?J??$DA$$Q'//15A$$Q'//15A  q{{1!'J K  q{{1!'J K   z2 . r c^URS:deTRS:wa>UR"TR/U4Sj[UR555Q76nUR S5$)Nrrc3J># UHoTR:wdMUv M g7frrO).0drJs r :_SpectralNorm._reshape_weight_to_matrix..sL':!488mAA':s# #)r}rPpermuterangeflatten)rJrKs` rr'_SpectralNorm._reshape_weight_to_matrixs\{{Q 88q=^^LuVZZ\':LF~~a  r rcURS:de[U5Hn[R"[R "XR 5SURURS9Ul[R"[R "URUR5SURUR S9UlM g)Nrr)rProut) r}rrrrmvrrrH)rJrr_s rr_SpectralNorm._power_methodsD""")*AkkWW-HHGG DG kktww/HHGG DG+r cURS:Xa [R"USURS9$UR U5nUR (aUR X R5 URR[RS9nURR[RS9n[R"U[R"X$55nX- $)Nrrr) memory_format)r}rrrrtrainingrrrrcrcontiguous_formatrvdotr)rJrKrrrsigmas rr]_SpectralNorm.forwards ;;! ;;v1$((; ;77?J}}"":/F/FG E,C,C DA E,C,C DAJJq%((:"9:E> !r valuecU$rr2)rJrs rrk_SpectralNorm.right_inverse s  r )rrrPrr)rr-q=)r3r4r5r6rrrZfloatrErrnrorr]rkr:rprqs@rrrs#$ #/ #/ #/ #/  #/  #/#/J ! ! ! ^^2 2#2RV22h"ell"u||""5<<ELLr rrrc v[XS5n[U[5(d[SUSUS35eUca[U[R R [R R[R R45(aSnOSn[R"X[XRXC55 U$)as Apply spectral normalization to a parameter in the given module. .. math:: \mathbf{W}_{SN} = \dfrac{\mathbf{W}}{\sigma(\mathbf{W})}, \sigma(\mathbf{W}) = \max_{\mathbf{h}: \mathbf{h} \ne 0} \dfrac{\|\mathbf{W} \mathbf{h}\|_2}{\|\mathbf{h}\|_2} When applied on a vector, it simplifies to .. math:: \mathbf{x}_{SN} = \dfrac{\mathbf{x}}{\|\mathbf{x}\|_2} Spectral normalization stabilizes the training of discriminators (critics) in Generative Adversarial Networks (GANs) by reducing the Lipschitz constant of the model. :math:`\sigma` is approximated performing one iteration of the `power method`_ every time the weight is accessed. If the dimension of the weight tensor is greater than 2, it is reshaped to 2D in power iteration method to get spectral norm. See `Spectral Normalization for Generative Adversarial Networks`_ . .. _`power method`: https://en.wikipedia.org/wiki/Power_iteration .. _`Spectral Normalization for Generative Adversarial Networks`: https://arxiv.org/abs/1802.05957 .. note:: This function is implemented using the parametrization functionality in :func:`~torch.nn.utils.parametrize.register_parametrization`. It is a reimplementation of :func:`torch.nn.utils.spectral_norm`. .. note:: When this constraint is registered, the singular vectors associated to the largest singular value are estimated rather than sampled at random. These are then updated performing :attr:`n_power_iterations` of the `power method`_ whenever the tensor is accessed with the module on `training` mode. .. note:: If the `_SpectralNorm` module, i.e., `module.parametrization.weight[idx]`, is in training mode on removal, it will perform another power iteration. If you'd like to avoid this iteration, set the module to eval mode before its removal. Args: module (nn.Module): containing module name (str, optional): name of weight parameter. Default: ``"weight"``. n_power_iterations (int, optional): number of power iterations to calculate spectral norm. Default: ``1``. eps (float, optional): epsilon for numerical stability in calculating norms. Default: ``1e-12``. dim (int, optional): dimension corresponding to number of outputs. Default: ``0``, except for modules that are instances of ConvTranspose{1,2,3}d, when it is ``1`` Returns: The original module with a new parametrization registered to the specified weight Example:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_LAPACK) >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> snm = spectral_norm(nn.Linear(20, 40)) >>> snm ParametrizedLinear( in_features=20, out_features=40, bias=True (parametrizations): ModuleDict( (weight): ParametrizationList( (0): _SpectralNorm() ) ) ) >>> torch.linalg.matrix_norm(snm.weight, 2) tensor(1.0081, grad_fn=) Nrurvrwrr) r{r|rrGrnnConvTranspose1dConvTranspose2dConvTranspose3drr~r)rrrsrrrPrKs rr r s`V4 (F ff % %vhFtfA N   {  ((((((   CC((mFI Mr r)rKN)rKr)rKrrN)enumrrtypingrrtorch.nn.functionalr functionalrrtorch.nn.modulesrtorch.nn.utilsr__all__rr.r0r<strboolr rrZr rrr r2r rrs* #& 92   S&Sp$(x $ x x xSMx  x  xv"&"(@@c@3@F{F{@ e e ee  e # e  er