o wZŽhW5ã@shddlmZddlmZddlZddlmmZddlmZGdd„deƒZ Gdd„dej ƒZ d ej d ej d e fd d „ZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGd d!„d!ejƒZGd"d#„d#ejƒZGd$d%„d%ejƒZGd&d'„d'ejƒZdS)(é)ÚAny)ÚBaseSparsifierN)ÚnncsReZdZdeeefddf‡fdd„ Zdejdedeeefddfdd „Z ‡Z S) ÚImplementedSparsifierÚkwargsÚreturnNc stƒj|ddS)N)Údefaults)ÚsuperÚ__init__)Úselfr©Ú __class__©úU/var/www/auris/lib/python3.10/site-packages/torch/testing/_internal/common_pruning.pyr szImplementedSparsifier.__init__ÚmoduleÚ tensor_namecKs4d|jjdjd<|jd}| dd¡d|d<dS)Nrzlinear1.weightZ step_counté)ZparametrizationsÚweightÚmaskÚstateÚget)r rrrZ linear_staterrrÚ update_mask s z!ImplementedSparsifier.update_mask) Ú__name__Ú __module__Ú __qualname__ÚdictÚstrrr rÚModulerÚ __classcell__rrr rr s,rc@s(eZdZdZedejddfdd„ƒZdS)ÚMockSparseLinearzË This class is a MockSparseLinear class to check convert functionality. It is the same as a normal Linear layer, except with a different type, as well as an additional from_dense method. ÚmodrcCs||j|jƒ}|S)z )Z in_featuresZ out_features)Úclsr ÚlinearrrrÚ from_densesÿzMockSparseLinear.from_denseN)rrrÚ__doc__Ú classmethodrÚLinearr#rrrrrsrÚ subset_tensorÚsuperset_tensorrcCsLd}|D]}|t|ƒkr t |||¡s|d7}nn |t|ƒks dSqdS)zW Checks to see if all rows in subset tensor are present in the superset tensor rrFT)ÚlenÚtorchÚequal)r'r(ÚiÚrowrrrÚrows_are_subset"s   üþr.có8eZdZdZd ‡fdd„ Zdejdejfdd„Z‡ZS) Ú SimpleLinearzžModel with only Linear layers without biases, some wrapped in a Sequential, some following the Sequential. Used to test basic pruned Linear-Linear fusion.rNc sftƒ ¡t tjddddtjddddtjdddd¡|_tjdddd|_tjdddd|_dS)NééF©Úbiasééé )r r rÚ Sequentialr&ÚseqÚlinear1Úlinear2©r r rrr 6s ýzSimpleLinear.__init__ÚxcCó"| |¡}| |¡}| |¡}|S©N)r9r:r;©r r=rrrÚforward@ó   zSimpleLinear.forward©rN© rrrr$r r*ÚTensorrArrrr rr02ó r0cr/) Ú LinearBiaszŠModel with only Linear layers, alternating layers with biases, wrapped in a Sequential. Used to test pruned Linear-Bias-Linear fusion.rNc s^tƒ ¡t tjddddtjddddtjddddtjddddtjdddd¡|_dS) Nr1r2Tr3r5Fér7)r r rr8r&r9r<r rrr Ks  ûzLinearBias.__init__r=cCs| |¡}|Sr?)r9r@rrrrAUs zLinearBias.forwardrCrDrrr rrGGrFrGcr/) ÚLinearActivationzùModel with only Linear layers, some with bias, some in a Sequential and some following. Activation functions modules in between each Linear in the Sequential, and each outside layer. Used to test pruned Linear(Bias)-Activation-Linear fusion.rNc s†tƒ ¡t tjddddt ¡tjddddt ¡tjdddd¡|_tjdddd|_t ¡|_ tjdd dd|_ t ¡|_ dS) Nr1r2Tr3r5Fr6rHr7) r r rr8r&ÚReLUÚTanhr9r:Úact1r;Úact2r<r rrr _s û zLinearActivation.__init__r=cCó6| |¡}| |¡}| |¡}| |¡}| |¡}|Sr?)r9r:rLr;rMr@rrrrAmó     zLinearActivation.forwardrCrDrrr rrIZsrIcr/) ÚLinearActivationFunctionala,Model with only Linear layers, some with bias, some in a Sequential and some following. Activation functions modules in between each Linear in the Sequential, and functional activationals are called in between each outside layer. Used to test pruned Linear(Bias)-Activation-Linear fusion.rNc sŽtƒ ¡t tjddddt ¡tjddddt ¡tjdddd¡|_tjdddd|_tjdd dd|_tjd d dd|_ t ¡|_ dS) Nr1r2Tr3r5Fr6rHér7) r r rr8r&rJr9r:r;Úlinear3rLr<r rrr |s ûz#LinearActivationFunctional.__init__r=cCsJ| |¡}| |¡}t |¡}| |¡}t |¡}| |¡}t |¡}|Sr?)r9r:ÚFÚrelur;rRr@rrrrAŠs       z"LinearActivationFunctional.forwardrCrDrrr rrPvsrPcr/) Ú SimpleConv2dz‡Model with only Conv2d layers, all without bias, some in a Sequential and some following. Used to test pruned Conv2d-Conv2d fusion.rNc shtƒ ¡t tjddddddtjdddddd¡|_tjdddddd|_tjdddddd|_dS) Nré rHFr3é@é0é4©r r rr8ÚConv2dr9Úconv2d1Úconv2d2r<r rrr ™s þzSimpleConv2d.__init__r=cCr>r?©r9r\r]r@rrrrA¢rBzSimpleConv2d.forwardrCrDrrr rrU•s rUcr/) Ú Conv2dBiaszˆModel with only Conv2d layers, some with bias, some in a Sequential and some outside. Used to test pruned Conv2d-Bias-Conv2d fusion.rNc sztƒ ¡t tjddddddtjddddddtjdddddd¡|_tjdddddd|_tjdd dddd|_dS© NrrVrHTr3rWFrXrYrZr<r rrr ­s ýzConv2dBias.__init__r=cCr>r?r^r@rrrrA·rBzConv2dBias.forwardrCrDrrr rr_©rFr_cr/) ÚConv2dActivationaModel with only Conv2d layers, some with bias, some in a Sequential and some following. Activation function modules in between each Sequential layer, functional activations called in-between each outside layer. Used to test pruned Conv2d-Bias-Activation-Conv2d fusion.rNc sŒtƒ ¡t tjddddddt ¡tjddddddt ¡tjddddddt ¡¡|_tjdddddd|_tjdd dddd|_ dSr`) r r rr8r[rJrKr9r\r]r<r rrr Äs úzConv2dActivation.__init__r=cCs6| |¡}| |¡}t |¡}| |¡}t |¡}|Sr?)r9r\rSrTr]Zhardtanhr@rrrrAÑrOzConv2dActivation.forwardrCrDrrr rra¾s racr/) Ú Conv2dPadBiasaQModel with only Conv2d layers, all with bias and some with padding > 0, some in a Sequential and some following. Activation function modules in between each layer. Used to test that bias is propagated correctly in the special case of pruned Conv2d-Bias-(Activation)Conv2d fusion, when the second Conv2d layer has padding > 0.rNcsÚtƒ ¡t tjdddddddt ¡tjddddddt ¡tjdddddddt ¡tjdddddddt ¡tjddddddt ¡¡ |_tjdd ddddd|_t ¡|_ tjd d ddddd|_ t ¡|_ dS) NrrVrHT)Úpaddingr4Fr3rWrXrY) r r rr8r[rJrKr9r\rLr]rMr<r rrr às" ö zConv2dPadBias.__init__r=cCrNr?)r9r\rLr]rMr@rrrrAórOzConv2dPadBias.forwardrCrDrrr rrbÚsrbcr/) Ú Conv2dPoolzèModel with only Conv2d layers, all with bias, some in a Sequential and some following. Activation function modules in between each layer, Pool2d modules in between each layer. Used to test pruned Conv2d-Pool2d-Conv2d fusion.rNc sÂtƒ ¡t tjddddddtjddddt ¡tjddddddt ¡tjdddd¡|_ tjdd dddd|_ tjdddd|_ t ¡|_ tjd d dddd|_ tjd d dddd|_dS) NrrVrHT©Ú kernel_sizercr4é©rfZstridercrWrXrY)r r rr8r[Ú MaxPool2drJrKÚ AvgPool2dr9r\ÚmaxpoolÚaf1r]Úconv2d3r<r rrr s ú zConv2dPool.__init__r=cCs\| |¡}| |¡}| |¡}| |¡}| |¡}tj|dddd}t |¡}| |¡}|S©Nrgrrh) r9r\rkrlr]rSZ avg_pool2drTrmr@rrrrAs       zConv2dPool.forwardrCrDrrr rrdüsrdcr/) ÚConv2dPoolFlattenFunctionalaModel with Conv2d layers, all with bias, some in a Sequential and some following, and then a Pool2d and a functional Flatten followed by a Linear layer. Activation functions and Pool2ds in between each layer also. Used to test pruned Conv2d-Pool2d-Flatten-Linear fusion.rNc s¸tƒ ¡t tjddddddtjddddt ¡tjddddddt ¡tjdddd¡|_ tjdddddd|_ t ¡|_ tjdd dddd|_ t  d ¡|_tjd d dd |_dS) NrrHTrergrhr2r1é )rré r3)r r rr8r[rirJrKrjr9r\rlr]ÚAdaptiveAvgPool2dÚavg_poolr&Úfcr<r rrr #s ú  z$Conv2dPoolFlattenFunctional.__init__r=cCs^| |¡}| |¡}tj|dddd}| |¡}| |¡}| |¡}t |d¡}|  |¡}|Srn) r9r\rSÚ max_pool2drlr]rsr*Úflattenrtr@rrrrA3s       z#Conv2dPoolFlattenFunctional.forwardrCrDrrr rrosrocr/) ÚConv2dPoolFlattenaModel with Conv2d layers, all with bias, some in a Sequential and some following, and then a Pool2d and a Flatten module followed by a Linear layer. Activation functions and Pool2ds in between each layer also. Used to test pruned Conv2d-Pool2d-Flatten-Linear fusion.rNc sÂtƒ ¡t tjddddddtjddddt ¡tjddddddt ¡tjdddd¡|_ tjdddddd|_ t ¡|_ tjdd dddd|_ t  d ¡|_t ¡|_tjd d dd |_dS)NrrHTrergrhr2r1rp)rgrgé,rqr3)r r rr8r[rirJrKrjr9r\rlr]rrrsZFlattenrvr&rtr<r rrr Es ú   zConv2dPoolFlatten.__init__r=cCs\| |¡}| |¡}tj|dddd}| |¡}| |¡}| |¡}| |¡}| |¡}|Srn) r9r\rSrurlr]rsrvrtr@rrrrAVs       zConv2dPoolFlatten.forwardrCrDrrr rrw?srwc óVeZdZdZdededededdf ‡fdd „ Zd ejdeejejffd d „Z ‡Z S) ÚLSTMLinearModelzCContainer module with an encoder, a recurrent module, and a linear.Ú input_dimÚ hidden_dimÚ output_dimÚ num_layersrNcs,tƒ ¡t |||¡|_t ||¡|_dSr?)r r rÚLSTMÚlstmr&r"©r r{r|r}r~r rrr es zLSTMLinearModel.__init__ÚinputcCs | |¡\}}| |¡}||fSr?)r€r")r r‚ÚoutputZ_hiddenÚdecodedrrrrAls zLSTMLinearModel.forward© rrrr$Úintr r*rEÚtuplerArrrr rrzbsÿÿÿÿþ(rzc ry) ÚLSTMLayerNormLinearModelz9Container module with an LSTM, a LayerNorm, and a linear.r{r|r}r~rNcs8tƒ ¡t |||¡|_t |¡|_t ||¡|_dSr?) r r rrr€Z LayerNormÚnormr&r"rr rrr us  z!LSTMLayerNormLinearModel.__init__r=cCs*| |¡\}}| |¡}| |¡}||fSr?)r€r‰r")r r=rrrrrA}s  z LSTMLayerNormLinearModel.forwardr…rrr rrˆrsÿÿÿÿþ(rˆ)ÚtypingrZtorch.ao.pruningrr*Ztorch.nn.functionalrZ functionalrSrr&rrEÚboolr.rr0rGrIrPrUr_rarbrdrorwrzrˆrrrrÚs*    "!"#