o ZhG-@sddlmZddlZgdZGdddejjZGdddejjZGdd d ejjZGd d d ejj Z Gd d d ejj Z Gdddejj Z Gdddej jj jZGdddejjZdS))warnN)ReLU6 HardswishELU LeakyReLUSigmoidSoftmaxMultiheadAttentionPReLUcs@eZdZdZd fdd ZddZddZed d d ZZ S) raApplies the element-wise function: :math:`\text{ReLU6}(x) = \min(\max(x_0, x), q(6))`, where :math:`x_0` is the zero_point, and :math:`q(6)` is the quantized representation of number 6. Args: inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(N, *)` where `*` means, any number of additional dimensions - Output: :math:`(N, *)`, same shape as the input .. image:: ../scripts/activation_images/ReLU6.png Examples:: >>> m = nn.quantized.ReLU6() >>> input = torch.randn(2) >>> # xdoctest: +SKIP >>> input = torch.quantize_per_tensor(input, 1.0, 0, dtype=torch.qint32) >>> output = m(input) Fcst|||_dSN)super__init__inplace)selfr __class__W/var/www/auris/lib/python3.10/site-packages/torch/ao/nn/quantized/modules/activation.pyr ,s  zReLU6.__init__cCstjj||jSr )torchops quantizedZrelu6rrinputrrrforward0sz ReLU6.forwardcCdS)NZQuantizedReLU6rrrrr _get_name3zReLU6._get_namecCs t|jSr )rr)moduse_precomputed_fake_quantrrr from_float6s zReLU6.from_floatF) __name__ __module__ __qualname____doc__r rr staticmethodr __classcell__rrrrrsrcLeZdZdZdfdd ZddZddZedd d Ze d d Z Z S)rzThis is the quantized version of :class:`~torch.nn.Hardswish`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor NcsL||d}t|dtj|fi||dtj|fi|dSN)devicedtypescale zero_pointr r Zregister_bufferrZtensor)rr,r-r*r+factory_kwargsrrrr Cs  zHardswish.__init__cCtjj||j|jSr )rrrZ hardswishr,r-rrrrrIszHardswish.forwardcCr)NZQuantizedHardswishrrrrrrLrzHardswish._get_nameFcCs |j\}}tt|t|Sr )activation_post_processcalculate_qparamsrfloatintrrr,r-rrrr OszHardswish.from_floatcCs|t|t|Sr )r3r4clsrr,r-rrrfrom_referenceTszHardswish.from_reference)NNr! r"r#r$r%r rrr&r classmethodr8r'rrrrr;s rcr()rzThis is the quantized equivalent of :class:`~torch.nn.ELU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor alpha: the alpha constant ?cst|||_||_dSr )r r r,r-)rr,r-alpharrrr bs  z ELU.__init__cCstjjjj||j|j|jSr ) raonnr functionalZelur,r-r<rrrrrgs z ELU.forwardcCr)NZ QuantizedELUrrrrrrlrz ELU._get_nameFcCs$|j\}}tt|t||jSr )r1r2rr3r4r<r5rrrr ozELU.from_floatcCs|t|t||jSr )r3r4r<r6rrrr8tzELU.from_reference)r;r!r9rrrrrYs rc sjeZdZdZ    ddedededed df fd d Zd d ZddZ e dddZ e ddZ Z S)raThis is the quantized equivalent of :class:`~torch.nn.LeakyReLU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor negative_slope: Controls the angle of the negative slope. Default: 1e-2 {Gz?FNr,r-negative_sloperreturncsP||d}t|||dtj|fi||dtj|fi|dSr)r.)rr,r-rCrr*r+r/rrrr s zLeakyReLU.__init__cCstjj||j|j|j|jSr )rrrZ leaky_relurCrr,r-rrrrrszLeakyReLU.forwardcCr)NZQuantizedLeakyReLUrrrrrrrzLeakyReLU._get_namecCs(|j\}}|t|t||j|jSr )r1r2r3r4rCr)r7rrr,r-rrrr szLeakyReLU.from_floatcCs|t|t||j|jSr )r3r4rCrr6rrrr8szLeakyReLU.from_reference)rBFNNr!)r"r#r$r%r3r4boolr rrr:r r8r'rrrrrys.  rcs@eZdZdZdedeffdd ZddZed d d Z Z S) rzThis is the quantized equivalent of :class:`~torch.nn.Sigmoid`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor output_scaleoutput_zero_pointcst||_||_dSr )r r rFrG)rrFrGrrrr s  zSigmoid.__init__cCr0r )rrrZsigmoidrFrGrrrrrs zSigmoid.forwardFcCs |j\}}|t|t|Sr )r1r2r3r4)r7rrrFrGrrrr s zSigmoid.from_floatr!) r"r#r$r%r3r4r rr:r r'rrrrrs rcsLeZdZdZdfdd ZddZd d Zedd d Ze ddZ Z S)ra,This is the quantized version of :class:`~torch.nn.Softmax`. Args: dim: A dimension along which Softmax will be computed (so every slice along dim will sum to 1). scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor Nr;rcs t||_||_||_dSr )r r dimr,r-)rrHr,r-rrrr s  zSoftmax.__init__cCs@|j}|durd}tjjd||}tjj|||j|j S)Nsoftmax) rHrr>r?Z_get_softmax_dimrrrJr,r-)rrrH stacklevelrrrrs zSoftmax.forwardcCr)NZQuantizedSoftmaxrrrrrrrzSoftmax._get_nameFcCs$|j\}}t|jt|t|Sr )r1r2rrHr3r4r5rrrr r@zSoftmax.from_floatcCs||jt|t|Sr )rHr3r4r6rrrr8rAzSoftmax.from_reference)Nr;rr!r9rrrrrs  rc@s8eZdZejjjjZddZ e ddZ e ddZ dS)r cCr)NZQuantizedMultiheadAttentionrrrrrrrzMultiheadAttention._get_namecCstd)NzpIt looks like you are trying to convert a non-observed MHA module. Please, see the examples on quantizable MHAs.)NotImplementedError)r7otherrrrr szMultiheadAttention.from_floatcCstjjj|ddddd}||_|jdur2|jd}tj|dd\}}t |||tj }t |d||j durU|jd}tj|dd\}}t |||tj }t |d||` |`|S)NFT)mappingrZremove_qconfigZconvert_custom_config_dictbias_k)Z reduce_rangebias_v)rr=Z quantizationconvertrrO _parameterspopZ_choose_qparams_per_tensorquantize_per_tensorquint8setattrrPZin_proj_weightZ in_proj_bias)r7rMZ convertedrOscZzprPrrr from_observeds.       z MultiheadAttention.from_observedN) r"r#r$rr=r> quantizabler Z _FLOAT_MODULErr:r rXrrrrr s   r c seZdZdZ ddedededdffdd Zd ejddfd d Z d ejdejfddZ ddZ e dddZ e ddZZS)r a%This is the quantized equivalent of :class:`~torch.nn.PReLU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor num_parameters: number of parameters: 1, or the number of channels at input. Default: 1 rFrGnum_parametersrDNcsNt||_||_||_tj|tjd}tj|ddtj d}| |dS)N)r+r;r)r,r-r+) r r r[r,r-rZrandnr3rTrU set_weight)rrFrGr[wZqwrrrr s zPReLU.__init__r]cCs ||_dSr )weight)rr]rrrr\+s zPReLU.set_weightrcCstjj||j|j|jSr )rrrZprelur^r,r-rrrrr.sz PReLU.forwardcCr)NZQuantizedPReLUrrrrrr3rzPReLU._get_nameFc Cs|j\}}|t|t||j}|j}|j}|||jtj kr.t d|j|\}} t |t|t| tj } | | |SNz9PReLU's weight observer should have dtype quint8 but got ) r1r2r3r4r[r^qconfigr+rrUrrTr\) r7rrr,r-qprelufloat_wtobserverwt_scalewt_zpqweightrrrr 6s      zPReLU.from_floatc Cs|t|t||j}|j}|j}|||jtjkr'td|j| \}}t |t|t|tj} | | |Sr_) r3r4r[r^r`r+rrUrr2rTr\) r7rr,r-rarbrcrdrerfrrrr8Hs      zPReLU.from_reference)rZr!)r"r#r$r%r3r4r rZTensorr\rrr:r r8r'rrrrr s&   r )warningsrr__all__r>ZReLUrrrrrrr=rYr Moduler rrrrs  ( )'3