o Zh@s$ ddlZddlmZddlZddlmZddlmZmZddl m Z m Z e ddZ ej ejejejejgZejejgZdd eDZed d eDd d Ze d e e dddejdededededejdejfddZe e dddejdededededejdejfddZe de e dddejdejdejdededejdejfddZ e e dddejdejdejdededejdejfdd Z!e d!e e d"ddejdejdejdejdejdejdejfd#d$Z"e e d"ddejdejdejdejdejdejdejfd%d&Z#e d'e e d(ddd)dejdededededejd*eejdejfd+d(Z$e e d(ddd)dejdejdejdededejd*eejdejfd,d-Z%e d.e e d/ddd)dejdejdejdededejd*eejdejfd0d1Z&e e d/ddd)dejdejdejdededejd*eejdejfd2d3Z'e d4e e d5ddd)dejdejdejdejdejdejd*eejdejfd6d7Z(e e d5ddd)d*eejdejfd8d9Z)e d:e e d;ddejdedejde*ejejff d?d@Z+e dAe e dBddejdedejde*ejejff dCdDZ,e e d;ddejdeded>edejde*ejejff dEdFZ-e e dBddejdeded>edejde*ejejff dGdHZ.dIdJZ/e dKe e dLddejdMejdNejdOedededejdejfdPdLZ0e e dLddejdMejdNejdOedededejdejfdQdRZ1e dSe e dTddd)dejdMejdNeejdOedededejd*eejdejfdUdTZ2e e dTddd)dejdMejdNeejdOedededejd*eejdejfdVdWZ3e dXe e dYddejdejde*ejejffdZdYZ4e e dYddejdejde*ejejffd[d\Z5e d]e e d^d_dejdejde*ejejffd`d^Z6e dae e dbddejdejde*ejejffdcdbZ7e e dbddejdejde*ejejffdddeZ8dfdgZ9e dhe e diddejdMejdNejdededejf djdiZ:e e diddejdMejdNejdededejf dkdlZ;e dme e dndeje dse e dtd uddejdMejdNejdededejf dvdtZ?e e dtd uddejdMejdNejdededejf dwdxZ@e dye e dzdduejsrcCs.i|]}|tt|jtt|jfqSr)intr finfor r r rrrrs.cCs^|tvr td|t|\}}||ksJd|d|||ks-Jd|d|dS)NzUnsupported dtype: z9quant_min out of bound for dtype, quant_min_lower_bound: z quant_min: z9quant_max out of bound for dtype, quant_max_upper_bound: z quant_max: )_DTYPE_TO_QVALUE_BOUNDS ValueError) quant_min quant_maxdtypeZquant_min_lower_boundZquant_max_upper_boundrrr_quant_min_max_bounds_checks"   rzxquantize_per_tensor(Tensor input, float scale, int zero_point, int quant_min, int quant_max, ScalarType dtype) -> Tensorquantize_per_tensorZCompositeExplicitAutogradinputscale zero_pointrrrreturncCsp|jtjtjfvr|tj}|jtjksJd|jt|||d|}tt||||||S)aAffine quantization for the Tensor using the same quantization parameters to map from floating point to quantized values Args: input (torch.Tensor): original float32 or bfloat16 Tensor scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead Tensorzquantize_per_tensor.tensorcCsV|dksJd||dksJd|t||||||SzAffine quantization for the Tensor using the same quantization parameters to map from floating point to quantized values Same as `quantize_per_tensor` but scale and zero_point are Scalar Tensor instead of scalar values >Expecting zero_point tensor to be one element, but received : 9Expecting scale tensor to be one element, but received : numelritemr(rrrquantize_per_tensor_tensorms  r1cCs|jtjtjfvr|tj}|dksJd||dks-Jd||jtjks;Jd|jtj||dS)Nr+r,r-rr&)rr r r!r"r#r/r'r(rrrquantize_per_tensor_tensor_metas    r2zquantize_per_tensor.tensor2(Tensor input, Tensor scale, Tensor zero_point, Tensor quant_min, Tensor quant_max, ScalarType dtype) -> Tensorzquantize_per_tensor.tensor2cCs^|dksJd||dksJd|t||||||Sr*r.r(rrrquantize_per_tensor_tensor2s  r3cCst||||||SN)r2r(rrr quantize_per_tensor_tensor2_metas r5zdequantize_per_tensor(Tensor input, float scale, int zero_point, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensordequantize_per_tensor out_dtyper8cCsV|j|ksJd|d|j|durtj}|tvr$||||Std|)aAffine dequantization for the Tensor using the same quantization parameters to map from quantized values to floating point values Args: input (torch.Tensor): Tensor with dtype matching `dtype` argument, e.g. (`torch.uint8`), it is a per tensor quantized Tensor if combined with quantization parameters in the argument of this function (scale/zero_point) scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization quant_min (int): minimum quantized value for input Tensor (not used in computation, reserved for pattern matching) quant_max (int): maximum quantized value for input Tensor (not used in computation, reserved for pattern matching) dtype (torch.dtype): dtype for input Tensor (not used in computation, reserved for pattern matching) out_dtype (torch.dtype?): optional dtype for output Tensor Returns: dequantized float32 Tensor Expecting input to have dtype: z , but got N,Unsupported dtype in dequantize_per_tensor: )rr r#rr"rrrrrrrr8rrrr6s &cCs|durtj}tj||dSNr&)r r#r'r;rrrdequantize_per_tensor_metas r=zdequantize_per_tensor.tensor(Tensor input, Tensor scale, Tensor zero_point, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensorzdequantize_per_tensor.tensorc CsZ|dksJd||dksJd|t|||||||dSzAffine dequantization for the Tensor using the same quantization parameters to map from quantized values to floating point values Same as `dequantize_per_tensor` but scale and zero_point are Scalar Tensor instead of scalar values r+r,r-r7r/r6r0r;rrrdequantize_per_tensor_tensor's  r@cCs|durtj}|dksJd||dks%Jd||j|ks1Jd||tvr Tensorzdequantize_per_tensor.tensor2c Csb|dksJd||dksJd|t|||||||dSr>r?r;rrrdequantize_per_tensor_tensor2ms  rBc Cst|||||||dS)Nr7)rAr;rrr"dequantize_per_tensor_tensor2_metas rCzrchoose_qparams.tensor(Tensor input, int quant_min, int quant_max, float eps, ScalarType dtype) -> (Tensor, Tensor)zchoose_qparams.tensorqminqmaxepsc Cs||jtjtjtjfvsJd|j|tvs#Jdtd|t||t|\}}t |||||t |gddS)[Given an input Tensor, derive the per tensor affine quantization parameter (scale and zero_point) for target quantized Tensor from the Tensor Args: input (torch.Tensor): floating point input Tensor quant_min (int): minimum quantized value for target quantized Tensor quant_max (int): maximum quantized value for target quantized Tensor dtype (torch.dtype): dtype for target quantized Tensor Returns: scale (float): quantization parameter for the target quantized Tensor zero_point (int): quantization parameter for the target quantized Tensor CExpecting input to have dtype torch.float32/16/b16, but got dtype: $Expecting target dtype to be one of , but got: F)has_customized_qrange) rr r#r r!rkeysraminmaxrTensorrrDrErFrmin_valmax_valrrrchoose_qparams_tensors*    rRz|choose_qparams_symmetric.tensor(Tensor input, int quant_min, int quant_max, float eps, ScalarType dtype) -> (Tensor, Tensor)zchoose_qparams_symmetric.tensorc Cs|jtjtjtjfvsJd|j|tvs#Jdtd|t||t|\}}t |||||t |gdtj dS)rGrHrIrJF)rKZqscheme) rr r#r r!rrLrrMrrNZper_tensor_symmetricrOrrrchoose_qparams_symmetric_tensors,    rScCsj|jtjtjtjfvsJd|j||ks!Jd|d|tjdtj|jdtjdtj|jdfS)NrHzKExpecting quant_min to be smaller than quant_max but received min: z max: r+rdevice) rr r#r r!emptydoublerUint64rrrrFrrrrchoose_qparams_tensor_metas"   rZcCs(tjdtj|jdtjdtj|jdfS)Nr+rT)r rVrWrUrXrYrrr$choose_qparams_symmetric_tensor_metas r[cCs6tt|}d||<||d<|t|}||fS)Nr)listrangedimpermutetuple)xaxisZ new_axis_listyrrr_permute_to_axis_zeros rdzquantize_per_channel(Tensor input, Tensor scales, Tensor zero_points, int axis, int quant_min, int quant_max, ScalarType dtype) -> Tensorquantize_per_channelscales zero_pointsrbc Cs|jtjtjfvr|tj}|jtjksJd|j||ks,Jd|t|||t||\}}dg|}|j d|d<| |}| |}t t |d||||} | t|} | |S)atAffine per channel quantization for the Tensor using the same quantization parameters for each channel/axis to map from floating point to quantized values Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (torch.Tensor): a list of scale quantization parameter for affine quantization, one per channel zero_point (torch.Tensor): a list of zero_point quantization parameter for affine quantization, one per channel quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead rExpecting axis to be < r+rr)rr r r!r"r#r^rrdshapeviewr$r%r_r`) rrfrgrbrrrpermute_axis_list new_shaperesoutrrrre,s"      cCsr|jtjtjfvr|tj}|jtjksJd|j||ks,Jd|t|||tj||dS)Nrrhr&) rr r r!r"r#r^rr')rrfrgrbrrrrrrquantize_per_channel_meta\s   rozdequantize_per_channel(Tensor input, Tensor scales, Tensor? zero_points, int axis, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensordequantize_per_channelc Cs|j|ksJd|d|j|durtj}||ks&Jd|t|||t||\}}dg|} |jd| d<|| }|durT||| |} n||} | |} | t |} | S)aAffine per channel dequantization for the Tensor using the same quantization parameters for each channel/axis to map from quantized values to floating point values Args: input (torch.Tensor): Tensor with dtype matching `dtype` argument, e.g. (`torch.uint8`), it is a per channel quantized Tensor if combined with quantization parameter in the argument of this function (scales/zero_points/axis) scales (torch.Tensor): a list of scale quantization parameter for affine quantization, one per channel zero_points (torch.Tensor): a list of zero_point quantization parameter for affine quantization, one per channel quant_min (int): minimum quantized value for output Tensor (not used in computation, reserved for pattern matching) quant_max (int): maximum quantized value for output Tensor (not used in computation, reserved for pattern matching) dtype (torch.dtype): requested dtype for output Tensor (not used in computation, reserved for pattern matching) out_dtype (torch.dtype?): optional dtype for output Tensor Returns: dequantized float32 Tensor Expecting input to have dtype , but got dtype: Nrhr+r) rr r#r^rrdrirjr"r_r`) rrfrgrbrrrr8rkrlrmrnrrrrpzs" )   cCsf|j|ksJd|d|j|durtj}||ks&Jd|t|||tj||dS)Nrqrrrhr&)rr r#r^rr')rrfrgrbrrrr8rrrdequantize_per_channel_metas  rszLchoose_qparams_per_token(Tensor input, ScalarType dtype) -> (Tensor, Tensor)choose_qparams_per_tokencCsx|jddd}|jtjkr|}|tjkr#d}d|dd}ntd||jdd  |}t |}||fS) Choose quantization parameters for per token quantization. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): original float32/float16 Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor Returns: scales and zero_points, both float32 Tensors Tr^Zkeepdimr+z/unsupported dtype in choose_qparams_per_token: gh㈵>r ) absamaxrr r floatint8 Exceptionr$div zeros_like)rrrfZn_bitsrrgrrrrts   cC@t|jdddg}tj|tj|jdtj|tj|jdfSNrvr+rTr\rir rVrWrUrXrrsizerrrchoose_qparams_per_token_meta  rz]_choose_qparams_per_token_asymmetric_impl(Tensor input, ScalarType dtype) -> (Tensor, Tensor))_choose_qparams_per_token_asymmetric_implZCompositeImplicitAutogradcCsd\}}tj|ddd}tj|ddd}t|t|}t|t|}ttjj}||t ||} | j |d} || } || } || } || } t | | dk|| || }t ||| }| tj| tjfS)ru)irvTrwrzr)r Zaminr|r rr rr#rFr}r$wherer%r"Zfloat64rX)rrrDrErPrQZ min_val_negZ max_val_posrFrZ descaled_minZ descaled_maxZzero_point_from_min_errorZzero_point_from_max_errorrrrrrs&  zWchoose_qparams_per_token_asymmetric(Tensor input, ScalarType dtype) -> (Tensor, Tensor)#choose_qparams_per_token_asymmetriccCs t||Sr4)rrrrrrrE cCrrrrrrr(choose_qparams_per_token_asymmetric_metaQrrcCsftt|dd}||ksJd|d|||ks1Jd|d|dS)Nrvz num_tokens: z scales: z zero_points: )mathprodr\rr/)rrfrgZ num_tokensrrr!_per_token_quant_qparam_dim_check`srz}quantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype) -> Tensorquantize_per_tokencCsBt|||t||||d|||||}|S)aPer token quantization for the Tensor using the quantization parameters to map from floating point to quantized values. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (float32 torch.Tensor): quantization parameter for per token affine quantization zero_points (int32 torch.Tensor): quantization parameter for per token affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead r)rrmuladdr%r$r"rrfrgrrrrrrrps   cCst|||tj||dSr<rr r'rrrrquantize_per_token_metas rzdequantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, ScalarType output_dtype) -> Tensordequantize_per_token output_dtypecCs||}||}||S)aPer token dequantization for the Tensor using the quantization parameters to map from floating point to quantized values. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): quantized Tensor (uint8, int8 etc.) scales (float64 torch.Tensor): quantization parameter for per token affine quantization zero_points (int64 torch.Tensor): quantization parameter for per token affine quantization quant_min (int): minimum quantized value for input Tensor quant_max (int): maximum quantized value for input Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor output_dtype (torch.dtype): dtype (e.g. torch.float32) for output Tensor Returns: dequantized Tensor with dtype `output_dtype` )r"rrfrgrrrrrrrrs cCst|||tj||dSr<rrrrrdequantize_per_token_metas rzquantize_per_channel_group(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, int group_size) -> Tensorquantize_per_channel_groupc Cs|dksJ||jdkr|jddkr|jd}|jd|dks$J|dks,J|d|}t|dks=J|dd}|dd}|d|| || | |}|S)Nr+rvrryr) rir^reshaper isnansumrrr%Zclamp_r" reshape_as) rrfrgrrr group_sizeZto_quantZ input_int8rrrrs"       cCsf|dksJ||jdkr|jddkr|jd}|jd|dks$J|dks,Jtj||dS)aXGroupwise quantization within each channel for an 2-d Tensor using the quantization parameters to map from floating point to quantized values. This means for each row of a 2-d Tensor (M, N), we calculate scales/zero_points for each `group_size` elements and quantize every `group_size` elements with the same quantization parameter. The dimension for scales/zero_points will be (M * ceil(N, group_size),) Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (float32 torch.Tensor): quantization parameter for per channel group affine quantization zero_points (int32 torch.Tensor): quantization parameter for per channel group affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead r+rvrryr&)rir^r r')rrfrgrrrrrrrquantize_per_channel_group_meta s  rzdequantize_per_channel_group(Tensor input, Tensor scales, Tensor? zero_points, int quant_min, int quant_max, ScalarType dtype, int group_size, ScalarType output_dtype) -> Tensordequantize_per_channel_groupw_int8rc Cs|dksJ||jdkr|jddkr|jd}|jd|dks$J|dks,J|d|}|dd}|durC|dd} n tjgtj|jd} || | | |} | S)a!Groupwise dequantization within each channel for an 2-d Tensor using the quantization parameters to map from floating point to quantized values. This means for each row of a 2-d Tensor (M, N), we calculate scales/zero_points for each `group_size` elements and quantize every `group_size` elements with the same quantization parameter. The dimension for scales/zero_points will be (M * ceil(N, group_size),) Args: input (torch.Tensor): quantized Tensor (uint8/int8 etc.) scales (float32 torch.Tensor): quantization parameter for per channel group affine quantization zero_points (int32 torch.Tensor): quantization parameter for per channel group affine quantization quant_min (int): minimum quantized value for input Tensor quant_max (int): maximum quantized value for input Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor output_dtype (torch.dtype): dtype (e.g. torch.float32) for output Tensor Returns: dequantized Tensor with dtype `output_dtype` r+rvrryNrT) rir^rr Zzerosint32rUsubrrr") rrfrgrrrrrZw_int8_groupedZzpZw_dqrrrr5s "   zyfake_quant_per_channel(Tensor input, Tensor scales, Tensor zero_points, int axis, int quant_min, int quant_max) -> Tensorc@s$eZdZeddZeddZdS)FakeQuantPerChannelc Cs|jtjkr |tj}|jtjkr|tj}|jtjks&Jd|j||ks5Jd|ttd|tt|d|j}t ||}t ||} t |d|| } t | ||| |} t | |k| |k} | | | S)Nrrhrr+r)rr r#r"rr^r\r]ndimrr%r$ logical_andZsave_for_backward) ctxrrfrgrbrrZbroadcast_dimsZunsqueeze_scalesZunsqueeze_zero_pointstemprnmaskrrrforwardos$     "   zFakeQuantPerChannel.forwardcCs|j\}||dddddfSr4)Z saved_tensors)rgyrrrrbackwardszFakeQuantPerChannel.backwardN)__name__ __module__ __qualname__ staticmethodrrrrrrrns  rfake_quant_per_channelZAutogradcCst||||||Sr4)rapplyrrfrgrbrrrrrrs cCs t|Sr4r r'rrrrfake_quant_per_channel_metarrzFconvert_element_type.no_fuse(Tensor input, ScalarType dtype) -> Tensorzconvert_element_type.no_fusecCstjjj||Sr4)r opsZprimsconvert_element_typedefaultrrrrrsrcCstj||dSr<rrrrrconvert_element_type_metasr)r)Irtypingrr Z torch._refsrZtorch.ao.quantization.utilsrrZ torch.libraryrrZquantized_decomposed_libZuint8r~Zuint16Zint16rZ_INTEGER_DTYPESZ float8_e5m2Z float8_e4m3fnZ _FLOAT_DTYPESrupdaterZdefinerNr}rrrr)r1r2r3r5r6r=r@rArBrCr`rRrSrZr[rdrerorprsrtrrrrrrrr#rrrrrZautogradFunctionrrrrrrrrrs       $        2            ' '     /     >    #   +      &    !   $ %  .     "