[Th 9|SSKrSSKJr SSKJr SSKJr /SQrSrSr Sr S r SS jr S \R4S jrS S \R4SjrSSjrSSjrSSjrS S \RS4SjrS S \R4SjrSSjrSSjrSSjrSSjrSSS S \R4SjrSSjrSSjrSSjrg) N)QConfig) QuantType)wrap_cpp_module) script_qconfigscript_qconfig_dictfuse_conv_bn_jit prepare_jitprepare_dynamic_jit convert_jitconvert_dynamic_jit quantize_jitquantize_dynamic_jitc[U[RR5(d [ S[ [ U55-5eg)Nz$input must be a script module, got: ) isinstancetorchjit ScriptModule ValueErrorstrtypemodels Z/var/www/auris/envauris/lib/python3.13/site-packages/torch/ao/quantization/quantize_jit.py_check_is_script_modulers8 eUYY33 4 4?#d5kBRRSS 5cZURRS5(d [S5eg)Nforwardz0input script module does not have forward method)_c _has_methodrrs r_check_forward_methodr s( 88   * *KLL +rc[[RRUR 55R [RRUR 55R S9$)zInstantiate the activation and weight observer modules and script them, these observer module instances will be deepcopied during prepare_jit step. ) activationweight)rrrscriptr"rr#)qconfigs rrr sQ 99##G$6$6$89<<yy 0144 rc|UR5VVs0sHupX(a [U5OS_M snn$s snnf)zrHelper function used by `prepare_jit`. Apply `script_qconfig` for all entries in `qconfig_dict` that is not None. N)itemsr) qconfig_dictkvs rrr+s8 =I_prepare_jit..G?+>az!S!!+>4qconfig_dict should only contain names(str) as keys.r) rr allkeysrrrrr,_jit_pass_insert_observersrr/r)rr(r0 quant_typescripted_qconfig_dictr1s r _prepare_jitrCDsE"% ?<+<+<+>? ? ?OPP/ = U ,Ehh11 )2ZG 7# L ( Lrrc[U5 [SUR555(d [S5e[ U5nUR R U5Rn[RRU5 [X5n[RRUR X%X45nU(aURU5 U$[U5nU$)Nc3B# UHn[U[5v M g7fr4r5r6s rr9(_prepare_ondevice_jit..]r;r<r=)rr>r?rrr _get_methodgraphrr,_jit_pass_inliner1_jit_pass_insert_observer_method_for_ondevice_ptqr/r)rr( method_namer0rArB method_graphr1s r_prepare_ondevice_jitrMUsE" ?<+<+<+>? ? ?OPP/ =88'' 4::L HHl+ U ,EhhHH +gG 7# L ( Lrcr[RRS5 [XU[R S9$)Nz)quantization_api.quantize_jit.prepare_jitrA)rr,r-rCrSTATICrr(r0s rr r ms* HH  !LM WAQAQ RRrcr[RRS5 [XU[R S9$)Nz1quantization_api.quantize_jit.prepare_dynamic_jitrO)rr,r-rCrDYNAMICrQs rr r rs* HH  !TU WARAR SSrc4[XX#[RS9$NrO)rMrrSrr(rKr0s r_prepare_ondevice_dynamic_jitrWws ! [i>O>O rcJ[U5 UR5 URn[RR USXU5nU(d\[ SUR555nU(dUR5 Uc/n[RRXSU5nU(aURU5 O [U5n[RRUR5 [RRUR5 U$)Nrc3R# UHoRRS:Hv M g7f)xpuN)devicer)r7ps rr9_convert_jit..sH5GXX]]e+5Gs%')revalrrr,_jit_pass_insert_quant_dequantr> parameterscpu_jit_pass_quant_finalizer/r_jit_pass_constant_propagationrH _jit_pass_dce)rr0debugrApreserved_attrsr1is_xpus r _convert_jitrhsE" JJLhhGhh55GJG HU5E5E5GHH IIK  " O((33   7#( HH++EKK8 HH5;;' Lrc[U5 U[R:XdS5eURS5(aS5eSU-nSU-nURn[ R RURXRU[R5n[ R RU[RU5nU(aURU5 U$[U5nU$)NzOThis API, while should work for static quant, is only tested for dynamic quant.observe_z2Pass in valid method to be quantized, e.g. forward quantize_) rrrS startswithrrr,/_jit_pass_insert_quant_dequant_for_ondevice_ptq)_jit_pass_quant_finalize_for_ondevice_ptqr/r)rrKr0rerAobserve_method_namequantize_method_namer1s r_convert_ondevice_jitrqsE"i'''YXY'%%<;< %{2&4hhGhhFF %y7H7HGhh@@""$8G 7# L ( Lrcv[RRS5 [UUU[R US9$)Nz)quantization_api.quantize_jit.convert_jitrArf)rr,r-rhrrPrr0rerfs rr r s7 HH  !LM   ##'  rcv[RRS5 [UUU[R US9$)Nz1quantization_api.quantize_jit.convert_dynamic_jitrs)rr,r-rhrrSrts rr r s7 HH  !TU   $$'  rc4[XX#[RS9$rU)rqrrS)rrKr0res r_convert_ondevice_dynamic_jitrws  Gy7H7H rc6[XX#5n[XU5nU$r4)rWrwrVs r#_quantize_ondevice_dynamic_jit_implrys! *%{ TE )%g FE LrcU[R:Xa[XU5n[USU5nO>U(dS5eU(dS5e[ XU5nU"U/UQ76 [ USU5n[ RRUR5 [ RRUR5 U$)NTzGMust provide calibration function for post training static quantizationzFMust provide calibration dataset for post training static quantization) rrSr r r r rr,rcrHrd)rr(run_fnrun_argsr0rerAs r _quantize_jitr}sY&&&#EA#E47  U T U   T S T E9u x E4/ HH++EKK8 HH5;;' Lrc z[RRS5 [UUUUUU[R S9$)a:Quantize the input float TorchScript model with post training static quantization. First it will prepare the model for calibration, then it calls `run_fn` which will run the calibration step, after that we will convert the model to a quantized model. Args: `model`: input float TorchScript model `qconfig_dict`: qconfig_dict is a dictionary with names of sub modules as key and qconfig for that module as value, empty key means the qconfig will be applied to whole model unless it's overwritten by more specific configurations, the qconfig for each module is either found in the dictionary or fallback to the qconfig of parent module. Right now qconfig_dict is the only way to configure how the model is quantized, and it is done in the granularity of module, that is, we only support one type of qconfig for each torch.nn.Module, and the qconfig for sub module will override the qconfig for parent module, empty string means global configuration. `run_fn`: a calibration function for calibrating the prepared model `run_args`: positional arguments for `run_fn` `inplace`: carry out model transformations in-place, the original module is mutated `debug`: flag for producing a debug friendly model (preserve weight attribute) Return: Quantized TorchSciprt model. Example: ```python import torch from torch.ao.quantization import get_default_qconfig from torch.ao.quantization import quantize_jit ts_model = torch.jit.script(float_model.eval()) # or torch.jit.trace(float_model, input) qconfig = get_default_qconfig('fbgemm') def calibrate(model, data_loader): model.eval() with torch.no_grad(): for image, target in data_loader: model(image) quantized_model = quantize_jit( ts_model, {'': qconfig}, calibrate, [data_loader_test]) ``` z*quantization_api.quantize_jit.quantize_jitrO)rr,r-r}rrP)rr(r{r|r0res rr r s@d HH  !MN   ## rcr[RRS5 [XX#[R S9$)aQuantize the input float TorchScript model with post training dynamic quantization. Currently only qint8 quantization of torch.nn.Linear is supported. Args: `model`: input float TorchScript model `qconfig_dict`: qconfig_dict is a dictionary with names of sub modules as key and qconfig for that module as value, please see detailed descriptions in :func:`~torch.ao.quantization.quantize_jit` `inplace`: carry out model transformations in-place, the original module is mutated `debug`: flag for producing a debug friendly model (preserve weight attribute) Return: Quantized TorchSciprt model. Example: ```python import torch from torch.ao.quantization import per_channel_dynamic_qconfig from torch.ao.quantization import quantize_dynamic_jit ts_model = torch.jit.script(float_model.eval()) # or torch.jit.trace(float_model, input) qconfig = get_default_qconfig('fbgemm') def calibrate(model, data_loader): model.eval() with torch.no_grad(): for image, target in data_loader: model(image) quantized_model = quantize_dynamic_jit( ts_model, {'': qconfig}, calibrate, [data_loader_test]) ``` z2quantization_api.quantize_jit.quantize_dynamic_jit)r0rerA)rr,r-r}rrS)rr(r0res rrr6s2L HH  !UV  WiFWFW rc[XX#S9$)a Prepares the input float TorchScript model with *on-device* post training dynamic quantization. Currently only qint8 quantization of torch.nn.Linear is supported. Args: `model`: input float TorchScript model `qconfig_dict`: qconfig_dict is a dictionary with names of sub modules as key and qconfig for that module as value, please see detailed `method_name`: Name of the method within the model, to be prepared for quantization descriptions in :func:`~torch.ao.quantization.quantize_jit` `inplace`: carry out model transformations in-place, the original module is mutated Return: TorchScript model that is ready for on device quantization. This means that the returned model has: - Method is inlined. - Model has observer modules inserted in the model. - Model has packed params inserted in the model. However they are empty as in they dont contain valid quantized weights. - observe_ is added that observe the values to be quantized. - reset_observers_ to reset observers. - quantize_ is added to the model. - This method extract scale, zero points. - Quantizes observed weights. - Creates packed params from it and update the attribute of the model with the new values for the packed params. - Reset the original fp32 weights with empty tensor using SetAttr. - quantized_ is added to the model. - This method uses quantized weights and quantized linear ops instead of fp32 op. - This method should be used for inference post PTQ. - Note that all method's signatures should be the same as method_name. Later on device: - Run reset_observers_ - Run observe_ - Run quantize_ - Now model can be saved and loaded later. - Run model with quantized_ Example: ```python import torch from torch.ao.quantization import per_channel_dynamic_qconfig from torch.ao.quantization.quantize_jit import _quantize_ondevice_dynamic_jit ts_model = torch.jit.script(float_model.eval()) # or torch.jit.trace(float_model, input) qconfig = get_default_qconfig('fbgemm') quant_ready_model = _quantize_ondevice_dynamic_jit( ts_model, {'': qconfig}, 'forward', True) ``` )r0)ryrVs r_quantize_ondevice_dynamic_jitrbsv / [ r)F)rF)FFN)FF)rtorch.ao.quantization.qconfigr torch.ao.quantization.quant_typertorch.jit._recursiver__all__rr rrrrPrCrMr r rWrhrqr r rwryr}r rrrrrs 160 T M R"/4 @P@P(  0S T 9>)2B2BTX<!&Uy?O?O4/4    <;|)Z9>=r