a h#y@s*ddlZddlZddlZddlZddlZddlZddlmmm Z ddl mZddl m Z ddlmZddlmZmZmZmZmZmZddlmZmZmZmZmZmZmZmZddl m!Z!m"Z"ddl#m$Z$dd l%m&Z&m'Z'm(Z(gd Z)eZ*ej+ej,j+ej-ej,j-iej,j+ej j+ej,j-ej j-id Z.d d Z/d7ddZ0d8ddZ1ddZ2ddZ3d9ddZ4d:ddZ5ddZ6ddZ7e8e&d;d d!Z9d"d#Z:d$d%Z;e8e&de8e&d=d*d+Z?e8e&d>d,d-Z@e8e&d?d/d0ZAd@d1d2ZBdAd3d4ZCdBd5d6ZDdS)CN) _FusedModule)_is_activation_post_process)_activation_is_memoryless_add_module_to_qconfig_obs_ctrdefault_dynamic_qconfigfloat16_dynamic_qconfig!float_qparams_weight_only_qconfig&float_qparams_weight_only_qconfig_4bit)_get_special_act_post_process_has_special_act_post_process)get_default_dynamic_quant_module_mappingsget_default_qat_module_mappings$get_default_qconfig_propagation_list(get_default_static_quant_module_mappings2get_default_static_quant_reference_module_mappingsno_observer_set) DeQuantStub QuantWrapper)type_before_parametrizations)DEPRECATION_WARNINGget_qparam_dict)has_no_children_ignoring_parametrizations) get_default_custom_config_dictpropagate_qconfig_add_quant_dequantpreparequantizequantize_dynamic prepare_qat quantize_qatconvert swap_module)%float_to_observed_custom_module_class)observed_to_quantized_custom_module_classcCstS)z'Defines the default custom config dict.)_DEFAULT_CUSTOM_CONFIG_DICTr&r&L/var/www/auris/lib/python3.9/site-packages/torch/ao/quantization/quantize.pyrGsrc Cs|t||}|||}t|d|}tjjj||t||}||_| D]V\}}|rj|d|n|} |dus||dgvsRt ||dgvsRt |||| qRdS)aThis is a helper function for `propagate_qconfig_` Args: module: input module qconfig_dict: dictionary that maps from name of submodule to quantization configuration qconfig_parent: quantization config of parent module, we will fallback to this config when there is no specified config for current module prefix: corresponding prefix of the current module, used as key in qconfig_dict prepare_custom_config_dict: dictionary for custom handling of modules see docs for :func:`~torch.ao.quantization.prepare_fx` Return: None, module is modified inplace with qconfig attached qconfig.NZnon_traceable_module_nameZnon_traceable_module_class) getrgetattrtorchaoZ quantizationr)Z_assert_valid_qconfigrnamed_childrentype_propagate_qconfig_helper) module qconfig_dictZqconfig_parentprefixprepare_custom_config_dictZmodule_qconfigZqconfig_with_device_checknamechild module_prefixr&r&r'r1Ls(    r1cCs*|dur i}|duri}t|||ddS)aPropagate qconfig through the module hierarchy and assign `qconfig` attribute on each leaf module Args: module: input module qconfig_dict: dictionary that maps from name or type of submodule to quantization configuration, qconfig applies to all submodules of a given module unless qconfig for the submodules are specified (when the submodule already has qconfig attribute) prepare_custom_config_dict: dictionary for custom handling of modules see docs for :func:`~torch.ao.quantization.prepare_fx` Return: None, module is modified inplace with qconfig attached N)r5)r1)r2r3r5r&r&r'r}srcCs ||S)z.Forward hook that calls observer on the outputactivation_post_process)selfinputoutputr&r&r'_observer_forward_hooksr>cCs||dS)z2Forward pre hook that calls observer on the outputrr9)r;r<r&r&r'_observer_forward_pre_hooksr?FcCs8t|dsJd|r&|jtddn|jtdddS)Nr:zGExpect activation_post_process attribute already attached to the moduleT)prepend)hasattrregister_forward_pre_hookr?register_forward_hookr>)r2pre_hookr&r&r'&_register_activation_post_process_hooks  rEc s.|durt}|duri}dur`t|}t|dksDJd|t|dkr\tt|ndddddddfd d }|D]8\}}t|tjfvrqqt t|t j t j fr|rt |d sJd t|d |j|_qt|tr|r||q|durBt||vrB|r||qt|r`t|} ||| q|rt||vr|t|} | |} t||| t | tts|| qt||||qt|rt|tjjst||vr||t |dr*t|tjjs*t||vr*||dS)asAdd observer for the leaf child of the module. This function insert observer module to all leaf child module that has a valid qconfig attribute. Args: module: input module with qconfig attributes for all the leaf modules that we want to quantize qconfig_propagation_list: a list of quantizable modules that will have observers added to them if they are leaf nodes device: parent device, if any non_leaf_module_list: list of non-leaf modules we want to add observer Return: None, module is modified inplace with added observer modules and forward_hooks NrzR_add_observer_ only works with cpu or single-device CUDA modules, but got devices rcSs,|dur|n|}|dur(|||SN) activationto)r)devicespecial_act_post_processrGr&r&r'get_activation_post_processs  z3_add_observer_..get_activation_post_processcSst|do|jduS)Nr)rAr))mr&r&r'needs_observationsz)_add_observer_..needs_observationcs>|r:t|ts:|d|j|t|t|jddS)zmAdds an activation post process module and register a pre or post hook that calls the module r:rDN) isinstancerZ add_moduler)rEr)rMrJrIrKrNr&r'insert_activation_post_processs z6_add_observer_..insert_activation_post_processr:zfunctional class z- has no pre-defined `activation_post_process`Zweight_fake_quant)N)N)r_get_unique_devices_lennextiterr/rnnZDropout issubclassnnqZFloatFunctionalZ QFunctionalrAr)r:rPrr r from_floatsetattrtupler_add_observer_rr-Z Sequential) r2qconfig_propagation_listZnon_leaf_module_listrIcustom_module_class_mappingdevicesrRr6r7rJZobserved_classZobserved_childr&rQr'r]s                   r]cCs$dd|Ddd|DBS)NcSsh|]}|jjdkr|jqS)meta)rIr0).0pr&r&r' 4z'_get_unique_devices_..) parametersbuffersr2r&r&r'rS3srScCsDt|r t|dr |jr t|S|D]\}}t||j|<q(|S)a{Wrap the leaf child module in QuantWrapper if it has a valid qconfig Note that this function will modify the children of module inplace and it can return a new module which wraps the input module as well. Args: module: input module with qconfig attributes for all the leaf modules that we want to quantize Return: Either the inplace modified module with submodules wrapped in `QuantWrapper` based on qconfig or a new `QuantWrapper` module which wraps the input module, the latter case only happens when the input module is a leaf module and we want to quantize it. r))rrAr)rr/r_modules)r2r6r7r&r&r'r9srcCstjd|durt}|di}|s4t|}|}|durFt}t|ddt dd| Dsrt dt ||||d|S) afPrepares a copy of the model for quantization calibration or quantization-aware training. Quantization configuration should be assigned preemptively to individual submodules in `.qconfig` attribute. The model will be attached with observer or fake quant modules, and qconfig will be propagated. Args: `model`: input model to be modified in-place `inplace`: carry out model transformations in-place, the original module is mutated `allow_list`: list of quantizable modules `observer_non_leaf_module_list`: list of non-leaf modules we want to add observer `prepare_custom_config_dict`: customization configuration dictionary for prepare function .. code-block:: python # Example of prepare_custom_config_dict: prepare_custom_config_dict = { # user will manually define the corresponding observed # module class which has a from_float class method that converts # float custom module to observed custom module "float_to_observed_custom_module_class": {CustomModule: ObservedCustomModule} } z!quantization_api.quantize.prepareNr#r3css|]}t|do|jVqdS)r)NrL)rbrMr&r&r' rezprepare..zNone of the submodule got qconfig applied. Make sure you passed correct configuration through `qconfig_dict` or by assigning the `.qconfig` attribute directly on submodules)r_)r-_C_log_api_usage_oncerr+copydeepcopyrranymoduleswarningswarnr])modelinplaceZ allow_listobserver_non_leaf_module_listr5r_r^r&r&r'rTs."   rcsDtdrtjrtddfdd }|dd|dddS)Nr:Fcs^|r jnj}|rtnt}t}|D]\}}||ur*||q*|D]}||qJdSrF)_forward_pre_hooks_forward_hooksr?r>setitemsaddpop)rDZhook_mapZ observer_hookZhandle_ids_to_removeZ handle_idhook_fnrhr&r' remove_hookss  z5_remove_activation_post_process..remove_hooksTrO)F)rArr:delattr)r2r~r&rhr'_remove_activation_post_processs   rcCs0|D] }t|qt|dr$|`t|dS)zClean up the qconfig left in the module so that new qconfig can be propagated. Args: module: module to be cleaned up r)N)children_remove_qconfigrAr)r)r2r7r&r&r'rs    rcCs^tjd|durt}|s(t|}|t|dd||g|Rt||dd|S)aQuantize the input float model with post training static quantization. First it will prepare the model for calibration, then it calls `run_fn` which will run the calibration step, after that we will convert the model to a quantized model. Args: model: input float model run_fn: a calibration function for calibrating the prepared model run_args: positional arguments for `run_fn` inplace: carry out model transformations in-place, the original module is mutated mapping: correspondence between original module types and quantized counterparts Return: Quantized model. z"quantization_api.quantize.quantizeNTru) r-rlrmrrnroevalrr!)rtrun_fnrun_argsmappingrur&r&r'rs   rc Csttjd|dur|tjkrHtjttjttjttj ttj ttj ti}nv|tj kr|tjt tjt tjt tj t tj t tj t i}nB|tjkrtjttjti}n&|tjkrtjti}ntd|dnpt|tr0|tjurt}n@|tj urt }n0|tjurt}n |tjurt}ntdt|tt|t|}|dur@t}|sPt|}| t!||t"||dd|S)avConverts a float model to dynamic (i.e. weights-only) quantized model. Replaces specified modules with dynamic weight-only quantized versions and output the quantized model. For simplest usage provide `dtype` argument that can be float16 or qint8. Weight-only quantization by default is performed for layers with large weights size - i.e. Linear and RNN variants. Fine grained control is possible with `qconfig` and `mapping` that act similarly to `quantize()`. If `qconfig` is provided, the `dtype` argument is ignored. Args: model: input model qconfig_spec: Either: - A dictionary that maps from name or type of submodule to quantization configuration, qconfig applies to all submodules of a given module unless qconfig for the submodules are specified (when the submodule already has qconfig attribute). Entries in the dictionary need to be QConfig instances. - A set of types and/or submodule names to apply dynamic quantization to, in which case the `dtype` argument is used to specify the bit-width inplace: carry out model transformations in-place, the original module is mutated mapping: maps type of a submodule to a type of corresponding dynamically quantized version with which the submodule needs to be replaced z*quantization_api.quantize.quantize_dynamicNz5Don't know how to quantize with default settings for z. Provide full qconfig pleasez.Unknown dtype specified for quantize_dynamic: Tr)#r-rlrmqint8rWZLinearrLSTMZGRUZLSTMCellZRNNCellZGRUCellfloat16rZquint8Z EmbeddingBagrZ EmbeddingZquint4x2r ValueErrorrPry RuntimeErrorstrdictzip itertoolsrepeatr rnrorrr!)rtZ qconfig_specZdtyperruZdefault_qconfigr&r&r'rsb              rcCsltjd|jsJd|dur(t}|s6t|}t|ddt||dddt |t | dd|S) a Prepares a copy of the model for quantization calibration or quantization-aware training and converts it to quantized version. Quantization configuration should be assigned preemptively to individual submodules in `.qconfig` attribute. Args: model: input model to be modified in-place mapping: dictionary that maps float modules to quantized modules to be replaced. inplace: carry out model transformations in-place, the original module is mutated z%quantization_api.quantize.prepare_qatz1prepare_qat only works on models in training modeNrjTF)rruremove_qconfig)rvru) r-rlrmZtrainingr rnrorr!rryvalues)rtrrur&r&r'r:s   rcCsNtjd|st|}|t|dd||g|Rt|dd|S)agDo quantization aware training and output a quantized model Args: model: input model run_fn: a function for evaluating the prepared model, can be a function that simply runs the prepared model or a training loop run_args: positional arguments for `run_fn` Return: Quantized model. z&quantization_api.quantize.quantize_qatTr)r-rlrmrnroZtrainrr!)rtrrrur&r&r'r Xs    r TcCs>tjd|st|}t||d|||d|r:t||S)aConverts submodules in input module to a different module according to `mapping` by calling `from_float` method on the target module class. And remove qconfig at the end if remove_qconfig is set to True. Args: `module`: prepared and calibrated module `mapping`: a dictionary that maps from source module type to target module type, can be overwritten to allow swapping user defined Modules `inplace`: carry out model transformations in-place, the original module is mutated `convert_custom_config_dict`: custom configuration dictionary for convert function `use_precomputed_fake_quant`: a flag to enable use of precomputed fake quant .. code-block:: python # Example of convert_custom_config_dict: convert_custom_config_dict = { # user will manually define the corresponding quantized # module class which has a from_observed class method that converts # observed custom module to quantized custom module "observed_to_quantized_custom_module_class": { ObservedCustomModule: QuantizedCustomModule } } z!quantization_api.quantize.convertT)ru is_referenceconvert_custom_config_dictuse_precomputed_fake_quant)r-rlrmrnro_convertr)r2rrurrrrr&r&r'r!ps%  r!c Cs|dur|rtnt}|dur&t}|di}|s@t|}i}|D]D\}} t| ts~t | |vr~t | |d|||dt | |||||<qL| D]\} } | |j | <q|S)aoConverts submodules in input module to a different module according to `mapping` by calling `from_float` method on the target module class Args: module: input module mapping: a dictionary that maps from source module type to target module type, can be overwritten to allow swapping user defined Modules inplace: carry out model transformations in-place, the original module is mutated is_reference: a flag to enable quantized reference module use_precomputed_fake_quant: a flag to enable use of precomputed fake quant Nr$Tr)rrrr+rnror/rPrrrr"rzri) r2rrurrrr_Zreassignr6modkeyvaluer&r&r'rs@    rcCs|}t|dr|jdurd}t||vrD|t||}d}nt||vr|t|}t|dr|jr|jduszJ|j}||jt|}|||}n0t |j} d| j vr|j||d}n ||}d}|r|j D]} | | q|j D]} | tur|| qt|} t| dks`t| d krRtd | vs`Jd | t| d krztt| nd} | r|| |S) a Swaps the module if it has a quantized counterpart and it has an `observer` attached. Args: mod: input module mapping: a dictionary that maps from nn module to nnq module Return: The corresponding quantized module of `mod` r)NFT _IS_REFERENCErrrrazOswap_module only works with cpu or single-device CUDA modules, but got devices r)rAr)rZ from_observedrZweightrrZinspect signaturerfrwrrBrxr>rCrSrTr-rIrUrVrH)rrr_rnew_modZswappedZqmodZweight_post_processZweight_qparamssigZ pre_hook_fnr}r`rIr&r&r'r"sV              r"cCsZdd}t|dr$|j|||d<|D](\}}|rD|||n|}t|||q,dS)a,Traverse the modules and save all observers into dict. This is mainly used for quantization accuracy debug Args: mod: the top module we want to save all observers prefix: the prefix for the current module target_dict: the dictionary used to save all the observers cSs|dkr |S|dS)Nr(r*r&)r4r&r&r' get_prefix*sz&_get_observer_dict..get_prefixr:N)rAr:r/_get_observer_dict)r target_dictr4rr6r7r8r&r&r'r!s  r)Nr(N)NN)F)NNNN)FNNN)NF)NF)F)NFTFNF)NFFNF)F)r()ErnrrZtyping_extensionsrrr-Ztorch.ao.nn.quantizedr.rWZ quantizedrYZtorch.nnZtorch.ao.nn.intrinsicrZtorch.ao.quantization.observerrZtorch.ao.quantization.qconfigrrrrrr Z+torch.ao.quantization.quantization_mappingsr r r r rrrrZtorch.ao.quantization.stubsrrZtorch.nn.utils.parametrizerutilsrrr__all__Zis_activation_post_processrZ quantizableZMultiheadAttentionr%rr1rr>r?rEr]rSr deprecatedrrrrrrrr r!rr"rr&r&r&r's    (        1     B  Z   6 ? >