eThcP SrSSKrSSKJr SSKJr SSKJr SSKJ r J r J r J r SSK JrJrJrJr \ "5(a SSKrSSKJr \ R*"\5r/S Q/S Q/S QS S ./S Q/SQ/S QS SS./S Q/S Q/S QS S ./S Q/S Q/S QS S .S.rSSS.SSS.SSS.SSS.SSS.SSS.SSS.SSS.S.rSrS$S\4SjjrSrSrSrS r S!r!S"r"S#r#g)%z;AWQ (Activation aware Weight Quantization) integration fileN)version)ACT2FN)PreTrainedModel)is_auto_awq_availableis_ipex_availableis_torch_availablelogging)AwqBackendPackingMethod AwqConfigAWQLinearVersionExllamaVersion)q_projk_projv_projo_proj) gate_projup_proj down_proj)input_layernormpost_attention_layernormnormF) attentionmlp layernorm use_alibi)w1w3w2g.A)rrrr rope_theta)mistralmixtralllamallavaactc_fc)r%layer_before_act dense_h_to_4hrfc_in gelu_impl) starcoder2RefinedWebModelfalconmptgptjgpt_neox gpt_bigcodebloomcfSSKJn U[;aU$UR5Hup4[USn[USnX5:Xa][ X5(aM[ U[US5nUR n[R"U5n U"XI5URU'[XA5n M U$)Nr)ScaledActivationr%r') awq.modules.actr4AWQ_SCALES_MAPPINGSnamed_childrenhasattrgetattr out_featurestorchones_modulesreplace_quantization_scales) model model_typer4namemoduleact_namelayer_before_act_namer'size scale_like_s U/var/www/auris/envauris/lib/python3.13/site-packages/transformers/integrations/awq.pyr>r>Ms0,, ,,. &z259 3J ?@R S   E E&u.A*.MN`.ab #00DD)J#3F#GENN4 ' ;/ Lreturnc B^Uc/nURn[5(d [S5eU[R:XGaUR [ R:Xa SSKJ n UnOUR [ R:Xa SSK J n UnOUR [ R:XaoURS[R :Xa SSKJn U nOURS[R&:Xa SSKJn U nOb[S URS35eUR [ R,:Xa SS KJn U nO [S UR 35eSS KJn U nUR75GH%upTc/mTR9U 5 [;U[<R>5(aX;a[AU4S jU55(dURBnURDnU"URFURHUUURJSLURLRNS9URPU 'SnURPU RSS5 [U[WURY555S:a[[UUTUUS9unnTR]S5 GM( X4$)a\ Public method that recursively replaces the Linear layers of the given model with AWQ quantized layers. `accelerate` is needed to use this method. Returns the converted model and a boolean that indicates if the conversion has been successful or not. During the module replacement, we also infer the backend to use through the `quantization_config` object. Args: model (`torch.nn.Module`): The model to convert, can be any `torch.nn.Module` instance. quantization_config (`AwqConfig`): The quantization config object that contains the quantization parameters. modules_to_not_convert (`list`, *optional*): A list of modules to not convert. If a module name is in the list (e.g. `lm_head`), it will not be converted. current_key_name (`list`, *optional*): A list that contains the current key name. This is used for recursion and should not be passed by the user. has_been_replaced (`bool`, *optional*): A boolean that indicates if the conversion has been successful or not. This is used for recursion and should not be passed by the user. NzAWQ (either `autoawq` or `llmawq`) is not available. Please install it with `pip install autoawq` or check out the installation guide in https://github.com/mit-han-lab/llm-awqr) WQLinear_GEMM) WQLinear_GEMVr)WQLinear_Exllama)WQLinear_ExllamaV2Unrecognized Exllama version:  WQLinear_IPEXzUnrecognized AWQ version: )WQLinearc3J># UHoSRT5;v M g7f).N)join).0keycurrent_key_names rH *replace_with_awq_linear..s [DZSchh'788DZs #)w_bit group_size in_featuresr:biasdevTF)modules_to_not_convertrYquantization_confighas_been_replaced)/backendr ValueErrorr AUTOAWQrr GEMMawq.modules.linear.gemmrLGEMVawq.modules.linear.gemvrMEXLLAMAexllama_configrONEawq.modules.linear.exllamarNTWOawq.modules.linear.exllamav2rOIPEXawq.modules.linear.gemm_ipexrRawq.quantize.qmodulerSr7append isinstancennLinearanyr^r:bitsr]r_weightdevicer=requires_grad_lenlistchildrenreplace_with_awq_linearpop)r?rarbrYrcrerL target_clsrMrNrOrRrSrArBr^r:rGs ` rHrr^sb8%!#!))G " " ~  )111  & &*:*?*? ? =&J ( (,<,A,A A =&J ( (,<,D,D D"11)<@R@RRG- $33I>.BTBTTK/  #ABUBdBdenBoAp!qrr ( (,<,A,A A B&J9:M:U:U9VWX X1 ,,.   #! % fbii ( (T-O[DZ[[[$00 %22 '1-222== +!-D0 ,, (t$%)!t$33E: tFOO%& '! +#:'=!1$7"3 $ A  R A/B  ##rIc[U[5(d"[SURR35eUR bUR nUR US'U$URR[;ax[URRnURRSS9nURnURn[USU5nXBS'XRS'XbS'UR US'U$[S5e) aJ Returns the fusing mapping given the quantization config and the model Args: model (`~PreTrainedModel`): The model to fuse - note this model should have been converted into AWQ format beforehand. quantization_config (`~transformers.quantization_config.AWQConfig`): The quantization configuration to use. z:The model should be an instance of `PreTrainedModel`, got max_seq_lenTdecodernum_key_value_heads hidden_sizenum_attention_headsaFusing mapping not found either on the quantization config or the supported `AWQ_FUSED_MAPPINGS`. Please pass a `fused_mapping` argument in the `quantization_config` or raise an issue on transformers https://github.com/huggingface/transformers to add its support.)rvr TypeError __class____name__modules_to_fusefuse_max_seq_lenconfigr@AWQ_FUSED_MAPPINGSget_text_configrrr9rf)r?rbcurrent_fused_mappingrrrrs rHget_modules_to_fusers e_ - -TUZUdUdUmUmTnopp**6 3 C C/B/S/Sm,, ! +  $6 6 25<<3J3J K--d-;(( $88%f.CEXY0;m,7J347J34/B/S/Sm, !  N  rIc^ ^ ^ [U[5(a[R"U5nURn[ X5n[ USS5nU[R:XaSSK J n SSK J n SSK Jn O [S5e/m UR!5Hum nUb[#U 4SjU55(aM%[%US X5 UR&S :wa[)UT US X5 O[*R-S 5 [/XUT U5n U (dMT R1T R3S 5S5 M [5T 5S:axUR!5Hdum n[#U U 4SjT 55(dM#[7US5(dM6[7UR8S5(dMSSUR8lMf U$)a. Optionally fuse some modules in the model to speedup inference. Args: model (`~PreTrainedModel`): The model to fuse - note this model should have been converted into AWQ format beforehand. quantization_config (`Union[AwqConfig, dict]`): The quantization configuration to use. raNr)QuantAttentionFused) QuantFusedMLP)FasterTransformerRMSNormz0Fusing is only supported for the AutoAWQ backendc3,># UH oT;v M g7fN)rWmodule_name_to_not_convertrAs rHrZ#fuse_awq_modules..soXn:T5Xnsripexrz7The IPEX version AWQ does not support fuse mlp for now.rUc3.># UH nTT;v M g7frr)rWfused_attention_parent_modulefused_attention_modules module_names rHrZr(s\s;X 66\ssr_attn_implementationcustom)rvdictr from_dictrerr9r rgawq.modules.fused.attnrawq.modules.fused.mlprawq.modules.fused.normrrf named_modulesry_fuse_awq_layernormr _fuse_awq_mlploggerinfo_fuse_awq_attention_layersrusplitr~r8rr) r?rbrerrarrrrBattention_has_been_fusedrrrAs @@@rHfuse_awq_modulesrs%t,,'112EF!))G)%EO$%8:RTXY)111>7CKLL ++- f ! -oXnooo OK8&[  & && 0 %u'=v U KKQ R$> ?D2E$   $ # # * *4::c?1+= >).2 "#a'#(#6#6#8 K\s68,,H^1_1_9AFMM6 $9 LrIcUHnn[X5(dM[X5nU"URUR5R URR 5UR U'AMp g)a Fuse the LayerNorm layers into a target class using autoawq Args: fuse_module_names (`List[str]`): The list of module names to fuse module (`nn.Module`): The pytorch parent module that has layernorm modules to fuse target_cls (`~autoawq.FasterTransformerRMSNorm`): The `FasterTransformerRMSNorm` class as it only supports that class for now. N)r8r9r{variance_epsilontor|r=)fuse_module_namesrBrr old_modules rHrr0sh) 6 ' ' 5J+5!!++,b""))* OOK ()rIc[U5S:Xag[X2S5(a[X2S5n[X2S5n[X2S5nURRnUR R SS9n U Rn [U n U"XWXk5n URSS5upURU 5n[XU RU55 AAAgg)a Fuse the MLP layers into a target class using autoawq Args: model (`~PreTrainedModel`): The input pretrained model current_module_name (`str`): The current submodule name fuse_module_names (`List[str]`): The list of module names to fuse. For the MLP layers it has to be an array of length 3 that consists of the 3 MLP layers in the order (gate (dense layer post-attention) / up / down layers) module (`nn.Module`): The pytorch parent module that has layernorm modules to fuse target_cls (`~autoawq.QuantFusedMLP`): The `QuantFusedMLP` class as it only supports that class for now. rNrTrrU) r~r8r9qweightr|rr hidden_actrrsplit get_submodulesetattrr)r?current_module_namerrBrrrrprevious_devicerr activation_fn new_module parent_name child_nameparents rHrrGs$ "v+,,Fa$89 &A"67Fa$89 #++22--d-;&& z*  gM "5"<"boolrrrrrrrrrrIrHrs5> ,YY   H %>4L >!L >4L >4L ): V<$/JAi 8w 7?C f= o F &  f$  f$R&!R=@.&*R\!~2 rI