fTh:3SrSSKJrJrJr SSKrSSKrSSKJr SSKJ r SSK J r SSK J r SS KJr SS KJr SS KJr S S KJrJrJrJrJrJrJr SSKJr \R<"\5r Sr!Sr""SS\RF5r$S"Sjr%"SS\RF5r&"SS\5r'"SS\5r("SS\\(5r)"SS\5r*"SS \5r+/S!Qr,g)#zPyTorch Phi-3 model.)CallableOptionalTupleN)nn)ACT2FN)Cache)FlashAttentionKwargs)ALL_ATTENTION_FUNCTIONS)Unpack)logging)MistralDecoderLayerMistralForCausalLM MistralForSequenceClassificationMistralForTokenClassificationMistralPreTrainedModeleager_attention_forward rotate_half) Phi3Configz microsoft/Phi-3-mini-4k-instructrcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)Phi3MLP0c >[TU]5 Xl[R"UR SUR -SS9Ul[R"UR UR SS9Ul[URUl g)NrFbias) super__init__configrLinear hidden_sizeintermediate_size gate_up_proj down_projr hidden_act activation_fn)selfr __class__s ]/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/phi3/modular_phi3.pyrPhi3MLP.__init__1sn  IIf&8&8!f>V>V:V]bc6#;#;V=O=OV[\#F$5$56 hidden_statesreturncURU5nURSSS9up2X RU5-nURU5$)Nrdim)r$chunkr'r%)r(r- up_statesgates r*forwardPhi3MLP.forward9sH%%m4 #//!/4 2 24 88 ~~i((r,)r'r r%r$) __name__ __module__ __qualname____firstlineno__rtorch FloatTensorr6__static_attributes__ __classcell__r)s@r*rr0s,7)U%6%6)5;L;L))r,rcNURU5nURU5nURSnUSSU24USUS24pUSSU24USUS24p[R"Xr-[ U5U--U/SS9n [R"X-[ U 5U--U /SS9n X4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. r0.Nr1) unsqueezeshaper<catr) qkcossin position_ids unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embeds r*apply_rotary_pos_embrRBs( -- &C -- &C2Jc;J;&'3 +;)<6c;J;&'3 +;)<6ii%++e*[TU]5 XlX l[ USUR UR -5UlUR UR-Ul URUlURS-Ul URUl SUl UR UR-SURUR---n[R"UR UR-UR SS9Ul[R"UR USS9Ulg)Nhead_dimgTrFr)rrr rVgetattrr"num_attention_headsrXnum_key_value_headsnum_key_value_groupsscalingattention_dropout is_causalrr!o_projqkv_proj)r(r rVop_sizer)s r*rPhi3Attention.__init__es  " F4F4F&JdJd4de $*$>$>&B\B\$\!#)#=#= }}d* !'!9!9,,t}}d>d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL"4;;0@$G %  % ! \"));;;;FFH kk+. L((r,) r^r rXr_rVr\r[r`rar])N)NN)r8r9r:r;__doc__rrintrr<Tensorrr LongTensorr r r6r>r?r@s@r*rTrTbsG Kz Khsm K K(+/59 6)||6)#5<<#=>6)!. 6) ! 6) !!1!12 6)-.6) u||Xell3XeELL>Q5RR S6)6)r,rTc^\rSrSrS\S\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \ S \ \RS \ \ \R\R4S \\S\ \R \ \ \R \R 444SjjrSrU=r$)Phi3DecoderLayerr rVc>[TU]X5 Xl[XS9Ul[ U5Ul[R"UR5Ul [R"UR5Ul g)N)r rV) rrr rT self_attnrmlprDropout resid_pdropresid_attn_dropoutresid_mlp_dropout)r(r rVr)s r*rPhi3DecoderLayer.__init__sZ + &fJ6?"$**V-?-?"@!#F,>,>!?r,r-rerIrfrl use_cachergrdrhr.c Un URU5nUR"SUUUUUUUUS.U D6upXRU5-nUn URU5nUR U5nXR U5-nU4n U(aX4- n U $)a5 Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_value (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )r-rerIrfrlrrgrd)input_layernormrrpost_attention_layernormrr) r(r-rerIrfrlrrgrdrhresidualself_attn_weightsoutputss r*r6Phi3DecoderLayer.forwardsD!,,]; ,0>> , ')%)/) 3 ,  , ( !#:#:=#II  55mD /  #9#9-#HH  "  + +Gr,)r rrrr)NNNFFNN)r8r9r:r;rrrr<rrrr boolrr r r=r6r>r?r@s@r*rrs@z@c@2637*.,1$)59KO=||=!.=u//0 = ! = $D> =D>=!!1!12=&eELL%,,,F&GH=-.= u  (51B1BEDUDU1U+V"WW X==r,rc\rSrSrSrSrg)Phi3PreTrainedModelz0.0.5rN)r8r9r:r;_versionr>rr,r*rrsHr,rc,\rSrSrSSjrSrg)Phi3ForCausalLMNc $U(aeURR(aJURSURRS-:a USn XRR::aSn[ 5R "SUUUUUUUUS.U D6n U $)Nrr) input_idspast_key_valuesre inputs_embedsrgrIrlogits_to_keepr)r rope_scalingrC original_max_position_embeddingsrprepare_inputs_for_generation) r(rrrerrgrIrrrh past_length model_inputss r*r-Phi3ForCausalLM.prepare_inputs_for_generations$  (("dkk&R&RUV&VV(+KkkJJJ"&*,JJ  +)')%)    r,r)NNNNNTN)r8r9r:r;rr>rr,r*rrs%r,rc\rSrSrSrg)Phi3ForSequenceClassificationi#rNr8r9r:r;r>rr,r*rr#r,rc\rSrSrSrg)Phi3ForTokenClassificationi'rNrrr,r*rr'rr,r)r Phi3Modelrrr)Nr)-rtypingrrrr<torch.utils.checkpointr activationsr cache_utilsr modeling_flash_attention_utilsr modeling_utilsr processing_utilsr utilsr mistral.modeling_mistralrrrrrrrconfiguration_phi3r get_loggerr8rt_CHECKPOINT_FOR_DOC_CONFIG_FOR_DOCModulerrRrTrrrrr__all__rr,r*rs ,, ! B5&+   H %8)bii)$@H)BIIH)VF*FR0&(*=&R $D  !>  r,