o Zh:3@sJdZddlmZmZmZddlZddlZddlmZddlm Z ddl m Z ddl m Z dd lmZdd lmZdd lmZd d lmZmZmZmZmZmZmZddlmZeeZ dZ!dZ"Gdddej#Z$d#ddZ%Gdddej#Z&GdddeZ'GdddeZ(Gdddee(Z)GdddeZ*Gd d!d!eZ+gd"Z,dS)$zPyTorch Phi-3 model.)CallableOptionalTupleN)nn)ACT2FN)Cache)FlashAttentionKwargs)ALL_ATTENTION_FUNCTIONS)Unpack)logging)MistralDecoderLayerMistralForCausalLM MistralForSequenceClassificationMistralForTokenClassificationMistralPreTrainedModeleager_attention_forward rotate_half) Phi3Configz microsoft/Phi-3-mini-4k-instructrcs2eZdZfddZdejdejfddZZS)Phi3MLPcsPt||_tj|jd|jdd|_tj|j|jdd|_t |j |_ dS)Nr FZbias) super__init__configrLinear hidden_sizeZintermediate_size gate_up_proj down_projrZ hidden_act activation_fn)selfr __class__T/var/www/auris/lib/python3.10/site-packages/transformers/models/phi3/modular_phi3.pyr1s zPhi3MLP.__init__ hidden_statesreturncCs4||}|jddd\}}|||}||S)Nr dim)rchunkr r)r!r&Z up_statesZgater$r$r%forward9s  zPhi3MLP.forward)__name__ __module__ __qualname__rtorch FloatTensorr, __classcell__r$r$r"r%r0s rc Cs||}||}|jd}|dd|f|d|df}}|dd|f|d|df} } tj||t|||gdd} tj| |t| || gdd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. r(.Nr))Z unsqueezeshaper0catr) qkcossin position_idsZ unsqueeze_dimZ rotary_dimZq_rotZq_passZk_rotZk_passZq_embedZk_embedr$r$r%apply_rotary_pos_embBs   """"r:cseZdZdZddedeeffdd Z  ddej de ej ej fd eej d ee d eej d e ed e ej eej ee ej ffddZZS) Phi3Attentionz=Multi-headed attention from 'Attention Is All You Need' paperNr layer_idxcst||_||_t|d|j|j|_|j|j|_ |j|_|jd|_ |j |_ d|_ |j|jd|j|j}t j|j|j|jdd|_t j|j|dd|_dS)Nhead_dimgTr Fr)rrrr<getattrrnum_attention_headsr=num_key_value_headsZnum_key_value_groupsscalingattention_dropoutZ is_causalrro_projqkv_proj)r!rr<Zop_sizer"r$r%res  zPhi3Attention.__init__r&position_embeddingsattention_maskpast_key_valuecache_positionkwargsr'cKs|jdd}g|d|jR}||} |jj|j} | dd| f} | d| | |j|jf} | d| |j|jdf} | |dd} | |dd} | |dd} |\}}t| | ||\} } |dur~|||d}| | | |j |\} } t }|jj dkr|jj dkr| dd rtd nt|jj }||| | | |f|jsd n|j|jt|jd dd |\}}|jg|dR}||}||fS)Nr(.rr )r8r7rHeagerZsdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.gsliding_window)ZdropoutrArL)r3r=rDrr?r@viewZ transposer:updater<rZ_attn_implementationgetloggerZ warning_oncer ZtrainingrBrAr>Zreshape contiguousrC)r!r&rErFrGrHrIZ input_shapeZ hidden_shapeZqkvZ query_posZ query_statesZ key_statesZ value_statesr7r8Z cache_kwargsZattention_interfaceZ attn_outputZ attn_weightsr$r$r%r,tsL        zPhi3Attention.forward)N)NN)r-r.r/__doc__rrintrr0Tensorrr LongTensorr r r,r2r$r$r"r%r;bs(r;cseZdZdedeffdd Z       ddejdeejd eej d ee d ee d ee d eej dee ejejfde ede ejee ejejfffddZZS)Phi3DecoderLayerrr<csLt||||_t||d|_t||_t|j |_ t|j |_ dS)N)rr<) rrrr; self_attnrmlprZDropoutZ resid_pdropresid_attn_dropoutresid_mlp_dropout)r!rr<r"r$r%rs  zPhi3DecoderLayer.__init__NFr&rFr9rGrK use_cacherHrErIr'c Ks|} ||}|jd||||||||d| \}} | ||}|} ||}||}| ||}|f} |r>| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_value (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )r&rFr9rGrKr[rHrENr$)Zinput_layernormrWrYZpost_attention_layernormrXrZ) r!r&rFr9rGrKr[rHrErIZresidualZself_attn_weightsZoutputsr$r$r%r,s."      zPhi3DecoderLayer.forward)NNNFFNN)r-r.r/rrSrr0rTrrUrboolrr r r1r,r2r$r$r"r%rVs<    rVc@seZdZdZdS)Phi3PreTrainedModelz0.0.5N)r-r.r/_versionr$r$r$r%r]sr]c@s$eZdZ       dddZdS)Phi3ForCausalLMNTc Ksb|r|jjr|jd|jjdkr|d} | |jjkrd}tjd||||||||d| } | S)Nrr) input_idspast_key_valuesrF inputs_embedsrHr9r[logits_to_keepr$)rZ rope_scalingr3Z original_max_position_embeddingsr]prepare_inputs_for_generation) r!r`rarFrbrHr9r[rcrIZ past_lengthZ model_inputsr$r$r%rds*   z-Phi3ForCausalLM.prepare_inputs_for_generation)NNNNNTN)r-r.r/rdr$r$r$r%r_sr_c@ eZdZdS)Phi3ForSequenceClassificationNr-r.r/r$r$r$r%rf#rfc@re)Phi3ForTokenClassificationNrgr$r$r$r%ri'rhri)r]Z Phi3Modelr_rfri)Nr)-rRtypingrrrr0Ztorch.utils.checkpointrZ activationsrZ cache_utilsrZmodeling_flash_attention_utilsr Zmodeling_utilsr Zprocessing_utilsr utilsr Zmistral.modeling_mistralrrrrrrrZconfiguration_phi3rZ get_loggerr-rPZ_CHECKPOINT_FOR_DOCZ_CONFIG_FOR_DOCModulerr:r;rVr]r_rfri__all__r$r$r$r%s2       $   KI)