o Zh¦@sddlmZmZmZmZddlZddlmZddlmZddl m Z m Z m Z m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-e*rddl.m/Z/ddl0m1Z1e+2e3Z4Gdddej5Z6ddZ7dDddZ8dej9de:dej9fddZ; dEd!ej5d"ej9d#ej9d$ej9d%eej9d&eed,Gd-d.d.ej5Z?Gd/d0d0eZ@e(Gd1d2d2e#ZAGd3d4d4ej5ZBe(Gd5d6d6eAZCGd7d8d8ee'ZDe(Gd9d:d:eAeZEe(d;d<Gd=d>d>eAZFe(Gd?d@d@eAZGe(GdAdBdBeAZHgdCZIdS)F)CallableOptionalTupleUnionN)nn)ACT2FN)Cache DynamicCacheSlidingWindowCache StaticCache)GenerationMixin)use_kernel_forward_from_hub)AttentionMaskConverter)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPastQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging) Qwen2Config) BlockMask)make_flex_block_causal_maskcs$eZdZfddZddZZS)Qwen2MLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFbias) super__init__config hidden_sizeZintermediate_sizerLinear gate_projup_proj down_projrZ hidden_actact_fnselfr+ __class__W/var/www/auris/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.pyr*+s zQwen2MLP.__init__cCs$||||||}|SN)r0r1r.r/)r3xr0r6r6r7forward5s zQwen2MLP.forward)__name__ __module__ __qualname__r*r: __classcell__r6r6r4r7r%*s  r%cCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)r9x1Zx2r6r6r7 rotate_half:srGcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezerG)qkcossin position_idsZ unsqueeze_dimZq_embedZk_embedr6r6r7apply_rotary_pos_embAs  rN hidden_statesn_repreturncCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r!N)rCexpandreshape)rOrPbatchnum_key_value_headsslenhead_dimr6r6r7 repeat_kv\s 0rXmodulequerykeyvalueattention_maskscalingdropoutc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr@rr?)rBdtype)ptrainingr!)rXnum_key_value_groupsrDmatmul transposerCrZ functionalZsoftmaxfloat32torbr`rd contiguous) rZr[r\r]r^r_r`kwargs key_states value_states attn_weights causal_mask attn_outputr6r6r7eager_attention_forwardhs & rqcseZdZdZdedeffdd Z  ddejde ejejfd e ejd e e d e ej d e ed e eje eje e ejffddZZS)Qwen2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperr+ layer_idxcst||_||_t|d|j|j|_|j|j|_ |jd|_ |j |_ d|_ t j|j|j|jdd|_t j|j|j|jdd|_t j|j|j|jdd|_t j|j|j|jdd|_dS)NrWgTr'F)r)r*r+rsgetattrr,Znum_attention_headsrWrUrer_attention_dropoutZ is_causalrr-q_projk_projv_projo_projr3r+rsr4r6r7r*s   zQwen2Attention.__init__NrOposition_embeddingsr^past_key_valuecache_positionrkrQcKs~|jdd}g|d|jR}|||dd} |||dd} |||dd} |\} } t| | | | \} } |durW| | |d}|| | |j |\} } d}|j j rqt |j dddurq|j |j j krq|j j}t}|j jdkr|j jdkr|dd rtd nt|j j}||| | | |f|jsd n|j|j|d |\}}|jg|dR}||}||fS) Nr?r!r@)rLrKr}sliding_windoweagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rY)r`r_r~)rCrWrvviewrgrwrxrNupdatersr+use_sliding_windowrtZmax_window_layersr~rq_attn_implementationgetlogger warning_oncerrdrur_rSrjry)r3rOr{r^r|r}rkZ input_shapeZ hidden_shapeZ query_statesrlrmrKrLZ cache_kwargsr~Zattention_interfacerprnr6r6r7r:sN      zQwen2Attention.forward)NN)r;r<r=__doc__r"intr*rDTensorrrr LongTensorrrr:r>r6r6r4r7rrs(rrZRMSNormcs.eZdZdfdd ZddZddZZS) Qwen2RMSNormư>cs&ttt||_||_dS)z; Qwen2RMSNorm is equivalent to T5LayerNorm N)r)r*r ParameterrDZonesweightvariance_epsilon)r3r,epsr4r6r7r*s  zQwen2RMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)Nr@r?T)Zkeepdim) rbrirDrhpowmeanZrsqrtrr)r3rOZ input_dtypeZvariancer6r6r7r:s  zQwen2RMSNorm.forwardcCst|jjd|jS)Nz, eps=)tuplerrCrr3r6r6r7 extra_reprszQwen2RMSNorm.extra_repr)r)r;r<r=r*r:rr>r6r6r4r7rsrcseZdZdedeffdd Z       ddejdeejd eej d ee d ee d ee d eej dee ejejfde ede ejee ejejfffddZZS)Qwen2DecoderLayerr+rscs~t|j|_t||d|_t||_t|j|jd|_ t|j|jd|_ |j r;|j dkr=t d|j ddSdSdS)N)r+rsrflash_attention_2z=Sliding Window Attention is enabled but not implemented for `z)`; unexpected results may be encountered.)r)r*r,rr self_attnr%mlpr rms_norm_epsinput_layernormpost_attention_layernormrrrrrzr4r6r7r*s   zQwen2DecoderLayer.__init__NFrOr^rMr|r use_cacher}r{rkrQc Kst|} ||}|jd||||||||d| \}} | |}|} ||}||}| |}|f} |r8| | f7} | S)N)rOr^rMr|rrr}r{r6)rrrr) r3rOr^rMr|rrr}r{rkZresidualZself_attn_weightsoutputsr6r6r7r:s.      zQwen2DecoderLayer.forward)NNNFFNN)r;r<r=r"rr*rDrrrr boolrrr FloatTensorr:r>r6r6r4r7rs<   rc@sHeZdZeZdZdZdgZdgZdZ dZ dZ dZ dZ dZdZddZdS)Qwen2PreTrainedModelmodelTrpast_key_valuescCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|tj rC|jjjd|d|j durA|jj|j  dSdSt|t rQ|jj ddSdS)NrY)rstdg?)r+Zinitializer_range isinstancerr-rdataZnormal_r(Zzero_ Embedding padding_idxrZfill_)r3rZrr6r6r7 _init_weights*s     z"Qwen2PreTrainedModel._init_weightsN)r;r<r=r"Z config_classbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attn_2Z_supports_sdpaZ_supports_flex_attnZ_supports_cache_classZ_supports_quantized_cacheZ_supports_static_cacheZ_supports_attention_backendrr6r6r6r7rs rcs8eZdZddeffdd ZeeddZZ S)Qwen2RotaryEmbeddingNr+cstt|dr|jdur|jd|jd|_nd|_|j|_|j|_||_ t |j|_ | |j |\}|_ |j d|dd|j|_dS)N rope_scaling rope_typetypedefaultinv_freqF) persistent)r)r*hasattrrrrZmax_position_embeddingsZmax_seq_len_cachedZoriginal_max_seq_lenr+rZ rope_init_fnattention_scalingZregister_bufferrZoriginal_inv_freq)r3r+devicerr4r6r7r*9s   zQwen2RotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t j |dd+|| dd}t j ||fdd }| |j}||j} Wdn1smwY|j|jd | j|jd fS) Nrr?r!ZmpscpuF) device_typeenabledr@rA)rb)rfloatrRrCrirrrstrrDZautocastrgrErKrrLrb) r3r9rMZinv_freq_expandedZposition_ids_expandedrZfreqsZembrKrLr6r6r7r:Js0&zQwen2RotaryEmbedding.forwardr8) r;r<r=r"r*rDZno_gradrr:r>r6r6r4r7r8s rcseZdZdeffdd ZddZddZee         d!d e e j d e e j d e e j d e e d e e jde ede ede ede e j deedefddZ d"d ee j dfde j de j d e def ddZed e j dedede jde j deded e fdd ZZS)# Qwen2Modelr+cstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr6)r).0rsr+r6r7 csz'Qwen2Model.__init__..rrF)r)r* pad_token_idr vocab_sizerrr, embed_tokensZ ModuleListrangenum_hidden_layerslayersrrnormr rotary_embgradient_checkpointing post_initr2r4rr7r*\s   zQwen2Model.__init__cC|jSr8rrr6r6r7get_input_embeddingslzQwen2Model.get_input_embeddingscC ||_dSr8rr3r]r6r6r7set_input_embeddingso zQwen2Model.set_input_embeddingsN input_idsr^rMr inputs_embedsrroutput_hidden_statesr}flash_attn_kwargsrQc  Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|du|duAr*td|jr9|jr9|r9tdd}t |t dt fsFtd|durO| |}|rX|durXt }| durt|durd|nd} tj| | |jd|jd} |dur}| d}|||| ||} |} || |}|rdnd}|rdnd}|jd|jjD]&}|r|| f7}|| f| ||||| |d | }|d} |r||df7}q|| } |r|| f7}t| |r|nd||d S) Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FzBThe `past_key_values` should be either a `Cache` object or `None`.rr!rr6)r^rMr|rrr}r{)last_hidden_staterrO attentions)r+rrr ValueErrorrrdrrrrr rr get_seq_lengthrDarangerCrrH_update_causal_maskrrrrr)r3rr^rMrrrrrr}rpast_seen_tokensrorOr{Zall_hidden_statesZall_self_attnsZ decoder_layerZ layer_outputsr6r6r7r:rsx             zQwen2Model.forwardFr# input_tensorc Cs|jjdkr2|dur&|dur&|dddf|dk}|r&td|dur0d|vr0|SdS|jjdkrDt|tjrBt |}|S|durL| nd}t|t }t|t } |jjdkrs|ss| ss|sst j||||jj|jdrsdS|j} t| j} |jd } | s|r|} nt|tjr|jdn|| d } |j|| | | ||jd|j|d }|jjdkr|dur|jjd vr|st || }|S) Nrr?rzYou are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Qwen2. Make sure to call `tokenizer.padding_side = 'left'` before tokenizing the input. rYZflex_attentionr)rZpast_key_values_lengthr~Z is_trainingr!)sequence_length target_lengthrbr} batch_sizer+r)cudaZxpuZnpu)r+rsumitemsizerrrDrr$rr r rZ_ignore_causal_mask_sdpar~rdrbfinfominrCZget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrrZ_unmask_unattended)r3r^rr}rrZis_padding_rightrZusing_static_cacheZusing_sliding_window_cacherb min_dtyperrror6r6r7rst $             zQwen2Model._update_causal_maskrrrbrcCs|dur|dkr|}|St|j} tj||f| ||jd}tj||jd|ddk} |} t | ddr\| j dur\t |t rF||kr\tj||jd|dd| j k} | | || 9}|ddddddf|ddd}|dur|}|jd|kr|ddd|f}|jd} |ddddddd| f|ddddddf|j}|d k}|ddddddd| f|| |ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. config (`Qwen2Config`): The model's configuration class past_key_values (`Cache`): The cache class that is being used currently to generate N)Z fill_valuerbrrr?r!rTr)rBrDrrfullrrrSZget_text_configrtr~rr Z bitwise_or_rRclonerCriZ masked_fill)r^rrrbr}rr+rrorZdiagonal_attend_maskZ text_configZsliding_attend_maskZ mask_lengthZ padding_maskr6r6r7r(s@!  $ 6  z@Qwen2Model._prepare_4d_causal_attention_mask_with_cache_position NNNNNNNNN)F)r;r<r=r"r*rrrrrrDrrr rrrrrr:rr staticmethodrrbrr>r6r6r4r7rZs    d  Vrc@s eZdZdS)KwargsForCausalLMN)r;r<r=r6r6r6r7rns rcs eZdZdgZddiZddgdgfiZfddZdd Zd d Zd d Z ddZ ddZ ddZ e e           d%deejdeejdeejdeedeejdeejdeedeedeedeejd eeejfd!eed"efd#d$ZZS)&Qwen2ForCausalLMzlm_head.weightlm_headZ colwise_reprOlogitscs@t|t||_|j|_tj|j|jdd|_| dSr&) r)r*rrrrr-r,rrr2r4r6r7r*ws   zQwen2ForCausalLM.__init__cC|jjSr8rrrr6r6r7rz%Qwen2ForCausalLM.get_input_embeddingscC ||j_dSr8rrr6r6r7r z%Qwen2ForCausalLM.set_input_embeddingscCrr8rrr6r6r7get_output_embeddingsrz&Qwen2ForCausalLM.get_output_embeddingscCrr8r)r3Znew_embeddingsr6r6r7set_output_embeddingsrz&Qwen2ForCausalLM.set_output_embeddingscCrr8r)r3decoderr6r6r7 set_decoderrzQwen2ForCausalLM.set_decodercCrr8rrr6r6r7 get_decoderrzQwen2ForCausalLM.get_decoderNrrr^rMrrlabelsrrrr}logits_to_keeprkrQc  Ks|dur|n|jj}| dur| n|jj} |jd|||||||| | d | } | j}t| tr4t| dn| }||dd|ddf}d}|durX|j d|||jj d| }t ||| j | j | jdS)at labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Qwen2ForCausalLM >>> model = Qwen2ForCausalLM.from_pretrained("meta-qwen2/Qwen2-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-qwen2/Qwen2-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rr^rMrrrrrr})rrrlossrrrOrr6)r+rrrrrrslicer loss_functionrrrrOr)r3rr^rMrrrrrrr}rrkrrOZ slice_indicesrrr6r6r7r:s:'  zQwen2ForCausalLM.forward) NNNNNNNNNNr)r;r<r=Z_tied_weights_keysZ_tp_planZ_pp_planr*rrrrrrrrrrDrrr rrrrrrrr:r>r6r6r4r7rqsf       ra The Qwen2 Model transformer with a sequence classification head on top (linear layer). [`Qwen2ForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). )Z custom_introceZdZfddZddZddZee         ddee j d ee j d ee j d ee d ee j d ee j deedeedeedefddZZS)Qwen2ForSequenceClassificationcs@t||j|_t||_tj|j|jdd|_| dSr&) r)r* num_labelsrrrr-r,scorerr2r4r6r7r*s   z'Qwen2ForSequenceClassification.__init__cCrr8rrr6r6r7rrz3Qwen2ForSequenceClassification.get_input_embeddingscCrr8rrr6r6r7rrz3Qwen2ForSequenceClassification.set_input_embeddingsNrr^rMrrrrrrrQc  Cs(|j|||||||| d} | j} || } |dur|jd} n|jd} |jjdur2| dkr2td|jjdur;d}n1|dur`||jjk| jt j }t j |jd| jt j d}|| d}n d}t |jjd| t j | | jd |f}d}|dur|j| |||jd }t||| j| j| jd S)  labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). r^rMrrrrrNrr!z=Cannot handle batch sizes > 1 if no padding token is defined.r?)rrbz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r)rr pooled_logitsr+r)rrr rCr+rrrirrDZint32rZargmaxrrr5r;rrrrOr)r3rr^rMrrrrrrZtransformer_outputsrOrrZlast_non_pad_tokenZ non_pad_maskZ token_indicesr rr6r6r7r:sL     z&Qwen2ForSequenceClassification.forwardr)r;r<r=r*rrrrrrDrrr rrrr:r>r6r6r4r7rsH     rcr)Qwen2ForTokenClassificationcs|t||j|_t||_t|dddur|j}nt|dddur'|j}nd}t ||_ t |j |j|_ |dS)Nclassifier_dropouthidden_dropoutg?)r)r*rrrrtrrrZDropoutr`r-r,r r)r3r+rr4r6r7r*Ds    z$Qwen2ForTokenClassification.__init__cCrr8rrr6r6r7rTrz0Qwen2ForTokenClassification.get_input_embeddingscCrr8rrr6r6r7rWrz0Qwen2ForTokenClassification.set_input_embeddingsNrr^rMrrrrrrrQc  Csd|j|||||||| d} | j} || } || } d} |dur(|| ||j} t| | | j| jdS)r r N)rrrOr) rrr`r rr+rrOr)r3rr^rMrrrrrrrsequence_outputrrr6r6r7r:Zs,  z#Qwen2ForTokenClassification.forwardr)r;r<r=r*rrrrrrDrrr rrrr:r>r6r6r4r7r BsH    r cseZdZdZfddZddZddZee         dd e e j d e e j d e e j d e e d e e jde e j de e j de ede edefddZZS)Qwen2ForQuestionAnswering transformercs2t|t||_t|jd|_|dS)Nr@) r)r*rrrr-r, qa_outputsrr2r4r6r7r*s   z"Qwen2ForQuestionAnswering.__init__cCrr8rrrr6r6r7rrz.Qwen2ForQuestionAnswering.get_input_embeddingscCrr8rrr6r6r7rrz.Qwen2ForQuestionAnswering.set_input_embeddingsNrr^rMrrstart_positions end_positionsrrrQc  Ks|j||||||| d} | j} || } | jddd\}}|d}|d}d}|durA|durA|j||||fi| }t|||| j| j dS)N)r^rMrrrrr!r?rA)r start_logits end_logitsrOr) rrrsplitZsqueezerjrrrOr)r3rr^rMrrrrrrrkrrrrrrr6r6r7r:s0 z!Qwen2ForQuestionAnswering.forwardr)r;r<r=rr*rrrrrrDrrr rrrr:r>r6r6r4r7rsJ    r)rrrrr r)Nr!)rY)JtypingrrrrrDrZ activationsrZ cache_utilsr r r r Z generationr Z integrationsrZmodeling_attn_mask_utilsrZmodeling_flash_attention_utilsrZmodeling_layersrZmodeling_outputsrrrrrZmodeling_rope_utilsrrZmodeling_utilsrrZprocessing_utilsrutilsrrrrr Zconfiguration_qwen2r"Z!torch.nn.attention.flex_attentionr#Zintegrations.flex_attentionr$Z get_loggerr;rModuler%rGrNrrrXrrqrrrrrrrrrrr r__all__r6r6r6r7s~              L8"lVF >