o Zh@sddlmZmZmZmZddlZddlmZddlmZddl m Z m Z m Z m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-e*rddl.m/Z/ddl0m1Z1e+2e3Z4edGdddej5Z6Gdddej5Z7ddZ8dDddZ9dej:de;d ej:fd!d"Z< #dEd$ej5d%ej:d&ej:d'ej:d(eej:d)e=d*e=fd+d,Z>Gd-d.d.ej5Z?Gd/d0d0eZ@e(Gd1d2d2e#ZAGd3d4d4ej5ZBe(Gd5d6d6eAZCGd7d8d8ee'ZDe(Gd9d:d:eAeZEe(d;d<Gd=d>d>eAZFe(Gd?d@d@eAZGe(GdAdBdBeAZHgdCZIdS)F)CallableOptionalTupleUnionN)nn)ACT2FN)Cache DynamicCacheSlidingWindowCache StaticCache)GenerationMixin)use_kernel_forward_from_hub)AttentionMaskConverter)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPastQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging) Qwen3Config) BlockMask)make_flex_block_causal_maskZRMSNormcs.eZdZdfdd ZddZddZZS) Qwen3RMSNormư>cs&ttt||_||_dS)z; Qwen3RMSNorm is equivalent to T5LayerNorm N)super__init__r ParametertorchZonesweightvariance_epsilon)self hidden_sizeeps __class__W/var/www/auris/lib/python3.10/site-packages/transformers/models/qwen3/modeling_qwen3.pyr(;s  zQwen3RMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)NT)Zkeepdim) dtypetor*float32powmeanZrsqrtr,r+)r- hidden_statesZ input_dtypeZvariancer2r2r3forwardCs  zQwen3RMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler+shaper,r-r2r2r3 extra_reprJszQwen3RMSNorm.extra_repr)r&)__name__ __module__ __qualname__r(r<r@ __classcell__r2r2r0r3r%9sr%cs$eZdZfddZddZZS)Qwen3MLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFbias) r'r(configr.Zintermediate_sizerLinear gate_projup_proj down_projrZ hidden_actact_fnr-rIr0r2r3r(Os zQwen3MLP.__init__cCs$||||||}|SN)rMrNrKrL)r-xrMr2r2r3r<Ys zQwen3MLP.forward)rArBrCr(r<rDr2r2r0r3rENs  rEcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr5r4dim)r>r*cat)rQx1Zx2r2r2r3 rotate_half^srVcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezerV)qkcossin position_idsZ unsqueeze_dimZq_embedZk_embedr2r2r3apply_rotary_pos_embes  r]r;n_repreturncCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r!N)r>expandreshape)r;r^batchnum_key_value_headsslenhead_dimr2r2r3 repeat_kvs 0rfmodulequerykeyvalueattention_maskscalingdropoutc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr4rr5)rSr6)ptrainingr!)rfnum_key_value_groupsr*matmul transposer>rZ functionalZsoftmaxr8r7r6rnrq contiguous) rhrirjrkrlrmrnkwargs key_states value_states attn_weights causal_mask attn_outputr2r2r3eager_attention_forwards & r|cseZdZdZdedeffdd Z  ddejde ejejfd e ejd e e d e ej d e ed e eje eje e ejffddZZS)Qwen3Attentionz=Multi-headed attention from 'Attention Is All You Need' paperrI layer_idxcs.t||_||_t|d|j|j|_|j|j|_ |jd|_ |j |_ d|_ t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_t|j|jd|_t|j|jd|_|j|_|jjrt|jdddur|j|jjksd|_dSdS)NregTrGr/sliding_window)r'r(rIr~getattrr.Znum_attention_headsrercrrrmattention_dropoutZ is_causalrrJZattention_biasq_projk_projv_projo_projr% rms_norm_epsq_normk_normruse_sliding_windowZmax_window_layersr-rIr~r0r2r3r(s:   zQwen3Attention.__init__Nr;position_embeddingsrlpast_key_valuecache_positionrvr_cKsX|jdd}g|d|jR}||||dd} ||||dd} |||dd} |\} } t | | | | \} } |dur]| | |d}| | | |j |\} } t }|j jdkr}|j jdkrw|ddrwtd nt|j j}||| | | |f|jsd n|j|j|jd |\}}|jg|dR}||}||fS) Nr5r!r4)r[rZreagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rg)rnrmr)r>rerrviewrtrrrr]updater~r|rI_attn_implementationgetlogger warning_oncerrqrrmrrarur)r-r;rrlrrrvZ input_shapeZ hidden_shapeZ query_statesrwrxrZr[Z cache_kwargsZattention_interfacer{ryr2r2r3r<sB      zQwen3Attention.forward)NN)rArBrC__doc__r"intr(r*Tensorrrr LongTensorrrr<rDr2r2r0r3r}s(%r}cseZdZdedeffdd Z       ddejdeejd eej d ee d ee d ee d eej dee ejejfde ede ejee ejejfffddZZS)Qwen3DecoderLayerrIr~cs~t|j|_t||d|_t||_t|j|jd|_ t|j|jd|_ |j r;|j dkr=t d|j ddSdSdS)N)rIr~rflash_attention_2z=Sliding Window Attention is enabled but not implemented for `z)`; unexpected results may be encountered.)r'r(r.r} self_attnrEmlpr%rinput_layernormpost_attention_layernormrrrrrr0r2r3r(s    zQwen3DecoderLayer.__init__NFr;rlr\rr use_cacherrrvr_c Kst|} ||}|jd||||||||d| \}} | |}|} ||}||}| |}|f} |r8| | f7} | S)N)r;rlr\rrrrrr2)rrrr) r-r;rlr\rrrrrrvZresidualZself_attn_weightsoutputsr2r2r3r< s.      zQwen3DecoderLayer.forward)NNNFFNN)rArBrCr"rr(r*rrrr boolrrr FloatTensorr<rDr2r2r0r3rs<   rc@sHeZdZeZdZdZdgZdgZdZ dZ dZ dZ dZ dZdZddZdS)Qwen3PreTrainedModelmodelTrpast_key_valuescCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|tj rC|jjjd|d|j durA|jj|j  dSdSt|t rQ|jj ddSdS)Nrg)r:stdg?)rIZinitializer_range isinstancerrJr+dataZnormal_rHZzero_ Embedding padding_idxr%Zfill_)r-rhrr2r2r3 _init_weightsEs     z"Qwen3PreTrainedModel._init_weightsN)rArBrCr"Z config_classbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attn_2Z_supports_sdpaZ_supports_flex_attnZ_supports_cache_classZ_supports_quantized_cacheZ_supports_static_cacheZ_supports_attention_backendrr2r2r2r3r6s rcs8eZdZddeffdd ZeeddZZ S)Qwen3RotaryEmbeddingNrIcstt|dr|jdur|jd|jd|_nd|_|j|_|j|_||_ t |j|_ | |j |\}|_ |j d|dd|j|_dS)N rope_scaling rope_typetypedefaultinv_freqF) persistent)r'r(hasattrrrrZmax_position_embeddingsZmax_seq_len_cachedZoriginal_max_seq_lenrIrZ rope_init_fnattention_scalingZregister_bufferrZoriginal_inv_freq)r-rIdevicerr0r2r3r(Ts   zQwen3RotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t j |dd+|| dd}t j ||fdd }| |j}||j} Wdn1smwY|j|jd | j|jd fS) Nrr5r!ZmpscpuF) device_typeenabledr4rR)r6)rfloatr`r>r7rrrstrr*ZautocastrtrTrZrr[r6) r-rQr\Zinv_freq_expandedZposition_ids_expandedrZfreqsZembrZr[r2r2r3r<es0&zQwen3RotaryEmbedding.forwardrP) rArBrCr"r(r*Zno_gradrr<rDr2r2r0r3rSs rcseZdZdeffdd ZddZddZee         d!d e e j d e e j d e e j d e e d e e jde ede ede ede e j deedefddZ d"d ee j dfde j de j d e def ddZed e j dedede jde j deded e fdd ZZS)# Qwen3ModelrIcstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr2)r).0r~rIr2r3 ~sz'Qwen3Model.__init__..rrF)r'r( pad_token_idr vocab_sizerrr. embed_tokensZ ModuleListrangenum_hidden_layerslayersr%rnormr rotary_embgradient_checkpointing post_initrOr0rr3r(ws   zQwen3Model.__init__cC|jSrPrr?r2r2r3get_input_embeddingszQwen3Model.get_input_embeddingscC ||_dSrPrr-rkr2r2r3set_input_embeddings zQwen3Model.set_input_embeddingsN input_idsrlr\r inputs_embedsrroutput_hidden_statesrflash_attn_kwargsr_c  Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|du|duAr*td|jr9|jr9|r9tdd}t |t dt fsFtd|durO| |}|rX|durXt }| durt|durd|nd} tj| | |jd|jd} |dur}| d}|||| ||} |} || |}|rdnd}|rdnd}|jd|jjD]&}|r|| f7}|| f| ||||| |d | }|d} |r||df7}q|| } |r|| f7}t| |r|nd||d S) Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FzBThe `past_key_values` should be either a `Cache` object or `None`.rr!rr2)rlr\rrrrr)last_hidden_staterr; attentions)rIrrr ValueErrorrrqrrrrr rr get_seq_lengthr*aranger>rrW_update_causal_maskrrrrr)r-rrlr\rrrrrrrpast_seen_tokensrzr;rZall_hidden_statesZall_self_attnsZ decoder_layerZ layer_outputsr2r2r3r<sx             zQwen3Model.forwardFr# input_tensorc Cs|jjdkr2|dur&|dur&|dddf|dk}|r&td|dur0d|vr0|SdS|jjdkrDt|tjrBt |}|S|durL| nd}t|t }t|t } |jjdkrs|ss| ss|sst j||||jj|jdrsdS|j} t| j} |jd } | s|r|} nt|tjr|jdn|| d } |j|| | | ||jd|j|d }|jjdkr|dur|jjd vr|st || }|S) Nrr5rzYou are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Qwen3. Make sure to call `tokenizer.padding_side = 'left'` before tokenizing the input. rgZflex_attentionr)rZpast_key_values_lengthrZ is_trainingr!)sequence_length target_lengthr6r batch_sizerIr)cudaZxpuZnpu)rIrsumitemsizerrr*rr$rr r rZ_ignore_causal_mask_sdparrqr6finfominr>Zget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrrZ_unmask_unattended)r-rlrrrrZis_padding_rightrZusing_static_cacheZusing_sliding_window_cacher6 min_dtyperrrzr2r2r3rst $             zQwen3Model._update_causal_maskrrr6rcCs|dur|dkr|}|St|j} tj||f| ||jd}tj||jd|ddk} |} t | ddr\| j dur\t |t rF||kr\tj||jd|dd| j k} | | || 9}|ddddddf|ddd}|dur|}|jd|kr|ddd|f}|jd} |ddddddd| f|ddddddf|j}|d k}|ddddddd| f|| |ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. config (`Qwen3Config`): The model's configuration class past_key_values (`Cache`): The cache class that is being used currently to generate N)Z fill_valuer6rrr5r!rTr)rSr*rrfullrrraZget_text_configrrrr Z bitwise_or_r`cloner>r7Z masked_fill)rlrrr6rrrIrrzrZdiagonal_attend_maskZ text_configZsliding_attend_maskZ mask_lengthZ padding_maskr2r2r3rCs@!  $ 6  z@Qwen3Model._prepare_4d_causal_attention_mask_with_cache_position NNNNNNNNN)F)rArBrCr"r(rrrrrr*rrr rrrrrr<rr staticmethodrr6rrDr2r2r0r3rus    d  Vrc@s eZdZdS)KwargsForCausalLMN)rArBrCr2r2r2r3rs rcs eZdZdgZddiZddgdgfiZfddZdd Zd d Zd d Z ddZ ddZ ddZ e e           d%deejdeejdeejdeedeejdeejdeedeedeedeejd eeejfd!eed"efd#d$ZZS)&Qwen3ForCausalLMzlm_head.weightlm_headZ colwise_repr;logitscs@t|t||_|j|_tj|j|jdd|_| dSrF) r'r(rrrrrJr.rrrOr0r2r3r(s   zQwen3ForCausalLM.__init__cC|jjSrPrrr?r2r2r3rz%Qwen3ForCausalLM.get_input_embeddingscC ||j_dSrPrrr2r2r3r z%Qwen3ForCausalLM.set_input_embeddingscCrrPrr?r2r2r3get_output_embeddingsrz&Qwen3ForCausalLM.get_output_embeddingscCrrPr)r-Znew_embeddingsr2r2r3set_output_embeddingsrz&Qwen3ForCausalLM.set_output_embeddingscCrrPr)r-decoderr2r2r3 set_decoderrzQwen3ForCausalLM.set_decodercCrrPrr?r2r2r3 get_decoderrzQwen3ForCausalLM.get_decoderNrrrlr\rrlabelsrrrrlogits_to_keeprvr_c  Ks|dur|n|jj}| dur| n|jj} |jd|||||||| | d | } | j}t| tr4t| dn| }||dd|ddf}d}|durX|j d|||jj d| }t ||| j | j | jdS)a^ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Qwen3ForCausalLM >>> model = Qwen3ForCausalLM.from_pretrained("Qwen/Qwen3-8B") >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrlr\rrrrrr)rrrlossrrr;rr2)rIrrrrrrslicer loss_functionrrrr;r)r-rrlr\rrrrrrrrrvrr;Z slice_indicesrrr2r2r3r<s:'  zQwen3ForCausalLM.forward) NNNNNNNNNNr)rArBrCZ_tied_weights_keysZ_tp_planZ_pp_planr(rrrrrrrrrr*rrr rrrrrrrr<rDr2r2r0r3rsf       ra The Qwen3 Model transformer with a sequence classification head on top (linear layer). [`Qwen3ForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). )Z custom_introceZdZfddZddZddZee         ddee j d ee j d ee j d ee d ee j d ee j deedeedeedefddZZS)Qwen3ForSequenceClassificationcs@t||j|_t||_tj|j|jdd|_| dSrF) r'r( num_labelsrrrrJr.scorerrOr0r2r3r(s   z'Qwen3ForSequenceClassification.__init__cCrrPrr?r2r2r3rrz3Qwen3ForSequenceClassification.get_input_embeddingscCrrPrrr2r2r3rrz3Qwen3ForSequenceClassification.set_input_embeddingsNrrlr\rrrrrrr_c  Cs(|j|||||||| d} | j} || } |dur|jd} n|jd} |jjdur2| dkr2td|jjdur;d}n1|dur`||jjk| jt j }t j |jd| jt j d}|| d}n d}t |jjd| t j | | jd |f}d}|dur|j| |||jd }t||| j| j| jd S)  labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rlr\rrrrrNrr!z=Cannot handle batch sizes > 1 if no padding token is defined.r5)rr6z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r)rr pooled_logitsrIr)rrr r>rIrrr7rr*Zint32rZargmaxrrr1rArrrr;r)r-rrlr\rrrrrrZtransformer_outputsr;rrZlast_non_pad_tokenZ non_pad_maskZ token_indicesrrr2r2r3r<sL     z&Qwen3ForSequenceClassification.forwardr)rArBrCr(rrrrrr*rrr rrrr<rDr2r2r0r3r sH     r cr)Qwen3ForTokenClassificationcs|t||j|_t||_t|dddur|j}nt|dddur'|j}nd}t ||_ t |j |j|_ |dS)Nclassifier_dropouthidden_dropoutg?)r'r(r rrrrrrZDropoutrnrJr.r r)r-rIrr0r2r3r(_s    z$Qwen3ForTokenClassification.__init__cCrrPrr?r2r2r3rorz0Qwen3ForTokenClassification.get_input_embeddingscCrrPrrr2r2r3rrrz0Qwen3ForTokenClassification.set_input_embeddingsNrrlr\rrrrrrr_c  Csd|j|||||||| d} | j} || } || } d} |dur(|| ||j} t| | | j| jdS)r r N)rrr;r) rrrnr rrIrr;r)r-rrlr\rrrrrrrsequence_outputrrr2r2r3r<us,  z#Qwen3ForTokenClassification.forwardr)rArBrCr(rrrrrr*rrr rrrr<rDr2r2r0r3r]sH    rcseZdZdZfddZddZddZee         dd e e j d e e j d e e j d e e d e e jde e j de e j de ede edefddZZS)Qwen3ForQuestionAnswering transformercs2t|t||_t|jd|_|dS)Nr4) r'r(rrrrJr. qa_outputsrrOr0r2r3r(s   z"Qwen3ForQuestionAnswering.__init__cCrrPrrr?r2r2r3rrz.Qwen3ForQuestionAnswering.get_input_embeddingscCrrPrrr2r2r3rrz.Qwen3ForQuestionAnswering.set_input_embeddingsNrrlr\rrstart_positions end_positionsrrr_c  Ks|j||||||| d} | j} || } | jddd\}}|d}|d}d}|durA|durA|j||||fi| }t|||| j| j dS)N)rlr\rrrrr!r5rR)r start_logits end_logitsr;r) rrrsplitZsqueezerurrr;r)r-rrlr\rrrrrrrvrrrrrrr2r2r3r<s0 z!Qwen3ForQuestionAnswering.forwardr)rArBrCrr(rrrrrr*rrr rrrr<rDr2r2r0r3rsJ    r)rrrrr r)Nr!)rg)Jtypingrrrrr*rZ activationsrZ cache_utilsr r r r Z generationr Z integrationsrZmodeling_attn_mask_utilsrZmodeling_flash_attention_utilsrZmodeling_layersrZmodeling_outputsrrrrrZmodeling_rope_utilsrrZmodeling_utilsrrZprocessing_utilsrutilsrrrrr Zconfiguration_qwen3r"Z!torch.nn.attention.flex_attentionr#Zintegrations.flex_attentionr$Z get_loggerrArModuler%rErVr]rrrfrr|r}rrrrrrr rr__all__r2r2r2r3s~              V:"lVF >