fTh¦SSKJrJrJrJr SSKrSSKJr SSKJr SSK J r J r J r J r SSKJr SSKJr SS KJr SS KJr SS KJr SS KJrJrJrJrJr SS KJrJ r SSK!J"r"J#r# SSK$J%r% SSK&J'r'J(r(J)r)J*r*J+r+ SSK,J-r- \*"5(a SSK.J/r/ SSK0J1r1 \+Rd"\35r4"SS\Rj5r6Sr7S?Sjr8S\RrS\:S\Rr4Sjr;S@S\RjS\RrS\RrS \RrS!\\RrS"\\"S'5"S(S)\Rj55r?"S*S+\5r@\("S,S-\#55rA"S.S/\Rj5rB\("S0S1\A55rC"S2S3\\'5rD\("S4S5\A\55rE\("S6S79"S8S9\A55rF\("S:S;\A55rG\("S<S=\A55rH/S>QrIg)A)CallableOptionalTupleUnionN)nn)ACT2FN)Cache DynamicCacheSlidingWindowCache StaticCache)GenerationMixin)use_kernel_forward_from_hub)AttentionMaskConverter)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPastQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging) Qwen2Config) BlockMask)make_flex_block_causal_maskc.^\rSrSrU4SjrSrSrU=r$)Qwen2MLP*c>[TU]5 XlURUlURUl[ R "URURSS9Ul[ R "URURSS9Ul[ R "URURSS9Ul [URUl gNFbias) super__init__config hidden_sizeintermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fnselfr/ __class__s `/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/qwen2/modeling_qwen2.pyr.Qwen2MLP.__init__+s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ cURURURU55URU5-5nU$N)r5r7r3r4)r9xr5s r;forwardQwen2MLP.forward5s6NN4;;t~~a/@#ADLLQRO#ST r=)r7r/r5r3r0r1r4)__name__ __module__ __qualname____firstlineno__r.rA__static_attributes__ __classcell__r:s@r;r'r'*s0r=r'cUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)r@x1x2s r; rotate_halfrT:sZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r=cURU5nURU5nX-[U5U--nX-[U5U--nXg4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezerT)qkcossin position_ids unsqueeze_dimq_embedk_embeds r;apply_rotary_pos_embr_AsS( -- &C -- &Cw;q>C/0Gw;q>C/0G  r= hidden_statesn_repreturncURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r"N)rOexpandreshape)r`rabatchnum_key_value_headsslenhead_dims r; repeat_kvrj\s_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr=modulequerykeyvalueattention_maskscalingdropoutc @[X R5n[X0R5n [R"XR SS55U-n Ub"USS2SS2SS2SUR S24n X-n [ RRU S[RS9RUR5n [ RRXURS9n [R"X5n U R SS5R5n X4$)NrLrrK)rNdtype)ptrainingr")rjnum_key_value_groupsrPmatmul transposerOr functionalsoftmaxfloat32tortrqrv contiguous) rkrlrmrnrorprqkwargs key_states value_states attn_weights causal_mask attn_outputs r;eager_attention_forwardrhs3 ; ;[TU]5 XlX l[ USUR UR -5UlUR UR-Ul URS-Ul URUl SUl [R"UR UR UR-SS9Ul[R"UR URUR-SS9Ul[R"UR URUR-SS9Ul[R"UR UR-UR SS9Ulg)NrigTr+F)r-r.r/rgetattrr0num_attention_headsrirgrwrpattention_dropout is_causalrr2q_projk_projv_projo_projr9r/rr:s r;r.Qwen2Attention.__init__s(  " F4F4F&JdJd4de $*$>$>&B\B\$\!}}d* !'!9!9ii 2 2F4N4NQUQ^Q^4^eij ii 2 2F4N4NQUQ^Q^4^eij ii 2 2F4N4NQUQ^Q^4^eij ii : :T]] JFL^L^ejk r=r`position_embeddingsropast_key_valuecache_positionrrbc URSSn/UQSPURP7nURU5RU5R SS5n UR U5RU5R SS5n UR U5RU5R SS5n Uup[XX5upUb$XUS.nURXURU5upSnURR(aR[URSS5b:URURR:aURRn[nURR S:wadURR S:Xa-UR#SS 5(a[$R'S 5 O[(URR nU"UU U U U4UR*(dS O UR,UR.US .UD6unnUR0"/UQSP76R35nUR5U5nUU4$) NrKr"rL)rZrYrsliding_windoweagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rqrpr)rOrirviewryrrr_updaterr/use_sliding_windowrmax_window_layersrr_attn_implementationgetlogger warning_oncerrvrrprer~r)r9r`rrorrr input_shape hidden_shape query_statesrrrYrZ cache_kwargsrattention_interfacerrs r;rAQwen2Attention.forwards!$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &#7 RU#[  %#&nUL'5'<'d>d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL) %  % ! \"));;;;FFH kk+. L((r=) rr/rirrrrwrrrpr)NN)rCrDrErF__doc__r#intr.rPTensorrrr LongTensorrrrArGrHrIs@r;rrsG l{ ls l&+/59 8)||8)#5<<#=>8)!. 8) ! 8) !!1!12 8)-.8) u||Xell3XeELL>Q5RR S8)8)r=rRMSNormc8^\rSrSrSU4SjjrSrSrSrU=r$) Qwen2RMSNormc>[TU]5 [R"[R "U55UlX lg)z+ Qwen2RMSNorm is equivalent to T5LayerNorm N)r-r.r ParameterrPonesweightvariance_epsilon)r9r0epsr:s r;r.Qwen2RMSNorm.__init__s/ ll5::k#:; #r=cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)NrLrKT)keepdim) rtr}rPr|powmeanrsqrtrr)r9r` input_dtypevariances r;rAQwen2RMSNorm.forwardsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r=c^[URR5SUR3$)Nz, eps=)tuplerrOrr9s r; extra_reprQwen2RMSNorm.extra_reprs* ))*+6$2G2G1HIIr=)rr)gư>) rCrDrErFr.rArrGrHrIs@r;rrs$;JJr=rc^\rSrSrS\S\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \ S \ \RS \ \ \R\R4S \\S\ \R \ \ \R \R 444SjjrSrU=r$)Qwen2DecoderLayerr/rc>[TU]5 URUl[XS9Ul[ U5Ul[URURS9Ul [URURS9Ul UR(a5URS:wa$[RSURS35 ggg)N)r/rrflash_attention_2z=Sliding Window Attention is enabled but not implemented for `z)`; unexpected results may be encountered.)r-r.r0r self_attnr'mlpr rms_norm_epsinput_layernormpost_attention_layernormrrrrrs r;r.Qwen2DecoderLayer.__init__s !--'vKF#+F,>,>FDWDWX(4V5G5GVM`M`(a%  $ $)D)DH[)[   OPVPkPkOlm99 *\ $r=r`ror[rr use_cacherrrrbc Un URU5nUR"SUUUUUUUUS.U D6upX-nUn URU5nURU5nX-nU4n U(aX4- n U $)N)r`ror[rrrrr)rrrr) r9r`ror[rrrrrrresidualself_attn_weightsoutputss r;rAQwen2DecoderLayer.forwards!,,]; ,0>> , ')%)/) 3 ,  , ( !0 !55mD /  0  "  + +Gr=)r0rrrr)NNNFFNN)rCrDrErFr#rr.rPrrrr boolrrr FloatTensorrArGrHrIs@r;rrs { s  2637*.,1$)59KO'||'!.'u//0 ' ! ' $D> 'D>'!!1!12'&eELL%,,,F&GH'-.' u  (51B1BEDUDU1U+V"WW X''r=rcN\rSrSr\rSrSrS/rS/r Sr Sr Sr Sr SrSrSrSrSrg) Qwen2PreTrainedModelimodelTrpast_key_valuescURRn[U[R5(aWUR R RSUS9 URb%URR R5 gg[U[R5(adUR R RSUS9 URb2UR R URR5 gg[U[5(a&UR R RS5 gg)Nr)rstdg?)r/initializer_range isinstancerr2rdatanormal_r,zero_ Embedding padding_idxrfill_)r9rkrs r; _init_weights"Qwen2PreTrainedModel._init_weights*skk++ fbii ( ( MM   & &CS & 9{{&   &&('  - - MM   & &CS & 9!!- ""6#5#56<<>.  - - MM   $ $S ).r=rN)rCrDrErFr# config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_flex_attn_supports_cache_class_supports_quantized_cache_supports_static_cache_supports_attention_backendrrGrr=r;rrsSL&*#,-#4"5!N  $!"& *r=rcl^\rSrSrSS\4U4Sjjjr\R"5\S55r Sr U=r $)Qwen2RotaryEmbeddingi8r/c>[TU]5 [US5(aHURb;URR SURR S55UlOSUlUR UlUR UlXl [UR Ul URURU5uo0l URSUSS9 URUlg)N rope_scaling rope_typetypedefaultinv_freqF) persistent)r-r.hasattrrrrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr/r rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)r9r/devicerr:s r;r.Qwen2RotaryEmbedding.__init__9s  6> * *v/B/B/N#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk ^^ U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs $BF  F.)rr/rr rrrr?) rCrDrErFr#r.rPno_gradrrArGrHrIs@r;rr8s6/{//" ]]_ < [TU]U5 URUlURUl[ R "URURUR5Ul[ R"[UR5Vs/sHn[X5PM sn5Ul [URURS9Ul[#US9UlSUlUR)5 gs snf)Nr)r/F)r-r. pad_token_idr vocab_sizerrr0 embed_tokens ModuleListrangenum_hidden_layersrlayersrrnormr rotary_embgradient_checkpointing post_initrs r;r.Qwen2Model.__init__\s  !.. ++LL):):F??ab b   --i8M 0*nO  !CRC^==?de "\\ ]5H5H5K"KTaThThN  )33A6L.. >L] & #oomJ#7BD0d![[)H4;;+H+HIM#!%55!) *)."3#-$7 $ M*!,M  =#3"55'J* -0  !1 1 &+/8Od+%   r=r$ input_tensorc URRS:Xa]UbNUbKUSS2S4R5R5UR 5S:gnU(a [ S5eUbSU;aU$gURRS:Xa,[ U[R5(a [U5nU$UbUR5OSn[ U[5n[ U[5n URRS:XaQU(dJU (dCU(d<[R"UUUURRUR S9(agUR"n [R$"U 5R&n UR(S n U (dU(aUR+5n O5[ U[R5(aUR(SOX|-S -n UR-UU U U UUR(SURUS 9nURRS:Xa:Ub7UR.R0S ;aU(d[R2"X5nU$) NrrKrzYou are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Qwen2. Make sure to call `tokenizer.padding_side = 'left'` before tokenizing the input. rflex_attentionr)r2past_key_values_lengthr is_trainingr")sequence_length target_lengthrtr batch_sizer/r)cudaxpunpu)r/rsumitemsizer9rrPrr%r:r r r_ignore_causal_mask_sdparrvrtfinfominrOget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionr r_unmask_unattended)r9rorCrrris_padding_rightr=using_static_cacheusing_sliding_window_cachert min_dtyperHrIrs r;r<Qwen2Model._update_causal_masks; ;; + +/B B)o.I#1!R%#8#<#<#>#C#C#EIZIZI\]^I_#_ #$a )c^.C%% ;; + +/? ?.%,,77!<^!L! ! @O?Z?99;`a'E%/AS%T" KK , , 6'+E%%>>*'7#{{99 MM ""KK&** &,,Q/ %);+??AM nell;;$$R(%7!; PP +')#))!,;;+Q   KK , , 6*%%**.DD% 1CCK[Kr=rHrIrtrJcUbUR5S:XaUnU$[R"U5Rn [R"X4XUR S9n[R "X$R S9URSS5:n UR5n [U SS5(avU Rbi[U[5(aX:aO[R "X$R S9URSS5U R- :*n U RU 5 X-nUSSSS2SS24RUSSS5nUbUR5nUR SU:a USS2SU24nUR Sn USS2SS2SS2SU 24USS2SSSS24R#UR 5-nUS :HnUSS2SS2SS2SU 24R%X5USS2SS2SS2SU 24'U$) aQ Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. config (`Qwen2Config`): The model's configuration class past_key_values (`Cache`): The cache class that is being used currently to generate N) fill_valuertr r6rKr"rTr)rNrPrRrSfullr r;reget_text_configrrrr bitwise_or_rdclonerOr} masked_fill)rorHrIrtrrJr/rrrZdiagonal_attend_mask text_configsliding_attend_mask mask_length padding_masks r;rU@Qwen2Model._prepare_4d_causal_attention_mask_with_cache_position(sB  %.*<*<*>!*C(K@= E*..I** 0Y\j\q\qK$)<< F[F[#\_m_u_uA`$ !002K{$8$??KD^D^Dj"/3EFF/Ji*/,,}MbMb*c&..r15 8R8RR+')445HI  /K%dD!Q&67>>z1bRTUK))//1 !''+m;%3A~ ~4E%FN,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12r=)rr&r#r$rr%r NNNNNNNNN)F)rCrDrErFr#r.r+r/rrrrPrrr rrrrrrArr< staticmethodrrtrUrGrHrIs@r;rrZs{ !"151537+/59$(,0/359\ E,,-\ !.\ u//0 \ "% \   1 12 \ D>\ $D>\ 'tn\ !!1!12\ $$89\  !\ \ H#( TellK78TllT T  T  TlB BBB{{ B  B  BBBBr=rc\rSrSrSrg)KwargsForCausalLMinrN)rCrDrErFrGrr=r;rmrmns3r=rmc^\rSrSrS/rSS0rSS/S/40rU4SjrSrS r S r S r S r S r \\SS\\R$S\\R&S\\R$S\\S\\R*S\\R$S\\S\\S\\S\\R$S\\\R&4S\\S\4Sjj55rSrU=r$)Qwen2ForCausalLMiqzlm_head.weightlm_head colwise_repr`logitsc>[TU]U5 [U5UlURUl[ R "URURSS9UlUR5 gr*) r-r.rrrrr2r0rpr'r8s r;r.Qwen2ForCausalLM.__init__wsU  '  ++yy!3!3V5F5FUS  r=c.URR$r?rrrs r;r+%Qwen2ForCausalLM.get_input_embeddingszz&&&r=c$XRlgr?rvr.s r;r/%Qwen2ForCausalLM.set_input_embeddings "' r=cUR$r?rprs r;get_output_embeddings&Qwen2ForCausalLM.get_output_embeddingss ||r=cXlgr?r})r9new_embeddingss r;set_output_embeddings&Qwen2ForCausalLM.set_output_embeddingss% r=cXlgr?r)r9decoders r; set_decoderQwen2ForCausalLM.set_decoders r=cUR$r?rrs r; get_decoderQwen2ForCausalLM.get_decoders zzr=r1ror[rr2labelsrrr3rlogits_to_keeprrbc UbUOURRnU bU OURRn UR"SUUUUUUUU U S. U D6n U Rn[ U [ 5(a [U *S5OU nURUSS2USS245nSnUb)UR"SUX`RRS.U D6n[UUU RU RU RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Qwen2ForCausalLM >>> model = Qwen2ForCausalLM.from_pretrained("meta-qwen2/Qwen2-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-qwen2/Qwen2-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) r1ror[rr2rrr3r)rrrrlossrrrr`r8r)r/rr3rr7rrslicerp loss_functionrrrr`r8)r9r1ror[rr2rrrr3rrrrr` slice_indicesrrrs r;rAQwen2ForCausalLM.forwardsN2C1N-TXT_T_TqTq$8$D $++JjJj  ,0:: , )%+'/!5) ,  ,  11 8B>SV8W8W~ot4]k mA}a,?@A  %%pVF{{OeOepiopD%#33!//))   r=)rprr) NNNNNNNNNNr)rCrDrErF_tied_weights_keys_tp_plan_pp_planr.r+r/r~rrrrrrrPrrr rrrrrrmrrArGrHrIs@r;roroqs*+=)H_-z:;H'(&151537+/59-1$(,0/35934G E,,-G !.G u//0 G "% G   1 12 G ))*G D>G $D>G 'tnG !!1!12G c5<</0G *+G  G G r=roa The Qwen2 Model transformer with a sequence classification head on top (linear layer). [`Qwen2ForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introc*^\rSrSrU4SjrSrSr\\SS\ \ RS\ \ RS\ \ RS\ \ S \ \ RS \ \ RS \ \S \ \S \ \S\4Sjj55rSrU=r$)Qwen2ForSequenceClassificationic>[TU]U5 URUl[U5Ul[ R "URURSS9UlUR5 gr*) r-r. num_labelsrrrr2r0scorer'r8s r;r.'Qwen2ForSequenceClassification.__init__sS   ++' YYv114??O  r=c.URR$r?rvrs r;r+3Qwen2ForSequenceClassification.get_input_embeddingsrxr=c$XRlgr?rvr.s r;r/3Qwen2ForSequenceClassification.set_input_embeddingsr{r=r1ror[rr2rrrr3rbc URUUUUUUUU S9n U Rn URU 5n UbURSn OURSn URR cU S:wa [ S5eURR cSnOUbXRR :gRU R[R5n[R"URSU R[RS9nUU-RS5nO.Sn[RURR S35 U [R"XRS 9U4nSnUbUR#XUURS 9n[%UUU R&U R(U R*S 9$) e labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). ror[rr2rrr3Nrr"z=Cannot handle batch sizes > 1 if no padding token is defined.rK)r rtz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r6)rrr pooled_logitsr/r)rr7rrOr/rr9r}r rPint32r;argmaxrrr:rCrrrr`r8)r9r1ror[rr2rrrr3transformer_outputsr`rrrJlast_non_pad_token non_pad_mask token_indicesrrs r;rA&Qwen2ForSequenceClassification.forwards*8!F!Fr!J !#    >>**+,ZZ  u||J}}MOaab   %%VR_hlhshs%tD/ /??-;;*55   r=)rrrrj)rCrDrErFr.r+r/rrrrPrrr rrrrArGrHrIs@r;rrs'(151537+/59-1$(,0/3A E,,-A !.A u//0 A "% A   1 12 A ))*A D>A $D>A 'tnA  *A A r=rc*^\rSrSrU4SjrSrSr\\SS\ \ RS\ \ RS\ \ RS\ \ S \ \ RS \ \ RS \ \S \ \S \ \S\4Sjj55rSrU=r$)Qwen2ForTokenClassificationiBc>[TU]U5 URUl[U5Ul[ USS5b UR nO[ USS5b URnOSn[R"U5Ul [R"URUR5Ul UR5 g)Nclassifier_dropouthidden_dropoutg?)r-r.rrrrrrrDropoutrqr2r0rr')r9r/rr:s r;r.$Qwen2ForTokenClassification.__init__Ds   ++' 6/ 6 B!'!:!:  V-t 4 @!'!6!6 !$ zz"45 YYv1163D3DE  r=c.URR$r?rvrs r;r+0Qwen2ForTokenClassification.get_input_embeddingsTrxr=c$XRlgr?rvr.s r;r/0Qwen2ForTokenClassification.set_input_embeddingsWr{r=r1ror[rr2rrrr3rbc  URUUUUUUUU S9n U Rn URU 5n URU 5n Sn UbUR XUR 5n [ U U U RU RS9$)rrN)rrrr`r8) rr7rqrrr/rr`r8)r9r1ror[rr2rrrr3rsequence_outputrrrs r;rA#Qwen2ForTokenClassification.forwardZs*,0:: )%+'/!5,6 , "33,,7O,  %%fdkkBD$!//))   r=)rqrrrrj)rCrDrErFr.r+r/rrrrPrrr rrrrArGrHrIs@r;rrBs '(151537+/59-1$(,0/3* E,,-* !.* u//0 * "% *   1 12 * ))** D>* $D>* 'tn*  * * r=rcB^\rSrSrSrU4SjrSrSr\\ SS\ \ RS\ \ RS\ \ RS \ \S \ \ RS \ \ RS \ \ RS \ \S\ \S\4Sjj55rSrU=r$)Qwen2ForQuestionAnsweringi transformerc>[TU]U5 [U5Ul[R "UR S5UlUR5 g)NrL) r-r.rrrr2r0 qa_outputsr'r8s r;r."Qwen2ForQuestionAnswering.__init__sA  %f-))F$6$6: r=c.URR$r?rrrs r;r+.Qwen2ForQuestionAnswering.get_input_embeddingss,,,r=c$XRlgr?rr.s r;r/.Qwen2ForQuestionAnswering.set_input_embeddingss(-%r=r1ror[rr2start_positions end_positionsrr3rbc URUUUUUUU S9n U Rn URU 5n U RSSS9upUR S5R 5nUR S5R 5nSnUbUbUR "XXg40U D6n[UUUU RU RS9$)N)ror[rr2rr3r"rKrM)r start_logits end_logitsr`r8) rr7rsplitsqueezer~rrr`r8)r9r1ror[rr2rrrr3rrrrrrrrs r;rA!Qwen2ForQuestionAnswering.forwards,0+;+; )%+'/!5,<, "331#)<<r<#: #++B/::< ''+668   &=+D%%libhiD+%!!//))   r=)rrrj)rCrDrErFrr.r+r/rrrrPrrr rrrrArGrHrIs@r;rrs%-.151537+/596:48,0/3( E,,-( !.( u//0 ( "% (   1 12 ( "%"2"23(   0 01( $D>( 'tn(  &( ( r=r)rrrorrr)Nr")r)JtypingrrrrrPr activationsr cache_utilsr r r r generationr integrationsrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrr r!configuration_qwen2r#!torch.nn.attention.flex_attentionr$integrations.flex_attentionr% get_loggerrCrModuler'rTr_rrrjrrrrrrrrrmrorrr__all__rr=r;rsC43 !OO)7>B9LF&hh, !!;J   H % ryy  (6 UU\\ U# U%,, U&% II% <<% % << % U\\* %  %%4I)RYYI)XY'J299J(J(525p*?**8<299i +_i i X  S %9S  S lC "6C C L; 4; ; | r=