fTh SrSSKrSSKJrJrJrJr SSKrSSKrSSKJ r SSK J r J r J r Jr SSK Jr SSKJr SS KJrJrJr SS KJr SS KJr SS KJrJr SS KJrJ r J!r!J"r"J#r# SSK$J%r%J&r& SSK'J(r( SSK)J*r*J+r+ SSK,J-r- \(aSSK.J/r/ \"5(aSSKJ0r0 \+Rb"\25r3"SS\ Rh5r5Sr6S>Sjr7"SS\ Rp5r9S\RtS\;S\RxS\Rt4Sjr=S \RtS!\RtS"\>S#\?S\Rt4 S$jr@"S%S&\ Rp5rA"S'S(\A5rB"S)S*\ Rp5rC\A\A\BS+.rD"S,S-\ Rp5rE\*"S.S/\(55rF\*"S0S1\F55rG\*"S2S39"S4S5\F\55rH\*"S6S39"S7S8\F55rI\*"S9S:\F55rJ\*"S;S<\F55rK/S=QrLg)?zPyTorch Falcon model.N) TYPE_CHECKINGOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLoss LayerNormMSELoss) functional)get_activation)Cache DynamicCache StaticCache)GenerationMixin)AttentionMaskConverter)!flash_attn_supports_top_left_maskis_flash_attn_available))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)PreTrainedModel)auto_docstringlogging) FalconConfig)PretrainedConfig)_flash_attention_forwardcN\rSrSrS\R S\R 4SjrSrg) FalconLinear<inputreturnclXRR-nURcU$X R-$N)weightTbias)selfr' hidden_statess b/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/falcon/modeling_falcon.pyforwardFalconLinear.forward=s/ - 99  yy((N)__name__ __module__ __qualname____firstlineno__torchTensorr1__static_attributes__r4r3r0r%r%<s)U\\)ell)r3r%cUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..Ndim)shaper9cat)xx1x2s r0 rotate_halfrFEsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r3cURU5nURU5nX-[U5U--nX-[U5U--nXg4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezerF)qkcossin position_ids unsqueeze_dimq_embedk_embeds r0apply_rotary_pos_embrQMsS( -- &C -- &Cw;q>C/0Gw;q>C/0G  r3cl^\rSrSrSS\4U4Sjjjr\R"5\S55r Sr U=r $)FalconRotaryEmbeddingiconfigc>[TU]5 [US5(aHURb;URR SURR S55UlOSUlUR UlUR UlXl [UR Ul URURU5uo0l URSUSS9 URUlg)N rope_scaling rope_typetypedefaultinv_freqF) persistent)super__init__hasattrrWgetrXmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrUr rope_init_fnattention_scalingregister_bufferr[original_inv_freq)r.rUdevicer[ __class__s r0r^FalconRotaryEmbedding.__init__js  6> * *v/B/B/N#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+r?dtype)r[floatexpandrAtorh isinstancerYstrr9autocast transposerBrKrerLrq) r.rCrMinv_freq_expandedposition_ids_expandedrnfreqsembrKrLs r0r1FalconRotaryEmbedding.forward{sR!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk ^^ U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs $BF  F.)rerUrbrgrcrdrXr*) r5r6r7r8r!r^r9no_gradrr1r; __classcell__ris@r0rSrSis6/|//" ]]_ < r rhrqr rr?r=).N)rAmathfloorlog2r9tensorrhfloat32arangeint32powminrBcumsumbfloat16reshapert)rrrq batch_size seq_lengthclosest_power_of_2basepowersslopes extra_basenum_remaining_heads extra_powers arange_tensoralibis r0build_alibi_tensorrs+11Jdjj9)=>> << tyy!34q899:;NDYDYafanan D\\!Q!33NI1dTU:VM 9  & & (= 8E ==/J ? B B5 IIr3rCresidualprobtrainingc8[R"XUS9nX-nU$)z Dropout add function Args: x (`torch.tensor`): input tensor residual (`torch.tensor`): residual tensor prob (`float`): dropout probability training (`bool`): training mode )pr)Fdropout)rCrrrouts r0 dropout_addrs  ))A 1C .C Jr3c^\rSrSrSS\4U4SjjjrS\RS\\R\R\R44Sjr S\RS\R4Sjr SS \RS \ \RS \RS \ \RS \ \ S\ \RS\S\S\ \RS\ \\R\R44SjjrSrU=r$)FalconAttentionrUc>[TU]5 XlURUlURUlURUR -UlURUlURUlURUl URUl SUl URS:HUl X lUc-[R!SUR"R$S35 UR UR -UR:wa&['SURSUR S35eS[(R*"UR 5- UlUR,UlUR0(a*UR2S -UR-UR -nO=UR4(aURS UR --nOS UR-n[7URX1R8S 9UlUR0UlUR4Ul[7URURUR8S 9Ul[>R@"URB5Ul!UR0(dUR4(d UR2OS UlURD(a[GURS 9Ul$gg)NTsdpazInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.zA`hidden_size` must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).?r>r r-r rU)%r]r^rU hidden_sizenum_attention_headsrhead_dim split_sizehidden_dropoutra rope_theta is_causal_attn_implementation _use_sdpa layer_idxlogger warning_oncerir5 ValueErrorrsqrtinv_norm_factorbetanew_decoder_architecture num_kv_heads multi_queryr%r-query_key_valuedenserDropoutattention_dropoutrotaryrS rotary_emb)r.rUr qkv_out_dimris r0r^FalconAttention.__init__sQ  !--33((DNN: **$33'-'E'E$ ++44>"      !8!8 9:,,  ==4>> )T-=-= =STXTdTdSefNN#2'  #TYYt}}%==((  * *!..2V5O5OOSWS`S``K   **Q->>Kd...K+D,<,Nr .rr ) rrAviewrrrr9 broadcast_toflattenr) r.rbatchseq_len_qkvquerykeyvaluerCrrthree_times_hidden_sizes r0 _split_headsFalconAttention._split_headss  ( ( ) EA..T^^tGXGX5X[\5\^b^k^klC1a" %EaAtm$C1a" &E$$S++6C&&ukk:E;@u:M N:MQ1a:M N Eu$ $!!>Goo ;J$;!zt~~qRVR_R_`IS!QY'319)=yaQR?SS S>Goo ;J$;!zt~~PQ?QSWS`S`aIS#2#q[)9S2$\+BIcTVSWYZlD[[ [!OsF4rCcURup#nX R-nURXPRX0R5nUR SSSS5nUR XSURUR-5$)z Merge heads together over the last dimension Args: x (`torch.tensor`): [batch_size * num_heads, seq_length, head_dim] Returns: torch.tensor: [batch_size, seq_length, num_heads * head_dim] rr>r r )rArrrpermuter)r.rCbatch_size_and_num_headsrrrs r0 _merge_headsFalconAttention._merge_heads so34''/ a-?  FF:~~z== I IIaAq !yy$--1OPPr3r/rrrM layer_past head_mask use_cacheoutput_attentionscache_positionposition_embeddingsc  URU5n UR(a URO URn UR U 5upnU R unn nU R SS5RUURUUR5n UR SS5RUU UUR5nUR SS5RUU UUR5nUcU unn[XUU5upUb:SU 0nUcURWWS.5 URXURU5upUR SnUR(aMU RRS:Xa3Ub0U R5n UR5nUR5nUbUSS2SS2SS2SUR S24nUGcKUR(aTU(dMUR (a UcUS:aSOSn["R$R&R)U UUUS US 9nSnO_XR S S5-nU[*R,"UR5-n[.R0"UU-S UR2S 9nUU-nUR5UURUUR5nUR7S SSS5nURUUURUR-5nUR9U5nU(aUUU4$UU4$UR(aU(dUcUR (a UcUS:aSOSn["R$R&R)U UUUUR:(aUR<R>OS US 9nUR SS5nURUUURUR-5nUR9U5nGOFXR S S5-nUR5UURUU5nUR2nU["R@:XdU["RB:XaURE["RF5nUUR5UURSS 5-nUURH-n[.R0"UU-S UR2S 9nUR=U5nUbUU-nUR5UURUU5nUU-RKS S5nURMU5nUR9U5nU(aUUW4$UU4$)Nr r>rrLrKrcudaTF) attn_mask dropout_prr=)r@rqrr )'rrrrrrArxrrrQupdaterrrhrY contiguousrr9rr scaled_dot_product_attentionrrrsoftmaxrqrrrrrrfloat16rrtrrrr)r.r/rrrMrrrrrrrr query_layer key_layer value_layerr query_lengthrrKrL cache_kwargs kv_lengthr attn_outputattention_scores matmul_result input_dtypeattention_logitsattention_probsattention_probs_reshapeds r0r1FalconAttention.forward$s((7 )-)F)Ft~~DL]L] 040A0A)0L-)4):):& L!Q!++Aq199*dnnVbdhdqdqr ''1-55j,P\^b^k^kl !++Aq199*lT`bfbobop =*HC%9+RUWZ%[ "K  !,n=L}##C$<=%/%6%6yt~~_k%l "IOOB' >>k0055?ND^&002K!,,.I%002K  %+Aq!5Jyr7J5J,JKN =~~&7 %)NN~7MR^abRbDhm #hh11NN,!' O $( #.1D1DR1L#L  DIIdmm$<< #$99-=-NTV^k^q^q#r .< %**:t~~|UYUbUbcK%--aAq9K%--j,Y]YfYfHfgK**[1K "J0@@@"J..~~&7I--d4466S' O *33Aq9 )11*lDNN]a]j]jLjk "jj5 +.A.A"b.I I $1#5#5j$..R^`i#j /44 %--/;%..3P'7':':5=='I$#3ejjT^^]^`b6c#c  D$8$88 "#)),<~,MSU]j]p]p"q"&"8"8"I(&5 &AO,;+?+? DNN\hjs+t( 8+ENNqRST #// < "jj5  "J??"J..r3)rrrrUrrrrrrrrarrrrrrrrr*NNNFFNN)r5r6r7r8r!r^r9r:rrrr LongTensorrboolr1r;rrs@r0rrsF-H|-H-H^\ell\uU\\5<^\rSrSrSrU4SjrSS\RS\\RS\RS\\RS\\ S \\RS \ S \ S \\RS \\ \R\R44Sjjr SrU=r$)FalconFlashAttention2ia8 Falcon flash attention module. This module inherits from `FalconAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cD>[TU]"U0UD6 [5Ulgr*)r]r^r_flash_attn_uses_top_left_mask)r.argskwargsris r0r^FalconFlashAttention2.__init__s# $)&) /P.Q+r3r/rrrMrrrrrrc "URU5n UR(a URO URn UR U 5upnU R unn nU R SS5RUURUUR5n UR SS5RUU UUR5nUR SS5RUU UUR5nUcU unn[XUU5upUb:SU 0nUcURWWS.5 URXURU5upU R SS5n UR SS5nUR SS5nUb [S5eUR(aURROSnU R nU["R$:Xa["R&"5(a["R("5nOR[+URS5(aURR,nO URR.R n[0R3SUS 35 U R5U5n UR5U5nUR5U5n[7U UUUUUUUR8UR:S 9 nURUUURUR-5nUR=U5nU(dSnUUU4$) Nr r>rrz6`alibi` is not supported when `use_flash_attn` is Truer_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)rMrruse_top_left_mask)rrrrrrArxrrrQrrrrrUrrqr9ris_autocast_enabledget_autocast_gpu_dtyper_rr+rrrtr#rrr)r.r/rrrMrrrrrrrrrrrrrrrKrLr attn_dropoutr target_dtyper attn_weightss r0r1FalconFlashAttention2.forwards((7 )-)F)Ft~~DL]L] 040A0A)0L-)4):):& L!Q!++Aq199*dnnVbdhdqdqr ''1-55j,P\^b^k^kl !++Aq199*lT`bfbobop =*HC%9+RUWZ%[ "K  !,n=L}##C$<=%/%6%6yt~~_k%l "I"++Aq1 ''1- !++Aq1  UV V8< t{{443 "'' %-- '((**$;;= &?@@#{{BB #33::@@    >$  &..6K! \2I%..6K.     % nn"AA  #**:|T^^VZVcVcEcd jj.  LJ 44r3)rr)r5r6r7r8__doc__r^r9r:rrrrrr1r;rrs@r0rrs R48&*,0"'59KOV5||V5 %V5 V5 u//0 V5 UO V5ELL)V5V5 V5!!1!12V5&eELL%,,,F&GHV5V5r3rcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $) FalconMLPirUc>[TU]5 URn[X!RUR S9Ul[UR5Ul [URX!R S9Ul URUl g)Nr) r]r^rr%ffn_hidden_sizer- dense_h_to_4hr activationact dense_4h_to_hr)r.rUrris r0r^FalconMLP.__init__sp (( )+7M7MTZT_T_`!&"3"34)&*@*@+T_T_`$33r3rCr(chURURU55nURU5nU$r*)rrr)r.rCs r0r1FalconMLP.forwards0 HHT''* +   q !r3)rrrr) r5r6r7r8r!r^r9r:r1r;rrs@r0rrs/4|4%,,r3r)eagerrflash_attention_2c^\rSrSrSS\4U4SjjjrSS\RS\\RS\RS\\RS\\ \ \ \R\R44S \\RS \ S \ S \\RS \\ \R\R44SjjrSrU=r$)FalconDecoderLayeri*rUcf>[TU]5 URnURUl[ UR "X5Ul[U5Ul URUl Xl URcUR(aSUl UR(d1[X1R S9Ul[X1R S9UlgURS:Xa1[X1R S9Ul[X1R S9Ulg[X1R S9Ulg)Nr>eps)r]r^rrrFALCON_ATTENTION_CLASSESrself_attentionrmlprrUnum_ln_in_parallel_attnr parallel_attnr layer_norm_epsilonpost_attention_layernorminput_layernormln_attnln_mlp)r.rUrrris r0r^FalconDecoderLayer.__init__+s (( 336v7R7RSTZfV$$33  ) ) 1f6U6U-.F *##,5kG`G`,aD )#,[>W>W#XD --2(:S:ST ' 9R9RS '0B[B['\$r3r/rrrMrrrrrrc Un URR(a=URRS:Xa#URU5n UR U5nOUR U5n UR U UUUUUUUU U S9 nUSnURR(dXURR(aU nO:[UXRRURS9n URU 5nURR(a7URR(aURRS:XaU nUSSnURW5nURR(dURR(aUU- n[UXRRURS9nU(aU4U-nU$U4USS-nU$)Nr>) rrrMrrrrrrr)rr )rUrr'r,r-r+r%r(rrrr*r&r)r.r/rrrMrrrrrrrrattention_layernorm_outmlp_layernorm_out attn_outputsattention_outputoutputs mlp_outputoutputs r0r1FalconDecoderLayer.forwardDs! ;; / /DKK4W4W[\4\&*ll=&A # $ M : &*&:&:=&I #** #!)%/) 3+  (?{{33{{(($;!&$h 0M0MX\XeXe%)$A$A($K! KK 0 0 )) 33q8 7 qr"XX/0 ;; / /4;;3L3L * *JZ;;3M3MX\XeXef i')Gi'!"+-Gr3) rUrr+r,r-r&rr*r%r*r)r5r6r7r8r!r^r9r:rrrrrrr1r;rrs@r0r r *s]|]]<48PT,0"'59KOE||E %E E u//0 E U5% ell0J*K#KLM EELL)EE E!!1!12E&eELL%,,,F&GHEEr3r c^\rSrSr\rSrSrS/rSr Sr Sr Sr Sr U4SjrS\R 4Sjr\S S\S S 4S jj5rS rU=r$)FalconPreTrainedModeli transformerTr c&>[TU]"U0UD6 gr*)r]r^)r.inputsrris r0r^FalconPreTrainedModel.__init__s &+F+r3modulec.[U[R5(d[U[5(akURR R SURRS9 URb%URR R5 gg[U[R5(axURR R SURRS9 URb2URR URR5 gg[U[5(aJURR R5 URR RS5 gg)zInitialize the weights.r)meanstdNr)rurLinearr%r+datanormal_rUinitializer_ranger-zero_ Embedding padding_idxr fill_)r.r>s r0 _init_weights#FalconPreTrainedModel._init_weightss fbii ( (Jv|,L,L MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  * * KK   " " $ MM   $ $S )+r3hard_check_onlyr(r"cN[USS5nU(aU$U(dSUlU$)Nuse_bettertransformerFr)getattrr)clsrUrL_is_bettertransformers r0_check_and_enable_sdpa,FalconPreTrainedModel._check_and_enable_sdpas* '-De L M*0F ' r3r4)F)r5r6r7r8r! config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_2_supports_sdpa_supports_cache_class_supports_quantized_cache_supports_static_cacher^rModulerJ classmethodrrRr;rrs@r0r9r9srL%&*#-.!N  $!,*BII*"TN`r3r9c^\rSrSrS\4U4SjjrSrS\R4Sjr \ SS\ \RS\ \ \\\\R\R4S 44S \ \RS \ \RS \ \RS \ \RS\ \S\ \S\ \S\ \S\ \RS\ \\RS 4\44Sjj5rS \RS\RS\RS\S\S \RS\R4Sjr\S \RS\S\S\R,S\RS\4 Sj5rSrU=r$) FalconModelirUc `>[TU]U5 URUlURUlUR Ul[R"URUR5Ul [R"[UR5Vs/sH n[XS9PM sn5UlUR"S:HUlUR"S:HUl[)URUR*S9Ul[/US9UlSUlUR55 gs snf)N)rrrr"rF)r]r^r embed_dimrrr use_alibirrG vocab_sizeword_embeddings ModuleListrangenum_hidden_layersr hr_use_flash_attention_2rr r)ln_frSrgradient_checkpointing post_init)r.rUiris r0r^FalconModel.__init__s  ++33 "||F,=,=t~~NQVW]WoWoQpqQpA 26 GQpqr&,&A&AEX&X#44>dnn&2K2KL /v>&+#  rsD+cUR$r*rer.s r0get_input_embeddings FalconModel.get_input_embeddingss###r3new_embeddingscXlgr*rqr.rus r0set_input_embeddings FalconModel.set_input_embeddingss-r3 input_idspast_key_values.rrMr inputs_embedsrroutput_hidden_states return_dictrr(c UbUOURRnU bU OURRn UbUOURRnU bU OURRn USLUSL- (a [ S5eUR (a/UR(aU(a[RS5 SnUcURU5nSn U(aP[U[5(d;Sn Uc [5nO+[R"U5n[RS5 Sn UbUR5OSnUR unnnUR"(aWUc3[$R&"UUU-4UR([$R*S9OUn[-UUR.UR0S 9n U c"[$R2"XU-UR(S 9n UcU R5S5nUR7X6XXU 5nUR9XPRR:5nUnUR=UU5nSnU(aS OSnU (aS OSn[?UR@5HunnU (aUU4-nUR (a:UR(a)URCURDUU UUUUUUUU U5 nOU"UUUUUUUUU U US 9 nUSnUSLaUS nU(dMUUU(aSOS 4-nM URGU5nU (aUU4-nU(aUOSnU (aURI5nU (d[KSUUUU455$[MUUUUS9$)h input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0][0].shape[2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Nz:You must specify exactly one of input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FTzWe detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)rrrprhr4) rrrMrrrrrrr r>c3.# UH ocMUv M g7fr*r4).0vs r0 &FalconModel.forward..ps^a^s )last_hidden_stater{r/ attentions)'rUrr}ruse_return_dictrrlrrrrerurrfrom_legacy_cacheget_seq_lengthrArcr9onesrhlongrrrqrrH_update_causal_mask get_head_maskrhr enumerateri_gradient_checkpointing_func__call__rkto_legacy_cachetupler)r.rzr{rrMrr|rrr}r~rreturn_legacy_cacherpast_key_values_lengthrrrmask causal_maskr/rnext_decoder_cacheall_self_attentionsall_hidden_statesrnblockr4 next_caches r0r1FalconModel.forwards82C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK> ") .D!DEmNbNbjojtjt$ 'tT^^=CVCVWE  !"\\&(KTaThThN  )33A6L.. >L]jo &&y++2O2OP % #oom\J!$5b4"6BD!$&&)HAu#$58H$H!**t}};;NN! aL#%"'  !.#.!-'l'&7#1(; $AJMD %,QZ"  &9W)QYZ=[<]&]#K*P -0  1]4D D +4'$ #335J):7HJ]^ 9+&+*   r3 input_tensorrc HURRS:Xa UbSU;aU$gUbUR5OSn[U[5n URRS:Xa;U (d4U(d-Uc*Uc'[ R "UUUURS9(agURURp[R"U 5Rn URupnU (aUR5nO2[U[R5(aURSOX-nUR!UUUU U UURSS9nUcwUbtUR""U S/URSSQ76n[R$"U[&R("URR*UR,-5- US:U 5nURRS:Xa;Ub8URR.S ;aU(d[ R0"UU 5nU$) Nrrrr)r|r is_trainingr=)sequence_length target_lengthrqrhrrr )rxpunpu)rUrrrurr_ignore_causal_mask_sdparrqrhr9finforrAget_max_cache_shaper:5_prepare_4d_causal_attention_mask_with_cache_positionr masked_fillrrrrrY_unmask_unattended)r.rrrr{rrrpast_seen_tokensusing_static_cacherqrh min_dtyperrrrrs r0rFalconModel._update_causal_mask{s ;; + +/B B)c^.C%% @O?Z?99;`a'E KK , , 6&%! %>>*'7 MM  $**L,?,?vKK&** )5););& Q +??AMnell;;$$R(%7 PP +')#))!,Q   !2MM*bC5;;qr?CE++ $++"9"9T^^"KLLb K KK , , 6*%%**.DD% 1CCKQZ[Kr3rrrqrc UbUR5S:XaUnU$[R"U5Rn[R"X4XUR S9nUS:wa[R "USS9nU[R"X$R S9URSS5:-nUSSSS2SS24RUSSS5nUbUR5nURSn USS2SS2SS2SU 24USS2SSSS24RUR 5-n U S:Hn USS2SS2SS2SU 24RX5USS2SS2SS2SU 24'U$) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuerqrhr )diagonalrr=r)r@r9rrfullrhtriurrrsclonerArtr) rrrrqrrrrr mask_length padding_masks r0rAFalconModel._prepare_4d_causal_attention_mask_with_cache_positions}>  %.*<*<*>!*C(K*' E*..I** 0Y\j\q\qK!##jjqA 5<< >S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12r3) rjrrbrlrirkrrrcre) NNNNNNNNNNN)r5r6r7r8r!r^rsr9r:rxrrrrrrrrr1r staticmethodintrqrr;rrs@r0r`r`s3|2$.5<<.15ae15370448$(,0/3&*59` E,,-` "%uU5<<;U5VX[5[/\(\"]^` !. ` u//0 ` E,,- `   0 01` D>` $D>` 'tn` d^` !!1!12`  uU\\3&')RR S` ` DU UllU U  U  U<<U||Un4 444{{ 4  4  44r3r`z The Falcon Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc ^\rSrSrS/rS\4U4SjjrSrS\R4Sjr \ SS\ \RS \ \\\\\R\R4S 44S \ \RS \ \RS \ \RS\ \RS\ \RS\ \S\ \S\ \S\ \S\ \RS\\\R4S\\\R\44Sjj5rS\\\R\R4S 4S\RS\\\R\R4S 44SjrSrU=r$)FalconForCausalLMi zlm_head.weightrUc>[TU]U5 [U5Ul[R "UR URSS9UlUR5 gNFr) r]r^r`r:rrBrrdlm_headrmr.rUris r0r^FalconForCausalLM.__init__sI  &v.yy!3!3V5F5FUS  r3cUR$r*rrrs r0get_output_embeddings'FalconForCausalLM.get_output_embeddingss ||r3rucXlgr*rrws r0set_output_embeddings'FalconForCausalLM.set_output_embeddingss% r3rzr{.rrMrr|labelsrrr}r~rlogits_to_keepr(c U bU OURRn URUUUUUUUU U U U S9 nUSn[U [5(a [ U *S5OU nUR USS2USS245nSnUb*UR"UU4SURR0UD6nU (dU4USS-nUbU4U-$U$[UUURURURS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0][0].shape[2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N) r{rrMrr|rrr}r~rrrdr losslogitsr{r/r) rUrr:rurslicer loss_functionrdrr{r/r)r.rzr{rrMrr|rrrr}r~rrrtransformer_outputsr/ slice_indices lm_logitsrr6s r0r1FalconForCausalLM.forward!s3H&1%SV8W8W~ot4]k LLq-/B!CD   %% ;;11 D\$7$;;F)-)9TGf$ Ev E0/??-;;*55   r3pastbeam_idxc ^UVVs0sH1o3H(oDRURUR5_M* M3 snnm[U4SjU55nU$s snnf)a$ This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or [`~PreTrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct beam_idx at every generation step. Output shares the same memory storage as `past`. c3># UHOnUSRSTUSR5USRSTUSR54v MQ g7f)rr N) index_selectrh)rrdevice_to_beam_idxs r0r3FalconForCausalLM._reorder_cache..}sf # 1 **1.@AAUAU.VW1 **1.@AAUAU.VW #sAA)rhrtr)r.rrr past_statereordered_pastrs @r0_reorder_cache FalconForCausalLM._reorder_cachensnQU PT*gqYc  x{{:+<+<= =gq PT  #    s8A)rr:) NNNNNNNNNNNNr)r5r6r7r8_tied_weights_keysr!r^rr9r:rrrrrrrrrrr1rr;rrs@r0rr s  ++|&ELL&15ae1537,004)-$(,0/3&*5934J E,,-J "%uU5<<;U5VX[5[/\(\"]^J !. J u//0 J ELL) J   -J &J D>J $D>J 'tnJ d^J !!1!12J c5<</0J uU\\"$EE F!J J X% ell :;S@AMRM]M] uU\\5<</0#5 6r3ra The Falcon Model transformer with a sequence classification head on top (linear layer). [`FalconForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-1) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). c^\rSrSrS\4U4Sjjr\SS\\RS\\ \ \R\R4S4S\\RS\\RS \\RS \\RS \\ S \\ S \\ S\\ S\ \ \R\44Sjj5rSrU=r$)FalconForSequenceClassificationirUc>[TU]U5 URUl[U5Ul[ R "URURSS9UlUR5 gr) r]r^ num_labelsr`r:rrBrscorermrs r0r^(FalconForSequenceClassification.__init__sV   ++&v.YYv1163D3D5Q  r3rzr{.rrr|rrrr}r~r(c U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UbURSnOURSnURR cUS:wa [ S5eURR cSnOUbXRR :gRU R[R5n[R"URSU R[RS9nUU-RS5nO.Sn[RURR S35 U [R"XRS 9U4nSnUGbgURR"cUR$S:XaS URlOoUR$S:aNUR&[R(:XdUR&[R*:XaS URlOS URlURR"S :XaJ[-5nUR$S:Xa&U"UR/5UR/55nOeU"UU5nO[URR"S :Xa[15nU"UU5nO-URR"S :Xa[35nU"UU5nU (dU4U SS-nUbU4U-$U$[5UUU R6U R8U R:S 9$) input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0][0].shape[2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr{rrr|rrr}r~rr z=Cannot handle batch sizes > 1 if no padding token is defined.r=rz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)rUrr:rrA pad_token_idrrtrhr9rrargmaxrrrir5 problem_typerrqrrr squeezer rrr{r/r)r.rzr{rrr|rrrr}r~rr/rrlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctr6s r0r1'FalconForSequenceClassification.forwards@&1%!F!Fr!J !#    >>**+,ZZ  u||J}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- v6))-II,. v6#%(;AB(??F)-)9TGf$ Ev E/ /??-;;*55   r3)rrr: NNNNNNNNNN)r5r6r7r8r!r^rrr9rrr:rrrr1r;rrs@r0rrs0|15SW15,004)-$(,0/3&*g E,,-g "%ellELL.H(I3(N"OPg !. g ELL) g   - g &g D>g $D>g 'tng d^g  uU\\"$DD Eg g r3rc^\rSrSrS\4U4Sjjr\SS\\RS\\ \ \R\R4S4S\\RS\\RS \\RS \\RS \\ S \\ S \\ S\\ S\ \ \R\44Sjj5rSrU=r$)FalconForTokenClassificationi rUc>[TU]U5 URUl[U5Ul[ USS5b UR nO[ USS5b URnOSn[R"U5Ul [R"URUR5Ul UR5 g)Nclassifier_dropoutrg?)r]r^rr`r:rOrrrrrrBr classifierrm)r.rUrris r0r^%FalconForTokenClassification.__init__ s   ++&v. 6/ 6 B!'!:!:  V-t 4 @!'!6!6 !$ zz"45 ))F$6$68I8IJ r3rzr{.rrr|rrrr}r~r(c U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUbQUR unn[ 5nU"U RUU-UR5URUU-55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)rNrrr>)rrr/r) rUrr:rrrAr rrrr/r)r.rzr{rrr|rrrr}r~rr/rrrrrr6s r0r1$FalconForTokenClassification.forwards@&1%B $D>B 'tnB d^B  uU\\"$99 :B B r3rc2^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\ S \\ S \\ S \ \ \ 44S jj5rSrU=r$)FalconForQuestionAnsweringicc>[TU]U5 [U5Ul[R "UR S5UlUR5 g)Nr>) r]r^r`r:rrBr qa_outputsrmrs r0r^#FalconForQuestionAnswering.__init__esA  &v.))F$6$6: r3rzrrr|start_positions end_positionsrr}r~r(c U bU OURRn URUUUUUUU S9n U Sn URU 5n U R SSS9upU R S5R 5n UR S5R 5nSnUbUb[UR55S:aUR S5n[UR55S:aUR S5nU RS5nURSU5nURSU5n[US9nU"X5nU"X5nUU-S- nU (dX4U SS-nUbU4U-$U$[UU UU RU RS 9$) rN)rrr|rr}r~rr r=r?) ignore_indexr>)r start_logits end_logitsr/r)rUrr:rsplitrrlensizeclampr rr/r)r.rzrrr|rrrr}r~r4sequence_outputrr r  total_loss ignored_indexr start_lossend_lossr6s r0r1"FalconForQuestionAnswering.forwardms4&1%G 'tnG d^G  u22 3G G r3r)rr`r9rrr)Nr )Mrrtypingrrrrr9torch.utils.checkpointrtorch.nnrr r r r r activationsr cache_utilsrrr generationrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrrmodeling_outputsrrrrrmodeling_rope_utilsrrmodeling_utilsrutilsrrconfiguration_falconr!configuration_utilsr"r# get_loggerr5rrBr%rFrQr]rSr:rrqrrrrrrrrr$r r9r`rrrr__all__r4r3r0r(sF 88 LL$);;)iL-/7J   H % )299)(8