fThlSrSSKrSSKrSSKJrJrJrJr SSKrSSKJ r SSK J r J r J r SSKJr SSKJrJrJr SS KJr SS KJr SS KJrJrJrJrJrJrJr SS K J!r! SS K"J#r#J$r$J%r%J&r&J'r'J(r(J)r) SSK*J+r+ \&"5(a SSK,J-r- SSK.J/r/ \)R`"\15r2"SS\ Rf5r4"SS\ Rf5r5"SS\ Rf5r6"SS\ Rf5r7"SS\ Rf5r8"SS\ Rf5r9"SS\ Rf5r:"S S!\ Rf5r;"S"S#\ Rf5r<\%"S$S%\!55r="S&S'\=5r>\%"S(S)\=55r?\%"S*S+9"S,S-\=\55r@\%"S.S/\=55rA\%"S0S+9"S1S2\=55rB\%"S3S4\=55rC\%"S5S6\=55rD/S7QrEg)8zPyTorch UMT5 model.N)ListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)AttentionMaskConverter)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringis_torch_flex_attn_availableis_torch_fx_proxyis_torchdynamo_compilinglogging) UMT5Config) BlockMask)make_flex_block_causal_maskc2^\rSrSrSU4SjjrSrSrU=r$) UMT5LayerNorm<c>[TU]5 [R"[R "U55UlX lg)zU Construct a layernorm module in the UMT5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s ^/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/umt5/modeling_umt5.pyr*UMT5LayerNorm.__init__=s/ ll5::k#:; #cUR[R5RS5R SSS9nU[R "X R -5-nURR[R[R4;a%URURR5nURU-$)NT)keepdim) tor,float32powmeanrsqrtr/r.dtypefloat16bfloat16)r0 hidden_statesvariances r4forwardUMT5LayerNorm.forwardEs !##EMM266q9>>r4>P% H?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r6)r/r.)gư>)__name__ __module__ __qualname____firstlineno__r*rE__static_attributes__ __classcell__r3s@r4r&r&<s$ + +r6r&c6^\rSrSrS\4U4SjjrSrSrU=r$)UMT5DenseActDenseVconfigcX>[TU]5 [R"URUR SS9Ul[R"UR URSS9Ul[R"UR5Ul [URUl gNFbias)r)r*rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr0rQr3s r4r*UMT5DenseActDense.__init__Wsn ))FNNFKKeD))FKKeDzz&"5"56 &--.r6cURU5nURU5nURU5n[URR [ R5(aURURR R:waaURR R[ R:wa/URURR R5nUR U5nU$N) rYr_r] isinstancerZr.r,Tensorr@int8r;r0rCs r4rEUMT5DenseActDense.forward^s . /  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r6)r_r]rYrZ rGrHrIrJr"r*rErKrLrMs@r4rOrOVs/z/  r6rOc6^\rSrSrS\4U4SjjrSrSrU=r$)UMT5DenseGatedActDensemrQc>[TU]5 [R"URUR SS9Ul[R"URUR SS9Ul[R"UR URSS9Ul[R"UR5Ul [URUlgrS)r)r*rrVrWrXwi_0wi_1rZr[r\r]r r^r_r`s r4r*UMT5DenseGatedActDense.__init__ns IIfnnfkkF IIfnnfkkF ))FKKeDzz&"5"56 &--.r6c8URURU55nURU5nX#-nURU5n[ UR R [R5(aURUR R R:waaUR R R[R:wa/URUR R R5nUR U5nU$rc) r_rnror]rdrZr.r,rer@rfr;)r0rC hidden_gelu hidden_linears r4rEUMT5DenseGatedActDense.forwardvshhtyy78  -0 #3  ]3 tww~~u|| 4 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r6)r_r]rnrorZrirMs@r4rkrkms/z/r6rkc6^\rSrSrS\4U4SjjrSrSrU=r$) UMT5LayerFFrQc>[TU]5 UR(a[U5UlO[ U5Ul[ URURS9Ul [R"UR5Ul g)Nr2)r)r* is_gated_actrkDenseReluDenserOr&rWlayer_norm_epsilon layer_normrr[r\r]r`s r4r*UMT5LayerFF.__init__s_    "8"@D "3F";D 'F[TU]5 URUlX lURUlUR UlUR UlURUlURUl URUl URUR-Ul X0lUc>UR(a-[R!SUR"R$S35 [&R("UR URSS9Ul[&R("UR URSS9Ul[&R("UR URSS9Ul[&R("URUR SS9UlUR(a0[&R2"URUR5Ul[75Ulg)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrT)r)r* is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerWd_kvkey_value_proj_dim num_headsn_headsr\r] inner_dimrlogger warning_oncer3rGrrVqkvo Embeddingrelative_attention_biasset pruned_heads)r0rQrrr3s r4r*UMT5Attention.__init__se  +++F(.4.S.S+/5/U/U,~~ "(++'' ** (?(??"     *4>>+B+B*CD,, 4<<eD4<<eD4<<eD4>>4<FFq!QPQRr6cSnURnURnUR(dCUS-nX!S:R[R 5U-- n[R "U5nO,[R"U[R"U55*nUS-nX:n[R"UR5U- 5[R"XE- 5- nXsU- -nXWR[R 5-n[R"U[R"XS- 55nU[R"XaU5- nU$)aR Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr8r!)rrrr;r,longabsmin zeros_likelogfloatmath full_likewhere) r0relative_positionrelative_buckets num_buckets max_distance max_exactis_small log_ratiorelative_position_if_larges r4_relative_position_bucket'UMT5Attention._relative_position_buckets/*99 ;;  A K Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $0II/557)CDtxxP\PhGii y!89 %.ejj1I%I"%*YY &8RbcTc(d& " EKKE_``r6cUc URRRnUc,[R"U[R US9SS2S4nO USS2S4n[R"U[R US9SSS24nXe- nUR U5nURU5n U R/SQ5RS5n U $)z%Compute binned relative position biasN)r@device)r8rr!r) rr.rr,arangerrr unsqueeze) r0 query_length key_lengthrcache_positioncontext_positionmemory_positionrrelative_position_bucketvaluess r4 compute_biasUMT5Attention.compute_biass >1188??F  !$||L SYZ[\^b[bc -ag6 ,,zFSTXZ[T[\+>#'#A#ABS#T --.FG *44Q7 r6rCencoder_hidden_statespast_key_valueattention_masklayer_head_maskrcURSSupxUSLn URU5n U RUSURUR5R SS5n UbEUR RUR5n U (a URn O URn U (aUOUn U (a=Ub:W (a3W RURnU RURnOURU 5nURU 5nURUSURUR5R SS5nURUSURUR5R SS5nUbLU (dUOSnW RXURSU05upU (aSUR UR'[ R""XR SS55nUbXR%5-OUnURSnUR&(d9[ R("SURUU4UR*UR,S9nO.UR/UUUR*US 9nUSS2SS2U*S2SS24nUb#USS2SS2SS2SURS24nUU-nUR0(aS[ R2"URS5nS U[5UR05'USS2UR754nOUnUU- n[8R:R=UR?5SS 9RAU5n[8R:RCUURBURDS 9nUbUU-n[ R""UU5nUR SS5RG5nURXxS5nURIU5nUUU4$) Nr8r9r!rTr )rr@)rrrdim)ptraining)%shaperrrr transpose is_updatedgetrcross_attention_cacheself_attention_cache key_cache value_cacherrupdater,matmulget_seq_lengthrzerosrr@rrr-listboolr functionalsoftmaxrtype_asr]r contiguousr)r0rCrrrrr batch_size seq_lengthis_cross_attention query_statesrcurr_past_key_valuecurrent_states key_states value_statesscoresreal_seq_lengthr position_bias causal_maskmaskposition_bias_masked attn_weights attn_outputs r4rEUMT5Attention.forwards"/!4!4Ra!8 3$>vvm, #((RtG^G^_iijkmno  %'2266t~~FJ!&4&J&J#&4&I&I#2D.- ."<,66t~~FJ.::4>>JL/J66.1L#RtG^G^_iijkmnoJ',,ZT\\4KbKbcmmnoqrsL)7It+>+E+Ednn?OQ_>`,( &@DN--dnn=l,@,@A,FGKYJd*'D'D'FFjt%%b) //!KKDLL*j9&--W]WcWcM!--FMMR`.M*!Q a*?@M  %(Aq2HJ4D4DR4H2H)HIK)K7M   ::m11!45D,-Dd''( )#0DIIK#@ #0 &&}},,V\\^,DLLVT }},,\T\\TXTaTa,b   &'/9Lll<> !++Aq1<<> !&&zrB ff[) L.88r6)rWr]rrrrrrrrrrrrrr)FN)NNNNNNN)rGrHrIrJ__doc__rintr*r,rerrrrrErKrLrMs@r4rrs"XVY]"":%,, - ^ $9=8<152615Y9||Y9 ( 5Y9!u||!45 Y9 !. Y9 "%,,/ Y9!.Y9Y9r6rcL^\rSrSrSS\\4U4SjjjrSSjrSrU=r $)UMT5LayerSelfAttentioni_rc>[TU]5 [USUS9Ul[ UR UR S9Ul[R"UR5Ul g)NTrrry) r)r*r SelfAttentionr&rWr|r}rr[r\r]r0rQrr3s r4r*UMT5LayerSelfAttention.__init__`sN *6t_hi'F[TU]5 [USUS9Ul[ UR UR S9Ul[R"UR5Ul g)NFrry) r)r*rEncDecAttentionr&rWr|r}rr[r\r]rs r4r* UMT5LayerCrossAttention.__init__|sO ,VQVbkl'F[TU]5 URUl[R"5UlUR R [XS95 UR(a"UR R [XS95 UR R [U55 g)Nr) r)r*rr ModuleListlayerappendrrrvrs r4r*UMT5Block.__init__sv  ++]]_  0MN ?? JJ  5fR S +f-.r6c URS"UUUUU S9upnUR[R:Xa}[R"UR5R n [R "[R"U5R5U S- U 5n [R"X*U S9nSnUR=(a USLnU(aURS"UUUUUU S9upnUR[R:Xa}[R"UR5R n [R "[R"U5R5U S- U 5n [R"X*U S9nURS"U5nUR[R:Xa}[R"UR5R n [R "[R"U5R5U S- U 5n [R"X*U S9nUU4nU (aUX4- nU$)Nrri)rmaxr!r r9) rr@r,rAfinforrisinfanyclampr)r0rCrrencoder_attention_maskrcross_attn_layer_head_maskr use_cacheoutput_attentionsrself_attn_weights max_dtype clamp_valuecross_attn_weightsdo_cross_attentionrs r4rEUMT5Block.forwards<@::a= )+)) < 8 .   %-- / M$7$78< >Gr6)rrrc) NNNNNNFFNrrMs@r4rrs?/(3-//"##'>>r6rcn^\rSrSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $) UMT5ClassificationHeadiz-Head for sentence-level classification tasks.rQc(>[TU]5 [R"URUR5Ul[R "URS9Ul[R"URUR5Ul g)N)r) r)r*rrVrWdenser[classifier_dropoutr] num_labelsout_projr`s r4r*UMT5ClassificationHead.__init__sZ YYv~~v~~> zzF$=$=>  &..&2C2CD r6rCrcURU5nURU5n[R"U5nURU5nUR U5nU$rc)r]r(r,tanhr+rgs r4rEUMT5ClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r6)r(r]r+) rGrHrIrJrr"r*r,rerErKrLrMs@r4r&r&s47EzE U\\ellr6r&cP\rSrSr\rSrSrSrSr S/r S/r \ S5r SrSrS rg ) UMT5PreTrainedModeli transformerTrrZcz[R"[5n[R"[5nUUUS.nU$)N)decoder_input_ids input_idsdecoder_attention_mask)r,tensorrr)r0r5 input_mask dummy_inputss r4r9 UMT5PreTrainedModel.dummy_inputss6LL. \\*- !*"&0 r6cURRn[U[5(a)URR R US-5 g [U[[[[45(GaURRR RSUS-S9 [US5(aLURR(d1URRR RSUS-S9 [US5(avUR RR RSX RR"S--S9 UR R$R R'5 g g [U[(5(ar[US5(a`UR*RR RSUS-S9 UR*R$R R'5 g g [U[,5(GaQUR.RR RSX RR"S--S9 [UR.S5(aEUR.R$b.UR.R$R R'5 UR0RR RSX RR"S--S9 [UR0S5(aGUR0R$b/UR0R$R R'5 g g g [U[25(GaQUR4RR RSX RR"S--S9 [UR4S5(aEUR4R$b.UR4R$R R'5 UR6RR RSX RR8S--S9 [UR6S5(aGUR6R$b/UR6R$R R'5 g g g [U[:5(GaUR<RR RSX RR"S--S9 [UR<S5(aEUR<R$b.UR<R$R R'5 UR>RR RSX RR"S--S9 [UR>S5(aEUR>R$b.UR>R$R R'5 UR6RR RSX RR8S--S9 [UR6S5(aGUR6R$b/UR6R$R R'5 g g g [U[@5(GaZURR"nURRBnURRDnURFRR RSX#U-S--S9 URHRR RSX#S--S9 URJRR RSX#S--S9 URLRR RSX%U-S--S9 URN(a4URPRR RSX#S--S9 g g g ) zInitialize the weights?)r>stdlm_head qa_outputs classifierrUN))rQinitializer_factorrdr&r.datafill_ UMT5ModelUMT5ForConditionalGenerationUMT5EncoderModelUMT5ForQuestionAnsweringsharednormal_hasattrtie_word_embeddingsr?r@rWrUzero_UMT5ForTokenClassificationrBr&r(r+rOrYrZrXrkrnrorrrrrrrrr)r0modulefactorrWrrs r4 _init_weights!UMT5PreTrainedModel._init_weights s// fm , , MM   $ $Vc\ 2  , (     MM % % - -3FSL - Ivy))$++2Q2Q%%**22#2Nv|,,!!((--553F{{ObObgkNkDl5m!!&&++113- : ; ;v|,,!!((--553FSL5Q!!&&++113- 6 7 7 LL   $ $ , ,#6kkFYFY^bEb;c , dv||V,,1B1B1N !!&&,,. OO " " ' ' / /SfI\I\aeHe>f / gv//FOO4H4H4T$$))//15U/  1 2 2 II   ! ! ) )s;;CVCV[_B_8` ) avyy&))fiinn.H ##))+ II   ! ! ) )s;;CSCSX\B\8] ) ^vyy&))fiinn.H ##))+/I)  6 7 7 KK   # # + +&[[EXEX]aDa:b + cv{{F++ 0@0@0L   %%++- KK   # # + +&[[EXEX]aDa:b + cv{{F++ 0@0@0L   %%++- II   ! ! ) )s;;CSCSX\B\8] ) ^vyy&))fiinn.H ##))+/I)  . .kk))G!%!1!1 kk++G HHOO ( (cvL^B^cgAg7h ( i HHOO ( (cv$7O ( P HHOO ( (cv$7O ( P HHOO ( (cvL^B^cgAg7h ( i11..55::BBQWhl[lQmBn2/r6cURRnURRnUc [S5e[ U5(aE[ R "URSSS-U5n[ R"XASSS24/SS9nO=URUR5nUSSS24R5USSS24'X$S'Uc [S5eURUS :HU5 U$) Nzself.model.config.decoder_start_token_id has to be defined. In UMT5 it is usually set to the pad_token_id. See UMT5 docs for more information.r9)r!.rr!).rz1self.model.config.pad_token_id has to be defined.) rQdecoder_start_token_id pad_token_id ValueErrorrr,fullrcat new_zerosclone masked_fill_)r0r5rVrWshifted_input_idss r4 _shift_right UMT5PreTrainedModel._shift_rightOs!%!C!C{{// ! )6  Y ' ' % 9??3B+?$+FH^ _  % +<SbS>Q*RXZ [  ) 3 3IOO D )238)<)B)B)D c12g &(>f %  PQ Q&&'8D'@,O  r6N)rGrHrIrJr" config_classbase_model_prefixsupports_gradient_checkpointing_supports_cache_class_supports_static_cache_no_split_modules_keep_in_fp32_modulespropertyr9rRr_rKrar6r4r1r1sNL%&*# !$ !F @oD!r6r1c .^\rSrSrSU4SjjrSrSrSSjrSS\\ RS4S\ RS \ RS \ S \ 4 S jjr \S\ RS \S\S\ R S \ RS\4 Sj5rSrU=r$) UMT5Stackikc >[TU]U5 X lURUl[R "[ UR5Vs/sH n[XS9PM sn5Ul [URURS9Ul [R"UR5UlSUlUR%5 gs snf)NrryF)r)r* embed_tokensrrrrange num_layersrblockr&rWr|final_layer_normr[r\r]gradient_checkpointing post_init)r0rQrmir3s r4r*UMT5Stack.__init__ls  ( ++]]ERXRcRcLd#eLdqIf$BLd#ef -fnn&B[B[ \zz&"5"56 ',#  $fsCcUR$rcrmr0s r4get_input_embeddingsUMT5Stack.get_input_embeddingsxs   r6cXlgrcrwr0new_embeddingss r4set_input_embeddingsUMT5Stack.set_input_embeddings{s*r6cv U bU OURRn U bU OURRn U bU OURRn U bU OURRn Ub*Ub'UR (aSOSn[ SUSUS35eUb&UR5nURSUS5nO>UbUR5SSnO'UR (aSOSn[ SUSUS 35eUR(a/UR(aU (a[RS 5 S n Uc)URc [ S 5eURU5nUunnU S La UR (d[ SUS35eS nS nUR (aU (dUb[U[5(a,[U[ 5(dS n[!U[#55nOv[U[ 5(d.S n[RS5 [ R$"U5nO3Uc[![#5[#55nOUR (dSnUbUR'5OSnU c#[(R*"UUU-UR,S9n Uc4[/5(d%UU-n[(R0"UUUR,S9nUR (a%UR3UUU Ub UR4OSU 5nO\UbWUSS2SSSS24nUR7UR8S9nSU- [(R:"UR85R<-nOSnUR (aOUbLUR5unnnUU4nUc[(R0"UUR,S9nUR?U5nOSnURAX`RRB5nURAXpRRB5nU (aSOSnU (aSOSnU (aUR (aSOSnUREU5n[GURH5Hun n!UU n"UU n#U (aUU4-nUR(a7UR(a&URKU!RLUUUUU"U#SU U U 5 n$OU!"UUUUU"U#UU U U S9 n$U$SnU (aU$Sn%U (dMUU$S4- nUR (dMUU$S4- nM UROU5nUREU5nU (aUU4-nU (aW%OSn&U(a UR4n&U(aURQ5n&U (d[SSUU&UUU455$[UUU&UUUS9$)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer9zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fz$UMT5Stack.forward..4s" As )last_hidden_statepast_key_valuesrC attentionscross_attentions)+rQrroutput_hidden_statesuse_return_dictrrXrrrrrrrrmrdr rrfrom_legacy_cacherr,rrrr-_update_causal_maskrr;r@rrinvert_attention_mask get_head_maskror] enumeraterp_gradient_checkpointing_funcrErqto_legacy_cachetupler)'r0r5rrrr head_maskcross_attn_head_maskrrrr return_dictrerr_msg_prefix input_shaperrreturn_legacy_cachereturn_self_attention_cachepast_key_values_lengthmask_seq_lengthrencoder_batch_sizeencoder_sequence_length_encoder_hidden_shapeencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsrCrt layer_modulerr layer_outputsnext_decoder_cache next_caches' r4rEUMT5Stack.forward~s "+!6IDKK+/??ZN*>*:.HXXvw  "#..*K!r;r?;I  &',,.s3K+/??ZN:>:J-XfWggtuv v  & &4==##p"    ( !_`` --i8M!, J  ?? #LTFRg!hii$&+# ?? _-H/511*_Vi:j:j.2+"5o|~"V1DEE&*###` #6"G"G"X ("5lnln"U#OETE`!?!?!Afg  !"\\&(>(KTaThThN  !*B*D*D4zAO"ZZ OML`L`aN ??228G8S44Y]! K '(D$)9:K%..}/B/B.CK, MT]n%>>*'7 MM  ""&,,Q/ !+??AMnell;;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCK[Kr6rrr@rc UbUR5S:XaUnU$[R"U5Rn[R"X4XUR S9nUS:wa[R "USS9nU[R"X$R S9URSS5:-nUSSSS2SS24RUSSS5nUbUR5nURSn USS2SS2SS2SU 24USS2SSSS24RUR 5-n U S:Hn USS2SS2SS2SU 24RX5USS2SS2SS2SU 24'U$) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuer@rr!)diagonalrr9r)rr,rrrYrtriurreshapeexpandr\rr; masked_fill) rrrr@rrkwargsrr mask_length padding_masks r4r?UMT5Stack._prepare_4d_causal_attention_mask_with_cache_positions}>  %.*<*<*>!*C(K*' E*..I** 0Y\j\q\qK!##jjqA 5<< >S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12r6)rpr]rmrqrrrrc) NNNNNNNNNNNNN)F)rGrHrIrJr*ryr~rErr,rer rr staticmethodrr@rrKrLrMs@r4rkrkks !+ "#!!G `#( BellK78BllB B  B  BH4 444{{ 4  4  44r6rkc&h^\rSrSrSrSr\rSS/rU4Sjr Sr Sr S r S r S rS r\S S \\R&S\\R(S\\R&S\\R*S\\R(S\\R(S\\R,S\\\\R(S\\\\R(S\\R,S\\R,S\\S\\S\\S\\S\\R&S\\\R(\44"Sjj5rSrU=r$)!rFia? Examples: ```python >>> from transformers import UMT5Model, AutoTokenizer >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> noisy_text = "UN Offizier sagt, dass weiter werden muss in Syrien." >>> label = " verhandelt" >>> inputs = tokenizer(inputs, return_tensors="pt") >>> labels = tokenizer(label=label, return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"], decoder_input_ids=labels["input_ids"]) >>> hidden_states = outputs.last_hidden_state ```umt5encoder.embed_tokens.weightdecoder.embed_tokens.weightc>[TU]U5 [R"URUR 5Ul[R"U5nSUl SUl SUl [X R 5Ul [R"U5nSUl SUl URUl[X0R 5UlUR#5 gNFT)r)r*rr vocab_sizerWrJcopydeepcopyrris_encoder_decoderrkencodernum_decoder_layersrodecoderrsr0rQencoder_configdecoder_configr3s r4r*UMT5Model.__init__s  ll6#4#4fnnE v.$)!#( ,1) = v.$(!,1)$*$=$=! =  r6cUR$rcrJrxs r4ryUMT5Model.get_input_embeddings {{r6c|XlURRU5 URRU5 grcrJrr~rr|s r4r~UMT5Model.set_input_embeddings+$  )).9 )).9r6cURR(aaURURRUR 5 URUR RUR 5 ggrcrQrM_tie_or_clone_weightsrrmrJrrxs r4 _tie_weightsUMT5Model._tie_weightsP ;; * *  & &t||'@'@$++ N  & &t||'@'@$++ N +r6cUR$rcrrxs r4 get_encoderUMT5Model.get_encoder ||r6cUR$rcrrxs r4 get_decoderUMT5Model.get_decoderrr6cUR5H7up#URRURR U5 M9 g) Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrr attention prune_headsr0heads_to_prunerheadss r4 _prune_headsUMT5Model._prune_headss< +002LE LL  u % / / ; ;E B3r6r5rr4r6rdecoder_head_maskrencoder_outputsrrdecoder_inputs_embedsrrrrrrc\U bU OURRn UbUOURRnUcURUUU UU UUS9nORU(aK[ U[ 5(d6[ US[ U5S:aUSOS[ U5S:aUSOSS9nUSnURUUU U UUUUU U UUUS9 nU(dUU-$[URURURURURURURURS9$) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: ```python >>> from transformers import AutoTokenizer, UMT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for UMT5Model. >>> # This is not needed for torch's UMT5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr5rrrrrrrr!r8rrCr r5rrrrrrrrrrrr)rrdecoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentions)rQrrrrdrlenrrrrrCrr)r0r5rr4r6rrrrrrrrrrrrrCdecoder_outputss r4rEUMT5Model.forwards^b"+!6IDKKD $D>D 'tnD d^!D "!!1!12#D $ uU&&');; <%D D r6rFz< UMT5 Model with a `language modeling` head on top. ) custom_introc(^\rSrSrSrSr/SQrU4SjrSrSr Sr S r S r S r S r\S#S \\R$S\\R&S\\R$S\\R(S\\R&S\\R&S\\R*S\\\\R*S\\\\R*S\\R&S\\R&S\\R$S\\S\\S\\S\\S\\R$S\\\R&\44$Sjj5rS\R*4S jr\S!5rS"rU=r$)$rGia Examples: ```python >>> from transformers import UMT5ForConditionalGeneration, AutoTokenizer >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, text_target=summary, return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ```r)rrzlm_head.weightcL>[TU]U5 URUl[R "UR UR5Ul[R"U5nSUl SUl SUl [X R5Ul[R"U5nSUl SUl URUl[X0R5Ul[R$"URUR SS9UlUR)5 g)NFTrT)r)r*rW model_dimrrrrJrrrrrrkrrrorrVr?rsrs r4r*%UMT5ForConditionalGeneration.__init__s  ll6#4#4fnnE v.$)!#( ,1) = v.$(!,1)$*$=$=! = yy1B1BO  r6cUR$rcrrxs r4ry1UMT5ForConditionalGeneration.get_input_embeddingsrr6c|XlURRU5 URRU5 grcrr|s r4r~1UMT5ForConditionalGeneration.set_input_embeddingsrr6cURR(aaURURRUR 5 URUR RUR 5 ggrcrrxs r4r)UMT5ForConditionalGeneration._tie_weightsrr6cXlgrcr?r|s r4set_output_embeddings2UMT5ForConditionalGeneration.set_output_embeddingss% r6cUR$rcr"rxs r4get_output_embeddings2UMT5ForConditionalGeneration.get_output_embeddingsrr6cUR$rcrrxs r4r(UMT5ForConditionalGeneration.get_encoderrr6cUR$rcrrxs r4r(UMT5ForConditionalGeneration.get_decoderrr6r5rr4r6rrrrrrrlabelsrrrrrrcU bU OURRn UbUOURRnUcURUUU UUUUS9nORU(aK[ U[ 5(d6[ US[ U5S:aUSOS[ U5S:aUSOSS9nUSnU bUcU cURU 5nURUUU U UUUUU UUUUS9 nUSnURR(aUURS--nURU5nSnU b[[S S 9nU RUR5n U"URS UR!S 55U RS 55nU(dU4USS-U-nUbU4U-$U$[#UUUR$UR&UR(UR*UR,UR&UR(S 9 $) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, UMT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer("Studies have shown that good for you", return_tensors="pt").input_ids >>> outputs = model.generate(input_ids) >>> tokenizer.decode(outputs[0], skip_special_tokens=True) ```Nrrr!r8rrrArU ignore_indexr9 losslogitsrr r rr rr )rQrrrrdrr r_rrMrr?r r;rrrrrrCrrr)r0r5rr4r6rrrrrrrr,rrrrrrCrsequence_output lm_logitsr1loss_fctoutputs r4rE$UMT5ForConditionalGeneration.forwards.j"+!6IDKK# UH/oRSTRUR55v M1 g7f)rN) index_selectr;r)r past_statebeam_idxs r4r>UMT5ForConditionalGeneration._reorder_cache..s1ncmU_--aZ=N=N1OPPcms7:)r)rr?reordered_past layer_pasts ` r4_reorder_cache+UMT5ForConditionalGeneration._reorder_caches8)J ncmnn N*r6)rrr?rrJ)NNNNNNNNNNNNNNNNN) rGrHrIrJrrrr*ryr~rr#r&rrrrr,rrrrerrrrrEr9rrCrKrLrMs@r4rGrGs4  Ji0: O &156:8<=A159=7;@D@D59=A-1$(,0/3&*59%_ E,,-_ !!2!23_ $E$4$45 _ !))9)9 : _ E--. _ $E$5$56_ 'u||4_ "%ell(;"<=_ "%ell(;"<=_   1 12_  ((9(9:_ ))*_ D>_ $D>_ 'tn!_ "d^#_ $!!1!12%_ & uU&&'8 9'_ _ D)ELL)r6rGc8^\rSrSrSrSrS/rU4SjrSrSr Sr S r S r \ SS \\R S \\R"S \\R"S\\R"S\\S\\S\\S\\\R"\44Sjj5rSrU=r$)rHia Examples: ```python >>> from transformers import UMT5EncoderModel, AutoTokenizer >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> input_ids = tokenizer(article, return_tensors="pt").input_ids >>> outputs = model(input_ids) >>> hidden_state = outputs.last_hidden_state ```rrc >[TU]U5 [R"URUR 5Ul[R"U5nSUl SUl [X R 5Ul UR5 gNF)r)r*rrrrWrJrrrrrkrrs)r0rQrr3s r4r*UMT5EncoderModel.__init__sf  ll6#4#4fnnE v.#( ,1) =  r6cUR$rcrrxs r4ry%UMT5EncoderModel.get_input_embeddingsrr6cFXlURRU5 grc)rJrr~r|s r4r~%UMT5EncoderModel.set_input_embeddingss$  )).9r6cURR(a1URURRUR 5 ggrc)rQrMrrrmrJrxs r4rUMT5EncoderModel._tie_weightss2 ;; * *  & &t||'@'@$++ N +r6cUR$rcrrxs r4rUMT5EncoderModel.get_encoderrr6cUR5HDup#URRURSRR U5 MF g)rrN)rrrprrrrs r4rUMT5EncoderModel._prune_headssG +002LE LL  u % + +A . < < H H O3r6r5rrrrrrrc fUbUOURRnURUUUUUUUS9nU$)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). Example: ```python >>> from transformers import AutoTokenizer, UMT5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```r)rQrr) r0r5rrrrrrrs r4rEUMT5EncoderModel.forwardsKF&1% -'tn-d^- uU&&'8 9--r6rHz UMT5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c$^\rSrSrS/rSS/rS\4U4Sjjr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ \ RS\ \ RS\ \ RS\ \ RS\ \S\ \S\ \S\ \S\\\44 Sjj5rSrU=r$)UMT5ForSequenceClassificationiFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weightrrrQc>[TU]U5 [U5Ul[ U5UlUR 5 SUlgrG)r)r*rFr2r&classification_headrsmodel_parallelr`s r4r*&UMT5ForSequenceClassification.__init__s>  $V,#9&#A  #r6r5rr4r6rrrrrrr,rrrrrc4UbUOURRnU bSn Uc%U b"[SURR35eUc"U cUc [ S5eUR U5nURUUUUUUUUU U U U UUS9nUSnURURR5RUR5n[[R"URS555S:a [ S5eUR unnnUUSS24R#US U5SS2S SS24nUR%U5nSnU GbU RUR5n URR&cURR(S:XaS URlOyURR(S:aNU R*[R,:XdU R*[R.:XaS URlOS URlURR&S :XaT[15nURR(S:Xa&U"UR35U R355nOU"UU 5nOURR&S :XaG[55nU"UR#S URR(5U R#S 55nO-URR&S :Xa[75nU"UU 5nU(dU4USS-nUbU4U-$U$[9UUUR:UR<UR>UR@URBURDURFS 9 $)ak input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rr4r6rrrrrrrrrrrr!z7All examples must have the same number of tokens.r9 regressionsingle_label_classificationmulti_label_classificationr0)$rQrNotImplementedErrorr3rGrXr_r2eq eos_token_idr;rr r,unique_consecutivesumrrrY problem_typer*r@rrr squeezer rrrr r rr rr )r0r5rr4r6rrrrrrr,rrrrrr3eos_maskrrr1sentence_representationr2r1r5r6s r4rE%UMT5ForSequenceClassification.forwardsR|&1%>KbKbJcd   $)>)F  U !% 1 1) < "" )/#9/!5+'"7/!5##  "!*<< 8 89<<_=S=ST u'' Q8 9A =VW W%4%:%:" A{"1(A+">"C"CJPRT_"`abdfhiai"j))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./Y,F)-)9TGf$ Ev E.#33")"?"?&99$55&-&G&G")"?"?&99  r6)rYrZr2)NNNNNNNNNNNNNNN)rGrHrIrJ"_keys_to_ignore_on_load_unexpectedrr"r*rrr,rrerrrrrrrErKrLrMs@r4rVrVs+s)s&79VW$z$15158<=A,0487;=A59=A-1$(,0/3&*!P E,,-P !.P $E$4$45 P !))9)9 : P ELL) P $ELL1P 'u||4P "$u'8'8"9:P   1 12P  ((9(9:P ))*P D>P $D>P 'tnP d^!P " u55 6#P P r6rVc@^\rSrSrS/rS/rS\4U4Sjjr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ S \ \ S \ \ S\ \\ R\44Sjj5rSrU=r$)rOirWz'transformer.encoder.embed_tokens.weightrQc0>[TU]U5 URUl[U5Ul[ R "UR5Ul[ R"URUR5Ul UR5 grc) r)r*r*rHr2rr[r)r]rVr1rBrsr`s r4r*#UMT5ForTokenClassification.__init__sj   +++F3zz&";";< ))F$6$68I8IJ r6r5rrrr,rrrrc UbUOURRnURUUUUUUUS9n U Sn URU 5n UR U 5n Sn Ub<[ 5n U "U R SUR5UR S55n U(dXSS4nU bU 4U-$U$[U U U RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. N)rrrrrrrr9r8)r1r2rCr) rQrr2r]rBr rr*rrCr)r0r5rrrr,rrrrrCr2r1r5r6s r4rE"UMT5ForTokenClassification.forwards6&1%7 'tn7 d^7  uU\\"$99 :7 7 r6rOc&J^\rSrSrSS/rU4SjrSrSrSrSr S r \ SS \ \ RS \ \ RS \ \ RS \ \ R S\ \ RS\ \ RS\ \ R"S\ \\\ R"S\ \ RS\ \ RS\ \ RS\ \ RS\ \S\ \S\ \S\ \S\\\ R\44"Sjj5rSrU=r$)rIirrcp>[TU]U5 URUl[R "UR UR5Ul[R"U5nSUl SUl SUl [X R5Ul[R"U5nSUl SUl URUl[X0R5UlUR$Ul[R&"URUR$5UlUR+5 gr)r)r*rWrrrrrJrrrrrrkrrrorr*rVr@rsrs r4r*!UMT5ForQuestionAnswering.__init__s  ll6#4#4fnnE v.$)!#( ,1) = v.$(!,1)$*$=$=! =  ++))FNNF4E4EF r6cUR$rcrrxs r4ry-UMT5ForQuestionAnswering.get_input_embeddingsrr6c|XlURRU5 URRU5 grcrr|s r4r~-UMT5ForQuestionAnswering.set_input_embeddingsrr6cURR(aaURURRUR 5 URUR RUR 5 ggrcrrxs r4r%UMT5ForQuestionAnswering._tie_weights"rr6cUR$rcrrxs r4r$UMT5ForQuestionAnswering.get_encoder(rr6cUR$rcrrxs r4r$UMT5ForQuestionAnswering.get_decoder,rr6r5rr4r6rrrrstart_positions end_positionsrrrrrrrcUbUOURRnU bU OURRn U bU bSn Uc"U cUc [S5eUR U5nU bU OURRn UbUOURRnUcUR UUU UUUUS9nORU(aK[ U[5(d6[US[U5S:aUSOS[U5S:aUSOSS9nUSnURUUU SUUUUU UUUS 9 nUSnURU5nURSS S 9unnURS 5R5nURS 5R5nSnU bU b[U R55S:a*U RS 5RUR 5n [U R55S:a*U RS 5RUR 5n URS5nU R#SU5n U R#SU5n [%US 9nU"UU 5nU"UU 5nUU-S- nU(dUU4USS-U-nUbU4U-$U$['UUUUR(UR*UR,UR.UR0UR*UR,S 9 $)aY input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. NFr]rrr!r8r) r5rrrrrrrrrrrr9rr.) r1 start_logits end_logitsrr r rr rr )rQrrrXr_rrdrr rr@splitrgrrr;rrr rrrCrrr)r0r5rr4r6rrrrr~rrrrrrrrCrr3r2rr total_loss ignored_indexr5 start_lossend_lossr6s r4rE UMT5ForQuestionAnswering.forward/s<x&1%)F  U !% 1 1) < !*!6IDKKZ $D>Z 'tn!Z "d^#Z $ uU&&')LL M%Z Z r6rI)rHrGrIrVrOrFr1)Frrrtypingrrrrr,rtorch.nnrr r activationsr cache_utilsr rr generationrmodeling_attn_mask_utilsrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrrr configuration_umt5r"!torch.nn.attention.flex_attentionr#integrations.flex_attentionr$ get_loggerrGrModuler&rOrkrvrrrrr&r1rkrFrGrHrVrOrI__all__rar6r4rs // AA!CC)>.+ !!;J   H %+BII+4 .RYY<"))$9BII9DRYY8bii