fTh0SrSSKrSSKrSSKJrJrJrJr SSKr SSK r SSK r SSK J r SSK Jr SSKJr SSKJrJr SS KJr SS KJrJr SS KJrJrJrJrJr SS KJ r SS K!J"r"J#r#J$r$J%r% SSK&J'r' \#"5(a SSK(J)r) SSK*J+r+ \%RX"\-5r.S\ R^S\0S\04Sjr1"SS\ Rd5r3"SS\ Rh5r5S\50r6"SS\ Rh5r7"SS\ Rh5r8\""SS \ 55r9"S!S"\95r:"S#S$\95r;\""S%S&\955r<\""S'S(9"S)S*\9\55r="S+S,\95r>"S-S.\9\5r?/S/Qr@g)0zPyTorch PEGASUS model.N)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)CacheEncoderDecoderCache)GenerationMixin)AttentionMaskConverter_prepare_4d_attention_mask)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)auto_docstringis_torch_flex_attn_availableis_torchdynamo_compilinglogging) PegasusConfig) BlockMask)make_flex_block_causal_mask input_ids pad_token_iddecoder_start_token_idcURUR5nUSS2SS24R5USS2SS24'X#SS2S4'Uc [S5eUR US:HU5 U$)z) Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)rrr shifted_input_idss d/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/pegasus/modeling_pegasus.pyshift_tokens_rightr*=sz"++IOO<(CRC0668ae4adLMM""#4#[TU]X5 gN)super__init__)selfr/r0r1 __class__s r)r6-PegasusSinusoidalPositionalEmbedding.__init__Qs 6r+cURRup[R"[ U5VVs/sH?n[ U5Vs/sH%oC[R "SSUS--U- 5- PM' snPMA snn5n[ R"XURRSS9nUS-S:XaUS-OUS-S-n[ R"[R"USS2SSS2455USS2SU24'[ R"[R"USS2SSS2455USS2US24'[R"USS9Ulgs snfs snnf) z Identical to the XLM create_sinusoidal_embeddings except features are not interleaved. The cos features are in the 2nd half of the vector. [dim // 2:] i'F)dtype requires_gradrrN)r=)weightr$nparrayrangepowertorchemptyr< FloatTensorsincosr Parameter)r7n_posdimposj position_encoutsentinels r) _init_weight1PegasusSinusoidalPositionalEmbedding._init_weightTs* [[&& xxX]^cXd eXdQTs LABHHUAaL3$677 LXd e kk%DKK,=,=US"Qw!|3!8#(a"..rvvl1add76K/LMAqzM!--bff\!QTT'5J.KLAxyLll3e< M esE ,E6E E input_ids_shapepast_key_values_length position_idsc>UcBUSSupE[R"X"U-[RURRS9n[ TU]U5$)z3`input_ids_shape` is expected to be [bsz x seqlen].Nr;)r<device)rCarangelongr>rVr5forward)r7rRrSrTbszseq_lenr8s r)rY,PegasusSinusoidalPositionalEmbedding.forwardcsX  *2A.LC <<&(HPUPZPZcgcncncucuLw|,,r+)r>r4rN)__name__ __module__ __qualname____firstlineno____doc__intrr6rPrCno_gradSizeTensorrY__static_attributes__ __classcell__r8s@r)r-r-NsN7c7#7HUXM7ei77 = ]]_sw -$zz -CF -ZbchcocoZp -  - -r+r-c^\rSrSrSrSS\S\S\S\S\S\S \\ S \\4U4S jjjr SS \ RS \\ RS\\ S\\ RS\\ RS\S\\ RS\\ R\\ R\\\ R44SjjrSrU=r$)PegasusAttentionqz=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsdropout is_decoderbias is_causalconfig layer_idxc t>[T U]5 XlX lX0lX-UlXplUR U-UR:wa[SURSUS35eUR S-UlX@l X`l Xl Uc>UR(a-[RSURRS35 [ R""XUS9Ul[ R""XUS9Ul[ R""XUS9Ul[ R""XUS9Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gzInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.rq)r5r6rmrnrohead_dimrsr&scalingrprrrtlogger warning_oncer8r^rLineark_projv_projq_projout_proj) r7rmrnrorprqrrrsrtr8s r)r6PegasusAttention.__init__ts  "" !.  MMI %$.. 8MdnnM]$YKr3 }}d* $""     *4>>+B+B*CD,,  ii 4@ ii 4@ ii 4@  )TB r+ hidden_stateskey_value_statespast_key_valueattention_masklayer_head_maskoutput_attentionscache_positionr2c USLnUR5upn URU5RU SURUR5R SS5n XR -n Ub][U[5(aFURRUR5n U(a URnOURnOUnU(aUOUnU(a=Ub:W (a3WRURnURURnOUR!U5nUR#U5nURU SURUR5R SS5nURU SURUR5R SS5nUbNU(dUOSnWR%UUURSU05unnU(aSURUR'XR-SUR4nU R&"U6n UR&"U6nUR&"U6nURS5n[(R*"U UR SS55nUR5XR-U U4:wa.[-SXR-U U4SUR535eUb]USS2SS2SS2SUR.S 24nURXRU U5U-nURXR-U U5n[0R2R5USS 9nUbUR5UR4:wa*[-S UR4SUR535eURSSSS5URXRU U5-nURXR-U U5nU(a=URXRU U5nURXR-U U5nOSn[0R2R7UUR6UR8S 9n[(R*"UU5nUR5XR-XR4:wa7[-S XR-XR4SUR535eURXRXR5nUR SS5nUR'XUR:5nUR=U5nUUU4$)z#Input shape: Batch x Time x ChannelNr"rr;rTz$Attention weights should be of size z , but is rJz/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )sizer~viewrnrw transposerx isinstancer is_updatedgetrtcross_attention_cacheself_attention_cache key_cache value_cacher|r}updatereshaperCbmmr&r$r functionalsoftmaxrorrmr)r7rrrrrrris_cross_attentionrZtgt_len_ query_statesrcurr_past_key_valuecurrent_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r)rYPegasusAttention.forwards.T9',,.a{{=166sBPTP]P]^hhijlmn #ll2  %.*=>>+66::4>>J %*8*N*N'*8*M*M'&4#-?)] ."<,66t~~FJ.::4>>JL^4J;;~6L#b$..$--PZZ[\^_`J',,S"dnndmmT^^_`bcdL)7It+>+E+E dnn?OQ_>`,( L&@DN--dnn=NN*B > #++Z8 ''4 #++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6nn8LgW^7_6`a %%'(*   %+Aq!5Kz7G7G7K5K,KLN',,S..'7SVddL',,S>>-A7GTL}},,\r,B  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVdVdfmov?wwL',,S>>-A7GTL  %1$5$5c>>7T[$\ !055cNN6JGU\]L$( !]]**<4<AAr+) rsrormrwrrrpr|rtrnrr~rxr})FTFNN)NNNNFN)r^r_r`rarbrcfloatboolrrr6rCrfr rrYrgrhris@r)rkrkqsYG  *.#'%C%C%C %C  %C  %C%C'%CC=%C%CT48*.1526"'15pB||pB#5<<0pB! pB !. pB "%,,/ pB pB!.pB u||Xell3XeELL>Q5RR SpBpBr+rkeagerc ^\rSrSrS\4U4SjjrS S\RS\RS\RS\S\R4 S jjr S r U=r $) PegasusEncoderLayerirsc>[TU]5 URUl[UR "URUR URUS9Ul[R"UR5Ul URUl [URUlUR Ul[R""URUR$5Ul[R""UR$UR5Ul[R"UR5Ulg)N)rmrnrors)r5r6d_modelrmPEGASUS_ATTENTION_CLASSES_attn_implementationencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normror activation_function activation_fnactivation_dropoutr{encoder_ffn_dimfc1fc2final_layer_normr7rsr8s r)r6PegasusEncoderLayer.__init__s 263N3NOnn44,,   %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r+rrrrr2cUnURU5nURUUUUS9upn[RR XRUR S9nXQ-nUnUR U5nURURU55n[RR XRUR S9nURU5n[RR XRUR S9nXQ-nUR[R:XaC[R"UR5RS- n[R "X*US9nU4n U(aX4- n U $)aW Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrrri)minmax)rrrrrorrrrrrr<rCfloat16finforclamp) r7rrrrresidualrr clamp_valueoutputss r)rYPegasusEncoderLayer.forward%sV$!11-@ )-')+/ *8* & Q  --m||VZVcVc-d  0  --m< **488M+BC  --m?V?Vaeanan-o /  --m||VZVcVc-d  0   %-- /++m&9&9:>>EK!KK <[YM "   &Gr+) rrrormrrrrrF) r^r_r`rarr6rCrfrrYrgrhris@r)rrs^=}=.#( .||. . .  .  ..r+rc`^\rSrSrSS\S\\4U4SjjjrSS\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\\RS\R4Sjjr Sr U=r$)PegasusDecoderLayeriWrsrtc T>[TU]5 URUl[UR "URUR URSSUUS9UlURUl [URUl URUl [R"UR5Ul[UR "URUR URSUUS9Ul[R"UR5Ul[R&"URUR(5Ul[R&"UR(UR5Ul[R"UR5Ulg)NT)rmrnrorprrrsrt)rorprsrt)r5r6rrmrrdecoder_attention_headsrrror rrrrrr encoder_attnencoder_attn_layer_normr{decoder_ffn_dimrrr)r7rsrtr8s r)r6PegasusDecoderLayer.__init__Xs6 263N3NOnn44,, ~~ #F$>$>?"(";";$&LL$@!5f6Q6QR NN  * *,,  (*||DNN'C$99T^^V-C-CD99V33T^^D " T^^ <r+rrencoder_hidden_statesencoder_attention_maskrcross_attn_layer_head_maskrr use_cacherr2c  Un URU5nURUUUUUU S9upn[RR XRUR S9nX-nSn Ub`Un UR U5nURUUUUUUS9upn[RR XRUR S9nX-nUn URU5nURURU55n[RR XRUR S9nURU5n[RR XRUR S9nX-nU4nU(aXU 4- nU (aX4- nU$)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. )rrrrrrrN)rrrrrr) rrrrrorrrrrrrr)r7rrrrrrrrrrrself_attn_weightscross_attn_weightsrs r)rYPegasusDecoderLayer.forwardwsD!11-@ <@>>'))+/) >G  ( (Gr+) rrrormrrrrrrrr4) NNNNNNFTN)r^r_r`rarrrcr6rCrfr rrYrgrhris@r)rrWs =}=#==D268<9=26=A*.,1$(15T||T!.T ( 5 T !) 6 T "%,,/ T%-U\\$:T!T$D>TD>T!.T TTr+rc \rSrSr\rSrSrSrSS\ \ RS4S\ RS\ RS \ S \ 4 S jjr\S\ RS \S \S\ R"S\ RS\4 Sj5rSrg)PegasusPreTrainedModelimodelTc@URRn[U[R5(aWUR R RSUS9 URb%URR R5 gg[U[5(aUR5 g[U[R5(adUR R RSUS9 URb2UR R URR5 gg[U[R5(aJUR R RS5 URR R5 gg)Nr)meanstd?)rsinit_stdrrr{r>datanormal_rqzero_r-rP Embeddingr1rfill_)r7modulers r) _init_weights$PegasusPreTrainedModel._init_weightss"kk"" fbii ( ( MM   & &CS & 9{{&   &&('  D E E    !  - - MM   & &CS & 9!!- ""6#5#56<<>.  - - MM   $ $S ) KK   " " $.r+rr input_tensorrpast_key_valuesrc URRS:XaUbUS:HR5(aU$gURRS:Xa,[U[R 5(a [ U5nU$UbUR5OSnUb UROSnURRS:Xa5U(d.U(d'[R"UUUURS9(agURnURSn U(aUR5n O5[U[R 5(aURS OXi-S-n URUU U UUURSS 9n URRS:XaZUbWUR R"S ;a=U(d6[R$"U5R&n [R("X5n U $) Nflash_attention_2rflex_attentionrFsdpa) inputs_embedsrS is_trainingrr")sequence_length target_lengthr<r batch_size)cudaxpunpu)rsranyrrCrfrget_seq_lengthis_compileabler_ignore_causal_mask_sdparr<r$get_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrVtyperr_unmask_unattended) r7rrrrrpast_seen_tokensusing_compilable_cacher<rr causal_mask min_dtypes r)_update_causal_mask*PegasusPreTrainedModel._update_causal_masks ;; + +/B B)~/D.I.I.K.K%% ;; + +/? ?.%,,77!<^!L! ! @O?Z?99;`aCRC^!?!?di ;; + +v 5>T]n%>>*'7 MM  ""&,,Q/ !+??AMnell;;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCK[Kr+rrr<rc UbUR5S:XaUnU$[R"U5Rn[R"X4XUR S9nUS:wa[R "USS9nU[R"X$R S9URSS5:-nUSSSS2SS24RUSSS5nUbUR5nURSn USS2SS2SS2SU 24USS2SSSS24RUR 5-n U S:Hn USS2SS2SS2SU 24RX5USS2SS2SS2SU 24'U$) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuer<rVr)diagonalrVr"r)rJrCrrfullrVtriurWrexpandr%r$to masked_fill) rrrr<rrkwargsrr mask_length padding_masks r)rLPegasusPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_position)s}>  %.*<*<*>!*C(K*' E*..I** 0Y\j\q\qK!##jjqA 5<< >S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12r+Nr)r^r_r`rar config_classbase_model_prefixsupports_gradient_checkpointingrrrCrfr rr  staticmethodrcr<rrgrr+r)rrs L&*#%.#( BellK78BllB B  B  BH4 444{{ 4  4  44r+rc^\rSrSrSrS S\S\\R4U4Sjjjr S\ 4Sjr S\R4S jr S S jr S rU=r$)PegasusEncoderibz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`PegasusEncoderLayer`]. Args: config: PegasusConfig embed_tokens (nn.Embedding): output embedding rs embed_tokensc>[TU]U5 URUlURUlUR nUR UlURUl UR(a[R"U5OSUl UbX lO0[R "UR"X0R5Ul[%URUUR5Ul[R("[+UR,5Vs/sHn[/U5PM sn5Ul[R2"UR 5UlSUlUR95 gs snf)NrF)r5r6roencoder_layerdrop layerdroprrr1max_position_embeddingsmax_source_positionsscale_embeddingmathsqrt embed_scaler rr vocab_sizer-embed_positions ModuleListrAencoder_layersrlayersr layer_normgradient_checkpointing post_init)r7rsr rmrr8s r)r6PegasusEncoder.__init__ls  ~~ 11NN !..$*$B$B!393I3I499Y/s  # ,  " V->-> K[K[ \D C  * *      mm%PVPePeJf$gJfQ%8%@Jf$gh ,,v~~6&+#  %hsE4new_num_position_embeddingsc\[RSUS35 XRl[ URRURR UR 5UlURR5 URRUR5 g Resizes position embeddings matrix of the model if `new_num_position_embeddings != config.max_position_embeddings`. Arguments: new_num_position_embeddings (`int`): The number of new position embeddings. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. z(Setting `config.max_position_embeddings=z`...N ryinforsr$r-rr1r+rPrrVr7r3s r)resize_position_embeddings)PegasusEncoder.resize_position_embeddings  >?Z>[[_`a.I +C KK / / KK       ))+  ,r+r2cUR$z( Returns the position embeddings matrix r+r7s r)get_position_embeddings&PegasusEncoder.get_position_embeddings###r+cUbUOURRnUbUOURRnUbUOURRnUbUb [ S5eUb7UR X5 UR 5nURSUS5nO"UbUR 5SSnO [ S5eUcURU5UR-nURU5n XI-n [RRXRURS9n Ub[X$R 5nU(aSOSn U(aSOSn Ub`UR 5S[#UR$5:wa6[ S[#UR$5S UR 5SS 35e['UR$5HupU(aX4-n S nUR(a([(R*"/5nUUR,:aS nU(aS nO^UR.(a8UR(a'UR1UR2U UUbX=OSU5nOU"U UUbX=OSUS9nUSn U(dMU US4-n M UR5U 5n U(aX4-n U(d[7SXU 455$[9XU S9$)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timer"z5You have to specify either input_ids or inputs_embedsrrrz&The head_mask should be specified for  layers, but it is for .FT)NN)rrrc3.# UH ocMUv M g7fr4r.0vs r) )PegasusEncoder.forward..&se$Sq$Ss last_hidden_stater attentions)rsroutput_hidden_statesuse_return_dictr&%warn_if_padding_and_no_attention_maskrrr r)r+rrrorrr<lenr. enumeraterCrandr#r0_gradient_checkpointing_func__call__r/tupler)r7rr head_maskrrrP return_dict input_shape embed_posrencoder_statesall_attentionsidx encoder_layerto_dropdropout_probability layer_outputss r)rYPegasusEncoder.forwards\2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K!r;r?;I  &',,.s3KTU U   --i84;K;KKM((5 %1  --m||VZVcVc-d   %7H[H[\N30d  ~~"c$++&66 >Ne]N$See e+Vd  r+) ror+r)r r0r/r#r.r%r1r4)NNNNNNN)r^r_r`rarbrrrrr6rcr:rArYrgrhris@r)rrbsh}HR\\[TU]U5 URUlURUlUR UlURUlUR(a [R"UR5OSUl UbX lO;[R "UR"URUR 5Ul[%URURUR 5Ul[R("[+UR,5Vs/sH n[/XS9PM sn5Ul[R2"UR5UlSUlUR95 gs snf)Nr)rtF)r5r6rodecoder_layerdropr#rr1r$max_target_positionsr&r'r(rr)r rrr*r-r+r,rAdecoder_layersrr.rr/r0r1)r7rsr ir8s r)r6PegasusDecoder.__init__5s  ~~ 11!..$*$B$B!8>8N8N499V^^4TW  # ,  " V->->PTP`P` aD C  * * NN     mmW\]c]r]rWs$tWsRS%8%MWs$tu ,,v~~6&+#  %us)FcUR$r4r r@s r)get_input_embeddings#PegasusDecoder.get_input_embeddingsNs   r+cXlgr4rnr7values r)set_input_embeddings#PegasusDecoder.set_input_embeddingsQs!r+r3c\[RSUS35 XRl[ URRURR UR 5UlURR5 URRUR5 gr5r7r9s r)r:)PegasusDecoder.resize_position_embeddingsTr<r+r2cUR$r>r?r@s r)rA&PegasusDecoder.get_position_embeddingslrCr+cF U bU OURRn U bU OURRn U bU OURRn U bU OURRn UR (a/UR (aU (a[RS5 Sn USLUSL- (a [S5eUbURSURS5nUcURU5nXR-nSnU (aB[U[5(d-Sn[RS5 [ R""U5nUR%5SSunnUbUR'5OSnU c#[(R*"UUU-UR,S 9n Uc4[/5(d%UU-n[(R0"UUUR,S 9n[U[ 5(a UR2OUnUR5UUU UU 5nUbUb[7XHR8US 9nUR;UU4UU S 9nUU-n[<R>RAUUR@UR S 9nU (aS OSnU (aS OSnU (aUbS OSnSn[CXV/SS/5HnunnUcM UR%5S[EURF5:wdM7[SUS[EURF5SUR%5SS35e [IURF5HunnU (aUU4- nUR (a([(RJ"/5nUURL:aMLUR (aGUR (a6UROURPUUUUUbUUOSUbUUOSSU U U 5 n OU"UUUUUbUUOSUbUUOSUU U U S9 n U SnU (aU U (aSOSnU (dMUU S4- nUcMUU S4- nGM URSU5nU (aUU4- nU (aUOSn!U(aURU5n!U (d[WSUU!UUU455$[YUU!UUUS9$)ax Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer"TzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.rr)r)rTrrrYcross_attn_head_maskzThe `z` should be specified for rErF) rrrrrrrrrr rr;c30# UH nUcMUv M g7fr4rrHs r)rK)PegasusDecoder.forward..[smAms )rNrrrOcross_attentions)-rsrrPrrQr0rryrzr&rr$r r)rr r from_legacy_cacherrrCrWrVronesrr rr<r+rrroziprSr.rTrUr#rVrWr/to_legacy_cacherXr)"r7rrrrrYr{rrrrrPrZrreturn_legacy_cacher seq_lengthrSmask_seq_lengthself_attn_cacher positionsrall_hidden_statesall_self_attnsall_cross_attentionsnext_decoder_cache attn_mask mask_namer_ decoder_layerrbrc next_caches" r)rYPegasusDecoder.forwardrsh2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK?I   --i8M&(8(88 $ Z??"&    \  2CCOTO!.!3!3!5cr!: JETE`!?!?!Afg  !"\\&(>(KTaThThN  !*B*D*D4zAO"ZZ OML`L`aN/+>??  0 0   ..        ! ,1G1S%?&(;(;Z& " ((*j)ACYhv(w % 1  --mt||VZVcVc-d #7BD0d&7>#A&#dkk*::$ {*DSEUDVW%NN,Q/03%r#,DKK"8 C#!m%55!}}&+jjn#&7**t}} $ A A!**!)*&/&;IcN1E1Q(-W[%" ! !.!#.*?+A7@7LYs^RV5I5U,S1[_#2&7'#1 ! *!,M%28I1q%Q"  =#3"55(4(]1-=,??(_#9b 6  -!1 1 +4'$ (88:J'5FXlm  9+&+%1   r+) ror+r)r r0r/r#r.rir1r4) NNNNNNNNNNNNN)r^r_r`rarbrrrrr6rortrcr:rArYrgrhris@r)rfrf,s}HR\\[TU]U5 URURp2[R "X1R U5Ul[XR5Ul [XR5Ul UR5 gr4) r5r6rr*rrrsharedrencoderrfdecoderr1)r7rsr1r*r8s r)r6PegasusModel.__init__mse  "("5"5v7H7HZll:~~{K %fkk: %fkk:  r+cUR$r4)rr@s r)ro!PegasusModel.get_input_embeddingsys {{r+c|XlURURlURURlgr4)rrr rrrs r)rt!PegasusModel.set_input_embeddings|s' $(KK !$(KK !r+cUR$r4)rr@s r) get_encoderPegasusModel.get_encoder ||r+cUR$r4rr@s r) get_decoderPegasusModel.get_decoderrr+r3cXRlURRU5 URRU5 gr6N)rsr$rr:rr9s r)r:'PegasusModel.resize_position_embeddingss5/J + //0KL //0KLr+r2cjURR5URR54$r>)rrArr@s r)rA$PegasusModel.get_position_embeddingss) 446 8\8\8^__r+rrdecoder_input_idsdecoder_attention_maskrYdecoder_head_maskr{encoder_outputsrrdecoder_inputs_embedsrrrPrZrcU bU OURRn UbUOURRnU bU OURRn UbUOURRnUcUR UUUU U UUS9nORU(aK[ U[5(d6[US[U5S:aUSOS[U5S:aUSOSS9nURUUUSUUUU U U U UUUS9 nU(dUU-$[URURURURURURURURS9$) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Pegasus uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: ```python >>> from transformers import AutoTokenizer, PegasusModel >>> tokenizer = AutoTokenizer.from_pretrained("google/pegasus-large") >>> model = PegasusModel.from_pretrained("google/pegasus-large") >>> inputs = tokenizer("Studies have been shown that owning a dog is good for you", return_tensors="pt") >>> decoder_inputs = tokenizer("Studies show that", return_tensors="pt") >>> outputs = model(input_ids=inputs.input_ids, decoder_input_ids=decoder_inputs.input_ids) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 4, 1024] ```N)rrrYrrrPrZrrr;rM rrrrrYr{rrrrrPrZr)rNrdecoder_hidden_statesdecoder_attentionsr~encoder_last_hidden_staterencoder_attentions)rsrrPrrQrrrrSrrrNrrrOr~)r7rrrrrYrr{rrrrrrrPrZrdecoder_outputss r)rYPegasusModel.forwardsv2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKB>B048<$(,0/3&*15#p ELL)p !.p $ELL1 p !) 6 p ELL) p $ELL1p 'u||4p "%(9(9":;p "%(9(9":;p   -p  ( 5p D>p $D>p 'tnp d^!p "!.#p $ u(( )%p p r+rzY The PEGASUS Model with a language modeling head. Can be used for summarization. ) custom_introc(^\rSrSrSrS/r/SQrS\4U4SjjrSr Sr S*S \ S \ \ S \ S \R4U4SjjjrS \ S S 4SjrSrSrS\ 4SjrS \\R4Sjr\S+S\ \R2S\ \R2S\ \R2S\ \R2S\ \R2S\ \R2S\ \R2S\ \\R4S\ \\R4S\ \R2S\ \R2S \ \R2S!\ \ S"\ \ S#\ \ S$\ \ S%\ \R2S \\\44$S&jj5rS \R24S'jr\S(5r S)r!U=r"$),PegasusForConditionalGenerationirfinal_logits_bias)rrlm_head.weightrscv>[TU]U5 [U5UlUR S[ R "SURRR455 [R"URURRRSS9Ul UR5 g)NrrFrv)r5r6rrregister_bufferrCzerosrnum_embeddingsrr{rlm_headr1rs r)r6(PegasusForConditionalGeneration.__init__s  !&)  0%++q$**BSBSBbBb>c2deyy1B1B1Q1QX]^  r+c6URR5$r4)rrr@s r)r+PegasusForConditionalGeneration.get_encoder%zz%%''r+c6URR5$r4)rrr@s r)r+PegasusForConditionalGeneration.get_decoder(rr+Nnew_num_tokenspad_to_multiple_of mean_resizingr2cx>[TU]XU5nURURRS5 U$)Nr)r5resize_token_embeddings_resize_final_logits_biasr>r$)r7rrrnew_embeddingsr8s r)r7PegasusForConditionalGeneration.resize_token_embeddings+s<8]jk &&~'<'<'B'B1'EFr+c,URRSnX::aURSS2SU24nON[R"SX- 4URRS9n[R "URU/SS9nUR SU5 g)Nr"rrrr)rr$rCrrVcatr)r7rold_num_tokensnew_bias extra_biass r)r9PegasusForConditionalGeneration._resize_final_logits_bias2s//55b9  +--a..@AHa)H%IRVRhRhRoRopJyy$"8"8*!E1MH 0(;r+cUR$r4rr@s r)get_output_embeddings5PegasusForConditionalGeneration.get_output_embeddings;rr+cXlgr4rr7rs r)set_output_embeddings5PegasusForConditionalGeneration.set_output_embeddings>% r+r3cXRlURRR U5 URR R U5 gr)rsr$rrr:rr9s r)r::PegasusForConditionalGeneration.resize_position_embeddingsAsA/J + 556QR 556QRr+cURRR5URRR54$r>)rrrArr@s r)rA7PegasusForConditionalGeneration.get_position_embeddingsRs5 ""::P>P>h>h>jkkr+rrrrrYrr{rrrrlabelsrrrPrZrcUbUOURRnU bXU (a[RS5 Sn Uc7U c4[ XRR URR 5nURUUUUUUUUU U U U UUUUS9nURUS5UR-nSnU bF[5nU"URSURR5U RS55nU(dU4USS-nUbU4U-$U$[UUURURUR UR"UR$UR&UR(S9 $) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Pegasus uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Summarization: ```python >>> from transformers import AutoTokenizer, PegasusForConditionalGeneration >>> model = PegasusForConditionalGeneration.from_pretrained("google/pegasus-xsum") >>> tokenizer = AutoTokenizer.from_pretrained("google/pegasus-xsum") >>> ARTICLE_TO_SUMMARIZE = ( ... "PG&E stated it scheduled the blackouts in response to forecasts for high winds " ... "amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were " ... "scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow." ... ) >>> inputs = tokenizer(ARTICLE_TO_SUMMARIZE, max_length=1024, return_tensors="pt") >>> # Generate Summary >>> summary_ids = model.generate(inputs["input_ids"]) >>> tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "California's largest electricity provider has turned off power to hundreds of thousands of customers." ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)rrrrrYrr{rrrrrrPrZrrr"r) losslogitsrrrr~rrr)rsrQrywarningr*rr rrrrrr*rrrrr~rrr)r7rrrrrYrr{rrrrrrrrPrZrr lm_logitsmasked_lm_lossloss_fctoutputs r)rY'PegasusForConditionalGeneration.forwardXsH&1%# UH/oRSTRUR55v M1 g7fr] index_selectrrVrI past_statebeam_idxs r)rKAPegasusForConditionalGeneration._reorder_cache..s1rcqU_--aZ=N=N1OPPcq7:r;rXrrreordered_past layer_pasts ` r)_reorder_cache.PegasusForConditionalGeneration._reorder_cachesO)J rcmnpopcqrrQR.! N* r+rr)NT)NNNNNNNNNNNNNNNNN)#r^r_r`rar_keys_to_ignore_on_load_missingrrr6rrrcrrrrrrrrr:rrArrCrfrErrrYrrrrgrhris@r)rrs~  ':&;#i}((dh!7?}\` <<<&ScS"lr||)<l -115489=,0487;>B>B048<)-$(,0/3&*15%u ELL)u !.u $ELL1 u !) 6 u ELL) u $ELL1u 'u||4u "%(9(9":;u "%(9(9":;u   -u  ( 5u &u D>u $D>u 'tn!u "d^#u $!.%u & uo% &'u u nhELLhr+rc2^\rSrSrSrU4SjrSrSrU=r$)PegasusDecoderWrapperiz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cD>[TU]U5 [U5Ulgr4)r5r6rfrrs r)r6PegasusDecoderWrapper.__init__s  %f- r+c&UR"U0UD6$r4r)r7argsrs r)rYPegasusDecoderWrapper.forwards||T,V,,r+r) r^r_r`rarbr6rYrgrhris@r)rrs .--r+rc".^\rSrSrS/rU4SjrSrSrSrSr Sr S r S \ R4S jrS \4S jr\SS\\R(S\\R*S\\R,S\\R,S\\R*S\\R*S\\\R,S\\R,S\\R(S\\S\\S\\S\\S\\R(S \\\44Sjj5r\S5rSrU=r $) PegasusForCausalLMirc >[R"U5nSUlSUl[TU]U5 [ U5Ul[R"URURSS9Ul UR5 g)NTFrv)copydeepcopyrpis_encoder_decoderr5r6rrrr{ hidden_sizer*rr1rs r)r6PegasusForCausalLM.__init__sfv& $)!  *62 yy!3!3V5F5FUS  r+cBURRR$r4rrr r@s r)ro'PegasusForCausalLM.get_input_embeddingsszz!!...r+c8XRRlgr4rrrs r)rt'PegasusForCausalLM.set_input_embeddingss*/ 'r+cUR$r4rr@s r)r(PegasusForCausalLM.get_output_embeddingsrr+cXlgr4rrs r)r(PegasusForCausalLM.set_output_embeddingsrr+c$XRlgr4rr)r7rs r) set_decoderPegasusForCausalLM.set_decoder s $ r+c.URR$r4rr@s r)rPegasusForCausalLM.get_decoder szz!!!r+r2cJURRR5$r>)rrrAr@s r)rA*PegasusForCausalLM.get_position_embeddingsszz!!99;;r+r3cnXRlURRR U5 gr)rsr$rrr:r9s r)r:-PegasusForCausalLM.resize_position_embeddingss(/J + 556QRr+rrrrrYr{rrrrrrPrZrcU bU OURRn U bU OURRn U bU OURRn URR UUUUUUUUU U U U US9 nUR US5nSnU baU RUR5n [5nU"URSURR5U RS55nU (dU4USS-nUbU4U-$U$[UUURURURUR S9$)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PegasusForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("google/pegasus-large") >>> model = PegasusForCausalLM.from_pretrained("google/pegasus-large", add_cross_attention=False) >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```Nrrr"r)rrrrrOr~)rsrrPrQrrrrrVrrr*rrrrOr~)r7rrrrrYr{rrrrrrPrZrrrrrrs r)rYPegasusForCausalLM.forward%s[^2C1N-TXT_T_TqTq$8$D $++JjJj &1%# UH/oRSTRUR55v M1 g7fr]rrs r)rK4PegasusForCausalLM._reorder_cache..s1ncmU_--aZ=N=N1OPPcmrrrs ` r)r!PegasusForCausalLM._reorder_caches8)J ncmnn N*r+r)NNNNNNNNNNNNNN)!r^r_r`rarr6rortrrrrrrrArcr:rrrC LongTensorrfrErrrrrrYrrrgrhris@r)r r s*+ /0&%"<< ScS 1515=A>B,07;=A59-1$(,0/3&*59W E,,-W !.W  ((9(9: W !)):): ; W ELL) W 'u||4W "$u'8'8"9:W   1 12W ))*W D>W $D>W 'tnW d^W !!1!12W u77 8!W W rr+r )r rrr)Arbr r'typingrrrrnumpyr?rCtorch.utils.checkpointrtorch.nnr activationsr cache_utilsr r generationr modeling_attn_mask_utilsrrmodeling_outputsrrrrrmodeling_utilsrutilsrrrrconfiguration_pegasusr!torch.nn.attention.flex_attentionrintegrations.flex_attentionr get_loggerr^ryrfrcr*rr-Modulerkrrrrrrfrrrr __all__rr+r)r<s // %!5). 1 !!;J   H % %,, c [^ "-2<<-FZBryyZBz%&67A"))AJt"))tnP_PPfG +G Tz +z z e )e e P E&