o ZhS5@sdZddlZddlZddlmZmZmZmZddlZ ddl Z ddl Z ddl m Z ddl mZddlmZddlmZmZdd lmZdd lmZmZdd lmZmZmZmZmZdd lm Z dd l!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'e#rddl(m)Z)ddl*m+Z+e%,e-Z.de j/de0de0fddZ1Gddde j2Z3Gddde j4Z5Gddde j4Z6de5iZ7Gddde j4Z8e"Gd d!d!e Z9Gd"d#d#e9Z:Gd$d%d%e9Z;e"Gd&d'd'e9ZGd.d/d/e9eZ?gd0Z@dS)1z=PyTorch MarianMTModel model, ported from the Marian C++ repo.N)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)CacheEncoderDecoderCache)GenerationMixin)AttentionMaskConverter_prepare_4d_attention_mask)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)auto_docstringis_torch_flex_attn_availableis_torchdynamo_compilinglogging) MarianConfig) BlockMask)make_flex_block_causal_mask input_ids pad_token_iddecoder_start_token_idcCsh||j}|ddddf|ddddf<||dddf<|dur*td||dk||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i)Z new_zerosshapeclone ValueErrorZ masked_fill_)rrrZshifted_input_idsr$Y/var/www/auris/lib/python3.10/site-packages/transformers/models/marian/modeling_marian.pyshift_tokens_right=s (r&c steZdZdZddededeeddffdd Zd d Ze dd ej d edeej dej ffdd Z Z S)#MarianSinusoidalPositionalEmbeddingzDThis module produces sinusoidal positional embeddings of any length.N num_positions embedding_dim padding_idxreturncst||dSN)super__init__)selfr(r)r* __class__r$r%r.Psz,MarianSinusoidalPositionalEmbedding.__init__c s|jj\}tfddt|D}tj||jjdd}ddkr)dndd}tt |dddddf|ddd|f<tt |dddddf|dd|df<t j |dd |_dS) z Identical to the XLM create_sinusoidal_embeddings except features are not interleaved. The cos features are in the 2nd half of the vector. [dim // 2:] cs$g|]fddtDqS)c s(g|]}tdd|dqS)i')nppower).0j)dimposr$r% Zs(zOMarianSinusoidalPositionalEmbedding._init_weight...)range)r5r7)r8r%r9Zs$zDMarianSinusoidalPositionalEmbedding._init_weight..F)dtype requires_gradr2rrN)r=) weightr!r3arrayr:torchemptyr< FloatTensorsincosr Parameter)r/Zn_posZ position_encoutsentinelr$r;r% _init_weightSs  22z0MarianSinusoidalPositionalEmbedding._init_weightrinput_ids_shapepast_key_values_length position_idscs@|dur|dd\}}tj|||tj|jjd}t|S)z3`input_ids_shape` is expected to be [bsz x seqlen].Nr2)r<device)r@arangelongr>rLr-forward)r/rIrJrKbszZseq_lenr0r$r%rObs  z+MarianSinusoidalPositionalEmbedding.forwardr,rN)__name__ __module__ __qualname____doc__intrr.rHr@Zno_gradSizeTensorrO __classcell__r$r$r0r%r'Ms$r'cseZdZdZ      ddededed ed ed ed eed eeffdd Z      dde j dee j dee dee j dee j dedee j de e j ee j ee e j ffddZZS)MarianAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsdropout is_decoderbias is_causalconfig layer_idxc st||_||_||_|||_||_|j||jkr*td|jd|d|jd|_||_ ||_ ||_ |durK|j rKt d|jjdtj|||d|_tj|||d|_tj|||d|_tj|||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gzInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.r`)r-r.r\r]r^head_dimrbr#scalingr_rarclogger warning_oncer1rRrLineark_projv_projq_projout_proj) r/r\r]r^r_r`rarbrcr0r$r%r.ss0    zMarianAttention.__init__ hidden_stateskey_value_statespast_key_valueattention_masklayer_head_maskoutput_attentionscache_positionr+cCsv|du}|\} } } ||| d|j|jdd} | |j} |dur=t|tr;|j |j } |r7|j }n|j }n|}|rA|n|}|rX|durX| rX|j|j }|j|j }nE||}||}|| d|j|jdd}|| d|j|jdd}|dur|s|nd}||||j d|i\}}|rd|j |j <| |jd|jf}| j|} |j|}|j|}|d}t| |dd}|| |j| |fkrtd| |j| |fd||dur|ddddddd|jd f}|| |j| ||}|| |j| |}tjj|dd }|durN||jfkr3td |jfd||dddd|| |j| |}|| |j| |}|re|| |j| |}|| |j| |}nd}tjj||j|jd }t||}|| |j| |jfkrtd | |j| |jfd||| |j| |j}|dd}|| | |j}||}|||fS)z#Input shape: Batch x Time x ChannelNr rr2rtTz$Attention weights should be of size z , but is r;z/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )sizerlviewr]reZ transposerf isinstancer is_updatedgetrcZcross_attention_cacheself_attention_cacheZ key_cacheZ value_cacherjrkupdatereshaper@Zbmmr#r!r functionalZsoftmaxr^rxr\rm)r/rnrorprqrrrsrtZis_cross_attentionrPtgt_len_Z query_statesr|Zcurr_past_key_valueZcurrent_statesZ key_statesZ value_statesZ proj_shapeZsrc_len attn_weightsZattn_weights_reshapedZ attn_probsZ attn_outputr$r$r%rOs"             &  "    zMarianAttention.forward)r[FTFNN)NNNNFN)rRrSrTrUrVfloatboolrrr.r@rXr rrOrYr$r$r0r%rZps` * rZcsheZdZddedeeffdd Z ddejdejd ejd ee d e ejeejff d d Z Z S)MarianEncoderLayerNrbrccst|j|_t|j|j|j|j||d|_t |j|_ |j |_ t |j|_|j|_t |j|j|_t |j|j|_t |j|_dS)N)r\r]r^rbrc)r-r.d_modelr\MARIAN_ATTENTION_CLASSES_attn_implementationZencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normr^r activation_function activation_fnactivation_dropoutriZencoder_ffn_dimfc1fc2final_layer_normr/rbrcr0r$r%r.s  zMarianEncoderLayer.__init__Frnrqrrrsr+c Cs|}|j||||d\}}}tjj||j|jd}||}||}|}|||}tjj||j|jd}| |}tjj||j|jd}||}| |}|j t j krvt |sdt |rvt |j jd}t j|| |d}|f} |r| |f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rnrqrrrsrvi)minmax)rrrr^rxrrrrrrr<r@Zfloat16isinfanyisnanfinforclamp) r/rnrqrrrsresidualrrZ clamp_valueoutputsr$r$r%rO"s8        zMarianEncoderLayer.forwardr,F) rRrSrTrrrVr.r@rBrrrOrYr$r$r0r%rsreagercseZdZddedeeffdd Z         ddejd eejd eejd eejd eejd eejdee dee dee deejde ej ee ej ej fffddZ ZS)MarianDecoderLayerNrbrcc st|j|_t|j|j|j|jdd||d|_|j |_ t |j |_ |j |_ t|j|_t|j|j|j|jd||d|_t|j|_t|j|j|_t|j|j|_t|j|_dS)NT)r\r]r^r_rarbrc)r^r_rbrc)r-r.rr\rrZdecoder_attention_headsrrr^r rrrrrr encoder_attnencoder_attn_layer_normriZdecoder_ffn_dimrrrrr0r$r%r.Zs6  zMarianDecoderLayer.__init__FTrnrqencoder_hidden_statesencoder_attention_maskrrcross_attn_layer_head_maskrprs use_cachertr+c Cs |} |j|||||| d\}} }tjj||j|jd}| |}||}d} |durN|} |j||||||d\}} }tjj||j|jd}| |}||}|} || |}tjj||j |jd}| |}tjj||j|jd}| |}| |}|f}|r|| | f7}| r||f7}|S)a8 Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. )rnrprqrrrsrtrvN)rnrorqrrrprs) rrrr^rxrrrrrrrr)r/rnrqrrrrrrprsrrtrZself_attn_weightsZcross_attn_weightsrr$r$r%rOysN"        zMarianDecoderLayer.forwardr,) NNNNNNFTN)rRrSrTrrrVr.r@rXr rrrBrOrYr$r$r0r%rYsD"    rc @seZdZeZdZdZdZdZde e j e j e ffddZ dde ejdfd ejd ejd ed ef d dZedejdededejd ejdef ddZeddZdS)MarianPreTrainedModelmodelTmodulecCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|t r-| dSt|tj rN|jjjd|d|j durL|jj|j  dSdSt|tjrc|jjd|jj dSdS)Nr[)meanstd?)rbZinit_stdr{rrir>dataZnormal_r`Zzero_r'rH Embeddingr*rZfill_)r/rrr$r$r% _init_weightss"       z#MarianPreTrainedModel._init_weightsFrqr input_tensorrtpast_key_valuesrsc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) NZflash_attention_2r[Zflex_attentionrFZsdpa) inputs_embedsrJZ is_trainingrr )sequence_length target_lengthr<rt batch_size)cudaZxpuZnpu)rbrrr{r@rXrget_seq_lengthZis_compileabler Z_ignore_causal_mask_sdparxr<r!Zget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrLtyperrZ_unmask_unattended) r/rqrrtrrsZpast_seen_tokensZusing_compilable_cacher<rr causal_mask min_dtyper$r$r%_update_causal_masksT           z)MarianPreTrainedModel._update_causal_maskrrr<rc KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valuer<rLr)ZdiagonalrLr r)r7r@rrfullrLZtriurMrexpandr"r!toZ masked_fill) rqrrr<rtrkwargsrrZ mask_lengthZ padding_maskr$r$r%r,s,  $ 6  zKMarianPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_positioncCs@|jj}tjgddddd|gg|jd}||||d}|S)N)r rr2r r2r)rqrdecoder_input_ids)rbrr@ZtensorrLne)r/Z pad_tokenr dummy_inputsr$r$r%rds"z"MarianPreTrainedModel.dummy_inputsNr)rRrSrTrZ config_classbase_model_prefixZsupports_gradient_checkpointingZ_supports_cache_classZ_supports_static_cacherrrirr'rr@rXr rr staticmethodrVr<rpropertyrr$r$r$r%rsF  D 6rcseZdZdZddedeejffdd ZddZ d d Z       dd ee j d ee j d ee j dee jdeedeedeedeee j effddZZS) MarianEncoderz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`MarianEncoderLayer`]. Args: config: MarianConfig embed_tokens (nn.Embedding): output embedding Nrb embed_tokenscstj|_j|_j}j|_j|_ j r!t |nd|_ |dur,||_n tj||j|_tj||j|_tfddtjD|_d|_|dS)Nrcsg|]}tqSr$)r)r5rrbr$r%r9sz*MarianEncoder.__init__..F)r-r.r^Zencoder_layerdrop layerdroprrr*max_position_embeddingsZmax_source_positionsscale_embeddingmathsqrt embed_scalerrr vocab_sizer'embed_positions ModuleListr:Zencoder_layerslayersgradient_checkpointing post_init)r/rbrr\r0rr%r.zs    zMarianEncoder.__init__cC|jSr,rr/r$r$r%get_input_embeddingsz"MarianEncoder.get_input_embeddingscC ||_dSr,rr/valuer$r$r%set_input_embeddings z"MarianEncoder.set_input_embeddingsrrq head_maskrrsoutput_hidden_states return_dictr+cCsJ|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|durA||||}|d|d}n|durN|dd}ntd|dur^|||j }| |} || } t j j | |j |jd} |dur|t||j}|rdnd} |rdnd} |dur|dt|jksJdt|jd |dd t|jD]X\} }|r| | f} d }|jrtg}||jkrd }|rd }n-|jr|jr||j| ||dur|| nd|}n|| ||dur|| nd|d}|d} |r| |df} q|r| | f} |stdd| | | fDSt| | | dS)a~ Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timer z5You have to specify either input_ids or inputs_embedsrvr$rz&The head_mask should be specified for  layers, but it is for .FT)NN)rrrsrcs|] }|dur|VqdSr,r$r5vr$r$r% sz(MarianEncoder.forward..last_hidden_statern attentions)rbrsruse_return_dictr#Z%warn_if_padding_and_no_attention_maskryrzrrrrrr^rxrr<lenr enumerater@randrr_gradient_checkpointing_func__call__tupler)r/rrqrrrsrrZ input_shapeZ embed_posrnZencoder_statesZall_attentionsidxZ encoder_layerZto_dropdropout_probability layer_outputsr$r$r%rOsz.          zMarianEncoder.forwardr,)NNNNNNN)rRrSrTrUrrrrr.rrr@ LongTensorrXrBrrrrrOrYr$r$r0r%rps:  rcseZdZdZddedeejffdd ZddZ d d Z             dd ee j d ee j d ee jdee j dee j dee j deeee jdee jdeedeedeedeedee j deee j effddZZS) MarianDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`MarianDecoderLayer`] Args: config: MarianConfig embed_tokens (nn.Embedding): output embedding Nrbrcstj|_j|_j|_j|_j rt j nd|_ |dur*||_n tjj |j|_tjj |j|_tfddtjD|_d|_|dS)Nrcsg|]}t|dqS))rc)r)r5irr$r%r93sz*MarianDecoder.__init__..F)r-r.r^Zdecoder_layerdroprrr*rZmax_target_positionsrrrrrrrrdecoder_vocab_sizer'rrr:Zdecoder_layersrrr)r/rbrr0rr%r.#s    zMarianDecoder.__init__cCrr,rrr$r$r%r9rz"MarianDecoder.get_input_embeddingscCrr,rrr$r$r%r<rz"MarianDecoder.set_input_embeddingsrrqrrrcross_attn_head_maskrrrrsrrrtr+c"Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} | dur$| n|jj} |jr7|jr7| r7tdd} |du|duArCt d|durP| d|j d}|durY| |}||j }d}| rst|tssd}tdt|}|dd\}}|dur|nd}| durtj||||jd } |durts||}tj|||jd }t|tr|jn|}|||| || }|dur|durt||j|d }|j||f|| d }||}tjj ||j |jd }| rd nd}| rd nd}| r|durd nd}d}t!||gddgD]+\}}|dur1|dt"|j#ks1Jd|dt"|j#d|ddqt$|j#D]\}}| rD||f7}|jrUt%g}||j&krUq8|jr|jr|'|j(|||||durn||nd|durx||ndd| | | } n ||||||dur||nd|dur||nd|| | | d } | d}| r| | rdnd}| r|| df7}|dur|| df7}q8| r||f7}| r|nd}!|r|)}!| st*dd||!|||fDSt+||!|||dS)a4 Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer TzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.rr)r)rKrvr$rrzThe `z` should be specified for rr) rqrrrrrrprsrrtrrr2csrr,r$rr$r$r%r)sz(MarianDecoder.forward..)rrrnrcross_attentions),rbrsrrrrrxrgrhr#rzr!rrr{r r Zfrom_legacy_cacheryrr@rMrLrZonesr~rrr<rrrr^ziprrrrrrrZto_legacy_cacherr)"r/rrqrrrrrrrrsrrrtZreturn_legacy_cacherZ seq_lengthrJZmask_seq_lengthZself_attn_cacherrKrnZall_hidden_statesZall_self_attnsZall_cross_attentionsZnext_decoder_cacheZ attn_maskZ mask_namerZ decoder_layerrrZ next_cacher$r$r%rO?sT                 zMarianDecoder.forwardr,) NNNNNNNNNNNNN)rRrSrTrUrrrrr.rrr@rrXrBrrrrrOrYr$r$r0r%rs^     rc$sNeZdZddgZdeffdd ZddZdd Zd d Zd d Z ddZ ddZ de de jfddZe                d)deejdeejdeejdeejdeejdeejdeejdeeeejefdeeeejd eejd!eejd"eed#eed$eed%eed&eejdef"d'd(ZZS)* MarianModelzencoder.embed_tokens.weightzdecoder.embed_tokens.weightrbcst||j|j}}t||j||_|jj r |j}}nt |j}t |j}d|_t |||_ t|||_|dSr,)r-r.rrrrrsharedrb share_encoder_decoder_embeddingscopydeepcopyrencoderrdecoderr)r/rbr*rZencoder_embed_tokensZdecoder_embed_tokensr0r$r%r.;s       zMarianModel.__init__cCs |Sr,) get_encoderrrr$r$r%rQs z MarianModel.get_input_embeddingscCs2|jjr||_|j|j_|j|j_dS||j_dSr,)rbrrr rr rr$r$r%rUs   z MarianModel.set_input_embeddingscCs|jjrtd|S)Nz`get_decoder_input_embeddings` should not be called if `config.share_encoder_decoder_embeddings` is `True`. Please use `get_input_embeddings` instead.)rbrr# get_decoderrrr$r$r%get_decoder_input_embeddings]s  z(MarianModel.get_decoder_input_embeddingscCs|jjrtd||j_dS)Na `config.share_encoder_decoder_embeddings` is set to `True` meaning the decoder input embeddings are shared with the encoder. In order to set the decoder input embeddings, you should simply set the encoder input embeddings by calling `set_input_embeddings` with the appropriate embeddings.)rbrr#r rrr$r$r%set_decoder_input_embeddingses  z(MarianModel.set_decoder_input_embeddingscCrr,)r rr$r$r%r nrzMarianModel.get_encodercCrr,r rr$r$r%r qrzMarianModel.get_decodernew_num_tokensr+cCsV|jjrtd|}|||}|||}|dur!|S||j_||SNz`resize_decoder_token_embeddings` should not be called if `config.share_encoder_decoder_embeddings` is `True`. Please use `resize_token_embeddings` instead.)rbrr#r _get_resized_embeddingsrr tie_weights)r/rold_embeddingsnew_embeddings model_embedsr$r$r%resize_decoder_token_embeddingsts  z+MarianModel.resize_decoder_token_embeddingsNrrqrdecoder_attention_maskrdecoder_head_maskrencoder_outputsrrdecoder_inputs_embedsrrsrrrtcCs| dur| n|jj} |dur|n|jj}| dur| n|jj} |dur$|n|jj}|dur9|j|||| | ||d}n$|r]t|ts]t|dt|dkrN|dndt|dkrY|dndd}|j |||d|||| | | | |||d }|sw||St |j |j |j |j|j|j |j |jdS) a? decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Marian uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: ```python >>> from transformers import AutoTokenizer, MarianModel >>> tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> model = MarianModel.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> inputs = tokenizer("Studies have been shown that owning a dog is good for you", return_tensors="pt") >>> decoder_inputs = tokenizer( ... " Studien haben gezeigt dass es hilfreich ist einen Hund zu besitzen", ... return_tensors="pt", ... add_special_tokens=False, ... ) >>> outputs = model(input_ids=inputs.input_ids, decoder_input_ids=decoder_inputs.input_ids) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 26, 512] ```N)rrqrrrsrrrrr2r rrqrrrrrrrrsrrrt)rrdecoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentions)rbrsrrrr r{rrr rrrrnrr)r/rrqrrrrrrrrrrrsrrrtZdecoder_outputsr$r$r%rOs`> zMarianModel.forward)NNNNNNNNNNNNNNNN)rRrSrT_tied_weights_keysrr.rrr rr r rVrrrrrr@rrXrrrrBrrrOrYr$r$r0r%r7s|      rzX The Marian Model with a language modeling head. Can be used for summarization. )Z custom_introc&seZdZdZgdZddgZgdZdeffdd Zd d Z d d Z d9de de e de dejffdd Zd:de dejfddZddZde dd fddZddZdejfddZd d!Ze d;d"e ejd#e ejd$e ejd%e ejd&e ejd'e ejd(e ejd)e eeejefd*e eeejd+e ejd,e ejd-e ejd.e e d/e e d0e e d1e e d2e ejdef$d3d4Z d-ejfd5d6Z!e"d7d8Z#Z$S)< MarianMTModelr)final_logits_biaszencoder.embed_positions.weightzdecoder.embed_positions.weightz$model.encoder.embed_positions.weightz$model.decoder.embed_positions.weight)z!model.encoder.embed_tokens.weightz!model.decoder.embed_tokens.weightlm_head.weightrbcs^t|t||_|jr|jn|j}|dt d|ft j |j |dd|_ |dS)Nr#rFrd)r-r.rrrrrregister_bufferr@zerosrrirlm_headr)r/rbZtarget_vocab_sizer0r$r%r.s   zMarianMTModel.__init__cC |jSr,)rr rr$r$r%r rzMarianMTModel.get_encodercCr(r,)rr rr$r$r%r  rzMarianMTModel.get_decoderNTrpad_to_multiple_of mean_resizingr+cs&t|||}|jjr|||Sr,)r-resize_token_embeddingsrbr_resize_final_logits_bias)r/rr)r*rr0r$r%r+#s z%MarianMTModel.resize_token_embeddingscGs~|}||||}|||jjd}|jjr||j_|jjr;|dur;|jj s;|}| ||}| ||S)Nr) rrrr>r!rbrrget_output_embeddingstie_word_embeddings_get_resized_lm_headset_output_embeddings)r/rr)argsrr old_lm_head new_lm_headr$r$r%_resize_token_embeddings,s     z&MarianMTModel._resize_token_embeddingscCs|jjrtd|j}|||}|j||dur2|jjs2|}| ||}| ||j}|dur=|S||j_ | | ||Sr)rbrr#rr rrr-r.r/r0rrr,)r/rrrr2r3rr$r$r%rBs$       z-MarianMTModel.resize_decoder_token_embeddingscCsj|jjd}||kr|jddd|f}ntjd||f|jjd}tj|j|gdd}|d|dS)Nr rrr;r#)r#r!r@r&rLcatr%)r/rZold_num_tokensZnew_biasZ extra_biasr$r$r%r,bs z'MarianMTModel._resize_final_logits_biascCrr,r'rr$r$r%r-krz#MarianMTModel.get_output_embeddingsrcCrr,r6r/rr$r$r%r0nrz#MarianMTModel.set_output_embeddingscCs|}|durt|jddr|}|||t|jddrCt|jddrCt||jr5t||j}||j |j |jd}||_ | D] }t|drR| qGdS) z Tie the weights between the input embeddings and the output embeddings. If the `torchscript` flag is set in the configuration, can't handle parameter sharing so we are cloning the weights instead. Nr.Tis_encoder_decoderFZtie_encoder_decoderr  _tie_weights)r-getattrrbr rZ_tie_or_clone_weightshasattrrZ_tie_encoder_decoder_weightsr r Z_dynamic_tied_weights_keysmodulesr9)r/Zoutput_embeddingsZword_embeddingsZ tied_weightsrr$r$r%rqs       zMarianMTModel.tie_weightsrrqrrrrrrrrrlabelsrrsrrrtcCs|dur|n|jj}| dur)| rtdd} |dur)| dur)t| |jj|jj}|j|f|||||||| | | | ||||d}||d|j }d}| durat }|| d|jj | d}|sw|f|dd}|duru|f|S|St |||j|j|j|j|j|j|jd S) uH decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Marian uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MarianMTModel >>> src = "fr" # source language >>> trg = "en" # target language >>> model_name = f"Helsinki-NLP/opus-mt-{src}-{trg}" >>> model = MarianMTModel.from_pretrained(model_name) >>> tokenizer = AutoTokenizer.from_pretrained(model_name) >>> sample_text = "où est l'arrêt de bus ?" >>> batch = tokenizer([sample_text], return_tensors="pt") >>> generated_ids = model.generate(**batch) >>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0] "Where's the bus stop?" ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)rqrrrrrrrrrrrsrrrtrr r) losslogitsrrrrrrr )rbrrgwarningr&rrrr'r#rrzrrrrrrrrr )r/rrqrrrrrrrrrr=rrsrrrtrZ lm_logitsZmasked_lm_lossloss_fctoutputr$r$r%rOs^C zMarianMTModel.forwardcCst||jj|jjSr,)r&rbrr)r/r=r$r$r%%prepare_decoder_input_ids_from_labelssz3MarianMTModel.prepare_decoder_input_ids_from_labelscsBd}|D]}|tfdd|ddD|ddf7}q|S)Nr$c3$|] }|d|jVqdSrQZ index_selectrrLr5Z past_statebeam_idxr$r%r "z/MarianMTModel._reorder_cache..r2rrrHZreordered_pastZ layer_pastr$rGr%_reorder_caches zMarianMTModel._reorder_cache)NTr,)NNNNNNNNNNNNNNNNN)%rRrSrTrZ_keys_to_ignore_on_load_missingZ_keys_to_ignore_on_saver!rr.r r rVrrrrr+r4rr,r-r0rrr@rrXrrrrBrrOrCrrLrYr$r$r0r%r"s          vr"cs(eZdZdZfddZddZZS)MarianDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cst|t||_dSr,)r-r.rr r/rbr0r$r%r.s zMarianDecoderWrapper.__init__cOs|j|i|Sr,r)r/r1rr$r$r%rOszMarianDecoderWrapper.forward)rRrSrTrUr.rOrYr$r$r0r%rMs rMc"seZdZdgZfddZddZddZdd Zd d Zd d Z ddZ e              d$de e jde e jde e jde e jde e jde e jde ee jde e jde e jde ede ede ede ede e jdeeeffd d!Zed"d#ZZS)%MarianForCausalLMr$csNt|}d|_d|_t|t||_tj |j |j dd|_ | dS)NTFrd)rrr_r8r-r.rMrrriZ hidden_sizerr'rrNr0r$r%r.&s    zMarianForCausalLM.__init__cCs |jjjSr,rr rrr$r$r%r2rz&MarianForCausalLM.get_input_embeddingscCs||jj_dSr,rPrr$r$r%r5sz&MarianForCausalLM.set_input_embeddingscCrr,r6rr$r$r%r-8rz'MarianForCausalLM.get_output_embeddingscCrr,r6r7r$r$r%r0;rz'MarianForCausalLM.set_output_embeddingscCs ||j_dSr,rr )r/r r$r$r% set_decoder>s zMarianForCausalLM.set_decodercCs|jjSr,rQrr$r$r%r AszMarianForCausalLM.get_decoderNrrqrrrrrrr=rrsrrrtr+cCs| dur| n|jj} | dur| n|jj} | dur| n|jj} |jj||||||||| | | | |d }||d}d}| durU| |j} t }|| d|jj | d}| sk|f|dd}|duri|f|S|St |||j |j|j|jdS)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MarianForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-fr-en") >>> model = MarianForCausalLM.from_pretrained("Helsinki-NLP/opus-mt-fr-en", add_cross_attention=False) >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```Nrrr r)r>r?rrnrr)rbrsrrrr r'rrLrrzrrrrnrr)r/rrqrrrrrrr=rrsrrrtrr?r>rArBr$r$r%rODsH. zMarianForCausalLM.forwardcs.d}|D]}|tfdd|Df7}q|S)Nr$c3rDrQrErFrGr$r%rrIz3MarianForCausalLM._reorder_cache..rJrKr$rGr%rLs z MarianForCausalLM._reorder_cache)NNNNNNNNNNNNNN)rRrSrTr!r.rrr-r0rRr rrr@rrXrBrrrrrrOrrLrYr$r$r0r%rO#sr          YrO)rOrr"r)ArUrrtypingrrrrnumpyr3r@Ztorch.utils.checkpointrZtorch.nnrZ activationsr Z cache_utilsr r Z generationr Zmodeling_attn_mask_utilsr rZmodeling_outputsrrrrrZmodeling_utilsrutilsrrrrZconfiguration_marianrZ!torch.nn.attention.flex_attentionrZintegrations.flex_attentionrZ get_loggerrRrgrXrVr&rr'ModulerZrrrrrrrr"rMrO__all__r$r$r$r%sb         #Gv!+L