o Zhl@sldZddlZddlZddlmZmZmZmZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZmZmZdd lmZdd lmZdd lmZmZmZmZmZmZmZdd l m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+e&rddl,m-Z-ddl.m/Z/e)0e1Z2Gddde j3Z4Gddde j3Z5Gddde j3Z6Gddde j3Z7Gddde j3Z8Gddde j3Z9Gddde j3Z:Gd d!d!e j3Z;Gd"d#d#e j3Ze%Gd(d)d)e=Z?e%d*d+Gd,d-d-e=eZ@e%Gd.d/d/e=ZAe%d0d+Gd1d2d2e=ZBe%Gd3d4d4e=ZCe%Gd5d6d6e=ZDgd7ZEdS)8zPyTorch UMT5 model.N)ListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)AttentionMaskConverter)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringis_torch_flex_attn_availableis_torch_fx_proxyis_torchdynamo_compilinglogging) UMT5Config) BlockMask)make_flex_block_causal_maskcs&eZdZdfdd ZddZZS) UMT5LayerNormư>cs&ttt||_||_dS)ze Construct a layernorm module in the UMT5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__U/var/www/auris/lib/python3.10/site-packages/transformers/models/umt5/modeling_umt5.pyr'=s  zUMT5LayerNorm.__init__cCs\|tjdjddd}|t||j}|jjtj tj fvr)||jj}|j|S)NT)Zkeepdim) tor)Zfloat32powmeanZrsqrtr,r+dtypefloat16Zbfloat16)r- hidden_statesZvariancer2r2r3forwardEs  zUMT5LayerNorm.forward)r%)__name__ __module__ __qualname__r'r< __classcell__r2r2r0r3r$<sr$c*eZdZdeffdd ZddZZS)UMT5DenseActDenseconfigcsTttj|j|jdd|_tj|j|jdd|_t|j |_ t |j |_ dSNFbias)r&r'rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr-rCr0r2r3r'Ws zUMT5DenseActDense.__init__cCsl||}||}||}t|jjtjr/|j|jjjkr/|jjjtj kr/| |jjj}||}|SN) rJrPrN isinstancerKr+r)Tensorr9int8r6r-r;r2r2r3r<^s    zUMT5DenseActDense.forwardr=r>r?r!r'r<r@r2r2r0r3rBVsrBcrA)UMT5DenseGatedActDenserCcsjttj|j|jdd|_tj|j|jdd|_tj|j|jdd|_t |j |_ t |j |_dSrD)r&r'rrGrHrIwi_0wi_1rKrLrMrNr rOrPrQr0r2r3r'ns zUMT5DenseGatedActDense.__init__cCsz|||}||}||}||}t|jjtjr6|j |jjj kr6|jjj tj kr6| |jjj }||}|SrR) rPrYrZrNrSrKr+r)rTr9rUr6)r-r;Z hidden_geluZ hidden_linearr2r2r3r<vs   zUMT5DenseGatedActDense.forwardrWr2r2r0r3rXmsrXcrA) UMT5LayerFFrCcsJt|jrt||_nt||_t|j|jd|_ t |j |_ dS)Nr/)r&r'Z is_gated_actrXDenseReluDenserBr$rHlayer_norm_epsilon layer_normrrLrMrNrQr0r2r3r's   zUMT5LayerFF.__init__cCs&||}||}|||}|SrR)r_r]rN)r-r;Zforwarded_statesr2r2r3r<s  zUMT5LayerFF.forwardrWr2r2r0r3r[s r[cseZdZdZddeeffdd Zdejdejfd d Z d d Z dd dZ     ddejdeejdee ejdeejdeejdeejf ddZ ZS) UMT5Attentionz7 T5's attention using relative_attention_bias. FN layer_idxcst|j|_||_|j|_|j|_|j|_|j|_|j |_ |j |_ |j |j|_ ||_|dur@|jr@td|jjdtj|j|j dd|_tj|j|j dd|_tj|j|j dd|_tj|j |jdd|_|jrxt|j|j |_t|_dS)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrE)r&r' is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerHd_kvkey_value_proj_dim num_headsn_headsrMrNZ inner_dimralogger warning_oncer1r=rrGqkvo Embeddingrelative_attention_biasset pruned_heads)r-rCrcrar0r2r3r's,  zUMT5Attention.__init__ projectionreturncCs6|dd|j|jf}||dddd}|S)Nr5rr4r r )sizerirgviewpermute)r-rtZnew_projection_shapeZnew_projectionr2r2r3_shapeszUMT5Attention._shapec Csd}|j}|j}|js!|d}||dktj|7}t|}n t|t| }|d}||k}t | |t ||}|||}||tj}t|t ||d}|t |||7}|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr4r )rdrerbr6r)longabsminZ zeros_likelogfloatmathZ full_likewhere) r-relative_positionZrelative_bucketsZ num_bucketsZ max_distanceZ max_exactZis_smallZ log_ratioZrelative_position_if_larger2r2r3_relative_position_buckets$   z'UMT5Attention._relative_position_bucketc Cs|dur |jjj}|durtj|tj|ddddf}n|dddf}tj|tj|ddddf}||}||}||} | gdd} | S)z%Compute binned relative position biasN)r9device)r4rr r) rqr+rr)arangerzrrxZ unsqueeze) r-Z query_length key_lengthrcache_positionZcontext_positionZmemory_positionrZrelative_position_bucketvaluesr2r2r3 compute_biass    zUMT5Attention.compute_biasr;encoder_hidden_statespast_key_valueattention_masklayer_head_maskrcCs|jdd\}}|du} ||} | |d|j|jdd} |dur4|j|j} | r1|j } n|j } | r8|n|} | rO|durO| rO| j |j}| j |j}nE| | }|| }||d|j|jdd}||d|j|jdd}|dur| s}|nd}| |||jd|i\}}| rd|j|j<t| |dd}|dur||n|}|jd}|jstjd|j||f|j|jd}n|j|||j|d }|dddd| dddf}|dur|ddddddd|jdf}||}|jrt|jd}d |t|j<|dd|f}n|}||7}tjj|dd  |}tjj!||j!|j"d }|dur=||}t||}|dd#}|||d}|$|}|||fS) Nr4r5r rTr )rr9)rrrdim)ptraining)%shaperlrwrirgZ transpose is_updatedgetraZcross_attention_cacheself_attention_cacheZ key_cacheZ value_cachermrnupdater)matmulget_seq_lengthrcZzerosrr9rrsr*listboolrZ functionalZsoftmaxr~Ztype_asrNr contiguousro)r-r;rrrrr batch_size seq_lengthZis_cross_attentionZ query_statesrZcurr_past_key_valueZcurrent_statesZ key_statesZ value_statesZscoresZreal_seq_lengthrZ position_bias causal_maskmaskZposition_bias_maskedZ attn_weightsZ attn_outputr2r2r3r<sj          "&    zUMT5Attention.forward)FN)NNNNNNN)r=r>r?__doc__rintr'r)rTryrrrr<r@r2r2r0r3r`s0 / r`cs:eZdZddeeffdd Z    dddZZS) UMT5LayerSelfAttentionNrac>tt|d|d|_t|j|jd|_t |j |_ dS)NTrcrar\) r&r'r` SelfAttentionr$rHr^r_rrLrMrNr-rCrar0r2r3r'` zUMT5LayerSelfAttention.__init__c CsF||}|j|||||d}|||d}|f|dd}|S)Nrrrrrr )r_rrN) r-r;rrrrnormed_hidden_statesattention_outputoutputsr2r2r3r<fs zUMT5LayerSelfAttention.forwardrR)NNNNr=r>r?rrr'r<r@r2r2r0r3r_s rcs<eZdZddeeffdd Z     dddZZS) UMT5LayerCrossAttentionNracr)NFrr\) r&r'r`EncDecAttentionr$rHr^r_rrLrMrNrr0r2r3r'|rz UMT5LayerCrossAttention.__init__c CsH||}|j||||||d}|||d} | f|dd} | S)Nrrrrrrr )r_rrN) r-r;rrrrrrrZ layer_outputrr2r2r3r<s zUMT5LayerCrossAttention.forwardrRrrr2r2r0r3r{s rcsDeZdZddeeffdd Z         d ddZZS) UMT5BlockNracs^t|j|_t|_|jt||d|jr%|jt||d|jt |dS)Nra) r&r'rbr ModuleListlayerappendrrr[rr0r2r3r's  zUMT5Block.__init__Fc CsV|jd||||| d\}} }|jtjkr3t|jj} tt|| d| } tj || | d}d}|j o;|du}|rr|jd|||||| d\}}}|jtjkrrt|jj} tt|| d| } tj || | d}|jd|}|jtjkrt|jj} tt|| d| } tj || | d}||f}| r|| |f7}|S)Nrri)r|maxr rr5) rr9r)r:finforrisinfanyclamprb)r-r;rrencoder_attention_maskrcross_attn_layer_head_maskr use_cacheoutput_attentionsrZself_attn_weightsZ max_dtypeZ clamp_valueZcross_attn_weightsZdo_cross_attentionrr2r2r3r<sJ     zUMT5Block.forwardrR) NNNNNNFFNrr2r2r0r3rs rcs<eZdZdZdeffdd ZdejdejfddZZ S) UMT5ClassificationHeadz-Head for sentence-level classification tasks.rCcsBtt|j|j|_tj|jd|_t|j|j |_ dS)N)r) r&r'rrGrHdenserLclassifier_dropoutrN num_labelsout_projrQr0r2r3r's zUMT5ClassificationHead.__init__r;rucCs6||}||}t|}||}||}|SrR)rNrr)tanhrrVr2r2r3r<s     zUMT5ClassificationHead.forward) r=r>r?rr!r'r)rTr<r@r2r2r0r3rsrc@sHeZdZeZdZdZdZdZdgZ dgZ e ddZ ddZ d d Zd S) UMT5PreTrainedModel transformerTrrKcCs$tt}tt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r)Ztensorrr)r-rZ input_mask dummy_inputsr2r2r3rs  z UMT5PreTrainedModel.dummy_inputscCs |jj}t|tr|jj|dd St|ttt t fr^|j jjj d|ddt |dr>|jjs>|jjjj d|ddt |dr\|jjjj d||jjdd|jjjd Sd St|trt |dr}|jjjj d|dd|jjjd Sd St|tr|jjjj d||jjddt |jdr|jjd ur|jjj|jjjj d||jjddt |jdr|jjd ur|jjjd Sd Sd St|tr$|jjjj d||jjddt |jdr|jjd ur|jjj|jjjj d||jjddt |jdr |jjd ur"|jjjd Sd Sd St|tr|jjjj d||jjddt |jdrO|jjd urO|jjj|jjjj d||jjddt |jdrt|jjd urt|jjj|jjjj d||jjddt |jdr|jjd ur|jjjd Sd Sd St|t r|jj}|jj!}|jj"}|j#jjj d|||dd|j$jjj d||dd|j%jjj d||dd|j&jjj d|||dd|j'r|j(jjj d||ddd Sd Sd S) zInitialize the weights?)r8Zstdlm_head qa_outputs classifierrFN))rCZinitializer_factorrSr$r+dataZfill_ UMT5ModelUMT5ForConditionalGenerationUMT5EncoderModelUMT5ForQuestionAnsweringsharedZnormal_hasattrtie_word_embeddingsrrrHrFZzero_UMT5ForTokenClassificationrrrrrBrJrKrIrXrYrZr`rfrhrlrmrnrorcrq)r-modulefactorrHrgrir2r2r3 _init_weights s|                     z!UMT5PreTrainedModel._init_weightscCs|jj}|jj}|durtdt|r1t|jddd|}tj||dddfgdd}n| |j}|dddf |dddf<||d<|durStd| |d k||S) Nzself.model.config.decoder_start_token_id has to be defined. In UMT5 it is usually set to the pad_token_id. See UMT5 docs for more information.r5)r .rr ).rz1self.model.config.pad_token_id has to be defined.) rCdecoder_start_token_id pad_token_id ValueErrorrr)fullrcatZ new_zeroscloneZ masked_fill_)r-rrrZshifted_input_idsr2r2r3 _shift_rightOs    z UMT5PreTrainedModel._shift_rightN)r=r>r?r! config_classZbase_model_prefixZsupports_gradient_checkpointingZ_supports_cache_classZ_supports_static_cacheZ_no_split_modulesZ_keep_in_fp32_modulespropertyrrrr2r2r2r3rs  Brc seZdZdfdd ZddZddZ             ddd Z dd eej d fd ej dej de de f ddZ e d ej dededejdej def ddZZS) UMT5StackNcslt||_j|_tfddtjD|_t j j d|_ t j|_d|_|dS)Ncsg|]}t|dqS)r)r).0irCr2r3 psz&UMT5Stack.__init__..r\F)r&r' embed_tokensrbrrrange num_layersblockr$rHr^final_layer_normrLrMrNgradient_checkpointing post_init)r-rCrr0rr3r'ls   zUMT5Stack.__init__cC|jSrRrr-r2r2r3get_input_embeddingsxzUMT5Stack.get_input_embeddingscC ||_dSrRrr-Znew_embeddingsr2r2r3set_input_embeddings{ zUMT5Stack.set_input_embeddingsc'Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} | dur$| n|jj} |durB|durB|jr5dnd}td|d|d|durS|}|d|d}n|dur`|dd}n|jrednd}td|d|d |j r|j r| rt d d } |dur|j durtd | |}|\}}| d ur|jstd|dd }d }|jr| s|durt|trt|tsd }t|t}n#t|tsd }t dt|}n|durttt}n|jsd}|dur|nd}| durtj||||jd} |durts||}tj|||jd}|jr/|||| |dur*|jnd| }n&|durS|ddddddf}|j|jd}d|t|jj}nd}|jr||dur||\}}}||f}|durvtj||jd}||}nd}| ||jj!}| ||jj!}| rdnd}| rdnd}| r|jrdnd}|"|}t#|j$D]\\} }!|| }"|| }#| r||f}|j r|j r|%|!j&|||||"|#d| | | }$n|!|||||"|#|| | | d }$|$d}| r|$d}%| r ||$df7}|jr ||$df7}q|'|}|"|}| r ||f}| r%|%nd}&|r-|j}&|r4|(}&| sEt)dd||&|||fDSt*||&|||dS)NZdecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer5zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fz4sz$UMT5Stack.forward..)last_hidden_statepast_key_valuesr; attentionscross_attentions)+rCrroutput_hidden_statesuse_return_dictrbrrvrwrrrjrkrrSr rr Zfrom_legacy_cacherr)rrrr*_update_causal_maskrr6r9rr|Zinvert_attention_maskZ get_head_maskrrN enumeraterZ_gradient_checkpointing_funcr<rZto_legacy_cachetupler)'r-rrrrr head_maskcross_attn_head_maskrrrr return_dictrZerr_msg_prefixZ input_shaperrZreturn_legacy_cacheZreturn_self_attention_cachepast_key_values_lengthZmask_seq_lengthrZencoder_batch_sizeZencoder_sequence_length_Zencoder_hidden_shapeZencoder_extended_attention_maskZall_hidden_statesZall_attentionsZall_cross_attentionsr;rZ layer_modulerrZ layer_outputsZnext_decoder_cacheZ next_cacher2r2r3r<~s$                 zUMT5Stack.forwardFrr" input_tensorrrrc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) NZflash_attention_2rZflex_attentionrFZsdpa)rr Z is_trainingr r5)sequence_length target_lengthr9rr)cudaZxpuZnpu)rCZ_attn_implementationrrSr)rTr#rZis_compileablerZ_ignore_causal_mask_sdparr9rZget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrtyperr|Z_unmask_unattended) r-rrrrrZpast_seen_tokensZusing_compilable_cacher9rrr min_dtyper2r2r3rHsT           zUMT5Stack._update_causal_maskrrr9rc KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valuer9rr )Zdiagonalrr5r)rr)rr|rrZtriurZreshapeexpandrrr6Z masked_fill) rrrr9rrkwargsrrZ mask_lengthZ padding_maskr2r2r3rs,  $ 6  z?UMT5Stack._prepare_4d_causal_attention_mask_with_cache_positionrR) NNNNNNNNNNNNN)F)r=r>r?r'rrr<rr)rTr rr staticmethodrr9rr@r2r2r0r3rksZ  Q  Drc&sJeZdZdZdZeZddgZfddZddZ d d Z d d Z d dZ ddZ ddZe                d'deejdeejdeejdeejdeejdeejdeejdeeeejdeeeejdeejdeejdeed eed!eed"eed#eejd$eeejeff"d%d&ZZS)(rao Examples: ```python >>> from transformers import UMT5Model, AutoTokenizer >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> noisy_text = "UN Offizier sagt, dass weiter werden muss in Syrien." >>> label = " verhandelt" >>> inputs = tokenizer(inputs, return_tensors="pt") >>> labels = tokenizer(label=label, return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"], decoder_input_ids=labels["input_ids"]) >>> hidden_states = outputs.last_hidden_state ```umt5encoder.embed_tokens.weightdecoder.embed_tokens.weightcst|t|j|j|_t|}d|_ d|_ d|_ t ||j|_ t|}d|_ d|_ |j|_t ||j|_|dSNFT)r&r'rrp vocab_sizerHrcopydeepcopyrbris_encoder_decoderrencodernum_decoder_layersrdecoderrr-rCencoder_configZdecoder_configr0r2r3r's    zUMT5Model.__init__cCrrRrrr2r2r3rrzUMT5Model.get_input_embeddingscC"||_|j||j|dSrRrr!rr#rr2r2r3r zUMT5Model.set_input_embeddingscC4|jjr||jj|j||jj|jdSdSrRrCr_tie_or_clone_weightsr!rrr#rr2r2r3 _tie_weightszUMT5Model._tie_weightscCrrRr!rr2r2r3 get_encoderrzUMT5Model.get_encodercCrrRr#rr2r2r3 get_decoderrzUMT5Model.get_decodercCs*|D]\}}|jj|j|qdS) Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr!rZ attention prune_headsr-Zheads_to_prunerZheadsr2r2r3 _prune_headsszUMT5Model._prune_headsNrrrrr decoder_head_maskr encoder_outputsrrdecoder_inputs_embedsrrrr rrucCs| dur| n|jj} |dur|n|jj}|dur%|j||| || ||d}n$|rIt|tsIt|dt|dkr:|dndt|dkrE|dndd}|d}|j||| | ||||| | |||d }|se||St|j |j |j |j |j |j |j |j dS) a+ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: ```python >>> from transformers import AutoTokenizer, UMT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for UMT5Model. >>> # This is not needed for torch's UMT5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nrrrr rrr rr r4rr;r rrrrrrr r rrrr r)rrdecoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentions)rCrrr!rSrlenr#rrrr;rr)r-rrrrr r8r r9rrr:rrrr rr;decoder_outputsr2r2r3r<s\Q zUMT5Model.forwardNNNNNNNNNNNNNNNN)r=r>r?r model_typer!r_tied_weights_keysr'rrr-r0r2r7rrr) LongTensor FloatTensor BoolTensorrTrrrrr<r@r2r2r0r3rs      rz< UMT5 Model with a `language modeling` head on top. )Z custom_introc(sveZdZdZdZgdZfddZddZdd Zd d Z d d Z ddZ ddZ ddZ e                 d-deejdeejdeejdeejdeejdeejdeejdeeeejdeeeejdeejdeejd eejd!eed"eed#eed$eed%eejd&eeejeff$d'd(Zd ejfd)d*Zed+d,ZZS).ra Examples: ```python >>> from transformers import UMT5ForConditionalGeneration, AutoTokenizer >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, text_target=summary, return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ```r)rrzlm_head.weightcst||j|_t|j|j|_t |}d|_ d|_ d|_ t ||j|_t |}d|_ d|_ |j|_t ||j|_tj|j|jdd|_|dS)NFTrE)r&r'rH model_dimrrprrrrrbrr rr!r"rr#rGrrr$r0r2r3r's    z%UMT5ForConditionalGeneration.__init__cCrrRr&rr2r2r3rrz1UMT5ForConditionalGeneration.get_input_embeddingscCr'rRr(rr2r2r3rr)z1UMT5ForConditionalGeneration.set_input_embeddingscCr*rRr+rr2r2r3r-r.z)UMT5ForConditionalGeneration._tie_weightscCrrRrrr2r2r3set_output_embeddingsrz2UMT5ForConditionalGeneration.set_output_embeddingscCrrRrKrr2r2r3get_output_embeddingsrz2UMT5ForConditionalGeneration.get_output_embeddingscCrrRr/rr2r2r3r0rz(UMT5ForConditionalGeneration.get_encodercCrrRr1rr2r2r3r2rz(UMT5ForConditionalGeneration.get_decoderNrrrrr r8r r9rrr:labelsrrrr rrucCs| dur| n|jj} |dur|n|jj}|dur%|j||| ||||d}n$|rIt|tsIt|dt|dkr:|dndt|dkrE|dndd}|d}| dur^|dur^| dur^|| }|j||| | ||||| ||||d }|d}|jj r||j d}| |}d}| durt d d }| |j} ||d |d | d }|s|f|dd|}|dur|f|S|St|||j|j|j|j|j|j|jd S) aK input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, UMT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer("Studies have shown that good for you", return_tensors="pt").input_ids >>> outputs = model.generate(input_ids) >>> tokenizer.decode(outputs[0], skip_special_tokens=True) ```Nr;rr r4r<r=rrZ ignore_indexr5 losslogitsrr>r?rr@rrA)rCrrr!rSrrBrr#rrJrrr6rrwrvrrr;rrr)r-rrrrr r8r r9rrr:rNrrrr rr;rCsequence_outputZ lm_logitsrQloss_fctoutputr2r2r3r<svU     z$UMT5ForConditionalGeneration.forwardcCs ||SrR)r)r-rNr2r2r3%prepare_decoder_input_ids_from_labelsrzBUMT5ForConditionalGeneration.prepare_decoder_input_ids_from_labelscs.d}|D]}|tfdd|Df7}q|S)Nr2c3s$|] }|d|jVqdS)rN)Z index_selectr6r)rZ past_statebeam_idxr2r3rs"z>UMT5ForConditionalGeneration._reorder_cache..)r)rrXZreordered_pastZ layer_pastr2rWr3_reorder_caches z+UMT5ForConditionalGeneration._reorder_cache)NNNNNNNNNNNNNNNNN)r=r>r?rrErFr'rrr-rLrMr0r2rrr)rGrHrIrTrrrrr<rVrrYr@r2r2r0r3rs       #rcseZdZdZdZdgZfddZddZdd Zd d Z d d Z ddZ e       dde ejde ejde ejde ejde ede ede edeeejeffddZZS)ra Examples: ```python >>> from transformers import UMT5EncoderModel, AutoTokenizer >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> input_ids = tokenizer(article, return_tensors="pt").input_ids >>> outputs = model(input_ids) >>> hidden_state = outputs.last_hidden_state ```rrcsNt|t|j|j|_t|}d|_ d|_ t ||j|_ | dSNF)r&r'rrprrHrrrrr rr!r)r-rCr%r0r2r3r's   zUMT5EncoderModel.__init__cCrrRr&rr2r2r3rrz%UMT5EncoderModel.get_input_embeddingscCs||_|j|dSrR)rr!rrr2r2r3rsz%UMT5EncoderModel.set_input_embeddingscCs"|jjr||jj|jdSdSrR)rCrr,r!rrrr2r2r3r-szUMT5EncoderModel._tie_weightscCrrRr/rr2r2r3r0rzUMT5EncoderModel.get_encodercCs0|D]\}}|jj|jdj|qdS)r3rN)r4r!rrrr5r6r2r2r3r7szUMT5EncoderModel._prune_headsNrrr rrrr ruc Cs0|dur|n|jj}|j|||||||d}|S)aQ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). Example: ```python >>> from transformers import AutoTokenizer, UMT5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr;)rCrr!) r-rrr rrrr r9r2r2r3r<s# zUMT5EncoderModel.forward)NNNNNNN)r=r>r?rrErFr'rrr-r0r7rrr)rGrHrrrrr<r@r2r2r0r3rsF   rz UMT5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c$seZdZdgZddgZdeffdd Ze               ddee j d ee j d ee j d ee j d ee j d ee j dee j dee e j dee j dee j dee j deedeedeedeedeeeff ddZZS)UMT5ForSequenceClassificationFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weightrrrCcs2t|t||_t||_|d|_dSrZ)r&r'rrrclassification_headrZmodel_parallelrQr0r2r3r's    z&UMT5ForSequenceClassification.__init__Nrrrrr r8r r9rr:rNrrrr rucCsh|dur|n|jj}| durd} |dur!| dur!td|jj|dur6| dur6|dur1td||}|j||||||||| | | | ||d}|d}||jj  |j }t t |ddkrhtd|j\}}}||ddf|d |ddd ddf}||}d}| dur| |j } |jjdur|jjdkrd |j_n|jjdkr| jt jks| jt jkrd |j_nd |j_|jjd krt}|jjdkr||| }n-||| }n'|jjd krt}||d |jj| d }n|jjd krt}||| }|s |f|dd}|dur|f|S|St|||j|j|j|j |j!|j"|j#d S)as input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rrrr r8r r9rr:rrrr rr z7All examples must have the same number of tokens.r5Z regressionZsingle_label_classificationZmulti_label_classificationrP)$rCrNotImplementedErrorr1r=rrreqZ eos_token_idr6rrBr)Zunique_consecutivesumrrwr]Z problem_typerr9rzrr squeezerrrrr>r?rr@rrA)r-rrrrr r8r r9rr:rNrrrr rrSZeos_maskrr r.Zsentence_representationrRrQrTrUr2r2r3r<s>   ,      $      z%UMT5ForSequenceClassification.forward)NNNNNNNNNNNNNNN)r=r>r?"_keys_to_ignore_on_load_unexpectedrFr!r'rrr)rGrTrrHrrrrr<r@r2r2r0r3r[sj       r[cseZdZdgZdgZdeffdd Ze        ddee j dee j d ee j d ee j d ee j d ee d ee dee de e e j effddZZS)rr\z'transformer.encoder.embed_tokens.weightrCcsJt||j|_t||_t|j|_t |j |j|_ | dSrR) r&r'rrrrrLrrNrGr.rrrQr0r2r3r's   z#UMT5ForTokenClassification.__init__Nrrr rrNrrr ruc  Cs|dur|n|jj}|j|||||||d} | d} || } || } d} |dur:t} | | d|j|d} |sO| | ddf}| durM| f|S|St| | | j | j dS)aB input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. N)rr rrrr rr5r4)rQrRr;r) rCrrrNrrrwrrr;r)r-rrr rrNrrr rr;rRrQrTrUr2r2r3r<s4  z"UMT5ForTokenClassification.forward)NNNNNNNN)r=r>r?rcrFr!r'rrr)rTrrrrr<r@r2r2r0r3rs@   rc&s.eZdZddgZfddZddZddZd d Zd d Zd dZ e                d#de e j de e jde e j de e jde e jde e jde e jde eee jde e j de e j de e jde e jde ede ede ede ed eee jeff"d!d"ZZS)$rrrcst||j|_t|j|j|_t |}d|_ d|_ d|_ t ||j|_t |}d|_ d|_ |j|_t ||j|_|j|_t|j|j|_|dSr)r&r'rHrJrrprrrrrbrr rr!r"rr#rrGrrr$r0r2r3r's    z!UMT5ForQuestionAnswering.__init__cCrrRr&rr2r2r3rrz-UMT5ForQuestionAnswering.get_input_embeddingscCr'rRr(rr2r2r3rr)z-UMT5ForQuestionAnswering.set_input_embeddingscCr*rRr+rr2r2r3r-"r.z%UMT5ForQuestionAnswering._tie_weightscCrrRr/rr2r2r3r0(rz$UMT5ForQuestionAnswering.get_encodercCrrRr1rr2r2r3r2,rz$UMT5ForQuestionAnswering.get_decoderNrrrrr r8r r9start_positions end_positionsrr:rrrr rucCsj|dur|n|jj}| dur| n|jj} | dur| durd} |dur3| dur3|dur.td||}| dur9| n|jj} |durC|n|jj}|durX|j||| ||||d}n$|r|t|ts|t|dt|dkrm|dndt|dkrx|dndd}|d}|j ||| d||||| |||d }|d}| |}|j dd d \}}| d  }| d  }d}| dur| durt| dkr| d |j} t| dkr| d |j} |d}| d|} | d|} t|d }||| }||| }||d}|s ||f|dd|}|dur|f|S|St||||j|j|j|j|j|j|jd S)aI input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. NFr^r;rr r4r<) rrrrrrr r rrrr r5rrO) rQ start_logits end_logitsrr>r?rr@rrA)rCrrrrr!rSrrBr#rsplitrbrrvr6rrrrrr;rrr)r-rrrrr r8r r9rdrerr:rrrr r;rCrSrRrfrgZ total_lossZ ignored_indexrTZ start_lossZend_lossrUr2r2r3r</s<          z UMT5ForQuestionAnswering.forwardrD)r=r>r?rFr'rrr-r0r2rrr)rGrHrIrTrrrrr<r@r2r2r0r3rsx      r)rrrr[rrr)Frrrtypingrrrrr)rZtorch.nnrrr Z activationsr Z cache_utilsr r rZ generationrZmodeling_attn_mask_utilsrZmodeling_outputsrrrrrrrZmodeling_utilsrutilsrrrrrrrZconfiguration_umt5r!Z!torch.nn.attention.flex_attentionr"Zintegrations.flex_attentionr#Z get_loggerr=rjModuler$rBrXr[r`rrrrrrrrrr[rr__all__r2r2r2r3sn    $ $    CMr\S|l$L R