o Zh @sdZddlZddlZddlZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z mZmZddlmZddlmZmZdd lmZdd lmZmZmZdd lmZmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%dd l&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,ddl-m.Z.e*rddl/m0Z0ddl1m2Z2erddlm3Z3e,4e5Z6de j7de8de8fddZ9Gddde j:Z;Gddde j:ZGdd d e>Z?Gd!d"d"e>Z@e>e@e?d#ZAGd$d%d%e j=ZBGd&d'd'e j=ZCGd(d)d)e j=ZDe)Gd*d+d+e'ZEGd,d-d-eEZFGd.d/d/eEZGGd0d1d1eEZHGd2d3d3eEZIe)Gd4d5d5eEZJe)d6d7Gd8d9d9eEeZKe)d:d7Gd;d<dd>eEZMGd?d@d@eEZNe)dAd7GdBdCdCeEeZOgdDZPdS)EzPyTorch BART model.N)ListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)CacheEncoderDecoderCache)GenerationMixin)AttentionMaskConverter_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa)!flash_attn_supports_top_left_maskis_flash_attn_available)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutput)PreTrainedModel)auto_docstringis_torch_flex_attn_availableis_torchdynamo_compilinglogging) BartConfig) BlockMask)make_flex_block_causal_mask)_flash_attention_forward input_ids pad_token_iddecoder_start_token_idcCsh||j}|ddddf|ddddf<||dddf<|dur*td||dk||S)z1 Shift input ids one token to the right. Nr rz1self.model.config.pad_token_id has to be defined.i)Z new_zerosshapeclone ValueErrorZ masked_fill_)r%r&r'Zshifted_input_idsr,U/var/www/auris/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.pyshift_tokens_rightDs (r.csJeZdZdZdedeffdd Zd dejd ed ejffd d ZZ S)BartLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcsd|_t||j|dSN)offsetsuper__init__)selfr0r1 __class__r,r-r6Ysz'BartLearnedPositionalEmbedding.__init__rNr%past_key_values_length position_idscs\|dur |jdd\}}tj|||tj|jjd|d}n|d}t ||j S)z3`input_ids' shape is expected to be [bsz x seqlen].Nr3)dtypedevicer(r) r)torcharangelongweightr=expandZ unsqueezer5forwardr4)r7r%r:r;bszZseq_lenr8r,r-rC_s z&BartLearnedPositionalEmbedding.forwardrN) __name__ __module__ __qualname____doc__intr6r>TensorrC __classcell__r,r,r8r-r/Ts(r/c sLeZdZdZd dedededeeffdd Zd ej ffd d Z Z S) BartScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. ?r0r1 padding_idx embed_scalecst|||||_dSN)r5r6rP)r7r0r1rOrPr8r,r-r6rs z BartScaledWordEmbedding.__init__r%cst||jSrQ)r5rCrP)r7r%r8r,r-rCvszBartScaledWordEmbedding.forward)rN) rFrGrHrIrJrfloatr6r>rKrCrLr,r,r8r-rMms$rMcseZdZdZ      ddededed ed ed ed eed eeffdd Z      dde j dee j dee dee j dee j dedee j de e j ee j ee e j ffddZZS) BartAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsdropout is_decoderbias is_causalconfig layer_idxc st||_||_||_|||_||_|j||jkr*td|jd|d|jd|_||_ ||_ ||_ |durK|j rKt d|jjdtj|||d|_tj|||d|_tj|||d|_tj|||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gzInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.rY)r5r6rUrVrWhead_dimr[r+scalingrXrZr\logger warning_oncer9rFrLineark_projv_projq_projout_proj) r7rUrVrWrXrYrZr[r\r8r,r-r6}s0    zBartAttention.__init__ hidden_stateskey_value_statespast_key_valueattention_masklayer_head_maskoutput_attentionscache_positionreturncCsv|du}|\} } } ||| d|j|jdd} | |j} |dur=t|tr;|j |j } |r7|j }n|j }n|}|rA|n|}|rX|durX| rX|j|j }|j|j }nE||}||}|| d|j|jdd}|| d|j|jdd}|dur|s|nd}||||j d|i\}}|rd|j |j <| |jd|jf}| j|} |j|}|j|}|d}t| |dd}|| |j| |fkrtd| |j| |fd||dur|ddddddd|jd f}|| |j| ||}|| |j| |}tjj|dd }|durN||jfkr3td |jfd||dddd|| |j| |}|| |j| |}|re|| |j| |}|| |j| |}nd}tjj||j|jd }t||}|| |j| |jfkrtd | |j| |jfd||| |j| |j}|dd}|| | |j}||}|||fS)#Input shape: Batch x Time x ChannelNr(r r3rmTz$Attention weights should be of size z , but is dimz/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )sizereviewrVr^ transposer_ isinstancer is_updatedgetr\cross_attention_cacheself_attention_cache key_cache value_cachercrdupdatereshaper>Zbmmr+r)r functionalZsoftmaxrWrurUrf)r7rgrhrirjrkrlrmis_cross_attentionrDtgt_len_ query_statesrzcurr_past_key_valuecurrent_states key_states value_statesZ proj_shapeZsrc_len attn_weightsZattn_weights_reshapedZ attn_probs attn_outputr,r,r-rCs"             &  "    zBartAttention.forward)rTFTFNNNNNNFN)rFrGrHrIrJrRboolrr!r6r>rKr rrCrLr,r,r8r-rSzs` * rScseZdZdZfddZ      ddejdeejdeed eejd eejd e d eejd e ejeejee ejffddZ Z S)BartFlashAttention2aD Bart flash attention module. This module inherits from `BartAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cstj|i|t|_dSrQ)r5r6r_flash_attn_uses_top_left_maskr7argskwargsr8r,r-r6s zBartFlashAttention2.__init__NFrgrhrirjrkrlrmrnc Cs|rtd|du}|\} } } ||| d|j|j} |dur:t|tr8|j |j } |r4|j }n|j }n|}|r>|n|}|rU|durU| rU|j |j }|j|j }nE||}||}|| d|j|jdd}|| d|j|jdd}|dur|s|nd}||||j d|i\}}|rd|j|j <|dd}|dd}| j}|tjkrtrt}nt|jdr|jj}n|jjj}td|d | |} ||}||}t| |||| |j r|j!nd |j"|j#d }|$| | d}|%|}|d|fS) NzBartSdpaAttention2 attention does not support `output_attentions`. Use the argument `attn_implementation='eager'` when loading the model.r(r r3rmT_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .rT)rWrZZuse_top_left_mask)&r+rvrerwrVr^ryr rzr{r\r|r}r~rrcrdrxrr<r>Zfloat32Zis_autocast_enabledZget_autocast_gpu_dtypehasattrr[rrAr`rator$rurWrZrrrf)r7rgrhrirjrkrlrmrrDZq_lenrrrzrrrrZ input_dtypeZ target_dtyperr,r,r-rC&sv                    zBartFlashAttention2.forwardr) rFrGrHrIr6r>rKrr rrrCrLr,r,r8r-rs4   rcseZdZ      d dejdeejdeedeejdeejded eejd eejeejeeejfffd d Z Z S)BartSdpaAttentionNFrgrhrirjrkrlrmrncs,|rtdtj||||||dS|du}|\} } } ||| d|j|j dd} |durKt |t rI|j |j} |rE|j}n|j}n|}|rO|n|}|rf|durf| rf|j|j}|j|j}nE||}||}|| d|j|j dd}|| d|j|j dd}|dur|s|nd}||||jd|i\}}|rd|j |j<d}|dur|ddddddd|jd f}| jjd kr|dur| } |}|}|jr|dur| dkrdnd }tjjj| ||||jr|j nd |d }| dd}|| | |j!}|"|}|d|fS)roaBartModel is using BartSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` . Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rhrirjrlrmNr(r r3rmTrpcudaFrT) attn_maskZ dropout_prZ)#r`rar5rCrvrerwrVr^rxryr rzr{r\r|r}r~rrcrdrr)r=type contiguousrZr>rrZscaled_dot_product_attentionrurWrUrf)r7rgrhrirjrkrlrmrrDrrrrzrrrr causal_maskrZrr8r,r-rCsr  "       &   zBartSdpaAttention.forwardr) rFrGrHr>rKrr rrrCrLr,r,r8r-rs0 r)eagersdpaflash_attention_2csheZdZddedeeffdd Z ddejdejd ejd ee d e ejeejff d d Z Z S)BartEncoderLayerNr[r\cst|j|_t|j|j|j|j||d|_t |j|_ |j |_ t |j|_|j|_t |j|j|_t |j|j|_t |j|_dS)N)rUrVrWr[r\)r5r6d_modelrUBART_ATTENTION_CLASSES_attn_implementationZencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrWr activation_function activation_fnactivation_dropoutrbZencoder_ffn_dimfc1fc2final_layer_normr7r[r\r8r,r-r6s  zBartEncoderLayer.__init__Frgrjrkrlrnc Cs|}|j||||d\}}}tjj||j|jd}||}||}|}|||}tjj||j|jd}| |}tjj||j|jd}||}| |}|j t j krvt |sdt |rvt |j jd}t j|| |d}|f} |r| |f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rgrjrkrlrsi)minmax)rrrrWrurrrrrrr<r>Zfloat16isinfanyisnanfinforclamp) r7rgrjrkrlresidualrrZ clamp_valueoutputsr,r,r-rCs8        zBartEncoderLayer.forwardrQF) rFrGrHr!rrJr6r> FloatTensorrrrCrLr,r,r8r-rsrcseZdZddedeeffdd Z         ddejd eejd eejd eejd eejd eejdee dee dee deejde ej ee ej ej fffddZ ZS)BartDecoderLayerNr[r\c st|j|_t|j|j|j|jdd||d|_|j |_ t |j |_ |j |_ t|j|_t|j|j|j|jd||d|_t|j|_t|j|j|_t|j|j|_t|j|_dS)NT)rUrVrWrXrZr[r\)rWrXr[r\)r5r6rrUrrZdecoder_attention_headsrrrWr rrrrrr encoder_attnencoder_attn_layer_normrbZdecoder_ffn_dimrrrrr8r,r-r6Fs6  zBartDecoderLayer.__init__FTrgrjencoder_hidden_statesencoder_attention_maskrkcross_attn_layer_head_maskrirl use_cachermrnc Cs |} |j|||||| d\}} }tjj||j|jd}| |}||}d} |durN|} |j||||||d\}} }tjj||j|jd}| |}||}|} || |}tjj||j |jd}| |}tjj||j|jd}| |}| |}|f}|r|| | f7}| r||f7}|S)a8 Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. )rgrirjrkrlrmrsN)rgrhrjrkrirl) rrrrWrurrrrrrrr)r7rgrjrrrkrrirlrrmrZself_attn_weightsZcross_attn_weightsrr,r,r-rCesN"        zBartDecoderLayer.forwardrQ) NNNNNNFTN)rFrGrHr!rrJr6r>rKr rrrrCrLr,r,r8r-rEsD"    rcsHeZdZdZdedededeffdd Zdejd ejfd d Z Z S) BartClassificationHeadz-Head for sentence-level classification tasks. input_dim inner_dim num_classespooler_dropoutcs8tt|||_tj|d|_t|||_dS)N)rt)r5r6rrbdenseZDropoutrWrf)r7rrrrr8r,r-r6s zBartClassificationHead.__init__rgrncCs6||}||}t|}||}||}|SrQ)rWrr>tanhrf)r7rgr,r,r-rCs     zBartClassificationHead.forward) rFrGrHrIrJrRr6r>rKrCrLr,r,r8r-rs rc @seZdZeZdZdZddgZddgZdZ dZ dZ dZ dZ dd Zed d Z dd eejdfdejdejdedef ddZed ejdededejdejdef ddZdS)BartPreTrainedModelmodelTzencoder.versionzdecoder.versionrrpast_key_valuescCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|tj rC|jjjd|d|j durA|jj|j  dSdSt|tj rX|jj d|jj dSdS)NrT)meanstdrN)r[Zinit_stdryrrbrAdataZnormal_rYZzero_ EmbeddingrOrZfill_)r7modulerr,r,r- _init_weightss     z!BartPreTrainedModel._init_weightscCs>|jj}tjgddddd|gg|jd}|||d}|S)N)r r3r r3r=)rjr%)r[r&r>Ztensorr=ne)r7Z pad_tokenr% dummy_inputsr,r,r-rs "z BartPreTrainedModel.dummy_inputsFrjr" input_tensorrmrlc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) NrrTZflex_attentionrFr) inputs_embedsr:Z is_trainingr r()sequence_length target_lengthr<rm batch_size)rZxpuZnpu)r[rrryr>rKr#get_seq_lengthZis_compileablerZ_ignore_causal_mask_sdparur<r)Zget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionr=rrrZ_unmask_unattended) r7rjrrmrrlZpast_seen_tokensZusing_compilable_cacher<rrr min_dtyper,r,r-_update_causal_masksT           z'BartPreTrainedModel._update_causal_maskrrr<rc KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr)Z fill_valuer<r=r )Zdiagonalrr(r)rrr>rrfullr=Ztriur?rrBr*r)rZ masked_fill) rjrrr<rmrrrrZ mask_lengthZ padding_maskr,r,r-r=s,  $ 6  zIBartPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_positionNr)rFrGrHr!Z config_classbase_model_prefixZsupports_gradient_checkpointingZ"_keys_to_ignore_on_load_unexpectedZ_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attn_2Z_supports_sdpaZ_supports_cache_classZ_supports_static_cacherpropertyrrr>rKr rr staticmethodrJr<rr,r,r,r-rsP   Drc@eZdZddZdS)PretrainedBartModelcCtdtdSNz_The class `PretrainedBartModel` has been depreciated, please use `BartPreTrainedModel` instead.warningswarn FutureWarningr7r,r,r-__init_subclass__wz%PretrainedBartModel.__init_subclass__NrFrGrHrr,r,r,r-rv rc@r)BartPretrainedModelcCrrrrr,r,r-rrz%BartPretrainedModel.__init_subclass__Nrr,r,r,r-r~rrcseZdZdZddedeejffdd ZddZ d d Z       dd ee j d ee j d ee j dee jdeedeedeedeeeffddZZS) BartEncoderz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`BartEncoderLayer`]. Args: config: BartConfig embed_tokens (nn.Embedding): output embedding Nr[ embed_tokenscstj|_j|_j}j|_j|_ j r!t |nd}t j||j|d|_|dur7|j|j_tj||_tfddtjD|_jdk|_jdk|_t||_d|_|dS)NrNrPcg|]}t|dqS)r\)r.0ir[r,r- z(BartEncoder.__init__..rrF)r5r6rWZencoder_layerdrop layerdroprr&rOmax_position_embeddingsZmax_source_positionsscale_embeddingmathsqrtrM vocab_sizerrAr/embed_positionsr ModuleListrangeZencoder_layerslayersr_use_flash_attention_2 _use_sdparlayernorm_embeddinggradient_checkpointing post_init)r7r[rrUrPr8rr-r6s,        zBartEncoder.__init__cC|jSrQrrr,r,r-get_input_embeddingsz BartEncoder.get_input_embeddingscC ||_dSrQrr7valuer,r,r-set_input_embeddings z BartEncoder.set_input_embeddingsr%rj head_maskrrloutput_hidden_states return_dictrncCs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur:|}|d|jd}n|durJ|dddddf}ntd|durW||}||} | |j } || } | | } t j j| |j|jd} |dur|jrd|vr|nd}n|jr|dur|st||j}nt||j}|rdnd} |rdnd} |dur|dt|jkrtdt|jd |dd t|jD]\\} }|r| | f} d }|jrtg}||jkrd }|rd }n1|jr |jr ||j| ||dur|| nd|}n|| ||dur|| nd|d}|d} |r*| |df} q|r3| | f} |sBtdd| | | fDSt | | | dS)a~ Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timer(z5You have to specify either input_ids or inputs_embedsrsrr,z&The head_mask should be specified for  layers, but it is for rFT)NN)rkrlr cs|] }|dur|VqdSrQr,rvr,r,r- <sz&BartEncoder.forward..last_hidden_staterg attentions)!r[rlruse_return_dictr+rwr)rrrr=r rrrWrur r rr<rrvlenr  enumerater>randrr _gradient_checkpointing_func__call__tupler)r7r%rjrrrlrrinputZ embed_posrgZencoder_statesZall_attentionsidxZ encoder_layerZto_dropdropout_probability layer_outputsr,r,r-rCs.            zBartEncoder.forwardrQ)NNNNNNN)rFrGrHrIr!rrrr6rrr> LongTensorrKrrrrrrCrLr,r,r8r-rs:  rcseZdZdZddedeejffdd ZddZ d d Z             dd ee j d ee j d ee jdee j dee j dee j deee jdee jdeedeedeedeedee j deeeffddZZS) BartDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`BartDecoderLayer`] Args: config: BartConfig embed_tokens (nn.Embedding): output embedding Nr[rcstj|_j|_j|_j|_j rt j nd}t jj |j|d|_|dur6|j|j_tjj |_tfddtjD|_jdk|_jdk|_tj |_d|_|dS)NrNrcrr)rrrr,r-r^rz(BartDecoder.__init__..rrF)r5r6rWZdecoder_layerdroprr&rOrZmax_target_positionsrrrrrMrrrAr/rrrrZdecoder_layersr rr r rr r r)r7r[rrPr8rr-r6Ks*      zBartDecoder.__init__cCrrQrrr,r,r-rhrz BartDecoder.get_input_embeddingscCrrQrrr,r,r-rkrz BartDecoder.set_input_embeddingsr%rjrrrcross_attn_head_maskrrrrlrrrmrnc"Cs<| dur| n|jj} | dur| n|jj} | dur| n|jj} | dur$| n|jj} |jr7|jr7| r7tdd} |du|duArCt d|durP| d|j d}|durY| |}d}| rnt |tsnd}tdt|}|dd\}}|dur|nd}| durtj||||jd } |durts||}tj|||jd }t |tr|jn|}|||| || }|dur|dur|jrd|vr|nd}n|jr|dur| st||j|d }nt||j|d }|jt || d }|!|j}||}|"|}t#j$j%||j%|jd }| r d nd}| rd nd}| r |dur d nd}d}t&||gddgD]+\}}|durW|dt'|j(krWt d|dt'|j(d|ddq-t)|j(D]\}}| rj||f7}|jr{t*g}||j+kr{q^|jr|jr|,|j-|||||dur||nd|dur||ndd| | | } n ||||||dur||nd|dur||nd|| | | d } | d}| r| | rdnd}| r|| df7}|dur|| df7}q^| r||f7}| r|nd}!|r|.}!| st/dd||!|||fDSt0||!|||dS)a4 Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer(TzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.rr)r)r;rsr,rr0zThe `z` should be specified for rr) rjrrrkrrirlrrmr r r3csrrQr,rr,r,r-rcsz&BartDecoder.forward..)r!rrgr"cross_attentions)1r[rlrrr#r rur`rar+rwr)rryr r Zfrom_legacy_cachervrr>r?r=rZonesr}rr r rr<rrr*rr rrrWzipr$r r%r&rr'r(Zto_legacy_cacher)r)"r7r%rjrrrr0rrrrlrrrmZreturn_legacy_cacherZ seq_lengthr:Zmask_seq_lengthZself_attn_cacherr;rgZall_hidden_statesZall_self_attnsZall_cross_attentionsZnext_decoder_cacherZ mask_namer+Z decoder_layerr,r-Z next_cacher,r,r-rCnsT               zBartDecoder.forwardrQ) NNNNNNNNNNNNN)rFrGrHrIr!rrrr6rrr>r.rKrrrrrrrCrLr,r,r8r-r/Bs^       r/c&s.eZdZddgZdeffdd ZddZdd Zd d Zd d Z ddZ e                d$de e jde e jde e jde e jde e jde e jde e jde ee jde ee jde e jde e jde ede ede ede ed e e jd!eeeff"d"d#ZZS)% BartModelencoder.embed_tokens.weightdecoder.embed_tokens.weightr[cslt||j|j}}|jrt|jnd}t||j||d|_ t ||j |_ t ||j |_ |dS)NrNr)r5r6r&rrrrrrMsharedrencoderr/decoderr)r7r[rOrrPr8r,r-r6us  zBartModel.__init__cCs|jjrB|jjjtdkr.|jjjjtdkr.||j j|jj||j|jjdS||j j|j||jj|jdSdS)Nmeta) r[tie_word_embeddingsr6rAr=r>r8r_tie_or_clone_weightsr7rr,r,r- _tie_weightss zBartModel._tie_weightscCrrQ)r6rr,r,r-rrzBartModel.get_input_embeddingscCs||_|j|j_|j|j_dSrQ)r6r7rr8rr,r,r-rs zBartModel.set_input_embeddingscCrrQ)r7rr,r,r- get_encoderrzBartModel.get_encodercCrrQr8rr,r,r- get_decoderrzBartModel.get_decoderNr%rjdecoder_input_idsdecoder_attention_maskrdecoder_head_maskr0encoder_outputsrrdecoder_inputs_embedsrrlrrrmrncCsJ|dur| dur|durtdt||jj|jj}| dur | n|jj} |dur*|n|jj}| dur4| n|jj} |dur>|n|jj}|durS|j |||| | ||d}n$|rwt |t swt |dt |dkrh|dndt |dkrs|dndd}|j |||d|||| | | | |||d }|s||St|j|j|j|j|j|j|j|jd S) $ decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. NzIf no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.)r%rjrrrlrrrr r3r  r%rjrrrr0rrrrlrrrm)r!rdecoder_hidden_statesdecoder_attentionsr1encoder_last_hidden_staterencoder_attentions)r+r.r[r&r'rlrrr#r7ryrr$r8rr!rrgr"r1)r7r%rjr@rArrBr0rCrrrDrrlrrrmZdecoder_outputsr,r,r-rCsp3 zBartModel.forwardNNNNNNNNNNNNNNNN)rFrGrH_tied_weights_keysr!r6r<rrr=r?rrr>r.rKrrrrrrrCrLr,r,r8r-r3qsx        r3zV The BART Model with a language modeling head. Can be used for summarization. )Z custom_introc(seZdZdZgdZdgZdeffdd ZddZd d Z d2d e de e de de jffdd Zd e dd fddZddZddZddZe d3de ejde ejde ejde ejde ejd e ejd!e ejd"e eejd#e eejd$e ejd%e ejd&e ejd'e e d(e e d)e e d*e e d+e ejdeeeff$d,d-Zd&ejfd.d/Zed0d1Z Z!S)4BartForConditionalGenerationr)r4r5lm_head.weightfinal_logits_biasr[csXt|t||_|dtd|jjjft j |j |jjjdd|_ | dS)NrOr Fr])r5r6r3rregister_bufferr>zerosr6r0rrbrlm_headrr7r[r8r,r-r6s   z%BartForConditionalGeneration.__init__cC |jSrQ)rr=rr,r,r-r='rz(BartForConditionalGeneration.get_encodercCrTrQ)rr?rr,r,r-r?*rz(BartForConditionalGeneration.get_decoderNTnew_num_tokenspad_to_multiple_of mean_resizingrncs&t|||}||jjd|S)Nr)r5resize_token_embeddings_resize_final_logits_biasrAr))r7rUrVrWnew_embeddingsr8r,r-rX-sz4BartForConditionalGeneration.resize_token_embeddingscCsj|jjd}||kr|jddd|f}ntjd||f|jjd}tj|j|gdd}|d|dS)Nr(r rrqrO)rOr)r>rQr=catrP)r7rUZold_num_tokensZnew_biasZ extra_biasr,r,r-rY4s z6BartForConditionalGeneration._resize_final_logits_biascCrrQrRrr,r,r-get_output_embeddings=rz2BartForConditionalGeneration.get_output_embeddingscCrrQr\r7rZr,r,r-set_output_embeddings@rz2BartForConditionalGeneration.set_output_embeddingscCs,|jjr|j||j|jjdSdSrQ)r[r:rr<r;rRr6rr,r,r-r<Cs z)BartForConditionalGeneration._tie_weightsr%rjr@rArrBr0rCrrrDlabelsrrlrrrmcCs.|dur|n|jj}| dur)| rtdd} |dur)| dur)t| |jj|jj}|j|f|||||||| | | | ||||d}||d}||j |j }d}| durm| |j } t }|| d|jj| d}|s|f|dd}|dur|f|S|St|||j|j|j|j|j|j|jd S) aa decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example summarization: ```python >>> from transformers import AutoTokenizer, BartForConditionalGeneration >>> model = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn") >>> ARTICLE_TO_SUMMARIZE = ( ... "PG&E stated it scheduled the blackouts in response to forecasts for high winds " ... "amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were " ... "scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow." ... ) >>> inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors="pt") >>> # Generate Summary >>> summary_ids = model.generate(inputs["input_ids"], num_beams=2, min_length=0, max_length=20) >>> tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] 'PG&E scheduled the blackouts in response to forecasts for high winds amid dry conditions' ``` Mask filling example: ```python >>> from transformers import AutoTokenizer, BartForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("facebook/bart-base") >>> model = BartForConditionalGeneration.from_pretrained("facebook/bart-base") >>> TXT = "My friends are but they eat too many carbs." >>> input_ids = tokenizer([TXT], return_tensors="pt")["input_ids"] >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['not', 'good', 'healthy', 'great', 'very'] ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)rjr@rCrArrBr0rrrDrrlrrrmrr(r losslogitsrrGrHr1rIrrJ)r[r#r`warningr.r&r'rrRrOrr=rrwrrrrGrHr1rIrrJ)r7r%rjr@rArrBr0rCrrrDr`rrlrrrmrZ lm_logitsZmasked_lm_lossloss_fctoutputr,r,r-rCHsb_  z$BartForConditionalGeneration.forwardcCst||jj|jjSrQ)r.r[r&r')r7r`r,r,r-%prepare_decoder_input_ids_from_labelsszBBartForConditionalGeneration.prepare_decoder_input_ids_from_labelscsBd}|D]}|tfdd|ddD|ddf7}q|S)Nr,c3$|] }|d|jVqdSrEZ index_selectrr=rZ past_statebeam_idxr,r-r"z>BartForConditionalGeneration._reorder_cache..r3r)rrlZreordered_pastZ layer_pastr,rkr-_reorder_caches z+BartForConditionalGeneration._reorder_cache)NTNNNNNNNNNNNNNNNNN)"rFrGrHrrLZ_keys_to_ignore_on_load_missingr!r6r=r?rJrrrrrXrYr]r_r<rr>r.rKrrrrrrCrgrrprLr,r,r8r-rMs         rMz Bart model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c&seZdZddgZdeffdd Ze                ddeej deej d eej d eej d eej d eej d eej dee ej deej deej deej dee dee dee dee deej deeeff"ddZZS)BartForSequenceClassificationr4r5r[c sBtj|fi|t||_t|j|j|j|j|_| dSrQ) r5r6r3rrr num_labelsZclassifier_dropoutclassification_headr)r7r[rr8r,r-r6s  z&BartForSequenceClassification.__init__Nr%rjr@rArrBr0rCrrDr`rrlrrrmrncCs<|dur|n|jj}| durd} |dur!| dur!td|jj|j||||||||| | | | |||d}|d}||jj|j }t t | ddkrTtd||ddf|dd|ddddddf}||}d}| dur| |j } |jjdur|jjdkrd |j_n|jjdkr| jt jks| jt jkrd |j_nd |j_|jjd krt}|jjdkr||| }n,||| }n&|jjd krt}||d|jj| d}n|jjd krt}||| }|s |f|dd}|dur|f|S|St|||j|j|j|j|j |j!|j"d S) aN decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for rjr@rArrBr0rCrrDrrlrrrmrr z7All examples must have the same number of tokens.r(Z regressionZsingle_label_classificationZmulti_label_classificationra)#r[r#NotImplementedErrorr9rFreqZ eos_token_idrr=r$r>Zunique_consecutivesumr+rwrvrtZ problem_typersr<r@rJr squeezerrrrrGrHr1rIrrJ)r7r%rjr@rArrBr0rCrrDr`rrlrrrmrrgZeos_maskZsentence_representationrcrbrerfr,r,r-rCs4 $     $       z%BartForSequenceClassification.forwardrK)rFrGrHrLr!r6rrr>r.rKrrrrrrrCrLr,r,r8r-rrsn       rrc(seZdZddgZfddZe                 ddeejdeejdeej d eej d eejd eejd eejd ee ej deej deej deej deej dee dee dee dee deej de eeff$ddZZS)BartForQuestionAnsweringr4r5csBt|d|_|j|_t||_t|j|j|_| dSr2) r5r6rsr3rrrb hidden_size qa_outputsrrSr8r,r-r6s   z!BartForQuestionAnswering.__init__Nr%rjr@rArrBr0rCstart_positions end_positionsrrDrrlrrrmrncCs||dur|n|jj}| dur| durd} |j||||||||| | | ||||d}|d}||}|jddd\}}|d}|d}d}| dur| durt| dkr_| d} t| dkrl| d} |d}| d|} | d|} t |d}||| }||| }||d }|s||f|dd}|dur|f|S|St ||||j |j |j|j|j|j|jd S) rENFrurr r(rq)Z ignore_indexr3) rb start_logits end_logitsrrGrHr1rIrrJ)r[r#rr|splitryrr$rvrrrrrGrHr1rIrrJ)r7r%rjr@rArrBr0rCr}r~rrDrrlrrrmrZsequence_outputrcrrZ total_lossZ ignored_indexreZ start_lossZend_lossrfr,r,r-rCsr2           z BartForQuestionAnswering.forwardrq)rFrGrHrLr6rrr>rKr.rrrrrrrCrLr,r,r8r-rzst        rzcs(eZdZdZfddZddZZS)BartDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cst|t||_dSrQ)r5r6r/r8rSr8r,r-r6s zBartDecoderWrapper.__init__cOs|j|i|SrQr>rr,r,r-rCszBartDecoderWrapper.forward)rFrGrHrIr6rCrLr,r,r8r-r s rzu BART decoder with a language modeling head on top (linear layer with weights tied to the input embeddings). c"seZdZdgZfddZddZddZdd Zd d Zd d Z ddZ e              d$de e jde e jde e jde e jde e jde e jde ee jde e jde e jde ede ede ede ede e jdeeeffd d!Zed"d#ZZS)%BartForCausalLMrNcsNt|}d|_d|_t|t||_tj |j |j dd|_ | dS)NTFr])copydeepcopyrXZis_encoder_decoderr5r6rrrrbr{rrRrrSr8r,r-r6s    zBartForCausalLM.__init__cCs |jjjSrQrr8rrr,r,r-r+rz$BartForCausalLM.get_input_embeddingscCs||jj_dSrQrrr,r,r-r.sz$BartForCausalLM.set_input_embeddingscCrrQr\rr,r,r-r]1rz%BartForCausalLM.get_output_embeddingscCrrQr\r^r,r,r-r_4rz%BartForCausalLM.set_output_embeddingscCs ||j_dSrQrr8)r7r8r,r,r- set_decoder7s zBartForCausalLM.set_decodercCs|jjSrQrrr,r,r-r?:szBartForCausalLM.get_decoderNr%rjrrrr0rrr`rrlrrrmrncCs| dur| n|jj} | dur| n|jj} | dur| n|jj} |jj||||||||| | | | |d }||d}d}| durU| |j} t }|| d|jj | d}| sk|f|dd}|duri|f|S|St |||j |j|j|jdS)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("facebook/bart-base") >>> model = BartForCausalLM.from_pretrained("facebook/bart-base", add_cross_attention=False) >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```NrFrr(r )rbrcrrgr"r1)r[rlrr#rr8rRrr=rrwrrrrgr"r1)r7r%rjrrrr0rrr`rrlrrrmrrcrbrerfr,r,r-rC=sH. zBartForCausalLM.forwardcs.d}|D]}|tfdd|Df7}q|S)Nr,c3rhrErirjrkr,r-rrmz1BartForCausalLM._reorder_cache..rnror,rkr-rps zBartForCausalLM._reorder_cache)NNNNNNNNNNNNNN)rFrGrHrLr6rrr]r_rr?rrr>r.rKrrrrrrrCrrprLr,r,r8r-rsr          Yr)rrMrzrrr3rrr)QrIrrrtypingrrrrr>Ztorch.utils.checkpointrZtorch.nnrrr Z activationsr Z cache_utilsr r Z generationrZmodeling_attn_mask_utilsrrrZmodeling_flash_attention_utilsrrZmodeling_outputsrrrrrrrZmodeling_utilsrutilsrrrrZconfiguration_bartr!Z!torch.nn.attention.flex_attentionr"Zintegrations.flex_attentionr#r$Z get_loggerrFr`rKrJr.rr/rMModulerSrrrrrrrrrrr/r3rMrrrzrr__all__r,r,r,r-s   $       wjGv#=1#U