o Zh@s,dZddlZddlZddlZddlZddlmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddlmZddlmZmZdd lmZdd lmZmZdd lmZmZmZmZmZdd lm Z dd l!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*e#rddl+m,Z,ddl-m.Z.e%/e0Z1de j2de3de3fddZ4Gddde j5Z6Gddde j5Z7Gddde j8Z9de9iZ:Gd d!d!e j8Z;Gd"d#d#e j8ZGd(d)d)e=Z?e"Gd*d+d+e=Z@e"d,d-Gd.d/d/e=eZAGd0d1d1e=ZBGd2d3d3e=eZCgd4ZDdS)5zPyTorch Blenderbot model.N)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)CacheEncoderDecoderCache)GenerationMixin)AttentionMaskConverter_prepare_4d_attention_mask)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)auto_docstringis_torch_flex_attn_availableis_torchdynamo_compilinglogging)'BlenderbotSmallForConditionalGenerationBlenderbotSmallModel)BlenderbotConfig) BlockMask)make_flex_block_causal_mask input_ids pad_token_iddecoder_start_token_idcCsh||j}|ddddf|ddddf<||dddf<|dur*td||dk||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i)Z new_zerosshapeclone ValueErrorZ masked_fill_)r r!r"Zshifted_input_idsr'a/var/www/auris/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.pyshift_tokens_right?s (r)csPeZdZdZdedeffdd Z d dejd ed eej ffd d Z Z S)$BlenderbotLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcst||dSN)super__init__)selfr+r, __class__r'r(r/Tz-BlenderbotLearnedPositionalEmbedding.__init__rNinput_ids_shapepast_key_values_length position_idscs@|dur|dd\}}tj|||tj|jjd}t|S)z3`input_ids_shape` is expected to be [bsz x seqlen].Nr)dtypedevice)torcharangelongweightr8r.forward)r0r4r5r6bszZseq_lenr1r'r(r=Ws  z,BlenderbotLearnedPositionalEmbedding.forwardrN) __name__ __module__ __qualname____doc__intr/r9SizerTensorr= __classcell__r'r'r1r(r*Osr*c sLeZdZdZd dedededeeffdd Zd ej ffd d Z Z S) BlenderbotScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. ?r+r, padding_idx embed_scalecst|||||_dSr-)r.r/rK)r0r+r,rJrKr1r'r(r/is z&BlenderbotScaledWordEmbedding.__init__r cst||jSr-)r.r=rK)r0r r1r'r(r=mr3z%BlenderbotScaledWordEmbedding.forward)rI) r@rArBrCrDrfloatr/r9rFr=rGr'r'r1r(rHds$rHcseZdZdZ      ddededed ed ed ed eed eeffdd Z      dde j dee j dee dee j dee j dedee j de e j ee j ee e j ffddZZS)BlenderbotAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsdropout is_decoderbias is_causalconfig layer_idxc st||_||_||_|||_||_|j||jkr*td|jd|d|jd|_||_ ||_ ||_ |durK|j rKt d|jjdtj|||d|_tj|||d|_tj|||d|_tj|||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gzInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.rS)r.r/rOrPrQhead_dimrUr&scalingrRrTrVlogger warning_oncer2r@rLineark_projv_projq_projout_proj) r0rOrPrQrRrSrTrUrVr1r'r(r/us0    zBlenderbotAttention.__init__ hidden_stateskey_value_statespast_key_valueattention_masklayer_head_maskoutput_attentionscache_positionreturncCsv|du}|\} } } ||| d|j|jdd} | |j} |dur=t|tr;|j |j } |r7|j }n|j }n|}|rA|n|}|rX|durX| rX|j|j }|j|j }nE||}||}|| d|j|jdd}|| d|j|jdd}|dur|s|nd}||||j d|i\}}|rd|j |j <| |jd|jf}| j|} |j|}|j|}|d}t| |dd}|| |j| |fkrtd| |j| |fd||dur|ddddddd|jd f}|| |j| ||}|| |j| |}tjj|dd }|durN||jfkr3td |jfd||dddd|| |j| |}|| |j| |}|re|| |j| |}|| |j| |}nd}tjj||j|jd }t||}|| |j| |jfkrtd | |j| |jfd||| |j| |j}|dd}|| | |j}||}|||fS)z#Input shape: Batch x Time x ChannelNr#rrrgTz$Attention weights should be of size z , but is dimz/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )sizer_viewrPrXZ transposerY isinstancer is_updatedgetrVZcross_attention_cacheself_attention_cacheZ key_cacheZ value_cacher]r^updatereshaper9Zbmmr&r$r functionalZsoftmaxrQrnrOr`)r0rarbrcrdrerfrgZis_cross_attentionr>tgt_len_Z query_statesrrZcurr_past_key_valueZcurrent_statesZ key_statesZ value_statesZ proj_shapeZsrc_len attn_weightsZattn_weights_reshapedZ attn_probsZ attn_outputr'r'r(r=s"             &  "    zBlenderbotAttention.forward)rNFTFNN)NNNNFN)r@rArBrCrDrLboolrrr/r9rFr rr=rGr'r'r1r(rMrs` * rMeagerc sLeZdZdeffdd Z d dejdejdejded ejf d d ZZ S) BlenderbotEncoderLayerrUcst|j|_t|j|j|j|j|d|_t |j|_ |j |_ t |j|_|j|_t |j|j|_t |j|j|_t |j|_dS)N)rOrPrQrU)r.r/d_modelrOBLENDERBOT_ATTENTION_CLASSES_attn_implementationZencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrQr activation_function activation_fnactivation_dropoutr\Zencoder_ffn_dimfc1fc2final_layer_normr0rUr1r'r(r/s  zBlenderbotEncoderLayer.__init__Frardrerfrhc Cs|}||}|j||||d\}}}tjj||j|jd}||}|}||}|||}tjj||j |jd}| |}tjj||j|jd}||}|j t j krht |j jd}t j|| |d}|f} |rr| |f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rardrerfrli)minmax)rrrrwrQrnrrrrrr7r9Zfloat16finforclamp) r0rardrerfresidualrzryZ clamp_valueoutputsr'r'r(r=&s0      zBlenderbotEncoderLayer.forwardF) r@rArBrr/r9rFr{r=rGr'r'r1r(r}sr}cseZdZddedeeffdd Z         ddejd eejd eejd eejd eejd eejdee dee dee deejdejfddZ Z S)BlenderbotDecoderLayerNrUrVc st|j|_t|j|j|j|jdd||d|_|j |_ t |j |_ |j |_ t|j|_t|j|j|j|jd||d|_t|j|_t|j|j|_t|j|j|_t|j|_dS)NT)rOrPrQrRrTrUrV)rQrRrUrV)r.r/r~rOrrZdecoder_attention_headsrrrQr rrrrrr encoder_attnencoder_attn_layer_normr\Zdecoder_ffn_dimrrr)r0rUrVr1r'r(r/Ys6  zBlenderbotDecoderLayer.__init__FTrardencoder_hidden_statesencoder_attention_maskrecross_attn_layer_head_maskrcrf use_cachergrhc Cs |} ||}|j|||||| d\}} }tjj||j|jd}| |}d} |durN|} ||}|j||||||d\}} }tjj||j|jd}| |}|} ||}| | |}tjj||j |jd}| |}tjj||j|jd}| |}|f}|r|| | f7}| r||f7}|S)a8 Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. )rarcrdrerfrgrlN)rarbrdrercrf) rrrrwrQrnrrrrrrr)r0rardrrrerrcrfrrgrZself_attn_weightsZcross_attn_weightsrr'r'r(r=xsN"        zBlenderbotDecoderLayer.forwardr-) NNNNNNFTN) r@rArBrrrDr/r9rFr r{r=rGr'r'r1r(rXsD"    rc @seZdZeZdZdZdZdZddZ e ddZ dde e jd fd e jd e jd ed ef ddZede jdedede jd e jdef ddZdS)BlenderbotPreTrainedModelmodelTcCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|tj rC|jjjd|d|j durA|jj|j  dSdSt|tj rX|jj d|jj dSdS)NrN)meanstdrI)rUZinit_stdrqrr\r<dataZnormal_rSZzero_ EmbeddingrJrZfill_)r0modulerr'r'r( _init_weightss     z'BlenderbotPreTrainedModel._init_weightscCs@|jj}tjgddddd|gg|jd}||||d}|S)N)r rr rr8)rdr decoder_input_ids)rUr!r9Ztensorr8ne)r0Z pad_tokenr dummy_inputsr'r'r(rs"z&BlenderbotPreTrainedModel.dummy_inputsFrdr input_tensorrgpast_key_valuesrfc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) NZflash_attention_2rNZflex_attentionrFZsdpa) inputs_embedsr5Z is_trainingrr#)sequence_length target_lengthr7rg batch_size)cudaZxpuZnpu)rUranyrqr9rFrget_seq_lengthZis_compileabler Z_ignore_causal_mask_sdparnr7r$Zget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionr8typerrZ_unmask_unattended) r0rdrrgrrfZpast_seen_tokensZusing_compilable_cacher7rr causal_mask min_dtyper'r'r(_update_causal_masksT           z-BlenderbotPreTrainedModel._update_causal_maskrrr7rc KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr)Z fill_valuer7r8r)Zdiagonalrr#r)rkr9rrfullr8Ztriur:rvexpandr%r$toZ masked_fill) rdrrr7rgrkwargsrrZ mask_lengthZ padding_maskr'r'r(r5s,  $ 6  zOBlenderbotPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_positionNr)r@rArBrZ config_classbase_model_prefixZsupports_gradient_checkpointingZ_supports_cache_classZ_supports_static_cacherpropertyrrr9rFr r{r staticmethodrDr7rr'r'r'r(rsF   DrcsJeZdZdZd dedeejffdd Z       d ddZ Z S) BlenderbotEncoderz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`BlenderbotEncoderLayer`]. Args: config: BlenderbotConfig embed_tokens (nn.Embedding): output embedding NrU embed_tokenscstj|_j|_j}j|_j|_ j r!t |nd}|dur+||_ n tj||j|d|_ tj||_tfddtjD|_tj|_d|_|dS)NrIrKcsg|]}tqSr')r}).0ryrUr'r( sz.BlenderbotEncoder.__init__..F)r.r/rQZencoder_layerdrop layerdropr~r!rJmax_position_embeddingsZmax_source_positionsscale_embeddingmathsqrtrrH vocab_sizer*embed_positionsr ModuleListrangeZencoder_layerslayersr layer_normgradient_checkpointing post_init)r0rUrrOrKr1rr(r/xs(    zBlenderbotEncoder.__init__cCsN|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|durA||||}|d|d}n|durN|dd}ntd|dur[||}| |} || } t j j | |j |j d} |duryt||j}|r}dnd} |rdnd} |dur|dt|jkrtdt|jd |dd t|jD]X\} }|r| | f} d }|j rtg}||jkrd }|rd }n-|jr|j r||j| ||dur|| nd|}n|| ||dur|| nd|d}|d} |r| |df} q|| } |r| | f} |s tdd| | | fDSt| | | dS)a~ Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timer#z5You have to specify either input_ids or inputs_embedsrlr'rz&The head_mask should be specified for  layers, but it is for .FT)NN)rerfrcs|] }|dur|VqdSr-r'rvr'r'r( sz,BlenderbotEncoder.forward..last_hidden_statera attentions)rUrfoutput_hidden_statesuse_return_dictr&Z%warn_if_padding_and_no_attention_maskrorprrrrwrQrnrr7lenr enumerater9randrr_gradient_checkpointing_func__call__rtupler)r0r rd head_maskrrfr return_dictZ input_shapeZ embed_posraZencoder_statesZall_attentionsidxZ encoder_layerZto_dropdropout_probability layer_outputsr'r'r(r=s.             zBlenderbotEncoder.forwardr-)NNNNNNN) r@rArBrCrrrrr/r=rGr'r'r1r(rns rcsreZdZdZddedeejffdd ZddZ d d Z             dd ee j fd d Z ZS)BlenderbotDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`BlenderbotDecoderLayer`] Args: config: BlenderbotConfig embed_tokens (nn.Embedding): output embedding NrUrcstj|_j|_j|_j|_j rt j nd}|dur)||_ n tjj |j|d|_ tjj |_tfddtjD|_tj |_d|_|dS)NrIrcsg|]}t|dqS))rV)r)rirr'r(r8sz.BlenderbotDecoder.__init__..F)r.r/rQZdecoder_layerdroprr!rJrZmax_target_positionsrrrr~rrHrr*rrrrZdecoder_layersrrrrr)r0rUrrKr1rr(r/$s*  zBlenderbotDecoder.__init__cC|jSr-rr0r'r'r(get_input_embeddings@z&BlenderbotDecoder.get_input_embeddingscC ||_dSr-rr0valuer'r'r(set_input_embeddingsC z&BlenderbotDecoder.set_input_embeddingsrgc"Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} | dur$| n|jj} |jr7|jr7| r7tdd} |du|duArCt d|durP| d|j d}|durY| |}d}| rnt |tsnd}tdt|}|dd\}}|dur|nd}| durtj||||jd } |durts||}tj|||jd }t |tr|jn|}|||| || }|dur|durt||j|d }|j||f|| d }||}tjj ||j |jd }| rd nd}| rd nd}| r|durd nd}d}t!||gddgD]+\}}|dur,|dt"|j#kr,t d|dt"|j#d|ddqt$|j#D]\}}| r?||f7}|jrPt%g}||j&krPq3|jr{|jr{|'|j(|||||duri||nd|durs||ndd| | | } n ||||||dur||nd|dur||nd|| | | d } | d}| r| | rdnd}| r|| df7}|dur|| df7}q3|)|}| r||f7}| r|nd}!|r|*}!| st+dd||!|||fDSt,||!|||dS)aS Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer#TzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.rr)rx)r6rlr'rcross_attn_head_maskzThe `z` should be specified for rr) rdrrrerrcrfrrgrrrcsrr-r'rr'r'r(r1sz,BlenderbotDecoder.forward..)rrrarcross_attentions)-rUrfrrrrrnrZwarningr&rpr$rrqr r[r Zfrom_legacy_cacherorr9r:r8rZonesrtrrr7rrrwrQziprrrrrrrrZto_legacy_cacherr)"r0r rdrrrrrrrrfrrrgZreturn_legacy_cacherZ seq_lengthr5Zmask_seq_lengthZself_attn_cacherr6raZall_hidden_statesZall_self_attnsZall_cross_attentionsZnext_decoder_cacheZ attn_maskZ mask_namerZ decoder_layerrrZ next_cacher'r'r(r=FsU                 zBlenderbotDecoder.forwardr-) NNNNNNNNNNNNN)r@rArBrCrrrrr/rrr9rFr=rGr'r'r1r(rs*rc&sReZdZddgZdeffdd Zedeee e j fffdd Z d d Z d d Zd dZddZe                d%deejdeejdeejdeejdeejdeejdeejdeeeefdeeejdeejdeejdeedeedeed eed!eejd"eeejeff"d#d$ZZS)&BlenderbotModeldecoder.embed_tokens.weightencoder.embed_tokens.weightrUcslt||j|j}}|jrt|jnd}t||j||d|_ t ||j |_ t ||j |_ |dS)NrIr)r.r/r!rrrrr~rHsharedrencoderrdecoderr)r0rUrJrrKr1r'r(r/Cs  zBlenderbotModel.__init__pretrained_model_name_or_pathc<|dkrtdtt|Stt|j|g|Ri|S)Nfacebook/blenderbot-90MzThe checkpoint `facebook/blenderbot-90M` is deprecated. In the future, please use the identical checkpoint `facebook/small_blenderbot-90M` with `BlenderbotSmallModel.from_pretrained('facebook/small_blenderbot-90M')` instead.)warningswarn FutureWarningrfrom_pretrainedr.rclsrZ model_argsrr1r'r(rOs zBlenderbotModel.from_pretrainedcCrr-)rrr'r'r(r\rz$BlenderbotModel.get_input_embeddingscCs||_|j|j_|j|j_dSr-)rrrrrr'r'r(r_s z$BlenderbotModel.set_input_embeddingscCrr-)rrr'r'r( get_encoderdrzBlenderbotModel.get_encodercCrr-rrr'r'r( get_decodergrzBlenderbotModel.get_decoderNr rdrdecoder_attention_maskrdecoder_head_maskrencoder_outputsrrdecoder_inputs_embedsrrfrrrgrhcCs| dur| n|jj} |dur|n|jj}| dur| n|jj} |dur$|n|jj}|dur9|j|||| | ||d}n$|r]t|ts]t|dt|dkrN|dndt|dkrY|dndd}|j |||d|||| | | | |||d }|sw||St |j |j |j |j|j|j |j |jdS) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Blenderbot uses the `bos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: ```python >>> from transformers import AutoTokenizer, BlenderbotModel >>> model = BlenderbotModel.from_pretrained("facebook/blenderbot-400M-distill") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/blenderbot-400M-distill") >>> inputs = tokenizer("Studies have been shown that owning a dog is good for you", return_tensors="pt") >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> outputs = model(input_ids=inputs.input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 6, 1280] ```N)r rdrrrfrrrrrr r rdrrrrrrrrfrrrg)rrdecoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentions)rUrfrrrrrqrrrrrrrarr)r0r rdrr rr rr rrrrrfrrrgZdecoder_outputsr'r'r(r=js`: zBlenderbotModel.forward)NNNNNNNNNNNNNNNN)r@rArB_tied_weights_keysrr/ classmethodrrstrosPathLikerrrrr rr9 LongTensorrFrrr FloatTensorr{rr=rGr'r'r1r(r?sz "      rz\ The Blenderbot Model with a language modeling head. Can be used for summarization. )Z custom_introc(seZdZdZdgZgdZdeffdd Zede e e e j fffdd Zd d Zd d Z d1dede ededejffdd ZdeddfddZddZddZe                 d2de ejde ejde ejde ejd e ejd!e ejd"e ejd#e e eefd$e eej d%e ejd&e ej d'e ejd(e ed)e ed*e ed+e ed,e ejde eej e!ff$d-d.Z"e#d/d0Z$Z%S)3"BlenderbotForConditionalGenerationrfinal_logits_bias)rrlm_head.weightrUcsXt|t||_|dtd|jjjft j |j |jjjdd|_ | dS)NrrFrW)r.r/rrregister_bufferr9zerosrr+rr\r~lm_headrrr1r'r(r/s   z+BlenderbotForConditionalGeneration.__init__rcr)NrzThe checkpoint `facebook/blenderbot-90M` is deprecated. In the future, please use the identical checkpoint `facebook/small_blenderbot-90M` with `BlenderbotSmallForConditionalGeneration.from_pretrained('facebook/small_blenderbot-90M')` instead.)rrrrrr.rrr1r'r(rs  z2BlenderbotForConditionalGeneration.from_pretrainedcC |jSr-)rrrr'r'r(rrz.BlenderbotForConditionalGeneration.get_encodercCr!r-)rr rr'r'r(r rz.BlenderbotForConditionalGeneration.get_decoderNTnew_num_tokenspad_to_multiple_of mean_resizingrhcs&t|||}||jjd|S)Nr)r.resize_token_embeddings_resize_final_logits_biasr<r$)r0r"r#r$new_embeddingsr1r'r(r%sz:BlenderbotForConditionalGeneration.resize_token_embeddingscCsj|jjd}||kr|jddd|f}ntjd||f|jjd}tj|j|gdd}|d|dS)Nr#rrrjr)rr$r9rr8catr)r0r"Zold_num_tokensZnew_biasZ extra_biasr'r'r(r& s z>> from transformers import AutoTokenizer, BlenderbotForConditionalGeneration >>> mname = "facebook/blenderbot-400M-distill" >>> model = BlenderbotForConditionalGeneration.from_pretrained(mname) >>> tokenizer = AutoTokenizer.from_pretrained(mname) >>> UTTERANCE = "My friends are cool but they eat too many carbs." >>> print("Human: ", UTTERANCE) Human: My friends are cool but they eat too many carbs. >>> inputs = tokenizer([UTTERANCE], return_tensors="pt") >>> reply_ids = model.generate(**inputs) >>> print("Bot: ", tokenizer.batch_decode(reply_ids, skip_special_tokens=True)[0]) Bot: That's unfortunate. Are they trying to lose weight or are they just trying to be healthier? >>> REPLY = "I'm not sure" >>> print("Human: ", REPLY) Human: I'm not sure >>> NEXT_UTTERANCE = ( ... "My friends are cool but they eat too many carbs. That's unfortunate. " ... "Are they trying to lose weight or are they just trying to be healthier? " ... " I'm not sure." ... ) >>> inputs = tokenizer([NEXT_UTTERANCE], return_tensors="pt") >>> next_reply_ids = model.generate(**inputs) >>> print("Bot: ", tokenizer.batch_decode(next_reply_ids, skip_special_tokens=True)[0]) Bot: I see. Well, it's good that they're trying to change their eating habits. ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)rdrr r rr rrrrrrfrrrgrr#r) losslogitsrrrrrrr)rUrrZrr)r!r"rr rrrprrrrrrrrr)r0r rdrr rr rr rrrr-rrfrrrgrZ lm_logitsZmasked_lm_lossloss_fctoutputr'r'r(r=s^O z*BlenderbotForConditionalGeneration.forwardcsBd}|D]}|tfdd|ddD|ddf7}q|S)Nr'c3$|] }|d|jVqdSr?Z index_selectrr8rZ past_statebeam_idxr'r(r"zDBlenderbotForConditionalGeneration._reorder_cache..rrrr6Zreordered_pastZ layer_pastr'r5r(_reorder_caches z1BlenderbotForConditionalGeneration._reorder_cache)NT)NNNNNNNNNNNNNNNNN)&r@rArBrZ_keys_to_ignore_on_load_missingrrr/rrrrrrrrr rDr{rrr%r&r*r,rr9rrFrrrrrr=rr:rGr'r'r1r(rs "       rcs(eZdZdZfddZddZZS)BlenderbotDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cst|t||_dSr-)r.r/rrrr1r'r(r/s z!BlenderbotDecoderWrapper.__init__cOs|j|i|Sr-r )r0argsrr'r'r(r=sz BlenderbotDecoderWrapper.forward)r@rArBrCr/r=rGr'r'r1r(r;s r;c"seZdZdgZfddZddZddZdd Zd d Zd d Z ddZ e              d$de e jde e jde e jde e jde e jde e jde ee jde e jde e jde ede ede ede ede e jdeeeffd d!Zed"d#ZZS)%BlenderbotForCausalLMrcsNt|}d|_d|_t|t||_tj |j |j dd|_ | dS)NTFrW)copydeepcopyrRZis_encoder_decoderr.r/r;rrr\Z hidden_sizerr rrr1r'r(r/s    zBlenderbotForCausalLM.__init__cCs |jjjSr-rrrrr'r'r(rrz*BlenderbotForCausalLM.get_input_embeddingscCs||jj_dSr-r@rr'r'r(rsz*BlenderbotForCausalLM.set_input_embeddingscCrr-r)rr'r'r(r*rz+BlenderbotForCausalLM.get_output_embeddingscCrr-r)r+r'r'r(r,rz+BlenderbotForCausalLM.set_output_embeddingscCs ||j_dSr-rr)r0rr'r'r( set_decoders z!BlenderbotForCausalLM.set_decodercCs|jjSr-rArr'r'r(r sz!BlenderbotForCausalLM.get_decoderNr rdrrrrrrr-rrfrrrgrhcCs| dur| n|jj} | dur| n|jj} | dur| n|jj} |jj||||||||| | | | |d }||d}d}| durU| |j} t }|| d|jj | d}| sk|f|dd}|duri|f|S|St |||j |j|j|jdS)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BlenderbotForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("facebook/blenderbot-400M-distill") >>> model = BlenderbotForCausalLM.from_pretrained("facebook/blenderbot-400M-distill", add_cross_attention=False) >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```Nrrr#r)r.r/rrarr)rUrfrrrrr rr8rrprrrrarr)r0r rdrrrrrrr-rrfrrrgrr/r.r0r1r'r'r(r=sH. zBlenderbotForCausalLM.forwardcs.d}|D]}|tfdd|Df7}q|S)Nr'c3r2r?r3r4r5r'r(r:r7z7BlenderbotForCausalLM._reorder_cache..r8r9r'r5r(r:5s z$BlenderbotForCausalLM._reorder_cache)NNNNNNNNNNNNNN)r@rArBrr/rrr*r,rBr rrr9rrFrrr{rrrr=rr:rGr'r'r1r(r=sr          Yr=)r=rrr)ErCr>rrrtypingrrrrr9Ztorch.utils.checkpointrZtorch.nnrZ activationsr Z cache_utilsr r Z generationr Zmodeling_attn_mask_utilsr rZmodeling_outputsrrrrrZmodeling_utilsrutilsrrrrZblenderbot_smallrrZconfiguration_blenderbotrZ!torch.nn.attention.flex_attentionrZintegrations.flex_attentionrZ get_loggerr@rZrFrDr)rr*rHModulerMrr}rrrrrrr;r=__all__r'r'r'r(sf         Ew.&J