o Zh@sddlZddlmZmZmZmZddlZddlZddlmZm Z m Z ddl m Z ddl mZddlmZddlmZmZmZdd lmZmZmZmZdd lmZd d lmZeeZ Gd dde j!Z"Gddde j!Z#Gddde j!Z$Gddde j!Z%Gddde j!Z&Gddde j!Z'Gddde j!Z(Gddde j!Z)Gddde j!Z*Gdd d e j!Z+Gd!d"d"e j!Z,Gd#d$d$e j!Z-Gd%d&d&eZ.Gd'd(d(e.Z/Gd)d*d*e.eZ0gd+Z1dS),N)ListOptionalTupleUnion)Tensordevicenn)CrossEntropyLoss)ACT2FN)GenerationMixin))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModelapply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)logging)BlipTextConfigc s\eZdZdZfddZ    d deejdeejdeejd e d ej f d d Z Z S)BlipTextEmbeddingsz;Construct the embeddings from word and position embeddings.csttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j |_|jdt|jdddt|dd|_||_dS) N)Z padding_idxZeps position_ids)rF) persistentposition_embedding_typeabsolute)super__init__r Embedding vocab_size hidden_sizeZ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutZregister_buffertorcharangeexpandgetattrrconfigselfr/ __class__Z/var/www/auris/lib/python3.10/site-packages/transformers/models/blip/modeling_blip_text.pyr1s  zBlipTextEmbeddings.__init__Nr input_idsr inputs_embedspast_key_values_lengthreturnc Cs|dur |}n|dd}|d}|dur&|jdd|||f}|dur/||}|}|jdkr?||}||7}||}||}|S)Nrrr)sizerr#rr%r&r*) r1r6rr7r8 input_shape seq_length embeddingsr%r4r4r5forwardCs      zBlipTextEmbeddings.forward)NNNr) __name__ __module__ __qualname____doc__rrr+Z LongTensor FloatTensorintrr> __classcell__r4r4r2r5r.s$ rcseZdZfddZddZddZddZd d Zd d Z dde j de e j de e j de e j de e j de e e e j de ede e j fddZZS)BlipTextSelfAttentioncs t||_|j|jdkrt|dstd|j|jf|j|_t|j|j|_|j|j|_ t |j|j |_ |rQt |j |j |_t |j |j |_nt |j|j |_t |j|j |_t |j|_t|dd|_|jdks{|jdkr|j|_t d|jd |j|_dSdS) NrZembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)rr relative_keyrelative_key_queryr)rrr/r"num_attention_headshasattr ValueErrorrDattention_head_size all_head_sizerLinearqueryZencoder_hidden_sizekeyvaluer(Zattention_probs_dropout_probr*r.rr$r distance_embeddingr1r/is_cross_attentionr2r4r5rcs.  zBlipTextSelfAttention.__init__cC ||_dSNattn_gradients)r1rYr4r4r5save_attn_gradients~ z)BlipTextSelfAttention.save_attn_gradientscC|jSrWrXr1r4r4r5get_attn_gradientsz(BlipTextSelfAttention.get_attn_gradientscCrVrW attention_map)r1rar4r4r5save_attention_mapr[z(BlipTextSelfAttention.save_attention_mapcCr\rWr`r]r4r4r5get_attention_mapr_z'BlipTextSelfAttention.get_attention_mapcCs6|dd|j|jf}|j|}|ddddS)NrrrIrr )r:rJrMviewpermute)r1xZ new_x_shaper4r4r5transpose_for_scoress z*BlipTextSelfAttention.transpose_for_scoresNF hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsr9cCsf||}|du} | r|||} |||} |}n;|durI|||} |||} tj|d| gdd} tj|d| gdd} n|||} |||} ||} | | f}t| | dd} |jdksv|jdkr| d}tj |tj |j d  dd}tj |tj |j d  dd}||}|||jd}|j| jd }|jdkrtd | |}| |} n|jdkrtd | |}td | |}| ||} | t|j} |dur| || j } tjdd| }||}|dur||}t|| }|dddd }| dd|jf}|j |}|r)||fn|f}||f}|S)NrrIdimrrrGrH)dtyperrrzbhld,lrd->bhlrzbhrd,lrd->bhlrr )rPrgrQrRr+catmatmulZ transposerr:r,longrrdrSr$torrZeinsummathsqrtrMrZSoftmaxr*re contiguousrN)r1rhrirjrkrlrmrnZmixed_query_layerrUZ key_layerZ value_layerZ query_layerZattention_scoresr<Zposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_probsZattention_probs_droppedZ context_layerZnew_context_layer_shapeoutputsr4r4r5r>sX            zBlipTextSelfAttention.forwardNNNNNF)r?r@rArrZr^rbrcrgr+rrrCrboolr>rEr4r4r2r5rFbs<  rFc8eZdZfddZdejdejdejfddZZS)BlipTextSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr) rrrrOr"denser&r'r(r)r*r0r2r4r5r zBlipTextSelfOutput.__init__rh input_tensorr9cC&||}||}|||}|SrWrr*r&r1rhrr4r4r5r>  zBlipTextSelfOutput.forwardr?r@rArr+rr>rEr4r4r2r5r $rcseZdZdfdd ZddZ      ddejdeejd eejd eejd eejd ee e ejd ee de ejfddZ Z S)BlipTextAttentionFcs,tt|||_t||_t|_dSrW)rrrFr1routputset pruned_headsrTr2r4r5rs    zBlipTextAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrro)lenrr1rJrMrrrPrQrRrrrNunion)r1headsindexr4r4r5 prune_headss zBlipTextAttention.prune_headsNrhrirjrkrlrmrnr9c Cs<||||||||}||d|} | f|dd} | S)Nrr)r1r) r1rhrirjrkrlrmrnZ self_outputsattention_outputr{r4r4r5r> s  zBlipTextAttention.forward)Fr|) r?r@rArrr+rrrCrr}r>rEr4r4r2r5rs4 rc2eZdZfddZdejdejfddZZS)BlipTextIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSrW) rrrrOr"intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr0r2r4r5r#s   zBlipTextIntermediate.__init__rhr9cC||}||}|SrW)rrr1rhr4r4r5r>+  zBlipTextIntermediate.forwardrr4r4r2r5r"s rcr~)BlipTextOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rrrrOrr"rr&r'r(r)r*r0r2r4r5r3rzBlipTextOutput.__init__rhrr9cCrrWrrr4r4r5r>9rzBlipTextOutput.forwardrr4r4r2r5r2rrcseZdZfddZ      ddejdeejdeejdeejd eejd eeeejd ee d eejfd dZ ddZ Z S) BlipTextLayercs`t||_|j|_d|_t||_||_|jjr$t||jjd|_ t ||_ t ||_ dS)Nr)rU)rrr/chunk_size_feed_forward seq_len_dimr attention layer_num is_decodercrossattentionr intermediaterr)r1r/rr2r4r5rAs   zBlipTextLayer.__init__NFrhrirjrkrlrmrnr9cCs|dur |ddnd}|j|||||d} | d} | dd} | d} |dur?|j| |||||d} | d} | | dd} t|j|j|j| }|f| } | | f} | S)NrI)rnrmrrr)rn)rrrfeed_forward_chunkrr)r1rhrirjrkrlrmrnZself_attn_past_key_valueZself_attention_outputsrr{Zpresent_key_valueZcross_attention_outputs layer_outputr4r4r5r>Ms8    zBlipTextLayer.forwardcCs||}|||}|SrW)rr)r1rZintermediate_outputrr4r4r5rys  z BlipTextLayer.feed_forward_chunkr|) r?r@rArr+rrrCrr}r>rrEr4r4r2r5r@s4  ,rcseZdZfddZ         ddejdeejdeejd eejd eejd eeeejd ee d ee dee dee de eeje ffddZ Z S)BlipTextEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}t|qSr4)r).0ir/r4r5 z,BlipTextEncoder.__init__..F) rrr/rZ ModuleListrangenum_hidden_layerslayergradient_checkpointingr0r2rr5rs   zBlipTextEncoder.__init__NFTrhrirjrkrlpast_key_values use_cachernoutput_hidden_states return_dictr9c  Cs^|jr|jr|rtdd}| rdnd} |rdnd} |r#|jjr#dnd} |r)dnd}t|jjD]]}|j|}| r?| |f} |durG||nd}|durQ||nd}|jrg|jrg| |j |||||||}n ||||||||}|d}|r~||df7}|r| |df} | |df} q1| r| |f} | st dd ||| | | fDSt ||| | | d S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr4rrrrIcss|] }|dur|VqdSrWr4)rvr4r4r5 sz*BlipTextEncoder.forward..)last_hidden_staterrh attentionscross_attentions) rZtrainingloggerwarningr/rrrrZ_gradient_checkpointing_func__call__tupler )r1rhrirjrkrlrrrnrrZall_hidden_statesZall_self_attentionsZall_cross_attentionsZnext_decoder_cacherZ layer_moduleZlayer_head_maskrmZ layer_outputsr4r4r5r>sz           zBlipTextEncoder.forward) NNNNNNFFT)r?r@rArr+rrrCrr}rr r>rEr4r4r2r5rsD      rcr)BlipTextPoolercs*tt|j|j|_t|_dSrW)rrrrOr"rZTanh activationr0r2r4r5rs zBlipTextPooler.__init__rhr9cCs(|dddf}||}||}|S)Nr)rr)r1rhZfirst_token_tensor pooled_outputr4r4r5r>s  zBlipTextPooler.forwardrr4r4r2r5rs rcr)BlipTextPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dSr) rrrrOr"rrrrr transform_act_fnr&r'r0r2r4r5rs  z(BlipTextPredictionHeadTransform.__init__rhr9cCs"||}||}||}|SrW)rrr&rr4r4r5r>s   z'BlipTextPredictionHeadTransform.forwardrr4r4r2r5rs  rcs,eZdZfddZddZddZZS)BlipTextLMPredictionHeadcsLtt||_tj|j|jdd|_t t |j|_ |j |j_ dS)NF)bias) rrr transformrrOr"r!decoder Parameterr+Zzerosrr0r2r4r5rs  z!BlipTextLMPredictionHead.__init__cCs|j|j_dSrW)rrr]r4r4r5 _tie_weightssz%BlipTextLMPredictionHead._tie_weightscCrrW)rrrr4r4r5r>rz BlipTextLMPredictionHead.forward)r?r@rArrr>rEr4r4r2r5rs  rcr)BlipTextOnlyMLMHeadcstt||_dSrW)rrr predictionsr0r2r4r5rs zBlipTextOnlyMLMHead.__init__sequence_outputr9cCs||}|SrW)r)r1rprediction_scoresr4r4r5r>s zBlipTextOnlyMLMHead.forwardrr4r4r2r5rs rc@s$eZdZdZeZdZgZddZdS)BlipTextPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. bertcCs~t|tjtjfr|jjjd|jjdnt|tj r(|j j |jj dt|tjr;|j dur=|j j dSdSdS)zInitialize the weightsg)meanZstd?N) rrrOr weightdataZnormal_r/Zinitializer_ranger&rZzero_Zfill_)r1moduler4r4r5 _init_weights-s  z%BlipTextPreTrainedModel._init_weightsN) r?r@rArBrZ config_classZbase_model_prefixZ_no_split_modulesrr4r4r4r5r#s  rc!seZdZdZd"fdd ZddZddZd d Zd ed e e d e de def ddZ              d#deejd eejdeejdeejdeejdeejdeejdeejdeeejdee dee dee dee dee dee ejeffd d!ZZS)$ BlipTextModela The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. argument and `is_decoder` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. TcsDt|||_t||_t||_|rt|nd|_| dSrW) rrr/rr=rencoderrpoolerZ post_init)r1r/add_pooling_layerr2r4r5rDs    zBlipTextModel.__init__cCs|jjSrWr=r#r]r4r4r5get_input_embeddingsNsz"BlipTextModel.get_input_embeddingscCs ||j_dSrWr)r1rRr4r4r5set_input_embeddingsQs z"BlipTextModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r1Zheads_to_prunerrr4r4r5 _prune_headsUszBlipTextModel._prune_headsrir;rrr9c CsX|dkr|dddddddf}n|dkr|r|\}}tj||d}|ddddf||d|ddddfk} | |j} | jd|jdkrl|jd| jd} tjtj||| f|| jd| gdd} | dddddddf|ddddddf}n|ddddddf}n t d ||j|j|jd }d |d }|S) a= Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`Tuple[int]`): The shape of the input to the model. device (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. r NrIrr)rrrr)ZaxiszAWrong shape for input_ids (shape {}) or attention_mask (shape {})rsrg) rpr+r,repeatrwrrshapertonesrLformat) r1rir;rrextended_attention_mask batch_sizer<Zseq_idsZ causal_maskZprefix_seq_lenr4r4r5get_extended_attention_mask]s8  . 6  z)BlipTextModel.get_extended_attention_maskNFr6rrjr7encoder_embedsrkrlrrrnrrc sv| dur| njj} | dur| njj} | dur| njj} |r+| dur&| njj} nd} |dur9|dur9td|durO|||}|\}}|j}n,|durc|dd}|\}}|j}n|durw|dd}|\}}|j}ntd| dur| ddj dnd}|durt |||f |} ||||}|durt|tr|d\}}}n|\}}}||f}t|trχfdd |D}n|durt j ||d }|}n|}nd}|jj}|durj||||d }n|}j|||||| | | | | d }|d}jdur|nd}| s-||f|d dSt|||j|j|j|jdS)a. encoder_hidden_states (`torch.FloatTensor`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timerzGYou have to specify either input_ids or inputs_embeds or encoder_embedsrrIcsg|]}|qSr4)invert_attention_mask)rmaskr]r4r5rrz)BlipTextModel.forward..r)r6rr7r8) rirjrkrlrrrnrrr)rZ pooler_outputrrhrr)r/rnruse_return_dictrrLZ%warn_if_padding_and_no_attention_maskr:rrr+rrwrrlistrZ get_head_maskrr=rrrrrhrr)r1r6rirrjr7rrkrlrrrnrrrr;rr<rr8rZencoder_batch_sizeZencoder_sequence_length_Zencoder_hidden_shapeZencoder_extended_attention_maskZembedding_outputZencoder_outputsrrr4r]r5r>s#      zBlipTextModel.forward)T)NNNNNNNNNNNNNF)r?r@rArBrrrrrrrDrr}rrr+rrCrrr>rEr4r4r2r5r;s|  B     rc%s"eZdZfddZddZddZddZd d Z d&dee j dee j dee j dee j dee j dee j dee j dee j dee e j dee dee dee dee dee dee dee deee j eff"d d!Zd'd"d#Zd$d%ZZS)(BlipTextLMHeadModelcs0t|t|dd|_t||_|j|_dS)NF)r)rrrrrclslabel_smoothingr0r2r4r5r+s   zBlipTextLMHeadModel.__init__cCs |jSrW)rrr]r4r4r5r2r[z(BlipTextLMHeadModel.get_input_embeddingscCs|j|dSrW)rrr1Znew_embeddingsr4r4r5r5sz(BlipTextLMHeadModel.set_input_embeddingscCs |jjjSrW)rrrr]r4r4r5get_output_embeddings8r[z)BlipTextLMHeadModel.get_output_embeddingscCs||jj_|j|jj_dSrW)rrrrrr4r4r5set_output_embeddings;s z)BlipTextLMHeadModel.set_output_embeddingsNFTrr6rirrjr7rkrllabelsrrrnrr return_logitsr reductionr9cCsT| dur| n|jj} |durd} |j|||||||| | | | | |d }|d}||}|r<|ddddddfSd}|dur|ddddddf}|ddddf|j}t||jd}|| d|jj | d}|dkr| | dd d}| s|f|d d}|dur|f|S|St |||j|j|j|jd S) a encoder_hidden_states (`torch.FloatTensor`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (`torch.LongTensor`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NF) rirrjr7rkrlrrrnrrrrrr)rrnonerI)ZlossZlogitsrrhrr)r/rrrrzrwrr rrdr!r:sumrrrhrr)r1r6rirrjr7rkrlrrrrnrrrrrr{rrZlm_lossZshifted_prediction_scoresZloss_fctrr4r4r5r>?sR)  zBlipTextLMHeadModel.forwardcKs|j}|dur ||}|dur4|ddjd}|jd|kr#|}n|jdd}|dd|df}||||dd|ddddS)NrrIrrkrlT)r6rirrkrlr)rZnew_onesget)r1r6rriZ model_kwargsr;Z past_lengthZremove_prefix_lengthr4r4r5prepare_inputs_for_generations    z1BlipTextLMHeadModel.prepare_inputs_for_generationcs.d}|D]}|tfdd|Df7}q|S)Nr4c3s$|] }|d|jVqdS)rN)Z index_selectrwr)rZ past_statebeam_idxr4r5rs"z5BlipTextLMHeadModel._reorder_cache..)r)r1rrZreordered_pastZ layer_pastr4rr5_reorder_caches z"BlipTextLMHeadModel._reorder_cache)NNNNNNNNNNNNNFTr)NN)r?r@rArrrrrrr+rrr}rrrrr>rrrEr4r4r2r5r*sv       Zr)rrr)2rxtypingrrrrr+Ztorch.utils.checkpointrrrZtorch.nnr Z activationsr Z generationr Zmodeling_outputsr rrZmodeling_utilsrrrrutilsrZconfiguration_bliprZ get_loggerr?rModulerrFrrrrrrrrrrrrr__all__r4r4r4r5s>      42@^ p