o ZhNQ@s(dZddlZddlmZddlmZddlmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZdd lmZmZdd lmZmZmZmZmZdd lmZmZdd lmZmZdd lm Z m!Z!m"Z"ddl#m$Z$m%Z%m&Z&e!'e(Z)dZ*eGdddeZ+eGdddeZ,Gddde j-Z.Gddde j-Z/Gddde j-Z0Gddde j-Z1Gddde j-Z2Gdd d e j-Z3Gd!d"d"e j-Z4Gd#d$d$e j-Z5Gd%d&d&e j-Z6Gd'd(d(e j-Z7d)e7iZ8Gd*d+d+e j-Z9Gd,d-d-e j-Z:Gd.d/d/e j-Z;Gd0d1d1e j-Ze Gd6d7d7eZ?Gd8d9d9e?Z@e d:d;Gdd;Gd?d@d@e?ZBGdAdBdBe j-ZCGdCdDdDe j-ZDGdEdFdFe j-ZEe dGd;GdHdIdIe?ZFe dJd;GdKdLdLe?ZGGdMdNdNe j-ZHe dOd;GdPdQdQe?ZIgdRZJdS)TzPyTorch BridgeTower ModelN) OrderedDict) dataclass)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FNQuickGELUActivation))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutput ModelOutputSequenceClassifierOutput)PreTrainedModelapply_chunking_to_forward) find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging torch_int)BridgeTowerConfigBridgeTowerTextConfigBridgeTowerVisionConfigZRobertaTokenizerc@steZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ ee ejed<dS)BridgeTowerModelOutputa Output type of [`BridgeTowerModel`]. Args: text_features (`torch.FloatTensor` of shape `(batch_size, text_sequence_length, hidden_size)`): Sequence of hidden-states at the text output of the last layer of the model. image_features (`torch.FloatTensor` of shape `(batch_size, image_sequence_length, hidden_size)`): Sequence of hidden-states at the image output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size x 2)`): Concatenation of last layer hidden-state of the first token of the text and image sequence (classification token), respectively, after further processing through layers used for auxiliary pretraining tasks. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N text_featuresimage_features pooler_output hidden_states attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rr r!rr"r*r*c/var/www/auris/lib/python3.10/site-packages/transformers/models/bridgetower/modeling_bridgetower.pyr.s rc@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZee ejed <dS) BridgeTowerContrastiveOutputaZ Output type of ['BridgeTowerForContrastiveLearning'] Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`: Image-text contrastive loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). text_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. image_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. cross_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The text-image cross-modal embeddings obtained by applying the projection layer to the pooler_output. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Nlosslogits text_embeds image_embeds cross_embedsr!r")r#r$r%r&r-rr'r(r)r.r/rr0r1r!r"r*r*r*r+r,Ns r,csNeZdZfddZdejdejfddZd dejdeejfdd ZZ S) BridgeTowerResidualAttentionc stt|j|jd|_tj|j|jd|_t t dt |j|jdfdt fdt |jd|jfg|_ tj|j|jd|_d|_dS)N@Zepsc_fcZgeluc_proj)super__init__rZMultiheadAttention hidden_sizeattn LayerNormlayer_norm_epsln_1Z ModuleDictrLinearr mlpln_2 attn_maskselfconfig __class__r*r+r9qs  z%BridgeTowerResidualAttention.__init__ hidden_stateattention_maskcCsZ|dur |jtj|jd}|jdur|jj|j|jdnd|_|j|||d|j|ddS)NdtypedeviceF)Z need_weightsrBZkey_padding_maskr)tor'boolrLrBrKr;)rDrHrIr*r*r+ attentions  z&BridgeTowerResidualAttention.attentionNcCsH|||||}||}|jD]\}}||}q||}|SN)rOr>rAr@items)rDrHrIZresidual_state_layerr*r*r+forwards   z$BridgeTowerResidualAttention.forwardrP) r#r$r%r9r'TensorrOrrT __classcell__r*r*rFr+r2ps $r2cs8eZdZfddZddejdeejfddZZS) BridgeTowerTransformercsrtj|_j|_jr#tfddt|jdD|_ntfddt|jD|_j |_ dS)Ncg|]}tqSr*r2.0rRrEr*r+ z3BridgeTowerTransformer.__init__..rcrXr*rYrZr\r*r+r]r^) r8r9r:num_hidden_layersZremove_last_layerr ModuleListrange resblocks stop_gradientrCrFr\r+r9s  zBridgeTowerTransformer.__init__NrHrIcCs>g}|jD]}|||}|jr||q||q|SrP)rbrcappenddetach)rDrHrIr!blockr*r*r+rTs   zBridgeTowerTransformer.forwardrP) r#r$r%r9r'rUrrTrVr*r*rFr+rWs $rWcsXeZdZdeffdd Zdejdededejfdd Zdd ej dejfd d Z Z S)BridgeTowerVisionEmbeddingsrEcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF)Z in_channelsZ out_channelsZ kernel_sizeZstridebiasr position_idsr persistent)r8r9rEr:Z embed_dim image_size patch_sizer Parameterr'Zrandnclass_embeddingConv2dZ num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferarangeexpandrCrFr*r+r9s"  "z$BridgeTowerVisionEmbeddings.__init__ embeddingsheightwidthreturnc Cs|jdd}|jjd}|jdd}tjs(||kr(||kr(||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNrl?r riZbicubicF)sizemodeZ align_cornersdim)shaperxweight unsqueezer'Zjit is_tracingrjrprZreshapepermuter functional interpolateviewcat) rDr|r}r~rurxrvZclass_pos_embedZpatch_pos_embedrZ new_heightZ new_widthZsqrt_num_positionsr*r*r+interpolate_pos_encodings*      z4BridgeTowerVisionEmbeddings.interpolate_pos_encodingF pixel_valuesc Cs|j\}}}}|s&||jks||jkr&td|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r[| | | ||} | S| ||j} | S) NzInput image size (*z) doesn't match model (z).rKrirrlr)rro ValueErrorrtrrKrMflatten transposerrr{r'rrrxrj) rDrr batch_sizerRr}r~Z target_dtypeZ patch_embedsZ class_embedsr|r*r*r+rTs  z#BridgeTowerVisionEmbeddings.forwardF) r#r$r%rr9r'rUintrr(rTrVr*r*rFr+rgs )rgcs\eZdZfddZ d dejdefddZ d dejdefdd Zd ejfd d Z Z S)BridgeTowerVisionTransformercs|tt|_tjjjd|_t |_ tjjjd|_ j |_ j s.)r8r9rgr|rr<r:r=ln_prerW transformerln_postshare_layernormr`rar_ ln_separaterCrFr\r+r9 s    z%BridgeTowerVisionTransformer.__init__FrrcCs|||}||}|ddd}|||}tj|dd}|dddd}|jr1||}|Sg}t||j D] \}}||}| |q9tj|dd}|S)Nrrrirr ) r|rrrr'stackrrziprrd)rDrrIrr!Zhidden_states_stacklnr*r*r+rTs     z$BridgeTowerVisionTransformer.forwardcCs*|j||d}||}|ddd}|S)Nrrrri)r|rr)rDrrr!r*r*r+ forward_pre3s z(BridgeTowerVisionTransformer.forward_prerHcCs|ddd}||}|S)Nrrri)rr)rDrHZvisual_output_postr*r*r+ forward_post>s z)BridgeTowerVisionTransformer.forward_postr) r#r$r%r9r'rUrNrTrrrVr*r*rFr+r s    rc$eZdZfddZddZZS)BridgeTowerLinkTowercst|j|_|j|_|jdvr;|jdkr!ttd|_n|jdkr/ttd|_ tj |j|j d|_ dSt d|jd) N)add scaled_addrr?rrr4link_tower_type  is not implemented) r8r9link_tower_typer:rrqr'tensor scaled_factorbetar<r=NotImplementedErrorrCrFr*r+r9Es    zBridgeTowerLinkTower.__init__cCsp|jdkr |||S|jdkr|||j|S|jdkr/||d|j||jStd|jd)Nrrrrrr)rr<rrr)rDr!Zcross_modal_hidden_statesrIr*r*r+rTRs   zBridgeTowerLinkTower.forwardr#r$r%r9rTrVr*r*rFr+rDs  rc8eZdZfddZdejdejdejfddZZS)BridgeTowerSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr4) r8r9rr?r:denser<r=Dropouthidden_dropout_probdropoutrCrFr*r+r9_ zBridgeTowerSelfOutput.__init__r! input_tensorrcC&||}||}|||}|SrPrrr<rDr!rr*r*r+rTe  zBridgeTowerSelfOutput.forwardr#r$r%r9r'rUrTrVr*r*rFr+r^ $rc2eZdZfddZdejdejfddZZS)BridgeTowerIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSrP) r8r9rr?r:intermediate_sizer isinstance hidden_actstrr intermediate_act_fnrCrFr*r+r9ns   z BridgeTowerIntermediate.__init__r!rcCs||}||}|SrP)rrrDr!r*r*r+rTvs  zBridgeTowerIntermediate.forwardrr*r*rFr+rms rcr)BridgeTowerOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r8r9rr?rr:rr<r=rrrrCrFr*r+r9~rzBridgeTowerOutput.__init__r!rrcCrrPrrr*r*r+rTrzBridgeTowerOutput.forwardrr*r*rFr+r}rrcr)BridgeTowerPoolercs*tt|j|j|_t|_dSrP)r8r9rr?r:rZTanh activationrCrFr*r+r9s zBridgeTowerPooler.__init__r!rcCs(|dddf}||}||}|S)Nr)rr)rDr!Zfirst_token_tensor pooled_outputr*r*r+rTs  zBridgeTowerPooler.forwardrr*r*rFr+rs rcseZdZdfdd ZdejdejfddZ      dd ejd eejd eejd eejd eejdee e ejdee de ejfddZ Z S)BridgeTowerSelfAttentionNcst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|p\t|dd|_|jdksh|jd kry|j|_t d |jd |j|_|j|_dS) NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()position_embedding_typeabsolute relative_keyrelative_key_queryrir)r8r9r:num_attention_headshasattrrrattention_head_size all_head_sizerr?querykeyvaluerZattention_probs_dropout_probrgetattrrmax_position_embeddingsrwdistance_embedding is_decoderrDrErrFr*r+r9s*   z!BridgeTowerSelfAttention.__init__xrcCs6|dd|j|jf}||}|ddddS)Nrlrrirr )rrrrr)rDrZ new_x_shaper*r*r+transpose_for_scoress z-BridgeTowerSelfAttention.transpose_for_scoresFr!rI head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionscCs||}|du} | r|dur|d} |d} |}nP| r/|||} |||} |}n;|durZ|||} |||} tj|d| gdd} tj|d| gdd} n|||} |||} ||} |du} |jrz| | f}t| | dd}|j dks|j dkr | j d| j d}}| rtj |dtj |j d dd}ntj|tj |j d dd}tj|tj |j d dd}||}|||jd}|j| jd }|j dkrtd | |}||}n|j dkr td | |}td | |}|||}|t|j}|dur||}tjj|dd}||}|dur0||}t|| }|dddd }|dd|jf}||}|rX||fn|f}|jrd||f}|S)NrrrirrlrrrJrzbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rrrrr'rrmatmulrrrrlongrLrrzrrrMrKZeinsummathsqrtrrrZsoftmaxrr contiguousrr)rDr!rIrrrrrZmixed_query_layerZis_cross_attentionZ key_layerZ value_layerZ query_layer use_cacheZattention_scoresZ query_lengthZ key_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_probsZ context_layerZnew_context_layer_shapeoutputsr*r*r+rTsn             z BridgeTowerSelfAttention.forwardrPNNNNNF) r#r$r%r9r'rUrrr(rrNrTrVr*r*rFr+rs4 reagercseZdZdfdd ZddZ      ddejdeejd eejd eejd eejd ee e ejd ee de ejfddZ Z S)BridgeTowerAttentionNcs4tt|j||d|_t||_t|_dS)Nr) r8r9#BRIDGE_TOWER_SELF_ATTENTION_CLASSESZ_attn_implementationrDroutputset pruned_headsrrFr*r+r9)s   zBridgeTowerAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)lenrrDrrrrrrrrrrunion)rDheadsindexr*r*r+ prune_heads1s z BridgeTowerAttention.prune_headsFr!rIrrrrrrc Cs<||||||||}||d|} | f|dd} | S)Nrr)rDr) rDr!rIrrrrrZ self_outputsattention_outputrr*r*r+rTCs  zBridgeTowerAttention.forwardrPr) r#r$r%r9rr'rUrr(rrNrTrVr*r*rFr+r(s4 rcs8eZdZfddZ     d ddZddZZS) BridgeTowerBertCrossLayercsTt|j|_d|_t||_|j|_|j|_t||_t ||_ t ||_ dS)Nr) r8r9chunk_size_feed_forward seq_len_dimrrOradd_cross_attentioncrossattentionr intermediaterrrCrFr*r+r9\s    z"BridgeTowerBertCrossLayer.__init__NFc Csz|j||d|dd}|d} |dd} |j| ||||||d} | d} | | dd} t|j|j|j| } | f| } | S)N)rIrrrrr)rIrrrrrrl)rOrrfeed_forward_chunkrr) rDr!rrIrrrrself_attention_outputsrrcross_attention_outputs layer_outputr*r*r+rTgs2    z!BridgeTowerBertCrossLayer.forwardcC||}|||}|SrPrrrDrZintermediate_outputrr*r*r+r  z,BridgeTowerBertCrossLayer.feed_forward_chunk)NNNNF)r#r$r%r9rTrrVr*r*rFr+r[s  ,rcseZdZfddZ      ddejdeejdeejdeejd eejd eeeejd ee d eejfd dZ ddZ Z S)BridgeTowerTextLayercsrt|j|_d|_t||_|j|_|j|_|jr-|js&t|dt|dd|_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is addedrr)r8r9rrrrOrrrrrrrrrCrFr*r+r9s   zBridgeTowerTextLayer.__init__NFr!rIrrrrrrc Cs|dur |ddnd}|j|||||d} | d} |jr(| dd} | d} n| dd} d} |jro|durot|dsDtd|d|durN|d dnd}|| ||||||}|d} | |dd} |d} | | } t|j|j|j| }|f| } |jr| | f} | S) Nri)rrrrrlrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rOrrrrrrrr)rDr!rIrrrrrZself_attn_past_key_valuerrrZpresent_key_valueZcross_attn_present_key_valueZcross_attn_past_key_valuerrr*r*r+rTsP         zBridgeTowerTextLayer.forwardcCrrPrrr*r*r+rrz'BridgeTowerTextLayer.feed_forward_chunkr) r#r$r%r9r'rUrr(rrNrTrrVr*r*rFr+r s4  Ar cseZdZfddZ         ddejdeejdeejd eejd eejd eeeejd ee d ee dee dee de eeje ffddZ Z S)BridgeTowerTextEncodercs:t|_tfddtjD|_d|_dS)NcrXr*)r rZr\r*r+r]r^z3BridgeTowerTextEncoder.__init__..F) r8r9rErr`rar_rSgradient_checkpointingrCrFr\r+r9s   zBridgeTowerTextEncoder.__init__NFTr!rIrrrpast_key_valuesrroutput_hidden_states return_dictrc  Cs^| rdnd} |r dnd} |r|jjrdnd} |jr%|jr%|r%tdd}|r)dnd}t|jD]^\}}| r;| |f} |durC||nd}|durM||nd}|jrc|jrc||j |||||||}n ||||||||}|d}|rz||df7}|r| |df} |jjr| |df} q0| r| |f} | st dd ||| | | fDSt ||| | | d S) Nr*zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frrlrrics|] }|dur|VqdSrPr*r[vr*r*r+ 9sz1BridgeTowerTextEncoder.forward..)last_hidden_stater r!r"cross_attentions) rErr ZtrainingloggerZ warning_once enumeraterSZ_gradient_checkpointing_func__call__tupler )rDr!rIrrrr rrr rall_hidden_statesall_self_attentionsZall_cross_attentionsZnext_decoder_cacheiZ layer_moduleZlayer_head_maskrZ layer_outputsr*r*r+rTsz         zBridgeTowerTextEncoder.forward) NNNNNNFFT)r#r$r%r9r'rUrr(rrNrr rTrVr*r*rFr+r sD      r cs4eZdZdZfddZ d ddZdd ZZS) BridgeTowerTextEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dd|j|_tj|j|j|jd|_ dS) N) padding_idxr4rrrjrkFrmtoken_type_idsr)r8r9rrw vocab_sizer:Z pad_token_idword_embeddingsrposition_embeddingsZtype_vocab_sizetoken_type_embeddingsr<r=rrrrrryr'rzr{zerosrjrrrrCrFr*r+r9Ts"   z"BridgeTowerTextEmbeddings.__init__Nrc Cs|dur|durt||j|}n||}|dur|}n|dd}|d}|durTt|drI|jddd|f}||d|} | }n tj|tj |j j d}|dur]| |}| |} || } |jdkrt||} | | 7} || } || } | S)NrlrrrrJr)"create_position_ids_from_input_idsr&create_position_ids_from_inputs_embedsrrrr{r'r#rrjrLr r"rr!r<r) rD input_idsrrj inputs_embedspast_key_values_length input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr"r|r!r*r*r+rTms0         z!BridgeTowerTextEmbeddings.forwardcCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor NrlrrJr)rr'rzrrrLrr{)rDr'r)Zsequence_lengthrjr*r*r+r%s  z@BridgeTowerTextEmbeddings.create_position_ids_from_inputs_embeds)NNNNr)r#r$r%r&r9rTr%rVr*r*rFr+rNs   (rcCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)nerr'ZcumsumZtype_asr)r&rr(maskZincremental_indicesr*r*r+r$s  r$c@s,eZdZeZdZdZddgZdZddZ dS) BridgeTowerPreTrainedModel bridgetowerFrr2r cCst|tr|jjjdd|jjjd}|jjjd}d|jjjd}|jjjD]=}tjj |j j ||j j dtjj |j jj||j j dtjj |jjj||j j dtjj |jjj||j j dq*tjj |jjj||j j dtjj |jjjj||j j dn,t|tjtjtjfr|jjj dd|j j dnt|tjr|jj|jjdt|tjr|jdur|jjdSdSdS)Ngri)stdgg?)meanr1r)rBridgeTowerVisionModelvisualrr:r_rbrinitZnormal_r;Zin_proj_weightrEZinitializer_factorZout_projrr@r5r7r|rrrxr?rsrwdatar<rhZzero_Zfill_)rDmoduleZproj_stdZattn_stdZfc_stdrfr*r*r+ _init_weightss.     z(BridgeTowerPreTrainedModel._init_weightsN) r#r$r%r config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementr8r*r*r*r+r/s r/cs6eZdZeZfddZeddZd ddZZ S) r3cst|t||_dSrP)r8r9rr4rCrFr*r+r9s zBridgeTowerVisionModel.__init__cCs|jjjjjSrP)r4r|rtrrKrDr*r*r+rKszBridgeTowerVisionModel.dtypeNFcCs|||j||SrP)r4typerK)rDimageZ image_maskrr*r*r+rTszBridgeTowerVisionModel.forward)NF) r#r$r%rr9r9propertyrKrTrVr*r*rFr+r3s   r3a( The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in *Attention is all you need*_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. .. _*Attention is all you need*: https://arxiv.org/abs/1706.03762 )Z custom_introc seZdZeZdfdd ZddZddZdd Ze dd e e j d e e j d e e j de e j de e j de e j de e j de e j de e e jde ede ede ede edeee j effddZZS)BridgeTowerTextModelTcsDt|||_t||_t||_|rt|nd|_| dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r8r9rErr|r encoderrpooler post_init)rDrEZadd_pooling_layerrFr*r+r9s    zBridgeTowerTextModel.__init__cC|jjSrPr|r r:r*r*r+get_input_embeddings z)BridgeTowerTextModel.get_input_embeddingscC ||j_dSrPrCrDrr*r*r+set_input_embeddings z)BridgeTowerTextModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)rQr?rSrOr)rDZheads_to_prunerSrr*r*r+ _prune_headssz!BridgeTowerTextModel._prune_headsNr&rIrrjrr'rrr rrr rrc CsP| dur| n|jj} | dur| n|jj} | dur| n|jj} |jjr-| dur(| n|jj} nd} |dur;|dur;td|durJ||||}n|durW|dd}ntd|\}}|durf|j n|j }| durv| ddj dnd}|durt j |||f|d}|durt |jdr|jjddd|f}|||}|}n t j|t j|d }|||}|jjr|dur|\}}}||f}|durt j ||d}||}nd}|||jj}|j|||||d }|j|||||| | | | | d }|d}|jdur ||nd}| s||f|d dSt|||j|j|j|jd S)NFzDYou cannot specify both input_ids and inputs_embeds at the same timerlz5You have to specify either input_ids or inputs_embedsrrirLrrJ)r&rjrr'r() rIrrrr rrr rr)rr r r!r"r)rErr use_return_dictrrrZ%warn_if_padding_and_no_attention_maskrrLrr'onesrr|rr{r#rget_extended_attention_maskZinvert_attention_maskZ get_head_maskr_r?r@rr r!r"r)rDr&rIrrjrr'rrr rrr rr)rr*rLr(r+r,Zextended_attention_maskZencoder_batch_sizeZencoder_sequence_lengthrRZencoder_hidden_shapeZencoder_extended_attention_maskZembedding_outputZencoder_outputsZsequence_outputrr*r*r+rTs       zBridgeTowerTextModel.forward)T) NNNNNNNNNNNNN)r#r$r%rr9r9rDrHrJrrr'rUrr(rNrrrrTrVr*r*rFr+r>sb     r>zv The bare BridgeTower Model transformer outputting BridgeTowerModelOutput object without any specific head on c"seZdZfddZddZddZe              dd eej d eej d eej d eej d eej deej deej deej dee dee dee dee deej de de eejeffddZddZZS)BridgeTowerModelcst|_jjjr%tjj|_ tjj|_ n"t fddt j D|_ t fddt j D|_ tdj|_t|_t|_jsyjry|jjjD]}|jjjjj|j_|jjjjj|j_qdt fddt j D|_t fddt j D|_t|_t|_ tj!jj"d|_#tj!jj"d|_$j%rt&|_'t&|_(n$t fddt j d D|_'t fd dt j d D|_(|)dS) Ncg|] }tjjqSr*rr?r:rZ)rE text_configr*r+r]z-BridgeTowerModel.__init__..crPr*rQrZ)rE vision_configr*r+r]rSricrXr*rrZrRr*r+r]r^crXr*rUrZrVr*r+r]r^r4crXr*rrZr\r*r+r]r^rcrXr*rWrZr\r*r+r]r^)*r8r9rErTrRZ$share_cross_modal_transformer_layersrr?r:cross_modal_text_transformcross_modal_image_transformr`rar_rwr"r3 vision_modelr> text_modelrZ"init_layernorm_from_vision_encoderr4Zcross_modal_ln_separaterrr6rhcross_modal_image_layerscross_modal_text_layersrcross_modal_image_poolercross_modal_text_poolerr<r=cross_modal_text_layernormcross_modal_image_layernormZshare_link_tower_layersrcross_modal_text_link_towercross_modal_image_link_towerrA)rDrErrF)rErRrTr+r9sP         zBridgeTowerModel.__init__cCs |jSrP)r[rDr:r*r*r+rDs z%BridgeTowerModel.get_input_embeddingscCs|j|dSrP)r[rHrGr*r*r+rHsz%BridgeTowerModel.set_input_embeddingsNFr&rIrr pixel_maskrr'r0image_token_type_idxrr rlabelsrrc-Cs| dur| n|jj} | dur| n|jj} | rdnd}| rdnd}| r$dnd}| r*dnd}| r0dnd}|dur>|dur>td| durD| n|jj} | rL| nd} |}|jj|d}| r`||f7}|durntj |tj |j d}|j || |j }t|jjj|jjd}|jjjd|D]}|||d}| r||f7}q|dur|jjj||jj|d}n|ddd }| r||f7}|jjjjd|D] }||}| r||f7}q|jj||jj}||}|tjdtj |j d|}|||}| |}|tj!d | tj |j d|}||}|"|}tj |d|dftj |j d}|j || |j }|j#d||||| d } | d}!|j$d||||| d }"|"d}#| ri||!|#ff7}| rw|| d|"dff7}d}$t%|t|jjjD]}%|jjj|%||d}|jjjj|%||jj}| |jj||}|j&|$}&|j'|$}'|&||||!|}(|'||#|})|j#|$d|(|)||| d } | d}!|j$|$d|)|(||| d }"|"d}#|$d7}$| r||f7}||f7}||!|#ff7}| r|| d|"dff7}q|!|#}*}+|(|*|+},| r&|||f}| s7t)d d |*|+|,||fDSt*|*|+|,||dS)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. output_hidden_states (`bool`, *optional*): If set to `True`, hidden states are returned as a list containing the hidden states of text, image, and cross-modal components respectively. i.e. `(hidden_states_text, hidden_states_image, hidden_states_cross_modal)` where each element is a list of the hidden states of the corresponding modality. `hidden_states_txt/img` are a list of tensors corresponding to unimodal hidden states and `hidden_states_cross_modal` is a list of tuples containing `cross_modal_text_hidden_states` and `cross_modal_image_hidden_states` of each brdige layer. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base") >>> model = BridgeTowerModel.from_pretrained("BridgeTower/bridgetower-base") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> outputs.keys() odict_keys(['text_features', 'image_features', 'pooler_output']) ```Nr*zYBridgeTowerModel does not use `inputs_embeds`. Make sure to pass in `input_ids` instead.r)r&rJrrrir)rIrrcsrrPr*rr*r*r+rsz+BridgeTowerModel.forward..)rrr r!r")+rErr rrLrr[r|r'rMrrLrNrMrr?rSr_rZr4rr;rKrrrbrrXr"r# expand_asr`rYfullrar]r\rarbrcget_cls_featuresrr)-rDr&rIrrrdrr'r0rerr rrfrZall_hidden_states_textZall_hidden_states_imageZall_hidden_states_crossrrr)r/Zextend_text_masksZ split_indexrSrfimage_embeds_with_lnZcross_modal_textZtext_token_type_embeddingsimage_token_type_embeddingsZcross_modal_imageZextend_image_masksZlayer_outputs_textZcross_text_featuresZlayer_outputs_imageZcross_image_featuresZlink_layer_indexrZtext_link_towerZimage_link_towerZcross_text_features_Zcross_image_features_rrZ cls_featuresr*r*r+rTs5                         zBridgeTowerModel.forwardcCs&||}||}tj||gddS)Nrlr)r_r^r'r)rDrrZcls_features_textZcls_features_imager*r*r+rjs  z!BridgeTowerModel.get_cls_features)NNNNNNNNNNNNNF)r#r$r%r9rDrHrrr' LongTensorr(rrNrrrUrrTrjrVr*r*rFr+rOsh 8      mrOcr)"BridgeTowerPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dSr) r8r9rr?r:rrrrr transform_act_fnr<r=rCrFr*r+r9s  z+BridgeTowerPredictionHeadTransform.__init__cCs"||}||}||}|SrP)rror<rr*r*r+rTs   z*BridgeTowerPredictionHeadTransform.forwardrr*r*rFr+rns  rncs&eZdZdfdd ZddZZS)BridgeTowerMLMHeadNcs`t||_t||_tj|j|jj dd|_ t t |jj |_|dur.||j _dSdS)NF)rh)r8r9rErn transformrr?r:rRrdecoderrqr'r#rhr)rDrErrFr*r+r9s   zBridgeTowerMLMHead.__init__cCs||}|||j}|SrP)rqrrrh)rDr mlm_scorer*r*r+rTs zBridgeTowerMLMHead.forwardrPrr*r*rFr+rps rpcr)BridgeTowerITMHeadcstt|d|_dSNrir8r9rr?fc)rDr:rFr*r+r9 zBridgeTowerITMHead.__init__cCs||}|SrPrw)rDr itm_scorer*r*r+rT zBridgeTowerITMHead.forwardrr*r*rFr+rt rtz\ BridgeTower Model with a language modeling head on top as done during pretraining. cseZdZdgZfddZddZddZe            dd ee j d ee j d ee j d ee j d ee j dee j dee j dee j dee dee dee dee j de eee j ffddZZS)BridgeTowerForMaskedLMzmlm_score.decoder.weightcs,t|t||_t||_|dSrP)r8r9rOr0rprsrArCrFr*r+r9s    zBridgeTowerForMaskedLM.__init__cCrBrPrsrrr:r*r*r+get_output_embeddingsrEz,BridgeTowerForMaskedLM.get_output_embeddingscCrFrPr~)rDZnew_embeddingsr*r*r+set_output_embeddingsrIz,BridgeTowerForMaskedLM.set_output_embeddingsNr&rIrrrdrr'r0rr rrfrc  Cs| dur| n|jj} |j||||||||| | | d } || r!| jn| d}d}| durDt}| |j} ||d|jj j | d}| sUt |}|durS|f|S|St ||| j | jdS)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000360943.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> text = "a looking out of the window" >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> model = BridgeTowerForMaskedLM.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist()) >>> print(results) .a cat looking out of the window. ```N rIrrrdrr'r0rr rrrlr-r.r!r")rErLr0rsrr rMrLrrRrrrr!r")rDr&rIrrrdrr'r0rr rrfrZ mlm_logitsZmasked_lm_lossloss_fctrr*r*r+rTs:2 zBridgeTowerForMaskedLM.forward NNNNNNNNNNNN)r#r$r%Z_tied_weights_keysr9rrrrr'rmr(rNrrrrTrVr*r*rFr+r}sZ       r}z BridgeTower Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text matching. cseZdZfddZe            ddeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee deejde e e ejffddZ ZS)#BridgeTowerForImageAndTextRetrievalcs2t|t||_t|jd|_|dSru)r8r9rOr0rtr:rzrArCrFr*r+r9`s   z,BridgeTowerForImageAndTextRetrieval.__init__Nr&rIrrrdrr'r0rr rrfrc  Cs| dur| n|jj} |j||||||||| | | d } | r| jn| d}||}d}| dur>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, 1].item() ```Nrrir) rErLr0r rzr rMrLrrr!r")rDr&rIrrrdrr'r0rr rrfrr r.Zitm_lossrrr*r*r+rTjs<.   z+BridgeTowerForImageAndTextRetrieval.forwardr)r#r$r%r9rrr'rmr(rNrrrrTrVr*r*rFr+rYsT       rcr)BridgeTowerContrastiveHeadcstt|||_dSrPrv)rDr:Z embed_sizerFr*r+r9rxz#BridgeTowerContrastiveHead.__init__cCs||}|SrPry)rDrr*r*r+rTr{z"BridgeTowerContrastiveHead.forwardrr*r*rFr+rr|rzl BridgeTower Model with a image-text contrastive head on top computing image-text contrastive loss. cseZdZfddZe            ddeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee dee de e e ejffddZ ZS)!BridgeTowerForContrastiveLearningcslt|t||_t|j|j|_t|j|j|_t|jd|j|_ t t |jj|_|dSru)r8r9rOr0rr:Zcontrastive_hidden_size itc_text_headitc_image_headitc_cross_modal_headrrqr'rrEZlogit_scale_init_value logit_scalerArCrFr*r+r9s   z*BridgeTowerForContrastiveLearning.__init__NTr&rIrrrdrr'r0rr r return_lossrc ! CsB| dur| n|jj} |j||||||||| d| d } | r| jn| d}| r(| jn| d\}}}|d}|d}|jjj|}|jt j ddt j |jjj j d |}|j||}tjj||ddd ddfddd }tjj||ddd ddfddd j|j d }tjj||ddd j|j d }t j|||gd d}|jj|j d }t |||}t |||}t |||}d}| rt jt||j d }tj||}tj||}tj||}|||d}| s||||f| dd} |dur|f| S| St|||||| j| j dS)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning >>> import requests >>> from PIL import Image >>> import torch >>> image_urls = [ ... "https://farm4.staticflickr.com/3395/3428278415_81c3e27f15_z.jpg", ... "http://images.cocodataset.org/val2017/000000039769.jpg", ... ] >>> texts = ["two dogs in a car", "two cats sleeping on a couch"] >>> images = [Image.open(requests.get(url, stream=True).raw) for url in image_urls] >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") >>> model = BridgeTowerForContrastiveLearning.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") >>> inputs = processor(images, texts, padding=True, return_tensors="pt") >>> loss = model(**inputs, return_loss=True).loss >>> inputs = processor(images, texts[::-1], padding=True, return_tensors="pt") >>> loss_swapped = model(**inputs, return_loss=True).loss >>> print("Loss", round(loss.item(), 4)) Loss 0.0019 >>> print("Loss with swapped images", round(loss_swapped.item(), 4)) Loss with swapped images 2.126 ```NTrrir rlrgrrJr)rprKrrg@)r-r.r/r0r1r!r")!rErLr0r r!rZr4rr"r'rirrrLrhrYrr normalizerrrMrrrexprtrzrZ cross_entropyr,r")!rDr&rIrrrdrr'r0rr rrrr Zhidden_states_txtZhidden_states_imgZhidden_states_cross_modalr/rkrlr1r.rZlogits_text_to_imageZlogits_text_to_crossZlogits_image_to_crossZitc_lossrfZtext_to_image_lossZtext_to_cross_lossZimage_to_cross_lossrr*r*r+rTsr5**z)BridgeTowerForContrastiveLearning.forward) NNNNNNNNNTNN)r#r$r%r9rrr'rmr(rNrr,rrTrVr*r*rFr+rsT       r)rrr}rOr/)r)Kr&r collectionsr dataclassesrtypingrrrrr'Ztorch.utils.checkpointrZtorch.nnr Z activationsr r Zmodeling_outputsr rrrrZmodeling_utilsrrZ pytorch_utilsrrutilsrrrZconfiguration_bridgetowerrrrZ get_loggerr#rZ_TOKENIZER_FOR_DOCrr,Moduler2rWrgrrrrrrrrrrr r rr$r/r3r>rOrnrprtr}rrr__all__r*r*r*r+s     !,S:3>W^ Z"5 g`