o ZhL>@sdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddlmZdd lmZdd lmZmZmZmZdd lmZmZdd lmZdd lmZmZm Z ddl!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.m/Z/e&0e1Z2eGddde#Z3Gddde j4Z5 dIde j4de j6de j6de j6dee j6de7de7fdd Z8Gd!d"d"e j4Z9Gd#d$d$e j4Z:Gd%d&d&e j4Z;e$Gd'd(d(eZGd-d.d.e j4Z?Gd/d0d0e j4Z@Gd1d2d2e j4ZAGd3d4d4e j4ZBGd5d6d6e j4ZCGd7d8d8e j4ZDGd9d:d:e j4ZEGd;d<dd>e<ZGGd?d@d@ee"ZHe$dAdBGdCdDdDe<ZIe$dEdBGdFdGdGeGs  zKInstructBlipForConditionalGenerationModelOutput.to_tuple..)tuplekeysr1r3r1r4r.Fs z8InstructBlipForConditionalGenerationModelOutput.to_tuple)__name__ __module__ __qualname____doc__r&rrtorch FloatTensor__annotations__r'r(r)r*rr.r3r3r3r4r%-s r%cs\eZdZdeffdd Zdejdededejfdd Zdd ej d e dejfd dZ Z S)InstructBlipVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt dd|j|_ tj d|j|j|jd|_ |j|jd|_|jd|_tt d|j|j|_dS)Nr!r )Z in_channelsZ out_channelsZ kernel_sizeZstrider)super__init__r@ hidden_size embed_dimZ image_size patch_sizer Parameterr<Zrandnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingr2r@ __class__r3r4rBQs  z%InstructBlipVisionEmbeddings.__init__ embeddingsheightwidthr+c Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||j} ||j} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r!Ng?rr rZbicubicF)sizemodeZ align_cornersdim)shaperLr<Zjit is_tracingrErreshapepermuter functionalZ interpolateviewcat) r2rPrQrRrJrKZclass_pos_embedZpatch_pos_embedrWZ new_heightZ new_widthZsqrt_num_positionsr3r3r4interpolate_pos_encodingcs(     z5InstructBlipVisionEmbeddings.interpolate_pos_encodingF pixel_valuesr_c Cs|j\}}}}|jjj}||j|d}|ddd}|j|dd|} t j | |gdd} |r<| | ||} n|j } | | ddd| dddf|} | S)Ndtyperr!rSrV)rXrIweightrbtoflatten transposerGexpandr<r^r_rLrT) r2r`r_ batch_size_rQrRZ target_dtypeZ patch_embedsZ class_embedsrPrLr3r3r4forwards *z$InstructBlipVisionEmbeddings.forwardF) r8r9r:r$rBr<Tensorintr_r=boolrj __classcell__r3r3rNr4r?Ps$(r?modulequerykeyvalueattention_maskscalingdropoutc Kspt||dd|}|dur||}tjj|dd}tjj|||jd}t||} | dd} | |fS)NrSrV)ptrainingr!r) r<matmulrfrr\Zsoftmaxrwrz contiguous) rqrrrsrtrurvrwkwargs attn_weights attn_outputr3r3r4eager_attention_forwards  rcseZdZdZfddZdejdedefddZ dd ejd e ejd e e de eje eje e ejffddZ Z S)InstructBlipAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ d|_ |j |_ t j |jd|jdd|_|jr]t t|j}t t|j}nd}d}|duryt|tj|dd|f}t ||j_t |j|j|_dS) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFr )bias)Z requires_grad)rArBr@rCrDnum_attention_heads num_headshead_dim ValueErrorscaleZ is_causalattention_dropoutrLinearqkvqkv_biasrFr<zerosr^Z zeros_liker projection)r2r@Zq_biasZv_biasrrNr3r4rBs0   zInstructBlipAttention.__init__tensorseq_lenbszcCs ||||j|jddS)Nr!r)r]rrrfr|)r2rrrr3r3r4_shapes zInstructBlipAttention._shapeNF hidden_states head_maskoutput_attentionsr+cKs|\}}}||}|||d|j||jddddd}|d|d|d} } } t} |jjdkrJ|jjdkrD|rDt dnt |jj} | || | | fd |j sVd n|j |j d |\} }| ||d } || } |rx| |f}|S| d f}|S) z#Input shape: Batch x Time x Channelr rrr!eagerZsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.Nrp)rurwrvrS)rTrrZrr[rr@Z_attn_implementationlogger warning_oncerrzrrr|r)r2rrrr}rZtgt_lenrDZ mixed_qkvZ query_statesZ key_statesZ value_statesZattention_interfacerr~outputsr3r3r4rjs>       zInstructBlipAttention.forward)NF)r8r9r:r;rBr<rlrmrrrnrrjror3r3rNr4rs rc2eZdZfddZdejdejfddZZS)InstructBlipMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) rArBr@r hidden_act activation_fnrrrCintermediate_sizefc1fc2rMrNr3r4rB s  zInstructBlipMLP.__init__rr+cCs"||}||}||}|Sr)rrrr2rr3r3r4rjs   zInstructBlipMLP.forwardr8r9r:rBr<rlrjror3r3rNr4rs rc sNeZdZdeffdd Z d dejdejdeede ej fd d Z Z S) InstructBlipEncoderLayerr@csRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSNZeps) rArBrCrDr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rMrNr3r4rBs   z!InstructBlipEncoderLayer.__init__Frrurr+cCsb|}||}|j|||d\}}||}|}||}||}||}|f}|r/||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrr)rrrr)r2rrurZresidualr~rr3r3r4rj!s      z InstructBlipEncoderLayer.forwardrk) r8r9r:r"rBr<rlrrnrr=rjror3r3rNr4rs rc@sDeZdZeZdZdZdZdZdZ dZ dZ dZ dZ gdZddZdS)InstructBlipPreTrainedModelZblipTF)InstructBlipQFormerEmbeddingsr%InstructBlipQFormerMultiHeadAttentionInstructBlipQFormerSelfOutputcCs|jj}t|tjtjfr%|jjjd|d|j dur#|j j dSdSt|tj r6|jjjd|ddSt|tj rK|j j |jj ddSt|trftjj|jd|dtjj|jd|ddSt|ttfru|jj dSdS)zInitialize the weightsrp)meanZstdN?)r@Zinitializer_range isinstancerrrHrcdataZnormal_rZzero_ EmbeddingrZfill_r?initZ trunc_normal_rLrG$InstructBlipForConditionalGenerationInstructBlipModel query_tokens)r2rqfactorr3r3r4 _init_weights\s"     z)InstructBlipPreTrainedModel._init_weightsN)r8r9r:r" config_classZbase_model_prefixZsupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_2_supports_sdpa_supports_flex_attn_supports_cache_class_supports_static_cache_supports_quantized_cache_no_split_modulesrr3r3r3r4rHs rc sheZdZdZdeffdd Z    d deejdee dee d ee d e e e ff d d Z ZS)InstructBlipEncodera Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipEncoderLayer`]. Args: config (`InstructBlipConfig`): The corresponding vision configuration for the `InstructBlipEncoder`. r@c:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr3)r)r/rir@r3r4 ~sz0InstructBlipEncoder.__init__..F) rArBr@r ModuleListrangenum_hidden_layerslayersgradient_checkpointingrMrNrr4rB{s   zInstructBlipEncoder.__init__Nruroutput_hidden_states return_dictr+c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|}t|jD]/\} } |r<||f}|jrL|jrL|| j |||} n| |||d} | d}|r`|| df}q1|rh||f}|svt dd|||fDSt |||dS) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr3rrr!cs|] }|dur|VqdSrr3r/vr3r3r4r5sz.InstructBlipEncoder.forward..)last_hidden_stater attentions) r@rruse_return_dict enumeraterrrz_gradient_checkpointing_func__call__r6r ) r2 inputs_embedsrurrrZencoder_statesZall_attentionsridxZ encoder_layer layer_outputsr3r3r4rjsB     zInstructBlipEncoder.forward)NNNN)r8r9r:r;r"rBrr<rlrnrrr rjror3r3rNr4rqs$   rcs~eZdZdZeZdeffdd Ze     ddee j dee dee d ee d e d e e eff d d ZddZZS)InstructBlipVisionModelr`r@csJt|||_|j}t||_t||_tj ||j d|_ | dSr) rArBr@rCr?rPrencoderrrrpost_layernorm post_init)r2r@rDrNr3r4rBs    z InstructBlipVisionModel.__init__NFrrrr_r+c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|j||d}|j||||d}|d}||}|dddddf} || } |s[|| f|ddSt|| |j |j dS)Nz You have to specify pixel_values)r_)rrrrrr!)r pooler_outputrr) r@rrrrrPrrrrr) r2r`rrrr_rencoder_outputsr pooled_outputr3r3r4rjs2   zInstructBlipVisionModel.forwardcC|jSr)rPr1r3r3r4get_input_embeddingsz,InstructBlipVisionModel.get_input_embeddingsNNNNF)r8r9r:main_input_namer$rrBrrr<r=rnrrrrjrror3r3rNr4rs0   *rcs\eZdZdfdd ZddZddZdd Zd d Zd d Z      dddZ Z S)rFcs"t||_|j|jdkrt|dstd|j|jf|j|_t|j|j|_|j|j|_ t |j|j |_ |rQt |j |j |_t |j |j |_nt |j|j |_t |j|j |_t |j|_t|dd|_|jdks{|jdkr|j|_t d|jd |j|_d |_dS) NrZembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)position_embedding_typeabsolute relative_keyrelative_key_queryrr!F)rArBr@rCrhasattrrrmattention_head_size all_head_sizerrrrZencoder_hidden_sizersrtDropoutZattention_probs_dropout_probrwr-rmax_position_embeddingsrdistance_embeddingsave_attentionr2r@is_cross_attentionrNr3r4rBs.   z.InstructBlipQFormerMultiHeadAttention.__init__cC ||_dSrattn_gradients)r2rr3r3r4save_attn_gradients# z9InstructBlipQFormerMultiHeadAttention.save_attn_gradientscCrrrr1r3r3r4get_attn_gradients&rz8InstructBlipQFormerMultiHeadAttention.get_attn_gradientscCrr attention_map)r2rr3r3r4save_attention_map)rz8InstructBlipQFormerMultiHeadAttention.save_attention_mapcCrrrr1r3r3r4get_attention_map,rz7InstructBlipQFormerMultiHeadAttention.get_attention_mapcCs6|dd|j|jf}|j|}|ddddS)NrSrrr!r )rTrrr]r[)r2xZ new_x_shaper3r3r4transpose_for_scores/s z:InstructBlipQFormerMultiHeadAttention.transpose_for_scoresNcCs|du}|r|||} |||} |}n;|durD|||} |||} tj|d| gdd} tj|d| gdd} n|||} |||} ||} || } | | f}t| | dd} |jdksv|jdkr| d}tj |tj |j d  dd}tj |tj |j d  dd}||}|||jd}|j| jd }|jdkrtd | |}| |} n|jdkrtd | |}td | |}| ||} | t|j} | j}|dur| |} tjdd| |}|r|jr||||j||}|dur||}t|| }|dddd }| dd|jf}|j |}|r=||fn|f}||f}|S)NrrrVr!rSrxrrrbdevicerazbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rrsrtr<r^rrr{rfrrTarangelongrr]rrrdrbZeinsummathsqrtrrZSoftmaxrr register_hookrrwr[r|r)r2rrurencoder_hidden_statesencoder_attention_maskpast_key_valuerrZ key_layerZ value_layerZmixed_query_layerZ query_layerZattention_scores seq_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_scores_dtypeZattention_probsZattention_probs_droppedZ context_layerZnew_context_layer_shaperr3r3r4rj4s`               z-InstructBlipQFormerMultiHeadAttention.forwardrkNNNNNF) r8r9r:rBrrrrrrjror3r3rNr4rsrc8eZdZfddZdejdejdejfddZZS)rcsBtt|j|j|_tj|j|jd|_t|j |_ dSr) rArBrrrCdenserrrhidden_dropout_probrwrMrNr3r4rB z&InstructBlipQFormerSelfOutput.__init__r input_tensorr+cC&||}||}|||}|Srrrwrr2rrr3r3r4rj  z%InstructBlipQFormerSelfOutput.forwardrr3r3rNr4r $rcseZdZdfdd ZddZ      ddejdeejd eejd eejd eejd ee e ejd ee de ejfddZ Z S)InstructBlipQFormerAttentionFcs,tt|||_t||_t|_dSr)rArBr attentionroutputset pruned_headsrrNr3r4rBs    z%InstructBlipQFormerAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrr!rV)lenrrrrrrrrrsrtrrrunion)r2headsindexr3r3r4 prune_headss z(InstructBlipQFormerAttention.prune_headsNrrurr r r rr+c Cs<||||||||}||d|} | f|dd} | S)Nrr!)rr) r2rrurr r r rZ self_outputsattention_outputrr3r3r4rjs  z$InstructBlipQFormerAttention.forwardrkr ) r8r9r:rBr!r<rlrr=rrnrjror3r3rNr4rs4 rcr)InstructBlipQFormerIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) rArBrrrCrrrrstrr intermediate_act_fnrMrNr3r4rBs   z(InstructBlipQFormerIntermediate.__init__rr+cCs||}||}|Sr)rr%rr3r3r4rjs  z'InstructBlipQFormerIntermediate.forwardrr3r3rNr4r#s r#cr)InstructBlipQFormerOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rArBrrrrCrrrrrrwrMrNr3r4rBrz"InstructBlipQFormerOutput.__init__rrr+cCrrrrr3r3r4rjrz!InstructBlipQFormerOutput.forwardrr3r3rNr4r&rr&csDeZdZfddZ       d ddZdd Zd d ZZS) InstructBlipQFormerLayercs~t|j|_d|_t||_||_||jdkr&t|dd|_d|_ nd|_ t ||_ t ||_ t ||_t ||_dS)Nr!rT)rF)rArBchunk_size_feed_forward seq_len_dimrr layer_idxZcross_attention_frequencycrossattentionhas_cross_attentionr# intermediater&rintermediate_query output_query)r2r@r*rNr3r4rBs     z!InstructBlipQFormerLayer.__init__NFrc  Cs:|dur |ddnd} |j||||| d} | d} | dd} | d} |dkr| ddd|ddf}|jrW|dur@td|j||||||d}|d}| |dd} t|j|j|j|}| jd|krt|j |j|j| dd|dddf}t j ||gdd}n t|j |j|j| }|f| } | | f} | S) Nr)rr rr!rSz>encoder_hidden_states must be given for cross-attention layersrrV) rr,rr+rfeed_forward_chunk_queryr(r)rXfeed_forward_chunkr<r^)r2rrurr r r r query_lengthZself_attn_past_key_valueZself_attention_outputsr"rZpresent_key_valueZquery_attention_outputZcross_attention_outputs layer_outputZlayer_output_textr3r3r4rjsd    z InstructBlipQFormerLayer.forwardcC||}|||}|Sr)r-rr2r"Zintermediate_outputr3r3r3r4r1G  z+InstructBlipQFormerLayer.feed_forward_chunkcCr4r)r.r/r5r3r3r4r0Lr6z1InstructBlipQFormerLayer.feed_forward_chunk_query)NNNNNFr)r8r9r:rBrjr1r0ror3r3rNr4r's  Gr'cs:eZdZfddZ          d ddZZS) InstructBlipQFormerEncodercr)Ncsg|]}t|qSr3)r')r/r*rr3r4rXz7InstructBlipQFormerEncoder.__init__..F) rArBr@rrrrlayerrrMrNrr4rBTs  z#InstructBlipQFormerEncoder.__init__NFTrc  CsV| rdnd} |r dnd} |rdnd}|rdnd}t|jjD]l}|j|}| r,| |f} |dur4||nd}|dur>||nd}t|jddr_|jr_|rStdd}||j |||||}n ||||||||| }|d}|rw||df7}|r| |df} |j r||df}q| r| |f} | st d d ||| | |fDSt ||| | |d S) Nr3rFzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...rrSr!rcsrrr3rr3r3r4r5sz5InstructBlipQFormerEncoder.forward..)rpast_key_valuesrrcross_attentions) rr@rr9r-rzrwarningrrr,r6r)r2rrurr r r: use_cacherrrr2Zall_hidden_statesZall_self_attentionsZall_cross_attentionsZnext_decoder_cacheiZ layer_moduleZlayer_head_maskr rr3r3r4rj\sx          z"InstructBlipQFormerEncoder.forward) NNNNNNFFTr)r8r9r:rBrjror3r3rNr4r7Ss  r7cs2eZdZdZfddZ    dddZZS) rz;Construct the embeddings from word and position embeddings.csttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j|_|jdt|jdddt|dd|_||_dS) N)Z padding_idxr position_ids)r!rSF) persistentrr)rArBrr vocab_sizerCZ pad_token_idword_embeddingsrposition_embeddingsrr layernormrrrwZregister_bufferr<rrgr-rr@rMrNr3r4rBs  z&InstructBlipQFormerEmbeddings.__init__NrcCs|dur |d}nd}|dur |jdd|||f}|durI||}|jdkr;|||j}||}|durHtj ||fdd}n|}||j j j }| |}| |}|S)Nr!rrrV)rTr?clonerBrrCrdrr<r^rDrcrbrw)r2 input_idsr? query_embedspast_key_values_lengthr rPrCr3r3r4rjs$    z%InstructBlipQFormerEmbeddings.forward)NNNr)r8r9r:r;rBrjror3r3rNr4rs rcs"eZdZdZdZdZdZdZdeffdd Z ddZ dd Z d d Z d!d e jd eede jdede jf ddZ           d"de jd ee jdee jdee jdee jdee jdee jdeeee jdeedeedeedeedeee jeffdd ZZS)#InstructBlipQFormerModelz Querying Transformer (Q-Former), used in InstructBLIP. Slightly modified from BLIP-2 as it also takes the instruction as input. Fr@cs2t|||_t||_t||_|dSr)rArBr@rrPr7rrrMrNr3r4rBs    z!InstructBlipQFormerModel.__init__cCs|jjSrrPrBr1r3r3r4rsz-InstructBlipQFormerModel.get_input_embeddingscCs ||j_dSrrJr2rtr3r3r4set_input_embeddingss z-InstructBlipQFormerModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrr9rr!)r2Zheads_to_pruner9rr3r3r4 _prune_headssz%InstructBlipQFormerModel._prune_headsru input_shaper has_queryr+cCs|dkr|dddddddf}n|dkr(|ddddddf}n td|d|jd|j|jd}d|d }|S) a> Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`Tuple[int]`): The shape of the input to the model. device: (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. r Nrz!Wrong shape for input_ids (shape z) or attention_mask (shape )rarg)rWrrXrdrb)r2rurOrrPextended_attention_maskr3r3r4get_extended_attention_masks   z4InstructBlipQFormerModel.get_extended_attention_maskNrFr?rGrr r r:r=rrrc  s| dur| njj} | dur| njj} | dur| njj} |dur*|dur*td|dur;|ddjdjjnd} |durF|jdnd}j|||| d}|dd}|\}}|j }|durot j ||| f|d} |||}|durt |tr|d\}}}n|\}}}||f}t |trfd d |D}n|durt j ||d}|}n|}nd}|jj}j||||||| | | | |d }|d}|dddddf}| s||f|ddSt|||j|j|j|jd S) a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of: shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Nz7You have to specify query_embeds when input_ids is Nonerrr!)rFr?rGrHrS)rcsg|]}|qSr3)invert_attention_mask)r/maskr1r3r4rr8z4InstructBlipQFormerModel.forward..) rurr r r:r=rrrr2)rrr:rrr;)r@rrrrrXr2rPrTrr<onesrSrlistrTZ get_head_maskrrrr:rrr;)r2rFrur?rGrr r r:r=rrrrHr2Zembedding_outputrOrhr rrRZencoder_batch_sizeZencoder_sequence_lengthriZencoder_hidden_shapeZencoder_extended_attention_maskrZsequence_outputrr3r1r4rj3sv"$     z InstructBlipQFormerModel.forwardrk) NNNNNNNNNNN)r8r9r:r;rrrrr#rBrrLrNr<rlrrmrrnrS LongTensorrr=rrrjror3r3rNr4rIsx   .     rIc@s eZdZdS)KwargsForCausalLMN)r8r9r:r3r3r3r4rYs rYz[ InstructBLIP base Model consisting of language model, qformer and vision encoder. )Z custom_introc!seZdZdZdgZdeffdd ZddZdd Zd d Z d d Z e e          dde jde jdee jdee jdee jdee jdee jdeedeedeededeedeedeeeffddZZS) rr`rr@cst|t|j|_ttd|j |j j |_ t |j |_t|j j |jj |_t|j|_|jjdur@|j|jj|jjdurN|j|jj|dSNr!)rArBr vision_config vision_modelrrFr<rnum_query_tokensqformer_configrCrrIqformerr text_configlanguage_projectionr from_configlanguage_modelrextend_keep_in_fp32_modulesrrMrNr3r4rBs      zInstructBlipModel.__init__cC |jSrrcrr1r3r3r4rrz&InstructBlipModel.get_input_embeddingscC|j|dSrrcrLrKr3r3r4rLz&InstructBlipModel.set_input_embeddingscC,|jjs|jj|jj_|jj|jj_dSdSrr@use_decoder_only_language_modelrcZsharedrZ embed_tokensdecoderr1r3r3r4 _tie_weightszInstructBlipModel._tie_weightscCP|j}t|dkrd|vrtjdkrtdt|jdr&d|jj _ dSdSz Some pre-processing hacks to make the model `accelerate` compatible. Check https://github.com/huggingface/transformers/pull/21707 for more details. r!rcaThe `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning. Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for more details on creating a `device_map` for large models._hf_hookTN hf_device_maprr<cudaZ device_countrr<rrcrsZio_same_devicer2rur3r3r4_preprocess_accelerate" z(InstructBlipModel._preprocess_accelerateNFqformer_input_idsqformer_attention_maskrFrudecoder_input_idsdecoder_attention_maskrrrr_r=r}r+c  Ks| dur| n|jj} |j||| | | d}|d}tj|ddtj|jd}|j |j ddd}tj|ddtj|jd}|durLt |}tj ||gdd}|j ||||||| | d}|dddd|dddf}||}|j|}|durt |}||jjkd|}|||<|jjr|jd |||| | | d | }n|jd |||||| | | d | }t|||d S) a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the language model. Input tokens can optionally be provided to serve as text prompt, which the language model can continue. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. N)r`rrrr_rrSrr!rV)rFrurGr r rrrrrurrrr=)rrur|r}rrrr=r,r3)r@rr\r<rVrTrrrrgrX ones_liker^r_rarcrimage_token_id unsqueeze expand_asrermr%)r2r`rzr{rFrur|r}rrrr_r=r}r( image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsrspecial_image_maskrr3r3r4rjsx0   $      zInstructBlipModel.forward) NNNNNNNNFN)r8r9r:rrer"rBrrLrorxrrr<r=rrXrnrr rrr%rjror3r3rNr4rsb      ra InstructBLIP Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. c#seZdZeZdZdZdZdZdgZ deffdd Z dd Z d d Z d d Z dejfddZddZddZddZddZ   d-dejdejdeejdeedeef ddZee           d.dejdejdeejd eejd!eejd"eejd#eejd$eed%eed&eejdeeded'eed(eedee e!ffd)d*Z"e#     d/dejdeejdeejd eejd!eejdedejfd+d,Z$Z%S)0rr`TFrr@cst|t|j|_tt d|j |j j |_ t|j |_t|j j |jj |_|jr7t|j}nt|j}|jdurI|j|j|jdurU|j|j||_|dSrZ)rArBrZ _from_configr[r\rrFr<rr]r^rCrrIr_rr`rarmrrbr rrdrercr)r2r@rcrNr3r4rBvs     z-InstructBlipForConditionalGeneration.__init__cCrfrrgr1r3r3r4rrz9InstructBlipForConditionalGeneration.get_input_embeddingscCrhrrirKr3r3r4rLrjz9InstructBlipForConditionalGeneration.set_input_embeddingscCrhr)rcset_output_embeddings)r2Znew_embeddingsr3r3r4rrjz:InstructBlipForConditionalGeneration.set_output_embeddingsr+cCrfr)rcget_output_embeddingsr1r3r3r4rrz:InstructBlipForConditionalGeneration.get_output_embeddingscCrfr)rc get_encoderr1r3r3r4rrz0InstructBlipForConditionalGeneration.get_encodercCrfr)rc get_decoderr1r3r3r4rrz0InstructBlipForConditionalGeneration.get_decodercCrkrrlr1r3r3r4rorpz1InstructBlipForConditionalGeneration._tie_weightscCrqrrrtrwr3r3r4rxryz;InstructBlipForConditionalGeneration._preprocess_accelerateNrzr{r_rcCs|j||dd}|d}tj|ddtj|jd}|j|jddd} tj| ddtj|jd} |dur@t |}tj | |gdd}|j ||| ||dd } | dddd| dddf} | | } |rr| || fS| S) a$ Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. T)r`r_rrNrSrr!rV)rFrurGr r r) r\r<rVrTrrrrgrXrr^r_ra)r2r`rzr{r_rr(rrrrrrrr3r3r4get_image_featuress2   $  z7InstructBlipForConditionalGeneration.get_image_featuresrFrur|r}rrlabelsr=r}c Ks| dur| n|jj} |j|||| dd\}}}| s|n|}| s%|n|}tj|ddtj|jd}|j |}|durGt |}t |jdddurc||jj kd|}|||<ntdtj|||jgdd }tj|||jgdd }|jjr|j d|||| | | d |}| r|jn|d }d}| dur|jd|| |jjjd |}n$|j d|||||| | | | d |}| r|jn|d }| r|jn|d}t|||||dS)a+ qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the language model. Input tokens can optionally be provided to serve as text prompt, which the language model can continue. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration >>> import torch >>> from PIL import Image >>> import requests >>> model = InstructBlipForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> model.to(device) # doctest: +IGNORE_RESULT >>> url = "https://raw.githubusercontent.com/salesforce/LAVIS/main/docs/_static/Confusing-Pictures.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> prompt = "What is unusual about this image?" >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... min_length=1, ... top_p=0.9, ... repetition_penalty=1.5, ... length_penalty=1.0, ... temperature=1, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) The unusual aspect of this image is that a man is ironing clothes on the back of a yellow SUV, which is parked in the middle of a busy city street. This is an unconventional approach to ironing clothes, as it requires the man to balance himself and his ironing equipment on top of the vehicle while navigating through traffic. Additionally, the presence of taxis and other vehicles in the scene further emphasizes the unusual nature of this situation. ```NTrzr{r_rrSrrAExpanding inputs for image tokens in InstructBLIP should be done in processing. Please follow instruction here (https://gist.github.com/zucchini-nlp/e9f20b054fa322f84ac9311d9ab67042) to update your InstructBLIP model. Using processors without these attributes in the config is deprecated and will throw an error in v4.50.r!rVr~r)r'rrA) rrur|r}rrrrr=)r&r'r(r)r*r3)r@rrr.r<rVrTrrrcrrr-rrrrerrr^rdrmr'Z loss_functionr`rAr&r%)r2r`rzr{rFrur|r}rrrrr_r=r}rr(rZlanguage_model_attention_maskrrrr'r&r3r3r4rjsW      z,InstructBlipForConditionalGeneration.forwardcKst|dr ||jd}|j||||dd\} } } tj| ddtj| jd} |durZ|j j j g} t |j dddurI|j j g|j j| } tj| gtj|jd}||d }|durct|}||}t |j dddur||j j kd|}| |j||<n@td tj| || jgd d }tj| || jgd d }|jj js|d d | jd d |d <|dd| jd |d<||d}|jj js||d<|jjdi||}|S)a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width)): Input images to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. rurTrNrSrrr!rrV max_lengthZ min_length)rrurFr3)rrxrXrr<rVrTrrr@r`Z bos_token_idr-rr]rrepeatrrrrrerdrrr^rcZis_encoder_decodergetgenerate)r2r`rzr{rFrur_Zgenerate_kwargsrhrr(rZlanguage_attention_maskZ start_tokensrrZinputsrr3r3r4rsR          z-InstructBlipForConditionalGeneration.generate)NFF) NNNNNNNNNFNr)&r8r9r:r"rrrrrrerBrrLrrModulerrrrorxr<r=rXrrnrrrrrYrrr%rjZno_gradrror3r3rNr4res  1      " r)rIrrrr)rp)Lr;r dataclassesrtypingrrrrrr<Ztorch.utils.checkpointrZ activationsr Z generationr Zmodeling_flash_attention_utilsr Zmodeling_outputsr rrrZmodeling_utilsrrZprocessing_utilsrZ pytorch_utilsrrrutilsrrrrrrautorrr Zconfiguration_instructblipr"r#r$Z get_loggerr8rr%rr?rlfloatrrrrrrrrrrr#r&r'r7rrIrYrr__all__r3r3r3r4s        !R U0(W>2h_3J0