o ZhG@sddlZddlmZddlmZmZmZmZmZddl Z ddl m Z ddl m Z ddl mZddlmZdd lmZmZmZmZdd lmZmZdd lmZdd lmZmZmZdd lm Z m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(m)Z)ddl*m+Z+m,Z,m-Z-e$.e/Z0Gddde j1Z2 dHde j1de j3de j3de j3dee j3de4de4fddZ5Gddde j1Z6Gd d!d!e j1Z7Gd"d#d#e j1Z8Gd$d%d%e j1Z9Gd&d'd'e j1Z:Gd(d)d)e j1Z;Gd*d+d+e j1ZGd0d1d1e j1Z?Gd2d3d3e j1Z@Gd4d5d5e j1ZAGd6d7d7ee ZBe"Gd8d9d9eZCGd:d;d;eCZDGdd?d?e!ZFe"d@dAGdBdCdCeCZGe"dDdAGdEdFdFeCeZHgdGZIdS)IN) dataclass)AnyCallableOptionalTupleUnion)nn)ACT2FN)GenerationMixin)FlashAttentionKwargs)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) LossKwargs ModelOutputauto_docstringcan_return_tuplelogging torch_int) AutoModelAutoModelForCausalLMAutoModelForSeq2SeqLM)InstructBlipVideoConfigInstructBlipVideoQFormerConfigInstructBlipVideoVisionConfigcs\eZdZdeffdd Zdejdededejfdd Zdd ej d e dejfd dZ Z S)!InstructBlipVideoVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt dd|j|_ tj d|j|j|jd|_ |j|jd|_|jd|_tt d|j|j|_dS)Nr!r )Z in_channelsZ out_channelsZ kernel_sizeZstrider)super__init__r& hidden_size embed_dimZ image_size patch_sizer ParametertorchZrandnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingselfr& __class__o/var/www/auris/lib/python3.10/site-packages/transformers/models/instructblipvideo/modeling_instructblipvideo.pyr(6s  z*InstructBlipVideoVisionEmbeddings.__init__ embeddingsheightwidthreturnc Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||j} ||j} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r!Ng?rr rZbicubicF)sizemodeZ align_cornersdim)shaper3r-Zjit is_tracingr+rreshapepermuter functionalZ interpolateviewcat) r5r:r;r<r1r2Zclass_pos_embedZpatch_pos_embedrBZ new_heightZ new_widthZsqrt_num_positionsr8r8r9interpolate_pos_encodingHs(     z:InstructBlipVideoVisionEmbeddings.interpolate_pos_encodingF pixel_valuesrJc Cs|j\}}}}|jjj}||j|d}|ddd}|j|dd|} t j | |gdd} |r<| | ||} n|j } | | ddd| dddf|} | S)Ndtyperr!r>rA)rCr0weightrMtoflatten transposer.expandr-rIrJr3r?) r5rKrJ batch_size_r;r<Z target_dtypeZ patch_embedsZ class_embedsr:r3r8r8r9forwardps *z)InstructBlipVideoVisionEmbeddings.forwardF) __name__ __module__ __qualname__r$r(r-TensorintrJ FloatTensorboolrU __classcell__r8r8r6r9r%5s$(r%modulequerykeyvalueattention_maskscalingdropoutc Kspt||dd|}|dur||}tjj|dd}tjj|||jd}t||} | dd} | |fS)Nr>rA)ptrainingr!r) r-matmulrQrrGZsoftmaxrfri contiguous) r`rarbrcrdrerfkwargs attn_weights attn_outputr8r8r9eager_attention_forwards  rocseZdZdZfddZdejdedefddZ dd ejd e ejd e e de eje eje e ejffddZ Z S)InstructBlipVideoAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ d|_ |j |_ t j |jd|jdd|_|jr]t t|j}t t|j}nd}d}|duryt|tj|dd|f}t ||j_t |j|j|_dS) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFr )bias)Z requires_grad)r'r(r&r)r*num_attention_heads num_headshead_dim ValueErrorscaleZ is_causalattention_dropoutrLinearqkvqkv_biasr,r-zerosrIZ zeros_likerq projection)r5r&Zq_biasZv_biasrzr6r8r9r(s0   z#InstructBlipVideoAttention.__init__tensorseq_lenbszcCs ||||j|jddS)Nr!r)rHrsrtrQrk)r5r}r~rr8r8r9_shapes z!InstructBlipVideoAttention._shapeNF hidden_states head_maskoutput_attentionsr=cKs|\}}}||}|||d|j||jddddd}|d|d|d} } } t} |jjdkrJ|jjdkrD|rDt dnt |jj} | || | | fd |j sVd n|j |j d |\} }| ||d } || } |rx| |f}|S| d f}|S) z#Input shape: Batch x Time x Channelr rrr!eagerZsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.Nr_)rdrfrer>)r?ryrErsrFror&Z_attn_implementationlogger warning_oncerrirwrvrkr|)r5rrrrlrZtgt_lenr*Z mixed_qkvZ query_statesZ key_statesZ value_statesZattention_interfacernrmoutputsr8r8r9rUs>       z"InstructBlipVideoAttention.forward)NF)rWrXrY__doc__r(r-rZr[rrr]rrUr^r8r8r6r9rps rpc2eZdZfddZdejdejfddZZS)InstructBlipVideoMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) r'r(r&r hidden_act activation_fnrrxr)intermediate_sizefc1fc2r4r6r8r9r(s  zInstructBlipVideoMLP.__init__rr=cCs"||}||}||}|Sr)rrrr5rr8r8r9rUs   zInstructBlipVideoMLP.forwardrWrXrYr(r-rZrUr^r8r8r6r9rs rc sNeZdZdeffdd Z d dejdejdeede ej fd d Z Z S) InstructBlipVideoEncoderLayerr&csRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSNZeps) r'r(r)r*rp self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2r4r6r8r9r(s   z&InstructBlipVideoEncoderLayer.__init__Frrdrr=cCsb|}||}|j|||d\}}||}|}||}||}||}|f}|r/||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrr)rrrr)r5rrdrZresidualrmrr8r8r9rUs      z%InstructBlipVideoEncoderLayer.forwardrV) rWrXrYr"r(r-rZrr]rr\rUr^r8r8r6r9rs rc sheZdZdZdeffdd Z    d deejdee dee d ee d e e e ff d d Z ZS)InstructBlipVideoEncodera" Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipVideoEncoderLayer`]. Args: config (`InstructBlipVideoConfig`): The corresponding vision configuration for the `InstructBlipVideoEncoder`. r&c:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr8)r).0rTr&r8r9 7sz5InstructBlipVideoEncoder.__init__..F) r'r(r&r ModuleListrangenum_hidden_layerslayersgradient_checkpointingr4r6rr9r(4s   z!InstructBlipVideoEncoder.__init__Nrdroutput_hidden_states return_dictr=c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|}t|jD]/\} } |r<||f}|jrL|jrL|| j |||} n| |||d} | d}|r`|| df}q1|rh||f}|svt dd|||fDSt |||dS) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr8rrr!cs|] }|dur|VqdSrr8rvr8r8r9 zsz3InstructBlipVideoEncoder.forward..)last_hidden_stater attentions) r&rruse_return_dict enumeraterrri_gradient_checkpointing_func__call__tupler ) r5 inputs_embedsrdrrrZencoder_statesZall_attentionsridxZ encoder_layer layer_outputsr8r8r9rU:sB     z InstructBlipVideoEncoder.forward)NNNN)rWrXrYrr"r(rr-rZr]rrr rUr^r8r8r6r9r*s$   rcs\eZdZdfdd ZddZddZdd Zd d Zd d Z      dddZ Z S)*InstructBlipVideoQFormerMultiHeadAttentionFcs"t||_|j|jdkrt|dstd|j|jf|j|_t|j|j|_|j|j|_ t |j|j |_ |rQt |j |j |_t |j |j |_nt |j|j |_t |j|j |_t |j|_t|dd|_|jdks{|jdkr|j|_t d|jd |j|_d |_dS) NrZembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)position_embedding_typeabsolute relative_keyrelative_key_queryrr!F)r'r(r&r)rrhasattrrur[attention_head_size all_head_sizerrxraZencoder_hidden_sizerbrcDropoutZattention_probs_dropout_probrfgetattrrmax_position_embeddings Embeddingdistance_embeddingsave_attentionr5r&is_cross_attentionr6r8r9r(s.   z3InstructBlipVideoQFormerMultiHeadAttention.__init__cC ||_dSrattn_gradients)r5rr8r8r9save_attn_gradients z>InstructBlipVideoQFormerMultiHeadAttention.save_attn_gradientscC|jSrrr5r8r8r9get_attn_gradientsz=InstructBlipVideoQFormerMultiHeadAttention.get_attn_gradientscCrr attention_map)r5rr8r8r9save_attention_maprz=InstructBlipVideoQFormerMultiHeadAttention.save_attention_mapcCrrrrr8r8r9get_attention_maprzrrr!r )r?rrrrHrF)r5xZ new_x_shaper8r8r9transpose_for_scoress z?InstructBlipVideoQFormerMultiHeadAttention.transpose_for_scoresNcCs|du}|r|||} |||} |}n;|durD|||} |||} tj|d| gdd} tj|d| gdd} n|||} |||} ||} || } | | f}t| | dd} |jdksv|jdkr| d}tj |tj |j d  dd}tj |tj |j d  dd}||}|||jd}|j| jd }|jdkrtd | |}| |} n|jdkrtd | |}td | |}| ||} | t|j} | j}|dur| |} tjdd| |}|r|jr||||j||}|dur||}t|| }|dddd }| dd|jf}|j |}|r=||fn|f}||f}|S)NrrrAr!r>rgrrrMdevicerLzbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rrbrcr-rIrarjrQrr?arangelongrrHrrrOrMZeinsummathsqrtrrZSoftmaxrr register_hookrrfrFrkr)r5rrdrencoder_hidden_statesencoder_attention_maskpast_key_valuerrZ key_layerZ value_layerZmixed_query_layerZ query_layerZattention_scores seq_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_scores_dtypeZattention_probsZattention_probs_droppedZ context_layerZnew_context_layer_shaperr8r8r9rUs`               z2InstructBlipVideoQFormerMultiHeadAttention.forwardrVNNNNNF) rWrXrYr(rrrrrrUr^r8r8r6r9rsrc8eZdZfddZdejdejdejfddZZS)"InstructBlipVideoQFormerSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSr) r'r(rrxr)denserrrhidden_dropout_probrfr4r6r8r9r( z+InstructBlipVideoQFormerSelfOutput.__init__r input_tensorr=cC&||}||}|||}|Srrrfrr5rrr8r8r9rU   z*InstructBlipVideoQFormerSelfOutput.forwardrr8r8r6r9r $rcseZdZdfdd ZddZ      ddejdeejd eejd eejd eejd ee e ejd ee de ejfddZ Z S)!InstructBlipVideoQFormerAttentionFcs,tt|||_t||_t|_dSr)r'r(r attentionroutputset pruned_headsrr6r8r9r(s    z*InstructBlipVideoQFormerAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrr!rA)lenrrrrrrrrarbrcrrrunion)r5headsindexr8r8r9 prune_headss z-InstructBlipVideoQFormerAttention.prune_headsNrrdrrrrrr=c Cs<||||||||}||d|} | f|dd} | S)Nrr!)rr) r5rrdrrrrrZ self_outputsattention_outputrr8r8r9rU,s  z)InstructBlipVideoQFormerAttention.forwardrVr) rWrXrYr(rr-rZrr\rr]rUr^r8r8r6r9rs4 rcr)$InstructBlipVideoQFormerIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r'r(rrxr)rr isinstancerstrr intermediate_act_fnr4r6r8r9r(Es   z-InstructBlipVideoQFormerIntermediate.__init__rr=cCs||}||}|Sr)rrrr8r8r9rUMs  z,InstructBlipVideoQFormerIntermediate.forwardrr8r8r6r9rDs rcr)InstructBlipVideoQFormerOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r'r(rrxrr)rrrrrrfr4r6r8r9r(Trz'InstructBlipVideoQFormerOutput.__init__rrr=cCrrrrr8r8r9rUZrz&InstructBlipVideoQFormerOutput.forwardrr8r8r6r9rSrrcsDeZdZfddZ       d ddZdd Zd d ZZS) InstructBlipVideoQFormerLayercs~t|j|_d|_t||_||_||jdkr&t|dd|_d|_ nd|_ t ||_ t ||_ t ||_t ||_dS)Nr!rT)rF)r'r(chunk_size_feed_forward seq_len_dimrr layer_idxZcross_attention_frequencycrossattentionhas_cross_attentionr intermediaterrintermediate_query output_query)r5r&rr6r8r9r(bs     z&InstructBlipVideoQFormerLayer.__init__NFrc  Cs:|dur |ddnd} |j||||| d} | d} | dd} | d} |dkr| ddd|ddf}|jrW|dur@td|j||||||d}|d}| |dd} t|j|j|j|}| jd|krt|j |j|j| dd|dddf}t j ||gdd}n t|j |j|j| }|f| } | | f} | S) Nr)rrrr!r>z>encoder_hidden_states must be given for cross-attention layersrrA) rrrurrfeed_forward_chunk_queryrrrCfeed_forward_chunkr-rI)r5rrdrrrrr query_lengthZself_attn_past_key_valueZself_attention_outputsrrZpresent_key_valueZquery_attention_outputZcross_attention_outputs layer_outputZlayer_output_textr8r8r9rUvsd    z%InstructBlipVideoQFormerLayer.forwardcC||}|||}|Sr)rrr5rZintermediate_outputr r8r8r9r  z0InstructBlipVideoQFormerLayer.feed_forward_chunkcCr r)rrr r8r8r9rr z6InstructBlipVideoQFormerLayer.feed_forward_chunk_query)NNNNNFr)rWrXrYr(rUrrr^r8r8r6r9ras  Grcs:eZdZfddZ          d ddZZS) InstructBlipVideoQFormerEncodercr)Ncsg|]}t|qSr8)r)rrrr8r9rz.F) r'r(r&rrrrlayerrr4r6rr9r(s  z(InstructBlipVideoQFormerEncoder.__init__NFTrc  CsV| rdnd} |r dnd} |rdnd}|rdnd}t|jjD]l}|j|}| r,| |f} |dur4||nd}|dur>||nd}t|jddr_|jr_|rStdd}||j |||||}n ||||||||| }|d}|rw||df7}|r| |df} |j r||df}q| r| |f} | st d d ||| | |fDSt ||| | |d S) Nr8rFzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...rr>r!rcsrrr8rr8r8r9rsz:InstructBlipVideoQFormerEncoder.forward..)rpast_key_valuesrrcross_attentions) rr&rrrrirwarningrrrrr)r5rrdrrrr use_cacherrrr Zall_hidden_statesZall_self_attentionsZall_cross_attentionsZnext_decoder_cacheiZ layer_moduleZlayer_head_maskrrr8r8r9rUsx          z'InstructBlipVideoQFormerEncoder.forward) NNNNNNFFTr)rWrXrYr(rUr^r8r8r6r9rs  rcs2eZdZdZfddZ    dddZZS) "InstructBlipVideoQFormerEmbeddingsz;Construct the embeddings from word and position embeddings.csttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j|_|jdt|jdddt|dd|_||_dS) N)Z padding_idxr position_ids)r!r>F) persistentrr)r'r(rr vocab_sizer)Z pad_token_idword_embeddingsrposition_embeddingsrr layernormrrrfZregister_bufferr-rrRrrr&r4r6r8r9r(*s  z+InstructBlipVideoQFormerEmbeddings.__init__NrcCs|dur |d}nd}|dur |jdd|||f}|durI||}|jdkr;|||j}||}|durHtj ||fdd}n|}||j j j }| |}| |}|S)Nr!rrrA)r?rclonerrrrOrr-rIrrNrMrf)r5 input_idsr query_embedspast_key_values_lengthrr:rr8r8r9rU:s$    z*InstructBlipVideoQFormerEmbeddings.forward)NNNr)rWrXrYrr(rUr^r8r8r6r9r's rc@s eZdZdS)KwargsForCausalLMN)rWrXrYr8r8r8r9r!Zs r!c@sDeZdZeZdZdZdZdZdZ dZ dZ dZ dZ gdZddZdS) InstructBlipVideoPreTrainedModelZblipTF)rrprrcCs|jj}t|tjtjfr%|jjjd|d|j dur#|j j dSdSt|tj r6|jjjd|ddSt|tj rK|j j |jj ddSt|trftjj|jd|dtjj|jd|ddSt|ttfru|jj dSdS)zInitialize the weightsr_)meanZstdN?)r&Zinitializer_rangerrrxr/rNdataZnormal_rqZzero_rrZfill_r%initZ trunc_normal_r3r.)InstructBlipVideoForConditionalGenerationInstructBlipVideoModel query_tokens)r5r`factorr8r8r9 _init_weightsqs"     z.InstructBlipVideoPreTrainedModel._init_weightsN)rWrXrYr" config_classZbase_model_prefixZsupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_2_supports_sdpa_supports_flex_attn_supports_cache_class_supports_static_cache_supports_quantized_cache_no_split_modulesr+r8r8r8r9r"]s r"cs~eZdZdZeZdeffdd Ze     ddee j dee dee d ee d e d e e eff d d ZddZZS)InstructBlipVideoVisionModelrKr&csJt|||_|j}t||_t||_tj ||j d|_ | dSr) r'r(r&r)r%r:rencoderrrrpost_layernorm post_init)r5r&r*r6r8r9r(s    z%InstructBlipVideoVisionModel.__init__NFrrrrJr=c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|j||d}|j||||d}|d}||}|dddddf} || } |s[|| f|ddSt|| |j |j dS)Nz You have to specify pixel_values)rJ)rrrrrr!)r pooler_outputrr) r&rrrrur:r6r7rrr) r5rKrrrrJrencoder_outputsr pooled_outputr8r8r9rUs2   z$InstructBlipVideoVisionModel.forwardcCrr)r:rr8r8r9get_input_embeddingsrz1InstructBlipVideoVisionModel.get_input_embeddingsNNNNF)rWrXrYmain_input_namer$r,r(rrr-r\r]rrrrUr<r^r8r8r6r9r5s0   *r5cs"eZdZdZdZdZdZdZdeffdd Z ddZ dd Z d d Z d!d e jd eede jdede jf ddZ           d"de jd ee jdee jdee jdee jdee jdee jdeeee jdeedeedeedeedeee jeffdd ZZS)#InstructBlipVideoQFormerModelz Querying Transformer (Q-Former), used in InstructBlipVideo. Slightly modified from BLIP-2 as it also takes the instruction as input. Fr&cs2t|||_t||_t||_|dSr)r'r(r&rr:rr6r8r4r6r8r9r(s    z&InstructBlipVideoQFormerModel.__init__cCs|jjSrr:rrr8r8r9r<sz2InstructBlipVideoQFormerModel.get_input_embeddingscCs ||j_dSrr@r5rcr8r8r9set_input_embeddingss z2InstructBlipVideoQFormerModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr6rrr)r5Zheads_to_prunerrr8r8r9 _prune_headssz*InstructBlipVideoQFormerModel._prune_headsrd input_shaper has_queryr=cCs|dkr|dddddddf}n|dkr(|ddddddf}n td|d|jd|j|jd}d|d }|S) a> Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`Tuple[int]`): The shape of the input to the model. device: (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. r Nrz!Wrong shape for input_ids (shape z) or attention_mask (shape )rLr$g)rBrurCrOrM)r5rdrErrFextended_attention_maskr8r8r9get_extended_attention_masks   z9InstructBlipVideoQFormerModel.get_extended_attention_maskNrrrrrrrrrrrc  s| dur| njj} | dur| njj} | dur| njj} |dur*|dur*td|dur;|ddjdjjnd} |durF|jdnd}j|||| d}|dd}|\}}|j }|durot j ||| f|d} |||}|durt |tr|d\}}}n|\}}}||f}t |trfd d |D}n|durt j ||d}|}n|}nd}|jj}j||||||| | | | |d }|d}|dddddf}| s||f|ddSt|||j|j|j|jd S) a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of: shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Nz7You have to specify query_embeds when input_ids is Nonerrr!)rrrr r>)rcsg|]}|qSr8)invert_attention_mask)rmaskrr8r9r_rz9InstructBlipVideoQFormerModel.forward..) rdrrrrrrrrr )rr9rrrr)r&rrrrurCr r:r?rr-onesrIrlistrJZ get_head_maskrr6rrrrr)r5rrdrrrrrrrrrrr r Zembedding_outputrErSrrrHZencoder_batch_sizeZencoder_sequence_lengthrTZencoder_hidden_shapeZencoder_extended_attention_maskr:Zsequence_outputr;r8rr9rUsv"$     z%InstructBlipVideoQFormerModel.forwardrV) NNNNNNNNNNN)rWrXrYrr-r.r/r0r#r(r<rBrDr-rZrr[rr]rI LongTensorrr\rrrUr^r8r8r6r9r?sx   .     r?c@seZdZUdZdZeeeje d<dZ eeeje d<dZ eeje d<dZ eeeje d<dZ eeeje d<deefd d ZdS) 4InstructBlipVideoForConditionalGenerationModelOutputa. Class defining the outputs of [`InstructBlipVideoForConditionalGeneration`]. Args: loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Language modeling loss from the language model. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head of the language model. vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). language_model_outputs (`CausalLMOutputWithPast` or `Seq2SeqLMOutput`): Outputs of the language model. Nlosslogitsvision_outputsqformer_outputslanguage_model_outputsr=cstfddDS)Nc3s.|]}|dvr |nt|VqdS)rRrSrTN)rto_tuple)rkrr8r9rs  zPInstructBlipVideoForConditionalGenerationModelOutput.to_tuple..)rkeysrr8rr9rVs z=InstructBlipVideoForConditionalGenerationModelOutput.to_tuple)rWrXrYrrPrrr-r\__annotations__rQrRrSrTrrVr8r8r8r9rOs rOz` InstructBlipVideo base Model consisting of language model, qformer and vision encoder. )Z custom_introc!seZdZdZdgZdeffdd ZddZdd Zd d Z d d Z e e          dde jde jdee jdee jdee jdee jdee jdeedeedeededeedeedeeeffddZZS) r(rKr)r&cst|t|j|_ttd|j |j j |_ t |j |_t|j j |jj |_t|j|_|jjdur@|j|jj|jjdurN|j|jj|dSNr!)r'r(r5 vision_config vision_modelrr,r-r{num_query_tokensqformer_configr)r)r?qformerrx text_configlanguage_projectionr from_configlanguage_modelr4extend_keep_in_fp32_modulesr8r4r6r8r9r(s      zInstructBlipVideoModel.__init__cC |jSrrcr<rr8r8r9r<rz+InstructBlipVideoModel.get_input_embeddingscC|j|dSrrcrBrAr8r8r9rBz+InstructBlipVideoModel.set_input_embeddingscC,|jjs|jj|jj_|jj|jj_dSdSrr&use_decoder_only_language_modelrcZsharedr6Z embed_tokensdecoderrr8r8r9 _tie_weightsz#InstructBlipVideoModel._tie_weightscCP|j}t|dkrd|vrtjdkrtdt|jdr&d|jj _ dSdSz Some pre-processing hacks to make the model `accelerate` compatible. Check https://github.com/huggingface/transformers/pull/21707 for more details. r!rcaThe `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning. Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for more details on creating a `device_map` for large models._hf_hookTN hf_device_maprr-cudaZ device_countrrrrcrsZio_same_devicer5rur8r8r9_preprocess_accelerate" z-InstructBlipVideoModel._preprocess_accelerateNFqformer_input_idsqformer_attention_maskrrddecoder_input_idsdecoder_attention_maskrrrrJrrlr=c  Ks| dur| n|jj} |j\}}}}}||||||}|j||| | | d}|d}tj|ddtj|j d}|j |jddd}tj|ddtj|j d}|dur^t |}|j |dd}|j |dd}tj||gdd}|j||||||| | d}|dddd|dddf}||}|||jj|d}|j|}|durt |}||jjkd|}|||<|jjr|jd |||| | | d | }n|jd |||||| | | d | }t|||d S) a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipVideoProcessor`]. See [`InstructBlipVideoProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the language model. Input tokens can optionally be provided to serve as text prompt, which the language model can continue. Indices can be obtained using [`InstructBlipVideoProcessor`]. See [`InstructBlipVideoProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. N)rKrrrrJrr>rrAr!)rrdrrrrrrrrdrrrr)rrdr|r}rrrrrUr8)r&rrCrEr\r-rLr?rrr)rR ones_likerepeat_interleaverIr_rar]rcr<video_token_id unsqueeze expand_asrPrmrO)r5rKrzr{rrdr|r}rrrrJrrlrSframeschannelr;r<rR image_embedsimage_attention_maskr)query_attention_mask query_outputs query_outputlanguage_model_inputsrspecial_image_maskrr8r8r9rUs0   $      zInstructBlipVideoModel.forward) NNNNNNNNFN)rWrXrYr>rer"r(r<rBrorxrrr-r\rrNr]rr rrrOrUr^r8r8r6r9r(sb      r(a InstructBlipVideo Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. c#seZdZeZdZdZdZdZdgZ deffdd Z dd Z d d Z d d Z dejfddZddZddZddZddZ   d/dejdejdeejdeedeef ddZee           d0dejdejdeejd eejd!eejd"eejd#eejd$eed%eed&eejdeeded'eed(eedee e!ffd)d*Z"e#     d1dejdeejdeejd eejd!eejdedejfd+d,Z$   d/dejdejdeejdeedeef d-d.Z%Z&S)2r'rKTFr)r&cst|t|j|_tt d|j |j j |_ t|j |_t|j j |jj |_|jr7t|j}nt|j}|jdurI|j|j|jdurU|j|j||_|dSrZ)r'r(r5Z _from_configr[r\rr,r-r{r]r^r)r)r?r_rxr`rarmrrbr r4rdrercr8)r5r&rcr6r8r9r(|s     z2InstructBlipVideoForConditionalGeneration.__init__cCrfrrgrr8r8r9r<rz>InstructBlipVideoForConditionalGeneration.get_input_embeddingscCrhrrirAr8r8r9rBrjz>InstructBlipVideoForConditionalGeneration.set_input_embeddingscCrhr)rcset_output_embeddings)r5Znew_embeddingsr8r8r9rrjz?InstructBlipVideoForConditionalGeneration.set_output_embeddingsr=cCrfr)rcget_output_embeddingsrr8r8r9rrz?InstructBlipVideoForConditionalGeneration.get_output_embeddingscCrfr)rc get_encoderrr8r8r9rrz5InstructBlipVideoForConditionalGeneration.get_encodercCrfr)rc get_decoderrr8r8r9rrz5InstructBlipVideoForConditionalGeneration.get_decodercCrkrrlrr8r8r9rorpz6InstructBlipVideoForConditionalGeneration._tie_weightscCrqrrrtrwr8r8r9rxryz@InstructBlipVideoForConditionalGeneration._preprocess_accelerateNrzr{rJrcCsdS)$ Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. Nr8)r5rKrzr{rJrr8r8r9get_image_featuressz>> from transformers import InstructBlipVideoProcessor, InstructBlipVideoForConditionalGeneration >>> import torch >>> from huggingface_hub import hf_hub_download >>> import av >>> import numpy as np >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> model = InstructBlipVideoForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b", device_map="auto") >>> processor = InstructBlipVideoProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample uniformly 4 frames from the videWhy is this video funny?o >>> total_frames = container.streams.video[0].frames >>> indices = np.arange(0, total_frames, total_frames / 4).astype(int) >>> clip = read_video_pyav(container, indices) >>> prompt = "What is happening in the video?" >>> inputs = processor(text=prompt, images=clip, return_tensors="pt").to(model.device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... repetition_penalty=1.5, ... length_penalty=1.0, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) "A person is eating a bowl of pasta, and they are using a fork to eat it. The person is sitting at a table, and the plate of pasta is on the table in front" ```NTrzr{rJrr>rrKExpanding inputs for video tokens in InstructBLIPVideo should be done in processing. Please follow instruction here (https://gist.github.com/zucchini-nlp/65f22892b054dc0d68228af56fbeaac2) to update your InstructBLIPVideo model. Using processors without these attributes in the config is deprecated and will throw an error in v4.47.r!rAr~r)rQrr) rrdr|r}rrrrr)rPrQrRrSrTr8)r&rget_video_featuresrVr-rLr?rrrcr<rrrrrrPrOrrrIrmrQZ loss_functionr`rrPrO)r5rKrzr{rrdr|r}rrrrrJrrlrrRrZlanguage_model_attention_maskrrrrQrPr8r8r9rUsn      z1InstructBlipVideoForConditionalGeneration.forwardcKst|dr ||jd}|j||||dd\} } } tj| ddtj| jd} |dur\|j j j g} t |j dddurK|j j g|j jd | } tj| gtj|jd}||d }|duret|}||}t |j dddur||j j kd|}| |j||<n@td tj| || jgd d }tj| || jgd d }|jj js|d d| jd d |d <|dd| jd |d<||d}|jj js||d<|jjdi||}|S)a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width) or (batch_size, num_frames, num_channels, height, width)): Input images or videos to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. rurTrNr>rrrr!rrA max_lengthZ min_length)rrdrr8)rrxrCrr-rLr?rrr&r`Z bos_token_idrrr]r}repeatrr<rrrPrOrrrIrcZis_encoder_decodergetgenerate)r5rKrzr{rrdrJZgenerate_kwargsrSrrRrZlanguage_attention_maskZ start_tokensrrZinputsrr8r8r9rsR          z2InstructBlipVideoForConditionalGeneration.generatecCs>|j\}}}} } ||||| | }|j||dd} | d} tj| ddtj| jd} |j | jddd}tj|ddtj| jd}|durRt |}|j |dd}|j |dd}tj ||gdd}|j |||| | dd }|dddd|dddf}||}|||jj|d}|r|| |fS|S) rT)rKrJrrNr>rrAr!)rrdrrrr)rCrEr\r-rLr?rrr)rRrrrIr_rar&r])r5rKrzr{rJrrSrrr;r<rRrrr)rrrrr8r8r9rs<   $  zr1r2r3rer(r<rBrrModulerrrrorxr-r\rNrr]rrrrr!rrrOrUZno_gradrrr^r8r8r6r9r'ks        9 _r')r5r"r?r(r')r_)Jr dataclassesrtypingrrrrrr-rZ activationsr Z generationr Zmodeling_flash_attention_utilsr Zmodeling_outputsr rrrZmodeling_utilsrrZprocessing_utilsrZ pytorch_utilsrrrutilsrrrrrrautorrr Zconfiguration_instructblipvideor"r#r$Z get_loggerrWrrr%rZfloatrorprrrrrrrrrrrr!r"r5r?rOr(r'__all__r8r8r8r9s        R T0V1g_3'>J :  3