o Zhm@sZddlmZddlmZmZmZddlZddlZddlm Z m Z ddl m Z m Z mZmZmZmZmZddlmZddlmZdd lmZdd lmZdd lmZd d lmZmZe e!Z"Gddde Z#Gddde Z$GdddeZ%GdddeZ&GdddeZ'GdddeZ(eGddde Z)GdddeZ*Gddde Z+gd Z,dS)!) dataclass)OptionalTupleUnionN)InstructBlipQFormerConfigInstructBlipVisionConfig)$InstructBlipForConditionalGeneration/InstructBlipForConditionalGenerationModelOutputInstructBlipModelInstructBlipPreTrainedModelInstructBlipQFormerModelInstructBlipVisionModelKwargsForCausalLM)PretrainedConfig)FlashAttentionKwargs)!MODEL_FOR_CAUSAL_LM_MAPPING_NAMES)Unpack)logging)CONFIG_MAPPING AutoConfigc@ eZdZdS)InstructBlipVideoVisionConfigN__name__ __module__ __qualname__rrn/var/www/auris/lib/python3.10/site-packages/transformers/models/instructblipvideo/modular_instructblipvideo.pyr/rc@r)InstructBlipVideoQFormerConfigNrrrrrr!3r r!cs^eZdZdZdZddiZeeedZ     dfdd Z e d ed ed e fd dZ ZS)InstructBlipVideoConfiga [`InstructBlipVideoConfig`] is the configuration class to store the configuration of a [`InstructBlipVideoForConditionalGeneration`]. It is used to instantiate a Instructblipvideo model according to the specified arguments, defining the vision model, Q-Former model and language model configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the Instructblipvideo [Salesforce/instruct-blip-flan-t5](https://huggingface.co/Salesforce/instruct-blip-flan-t5) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vision_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`InstructBlipVideoVisionConfig`]. qformer_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`InstructBlipVideoQFormerConfig`]. text_config (`dict`, *optional*): Dictionary of configuration options used to initialize any [`PretrainedConfig`]. num_query_tokens (`int`, *optional*, defaults to 32): The number of query tokens passed through the Transformer. video_token_index (`int`, *optional*): Token index of special video token. kwargs (*optional*): Dictionary of keyword arguments. Example: ```python >>> from transformers import ( ... InstructBlipVideoVisionConfig, ... InstructBlipVideoQFormerConfig, ... OPTConfig, ... InstructBlipVideoConfig, ... InstructBlipVideoForConditionalGeneration, ... ) >>> # Initializing a InstructBlipVideoConfig with Salesforce/instruct-blip-flan-t5 style configuration >>> configuration = InstructBlipVideoConfig() >>> # Initializing a InstructBlipVideoForConditionalGeneration (with random weights) from the Salesforce/instruct-blip-flan-t5 style configuration >>> model = InstructBlipVideoForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config >>> # We can also initialize a InstructBlipVideoConfig from a InstructBlipVideoVisionConfig, InstructBlipVideoQFormerConfig and any PretrainedConfig >>> # Initializing Instructblipvideo vision, Instructblipvideo Q-Former and language model configurations >>> vision_config = InstructBlipVideoVisionConfig() >>> qformer_config = InstructBlipVideoQFormerConfig() >>> text_config = OPTConfig() >>> config = InstructBlipVideoConfig.from_text_vision_configs(vision_config, qformer_config, text_config) ```Zinstructblipvideovideo_token_idvideo_token_index) text_configqformer_config vision_configN c stjdi||duri}td|duri}td|dur*i}tdtdi||_tdi||_d|vrB|dnd}t|di||_ ||_ ||_ |jj |j_ |j jtv|_d|_d|_dS) NzZvision_config is None. initializing the InstructBlipVideoVisionConfig with default values.z\qformer_config is None. Initializing the InstructBlipVideoQFormerConfig with default values.zTtext_config is None. Initializing the text config with default values (`OPTConfig`). model_typeoptg?g{Gz?r)super__init__loggerinforr'r!r&rr%num_query_tokensr$Z hidden_sizeZencoder_hidden_sizer)ruse_decoder_only_language_modelZinitializer_factorZinitializer_range)selfr'r&r%r/r$kwargsZtext_model_type __class__rrr,ys(      z InstructBlipVideoConfig.__init__r'r&r%cKs"|d|||d|S)a  Instantiate a [`InstructBlipVideoConfig`] (or a derived class) from a InstructBlipVideo vision model, Q-Former and language model configurations. Returns: [`InstructBlipVideoConfig`]: An instance of a configuration object )r'r&r%Nr)to_dict)clsr'r&r%r2rrr from_vision_qformer_text_configssz8InstructBlipVideoConfig.from_vision_qformer_text_configs)NNNr(N)rrr__doc__r)Z attribute_maprr!rZ sub_configsr, classmethodrr7 __classcell__rrr3rr"7s.7#r"c@r) InstructBlipVideoPreTrainedModelNrrrrrr;r r;c@r)InstructBlipVideoVisionModelNrrrrrr<r r<c@r)InstructBlipVideoQFormerModelNrrrrrr=r r=c@r)4InstructBlipVideoForConditionalGenerationModelOutputNrrrrrr>sr>c@seZdZ          ddejdejdeejdeejdeejdeejd eejd eed eed eed edeedee de e e ffddZ dS)InstructBlipVideoModelNF pixel_valuesqformer_input_idsqformer_attention_mask input_idsattention_maskdecoder_input_idsdecoder_attention_maskoutput_attentionsoutput_hidden_states return_dictinterpolate_pos_encoding use_cacher2returnc  Ks| dur| n|jj} |j\}}}}}||||||}|j||| | | d}|d}tj|ddtj|j d}|j |jddd}tj|ddtj|j d}|dur^t |}|j |dd}|j |dd}tj||gdd}|j||||||| | d}|dddd|dddf}||}|||jj|d}|j|}|durt |}||jjkd|}|||<|jjr|jd |||| | | d| }n|jd |||||| | | d | }t|||d S) N)r@rGrHrIrJrZdtypedevicedim)rCrD query_embedsencoder_hidden_statesencoder_attention_maskrGrHrI inputs_embedsrDrGrHrIrK)rWrDrErFrGrHrIrK)vision_outputsqformer_outputslanguage_model_outputsr)configuse_return_dictshapereshape vision_modeltorchonessizelongrO query_tokensexpand ones_likerepeat_interleavecatqformerlanguage_projectionr/language_modelget_input_embeddingsr# unsqueeze expand_asflattenr0r>)r1r@rArBrCrDrErFrGrHrIrJrKr2 batch_sizeframeschannelheightwidthrX image_embedsimage_attention_maskrdquery_attention_mask query_outputs query_outputlanguage_model_inputsrWspecial_image_maskoutputsrrrforwards   $      zInstructBlipVideoModel.forward) NNNNNNNNFN)rrrr` FloatTensorr LongTensorboolrrrrr>r}rrrrr?sP      r?c!@steZdZ   ddejdejdeejdeedeef dd Z   ddejdejdeejdeedeef d d Z           ddejdejdeejd eejd eejdeejdeejdeedeedeejdeededeede e de e effddZe     ddejdeejdeejd eejd eejdedejfddZdS))InstructBlipVideoForConditionalGenerationNFr@rArBrJrIcCs>|j\}}}} } ||||| | }|j||dd} | d} tj| ddtj| jd} |j | jddd}tj|ddtj| jd}|durRt |}|j |dd}|j |dd}tj ||gdd}|j |||| | dd }|dddd|dddf}||}|||jj|d}|r|| |fS|S) a$ Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. T)r@rJrIrNrMrNrPrR)rCrDrSrTrUrI)r]r^r_r`rarbrcrOrdrerfrgrhrirjr[r/)r1r@rArBrJrIrprqrrrsrtrXrurvrdrwrxryrzrrrget_video_features*s<   $  z>> from transformers import InstructBlipVideoProcessor, InstructBlipVideoForConditionalGeneration >>> import torch >>> from huggingface_hub import hf_hub_download >>> import av >>> import numpy as np >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> model = InstructBlipVideoForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b", device_map="auto") >>> processor = InstructBlipVideoProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample uniformly 4 frames from the videWhy is this video funny?o >>> total_frames = container.streams.video[0].frames >>> indices = np.arange(0, total_frames, total_frames / 4).astype(int) >>> clip = read_video_pyav(container, indices) >>> prompt = "What is happening in the video?" >>> inputs = processor(text=prompt, images=clip, return_tensors="pt").to(model.device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... repetition_penalty=1.5, ... length_penalty=1.0, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) "A person is eating a bowl of pasta, and they are using a fork to eat it. The person is sitting at a table, and the plate of pasta is on the table in front" ```NTrArBrJrIrMrNr#KExpanding inputs for video tokens in InstructBLIPVideo should be done in processing. Please follow instruction here (https://gist.github.com/zucchini-nlp/65f22892b054dc0d68228af56fbeaac2) to update your InstructBLIPVideo model. Using processors without these attributes in the config is deprecated and will throw an error in v4.47.rRrPrVr)logitsr vocab_size) rWrDrErFrGrHrIrrK)lossrrXrYrZr)r[r\rZto_tupler`rarbrcrOrkrlrfgetattrr#rmrnrotor- warning_oncerhr0rZ loss_functionr%rrr>)r1r@rArBrCrDrErFrGrHrrIrJrKr2rzrXrxZlanguage_model_attention_maskrWr{r|rrrrrr}psI      z1InstructBlipVideoForConditionalGeneration.forwardcKst|dr ||jd}|j||||dd\} } } tj| ddtj| jd} |dur\|j j j g} t |j dddurK|j j g|j jd | } tj| gtj|jd}||d }|duret|}||}t |j dddur||j j kd|}| |j||<n@td tj| || jgd d }tj| || jgd d }|jj js|d d| jd d |d <|dd| jd |d<||d}|jj js||d<|jjdi||}|S)a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width) or (batch_size, num_frames, num_channels, height, width)): Input images or videos to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. Z hf_device_maprTrNrMrNr#rRrrP max_lengthZ min_length)rWrDrCr)hasattrZ_preprocess_accelerater]rr`rarbrcrOr[r%Z bos_token_idrr#r/Ztensorrepeatrfrlrmrnrorr-rrhrkZis_encoder_decodergetgenerate)r1r@rArBrCrDrJZgenerate_kwargsrprzrXrxZlanguage_attention_maskZ start_tokensrWr{Zinputsr|rrrrsR          z2InstructBlipVideoForConditionalGeneration.generate)NFF) NNNNNNNNNFN)NNNNF)rrrr`r~rrrrrrrrrr>r}Zno_gradrrrrrr)s @         r)r"r!rr<r;r=r?r)- dataclassesrtypingrrrr`Ztorch.utils.checkpointZ;transformers.models.instructblip.configuration_instructbliprrZ6transformers.models.instructblip.modeling_instructbliprr r r r r rZconfiguration_utilsrZmodeling_flash_attention_utilsrZmodels.auto.modeling_autorZprocessing_utilsrutilsrautorrZ get_loggerrr-rr!r"r;r<r=r>r?r__all__rrrrs4 $     }d ;