o Zhcg@spdZddlmZddlmZmZmZmZddlZddl Zddlm Z ddl m Z ddl mZdd lmZdd lmZmZdd lmZdd lmZdd lmZmZmZmZmZddlmZddl m!Z!e"e#Z$eGdddeZ%eGdddeZ&Gddde j'Z(eGdddeZ)eddGddde)Z*GdddeeZ+ed dGd!d"d"e)eZ,gd#Z-dS)$zPyTorch Llava model.) dataclass)ListOptionalTupleUnionN)nn)ACT2FN)GenerationMixin)FlashAttentionKwargs)BaseModelOutputWithPast ModelOutput)PreTrainedModel)Unpack) LossKwargsauto_docstringcan_return_tupleis_torchdynamo_compilinglogging) AutoModel) LlavaConfigc@s$eZdZUdZdZeejed<dS)LlavaModelOutputWithPasta Base class for Llava outputs, with hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__r"r"W/var/www/auris/lib/python3.10/site-packages/transformers/models/llava/modeling_llava.pyr&s rc@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZeejed<dS) LlavaCausalLMOutputWithPasta Base class for Llava causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)rrrrr%rrr r!r&r'rr(rr)rr"r"r"r#r$Gs r$cs*eZdZdeffdd ZddZZS)LlavaMultiModalProjectorconfigcsptt|jtr dnt|j}tj|jj ||j j |j d|_ t |j|_tj|j j |j j |j d|_dS)Nrbias)super__init__ isinstancevision_feature_layerintlenrLinear vision_config hidden_size text_configZmultimodal_projector_biaslinear_1r Zprojector_hidden_actactlinear_2)selfr+Znum_feature_layers __class__r"r#r/ps    z!LlavaMultiModalProjector.__init__cCs"||}||}||}|SN)r8r9r:)r;image_featuresr(r"r"r#forward~s   z LlavaMultiModalProjector.forward)rrrrr/r@ __classcell__r"r"r<r#r*osr*c@s<eZdZeZdZdZdZdZdZ dZ dZ dZ dZ ddZdS)LlavaPreTrainedModelTr'cCst|jd|jj}t|tjr)|jjj d|d|j dur'|j j dSdSt|tj r>|jj d|j j dSdS)Ninitializer_rangeg)meanstdg?)getattrr+Zget_text_configrDr0rr4weightdataZnormal_r-Zzero_Z LayerNormZfill_)r;modulerFr"r"r# _init_weightss   z"LlavaPreTrainedModel._init_weightsN)rrrrZ config_classZbase_model_prefixZsupports_gradient_checkpointingZ_skip_keys_device_placementZ_supports_cache_classZ_supports_flash_attn_2Z_supports_sdpaZ_supports_quantized_cacheZ_supports_static_cacheZ_supports_attention_backendrKr"r"r"r#rBs rBzu The Llava model which consists of a vision backbone and a language model, without a language modeling head. )Z custom_introc%s4eZdZddiZdeffdd ZddZdd Z dd ej d e e e e e fd e efddZee d dejd ej de ejde ejde e ej de ej d e e e e e fd e ede ede ede ede ede ejdejdeede eeff ddZZS)! LlavaModelzlanguage_model.modellanguage_modelr+cs>t|t|j|_t||_t|j|_ | dSr>) r.r/rZ from_configr5 vision_towerr*multi_modal_projectorr7rM post_initr;r+r<r"r#r/s   zLlavaModel.__init__cC |jSr>)rMget_input_embeddingsr;r"r"r#rS zLlavaModel.get_input_embeddingscC|j|dSr>)rMset_input_embeddingsr;valuer"r"r#rWzLlavaModel.set_input_embeddingsN pixel_valuesr1vision_feature_select_strategyc s|dur|n|jj}|dur|n|jj}|dvr!td|jjdd|D}|j|fddi|t|trNj|}|dkrM|ddd df}nfd d |D}|dkrbd d |D}t j |d d}| |}|S)a Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`): The tensors corresponding to the input images. vision_feature_layer (`Union[int, List[int]]`, *optional*): The index of the layer to select the vision feature. If multiple indices are provided, the vision feature of the corresponding indices will be concatenated to form the vision features. vision_feature_select_strategy (`str`, *optional*): The feature selection strategy used to select the vision feature from the vision backbone. Can be one of `"default"` or `"full"` Returns: image_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`). N)defaultfullz$Unexpected select feature strategy: cSsi|] \}}|dur||qSr>r").0kvr"r"r# sz1LlavaModel.get_image_features..output_hidden_statesTr]rcsg|]}j|qSr")r()r_Z layer_idxZ image_outputsr"r# sz1LlavaModel.get_image_features..cSs g|] }|ddddfqS)Nrr")r_hsr"r"r#res dim) r+r1r\ ValueErroritemsrNr0r2r(rcatrO)r;r[r1r\kwargsZselected_image_featureZhs_poolr?r"rdr#get_image_featuress*   zLlavaModel.get_image_features input_idsattention_mask position_idsr' inputs_embeds use_cacheoutput_attentionsrc return_dictcache_position image_sizesrmreturnc Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|dur.|n|jj}|du|duAr>td|durH||}|dur|j||||d}|durx||t j |jj t j |j dk}|jddjddd}n||jj kd}|||j }||jj k}ts|||kr||jj k}|jd|jd}td|d |||j |j}|||}|jd ||||| | | d | d |}t|j|j|j|j|dur|d Sdd S)Nz:You must specify exactly one of input_ids or inputs_embeds)r[r1r\rw)dtypedevicerrhrrgz6Image features and image tokens do not match: tokens: z , features T) rprqr'rrrsrtrcrurv)last_hidden_stater'r(r)rr")r+rtrcuse_return_dictr1r\rjrSrnrZtensorZimage_token_idlongrzsumZ unsqueezeZ expand_astorZnumelshaperyZmasked_scatterrMrr{r'r(r))r;ror[rprqr'rrr1r\rsrtrcrurvrwrmr?Zspecial_image_maskZn_image_tokensZn_image_featuresoutputsr"r"r#r@sv     zLlavaModel.forward)NN)NNNNNNNNNNNNNN)rrr_checkpoint_conversion_mappingrr/rSrWrr rrr2rstrrnrr LongTensorTensorboolrr rrr@rAr"r"r<r#rLs~ 7       rLc@s eZdZdS)KwargsForCausalLMN)rrrr"r"r"r#rGs rzS The LLAVA model which consists of a vision backbone and a language model. c)seZdZdddddZdgZdeffdd Zd d Zd d Zde j fddZ ddZ e ddZe ddZe ddZee                d6dejdejdeejdeejdeeejd eejd!eeeeefd"eed#eejd$eed%eed&eed'eed(eejd)eeejfd*eejd+eedeee ff$d,d-Z!      d7fd.d/ Z"e#dejd0ed1ed2ej$d(ejd3ef d4d5Z%Z&S)8LlavaForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)z^language_model.modelz ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightr+cs<t|t||_tj|jj|jjdd|_ | dS)NFr,) r.r/rLmodelrr4r7r6 vocab_sizerrPrQr<r"r#r/Xs   z&LlavaForConditionalGeneration.__init__cCrRr>)rrSrTr"r"r#rS^rUz2LlavaForConditionalGeneration.get_input_embeddingscCrVr>)rrWrXr"r"r#rWarZz2LlavaForConditionalGeneration.set_input_embeddingsrxcCs|jSr>rrTr"r"r#get_output_embeddingsdsz3LlavaForConditionalGeneration.get_output_embeddingscCs ||_dSr>r)r;Znew_embeddingsr"r"r#set_output_embeddingsgrUz3LlavaForConditionalGeneration.set_output_embeddingscC|jjSr>)rrMrTr"r"r#rMkz,LlavaForConditionalGeneration.language_modelcCrr>)rrNrTr"r"r#rNorz*LlavaForConditionalGeneration.vision_towercCrr>)rrOrTr"r"r#rOsrz3LlavaForConditionalGeneration.multi_modal_projectorNrror[rprqr'rrr1r\labelsrsrtrcrurvlogits_to_keeprwrmcKs| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|dur.|n|jj}|jd||||||||| | | d||d|}|d}t|trXt | dn|}| |dd|ddf}d}| dur}|j d|| |jj j d|}t|||j|j|j|jdS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, LlavaForConditionalGeneration >>> model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> prompt = "USER: \nWhat's the content of the image? ASSISTANT:" >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=15) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "USER: \nWhat's the content of the image? ASSISTANT: The image features a busy city street with a stop sign prominently displayed" ```NT)ror[rprqr'rrr1r\rsrtrcrurvrwr)r&rr)r%r&r'r(r)rr")r+rtrcr|r1r\rr0r2slicerZ loss_functionr7rr$r'r(r)r)r;ror[rprqr'rrr1r\rrsrtrcrurvrrwrmrr(Z slice_indicesr&r%r"r"r#r@ws\1 z%LlavaForConditionalGeneration.forwardc  s8tj|f|||||d|} |ddkr|| d<| S)N)r'rrrprvrrr[)r.prepare_inputs_for_generation) r;ror'rrr[rprvrrmZ model_inputsr<r"r#rs  z;LlavaForConditionalGeneration.prepare_inputs_for_generationsequence_length target_lengthry batch_sizec KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valueryrzr)Zdiagonal)rzrgr)rirZfinfominr^rzZtriuZarangeZreshapeexpandclonerrZ masked_fill) rprrryrvrrmZ causal_maskZ min_dtypeZ mask_lengthZ padding_maskr"r"r#5_prepare_4d_causal_attention_mask_with_cache_positions,  $ 6  zSLlavaForConditionalGeneration._prepare_4d_causal_attention_mask_with_cache_position)NNNNNNNNNNNNNNrN)NNNNNN)'rrrrZ_tied_weights_keysrr/rSrWrModulerrpropertyrMrNrOrrrrr rrrrr2rrrrrr$r@r staticmethodryrrAr"r"r<r#rJs           fr)rrBrL).r dataclassesrtypingrrrrrZtorch.utils.checkpointrZ activationsr Z generationr Zmodeling_flash_attention_utilsr Zmodeling_outputsr r Zmodeling_utilsrZprocessing_utilsrutilsrrrrrautorZconfiguration_llavarZ get_loggerrloggerrr$rr*rBrLrr__all__r"r"r"r#sF           '" e