o ZhO1@sddlmZmZmZmZddlZddlmZddlmZm Z m Z m Z m Z ddl mZddlmZmZmZdd lmZeeZGd d d e ZGd d d eZGdddejZGddde ZGddde ZGddde ZgdZdS))ListOptionalTupleUnionN)nn)LlavaCausalLMOutputWithPastLlavaForConditionalGeneration LlavaModelLlavaModelOutputWithPastLlavaPreTrainedModel)ACT2FN)auto_docstringis_torchdynamo_compilinglogging)VipLlavaConfigc@ eZdZdS)VipLlavaModelOutputWithPastN__name__ __module__ __qualname__rr\/var/www/auris/lib/python3.10/site-packages/transformers/models/vipllava/modular_vipllava.pyr%rc@r)VipLlavaCausalLMOutputWithPastNrrrrrr)rrcs*eZdZdeffdd ZddZZS)VipLlavaMultiModalProjectorconfigcstt|jtr dnt|j}tj||jj |j d|_ tj ||jj |j j dd|_t|j|_tj |j j |j j dd|_dS)Nr)ZepsT)Zbias)super__init__ isinstancevision_feature_layersintlenrZ LayerNormZ vision_configZ hidden_sizeZprojector_layernorm_epsprojector_layernormZLinear text_configlinear_1r Zprojector_hidden_actactlinear_2)selfrZnum_feature_layers __class__rrr .s   z$VipLlavaMultiModalProjector.__init__cCs,||}||}||}||}|S)N)r%r'r(r))r* hidden_statesrrrforward=s    z#VipLlavaMultiModalProjector.forward)rrrrr r. __classcell__rrr+rr-src@r)VipLlavaPreTrainedModelNrrrrrr0Err0c@seZdZdejdeeeeffddZe            ddej dejde ej de ej d e eejd e ejde eeeefd e e d e e d e e de e de ej deeeffddZdS) VipLlavaModel pixel_valuesr"csb|j|ddt|trj|ddddf}nfdd|D}tj|dd}||}|S) aW Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layers (`Union[int, List[int]]`): The vision feature layer, or the list of indexes of the layers to select the vision feature. Returns: image_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`). T)output_hidden_statesNrcs&g|]}j|ddddfqS)Nr)r-).0indexZ image_outputsrr _s&z4VipLlavaModel.get_image_features..)dim)Z vision_towerr!r#r-torchcatZmulti_modal_projector)r*r2r"image_featuresrr6rget_image_featuresJs  z VipLlavaModel.get_image_featuresN input_idsattention_mask position_idspast_key_values inputs_embeds use_cacheoutput_attentionsr3 return_dictcache_positionreturnc  Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|du|duAr4td|dur@|dur@td|durJ||}|dur|j||d}||jjk d}| | |j }t s|||kr||jjk}|jd|jd}td|d || |j |j}|||}|jd |||||| | d | d | }t|j|j|j|j|dur|ndd }| r|S|S)z vision_feature_layers (`Union[int, List[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. Nz:You must specify exactly one of input_ids or inputs_embedszdYou cannot specify both pixel_values and inputs_embeds at the same time, and must specify either one)r2r"r8rrz6Image features and image tokens do not match: tokens: z , features T) r?r@rArBrCrDr3rErF)last_hidden_staterAr- attentionsimage_hidden_statesr)rrDr3use_return_dictr" ValueErrorZget_input_embeddingsr=Zimage_token_idZ unsqueezeZ expand_astoZdevicerZnumelsumshapeZdtypeZmasked_scatterlanguage_modelrrHrAr-rIZto_tuple)r*r>r2r?r@rArBr"rCrDr3rErF lm_kwargsr<Zspecial_image_maskZn_image_tokensZn_image_featuresoutputsoutputrrrr.dsb    zVipLlavaModel.forward) NNNNNNNNNNNN)rrrr: FloatTensorrr#rr=r LongTensorrTensorboolrrr.rrrrr1IsT        r1c!@seZdZ              ddejdejdeejdeejdeeejdeejd ee e ee fd eejd ee d ee d ee dee deejde e ejfde e e ffddZdS) VipLlavaForConditionalGenerationNrr>r2r?r@rArBr"labelsrCrDr3rErFlogits_to_keeprGcKs| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|jd||||||| || | d| d |}|d}t|trLt| dn|}| |dd|ddf}d}|durm|j |||jj j d}t |||j|j|j|jdS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. vision_feature_layers (`Union[int, List[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. Example: ```python >>> import torch >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, VipLlavaForConditionalGeneration >>> model = VipLlavaForConditionalGeneration.from_pretrained("llava-hf/vip-llava-7b-hf", device_map="auto", torch_dtype=torch.float16) >>> processor = AutoProcessor.from_pretrained("llava-hf/vip-llava-7b-hf") >>> prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: \n{}###Assistant:" >>> question = "Can you please describe this image?" >>> prompt = prompt.format(question) >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=text, images=image, return_tensors="pt").to(0, torch.float16) >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=20) >>> processor.decode(generate_ids[0][len(inputs["input_ids"][0]):], skip_special_tokens=True) The image features a brown and white cat sitting on a green surface, with a red ball in its ```NT) r>r2r?r@rArBrCr"rDr3rErFr)logitsrY vocab_size)lossr[rAr-rIrJr)rrDr3rKr"modelr!r#sliceZlm_headZ loss_functionr&r\rrAr-rIrJ)r*r>r2r?r@rArBr"rYrCrDr3rErFrZrQrRr-Z slice_indicesr[r]rrrr.sH4 z(VipLlavaForConditionalGeneration.forward)NNNNNNNNNNNNNr)rrrr:rUrTrrVrrr#rWrrr.rrrrrXs\        rX)r1rXr0) typingrrrrr:rZ(transformers.models.llava.modeling_llavarrr r r Z activationsr utilsrrrZconfiguration_vipllavarZ get_loggerrloggerrrModulerr0r1rX__all__rrrrs    l a