o Zh_r@sxdZddlmZddlmZmZmZmZddlZddl Zddlm Z ddl m Z m Z mZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZddl m!Z!ddl"m#Z#e$e%Z&eGdddeZ'eGdddeZ(Gddde j)Z*eGdddeZ+eddGddde+Z,GdddeeZ-ed dGd!d"d"e+eZ.gd#Z/dS)$zPyTorch PaliGemmamodel.) dataclass)ListOptionalTupleUnionN)nn)Cache HybridCache StaticCache)GenerationMixin)FlashAttentionKwargs)BaseModelOutputWithPast)PreTrainedModel)Unpack) LossKwargs ModelOutputauto_docstringcan_return_tupleis_torchdynamo_compilinglogging) AutoModel)PaliGemmaConfigc@s$eZdZUdZdZeejed<dS)PaligemmaModelOutputWithPasta Base class for Paligemma outputs, with hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__r$r$_/var/www/auris/lib/python3.10/site-packages/transformers/models/paligemma/modeling_paligemma.pyr&s rc@seZdZUdZdZeejed<dZ eejed<dZ ee e eje fed<dZeeejed<dZeeejed<dZeejed<dS) PaliGemmaCausalLMOutputWithPasta Base class for PaliGemma causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)rrrr r'rr!r"r#r(r)rrr r*rr+rr$r$r$r%r&Gs r&cs*eZdZdeffdd ZddZZS)PaliGemmaMultiModalProjectorconfigcs(ttj|jj|jjdd|_dS)NTbias)super__init__rLinear vision_config hidden_sizeZprojection_dimlinearselfr- __class__r$r%r1ps z%PaliGemmaMultiModalProjector.__init__cCs||}|SN)r5)r7image_featuresr*r$r$r%forwardts z$PaliGemmaMultiModalProjector.forward)rrrrr1r< __classcell__r$r$r8r%r,osr,c@sBeZdZeZdZdZdgZdZdZ dZ dZ dZ dZ dZddZdS)PaliGemmaPreTrainedModelTr,r)cCsVt|jd|jj}t|tjr'|jjj d|d|j dur)|j j dSdSdS)Ninitializer_range)meanstd) getattrr-Zget_text_configr@ isinstancerr2weightdataZnormal_r/Zzero_)r7modulerCr$r$r% _init_weightss  z&PaliGemmaPreTrainedModel._init_weightsN)rrrrZ config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_supports_cache_classZ_supports_quantized_cacheZ_supports_static_cacheZ_supports_flash_attn_2Z_supports_sdpaZ_supports_attention_backendrIr$r$r$r%r>zs r>z{ The Base Paligemma model which consists of a vision backbone and a language model withou language modeling head., )Z custom_introc#s(eZdZddiZdeffdd ZddZdd Z d!d ee fd d Z de j fddZ ee d"de jde j dee jdee jdeeee j efdee jdee jdee j dee jdee dee dee dee deedeeeffdd ZZS)#PaliGemmaModelzlanguage_model.modellanguage_modelr-csjt|tj|jd|_t||_|jj |_ tj|jd}||_ |j j dur,|j j nd|_ | dS)N)r-)r0r1rZ from_configr3 vision_towerr,multi_modal_projector text_config vocab_sizerKr-Z pad_token_id post_init)r7r-rKr8r$r%r1s    zPaliGemmaModel.__init__cC |jSr:)rKget_input_embeddingsr7r$r$r%rS z#PaliGemmaModel.get_input_embeddingscC|j|dSr:)rKset_input_embeddingsr7valuer$r$r%rWz#PaliGemmaModel.set_input_embeddingsN is_trainingcCs|jjjdkr|durd|vr|SdS|dur|n|j}t|t}t|jj }|dur.|}|j dd\} } |r>| } nt|t rH| } nt|tj rS|j dn|d| d} |durg|dkrg|Stj| | f||j|jd} | dkr|rtj| dd } n d| ddd| f<| tj| |jd |ddk9} | ddddddf| ddd} |dur>| } |j d} |r|durtd | ddddddd| f|ddddddf| jdkd| ddddddd| f<| ddddddd| f|ddddddf| j}|dk}| ddddddd| f||| ddddddd| f<| S) NZflash_attention_2rArrLrrZ fill_valuedtypedeviceZdiagonalr_z/Token type ids must be provided during training)r-rOZ_attn_implementationZtrainingrEr r!finfor^minshapeZget_max_cache_shaper Tensordimfullr_triuarangereshapeexpandclone ValueError masked_fillto)r7attention_masktoken_type_idsr)cache_position input_tensorr[Zusing_static_cache min_dtypeZinputs_lead_dimsequence_length target_length causal_mask mask_length padding_maskr$r$r%_update_causal_masksX        $   $ @  z"PaliGemmaModel._update_causal_mask pixel_valuescCs0||}|j}||}||jjjd}|S)a Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. Returns: image_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`). g?)rMlast_hidden_staterNr-rOr4)r7r{Z image_outputsZselected_image_featurer;r$r$r%get_image_featuress  z!PaliGemmaModel.get_image_features input_idsrp position_idsr)rqrr inputs_embedslabels use_cacheoutput_attentionsoutput_hidden_states return_dictkwargsreturnc Ks:|du|duAr td| dur| n|jj} | dur| n|jj} | dur&| n|jj} |duo1| du}|durL|jj|jkrL||jjk}|}d||<n|}|durX||}|durt|durd| nd}t j |||j d|j d}|dur|dd}|dur||}|dur||t j|jjt j|j dk}n||jjkd}|||j }ts|||kr|jddjddd}td |d |j d|j dd ||j |j}|||}|||||||}|jd||||| | | d |d |}t|j|j|j|j|dur|dSddS)i labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, PaliGemmaForConditionalGeneration >>> model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma2-3b-mix-224") >>> processor = AutoProcessor.from_pretrained("google/paligemma2-3b-mix-224") >>> prompt = "Where is the cat standing?" >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs,) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Where is the cat standing?\nsnow" ```Nz:You must specify exactly one of input_ids or inputs_embedsrrra)r^r_rL)rfzVNumber of images does not match number of special image tokens in the input text. Got z image tokens in the text but z tokens from image embeddings.T) rprr)rrrrrrr)r|r)r*r+rr$) rmr-rruse_return_dictZimage_token_idrPrlrSZget_seq_lengthr!rirdr_Z unsqueezer}ZtensorlongZ expand_asrorZnumelsumr^Zmasked_scatterrzrKrr|r)r*r+)r7r~r{rprr)rqrrrrrrrrrr[Zspecial_image_maskZ llm_input_idsZpast_seen_tokensr;Zimage_tokens_in_textrwoutputsr$r$r%r<s/         zPaliGemmaModel.forward)NNNNN) NNNNNNNNNNNNN)rrr_checkpoint_conversion_mappingrr1rSrWrboolrzr!r"r}rr LongTensorrerrr rr rrr<r=r$r$r8r%rJsx  D      rJc@s eZdZdS)KwargsForCausalLMN)rrrr$r$r$r%rs rz| The Base Paligemma model which consists of a vision backbone and a language model without language modeling head., c%seZdZdddddZdgZdeffdd Zd d Zd d ZddZ ddZ e ddZ e ddZ e ddZee              d5dejdejdeejdeejdeeeejefdeejd eejd!eejd"eejd#eed$eed%eed&eed'eeejfd(eed)eeeff d*d+Z        ,  d6fd-d. Z e!dejd/ed0ed1ej"d ejd2ef d3d4Z#Z$S)7!PaliGemmaForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)z^language_model.modelz ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightr-cs<t|t||_tj|jj|jjdd|_ | dS)NFr.) r0r1rJmodelrr2rOr4rPrrQr6r8r$r%r1s   z*PaliGemmaForConditionalGeneration.__init__cCrRr:)rrSrTr$r$r%rSrUz6PaliGemmaForConditionalGeneration.get_input_embeddingscCrVr:)rrWrXr$r$r%rWrZz6PaliGemmaForConditionalGeneration.set_input_embeddingscCs|jSr:rrTr$r$r%get_output_embeddingssz7PaliGemmaForConditionalGeneration.get_output_embeddingscCs ||_dSr:r)r7Znew_embeddingsr$r$r%set_output_embeddingsrUz7PaliGemmaForConditionalGeneration.set_output_embeddingscC|jjSr:)rrKrTr$r$r%rKz0PaliGemmaForConditionalGeneration.language_modelcCrr:)rrMrTr$r$r%rMrz.PaliGemmaForConditionalGeneration.vision_towercCrr:)rrNrTr$r$r%rNrz7PaliGemmaForConditionalGeneration.multi_modal_projectorNrr~r{rprr)rqrrrrrrrrlogits_to_keeprrcKs| dur| n|jj} | dur| n|jj} | dur| n|jj} |jd|||||||| | | | d|d |}|d}t|trCt| dn|}||dd|ddf}d}| durh|j d|| |jj j d|}t |||j |j|j|jdS)rNT) r~r{rqrprr)rrrrrrrrr)r(rrP)r'r(r)r*r+rr$)r-rrrrrEintslicerZ loss_functionrOrPr&r)r*r+r)r7r~r{rprr)rqrrrrrrrrrrrr*Z slice_indicesr(r'r$r$r%r<sN/ z)PaliGemmaForConditionalGeneration.forwardTc  stj|f|||||| | |d| } | ddur"| dd7<|ddkr,|| d<|duo3| du}|ddkrVt|trV|durE|n|}|j||||||}|| d<| S)N)r)rrprrrrrrqrrrr{rp)r0prepare_inputs_for_generationgetrEr rrz)r7r~r)rrrrr{rprqrrrrZ model_inputsr[rsrwr8r$r%r s4   z?PaliGemmaForConditionalGeneration.prepare_inputs_for_generationrurvr^ batch_sizec KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr\r]rr`rarLr)rfr!rbrcrgr_rhrirjrkrlrdrorn) rprurvr^rrrrrwrtrxryr$r$r%5_prepare_4d_causal_attention_mask_with_cache_position8s,  $ 6  zWPaliGemmaForConditionalGeneration._prepare_4d_causal_attention_mask_with_cache_position)NNNNNNNNNNNNNr) NNNNNNNTNN)%rrrrZ_tied_weights_keysrr1rSrWrrpropertyrKrMrNrrr!rr"rrerrr rrrrrr&r<r staticmethodr^rr=r$r$r8r%rs          [.r)rr>rJ)0r dataclassesrtypingrrrrr!Ztorch.utils.checkpointrZ cache_utilsr r r Z generationr Zmodeling_flash_attention_utilsr Zmodeling_outputsrZmodeling_utilsrZprocessing_utilsrutilsrrrrrrautorZconfiguration_paligemmarZ get_loggerrloggerrr&Moduler,r>rJrr__all__r$r$r$r%sF            ' i j