o Zhh@sddlmZddlZddlmZddlmmZddlmZm Z m Z ddl m Z m Z mZddlmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZGdd d e ZGd d d eZ Gd d d e Z!GdddeZ"GdddeZ#GdddeZ$Gdddej%Z&GdddeZ'GdddeZ(GdddeZ)GdddeZ*GdddeZ+Gd d!d!eZ,Gd"d#d#eZ-gd$Z.dS)%)OptionalN)BCEWithLogitsLossCrossEntropyLossMSELoss) SiglipConfigSiglipTextConfigSiglipVisionConfig) BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputSiglipForImageClassification SiglipModel#SiglipMultiheadAttentionPoolingHead SiglipOutputSiglipPreTrainedModelSiglipTextModelSiglipTextModelOutputSiglipVisionModelSiglipVisionModelOutputSiglipVisionTransformer)_prepare_4d_attention_maskc@ eZdZdS)Siglip2TextConfigN__name__ __module__ __qualname__rrZ/var/www/auris/lib/python3.10/site-packages/transformers/models/siglip2/modular_siglip2.pyr*rcs6eZdZdZ         d fd d ZZS)Siglip2VisionConfigaO This is the configuration class to store the configuration of a [`Siglip2VisionModel`]. It is used to instantiate a Siglip2 vision encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the vision encoder of the Siglip2 [google/siglip2-base-patch16-naflex](https://huggingface.co/google/siglip2-base-patch16-naflex) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 768): Dimensionality of the encoder layers and the pooler layer. intermediate_size (`int`, *optional*, defaults to 3072): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. num_hidden_layers (`int`, *optional*, defaults to 12): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer encoder. num_channels (`int`, *optional*, defaults to 3): Number of channels in the input images. num_patches (`int`, *optional*, defaults to 256): The number of patches in the image with the size of (`patch_size`, `patch_size`). The image is resized to fill maximum of this number of patches, and to preserve the aspect ratio. In case the resulted number of patches is lower, the image is padded in "patch" dimension. patch_size (`int`, *optional*, defaults to 16): The size (resolution) of each patch. hidden_act (`str` or `function`, *optional*, defaults to `"gelu_pytorch_tanh"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` `"quick_gelu"` are supported. layer_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the layer normalization layers. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. Example: ```python >>> from transformers import Siglip2VisionConfig, Siglip2VisionModel >>> # Initializing a Siglip2VisionConfig with google/siglip2-base-patch16-naflex style configuration >>> configuration = Siglip2VisionConfig() >>> # Initializing a Siglip2VisionModel (with random weights) from the google/siglip2-base-patch16-naflex style configuration >>> model = Siglip2VisionModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` rgelu_pytorch_tanhư>c  s tjdi| ||_|`dS)Nr)super__init__ num_patchesZ image_size) self hidden_sizeZintermediate_sizeZnum_hidden_layersnum_attention_heads num_channelsr, patch_sizeZ hidden_actZlayer_norm_epsZattention_dropoutkwargs __class__rrr+aszSiglip2VisionConfig.__init__) r"r#r$r$rr%r&r'r(r))rrr__doc__r+ __classcell__rrr3rr!.s4r!c@r) Siglip2ConfigNrrrrrr7tr r7c@r)Siglip2VisionOutputNrrrrrr8xr r8c@r)Siglip2TextOutputNrrrrrr9|r r9c@r) Siglip2OutputNrrrrrr:r r:c sbeZdZdeffdd Zedejdejde dejfdd Z d ej dejdejfd d Z Z S) Siglip2VisionEmbeddingsconfigcsnt||_|j|_|j|_tj|j|j|j|jd|_ |j |_ t |j d|_ t |j |j|_dS)N)Z in_featuresZ out_featuresg?)r*r+r<r. embed_dimr1nnZLinearr0patch_embeddingr,intposition_embedding_sizeZ Embeddingposition_embeddingr-r<r3rrr+s z Siglip2VisionEmbeddings.__init__positional_embeddingsspatial_shapes max_lengthreturnc Cs|jd}|jd}|j}tj|||f|j|d}|dddd}|jjdkr/|tj }t |D];}||\}} t j ||| fddd d } | ||| dd} | |} | ||d || f<| d|||| d f<q3|S) ac Resize positional embeddings to image-specific size and pad to a fixed size. Args: positional_embeddings (`torch.Tensor`): Position embeddings of shape (height, width, embed_dim) spatial_shapes (`torch.LongTensor`): Spatial shapes of shape (batch_size, 2) to resize the positional embeddings to max_length (`int`): Maximum length of the positional embeddings to pad resized positional embeddings to Returns: `torch.Tensor`: Embeddings of shape (batch_size, max_length, embed_dim) r)devicedtypecpuZbilinearFT)sizemodeZ align_cornersZ antialiasN)shaperJtorchemptyrIZpermuteZ unsqueezetypetoZfloat32rangeFZ interpolatereshapeZ transpose) rDrErF batch_sizer=Z source_dtypeZresulted_positional_embeddingsiheightwidthZresized_embeddingsrrrresize_positional_embeddingss2       z4Siglip2VisionEmbeddings.resize_positional_embeddings pixel_valuescCsT|jjj}||j|d}|jj|j|jd}|j|||jdd}||}|S)aH Args: pixel_values (`torch.FloatTensor`): Pixel values of shape (batch_size, max_num_patches, num_channels * patch_size * patch_size) spatial_shapes (`List[Tuple[int, int]]`): Spatial shapes of shape (batch_size, 2) to resize the positional embeddings to )rJrHrL)rF) r?weightrJrTrBrWrAr\rP)r-r]rEZ target_dtypeZ patch_embedsrDZresized_positional_embeddings embeddingsrrrforwards   zSiglip2VisionEmbeddings.forward)rrrr!r+ staticmethodrQTensor LongTensorr@r\ FloatTensorr`r6rrr3rr;s $:r;c sXeZdZdeffdd Z  d dejdejdejde e d e e d e f d d Z Z S)Siglip2VisionTransformerr<cst|jdk|_dS)NZflash_attention_2)r*r+Z_attn_implementation_use_flash_attention_2rCr3rrr+s z!Siglip2VisionTransformer.__init__Nr]attention_maskrEoutput_attentionsoutput_hidden_statesrGc Cs|dur|n|jj}|dur|n|jj}|||}|dur(|js(t||j}n|}|j||||d}|j} | | } |j rD| | |nd} t | | |j |jdS)z spatial_shapes (`torch.LongTensor` of shape `(batch_size, 2)`): Tensor containing the spatial dimensions (height, width) of the input images. N)Z inputs_embedsrgrhri)last_hidden_state pooler_output hidden_states attentions)r<rhrir_rfrrJencoderrjZpost_layernormZuse_headheadr rlrm) r-r]rgrErhrirlZencoder_attention_maskZencoder_outputsrjrkrrrr`s,   z Siglip2VisionTransformer.forwardNN)rrrr!r+rQrdrbrcrboolr r`r6rrr3rres" rec@r)Siglip2PreTrainedModelNrrrrrrrr rrc@r)Siglip2TextModelNrrrrrrs r rscsDeZdZdeffdd Zd dejdeejdejfdd ZZ S) $Siglip2MultiheadAttentionPoolingHeadr<cst||j|_dSN)r*r+r/ num_headsrCr3rrr+%s  z-Siglip2MultiheadAttentionPoolingHead.__init__N hidden_statergrGcCs|jd}|j|dd}|dur3|jd|jd}}t||j|}|d|j|d}|d||}|j||||dd}|}||}|| |}|dddfS)NrrLrH)Z attn_mask) rPproberepeatrrJrvrWZ attentionZ layernormZmlp)r-rwrgrXrxZ target_lenZ source_lenZresidualrrrr`)s  z,Siglip2MultiheadAttentionPoolingHead.forwardru) rrrr!r+rQrbrr`r6rrr3rrt$s*rtc @sBeZdZ  d dejdejdejdeedeede f dd Z dS) Siglip2VisionModelNr]pixel_attention_maskrErhrirGcCs|j|||||dS)a9 pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. spatial_shapes (`torch.LongTensor` of shape `(batch_size, 2)`): Tensor containing the spatial dimensions (height, width) of the input images. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Siglip2VisionModel >>> model = Siglip2VisionModel.from_pretrained("google/siglip2-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled features ```r]rgrErhri) vision_model)r-r]r{rErhrirrrr`>s!zSiglip2VisionModel.forwardrp) rrrrQrdrbrcrrqr r`rrrrrz<s rzc@seZdZ     ddeejdeejdeejdeedeedejf dd Z         dd eejdeejdeejdeejd eejd eejd eedeedeede fddZ dS) Siglip2ModelNr]r{rErhrirGcCsF|dur|n|jj}|dur|n|jj}|j|||||d}|j}|S)a pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. spatial_shapes (`torch.LongTensor` of shape `(batch_size, 2)`): Tensor containing the spatial dimensions (height, width) of the input images. Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`Siglip2VisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip2-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... image_features = model.get_image_features(**inputs) ``` Nr|)r<rhrir}rk)r-r]r{rErhrivision_outputsZ pooled_outputrrrget_image_featuresjs'zSiglip2Model.get_image_features input_idsrg position_ids return_lossc  CsD|dur|n|jj}| dur| n|jj} |j||||| d} |j||||| d} | j} | j} | | jdddd} | | jdddd} t| | | j }|j | j |j | j }}|||}| }d}|rtj|d|j d }t| d|}tjj||}tj|dd  }|}t|||| | | | d S) a pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. spatial_shapes (`torch.LongTensor` of shape `(batch_size, 2)`): Tensor containing the spatial dimensions (height, width) of the input images. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip2-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["a photo of 2 cats", "a photo of 2 dogs"] >>> # important: we pass `padding=max_length` since the model was trained with this >>> inputs = processor(text=texts, images=image, padding="max_length", return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image >>> probs = torch.sigmoid(logits_per_image) # these are the probabilities >>> print(f"{probs[0][0]:.1%} that image 0 is '{texts[0]}'") 31.9% that image 0 is 'a photo of 2 cats' ``` Nr|)rrgrrhrirKrHT)pdimZkeepdimr)rIr)losslogits_per_imagelogits_per_text text_embeds image_embedsZtext_model_outputZvision_model_output)r<rhrir}Z text_modelrkZnormrQmatmultrTrI logit_scale logit_biasexpeyerNZ ones_liker> functionalZ logsigmoidsummeanr:)r-rr]r{rErgrrrhrirZ text_outputsrrrrrrrrZm1_diag1ZloglikZnllrrrr`sR0zSiglip2Model.forward)NNNNN) NNNNNNNNN) rrrrrQrdrbrcrqrr:r`rrrrr~hsb ;   r~c@s`eZdZ      d deejdeejdeejdeejdeedeedefd d Z dS) Siglip2ForImageClassificationNr]r{rElabelsrhrirGc Cs|dur|n|jj}|dur|n|jj}|j|||||d}|j}|dur>|d|j} tj|| ddtj| dd}ntj |dd}| |} d} |dur|| j}|jj dur|j dkrfd|j_ n|j dkr||j tjksw|j tjkr|d|j_ nd|j_ |jj dkrt} |j dkr| | |} n+| | |} n%|jj dkrt} | | d |j |d } n|jj dkrt} | | |} t| | |j|jd S) a pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. spatial_shapes (`torch.LongTensor` of shape `(batch_size, 2)`): Tensor containing the spatial dimensions (height, width) of the input images. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, Siglip2ForImageClassification >>> import torch >>> from PIL import Image >>> import requests >>> torch.manual_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # note: we are loading a `Siglip2Model` from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random if seed is not set above. >>> image_processor = AutoImageProcessor.from_pretrained("google/siglip2-base-patch16-224") >>> model = Siglip2ForImageClassification.from_pretrained("google/siglip2-base-patch16-224") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the two classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: LABEL_1 ``` N)rgrErhri).NrLrZ regressionZsingle_label_classificationZmulti_label_classificationrH)rlogitsrlrm)r<rhrir}rjrTrIrQrrZ classifierZ problem_typeZ num_labelsrJlongr@rZsqueezerviewrr rlrm) r-r]r{rErrhriZoutputsZsequence_outputZ pool_maskrrZloss_fctrrrr` sT-"     "       z%Siglip2ForImageClassification.forward)NNNNNN) rrrrrQrbrcrqr r`rrrrr s,r)r7rr!r~rrrsrzr)/typingrrQZtorch.nnr>Ztorch.nn.functionalrrVrrrZ/transformers.models.siglip.configuration_sigliprrrZ*transformers.models.siglip.modeling_siglipr r r r r rrrrrrrrZmodeling_attn_mask_utilsrrr!r7r8r9r:Moduler;rerrrsrtrzr~r__all__rrrrs0  < Fe3,$ i