o Zh@sddlZddlZddlmZddlmZmZmZmZm Z ddl Z ddl Z ddl mZddlmmZddl mZmZmZddlmZddlmZddlmZdd lmZdd lmZmZm Z dd l!m"Z"m#Z#dd l$m%Z%m&Z&m'Z'm(Z(d dl)m*Z*m+Z+m,Z,e(-e.Z/eGddde%Z0eGddde%Z1eGddde%Z2Gdddej3Z4 dUdej3de j5de j5de j5dee j5de6de6fdd Z7Gd!d"d"ej3Z8Gd#d$d$ej3Z9Gd%d&d&eZ:Gd'd(d(ej3Z;Gd)d*d*ej3Z 1dVd2e j5d3e6d4e6d5e6d6e6d7e j5f d8d9Z?dWdd?ZAd@dAZBGdBdCdCej3ZCe&GdDdEdEe#ZDe&dFdGGdHdIdIeDZEGdJdKdKej3ZFe&dLdGGdMdNdNeDZGe&GdOdPdPeDZHe&dQdGGdRdSdSeDZIgdTZJdS)XN) dataclass)AnyCallableOptionalTupleUnion)BCEWithLogitsLossCrossEntropyLossMSELoss)_calculate_fan_in_and_fan_out)ACT2FN)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) ModelOutputauto_docstringcan_return_tuplelogging) Siglip2ConfigSiglip2TextConfigSiglip2VisionConfigc@jeZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS)Siglip2VisionOutputa Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__r r!rr"r*r*[/var/www/auris/lib/python3.10/site-packages/transformers/models/siglip2/modeling_siglip2.pyr- rc@r)Siglip2TextOutputa Base class for text model's outputs that also contains a pooling of the last hidden states. Args: text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N text_embedsr .r!r") r#r$r%r&r.rr'r(r)r r!rr"r*r*r*r+r-Jr,r-c@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd d ZdS) Siglip2Outputa Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`Siglip2TextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`Siglip2VisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`Siglip2TextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`Siglip2VisionModel`]. Nlosslogits_per_imagelogits_per_textr.rtext_model_outputvision_model_outputreturncstfddDS)Nc3s.|]}|dvr |nt|VqdS))r3r4N)getattrto_tuple).0kselfr*r+ s  z)Siglip2Output.to_tuple..)tuplekeysr:r*r:r+r7s zSiglip2Output.to_tuple)r#r$r%r&r0rr'r(r)r1r2r.rr3rr4rrr7r*r*r*r+r/gs   r/c sbeZdZdeffdd Zedejdejde dejfdd Z d ej dejdejfd d Z Z S) Siglip2VisionEmbeddingsconfigcsnt||_|j|_|j|_tj|j|j|j|jd|_ |j |_ t |j d|_ t |j |j|_dS)N)Z in_featuresZ out_featuresg?)super__init__r@ hidden_size embed_dimZ patch_sizennLinearZ num_channelspatch_embeddingZ num_patchesintposition_embedding_size Embeddingposition_embeddingr;r@ __class__r*r+rBs z Siglip2VisionEmbeddings.__init__positional_embeddingsspatial_shapes max_lengthr5c Cs|jd}|jd}|j}tj|||f|j|d}|dddd}|jjdkr/|tj }t |D];}||\}} t j ||| fddd d } | ||| dd} | |} | ||d || f<| d|||| d f<q3|S) ac Resize positional embeddings to image-specific size and pad to a fixed size. Args: positional_embeddings (`torch.Tensor`): Position embeddings of shape (height, width, embed_dim) spatial_shapes (`torch.LongTensor`): Spatial shapes of shape (batch_size, 2) to resize the positional embeddings to max_length (`int`): Maximum length of the positional embeddings to pad resized positional embeddings to Returns: `torch.Tensor`: Embeddings of shape (batch_size, max_length, embed_dim) r)devicedtypercpuZbilinearFT)sizemodeZ align_cornersZ antialiasN)shaperTr'emptyrSZpermuteZ unsqueezetypetofloat32rangeFZ interpolatereshape transpose) rOrPrQ batch_sizerDZ source_dtypeZresulted_positional_embeddingsiheightwidthZresized_embeddingsr*r*r+resize_positional_embeddingss2       z4Siglip2VisionEmbeddings.resize_positional_embeddings pixel_valuescCsT|jjj}||j|d}|jj|j|jd}|j|||jdd}||}|S)aH Args: pixel_values (`torch.FloatTensor`): Pixel values of shape (batch_size, max_num_patches, num_channels * patch_size * patch_size) spatial_shapes (`List[Tuple[int, int]]`): Spatial shapes of shape (batch_size, 2) to resize the positional embeddings to )rTrRr)rQ) rGweightrTr\rKr`rIrfrY)r;rgrPZ target_dtypeZ patch_embedsrOZresized_positional_embeddings embeddingsr*r*r+forwards   zSiglip2VisionEmbeddings.forward)r#r$r%rrB staticmethodr'Tensor LongTensorrHrfr(rj __classcell__r*r*rMr+r?s $:r?modulequerykeyvalueattention_maskscalingdropoutc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)NrR)dimrT)ptrainingrrU) r'matmulrarE functionalZsoftmaxr]r\rTrvrz contiguous) rprqrrrsrtrurvkwargs attn_weights attn_outputr*r*r+eager_attention_forwards  rc sjeZdZdZdeeefffdd Z  d dej de ej d e e d e ej e ej ffd d Z ZS)Siglip2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperr@cst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).F)rArBr@rCrDnum_attention_heads num_headshead_dim ValueErrorscaleZattention_dropoutrv is_causalrErFk_projv_projq_projout_projrLrMr*r+rB s$   zSiglip2Attention.__init__NFr!rtoutput_attentionsr5c Cs|j\}}}||}||}||} ||||j|jdd}||||j|jdd}| |||j|jdd} t} |j j dkr[|j j dkrU|rUt dnt |j j } | |||| ||j|j|jsjdn|jd\} } | |||} || } |sd} | | fS) z#Input shape: Batch x Time x ChannelrrUeagerZsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.ro)rrurvN)rYrrrviewrrrarr@_attn_implementationloggerZ warning_oncerrrrzrvr`r}r) r;r!rtrrb seq_lengthrDZqueriesr>valuesZattention_interfacerrr*r*r+rjs:        zSiglip2Attention.forward)NF)r#r$r%r&rrrrBr'rlrboolrrjrnr*r*rMr+rsrcs2eZdZfddZdejdejfddZZS) Siglip2MLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) rArBr@r Z hidden_act activation_fnrErFrCZintermediate_sizefc1fc2rLrMr*r+rBPs  zSiglip2MLP.__init__r!r5cCs"||}||}||}|Sr)rrr)r;r!r*r*r+rjWs   zSiglip2MLP.forward)r#r$r%rBr'rlrjrnr*r*rMr+rOs rc sVeZdZdeeefffdd Z d dejdejde e de ej fd d Z ZS) Siglip2EncoderLayerr@csRt|j|_tj|j|jd|_t||_ tj|j|jd|_ t ||_ dS)NZeps) rArBrCrDrE LayerNormlayer_norm_eps layer_norm1r self_attn layer_norm2rmlprLrMr*r+rB_s  zSiglip2EncoderLayer.__init__Fr!rtrr5cCsb|}||}|j|||d\}}||}|}||}||}||}|f}|r/||f7}|S)a= Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r!rtr)rrrr)r;r!rtrresidualroutputsr*r*r+rjgs      zSiglip2EncoderLayer.forward)F)r#r$r%rrrrBr'rlrrrr(rjrnr*r*rMr+r^s rc sZeZdZdZdeffdd Ze   d deej dee dee d e fd d Z Z S) Siglip2Encoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`Siglip2EncoderLayer`]. Args: config: Siglip2Config r@cs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr*)r)r8_r@r*r+ sz+Siglip2Encoder.__init__..F) rArBr@rEZ ModuleListr^Znum_hidden_layerslayersZgradient_checkpointingrLrMrr+rBs   zSiglip2Encoder.__init__Nrtroutput_hidden_statesr5c Cs|dur|n|jj}|dur|n|jj}|rdnd}|rdnd}|}|jD]}|r.||f}||||d} | d}|rB|| df}q%|rJ||f}t|||dS)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr*)rrr)r r!r")r@rrrr) r; inputs_embedsrtrrZencoder_statesZall_attentionsr!Z encoder_layerZ layer_outputsr*r*r+rjs2     zSiglip2Encoder.forwardNNN)r#r$r%r&rrBrrr'rlrrrjrnr*r*rMr+rs rcs`eZdZdeffdd Zee  d dejdej dej de e d e e d e f d d ZZS)Siglip2VisionTransformerr@csrt||_|j}t||_t||_tj ||j d|_ t |ds%dn|j |_|jr1t||_|jdk|_dS)Nrvision_use_headTflash_attention_2)rArBr@rCr?rirencoderrErrpost_layernormhasattrruse_head$Siglip2MultiheadAttentionPoolingHeadheadr_use_flash_attention_2r;r@rDrMr*r+rBs    z!Siglip2VisionTransformer.__init__NrgrtrPrrr5c Cs|dur|n|jj}|dur|n|jj}|||}|dur(|js(t||j}n|}|j||||d}|j} | | } |j rD| | |nd} t | | |j |jdS)z spatial_shapes (`torch.LongTensor` of shape `(batch_size, 2)`): Tensor containing the spatial dimensions (height, width) of the input images. Nrrtrrr pooler_outputr!r")r@rrrirrrTrr rrrrr!r") r;rgrtrPrrr!Zencoder_attention_maskencoder_outputsr rr*r*r+rjs,  z Siglip2VisionTransformer.forwardNN)r#r$r%rrBrrr'r(rlrmrrrrjrnr*r*rMr+rs& rc sXeZdZdeffdd Z   d deejdeejdeejdej fd d Z Z S) Siglip2TextEmbeddingsr@csRt|j}t|j||_t|j||_|j dt |j ddddS)N position_ids)rrRF) persistent) rArBrCrErJZ vocab_sizetoken_embeddingZmax_position_embeddingsrKZregister_bufferr'ZarangeexpandrrMr*r+rBs  zSiglip2TextEmbeddings.__init__N input_idsrrr5cCs|dur |jdn|jd}|jjjd}||kr#td|d||dur2|jddd|f}|dur;||}||}||}|S)NrRrwrzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rYrKrhrrr)r;rrrrZmax_position_embeddingZposition_embeddingsrir*r*r+rj(s"  zSiglip2TextEmbeddings.forwardr) r#r$r%rrBrr'rmr(rlrjrnr*r*rMr+rsrcCsdd}||d|ks||d|krtjddd||||}||||}|d|dd|d|||td|||j||ddS) NcSsdt|tddS)N?@)matherfsqrt)xr*r*r+norm_cdfFsz _trunc_normal_..norm_cdfrUzjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect.) stacklevelrr)minmax) warningswarnuniform_Zerfinv_mul_rradd_Zclamp_)tensormeanstdabrlur*r*r+_trunc_normal_Cs   rrrrrrrrr5cCsNtt|dd|||||WddS1s wYdS)anFills the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}( ext{mean}, ext{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq ext{mean} \leq b`. NOTE: this 'tf' variant behaves closer to Tensorflow / JAX impl where the bounds [a, b] are applied when sampling the normal distribution with mean=0, std=1.0 and the result is subsequently scaled and shifted by the mean and std args. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value rrN)r'no_gradrrr)rrrrrr*r*r+trunc_normal_tf_gs "rfan_innormalc Cst|\}}|dkr |}n|dkr|}n |dkr||d}||}|dkr3t|t|dddS|dkrWt|jt|dWddS1sPwYdS|d krtd |}t|| |WddS1sywYdStd |) Nrfan_outZfan_avgrUtruncated_normalg۶%?rruniformr zinvalid distribution ) r rrrr'rnormal_rr) rrrX distributionrrdenomZvarianceboundr*r*r+variance_scaling_s(   " "rcCt|ddddS)NrrrXrrrr*r*r+ lecun_normal_rcCr)Nrrrrrr*r*r+default_flax_embed_initrrcsreZdZdeffdd Zee     d deej deej deej dee d ee d e f d d Z Z S)Siglip2TextTransformerr@cs\t||_|j}t||_t||_tj ||j d|_ t ||j |_|jdk|_dS)Nrr)rArBr@rCrrirrrErrfinal_layer_normrFZprojection_sizerrrrrMr*r+rBs   zSiglip2TextTransformer.__init__Nrrtrrrr5c Cs|dur|n|jj}|dur|n|jj}|durtd|}|d|d}|j||d}|dur<|jsrrN)/ isinstancer?r@r vision_configrCrEinitrrKrhnprrJrrZxavier_uniform_rrrrZzeros_Zbiasrrrrprobedata attentionZin_proj_weightZ in_proj_bias Siglip2Modelr'logr logit_scaleZfill_ logit_biasZzero_Siglip2ForImageClassification classifierZinitializer_factorrFZConv2drr)r;rpreZlogit_scale_initr*r*r+ _init_weightssX   "          z$Siglip2PreTrainedModel._init_weightsN) r#r$r%r config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_flash_attn_2Z_supports_sdparr*r*r*r+rs rzL The text model from Siglip2 without any head or projection on top. )Z custom_introcseZdZeZdeffdd ZdejfddZddZ e e dd e e jd e e jd e e jd e ede edef ddZZS)Siglip2TextModelr@c"t|t||_|dSr)rArBr text_model post_initrLrMr*r+rB(s   zSiglip2TextModel.__init__r5cC |jjjSrrrirr:r*r*r+get_input_embeddings. z%Siglip2TextModel.get_input_embeddingscCs||jj_dSrr)r;rsr*r*r+set_input_embeddings1sz%Siglip2TextModel.set_input_embeddingsNrrtrrrcC|j|||||dS)a Examples: ```python >>> from transformers import AutoTokenizer, Siglip2TextModel >>> model = Siglip2TextModel.from_pretrained("google/siglip2-base-patch16-224") >>> tokenizer = AutoTokenizer.from_pretrained("google/siglip2-base-patch16-224") >>> # important: make sure to set padding="max_length" as that's how the model was trained >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding="max_length", return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrtrrr)r)r;rrtrrrr*r*r+rj4szSiglip2TextModel.forwardr)r#r$r%rrrBrEModuler r rrrr'rlrrrjrnr*r*rMr+r s2rcsHeZdZdZdeffdd Zd dejdeejdejfd d Z Z S) rzMultihead Attention Pooling.r@csdtttdd|j|_tjj|j|j dd|_ tj |j|j d|_ t||_|j |_dS)NrT)Z batch_firstr)rArBrE Parameterr'randnrCrZMultiheadAttentionrrrr layernormrrrrLrMr*r+rB[s   z-Siglip2MultiheadAttentionPoolingHead.__init__N hidden_statertr5cCs|jd}|j|dd}|dur3|jd|jd}}t||j|}|d|j|d}|d||}|j||||dd}|}||}|| |}|dddfS)NrrrR)Z attn_mask) rYrrepeatrrTrr`rrr)r;rrtrbrZ target_lenZ source_lenrr*r*r+rjds  z,Siglip2MultiheadAttentionPoolingHead.forwardr) r#r$r%r&rrBr'rlrrjrnr*r*rMr+rXs* rzN The vision model from Siglip2 without any head or projection on top. csxeZdZeZdZdeffdd ZdejfddZ e e  dde j d e jd e jd eed eedef d dZZS)Siglip2VisionModelrgr@crr)rArBr vision_modelrrLrMr*r+rBs   zSiglip2VisionModel.__init__r5cCrr)rrirGr:r*r*r+r r z'Siglip2VisionModel.get_input_embeddingsNpixel_attention_maskrPrrcCr )a9 pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. spatial_shapes (`torch.LongTensor` of shape `(batch_size, 2)`): Tensor containing the spatial dimensions (height, width) of the input images. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Siglip2VisionModel >>> model = Siglip2VisionModel.from_pretrained("google/siglip2-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled features ```rgrtrPrr)r)r;rgrrPrrr*r*r+rjs#zSiglip2VisionModel.forwardr)r#r$r%rrmain_input_namerBrErr rrr'r(rlrmrrrrjrnr*r*rMr+rws,rcs@eZdZeZdeffdd Ze     ddeej deej deej dee d ee d ej f d d Z e     dd eej deej deej dee d ee d ej f ddZee         ddeej d eej deej deej deej deej dee dee d ee d efddZZS)rr@cst|t|jtstdt|jdt|jts(tdt|jd|j}|j}t |}t |}|j |_ |j |_ ttd|_ttd|_|dS)NzNconfig.text_config is expected to be of type Siglip2TextConfig but is of type .zRconfig.vision_config is expected to be of type Siglip2VisionConfig but is of type r)rArBr text_configr TypeErrorr[rrr _from_configrrrrErr'rrrr)r;r@rrrrrMr*r+rBs,      zSiglip2Model.__init__Nrrtrrrr5cCF|dur|n|jj}|dur|n|jj}|j|||||d}|j}|S)aM Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`Siglip2TextModel`]. Examples: ```python >>> from transformers import AutoTokenizer, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip2-base-patch16-224") >>> tokenizer = AutoTokenizer.from_pretrained("google/siglip2-base-patch16-224") >>> # important: make sure to set padding="max_length" as that's how the model was trained >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding="max_length", return_tensors="pt") >>> with torch.no_grad(): ... text_features = model.get_text_features(**inputs) ```Nr )r@rrrr)r;rrtrrr text_outputsrr*r*r+get_text_featuresszSiglip2Model.get_text_featuresrgrrPcCr)a pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. spatial_shapes (`torch.LongTensor` of shape `(batch_size, 2)`): Tensor containing the spatial dimensions (height, width) of the input images. Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`Siglip2VisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip2-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... image_features = model.get_image_features(**inputs) ``` Nr)r@rrrr)r;rgrrPrrvision_outputsrr*r*r+get_image_features s(zSiglip2Model.get_image_features return_lossc  CsD|dur|n|jj}| dur| n|jj} |j||||| d} |j||||| d} | j} | j} | | jdddd} | | jdddd} t| | | j }|j | j |j | j }}|||}| }d}|rtj|d|j d }t| d|}tjj||}tj|dd  }|}t|||| | | | d S) a pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. spatial_shapes (`torch.LongTensor` of shape `(batch_size, 2)`): Tensor containing the spatial dimensions (height, width) of the input images. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip2-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["a photo of 2 cats", "a photo of 2 dogs"] >>> # important: we pass `padding=max_length` since the model was trained with this >>> inputs = processor(text=texts, images=image, padding="max_length", return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image >>> probs = torch.sigmoid(logits_per_image) # these are the probabilities >>> print(f"{probs[0][0]:.1%} that image 0 is '{texts[0]}'") 31.9% that image 0 is 'a photo of 2 cats' ``` Nrr rUrRT)ryrxZkeepdimr)rSrx)r0r1r2r.rr3r4)r@rrrrrZnormr'r{tr\rSrrexpeyerWZ ones_likerEr|Z logsigmoidsumrr/)r;rrgrrPrtrr"rrr rrr.r2rrr1r0r&Zm1_diag1ZloglikZnllr*r*r+rjBsR2zSiglip2Model.forwardr) NNNNNNNNN)r#r$r%rrrBrrr'rlrr(rrmr!rr/rjrnr*r*rMr+rs  - 8   rz Siglip2 vision encoder with an image classification head on top (a linear layer on top of the pooled final hidden states of the patch tokens) e.g. for ImageNet. cseZdZdZdeddffdd Zee      ddee j dee j dee j d ee j d ee d ee de fd d ZZS)rrgr@r5NcsZt||j|_t|j}|j|_|jdkr"t|jj |jnt |_ | dS)Nr) rArB num_labelsrrrrrErFrCZIdentityrr)r;r@rrMr*r+rBs  " z&Siglip2ForImageClassification.__init__rrPlabelsrrc Cs|dur|n|jj}|dur|n|jj}|j|||||d}|j}|dur>|d|j} tj|| ddtj| dd}ntj |dd}| |} d} |dur|| j}|jj dur|j dkrfd|j_ n|j dkr||j tjksw|j tjkr|d|j_ nd|j_ |jj dkrt} |j dkr| | |} n+| | |} n%|jj dkrt} | | d |j |d } n|jj dkrt} | | |} t| | |j|jd S) a pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. spatial_shapes (`torch.LongTensor` of shape `(batch_size, 2)`): Tensor containing the spatial dimensions (height, width) of the input images. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, Siglip2ForImageClassification >>> import torch >>> from PIL import Image >>> import requests >>> torch.manual_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # note: we are loading a `Siglip2Model` from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random if seed is not set above. >>> image_processor = AutoImageProcessor.from_pretrained("google/siglip2-base-patch16-224") >>> model = Siglip2ForImageClassification.from_pretrained("google/siglip2-base-patch16-224") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the two classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: LABEL_1 ``` N)rtrPrr).Nrr#Z regressionZsingle_label_classificationZmulti_label_classificationrR)r0logitsr!r")r@rrrr r\rSr'r'rrZ problem_typer(rTlongrHr Zsqueezer rrrr!r") r;rgrrPr)rrrZsequence_outputZ pool_maskr*r0Zloss_fctr*r*r+rjsT/"     "       z%Siglip2ForImageClassification.forward)NNNNNN)r#r$r%rrrBrrrr'rlrmrrrjrnr*r*rMr+rs4r)rrrrr)ro)rorrr)rrr)Krr dataclassesrtypingrrrrrnumpyrr'Ztorch.nnrEZtorch.nn.functionalr|r_rr r Z torch.nn.initr Z activationsr Zmodeling_attn_mask_utilsrZmodeling_layersrZmodeling_outputsrrrZmodeling_utilsrrutilsrrrrZconfiguration_siglip2rrrZ get_loggerr#rrr-r/rr?rlfloatrrrrrrrrrrrrrrrrrrr__all__r*r*r*r+s       $l G0P=(% ?>3;u ~