o Zh@sdZddlZddlZddlmZddlmZmZmZm Z m Z ddl Z ddl Z ddlZ ddl mZddlmZmZmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZdd l m!Z!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,e'-e.Z/ddZ0 dWde j1de2de2de2de2de j1f ddZ3dXd!d"Z4d#d$Z5d%d&Z6eGd'd(d(e$Z7eGd)d*d*e$Z8eGd+d,d,e$Z9Gd-d.d.ej:Z;Gd/d0d0ej:Z< dYd1ej:d2e j1d3e j1d4e j1d5ee j1d6e2d7e2fd8d9Z=Gd:d;d;ej:Z>Gdd?d?eZ@e%Gd@dAdAe"ZAGdBdCdCej:ZBGdDdEdEej:ZCe%dFdGGdHdIdIeAZDGdJdKdKej:ZEGdLdMdMej:ZFe%dNdGGdOdPdPeAZGe%GdQdRdReAZHe%dSdGGdTdUdUeAZIgdVZJdS)ZzPyTorch Siglip model.N) dataclass)AnyCallableOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)_calculate_fan_in_and_fan_out)ACT2FN)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) ModelOutputauto_docstringcan_return_tuplelogging torch_int) SiglipConfigSiglipTextConfigSiglipVisionConfigcCsdd}||d|ks||d|krtjddd||||}||||}|d|dd|d|||td|||j||ddS) NcSsdt|tddS)N?@)matherfsqrt)xr%Y/var/www/auris/lib/python3.10/site-packages/transformers/models/siglip/modeling_siglip.pynorm_cdf,sz _trunc_normal_..norm_cdfzjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect.) stacklevelrr )minmax) warningswarnuniform_Zerfinv_mul_r!r#add_Zclamp_)tensormeanstdabr'lur%r%r&_trunc_normal_)s   r8rr r1r2r3r4r5returncCsNtt|dd|||||WddS1s wYdS)anFills the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}( ext{mean}, ext{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq ext{mean} \leq b`. NOTE: this 'tf' variant behaves closer to Tensorflow / JAX impl where the bounds [a, b] are applied when sampling the normal distribution with mean=0, std=1.0 and the result is subsequently scaled and shifted by the mean and std args. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value rrN)torchno_gradr8r/r0)r1r2r3r4r5r%r%r&trunc_normal_tf_Ms "r>fan_innormalc Cst|\}}|dkr |}n|dkr|}n |dkr||d}||}|dkr3t|t|dddS|dkrWt|jt|dWddS1sPwYdS|d krtd |}t|| |WddS1sywYdStd |) Nr?fan_outZfan_avgr(truncated_normalg۶%?r3r@uniformr zinvalid distribution ) r r>r!r#r<r=normal_r. ValueError) r1scalemode distributionr?rAdenomZvarianceboundr%r%r&variance_scaling_gs(   " "rLcCt|ddddS)Nr?rBrHrIrLr1r%r%r& lecun_normal_rQcCrM)Nr?r@rNrOrPr%r%r&default_flax_embed_initrRrSc@jeZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS)SiglipVisionModelOutputa Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__rVrr< FloatTensor__annotations__rWrXrrYr%r%r%r&rU rUc@rT)SiglipTextModelOutputa Base class for text model's outputs that also contains a pooling of the last hidden states. Args: text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N text_embedsrW.rXrY) rZr[r\r]rbrr<r^r_rWrXrrYr%r%r%r&rar`rac@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd d ZdS) SiglipOutputa Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`SiglipTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`SiglipVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`SiglipTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`SiglipVisionModel`]. Nlosslogits_per_imagelogits_per_textrbrVtext_model_outputvision_model_outputr;cstfddDS)Nc3s.|]}|dvr |nt|VqdS))rgrhN)getattrto_tuple).0kselfr%r& s  z(SiglipOutput.to_tuple..)tuplekeysrmr%rmr&rjs zSiglipOutput.to_tuple)rZr[r\r]rdrr<r^r_rerfrbrVrgrrhrrrjr%r%r%r&rcs   rccsXeZdZdeffdd Zdejdededejfdd Zdd ej dejfd d Z Z S)SiglipVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tj|j |j|j|jdd|_ |j|jd|_ |j |_ t |j |j|_|jdt|j ddddS)NZvalid)Z in_channelsZ out_channelsZ kernel_sizeZstridepaddingr( position_idsrF persistent)super__init__rs hidden_size embed_dimZ image_size patch_sizerConv2dZ num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferr<arangeexpandrnrs __class__r%r&r{s "zSiglipVisionEmbeddings.__init__ embeddingsheightwidthr;c Cs|jd}|jjjd}tjs||kr||kr||jS|jjd}|jd}||j}||j} t |d} | d| | |}| dddd}t j j||| fddd }| dddddd|}|S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing and no class embeddings. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrrwg?r r(ZbicubicF)sizerHZ align_corners)shaperweightr<Zjit is_tracingruZ unsqueezer~rreshapeZpermuter functionalZ interpolateview) rnrrrrrZpatch_pos_embeddimZ new_heightZ new_widthZsqrt_num_positionsr%r%r&interpolate_pos_encodings&      z/SiglipVisionEmbeddings.interpolate_pos_encodingF pixel_valuesc Csj|j\}}}}|jjj}||j|d}|ddd}|r+|||||}|S|||j }|S)N)dtyper(r) rrrrtoflatten transposerrru) rnrr_rrZ target_dtypeZ patch_embedsrr%r%r&forward%s zSiglipVisionEmbeddings.forwardF) rZr[r\rr{r<Tensorintrr^r __classcell__r%r%rr&rrs &rrc sXeZdZdeffdd Z   d deejdeejdeejdej fd d Z Z S) SiglipTextEmbeddingsrscsRt|j}t|j||_t|j||_|j dt |j ddddS)NrurvFrx) rzr{r|rrZ vocab_sizetoken_embeddingZmax_position_embeddingsrrr<rrrnrsr}rr%r&r{4s  zSiglipTextEmbeddings.__init__N input_idsru inputs_embedsr;cCs|dur |jdn|jd}|jjjd}||kr#td|d||dur2|jddd|f}|dur;||}||}||}|S)NrwrzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rrrrFrur)rnrrur seq_lengthZmax_position_embeddingZposition_embeddingsrr%r%r&r@s"  zSiglipTextEmbeddings.forwardNNN) rZr[r\rr{rr< LongTensorr^rrrr%r%rr&r3srmodulequerykeyvalueattention_maskscalingdropoutc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nrwr)rr)ptrainingrr() r<matmulrrrZsoftmaxZfloat32rrrr contiguous) rrrrrrrkwargs attn_weights attn_outputr%r%r&eager_attention_forward[s  rc sjeZdZdZdeeefffdd Z  d dej de ej d e e d e ej e ej ffd d Z ZS)SiglipAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrscst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).F)rzr{rsr|r}num_attention_heads num_headshead_dimrFrGZattention_dropoutr is_causalrLineark_projv_projq_projout_projrrr%r&r{us$   zSiglipAttention.__init__NFrXroutput_attentionsr;c Cs|j\}}}||}||}||} ||||j|jdd}||||j|jdd}| |||j|jdd} t} |j j dkr[|j j dkrU|rUt dnt |j j } | |||| ||j|j|jsjdn|jd\} } | |||} || } |sd} | | fS) z#Input shape: Batch x Time x Channelrr(eagerZsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r9)rrrN)rrrrrrrrrrs_attn_implementationloggerZ warning_oncerrrGrrrrr) rnrXrr batch_sizerr}ZqueriesrqvaluesZattention_interfacerrr%r%r&rs:        zSiglipAttention.forward)NF)rZr[r\r]rrrr{r<rrboolrrrr%r%rr&rrsrcs2eZdZfddZdejdejfddZZS) SiglipMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) rzr{rsrZ hidden_act activation_fnrrr|Zintermediate_sizefc1fc2rrr%r&r{s  zSiglipMLP.__init__rXr;cCs"||}||}||}|Sr)rrr)rnrXr%r%r&rs   zSiglipMLP.forward)rZr[r\r{r<rrrr%r%rr&rs rc sVeZdZdeeefffdd Z d dejdejde e de ej fd d Z ZS) SiglipEncoderLayerrscsRt|j|_tj|j|jd|_t||_ tj|j|jd|_ t ||_ dS)NZeps) rzr{r|r}r LayerNormlayer_norm_eps layer_norm1r self_attn layer_norm2rmlprrr%r&r{s  zSiglipEncoderLayer.__init__FrXrrr;cCsb|}||}|j|||d\}}||}|}||}||}||}|f}|r/||f7}|S)a= Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rXrr)rrrr)rnrXrrresidualroutputsr%r%r&rs      zSiglipEncoderLayer.forwardr)rZr[r\rrrr{r<rrrrr^rrr%r%rr&rs rc@s0eZdZeZdZdZgdZdZdZ ddZ dS)SiglipPreTrainedModelZsiglipT)rrrrr#SiglipMultiheadAttentionPoolingHeadcCsft|tr%t|jtr|jjjn|jj}tjj|j j dt |ddSt|tj r2t|j dSt|trytj|jj tj|jj tj|jj tj|jj tj|jjtj|jjtj|jjtj|jjdSt|trtj|jj tj|jj tjj|jjddtjj|jjdddSt|trtj|jjtj|jjjtj|jjjdSt|t rt!"t!#d}|j$j%||j&j'dSt|t(rtjj|j)j |jjjd|jj*ddSt|tj+tj,frt-|j |jdurtj|jdSdSt|tj.r1|jj'|j j%ddSdS)zInitialize the weightsrrCgư>rrN)/ isinstancerrrsr vision_configr|rinitrErrnpr#rrSrZxavier_uniform_rrrrZzeros_Zbiasrrrrprobedata attentionZin_proj_weightZ in_proj_bias SiglipModelr<logr1 logit_scaleZfill_ logit_biasZzero_SiglipForImageClassification classifierZinitializer_factorrrrQr)rnrrZlogit_scale_initr%r%r& _init_weights sX   "          z#SiglipPreTrainedModel._init_weightsN) rZr[r\r config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_flash_attn_2Z_supports_sdparr%r%r%r&rs rc sZeZdZdZdeffdd Ze   d deej dee dee d e fd d Z Z S) SiglipEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`SiglipEncoderLayer`]. Args: config: SiglipConfig rscs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr%)r)rkrrsr%r& Esz*SiglipEncoder.__init__..F) rzr{rsrZ ModuleListrangeZnum_hidden_layerslayersZgradient_checkpointingrrrr&r{Bs   zSiglipEncoder.__init__Nrroutput_hidden_statesr;c Cs|dur|n|jj}|dur|n|jj}|rdnd}|rdnd}|}|jD]}|r.||f}||||d} | d}|rB|| df}q%|rJ||f}t|||dS)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr%)rrr)rWrXrY)rsrrrr) rnrrrrZencoder_statesZall_attentionsrXZ encoder_layerZ layer_outputsr%r%r&rIs2     zSiglipEncoder.forwardr)rZr[r\r]rr{rrr<rrrrrr%r%rr&r9s rcsreZdZdeffdd Zee     d deej deej deej dee d ee d e f d d Z Z S)SiglipTextTransformerrscs\t||_|j}t||_t||_tj ||j d|_ t ||j |_|jdk|_dS)NrZflash_attention_2)rzr{rsr|rrrencoderrrrfinal_layer_normrZprojection_sizeheadr_use_flash_attention_2rrr%r&r{s   zSiglipTextTransformer.__init__Nrrrurrr;c Cs|dur|n|jj}|dur|n|jj}|durtd|}|d|d}|j||d}|dur<|js>> from transformers import AutoTokenizer, SiglipTextModel >>> model = SiglipTextModel.from_pretrained("google/siglip-base-patch16-224") >>> tokenizer = AutoTokenizer.from_pretrained("google/siglip-base-patch16-224") >>> # important: make sure to set padding="max_length" as that's how the model was trained >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding="max_length", return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrurr)r)rnrrrurrr%r%r&rszSiglipTextModel.forwardr)rZr[r\rrr{rModulerrrrrr<rrrrrr%r%rr&rs2rc sXeZdZdeffdd Zee   d deedeedeed e fd d Z Z S) SiglipVisionTransformerrscsjt||_|j}t||_t||_tj ||j d|_ t |ds%dn|j |_|jr3t||_dSdS)Nrvision_use_headT)rzr{rsr|rrrrrrrrpost_layernormhasattrruse_headrrrrr%r&r{s   z SiglipVisionTransformer.__init__NFrrrr;c Cs~|dur|n|jj}|dur|n|jj}|j||d}|j|||d}|j}||}|jr3||nd}t |||j |j dS)N)r)rrrr) rsrrrrrWr r rrrXrY) rnrrrrrXrrWrr%r%r&r s$  zSiglipVisionTransformer.forwardNNF) rZr[r\rr{rrrrrrrr%r%rr&rs  rcs.eZdZdZdeffdd ZddZZS)rzMultihead Attention Pooling.rscs\tttdd|j|_tjj|j|j dd|_ tj |j|j d|_ t||_dS)NrT)Z batch_firstr)rzr{r Parameterr<randnr|rZMultiheadAttentionrrrr layernormrrrrr%r&r{3s z,SiglipMultiheadAttentionPoolingHead.__init__cCsX|jd}|j|dd}||||d}|}||}|||}|dddfS)Nrr)rrrepeatrrr)rnZ hidden_staterrrr%r%r&r;s  z+SiglipMultiheadAttentionPoolingHead.forward)rZr[r\r]rr{rrr%r%rr&r0srzM The vision model from SigLIP without any head or projection on top. c sleZdZeZdZdeffdd ZdejfddZ e e   dd e e d e e d e defd dZZS)SiglipVisionModelrrscrr)rzr{r vision_modelrrrr%r&r{Qs   zSiglipVisionModel.__init__r;cCrr)rrrrmr%r%r&rYrz&SiglipVisionModel.get_input_embeddingsNFrrrcCs|j||||dS)a Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, SiglipVisionModel >>> model = SiglipVisionModel.from_pretrained("google/siglip-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled features ```rrrr)r)rnrrrrr%r%r&r\s zSiglipVisionModel.forwardr )rZr[r\rrmain_input_namer{rrrrrrrrrrr%r%rr&rHs&rcseZdZeZdeffdd Ze     ddeej deej deej dee d ee d ej f d d Z e    ddeej dee d ee de d ej f ddZ ee        ddeejdeej deej deejdee dee d ee de d efddZZS)rrscst|t|jtstdt|jdt|jts(tdt|jd|j}|j}t |}t |}|j |_ |j |_ ttd|_ttd|_|dS)NzMconfig.text_config is expected to be of type SiglipTextConfig but is of type .zQconfig.vision_config is expected to be of type SiglipVisionConfig but is of type r)rzr{r text_configr TypeErrortyperrr _from_configrrrrr r<rrrr)rnrsrrrrrr%r&r{s,      zSiglipModel.__init__Nrrrurrr;cCsF|dur|n|jj}|dur|n|jj}|j|||||d}|j}|S)aJ Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`SiglipTextModel`]. Examples: ```python >>> from transformers import AutoTokenizer, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip-base-patch16-224") >>> tokenizer = AutoTokenizer.from_pretrained("google/siglip-base-patch16-224") >>> # important: make sure to set padding="max_length" as that's how the model was trained >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding="max_length", return_tensors="pt") >>> with torch.no_grad(): ... text_features = model.get_text_features(**inputs) ```Nr)rsrrrr)rnrrrurr text_outputsrr%r%r&get_text_featuresszSiglipModel.get_text_featuresFrrcCsD|dur|n|jj}|dur|n|jj}|j||||d}|j}|S)a Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`SiglipVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... image_features = model.get_image_features(**inputs) ```Nr)rsrrrr)rnrrrrvision_outputsrr%r%r&get_image_featuress!zSiglipModel.get_image_features return_lossc  CsB|dur|n|jj}|dur|n|jj}|j||||d} |j|||||d} | j} | j} | | jdddd} | | jdddd} t| | | j } |j | j |j | j }}| ||} | }d}|rtj| d| j d }t|  d|}tjj|| }tj|dd  }|}t||| | | | | d S) a return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["a photo of 2 cats", "a photo of 2 dogs"] >>> # important: we pass `padding=max_length` since the model was trained with this >>> inputs = processor(text=texts, images=image, padding="max_length", return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image >>> probs = torch.sigmoid(logits_per_image) # these are the probabilities >>> print(f"{probs[0][0]:.1%} that image 0 is '{texts[0]}'") 31.9% that image 0 is 'a photo of 2 cats' ```Nrrr(rwT)rrZkeepdimr)devicer)rdrerfrbrVrgrh)rsrrrrrZnormr<rtrrrrexpeyerZ ones_likerrZ logsigmoidsumr2rc)rnrrrrurrrrrrrVrbrfrrrerdr#Zm1_diag1ZloglikZnllr%r%r&rsP,zSiglipModel.forwardr)NNNF)NNNNNNNF)rZr[r\rrr{rrr<rrr^rrrrrcrrr%r%rr&rs  - 0  rz SigLIP vision encoder with an image classification head on top (a linear layer on top of the pooled final hidden states of the patch tokens) e.g. for ImageNet. csteZdZdZdeddffdd Zee     ddee j dee j d ee d ee d e de f d d Z ZS)rrrsr;NcsZt||j|_t|j}|j|_|jdkr"t|jj |jnt |_ | dS)Nr) rzr{ num_labelsrrrrrrr|ZIdentityrr)rnrsrrr%r&r{qs  " z%SiglipForImageClassification.__init__Flabelsrrrc Cs^|dur|n|jj}|dur|n|jj}|j||||d}|j}tj|dd}||}d} |dur||j }|jj durb|j dkrHd|j_ n|j dkr^|j tj ksY|j tjkr^d|j_ nd|j_ |jj dkrt} |j dkrz| ||} n+| ||} n%|jj dkrt} | |d|j |d} n|jj dkrt} | ||} t| ||j|jd S) a$ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, SiglipForImageClassification >>> import torch >>> from PIL import Image >>> import requests >>> torch.manual_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # note: we are loading a `SiglipModel` from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random if seed is not set above. >>> image_processor = AutoImageProcessor.from_pretrained("google/siglip-base-patch16-224") >>> model = SiglipForImageClassification.from_pretrained("google/siglip-base-patch16-224") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the two classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: LABEL_1 ```N)rrrrr Z regressionZsingle_label_classificationZmulti_label_classificationrw)rdlogitsrXrY)rsrrrrWr<r2rrrZ problem_typer%rlongrr Zsqueezer rr rrXrY) rnrr&rrrrZsequence_outputr'rdZloss_fctr%r%r&rsL)     "       z$SiglipForImageClassification.forward)NNNNF)rZr[r\rrr{rrrr<rrrrrr%r%rr&rhs.r)rrrrr)r9rr:r )rr?r@)r9)Kr]r!r, dataclassesrtypingrrrrrnumpyrr<Ztorch.utils.checkpointrZtorch.nnr r r Z torch.nn.initr Z activationsrZmodeling_attn_mask_utilsrZmodeling_layersrZmodeling_outputsrrrZmodeling_utilsrrutilsrrrrrZconfiguration_sigliprrrZ get_loggerrZrr8rfloatr>rLrQrSrUrarcrrrrrrrrrrrrrrrrr__all__r%r%r%r&s       % $I/ H0?P?305f r