o Zh6@sdZddlmZddlmZmZmZmZmZddl Z ddl m Z ddl m Z m Z mZddlmZdd lmZmZdd lmZmZmZdd lmZmZdd lmZmZmZmZm Z d dl!m"Z"m#Z#m$Z$e%e&Z'de j(de j(fddZ)de j(de j(fddZ*de j(de j(fddZ+eGdddeZ,eGdddeZ-eGdddeZ.Gdd d e j/Z0Gd!d"d"e j/Z1 # $dNd%e j/d&e j(d'e j(d(e j(d)ee j(d*e2d+e2d,e3fd-d.Z4Gd/d0d0e j/Z5Gd1d2d2e j/Z6Gd3d4d4e j/Z7eGd5d6d6eZ8Gd7d8d8e j/Z9Gd9d:d:e j/Z:ed;d<Gd=d>d>e8Z;Gd?d@d@e j/ZeGdFdGdGe8Z?eGdHdIdIe8Z@edJd<GdKdLdLe8ZAgdMZBdS)OzPyTorch CLIP model.) dataclass)AnyCallableOptionalTupleUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) ModelOutputauto_docstringcan_return_tuplelogging torch_int) CLIPConfigCLIPTextConfigCLIPVisionConfiglogitsreturncCstj|tjt||jdS)Ndevice)r functionalZ cross_entropytorcharangelenr!)rr&U/var/www/auris/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.pycontrastive_loss%sr( similaritycCs t|}t|}||dS)Ng@)r(t)r)Z caption_lossZ image_lossr&r&r' clip_loss)s  r+tensorcCs,t|d}tj|ddd}t|d}|S)z This method is equivalent to tensor.norm(p=2, dim=-1, keepdim=True) and used to make model `executorch` exportable. See issue https://github.com/pytorch/executorch/issues/3566 T)dimZkeepdim?)r#powsum)r,Z square_tensorZ sum_tensorZ normed_tensorr&r&r'_get_vector_norm/s  r3c@jeZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS)CLIPVisionModelOutputa Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r6rr# FloatTensor__annotations__r7r8rr9r&r&r&r'r5: r5c@r4)CLIPTextModelOutputa Base class for text model's outputs that also contains a pooling of the last hidden states. Args: text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N text_embedsr7.r8r9) r:r;r<r=rBrr#r>r?r7r8rr9r&r&r&r'rAWr@rAc@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd d ZdS) CLIPOutputa Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPVisionModel`]. Nlosslogits_per_imagelogits_per_textrBr6text_model_outputvision_model_outputrcstfddDS)Nc3s.|]}|dvr |nt|VqdS))rGrHN)getattrto_tuple).0kselfr&r' s  z&CLIPOutput.to_tuple..)tuplekeysrMr&rMr'rJs zCLIPOutput.to_tuple)r:r;r<r=rDrr#r>r?rErFrBr6rGrrHrrrJr&r&r&r'rCts   rCcsXeZdZdeffdd Zdejdededejfdd Zdd ej dejfd d Z Z S)CLIPVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF)Z in_channelsZ out_channelsZ kernel_sizeZstridebiasr-r position_idsrr. persistent)super__init__rS hidden_size embed_dim image_size patch_sizer Parameterr#Zrandnclass_embeddingZConv2dZ num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferr$expandrNrS __class__r&r'rZs"  "zCLIPVisionEmbeddings.__init__ embeddingsheightwidthrc Cs|jdd}|jjd}|jdd}tjs(||kr(||kr(||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr.r0r r-ZbicubicF)sizemodeZ align_cornersr/)shapereweightZ unsqueezer#Zjit is_tracingrUr^rreshapeZpermuterr"Z interpolateviewcat) rNrkrlrmrbrercZclass_pos_embedZpatch_pos_embedr/Z new_heightZ new_widthZsqrt_num_positionsr&r&r'interpolate_pos_encodings*      z-CLIPVisionEmbeddings.interpolate_pos_encodingF pixel_valuesc Cs|j\}}}}|s&||jks||jkr&td|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r[| | | ||} | S| ||j} | S) NzInput image size (*z) doesn't match model ().)dtyper-rr.rp)rqr] ValueErrorrarrr{toflatten transposer`rgr#rvrwrerU) rNrxrw batch_size_rlrmZ target_dtypeZ patch_embedsZ class_embedsrkr&r&r'forwards  zCLIPVisionEmbeddings.forwardF) r:r;r<rrZr#Tensorintrwr>r __classcell__r&r&rir'rRs )rRc sXeZdZdeffdd Z   d deejdeejdeejdej fd d Z Z S) CLIPTextEmbeddingsrScsRt|j}t|j||_t|j||_|j dt |j ddddS)NrUrVFrW) rYrZr[rrdZ vocab_sizetoken_embeddingZmax_position_embeddingsrerfr#r$rgrNrSr\rir&r'rZs  zCLIPTextEmbeddings.__init__N input_idsrU inputs_embedsrcCs|dur |jdn|jd}|jjjd}||kr#td|d||dur2|jddd|f}|dur;||}||}||}|S)Nr.rzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rqrerrr|rUr)rNrrUr seq_lengthZmax_position_embeddingZposition_embeddingsrkr&r&r'rs"  zCLIPTextEmbeddings.forward)NNN) r:r;r<rrZrr# LongTensorr>rrrr&r&rir'rsrTmodulequerykeyvalueattention_maskscalingdropoutoutput_attentionsc Kst||dd|} |dur| |} tjj| dtjd|j} tjj | ||j d} t| |} | dd } |s>d} | | fS)Nr.r)r/r{)ptrainingrr-) r#matmulrrr"ZsoftmaxZfloat32r}r{rr contiguous) rrrrrrrrkwargs attn_weights attn_outputr&r&r'eager_attention_forwards  rcsveZdZdZdeeefffdd Z   ddej de ej d e ej d e e d e ej e ej ff d d Z ZS) CLIPAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrScst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rzF)rYrZrSr[r\Znum_attention_headsZ num_headshead_dimr|scaleZattention_dropoutr is_causalrLineark_projv_projq_projout_projrhrir&r'rZ/s$   zCLIPAttention.__init__NFr8rcausal_attention_maskrrc CsH|j\}}}||}||} ||} |||d|jdd}| ||d|jdd} | ||d|jdd} |jjdkrH|du|_ n|durU|durU||}n|dur[|}t } |jjdkrw|jjdkrq|rqt dnt |jj} | ||| | ||j |j|jsd n|j|d \} } | |||} || } |sd} | | fS) z#Input shape: Batch x Time x Channelr.rr-flash_attention_2NeagerZsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)rrrr)rqrrrrurrrS_attn_implementationrrloggerZ warning_oncerrrrrtrr)rNr8rrrrrr\ZqueriesrQvaluesZattention_interfacerrr&r&r'rCsH          zCLIPAttention.forward)NNF)r:r;r<r=rrrrZr#rrboolrrrr&r&rir'r,s"rcs2eZdZfddZdejdejfddZZS)CLIPMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) rYrZrSr Z hidden_act activation_fnrrr[Zintermediate_sizefc1fc2rhrir&r'rZ~s  zCLIPMLP.__init__r8rcCs"||}||}||}|Sr)rrr)rNr8r&r&r'rs   zCLIPMLP.forward)r:r;r<rZr#rrrr&r&rir'r}s rc s\eZdZdeeefffdd Z d dejdejdejde e d e ej f d d Z ZS) CLIPEncoderLayerrScsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSNZeps) rYrZr[r\r self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rhrir&r'rZs   zCLIPEncoderLayer.__init__Fr8rrrrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r0||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r8rrr)rrrr)rNr8rrrZresidualroutputsr&r&r'rs"     zCLIPEncoderLayer.forwardr)r:r;r<rrrrZr#rrrrr>rrr&r&rir'rs rc@s(eZdZeZdZdZdZdZddZ dS)CLIPPreTrainedModelZclipTcCs|jj}t|tr#|jjjjd|dd|jjjjd|ddn)t|t rX|jj}t j j|j d|j d|dt j j|jj|jj|dt j j|jj|jj|dnt|tr|jj}|j dd|jjd|}|j d|}t j j|jj|dt j j|jj|dt j j|jj|dt j j|jj|dnt|tr|jj}|jjdd|jjd|}d|jjd|}t j j|jj|dt j j|jj|dntt|trt j j|jj|jd|jjdt j j|jj|jd|jjdnKt|trt j j|jj|jjd|jjdn2t|t r3t j j|jj|jjd|jjdnt|t!rLt j j|j"j|jj#jd|jjdt|t j$r`|j%j&|jj'dt|t j(ru|j%durw|j%j&dSdSdS) zInitialize the weightsrg{Gz?)meanstdr)rr-g?N))rSZinitializer_factor isinstancerrrrdataZnormal_rerRrinitr`r\raZinitializer_rangernum_hidden_layersrrrrrr[rr CLIPModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimCLIPVisionModelWithProjectionCLIPTextModelWithProjectionCLIPForImageClassification classifier vision_configrrTZzero_Zfill_r)rNrfactorZ in_proj_stdZ out_proj_stdZfc_stdr&r&r' _init_weightssj          z!CLIPPreTrainedModel._init_weightsN) r:r;r<r config_classZbase_model_prefixZsupports_gradient_checkpointingZ_supports_sdpaZ_supports_flash_attn_2rr&r&r&r'rs rc sfeZdZdZdeffdd Ze    d deej deej dee d ee d e f d d Z Z S) CLIPEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPEncoderLayer`]. Args: config: CLIPConfig rScs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr&)r)rKrrSr&r' sz(CLIPEncoder.__init__..F) rYrZrSrZ ModuleListrangerlayersgradient_checkpointingrhrirr'rZs   zCLIPEncoder.__init__Nrrroutput_hidden_statesrc Cs|dur|n|jj}|dur|n|jj}|rdnd}|rdnd}|}t|jD]1\} } |r2||f}|jrC|jrC|| j||||} n| ||||d} | d}|rX|| df}q'|r`||f}t |||dS)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr&)rrr)r7r8r9) rSrr enumeraterrrZ_gradient_checkpointing_func__call__r) rNrrrrrZencoder_statesZall_attentionsr8idxZ encoder_layerZ layer_outputsr&r&r'rsD&     zCLIPEncoder.forwardNNNN)r:r;r<r=rrZrrr#rrrrrr&r&rir'rs&rcsreZdZdeffdd Zee     d deej deej deej dee d ee d e f d d Z Z S)CLIPTextTransformerrScsTt||_|j}t||_t||_tj ||j d|_ |j |_ |j dk|_dS)Nrr)rYrZrSr[rrkrencoderrrrfinal_layer_norm eos_token_idr_use_flash_attention_2rrir&r'rZ`s   zCLIPTextTransformer.__init__NrrrUrrrc Cs@|dur|n|jj}|dur|n|jj}|durtd|}|d|d}|j||d}t||j|j d}|durE|j sEt ||j}|j |||||d} | j } || } |jdkrw| tj| jd| j d|jtj| j djdd f} n| tj| jd| j d|jtj| j d|jkjdd f} t| | | j| jd S) NzYou have to specify input_idsr.)rrUr )rrrrrr-r)r{r!rpr7 pooler_outputr8r9)rSrrr|rnrurkrr{r!rrrr7rrr#r$rqr}rZargmaxrr8r9) rNrrrUrrZ input_shaper8rencoder_outputsr7 pooled_outputr&r&r'rnsT      zCLIPTextTransformer.forwardNNNNN)r:r;r<rrZrrrr#rrrrrr&r&rir'r_s,rzI The text model from CLIP without any head or projection on top. )Z custom_introceZdZeZddgZdeffdd ZdejfddZ d d Z e e dd e ejd e ejde ejde ede edef ddZZS) CLIPTextModelrrrSc"t|t||_|dSr)rYrZr text_model post_initrhrir&r'rZ   zCLIPTextModel.__init__rcC |jjjSrrrkrrMr&r&r'get_input_embeddings z"CLIPTextModel.get_input_embeddingscC||jj_dSrrrNrr&r&r'set_input_embeddingsz"CLIPTextModel.set_input_embeddingsNrrrUrrcCs|j|||||dS)a9 Examples: ```python >>> from transformers import AutoTokenizer, CLIPTextModel >>> model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrUrr)r)rNrrrUrrr&r&r'rszCLIPTextModel.forwardr)r:r;r<rr_no_split_modulesrZrModulerrrrrr#rrrrrr&r&rir'rs4rc sdeZdZdeffdd Zee    d deej dee dee d ee d e f d d Z Z S)CLIPVisionTransformerrScsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) rYrZrSr[rRrkrrr pre_layrnormrrpost_layernormrrir&r'rZs   zCLIPVisionTransformer.__init__NFrxrrrwrc Cs|dur|n|jj}|dur|n|jj}|durtd|j||d}||}|j|||d}|j}|dddddf}||}t |||j |j dS)Nz You have to specify pixel_values)rw)rrrrr) rSrrr|rkrrr7rrr8r9) rNrxrrrwr8rr7rr&r&r'rs*   zCLIPVisionTransformer.forwardNNNF)r:r;r<rrZrrrr#r>rrrrr&r&rir'rs& rzK The vision model from CLIP without any head or projection on top. c s~eZdZeZdZdgZdeffdd Zdej fddZ e e dde ejd e ed e ed edef ddZZS)CLIPVisionModelrxrrScrr)rYrZr vision_modelrrhrir&r'rZ-rzCLIPVisionModel.__init__rcCrrrrkrarMr&r&r'r3rz$CLIPVisionModel.get_input_embeddingsNFrrrwcCs|j||||dS)a Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPVisionModel >>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rxrrrw)r)rNrxrrrwr&r&r'r6s zCLIPVisionModel.forwardr)r:r;r<rrmain_input_namerrZrrrrrrr#r>rrrrr&r&rir'r#s.rcs$eZdZeZgdZdeffdd Ze     ddee j dee j dee j d ee d ee d e j f d d Z e    ddee j d ee d ee de d e j f ddZee        ddee jdee j dee j dee jdee d ee d ee de d efddZZS)r)rrrRrScst|t|jtstdt|jdt|jts(tdt|jd|j}|j}|j |_ |j |_ |j |_ t |}|j|_t|}|j|_tj|j |j dd|_tj|j |j dd|_tt|jj|_|dS)NzKconfig.text_config is expected to be of type CLIPTextConfig but is of type .zOconfig.vision_config is expected to be of type CLIPVisionConfig but is of type FrT)rYrZr text_configr TypeErrortyperrprojection_dimr[rrr _from_configrrrrrrrr_r#r,rSZlogit_scale_init_value logit_scaler)rNrSrrrrrir&r'rZas4      zCLIPModel.__init__NrrrUrrrc CsP|dur|n|jj}|dur|n|jj}|j|||||d}|j}||}|S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. Examples: ```python >>> from transformers import AutoTokenizer, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```Nr)rSrrrrr) rNrrrUrr text_outputsrZ text_featuresr&r&r'get_text_featuress zCLIPModel.get_text_featuresFrxrwcCsN|dur|n|jj}|dur|n|jj}|j||||d}|j}||}|S)aD Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```Nr)rSrrrrr)rNrxrrrwvision_outputsrZimage_featuresr&r&r'get_image_featuress zCLIPModel.get_image_features return_lossc  Cs|dur|n|jj}|dur|n|jj}|j||||d} |j|||||d} | j} || } | j} || } | t| } | t| } t | | | j } | |j | j } | }d}|rft| }t||| | | | | dS)a return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```Nrr)rDrErFrBr6rGrH)rSrrrrrrrr3r#rr*r}r!r expr+rC)rNrrxrrUrrrrwr r r6rBrFrErDr&r&r'rsJ'    zCLIPModel.forwardrr)NNNNNNNF)r:r;r<rrrrZrrr#rrr>r rrrrCrrr&r&rir'r\s# + /  rcr)rrrrSc@t|t|}|j|_tj|j|jdd|_ | dSNFr) rYrZrr rrrr[rrr)rNrSrrir&r'rZ@   z$CLIPTextModelWithProjection.__init__rcCrrrrMr&r&r'rKrz0CLIPTextModelWithProjection.get_input_embeddingscCrrrrr&r&r'rNrz0CLIPTextModelWithProjection.set_input_embeddingsNrrrUrrc Cs:|j|||||d}|j}||}t||j|j|jdS)a Examples: ```python >>> from transformers import AutoTokenizer, CLIPTextModelWithProjection >>> model = CLIPTextModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> text_embeds = outputs.text_embeds ```r)rBr7r8r9)rrrrAr7r8r9) rNrrrUrrr rrBr&r&r'rQs z#CLIPTextModelWithProjection.forwardr)r:r;r<rrrrZrrrrrrrr#rrrArrr&r&rir'r:s4 rc sxeZdZeZdZdeffdd ZdejfddZ e e    dde e jd e ed e ed edef d dZZS)rrxrScrr) rYrZrr rrrr[rrrrNrSrrir&r'rZrz&CLIPVisionModelWithProjection.__init__rcCrrrrMr&r&r'rrz2CLIPVisionModelWithProjection.get_input_embeddingsNFrrrwcCs8|j||||d}|j}||}t||j|j|jdS)a Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPVisionModelWithProjection >>> model = CLIPVisionModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> image_embeds = outputs.image_embeds ```r)r6r7r8r9)rrrr5r7r8r9)rNrxrrrwr rr6r&r&r'rs z%CLIPVisionModelWithProjection.forwardr)r:r;r<rrrrZrrrrrrr#r>rr5rrr&r&rir'r|s, rz CLIP vision encoder with an image classification head on top (a linear layer on top of the pooled final hidden states of the patch tokens) e.g. for ImageNet. c sneZdZdZdeddffdd Zee    d dee j dee j dee d ee de f d d Z ZS) rrxrSrNcsZt||j|_t|j}|j|_|jdkr"t|jj |jnt |_ | dS)Nr) rYrZ num_labelsrr rrrrr[ZIdentityrrrrir&r'rZs  " z#CLIPForImageClassification.__init__labelsrrc Csr|dur|n|jj}|dur|n|jj}|j|||d}|j}tj|ddddddfdd}||}d}|dur||j }|jj durl|j dkrRd|j_ n|j dkrh|j tj ksc|j tjkrhd|j_ nd|j_ |jj dkrt} |j dkr| ||}n+| ||}n%|jj dkrt} | |d|j |d}n|jj dkrt} | ||}t|||j|jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrrpZ regressionZsingle_label_classificationZmulti_label_classificationr.)rDrr8r9)rSrrrr7r#rrr}r!Z problem_typerr{longrr Zsqueezer rur rr8r9) rNrxrrrrZsequence_outputrrDZloss_fctr&r&r'rsJ$     "       z"CLIPForImageClassification.forwardr)r:r;r<rrrZrrrr#rrrrrr&r&rir'rs(r)rrrrrrr)rT)Cr= dataclassesrtypingrrrrrr#rZtorch.nnr r r Z activationsr Zmodeling_attn_mask_utilsrrZmodeling_outputsrrrZmodeling_utilsrrutilsrrrrrZconfiguration_cliprrrZ get_loggerr:rrr(r+r3r5rArCrrRrfloatrrrrrrrrrrrrrrr__all__r&r&r&r's     $S/ Q2@`Z414^A@ T