o Zh@sdZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd lm Z m!Z!d dl"m#Z#e!$e%Z&eGdddeZ'Gddde j(Z)Gddde j(Z*Gddde j(Z+Gddde j(Z,Gddde j(Z-Gddde j(Z.Gddde j(Z/Gdd d e j(Z0Gd!d"d"e j(Z1Gd#d$d$e j(Z2e Gd%d&d&eZ3e Gd'd(d(e3Z4Gd)d*d*e j(Z5e d+d,Gd-d.d.e3Z6Gd/d0d0e j(Z7Gd1d2d2e j(Z8e d3d,Gd4d5d5e3Z9e d6d,Gd7d8d8e3Z:e d9d,Gd:d;d;e3Z;e GdZ=dS)?zPyTorch ViLT model.N) dataclass)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesmeshgridprune_linear_layer)auto_docstringlogging) ViltConfigc@sjeZdZUdZdZeejed<dZ eejed<dZ ee e ejed<dZ ee e ejed<dS)(ViltForImagesAndTextClassificationOutputa Class for outputs of [`ViltForImagesAndTextClassification`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`List[tuple(torch.FloatTensor)]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`List[tuple(torch.FloatTensor)]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the attention weights of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlosslogits hidden_states attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrrr%r%U/var/www/auris/lib/python3.10/site-packages/transformers/models/vilt/modeling_vilt.pyr-s rcs6eZdZdZfddZd ddZ d dd ZZS) ViltEmbeddingsz Construct the text and patch embeddings. Text embeddings are equivalent to BERT embeddings. Patch embeddings are equivalent to ViT embeddings. cstt||_ttdd|j|_ t ||_ |j j }ttd|d|j|_ t|j|j|_t|j|_||_dSNr)super__init__TextEmbeddingstext_embeddingsr Parameterr"zeros hidden_size cls_tokenViltPatchEmbeddingspatch_embeddings num_patchesposition_embeddings EmbeddingZmodality_type_vocab_sizetoken_type_embeddingsDropouthidden_dropout_probdropoutconfig)selfr:r3 __class__r%r&r*Ps    zViltEmbeddings.__init__cs|jjjj\}}}}||}|dddddddf}tjj||jd|jdfd}|dddfj dddddf} |dddfj dddddf} |j\} } |j j |j j } |j ddddddfddd| | | tjfddt| | Ddd}|ddd}|ddd}tjtt|jd t|jd d d d dj|jd }|ddddddddf}||jd|jdd d d }|dd}|d}dksdustts| | }|n | | }t||jddd|jdddddf}fdd|D}fdd|D}dd|D}dd|D}fdd|D}g}t t|||D]B\}\}}}|dkrut!t"|}|#|||qStj!t"||dd}|#tj|||||gddqStj|dd}||dddf|dddff| d | }||dddf|dddff| d }||dddf|dddff| d d}||dddf|dddff| d | }|j$| d d }tj||fdd}tj|j dddddfdddddf| d d |fdd}||}|%|}tjt"|jdd||gdd}|||fffS)Nr sizerrdimc sBg|]\}}tjtjj||fdddd|d|fqS)ZbilinearT)rAmodeZ align_cornersr)r functionalpad interpolate).0hw)height spatial_poswidthr%r& ls z/ViltEmbeddings.visual_embed..Zij)ZindexingdeviceF)as_tuplec$g|]}dddf|kqSNrr%rHu) valid_idxr%r&rN$crTrUr%rV) non_valid_idxr%r&rNrYcSg|]}|dqSrr@rHvr%r%r&rNcSr[r\r@r]r%r%r&rNr_csg|]}|qSr%r%r]max_image_lengthr%r&rNT) replacement)&r2 projectionweightshapefloatrrErGlongsumr: image_size patch_sizer4 transposeviewr"catzipflattenstackrarangetorRexpand isinstanceintmaxminZnonzerounique enumerateZ multinomialonesappendr0r9)r; pixel_values pixel_maskra_phpwxZx_maskZx_hZx_w batch_size num_channelsZ patch_dimZ pos_embed patch_indexZeffective_resolutionZ unique_rowsZ valid_row_idxZnon_valid_row_idxZ valid_numsZnon_valid_numsZpad_numsselectir^nvpZ valid_choiceZ pad_choiceZ cls_tokensr%)rKrarZrLrXrMr& visual_embed_sx  $$$0  &     (.,..8 &zViltEmbeddings.visual_embedrc  Cs|j|||d} |dur|j|||jjd\}} } n|d} |dur%d}| |tj|tj| j d} ||tj | |tj| j d}tj | |gdd} tj || gdd} | | fS)N) input_idstoken_type_ids inputs_embedsr`rdtyperRrB) r,rr:rarpr6r"Z zeros_likerhrRZ full_likern)r;rattention_maskrr}r~r image_embedsimage_token_type_idxZ text_embedsZ image_masksr embeddingsmasksr%r%r&forwards&   zViltEmbeddings.forward)r>)r)rrr r!r*rr __classcell__r%r%r<r&r'Gs   ar'cs*eZdZdZfddZdddZZS)r+zGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dddS) N) padding_idxZepsposition_embedding_typeabsolute position_ids)rrPF) persistentrr)r)r*rr5 vocab_sizer/Z pad_token_idword_embeddingsZmax_position_embeddingsr4Ztype_vocab_sizer6 LayerNormlayer_norm_epsr7r8r9getattrrZregister_bufferr"rrrtr.rrArhr;r:r<r%r&r*s  zTextEmbeddings.__init__Nc Cs|dur |}n|dd}|d}|dur$|jddd|f}|durNt|drC|jddd|f}||d|}|}n tj|tj|jjd}|durW| |}| |} || } |j dkrn| |} | | 7} | | } || } | S)NrPrrrrr)rArhasattrrrtr"r.rhrRrr6rr4rr9) r;rrrr input_shape seq_lengthZbuffered_token_type_idsZ buffered_token_type_ids_expandedr6rr4r%r%r&rs,        zTextEmbeddings.forward)NNNNrrr r!r*rrr%r%r<r&r+s r+cs(eZdZdZfddZddZZS)r1z# Image to Patch Embedding. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr)Z kernel_sizeZstride)r)r*rjrkrr/ru collectionsabcIterabler3rConv2drd)r;r:rjrkrr/r3r<r%r&r*s  zViltPatchEmbeddings.__init__cCs@|j\}}}}||jkrtd|jjj}||j|d}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r)rfr ValueErrorrdrerrs)r;r}rrrKrMZ target_dtyperr%r%r&r.s  zViltPatchEmbeddings.forwardrr%r%r<r&r1s r1c.eZdZfddZddZd ddZZS) ViltSelfAttentioncst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r)r*r/num_attention_headsrrrvattention_head_size all_head_sizerLinearZqkv_biasquerykeyvaluer7Zattention_probs_dropout_probr9rr<r%r&r*:s  zViltSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)NrPrr?rr )rArrrmpermute)r;rZ new_x_shaper%r%r&transpose_for_scoresLs z&ViltSelfAttention.transpose_for_scoresNFcCs||}|||}|||}||}t||dd} | t|j } |dur4| |} t j dd| } | | } |durI| |} t| |} | dddd} | dd|jf} | j| } |rr| | f} | S| f} | S)NrPrOrBrr?rr )rrrrr"matmulrlmathsqrtrrZSoftmaxr9r contiguousrArrm)r;rr head_maskoutput_attentionsZmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr%r%r&rQs(      zViltSelfAttention.forwardNNF)rrr r*rrrr%r%r<r&r9s rcsFeZdZdZdeddffdd Zdejdejdejfd d ZZ S) ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r:returnNcs.tt|j|j|_t|j|_dSN) r)r*rrr/denser7r8r9rr<r%r&r*| zViltSelfOutput.__init__r input_tensorcC||}||}|Srrr9r;rrr%r%r&r  zViltSelfOutput.forward) rrr r!rr*r"Tensorrrr%r%r<r&rvs$rcr) ViltAttentioncs*tt||_t||_t|_dSr)r)r*r attentionroutputset pruned_headsrr<r%r&r*s    zViltAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrB)lenrrrrrrrrrrrrunion)r;headsindexr%r%r& prune_headss zViltAttention.prune_headsNFcCs6|||||}||d|}|f|dd}|S)Nrr)rr)r;rrrrZ self_outputsattention_outputrr%r%r&rszViltAttention.forwardr)rrr r*rrrr%r%r<r&rs rcs<eZdZdeddffdd ZdejdejfddZZS) ViltIntermediater:rNcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r)r*rrr/intermediate_sizerru hidden_actstrr intermediate_act_fnrr<r%r&r*s   zViltIntermediate.__init__rcCrr)rrr;rr%r%r&rrzViltIntermediate.forward rrr rr*r"rrrr%r%r<r&rsrcsBeZdZdeddffdd Zdejdejdejfdd ZZS) ViltOutputr:rNcs.tt|j|j|_t|j|_ dSr) r)r*rrrr/rr7r8r9rr<r%r&r*rzViltOutput.__init__rrcCs ||}||}||}|Srrrr%r%r&rs  zViltOutput.forwardrr%r%r<r&rs$rcs*eZdZdZfddZdddZZS) ViltLayerz?This corresponds to the Block class in the timm implementation.csbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nrr)r)r*Zchunk_size_feed_forwardZ seq_len_dimrrr intermediaterrrrr/rlayernorm_beforelayernorm_afterrr<r%r&r*s    zViltLayer.__init__NFc Csj|j|||||d}|d}|dd}|||j}||}||}|||}|f|}|S)N)rrr)rrrsrRrrr) r;rrrrZself_attention_outputsrrZ layer_outputr%r%r&rs     zViltLayer.forwardrrr%r%r<r&rs  rcs0eZdZfddZ     dddZZS) ViltEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr%)r)rHrr:r%r&rNrbz(ViltEncoder.__init__..F) r)r*r:rZ ModuleListrangenum_hidden_layerslayergradient_checkpointingrr<rr&r*s   zViltEncoder.__init__NFTc Cs|rdnd}|r dnd}t|jD]:\} } |r||f}|dur$|| nd} |jr7|jr7|| j||| |} n| ||| |} | d}|rK|| df}q|rS||f}|satdd|||fDSt|||dS)Nr%rrcss|] }|dur|VqdSrr%r]r%r%r& sz&ViltEncoder.forward..)last_hidden_staterr)rzrrZtrainingZ_gradient_checkpointing_func__call__tupler ) r;rrrroutput_hidden_states return_dictZall_hidden_statesZall_self_attentionsrZ layer_moduleZlayer_head_maskZ layer_outputsr%r%r&rs8    zViltEncoder.forward)NNFFTrrr r*rrr%r%r<r&rs  rc@s(eZdZeZdZdZddgZddZdS)ViltPreTrainedModelviltTr'rcCst|tjtjfr#|jjjd|jjd|j dur!|j j dSdSt|tj rF|jjjd|jjd|j durD|jj|j  dSdSt|tj r[|j j |jjddSdS)zInitialize the weightsg)meanZstdNg?)rurrrredataZnormal_r:Zinitializer_rangerZzero_r5rrZfill_)r;moduler%r%r& _init_weights-s     z!ViltPreTrainedModel._init_weightsN) rrr rZ config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesrr%r%r%r&r&s  rcseZdZdfdd ZddZddZdd Ze dd ee j d ee j d ee j dee j dee j dee j dee j dee j dee dee dee dee deeee j ffddZZS) ViltModelTcsXt|||_t||_t||_tj|j |j d|_ |r#t |nd|_ |dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r)r*r:r'rrencoderrrr/r layernorm ViltPoolerpooler post_init)r;r:add_pooling_layerr<r%r&r*@s    zViltModel.__init__cCs |jjjSrrr,rr;r%r%r&get_input_embeddingsQs zViltModel.get_input_embeddingscCs||jj_dSrr)r;rr%r%r&set_input_embeddingsTzViltModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r;Zheads_to_prunerrr%r%r& _prune_headsWszViltModel._prune_headsNrrrr}r~rrrrrrrrc  Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur*|dur*td|dur9||||} n|durF|dd} ntd| \}}|durU|jn|j}|duretj ||f|d}|durq|durqtd|dur}|dur}td|dur|j dn|j d}||krtd |durtj ||jj |jj f|d}| ||jj }|j|||||||| d \}}||| }|j|||| | | d }|d}||}|jdur||nd}| s||f|d dSt|||j|jd S)ak image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerPz5You have to specify either input_ids or inputs_embedsrQzFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)rrrrrr)r pooler_outputrr)r:rruse_return_dictrZ%warn_if_padding_and_no_attention_maskrArRr"r{rfrjZ get_head_maskrrZget_extended_attention_maskrrrr rr)r;rrrr}r~rrrrrrrrZtext_batch_sizerrRZimage_batch_sizeZembedding_outputZextended_attention_maskZencoder_outputssequence_output pooled_outputr%r%r&r_sp*    zViltModel.forward)T NNNNNNNNNNNN)rrr r*rrrrrr" LongTensorr#rvboolrr rrrr%r%r<r&r>sZ     rc$eZdZfddZddZZS)rcs*tt|j|j|_t|_dSr)r)r*rrr/rZTanh activationrr<r%r&r*s zViltPooler.__init__cCs(|dddf}||}||}|SrU)rr)r;rZfirst_token_tensorr r%r%r&rs  zViltPooler.forwardrr%r%r<r&rs rzU ViLT Model with a language modeling head on top as done during pretraining. )Z custom_introcseZdZddgZfddZddZddZe dd ee j d ee j d ee j d ee j dee j dee j dee j dee j dee j dee dee dee de eee j ffddZZS)ViltForMaskedLMzmlm_score.decoder.weightzmlm_score.decoder.biascs,t|t||_t||_|dSr)r)r*rr ViltMLMHead mlm_scorerrr<r%r&r*s    zViltForMaskedLM.__init__cCs|jjSr)rdecoderrr%r%r&get_output_embeddingssz%ViltForMaskedLM.get_output_embeddingscCs||j_|j|j_dSr)rrr)r;Znew_embeddingsr%r%r&set_output_embeddingssz%ViltForMaskedLM.set_output_embeddingsNrrrr}r~rrrlabelsrrrrc  Cs| dur| n|jj} |j||||||||| | | d } | dd\}}|dur+|jdn|jd}|ddd|f|dd|df}}||}d}| durgt}| |j} ||d|jj | d}| s}|f| dd}|dur{|f|S|St ||| j | j dS)a/ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import requests >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrr}r~rrrrrrr?rrPrrrr) r:rrrfrrrsrRrmrr rr)r;rrrr}r~rrrrrrrrrr Z text_seq_lenZ text_featuresrZ mlm_logitsZmasked_lm_lossloss_fctrr%r%r&rs@I*  zViltForMaskedLM.forwardr )rrr Z_tied_weights_keysr*rrrrr"r r#r rr rrrr%r%r<r&rsZ       rcr )ViltPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dS)Nr) r)r*rrr/rrurrr transform_act_fnrrrr<r%r&r*rs  z$ViltPredictionHeadTransform.__init__cCs"||}||}||}|Sr)rrrrr%r%r&r{s   z#ViltPredictionHeadTransform.forwardrr%r%r<r&rqs  rcs.eZdZdfdd ZddZddZZS) rNcsbt||_t||_tj|j|jdd|_ t t |j|_ |dur*||j _|j |j _ dS)NFr)r)r*r:r transformrrr/rrr-r"r.rre)r;r:rer<r%r&r*s  zViltMLMHead.__init__cCs|j|j_dSr)rrrr%r%r& _tie_weightsrzViltMLMHead._tie_weightscCrr)rr)r;rr%r%r&rs  zViltMLMHead.forwardr)rrr r*rrrr%r%r<r&rs rz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. ceZdZfddZe            ddeejdeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee de e e ejffddZ ZS)ViltForQuestionAnsweringc slt||j|_t||_tt|j|jdt |jdt t|jd|j|_ | dS)Nr?) r)r* num_labelsrrr Sequentialrr/rGELU classifierrrr<r%r&r*s   z!ViltForQuestionAnswering.__init__Nrrrr}r~rrrrrrrrc  Cs| dur| n|jj} |j||||||||| | | d } | r| jn| d}||}d}| dur@| |j} tj || | j d}| sV|f| dd}|durT|f|S|St ||| j | j dS)aX image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.FloatTensor` of shape `(batch_size, num_labels)`, *optional*): Labels for computing the visual question answering loss. This tensor must be either a one-hot encoding of all answers that are applicable for a given example in the batch, or a soft encoding indicating which answers are applicable, where 1.0 is the highest score. Examples: ```python >>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```Nrrr?r)r:rrrr"rsrRrrEZ binary_cross_entropy_with_logitsrfrrr)r;rrrr}r~rrrrrrrrrrrrr%r%r&rs:1  z ViltForQuestionAnswering.forwardr rrr r*rrr"r r#r rrrrrr%r%r<r&rsT      rz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text or text-to-image retrieval, e.g. MSCOCO and F30K. cr)ViltForImageAndTextRetrievalcs2t|t||_t|jd|_|dSr() r)r*rrrrr/ rank_outputrrr<r%r&r*s   z%ViltForImageAndTextRetrieval.__init__Nrrrr}r~rrrrrrrrc  Cs| dur| n|jj} d} | durtd|j||||||||| | | d }| r)|jn|d}||}| sH|f|dd}| durF| f|S|St| ||j|jdS)ad image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Examples: ```python >>> from transformers import ViltProcessor, ViltForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> model = ViltForImageAndTextRetrieval.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, :].item() ```NzTraining is not yet supported.rrr?r) r:rNotImplementedErrorrrr%rrr)r;rrrr}r~rrrrrrrrrrrrr%r%r&rs8- z$ViltForImageAndTextRetrieval.forwardr r#r%r%r<r&r$sT       r$zq Vilt Model transformer with a classifier head on top for natural language visual reasoning, e.g. NLVR2. cr)"ViltForImagesAndTextClassificationc svt||j|_t||_|j}tt|j ||j |t |j |t t|j ||j|_ | dSr)r)r*rrr num_imagesrr rr/rr!r"r)r;r:r(r<r%r&r*qs   z+ViltForImagesAndTextClassification.__init__Nrrrr}r~rrrrrrrrc CsD| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur,|jdkr,|d}|dur:|jdkr:|d}|durC|jdnd} | durT|durR|jdnd} | |jjkr^tdg}| rdgnd}| rjgnd}t | D]l}|j ||||dur|dd|ddddddfnd|dur|dd|ddddfnd|||dur|dd|ddddfnd|d| | | d }| r|j n|d}| || r| |j | r| |jqptj|dd}||}d}| durt}| |j} ||d|j| d}| s|||f}|dur|f|S|St||||d S) a3 image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Binary classification labels. Examples: ```python >>> from transformers import ViltProcessor, ViltForImagesAndTextClassification >>> import requests >>> from PIL import Image >>> image1 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_0.jpg", stream=True).raw) >>> image2 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_1.jpg", stream=True).raw) >>> text = "The left image contains twice the number of dogs as the right image." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> model = ViltForImagesAndTextClassification.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> # prepare inputs >>> encoding = processor([image1, image2], text, return_tensors="pt") >>> # forward pass >>> outputs = model(input_ids=encoding.input_ids, pixel_values=encoding.pixel_values.unsqueeze(0)) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: True ```Nrr z\Make sure to match the number of images in the model with the number of images in the input.) rrr}r~rrrrrrrrPrBr)r:rrrndimZ unsqueezerfr(rrrrr|rrr"rnr"rrsrRrmrr)r;rrrr}r~rrrrrrrr(Zpooler_outputsrrrrrr rrrrr%r%r&rsp/      ,&&       z*ViltForImagesAndTextClassification.forwardr )rrr r*rrr"r r#r rrrrrr%r%r<r&r'ksT      r'cr)ViltForTokenClassificationcsNt||j|_t|dd|_t|j|_t |j |j|_ | dS)NF)r) r)r*rrrrr7r8r9rr/r"rrr<r%r&r*s  z#ViltForTokenClassification.__init__Nrrrr}r~rrrrrrrrc  Cs| dur| n|jj} |j||||||||| | | d } | d}|dur'|jdn|jd}||}||ddd|f}d}| durZt}| |j} || d|j | d}| sp|f| dd}|durn|f|S|St ||| j | j dS)a/ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, text_sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. NrrrrPr?r)r:rrrfr9r"rrsrRrmrrrr)r;rrrr}r~rrrrrrrrrZtext_input_sizerrrrr%r%r&rs@  z"ViltForTokenClassification.forwardr )rrr r*rrr"r r#r rrrrrr%r%r<r&r+sT       r+)r$r'r+rrrrr)>r!collections.abcrr dataclassesrtypingrrrrr"Ztorch.utils.checkpointrZtorch.nnrZ activationsr Zmodeling_outputsr r r rrrZmodeling_utilsrZ pytorch_utilsrrrutilsrrZconfiguration_viltrZ get_loggerrloggerrModuler'r+r1rrrrrrrrrrrrrrr$r'r+__all__r%r%r%r&st        9=#&5j] O