o Zh @sdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddlmZmZdd lmZdd lmZdd lmZmZmZmZdd lmZmZdd lmZm Z m!Z!ddl"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)e%*e+Z,eGddde#Z-Gddde j.Z/Gddde j.Z0Gddde j.Z1de0iZ2Gddde j.Z3Gddde j.Z4Gddde j.Z5Gd d!d!e j.Z6Gd"d#d#e j.Z7e$Gd$d%d%eZ8Gd&d'd'e j.Z9Gd(d)d)e j.Z: *dId+e j.d,e j;d-e j;d.e j;d/ee j;d0eGd6d7d7e j.Z?Gd8d9d9e j.Z@Gd:d;d;e j.ZAe$dd?d?e8ZBGd@dAdAe j.ZCe$dBd=GdCdDdDe8ZDe$dEd=GdFdGdGe8eZEgdHZFdS)JzPyTorch GIT model.N) dataclass)CallableListOptionalTupleUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)_prepare_4d_attention_mask)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging torch_int) GitConfigGitVisionConfigc@sjeZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS)GitVisionModelOutputa Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r rtorch FloatTensor__annotations__r!r"rr#r+r+S/var/www/auris/lib/python3.10/site-packages/transformers/models/git/modeling_git.pyr2s rc s\eZdZdZfddZ    d deejdeejdeejd e d ej f d d Z Z S) GitEmbeddingsz;Construct the embeddings from word and position embeddings.csttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j |_t|dd|_|jdt|jddddS) N) padding_idxZepsposition_embedding_typeabsolute position_idsrF persistent)super__init__r Embedding vocab_size hidden_sizeZ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr0register_bufferr(arangeexpandselfconfig __class__r+r,r8Ss  zGitEmbeddings.__init__Nr input_idsr2 inputs_embedspast_key_values_lengthreturnc Cs|dur |}n|dd}|d}|dur&|jdd|||f}|dur0||}n|}|jdkr@||}||7}||}||}|S)Nr4rr1)sizer2r<r0r>r?rC) rIrMr2rNrO input_shape seq_length embeddingsr>r+r+r,forwardbs      zGitEmbeddings.forward)NNNr) r$r%r&r'r8rr(Z LongTensorr)intTensorrU __classcell__r+r+rKr,r-Ps$ r-cseZdZdfdd ZdejdejfddZ     dd ejd eejd eejd ee d ee dee de ejfddZ Z S)GitSelfAttentionNcsVt|j|jdkrt|dstd|jd|jd||_|dur1td|j j d|j|_t |j|j|_ |j|j |_ t |jj|jjdd |_|jdura|j|j9_t|j|j |_t|j|j |_t|j|j |_t|j|_|pt|d d |_|jd ks|jd kr|j|_td|jd |j |_dSdS)NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()zInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.rr0r1 relative_keyrelative_key_query) r7r8r;num_attention_headshasattr ValueError layer_idxlogger warning_oncerLr$rVattention_head_size all_head_size vision_config image_size patch_sizeimage_patch_tokensnum_image_with_embeddingrLinearquerykeyvaluerAZattention_probs_dropout_probrCrDr0r=r9distance_embeddingrIrJr0rarKr+r,r8s:   zGitSelfAttention.__init__xrPcCs6|dd|j|jf}||}|ddddS)Nr4rr[rr )rQr^rdviewpermute)rIrqZ new_x_shaper+r+r,transpose_for_scoress z%GitSelfAttention.transpose_for_scoresFr"attention_mask head_maskpast_key_valueoutput_attentionspixel_values_presentc Cs||}|r |jnd}|||} |||} |durt|| dddd|dddf| dddd|dddf|j\} } tj| ddddd|ddf| gdd} tj| ddddd|ddf| gdd} ||} t | | dd}|j dks|j dkr | j d| j d}}|durtj |dtj|jd dd}ntj|tj|jd dd}tj|tj|jd dd}||}|||jd}|j| jd }|j dkrtd | |}||}n|j dkr td | |}td | |}|||}|t|j}|dur||}tjj|dd}||}|dur1||}t || }|dddd }| dd|j!f}||}|rY||fn|f}||f}|S)Nrr[dimr4r\r]rdtypedevicer~zbhld,lrd->bhlrzbhrd,lrd->bhlrr )"rlrirtrmrnupdaterar(catmatmul transposer0shapeZtensorlongrrrrFror=tor~Zeinsummathsqrtrdr functionalsoftmaxrCrs contiguousrQre)rIr"rurvrwrxryZmixed_query_layercutoffZ key_layerZ value_layerZkey_layer_pastZvalue_layer_pastZ query_layerZattention_scoresZ query_lengthZ key_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_probsZ context_layerZnew_context_layer_shapeoutputsr+r+r,rUsX @..           zGitSelfAttention.forwardNNNNNFF)r$r%r&r8r(rWrtrr)r boolrrUrXr+r+rKr,rYs."rYc8eZdZfddZdejdejdejfddZZS) GitSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr/) r7r8rrkr;denser?r@rArBrCrHrKr+r,r8 zGitSelfOutput.__init__r" input_tensorrPcC&||}||}|||}|SNrrCr?rIr"rr+r+r,rU  zGitSelfOutput.forwardr$r%r&r8r(rWrUrXr+r+rKr,r $reagercsxeZdZdfdd ZddZ     ddejdeejd eejd ee d ee d ee d e ejfddZ Z S) GitAttentionNcs6tt|j|||d|_t||_t|_dS)N)r0ra) r7r8GIT_SELF_ATTENTION_CLASSES_attn_implementationrIroutputset pruned_headsrprKr+r,r8 s   zGitAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrrz)lenrrIr^rdrrrlrmrnrrreunion)rIheadsindexr+r+r, prune_headss zGitAttention.prune_headsFr"rurvrwrxryrPc Cs:|||||||}||d|}|f|dd} | S)Nrr)rIr) rIr"rurvrwrxryZ self_outputsattention_outputrr+r+r,rU%s zGitAttention.forwardrr)r$r%r&r8rr(rWrr)r rrrUrXr+r+rKr,r s. rc2eZdZfddZdejdejfddZZS)GitIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r7r8rrkr;intermediate_sizer isinstance hidden_actstrr intermediate_act_fnrHrKr+r,r8=s   zGitIntermediate.__init__r"rPcCs||}||}|Sr)rrrIr"r+r+r,rUEs  zGitIntermediate.forwardrr+r+rKr,r<s rcr) GitOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r7r8rrkrr;rr?r@rArBrCrHrKr+r,r8MrzGitOutput.__init__r"rrPcCrrrrr+r+r,rUSrzGitOutput.forwardrr+r+rKr,rLrrcsxeZdZdfdd Z     ddejdeejdeejdeed ee d ee d e ejfd d Z ddZ Z S)GitLayerNcs>t|j|_d|_t||d|_t||_t||_ dS)Nr)ra) r7r8chunk_size_feed_forward seq_len_dimr attentionr intermediaterr)rIrJrarKr+r,r8[s  zGitLayer.__init__Fr"rurvrwrxryrPc Cs^|j||||||d}|d}|dd} |d} t|j|j|j|} | f| } | | f} | S)N)rxrwryrrr4)rrfeed_forward_chunkrr) rIr"rurvrwrxryZself_attention_outputsrrZpresent_key_value layer_outputr+r+r,rUcs"    zGitLayer.forwardcCs||}|||}|Sr)rr)rIrZintermediate_outputrr+r+r,rs  zGitLayer.feed_forward_chunkrr)r$r%r&r8r(rWrr)r rrrUrrXr+r+rKr,rZs.  "rcseZdZfddZ        ddejdeejdeejd eee e e ejfd ee d ee d ee d ee dee dee eje ffddZ ZS) GitEncoderc:t|_tfddtjD|_d|_dS)Ncsg|]}t|qSr+)r).0irJr+r, sz'GitEncoder.__init__..F) r7r8rJr ModuleListrangenum_hidden_layerslayergradient_checkpointingrHrKrr,r8   zGitEncoder.__init__NFTr"rurvpast_key_values use_cacherxoutput_hidden_statesry return_dictrPc  CsZ|jr|jr|rtdd}d} |r,t|ts,d} |dur"t}n t|}td|r0dnd} |r6dnd} d} t|j D]C\}}|rJ| |f} |durR||nd}|jrf|jrf| |j |||||}n |||||||}|d}|ry|d} |r| |df} q?|r| |f} |r| nd}| r| }| st d d ||| | fDSt||| | d S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FTzWe detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)r+rr4rcs|] }|dur|VqdSrr+rvr+r+r, sz%GitEncoder.forward..r!rr"r#)rtrainingrbrcrr r Zfrom_legacy_cache enumerater_gradient_checkpointing_func__call__Zto_legacy_cachetupler)rIr"rurvrrrxrryrZreturn_legacy_cacheZall_hidden_statesZall_self_attentionsZnext_decoder_cacherZ layer_moduleZlayer_head_mask layer_outputsZ next_cacher+r+r,rUs           zGitEncoder.forward)NNNNFFFT)r$r%r&r8r(rWrr)rr rrrrUrXr+r+rKr,rs>     rc@s(eZdZeZdZdZdZdZddZ dS)GitPreTrainedModelgitTcCst|tr)tjj|jd|jjdtjj|jj |jjdtjj|j j |jjdt|tj rI|j j jd|jjd|j durG|j j dSdSt|tjrl|j j jd|jjd|jdurj|j j |jdSdSt|tjr|j j |j j ddSdS)zInitialize the weights)meanstd)rNg?)rGitVisionEmbeddingsrinitZnormal_class_embeddingrJZinitializer_rangepatch_embeddingweightposition_embeddingrkdatabiasZzero_r9r.r?Zfill_)rImoduler+r+r, _init_weightss$       z GitPreTrainedModel._init_weightsN) r$r%r&r config_classZbase_model_prefixZsupports_gradient_checkpointingZ_supports_cache_classZ_supports_quantized_cacherr+r+r+r,rs rcsXeZdZdeffdd Zdejdededejfdd Zdd ej dejfd d Z Z S)rrJcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF)Z in_channelsZ out_channelsZ kernel_sizeZstriderr[rr2r3r5)r7r8rJr; embed_dimrgrhr Parameterr(ZrandnrZConv2dZ num_channelsr num_patches num_positionsr9rrErFrGrHrKr+r,r8s"  "zGitVisionEmbeddings.__init__rTheightwidthrPc Cs|jdd}|jjd}|jdd}tjs(||kr(||kr(||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr4g?r r[ZbicubicF)rQmodeZ align_cornersrz)rrrZ unsqueezer(Zjit is_tracingr2rhrreshapersrrZ interpolaterrr) rIrTrrrrrZclass_pos_embedZpatch_pos_embedr{Z new_heightZ new_widthZsqrt_num_positionsr+r+r,interpolate_pos_encoding$s*      z,GitVisionEmbeddings.interpolate_pos_encodingF pixel_valuesc Cs|j\}}}}|s&||jks||jkr&td|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r[| | | ||} | S| ||j} | S) NzInput image size (*z) doesn't match model ().rr[rr4rz)rrgr`rrr~rflattenrrrGr(rrrr2) rIrr batch_size_rrZ target_dtypeZ patch_embedsZ class_embedsrTr+r+r,rUMs  zGitVisionEmbeddings.forwardF) r$r%r&rr8r(rWrVrr)rUrXr+r+rKr,r s )rcr) GitVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSr) r7r8rJr r activation_fnrrkr;rfc1fc2rHrKr+r,r8as  zGitVisionMLP.__init__r"rPcCs"||}||}||}|Sr)rrrrr+r+r,rUhs   zGitVisionMLP.forwardrr+r+rKr,r`s rrrrlrmrnruscalingrCc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr4r|)r{r~)prrr[) r(rrrrrZfloat32rr~rCrr) rrlrmrnrurrCkwargs attn_weights attn_outputr+r+r,eager_attention_forwardps  rcsheZdZdZfddZ   d dejdeejdeejd eed e ejeejff d d Z Z S)GitVisionAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rgF)r7r8rJr;rr^ num_headshead_dimr`scaleZattention_dropoutrC is_causalrrkk_projv_projq_projout_projrHrKr+r,r8s$   zGitVisionAttention.__init__NFr"rucausal_attention_maskrxrPc CsL|j\}}}||}||} ||} ||||j|jdd}| |||j|jdd} | |||j|jdd} |jj dkrY|durR|durR||}n |durX|}n|du|_ t } |jj dkrz|jj dkrt|rtt dnt|jj } | ||| | ||j |j|jsdn|jd \} } | |||} || } |sd} | | fS) z#Input shape: Batch x Time x Channelrr[Zflash_attention_2NrZsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)rrrC)rr rrrrrrrrJrrrrbrcrrrrCrrr )rIr"rur rxrrSrZquerieskeysvaluesZattention_interfacerrr+r+r,rUsH          zGitVisionAttention.forward)NNF) r$r%r&r'r8r(rWrrrrUrXr+r+rKr,rs" rc sTeZdZdeffdd Z d dejdejdejdeed e ej f d d Z Z S) GitVisionEncoderLayerrJcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSr) r7r8r;rr self_attnrr?r@ layer_norm1rmlp layer_norm2rHrKr+r,r8s   zGitVisionEncoderLayer.__init__Fr"rur rxrPcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r0||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r"rur rx)rrrr)rIr"rur rxZresidualrrr+r+r,rUs"     zGitVisionEncoderLayer.forwardr) r$r%r&rr8r(rWrrrr)rUrXr+r+rKr,rs rcsteZdZdZdeffdd Z     ddeejdeejdee d ee d ee d e e e ff d d Z ZS)GitVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`GitVisionEncoderLayer`]. Args: config: GitVisionConfig rJcr)Ncsg|]}tqSr+)rrrrr+r,rsz-GitVisionEncoder.__init__..F) r7r8rJrrrrlayersrrHrKrr,r8rzGitVisionEncoder.__init__Nrur rxrrrPc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|} t|jD]1\} } |r<|| f}|jrM|jrM|| j | |||} n| | |||d} | d} |rb|| df}q1|rj|| f}|sxt dd| ||fDSt | ||dS) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr+)rxrrcsrrr+rr+r+r,rdsz+GitVisionEncoder.forward..r!r"r#) rJrxruse_return_dictrrrrrrrr) rIrNrur rxrrZencoder_statesZall_attentionsr"idxZ encoder_layerrr+r+r,rUsF&     zGitVisionEncoder.forward)NNNNN)r$r%r&r'rr8rr(rWrrrrrUrXr+r+rKr,r s*  rcsreZdZdeffdd Ze     ddeejdee dee d ee d ee d e e e ff d d Z ZS)GitVisionTransformerrJcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) r7r8rJr;rrTrr?r@ pre_layrnormrencoderpost_layernorm)rIrJrrKr+r,r8ls   zGitVisionTransformer.__init__NFrrxrrrrPc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|j||d}||}|j||||d}|d}||}|sO|f|ddSt ||j |j dS)Nz You have to specify pixel_valuesr)rNrxrrrrr) rJrxrrr`rTrrrrr"r#) rIrrxrrrr"encoder_outputsr!r+r+r,rUvs.   zGitVisionTransformer.forwardNNNFN)r$r%r&rr8rrr(r)rrrrrUrXr+r+rKr,rjs*  rzY The vision model from CLIP, used in GIT, without any head or projection on top. )Z custom_introcseZdZeZdZdeffdd ZdejfddZ e    dde e j d e ed e ed ed e edeeeff ddZZS)GitVisionModelrrJcs"t|t||_|dSr)r7r8r vision_model post_initrHrKr+r,r8s   zGitVisionModel.__init__rPcCs |jjjSr)r!rTrrIr+r+r,get_input_embeddings z#GitVisionModel.get_input_embeddingsNFrxrrrcCs(|dur|n|jj}|j|||||dS)a{ Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, GitVisionModel >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = GitVisionModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```N)rrxrrr)rJrr!)rIrrxrrrr+r+r,rUszGitVisionModel.forwardr)r$r%r&rrZmain_input_namer8rModuler$rrr(r)rrrrrUrXr+r+rKr,r s0 r cs8eZdZdeffdd ZdejdejfddZZS) GitProjectionrJcs@t||_tt|jj|jtj|j|jj d|_ dSr) r7r8rJrZ Sequentialrkrfr;r?r@visual_projectionrHrKr+r,r8s  zGitProjection.__init__rTrPcCs ||Sr)r()rIrTr+r+r,rUr%zGitProjection.forward) r$r%r&rr8r(rWrUrXr+r+rKr,r'sr'zy The bare GIT Model transformer consisting of a CLIP image encoder and text decoder outputting raw hidden-states cseZdZfddZddZddZddZd ed ej d ej d ej fd dZ d!ddZ e            d"deej deej deej deej deej deej deeeeejfdeedeedeededeed eeej effdd ZZS)#GitModelcsrt|_t|_tj|_t|_ t |_ j dur3t fddtj D|_|dS)Nc3s(|]}ttddjjVqdS)rN)rrr(zerosrfr;rrr+r,rs  z$GitModel.__init__..)r7r8rJr-rTr rf image_encoderrrr'r(rjrZ ParameterListrimg_temperal_embeddingr"rHrKrr,r8s       zGitModel.__init__cCs|jjSrrTr<r#r+r+r,r$szGitModel.get_input_embeddingscCs ||j_dSrr-)rIrnr+r+r,set_input_embeddingss zGitModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rIZheads_to_prunerrr+r+r, _prune_headsszGitModel._prune_headsrQr~rrPcCs4tjtj||||ddd}||dktd}|S)Nrr~r)Zdiagonal-inf)r(ZtriuZonesZ masked_fillfloat)rIrQr~rmaskr+r+r,_generate_future_maskszGitModel._generate_future_maskNcCs|jd}|jd}|j}|j} tj||f|| d} tj|||ftd|j| d} tj||f| |jd} |dkrLtj|jd|jd|f| |jd}tj| | fdd} tj| || fdd}tj| |fdddddf}|durtj|jd|jdfd|d}|jtj krt d tj ||jd }td||<| |jd|||||f}| }|ddddd|f}|dddddf}|||ddddd|f<|dddddddf}|S) Nrr1r2r}rrzF)Z fill_valuerz1Memory key padding mask must be a boolean tensor.r)rrr~r(r*fullr3rrrr`Z zeros_likerGclone)rItgtmemorytgt_maskrOZmemory_key_padding_maskZnum_tgtZ num_memoryrr~top_left top_right bottom_leftleftrightZfull_attention_maskZzero_negative_infinityZ origin_leftrr+r+r,create_attention_masksP      zGitModel.create_attention_maskFrMrur2rrvrNrrrxrrrc  Cs| dur| n|jj} | dur| n|jj} |dur|n|jj}| dur$| n|jj} |dur4|dur4td|durC||||} n|durP|dd} ntd| d}d}|durpt|t sl|ddj dn| }| ||jj }d}|dur|jdkr|j|| d j}n=|jd krg}t|j dD]"}|j|dd|ddddf| d j}||j|7}||qtj|dd }ntd ||}|j||||d }|durtj|j dd|j df|j|jd}||d|ddd}tj||fdd }|||j|j}|j||||d}|dur\t||j| dd|j}|dkrB|dddd| dddf}n|dddd| d d| d df|7<|j |||||| | | |dud }|d}| s||f|ddSt!||j"|j#|j$dS)a Examples: ```python >>> from transformers import AutoProcessor, AutoModel >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = AutoModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "this is an image of two cats" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer4z5You have to specify either input_ids or inputs_embedsrrr[rrzz#pixel_values must be of rank 4 or 5)rMr2rNrOr})r8r9r:rO)Ztgt_len)rurvrrrxrrryr)%rJrxrrrr`Z%warn_if_padding_and_no_attention_maskrQrr rget_seq_lengthZ get_head_maskrndimr+r!rr,appendr(rr(rTr*r~rrepeatr5r@rrrrrr"r#)rIrMrur2rrvrNrrrxrrrrRrSrOZprojected_visual_featuresZvisual_featuresZ frame_idxZvisual_features_frameZembedding_outputr"r:Zcombined_attention_maskZexpanded_attn_maskrsequence_outputr+r+r,rUFs%          $4 zGitModel.forwardr) NNNNNNNNNNFN)r$r%r&r8r$r.r0rVr(r~rrWr5r@rrrr rr)rrrrUrXr+r+rKr,r)s^   2     r)z` GIT Model with a `language modeling` head on top for autoregressive language modeling. c seZdZdgZfddZddZddZe            dd ee j d ee j d ee j d ee j dee j dee j dee j dee e e e j fdeedeedeededeede ee j effddZ dddZddZZS) GitForCausalLMz output.weightcs4t|t||_t|j|j|_| dSr) r7r8r)rrrkr;r:rr"rHrKr+r,r8s   zGitForCausalLM.__init__cCs|jSrrr#r+r+r,get_output_embeddingssz$GitForCausalLM.get_output_embeddingscCs ||_dSrrI)rIZnew_embeddingsr+r+r,set_output_embeddingsr%z$GitForCausalLM.set_output_embeddingsNFrMrur2rrvrNlabelsrrrxrrrrPcKs| dur| n|jj} |durd} |j|||||||| | | | | d }|d}||}d}|durl|jjjdjjj}|dd|dddf }|ddddf }|j | d|jj | dfd|jj i|}| s|f|dd}|dur|f|S|St |||j|j|jdS) a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` Examples: Image captioning example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-coco") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_caption) two cats sleeping on a pink blanket next to remotes. ``` Visual question answering (VQA) example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> from huggingface_hub import hf_hub_download >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-textvqa") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-textvqa") >>> file_path = hf_hub_download(repo_id="nielsr/textvqa-sample", filename="bus.png", repo_type="dataset") >>> image = Image.open(file_path).convert("RGB") >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> question = "what does the front of the bus say at the top?" >>> input_ids = processor(text=question, add_special_tokens=False).input_ids >>> input_ids = [processor.tokenizer.cls_token_id] + input_ids >>> input_ids = torch.tensor(input_ids).unsqueeze(0) >>> generated_ids = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_length=50) >>> print(processor.batch_decode(generated_ids, skip_special_tokens=True)) ['what does the front of the bus say at the top? special'] ``` Video captioning example: ```python >>> import av >>> import numpy as np >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-vatex") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-vatex") >>> # set seed for reproducibility >>> np.random.seed(45) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # load video >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample frames >>> num_frames = model.config.num_image_with_embedding >>> indices = sample_frame_indices( ... clip_len=num_frames, frame_sample_rate=4, seg_len=container.streams.video[0].frames ... ) >>> frames = read_video_pyav(container, indices) >>> pixel_values = processor(images=list(frames), return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> print("Generated caption:", processor.batch_decode(generated_ids, skip_special_tokens=True)) Generated caption: ['a woman is sitting at a table and she is talking about the food she is holding.'] ``` NF) rur2rrvrNrrrxrrrrr4rr:)losslogitsrr"r#)rJrrrrrrrIrirZ loss_functionrrr:rrr"r#)rIrMrur2rrvrNrLrrrxrrrrrrGrNrMZnum_image_tokensZshifted_logitsrr+r+r,rUsV zGitForCausalLM.forwardc Ksv|dur#|}|jd|kr|}n|jdd}|dd|df}|j}|dur/||}|||dd||dS)Nrr)rMrurrr)rCrZnew_onesget) rIrMrrurrZ past_lengthZremove_prefix_lengthrRr+r+r,prepare_inputs_for_generations  z,GitForCausalLM.prepare_inputs_for_generationcs.d}|D]}|tfdd|Df7}q|S)Nr+c3s$|] }|d|jVqdS)rN)Z index_selectrr)rZ past_statebeam_idxr+r,rs"z0GitForCausalLM._reorder_cache..)r)rIrrRZreordered_pastZ layer_pastr+rQr,_reorder_caches zGitForCausalLM._reorder_cache) NNNNNNNNNNNFN)NNN)r$r%r&Z_tied_weights_keysr8rJrKrrr(rWrr rrrrrUrPrSrXr+r+rKr,rHsh        E rH)rHr)rr )r)Gr'r dataclassesrtypingrrrrrr(Ztorch.utils.checkpointrZ activationsr Z cache_utilsr r Z generationr Zmodeling_attn_mask_utilsrZmodeling_outputsrrrrZmodeling_utilsrrZ pytorch_utilsrrrutilsrrrrZconfiguration_gitrrZ get_loggerr$rbrr&r-rYrrrrrrrrrrrWr3rrrrrr r'r)rH__all__r+r+r+r,s      0v31dS P3a65