o Zh@sddlZddlmZddlmZmZmZmZm Z ddl Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZmZmZdd lmZmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%d dl&m'Z'ddl(m)Z)m*Z*e$+e,Z-edGddde j.Z/ dCde j.de j0de j0de j0dee j0de1de1fddZ2Gddde j.Z3e!Gd d!d!eZ4eGd"d#d#eZ5Gd$d%d%e j.Z6Gd&d'd'e j.Z7Gd(d)d)e j.Z8e j9e/d*Z:Gd+d,d,e j.Z;Gd-d.d.e j.ZGd3d4d4e j.Z?eGd5d6d6eZ@e!d7d8Gd9d:d:e>ZAeGd;d<dd>eeZCe!d?d8Gd@dAdAe>eZDgdBZEdS)DN) dataclass)CallableListOptionalTupleUnion)ACT2FN)GenerationMixin)use_kernel_forward_from_hub)FlashAttentionKwargs)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargs ModelOutputauto_docstringcan_return_tupleis_torchdynamo_compilinglogging torch_int) AutoModel)InternVLConfigInternVLVisionConfigZRMSNormcs.eZdZdfdd ZddZddZZS) InternVLVisionRMSNormư>cs&ttt||_||_dS)zD InternVLVisionRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__]/var/www/auris/lib/python3.10/site-packages/transformers/models/internvl/modeling_internvl.pyr"7s  zInternVLVisionRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)NrT)Zkeepdim) dtypetor%Zfloat32powmeanZrsqrtr(r')r) hidden_statesZ input_dtypeZvariancer.r.r/forward?s  zInternVLVisionRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler'shaper(r)r.r.r/ extra_reprFsz InternVLVisionRMSNorm.extra_repr)r )__name__ __module__ __qualname__r"r6r: __classcell__r.r.r,r/r5srmodulequerykeyvalueattention_maskscalingdropoutc Ks|}|} t||dd|} |dur+|ddddddd|jdf} | | } tjj| dd} tjj| ||jd} t| | } | dd } | | fS)Nrrr0dim)ptrainingr) r%matmul transposer8r# functionalZsoftmaxrFrK contiguous) r@rArBrCrDrErFkwargs key_states value_states attn_weights causal_mask attn_outputr.r.r/eager_attention_forwardJs & rVc sXeZdZdZdeffdd Z  d dejdeejdeejd e e fd d Z Z S) InternVLVisionAttentionz+Attention Class for InternVL Vision EncoderconfigcsBt||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ |j }|j }d|_ tj|j|j|j|jd|_tj|j|j|j|jd|_tj|j|j|j|jd|_t|j|j|_|dkrt|nt|_ |rt|jnt|_|rt|j|_dSt|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFbiasr)r!r"rXr* embed_dimZnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutprojection_dropoutZ use_qk_norm is_causalr#LinearZattention_biasq_projk_projv_projprojection_layerDropoutIdentityrq_normk_norm)r)rXZ proj_dropoutZqk_normr,r.r/r"hs.   "z InternVLVisionAttention.__init__Nr5rDoutput_attentionsrPcKsF|\}}}||}||} ||} ||}|| } ||||j|j dd}| |||j|j dd} | |||j|j dd} t } |j j dkrj|j j dkrd|ddrdtdnt|j j } | ||| | |f|jsvdn|j|jdd |\} } | |||j} || }||}|r|| f}|S|df}|S) NrreagerZsdparlFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r?)rFrErb)sizerdrerfrjrkreshaper\r]rMviewrVrXZ_attn_implementationgetloggerZ warning_oncerrKr`r_r[rgra)r)r5rDrlrP batch_sizeseq_len_Z query_statesrQrRZattention_interfacerUrSoutputoutputsr.r.r/r6sH           zInternVLVisionAttention.forward)NN) r;r<r=__doc__rr"r%Tensorrrr r6r>r.r.r,r/rWesrWc@s2eZdZeZdZdZdZdgZdZ dZ ddZ dS)InternVLVisionPreTrainedModelZinternvl_vision pixel_valuesTInternVLVisionLayercCs:t|tjtjtjfr%|jjjd|jj d|j dur#|j j dSdSt|tj rH|jjjd|jj d|j durF|jj|j  dSdSt|tjr]|j j |jjddSt|tr|jj |jdurs|jj |jdur|jj dSdSt|tr|jj|jj|jj|jjdSdS)zInitialize the weightsr?r4stdN?) isinstancer#rcConv2dZConvTranspose2dr'datanormal_rXinitializer_rangerZzero_Z EmbeddingZ padding_idx LayerNormfill_InternVLVisionEmbeddings cls_token mask_tokenposition_embeddingsr|lambda_1layer_scale_init_valuelambda_2)r)r@r.r.r/ _init_weightss0           z+InternVLVisionPreTrainedModel._init_weightsN) r;r<r=r config_classbase_model_prefixZmain_input_namesupports_gradient_checkpointingZ_no_split_modules_supports_sdpa_supports_flash_attn_2rr.r.r.r/rzs rzc@seZdZdZdS)$InternVLVisionModelOutputWithPoolinga Class for outputs of [`InternVLVisionModel`]. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N)r;r<r=rxr.r.r.r/rsrcs6eZdZdZfddZdejdejfddZZS)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}|d|d|d|d}|d|d|d|df}||_||_||_||_||_tj ||||d|_ dS)Nrr)Z kernel_sizeZstride) r!r" image_size patch_size num_channelsr* num_patches patch_shaper#r projection)r)rXrrrr*rrr,r.r/r"s   z&InternVLVisionPatchEmbeddings.__init__r{returnc Cs^|j\}}}}||jkrtd||}|jd|jd}}|ddd}|||ffS)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrr)r8rr^rflattenrM) r)r{rsrheightwidth embeddings patch_height patch_widthr.r.r/r6 s   z%InternVLVisionPatchEmbeddings.forward) r;r<r=rxr"r%ryr6r>r.r.r,r/rs rcsleZdZdZdeddffdd Zdejded edejfd d Z dd ejd e ej dejfddZ Z S)rzc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. rXrNcstttdd|j|_|jr!ttdd|j|_ nd|_ t ||_ |j |_ t |jtjjr8|jn|j|jf|_|j j}|jrUttd|d|j|_nd|_t|j|_dS)Nr)r!r"r#r$r%Zzerosr*rZuse_mask_tokenrrpatch_embeddingsrrr collectionsabcIterablerZ use_absolute_position_embeddingsrrhhidden_dropout_probrF)r)rXrr,r.r/r" s   z!InternVLVisionEmbeddings.__init__rrrc Cs|jdd}|jjdd}tjs||kr||kr|jS|jddddf}|jddddf}|jd}||jd} ||jd} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr0r?rrZbicubicF)rnmodeZ align_cornersrH)r8rr%Zjit is_tracingrrropermuter#rNZ interpolaterpcat) r)rrrrZ num_positionsZclass_pos_embedZpatch_pos_embedrIZ new_heightZ new_widthZsqrt_num_positionsr.r.r/interpolate_pos_encoding6s(   z1InternVLVisionEmbeddings.interpolate_pos_encodingr{bool_masked_poscCs|j\}}}}||\}\}}|\} } }|dur5|j| | d} |d| } |d| | | }|j| dd} tj | |fdd}|j durT|| |||}| |}|||ffS)Nr0rrH) r8rrnrexpand unsqueezeZtype_asrr%rrrrF)r)r{rrurrrrrrsrtZ mask_tokenswZ cls_tokensr.r.r/r6^s   z InternVLVisionEmbeddings.forwardN)r;r<r=rxrr"r%ryintrr BoolTensorr6r>r.r.r,r/rs+rcs2eZdZfddZdejdejfddZZS)InternVLVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSr) r!r"rXr Z hidden_act activation_fnr#rcr*Zintermediate_sizefc1fc2r)rXr,r.r/r"ys  zInternVLVisionMLP.__init__r5rcCs"||}||}||}|Sr)rrr)r)r5r.r.r/r6s   zInternVLVisionMLP.forward)r;r<r=r"r%ryr6r>r.r.r,r/rxs r) layer_normZrms_normc s`eZdZdZdeddffdd Z d dejd ede e eje ejejfffd d Z Z S) r|z?This corresponds to the Block class in the timm implementation.rXrNcst|j|_d|_t||_t||_t|j |j |j d|_ t|j |j |j d|_ |j}tj|t|j dd|_tj|t|j dd|_t|j|_dS)Nrr+T)Z requires_grad)r!r"Zchunk_size_feed_forwardZ seq_len_dimrW attentionrmlpNORM2FNZ norm_typer*layer_norm_epslayernorm_beforelayernorm_afterrr#r$r%r&rrrhrrF)r)rXZ init_valuesr,r.r/r"s   zInternVLVisionLayer.__init__Fr5rlcCsl|j|||d\}}|j|}||}||}||}||}|jdur.|j|}||}||fS)N)rl)rrrrrrFr)r)r5rlZattention_outputZattention_weightsZ layer_outputr.r.r/r6s      zInternVLVisionLayer.forward)F) r;r<r=rxrr"r%ryboolrrr6r>r.r.r,r/r|sr|c sTeZdZdeddffdd Ze  d dejded ede e e ffd d Z Z S) InternVLVisionEncoderrXrNcs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr.)r|).0irXr.r/ sz2InternVLVisionEncoder.__init__..F) r!r"rXr#Z ModuleListrangeZnum_hidden_layerslayergradient_checkpointingrr,rr/r"s   zInternVLVisionEncoder.__init__Fr5rloutput_hidden_statesc Cs|rdnd}|r dnd}t|jD],\}}|r||f}|jr+|jr+||j||}n|||}|d}|r=||df}q|rE||f}t|||dS)Nr.rrlast_hidden_stater5 attentions) enumeraterrrKZ_gradient_checkpointing_func__call__r ) r)r5rlrZall_hidden_statesZall_self_attentionsrZ layer_moduleZ layer_outputsr.r.r/r6s*      zInternVLVisionEncoder.forward)FF)r;r<r=rr"rr%ryrrr7r r6r>r.r.r,r/rs rcsteZdZdeddffdd ZddZee   ddej d e ej d e e d e e de eeff d d ZZS)InternVLVisionModelrXrNcsTt|||_t||_t||_|jrt ntj |j |j d|_ |dS)Nr)r!r"rXrrrencoderZuse_mean_poolingr#rirr*r layernorm post_initrr,r.r/r"s    zInternVLVisionModel.__init__cC|jjSr)rrr9r.r.r/get_input_embeddingssz(InternVLVisionModel.get_input_embeddingsr{rrlrc Csn|dur|n|jj}|dur|n|jj}|j||d\}}|j|||d}|d}||}t||j|jdS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). N)r)rlrrr) rXrlrrrrrr5r) r)r{rrlrZembedding_outputruZencoder_outputsZsequence_outputr.r.r/r6s   zInternVLVisionModel.forward)NNN)r;r<r=rr"rrrr%ryrrrrr7rr6r>r.r.r,r/rs& rc@s<eZdZeZdZdZdZdZdZ dZ dZ dZ dZ ddZdS)InternVLPreTrainedModelTpast_key_valuescCst|jd|jj}t|tjr)|jjj d|d|j dur'|j j dSdSt|tj r>|j j |jj ddSdS)Nrr?r}r)getattrrXZget_text_configrrr#rcr'rrrZrrr)r)r@r~r.r.r/r(s    z%InternVLPreTrainedModel._init_weightsN)r;r<r=rrrrZ_skip_keys_device_placementZ_supports_cache_classrrZ_supports_quantized_cacheZ_supports_static_cacheZ_supports_attention_backendrr.r.r.r/rs rcs*eZdZdeffdd ZddZZS)InternVLMultiModalProjectorrXcsztt|jjtd|jd|_t |jjtd|jd|j j|_ t |j |_t |j j|j j|_dS)Nrr)r!r"r#r vision_configr*rdownsample_ratiorrc text_configlinear_1r Zprojector_hidden_actactlinear_2rr,r.r/r"5s " z$InternVLMultiModalProjector.__init__cCs,||}||}||}||}|Sr)rrrr)r)image_featuresr5r.r.r/r6>s    z#InternVLMultiModalProjector.forward)r;r<r=rr"r6r>r.r.r,r/r4s rc@s$eZdZUdZdZeejed<dS)InternVLModelOutputWithPasta Base class for InternVL outputs, with hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) r;r<r=rxrrr% FloatTensor__annotations__r.r.r.r/rFs rzx The InternVL model which consists of a vision backbone and a language model, without a language modeling head. )Z custom_introc%s<eZdZddiZdeffdd ZddZdd Zd ej d e e e e fd e fd dZee              d$dejd ej deejdeejdee ej deej d ee e e e fd ee deedeedeedeedeejdejdeede eeff ddZd%d ejd!efd"d#ZZS)& InternVLModelzlanguage_model.modellanguage_modelrXcs>t|t|j|_t||_t|j|_ | dSr) r!r"rZ from_configr vision_towerrmulti_modal_projectorrrrrr,r.r/r"os   zInternVLModel.__init__cC |jSr)rrr9r.r.r/rw z"InternVLModel.get_input_embeddingscC|j|dSr)rset_input_embeddingsr)rCr.r.r/rzz"InternVLModel.set_input_embeddingsr{vision_feature_layervision_feature_select_strategyc Ks|jj}|dkr|j|dj}n |j|dj|}|dkr*|ddddddf}|jd}t|d}|jd} || ||d}|j ||d}|| d|jd}| |}|S) a% Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `List[int]`): Layer index or list of layer indices to extract features from. Returns: vision_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`. r0)r{defaultNrrr) scale_factor) rXrrrZ vision_modelr5r8rro pixel_shuffler) r)r{rrrPrvision_featureschannelsZ feature_sizersr.r.r/get_image_features}s    z InternVLModel.get_image_featuresN input_idsrD position_idsr inputs_embeds use_cacherlr return_dictcache_position image_sizesrPrc Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|dur.|n|jj}|du|duAr>td|durH||}|dur|j||||d}|durx||t j |jj t j |j dk}|jddjddd}n||jj kd}|||j }||jj k}ts|||kr||jj k}|jd|jd}td|d |||j |j}|||}|jd ||||| | | d | d |}t|j|j|j|j|dur|d Sdd S)Nz:You must specify exactly one of input_ids or inputs_embeds)r{rrr)r1devicerrHrr0z6Image features and image tokens do not match: tokens: z , features T) rDrrrrrlrrr)rrr5rrr.)rXrlruse_return_dictrrr^rrr%ZtensorZimage_token_idlongrsumrZ expand_asr2rZnumelr8r1Zmasked_scatterrrrrr5r)r)rr{rDrrrrrrrlrrrrrPrZspecial_image_maskZn_image_tokensZn_image_featuresrwr.r.r/r6sv     zInternVLModel.forwardrrrc Cs|\}}}}||dks||dkrtd|||t||t||}|dddd}||t||t||t||d}|dddd}|S)a&Perform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr)rnr^rprrrO)r)rrrsrrrr.r.r/rs $zInternVLModel.pixel_shuffle)NNNNNNNNNNNNNN)r)r;r<r=_checkpoint_conversion_mappingrr"rrr%rrrrstrrrr LongTensorrryrrr rrr6floatrr>r.r.r,r/rgs| -       Vrc@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZeejed<dS) InternVLCausalLMOutputWithPasta Base class for InternVL causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitsrr5rr)r;r<r=rxr rr%rrr rrr5rrrr.r.r.r/r &s r c@s eZdZdS)KwargsForCausalLMN)r;r<r=r.r.r.r/r Ns r zV The INTERNVL model which consists of a vision backbone and a language model. c)seZdZdddddZdgZdeffdd Zd d Zd d Zde j fddZ ddZ e ddZe ddZe ddZee                d6dejdejdeejdeejdeeejd eejd!eeeeefd"eed#eejd$eed%eed&eed'eed(eejd)eeejfd*eejd+eedeee ff$d,d-Z!      d7fd.d/ Z"e#dejd0ed1ed2ej$d(ejd3ef d4d5Z%Z&S)8 InternVLForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)z^language_model.modelz ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightrXcs<t|t||_tj|jj|jjdd|_ | dS)NFrY) r!r"rmodelr#rcrr* vocab_sizerrrr,r.r/r"_s   z)InternVLForConditionalGeneration.__init__cCrr)rrr9r.r.r/rerz5InternVLForConditionalGeneration.get_input_embeddingscCrr)rrrr.r.r/rhrz5InternVLForConditionalGeneration.set_input_embeddingsrcCs|jSrrr9r.r.r/get_output_embeddingsksz6InternVLForConditionalGeneration.get_output_embeddingscCs ||_dSrr)r)Znew_embeddingsr.r.r/set_output_embeddingsnrz6InternVLForConditionalGeneration.set_output_embeddingscCrr)rrr9r.r.r/rrz/InternVLForConditionalGeneration.language_modelcCrr)rrr9r.r.r/rvrz-InternVLForConditionalGeneration.vision_towercCrr)rrr9r.r.r/rzrz6InternVLForConditionalGeneration.multi_modal_projectorNrrr{rDrrrrrlabelsrrlrrrlogits_to_keeprrPcKs| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|dur.|n|jj}|jd||||||||| | | d||d|}|d}t|trXt | dn|}| |dd|ddf}d}| dur}|j d|| |jj j d|}t|||j|j|j|jdS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", torch_dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```NT)rr{rDrrrrrrrlrrrrr)r rr)r r rr5rrr.)rXrlrrrrrrrslicerZ loss_functionrrr rr5rr)r)rr{rDrrrrrrrrlrrrrrrPrwr5Z slice_indicesr r r.r.r/r6~s\> z(InternVLForConditionalGeneration.forwardc  s8tj|f|||||d|} |ddkr|| d<| S)N)rrrDrrrr{)r!prepare_inputs_for_generation) r)rrrr{rDrrrPZ model_inputsr,r.r/rs  z>InternVLForConditionalGeneration.prepare_inputs_for_generationsequence_length target_lengthr1rsc KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valuer1rr)Zdiagonal)rr0r)rIr%ZfinfominfullrZtriuZarangerorcloner8r2Z masked_fill) rDrrr1rrsrPrTZ min_dtypeZ mask_lengthZ padding_maskr.r.r/5_prepare_4d_causal_attention_mask_with_cache_positions,  $ 6  zVInternVLForConditionalGeneration._prepare_4d_causal_attention_mask_with_cache_position)NNNNNNNNNNNNNNrN)NNNNNN)'r;r<r=rZ_tied_weights_keysrr"rrr#Modulerrpropertyrrrrrr%rrrryrrrrrrr rr r6r staticmethodr1rr>r.r.r,r/r Qs           sr )rzrrrr )r?)Fcollections.abcr dataclassesrtypingrrrrrr%Ztorch.nnr#Z activationsr Z generationr Z integrationsr Zmodeling_flash_attention_utilsr Zmodeling_outputsr rrZmodeling_utilsrrZprocessing_utilsrutilsrrrrrrrautorZconfiguration_internvlrrZ get_loggerr;rrr rryrrVrWrzrrrrrrr|rrrrrrr r r __all__r.r.r.r/s       $   Q#&^ 0+5 ;' q