fThXSSKrSSKJr SSKJrJrJrJrJ r SSK r SSK J r SSK Jr SSKJr SSKJr SSKJr SS KJrJrJr SS KJrJr SS KJr SS KJrJ r J!r!J"r"J#r#J$r$J%r% S SK&J'r' SSK(J)r)J*r* \$RV"\,5r-\"S5"SS\ R\55r/SAS\ R\S\ R`S\ R`S\ R`S\\ R`S\1S\14Sjjr2"SS\ R\5r3\!"SS\55r4\"S S!\55r5"S"S#\ R\5r6"S$S%\ R\5r7"S&S'\ R\5r8\ Rr\/S(.r:"S)S*\ R\5r;"S+S,\ R\5r<\!"S-S.\455r=\!"S/S0\55r>"S1S2\ R\5r?\"S3S4\55r@\!"S5S69"S7S8\>55rA\"S9S:\ 55rB"S;S<\\5rC\!"S=S69"S>S?\>\55rD/S@QrEg)BN) dataclass)CallableListOptionalTupleUnion)ACT2FN)GenerationMixin)use_kernel_forward_from_hub)FlashAttentionKwargs)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargs ModelOutputauto_docstringcan_return_tupleis_torchdynamo_compilinglogging torch_int) AutoModel)InternVLConfigInternVLVisionConfigRMSNormc8^\rSrSrSU4SjjrSrSrSrU=r$)InternVLVisionRMSNorm5c>[TU]5 [R"[R "U55UlX lg)z4 InternVLVisionRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s f/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/internvl/modeling_internvl.pyr&InternVLVisionRMSNorm.__init__7s/ ll5::k#:; #cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)NrT)keepdim) dtypetor)float32powmeanrsqrtr,r+)r- hidden_states input_dtypevariances r1forwardInternVLVisionRMSNorm.forward?sw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r3c^[URR5SUR3$)Nz, eps=)tupler+shaper,r-s r1 extra_repr InternVLVisionRMSNorm.extra_reprFs* ))*+6$2G2G1HIIr3)r,r+)gư>) __name__ __module__ __qualname____firstlineno__r&r@rF__static_attributes__ __classcell__r0s@r1r"r"5s$;JJr3r"modulequerykeyvalueattention_maskscalingdropoutc UnUn [R"XRSS55U-n Ub"USS2SS2SS2SURS24n X-n [R R U SS9n [R RXURS9n [R"X5n U RSS5R5n X4$)Nrr r5dim)ptrainingr) r)matmul transposerDr' functionalsoftmaxrUr[ contiguous) rOrPrQrRrSrTrUkwargs key_states value_states attn_weights causal_mask attn_outputs r1eager_attention_forwardrgJsJL<<';';Aq'ABWLL!$Q1.D 0@0@0D.D%DE #1 ==((2(>L==((6??([L,,|:K''1-88:K  $$r3c ^\rSrSrSrS\4U4SjjrS S\RS\ \RS\ \RS\ \ 4S jjr S r U=r$) InternVLVisionAttentionez+Attention Class for InternVL Vision Encoderconfigc$>[TU]5 XlURUlUR UlURUR -UlURUR -UR:wa&[SURSUR S35eURS-Ul URUl URnURnSUl [R"URUR UR-UR S9Ul[R"URUR UR-UR S9Ul[R"URUR UR-UR S9Ul[R"URUR5UlUS:a[R*"U5O[R,"5Ul U(a[/UR5O[R,"5UlU(a[/UR5Ulg[R,"5Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFbiasr)r%r&rkr. embed_dimnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutprojection_dropout use_qk_norm is_causalr'Linearattention_biasq_projk_projv_projprojection_layerDropoutIdentityr"q_normk_norm)r-rk proj_dropoutqk_normr0s r1r& InternVLVisionAttention.__init__hs  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!900 $$ii0NU[UjUjk ii0NU[UjUjk ii0NU[UjUjk " $..$.. I>JQ>N"**\":TVT_T_Ta?F+DNN;BKKM ?F+DNN; BKKM r3r=rSoutput_attentionsrac 8UR5upVnURU5nURU5n URU5n UR U5nUR U 5n UR XVURUR5RSS5nU R XVURUR5RSS5n U RXVURUR5RSS5n [n URRS:wadURRS:Xa-URSS5(a[R!S5 O["URRn U "UUU U U4UR$(dSO UR&UR(SS .UD6upU R XVUR*5n UR-U 5nUR/U5nU(aX4nU$US4nU$) NrreagersdparFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rUrTrx)sizer{r|r}rrreshaperqrrr]viewrgrk_attn_implementationgetlogger warning_oncerr[rurtror~rv)r-r=rSrra batch_sizeseq_len_ query_statesrbrcattention_interfacerfrdoutputoutputss r1r@InternVLVisionAttention.forwards"/!3!3!5 Q{{=1 [[/ {{=1 {{<0 [[, #++JQUQ^Q^_iijkmno '' T^^T]][eefgijk #((dnndmm\ffghjkl (? ;; + +w 6{{//69fjjI\^c>d>d##L '>dkk>^>^&_#$7      % $}}C$2H2HJJ %  % ! "))*t~~N &&{3((0,=6(EKD>r3)rurkrorrrxrr|rqrvr~rr{rtr})NN)rHrIrJrK__doc__rr&r)Tensorrrr r@rLrMrNs@r1ririesi5Z3Z>2648 /||/!./$ELL1 / -. //r3ric8\rSrSr\rSrSrSrS/r Sr Sr Sr Sr g) InternVLVisionPreTrainedModelinternvl_vision pixel_valuesTInternVLVisionLayerc[U[R[R[R45(akUR R RSURRS9 URb%URR R5 gg[U[R5(axUR R RSURRS9 URb2UR R URR5 gg[U[R5(aJURR R5 UR R RS5 g[U[ 5(aUR"R R5 UR$b$UR$R R5 UR&b%UR&R R5 gg[U[(5(asUR*R RURR,5 UR.R RURR,5 gg)zInitialize the weightsrr;stdN?) isinstancer'ryConv2dConvTranspose2dr+datanormal_rkinitializer_rangernzero_ Embedding padding_idx LayerNormfill_InternVLVisionEmbeddings cls_token mask_tokenposition_embeddingsrlambda_1layer_scale_init_valuelambda_2)r-rOs r1 _init_weights+InternVLVisionPreTrainedModel._init_weightss fryy"))R5G5GH I I MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S )  8 9 9    ! ! ' ' )  ,!!&&,,.))5**//5576  3 4 4 OO & &t{{'I'I J OO & &t{{'I'I J5r3N)rHrIrJrKr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_2rrLrr3r1rrs2'L)$O&*#./N!Kr3rc\rSrSrSrSrg)$InternVLVisionModelOutputWithPoolingav Class for outputs of [`InternVLVisionModel`]. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. rN)rHrIrJrKrrLrr3r1rrsr3rcf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cH>[TU]5 URURp2URUR pTUSUS-USUS--nUSUS-USUS-4nX lX0lX@lX`lXpl[R"XEX3S9Ul g)Nrr) kernel_sizestride) r%r& image_size patch_size num_channelsr. num_patches patch_shaper'r projection) r-rkrrrr.rrr0s r1r&&InternVLVisionPatchEmbeddings.__init__s !'!2!2F4E4EJ$*$7$79K9Kk!!} 1 5*Q-:VW=:XY !!} 1 5z!} ST 7UV $$(&&))L:ir3rreturncURup#pEX0R:wa [S5eURU5nURSURSpUR S5R SS5nXgU44$)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rr r)rDrrsrflattenr]) r-rrrheightwidth embeddings patch_height patch_widths r1r@%InternVLVisionPatchEmbeddings.forward s2>2D2D/ & ,, ,w __\2 $.$4$4Q$79I9I!9Lk''*44Q: +666r3)rrrrrr) rHrIrJrKrr&r)rr@rLrMrNs@r1rrs. j 7ELL 7U\\ 7 7r3rc^\rSrSrSrS\SS4U4SjjrS\RS\ S \ S\R4S jr SS \RS \ \RS\R4S jjr SrU=r$)riz[ Construct the CLS token, position and patch embeddings. Optionally, also the mask token. rkrNc^>[TU]5 [R"[R "SSUR 55UlUR(a<[R"[R "SSUR 55Ul OSUl [U5Ul URUl [UR[R R"5(a UROURUR4UlURR$nUR&(a?[R"[R "SUS-UR 55UlOSUl[R*"UR,5Ulg)Nr)r%r&r'r(r)zerosr.ruse_mask_tokenrrpatch_embeddingsrrr collectionsabcIterabler use_absolute_position_embeddingsrrhidden_dropout_probrU)r-rkrr0s r1r&!InternVLVisionEmbeddings.__init__ s' ekk!Q8J8J&KL   ll5;;q!V=O=O+PQDO"DO =f E ++&++[__-E-EFF   ##V%6%67  ++77  2 2')||EKK;QR?TZTfTf4g'hD $'+D $zz&"<"<= r3rrrcURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R S-n X0R S-n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr5r?r rbicubicF)rmode align_cornersrX)rDrr)jit is_tracingrrrpermuter'r^ interpolatercat) r-rrrr num_positionsclass_pos_embedpatch_pos_embedrY new_height new_widthsqrt_num_positionss r1interpolate_pos_encoding1InternVLVisionEmbeddings.interpolate_pos_encoding6s]!&&q)A- 0066q9A= yy##%%+*F6?++ +221bqb59221ab59r"q11 __Q// &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr3rbool_masked_poscURu p4nURU5unupxUR5upnUbIURR XS5n UR S5R U 5n USU - -X--nURR U SS5n [R"X4SS9nURbX`RXdU5-nURU5nXgU44$)Nr5rrX) rDrrrexpand unsqueezetype_asrr)rrrrU)r-rrrrrrrrrr mask_tokensw cls_tokenss r1r@ InternVLVisionEmbeddings.forward^s +001e262G2G 2U/ /\!+!2 Q  &//00bIK))"-55kBA#q1u- ?J^^**:r2> YY 7Q?  # # /#&C&CJX]&^^J\\*- +666r3)rrUrrrrrN)rHrIrJrKrrr&r)rintrr BoolTensorr@rLrMrNs@r1rrs >3>>,&D5<<&D&DUX&D]b]i]i&DV7;7ll7"%"2"237  77r3rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)InternVLVisionMLPixc>[TU]5 Xl[URUl[ R"URUR5Ul [ R"URUR5Ul gr) r%r&rkr hidden_act activation_fnr'ryr.intermediate_sizefc1fc2r-rkr0s r1r&InternVLVisionMLP.__init__ysb  #F$5$5699V//1I1IJ99V55v7I7IJr3r=rclURU5nURU5nURU5nU$r)r rr )r-r=s r1r@InternVLVisionMLP.forwards4/ **=9 / r3)rrkr r ) rHrIrJrKr&r)rr@rLrMrNs@r1rrxs)KU\\ellr3r) layer_normrms_normc ^\rSrSrSrS\SS4U4SjjrS S\RS\ S\ \ \R\ \R\R444S jjr S r U=r$) riz?This corresponds to the Block class in the timm implementation.rkrNc>[TU]5 URUlSUl[ U5Ul[ U5Ul[UR"URURS9Ul [UR"URURS9Ul URn[R "U["R$"UR5-SS9Ul[R "U["R$"UR5-SS9Ul[R*"UR,5Ulg)Nrr/T) requires_grad)r%r&chunk_size_feed_forward seq_len_dimri attentionrmlpNORM2FN norm_typer.layer_norm_epslayernorm_beforelayernorm_afterrr'r(r)r*rrrrrU)r-rk init_valuesr0s r1r&InternVLVisionLayer.__init__s '-'E'E$08$V, '(8(8 9&:L:LRXRgRg h&v'7'789K9KQWQfQfg33  [5::v?Q?Q3S%Scgh  [5::v?Q?Q3S%Scgh zz&"<"<= r3r=rcURURU5US9up4URU-nX1-nURU5nUR U5nUR U5nUR bUR U-nXQ-nXT4$)N)r)rrrrrrUr)r-r=rattention_outputattention_weights layer_outputs r1r@InternVLVisionLayer.forwards /3nn  ! !- 0//=/ +  ==+;;)8 ++M: xx - ||L1 == $==<7L$3 ..r3) rrrUrrrrrr)F)rHrIrJrKrrr&r)rboolrrr@rLrMrNs@r1rrsnI >3 > >$#(/||/ / uU\\"E%,, *D$EE F //r3rc ~^\rSrSrS\SS4U4Sjjr\S S\RS\ S\ S\ \ \ 44S jj5r S rU=r$) InternVLVisionEncoderirkrNc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snf)NF) r%r&rkr' ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r-rkir0s r1r&InternVLVisionEncoder.__init__sS  ]]vOgOgIh#iIhA$7$?Ih#ij &+#$jsA&r=routput_hidden_statescjU(aSOSnU(aSOSn[UR5HnupgU(aXA4-nUR(a.UR(aUR UR X5nOU"X5nUSnU(dMfXXS4-nMp U(aXA4-n[ UUUS9$)Nrrrlast_hidden_stater= attentions) enumerater-r.r[_gradient_checkpointing_func__call__r) r-r=rr1all_hidden_statesall_self_attentionsr/ layer_module layer_outputss r1r@InternVLVisionEncoder.forwards#7BD$5b4(4OA#$58H$H!**t}} $ A A ))=! !-] N )!,M  &91=M[TU]U5 Xl[U5Ul[ U5UlUR(a[R"5O([R"URURS9Ul UR5 g)Nr)r%r&rkrrr(encoderuse_mean_poolingr'rrr.r layernorm post_initr s r1r&InternVLVisionModel.__init__sm   26:,V4 $44BKKM",,vGYGY_e_t_t:u  r3c.URR$r)rrrEs r1get_input_embeddings(InternVLVisionModel.get_input_embeddingss///r3rrrr1cUbUOURRnUbUOURRnURXS9upVUR UUUS9nUSnUR U5n[ UURURS9$)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )r)rr1rr3) rkrr1rrArCrr=r5) r-rrrr1embedding_outputrencoder_outputssequence_outputs r1r@InternVLVisionModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj #oolo\,, /!5'  *!,..93-)77&11  r3)rkrrArC)NNN)rHrIrJrKrr&rGrrr)rrrr&rrCrr@rLrMrNs@r1r?r?s 3  07;,0/3  ll "%"2"23 $D>  'tn  u:: ;   r3r?cB\rSrSr\rSrSrSrSr Sr Sr Sr Sr SrSrSrg)InternVLPreTrainedModeliTpast_key_valuesc4[URSURR5R5n[ U[ R 5(aWURRRSUS9 URb%URRR5 gg[ U[ R5(aJURRR5 URRRS5 gg)Nrrrr)getattrrkget_text_configrrr'ryr+rrrnrrr)r-rOrs r1r%InternVLPreTrainedModel._init_weights(sdkk#6 8S8S8U8g8gh fbii ( ( MM   & &CS & 9{{&   &&('  - - KK   " " $ MM   $ $S ).r3rN)rHrIrJrKrrrr_skip_keys_device_placement_supports_cache_classrr_supports_quantized_cache_supports_static_cache_supports_attention_backendrrLrr3r1rOrOsA!L&*#"3 !N $!"& *r3rOc6^\rSrSrS\4U4SjjrSrSrU=r$)InternVLMultiModalProjectori4rkc0>[TU]5 [R"URR [ SUR- 5S--5Ul[R"URR [ SUR- 5S--URR 5Ul [URUl[R"URR URR 5Ulg)Nrr)r%r&r'r vision_configr.rdownsample_ratiorry text_configlinear_1r projector_hidden_actactlinear_2r s r1r&$InternVLMultiModalProjector.__init__5s ,,v';';'G'G#aRXRiRiNiJjnoJo'op  , ,s1v7N7N3N/OST/T TV\VhVhVtVt  &556 &"4"4"@"@&BTBTB`B`a r3cURU5nURU5nURU5nURU5nU$r)rrarcrd)r-image_featuresr=s r1r@#InternVLMultiModalProjector.forward>s@7  m4 /  m4 r3)rcrrard) rHrIrJrKrr&r@rLrMrNs@r1r\r\4sb~br3r\cB\rSrSr%SrSr\\R\ S'Sr g)InternVLModelOutputWithPastiFa Base class for InternVL outputs, with hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_statesr) rHrIrJrKrrkrr) FloatTensor__annotations__rLrr3r1rjrjFs88<%"3"34;r3rjzx The InternVL model which consists of a vision backbone and a language model, without a language modeling head. ) custom_introc%.^\rSrSrSS0rS\4U4SjjrSrSrS\ RS \ \ \ \ 4S \4S jr\\SS \ R$S\ RS \\ R(S\\ R$S\\ \ RS\\ RS \\ \ \ \ 4S \\S\\S\\S\\S\\S\\ R$S\ R(S\\S\ \\44 Sjj55rSS\ R(S\4SjjrSrU=r$) InternVLModeligzlanguage_model.modellanguage_modelrkc>[TU]U5 [R"UR5Ul[ U5Ul[R"UR5Ul UR5 gr) r%r&r from_configr^ vision_towerr\multi_modal_projectorr`rqrDr s r1r&InternVLModel.__init__osY  %11&2F2FG%@%H"'33F4F4FG r3c6URR5$r)rqrGrEs r1rG"InternVLModel.get_input_embeddingsws""7799r3c:URRU5 gr)rqset_input_embeddingsr-rRs r1rz"InternVLModel.set_input_embeddingszs 007r3rvision_feature_layervision_feature_select_strategyc URRnUS:XaURUS9RnOUR US9R UnUS:XaUSS2SS2SS24nUR Sn[US-5nUR Sn URXUS5nURXeS9nURU SUR S5nURU5nU$) a Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `List[int]`): Layer index or list of layer indices to extract features from. Returns: vision_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`. r5)rdefaultNrrr) scale_factor) rkr_rtr4 vision_modelr=rDrr pixel_shuffleru) r-rr}r~rar_vision_featureschannels feature_sizers r1get_image_features InternVLModel.get_image_features}s$ ;;77 2 %"//\/J\\O"//\/JXXYmnO )Y 6-aQh7O#((+8S=) $**1- *11*LZ\],,_,\*11*b/BWBWXZB[\44_Er3 input_idsrS position_idsrQ inputs_embeds use_cacherr1 return_dictcache_position image_sizesrarc U bU OURRn U bU OURRn U bU OURRn UbUOURRnUbUOURR nUSLUSL- (a [ S5eUcUR5"U5nUGbURUUUUS9nUcxX`R5"[R"URR[RURS95:HnURSS9RSS9SnOwXRR:HRS5nUR!U5R#UR5nXRR:HR5n[%5(d{UUR'5UR'5:waVXRR:HR5nUR(SUR(S-n[ SUS U35eUR#URUR*5nUR-UU5nUR."S UUUUU U U S U S . UD6n[1UR2UR4UR6UR8UbWS 9$SS 9$)Nz:You must specify exactly one of input_ids or inputs_embeds)rr}r~r)r7devicerrXrr5z6Image features and image tokens do not match: tokens: z , features T) rSrrQrrrr1rr)r4rQr=r5rkr)rkrr1use_return_dictr}r~rsrGrr)tensorimage_token_idlongrsumr expand_asr8rnumelrDr7masked_scatterrqrjr4rQr=r5)r-rrrSrrQrr}r~rrr1rrrrargspecial_image_maskn_image_tokensn_image_featuresrs r1r@InternVLModel.forwards(2C1N-TXT_T_TqTq$8$D $++JjJj &1%2J   QU   r3rrc UR5up4pVXR-S:wdXB-S:wa [S5eURX4[XR-5[Xb- 55nUR SSSS5R 5nURU[XR-5[XB-5[XbS-- 55nUR SSSS5R 5nU$)aPerform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr )rrsrrrr`)r-rrrrrrs r1rInternVLModel.pixel_shuffles/>.B.B.D+ 6  A %)=)Bjk k*.. s6#893x?V;W *11!Q1=HHJ*.. F12C8L4MsS[mn_nSoOp  *11!Q1=HHJr3)rqrurt)NNNNNNNNNNNNNN)r)rHrIrJrK_checkpoint_conversion_mappingrr&rGrzr)rlrrrstrrrr LongTensorrrr&rr rrjr@floatrrLrMrNs@r1rprpgs '=>N%O"~:8+''+$CcN3+), +Z'+*.1537=A59@D8<$(,0/3&*59$(T ##T ''T !. T u//0 T "$u'8'8"9: T   1 12T 'uS$s)^'<=T )1 T D>T $D>T 'tnT d^T !!1!12T \\T -.!T " u11 2#T T l!U\\!!!r3rpc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \R\ S'Sr \\\R\ S'Sr\\\R\ S'Sr\\R\ S 'S rg) InternVLCausalLMOutputWithPasti&a@ Base class for InternVL causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. NlosslogitsrQr=r5rkr)rHrIrJrKrrrr)rlrmrrQrr=rr5rkrLrr3r1rr&s<)-D(5$$ %,*.FHU&& '.9=OXd5#4#456=8[TU]U5 [U5Ul[R "UR RUR RSS9Ul UR5 g)NFrm) r%r&rpmodelr'ryr`r. vocab_sizerrDr s r1r&)InternVLForConditionalGeneration.__init___sS  "6* yy!3!3!?!?ASASA^A^ejk  r3c6URR5$r)rrGrEs r1rG5InternVLForConditionalGeneration.get_input_embeddingseszz..00r3c:URRU5 gr)rrzr{s r1rz5InternVLForConditionalGeneration.set_input_embeddingshs ''.r3rcUR$rrrEs r1get_output_embeddings6InternVLForConditionalGeneration.get_output_embeddingsks ||r3cXlgrr)r-new_embeddingss r1set_output_embeddings6InternVLForConditionalGeneration.set_output_embeddingsns% r3c.URR$r)rrqrEs r1rq/InternVLForConditionalGeneration.language_modelrszz(((r3c.URR$r)rrtrEs r1rt-InternVLForConditionalGeneration.vision_towervszz&&&r3c.URR$r)rrurEs r1ru6InternVLForConditionalGeneration.multi_modal_projectorzszz///r3rrrSrrQrr}r~labelsrrr1rrlogits_to_keeprrac U bU OURRn U bU OURRn U bU OURRn UbUOURRnUbUOURR nUR "SUUUUUUUUU U U SUUS.UD6nUSn[U[5(a [U*S5OUnURUSS2USS245nSnU b3UR"SUXRRRS.UD6n[UUURUR UR"UR$S9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", torch_dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```NT)rrrSrrQrr}r~rrr1rrrr)rrr)rrrQr=r5rkr)rkrr1rr}r~rrrslicer loss_functionr`rrrQr=r5rk)r-rrrSrrQrr}r~rrrr1rrrrrarr= slice_indicesrrs r1r@(InternVLForConditionalGeneration.forward~s|2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W8W~ot4]k mA}a,?@A  %%f9P9P9[9[_eD.#33!//)) ' ; ;   r3c P>[T U]"U4UUUUUS.UD6n USS:XaXIS'U $)N)rQrrSrrrr)r%prepare_inputs_for_generation) r-rrQrrrSrrra model_inputsr0s r1r>InternVLForConditionalGeneration.prepare_inputs_for_generationsTw<  +')))    !  !,8 (r3sequence_length target_lengthr7rc UbUR5S:XaUnU$[R"U5Rn[R"X4XUR S9nUS:wa[R "USS9nU[R"X$R S9URSS5:-nUSSSS2SS24RUSSS5nUbUR5nURSn USS2SS2SS2SU 24USS2SSSS24RUR 5-n U S:Hn USS2SS2SS2SU 24RX5USS2SS2SS2SU 24'U$) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuer7rr)diagonal)rr5r)rYr)finfominfullrtriuarangerrclonerDr8 masked_fill) rSrrr7rrrare min_dtype mask_length padding_masks r15_prepare_4d_causal_attention_mask_with_cache_positionVInternVLForConditionalGeneration._prepare_4d_causal_attention_mask_with_cache_positions}<  %.*<*<*>!*C(K*' E*..I** 0Y\j\q\qK!##jjqA 5<< >S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12r3)rr)NNNNNNNNNNNNNNrN)NNNNNN))rHrIrJrKr_tied_weights_keysrr&rGrzr'Modulerrpropertyrqrtrurrr)rrlrrrrrrr&rrrrr@r staticmethodr7rrLrMrNs@r1rrQs"8-"?#, &" ++~ 1/ryy&))''00'+*.1537=A59@D8<-1$(,0/3&*5934.2#n ##n ''n !. n u//0 n "$u'8'8"9: n   1 12n 'uS$s)^'<=n )1 n ))*n D>n $D>n 'tnn d^n !!1!12n c5<</0!n "ell+#n $*+%n & u44 5'n n f<4 444{{ 4  4  44r3r)rr?rOrpr)r)Fcollections.abcr dataclassesrtypingrrrrrr)torch.nnr' activationsr generationr integrationsr modeling_flash_attention_utilsr modeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrrrrrrautorconfiguration_internvlrr get_loggerrHrrr"rrrgrirrrrrrrrr(r?rOr\rjrprrr__all__rr3r1rs.!99 !)7BddF&H   H %Y'JBIIJ(J6% II% <<% % << % U\\* %  %%6NbiiNb KO K KF +E 2!7BII!7L[7ryy[7|  3H I-/"))-/`( BII( V2 72 2 j*o**0"))$ <"9< <@ w+w  wt $<[$< $ m'>m  m` r3