o ZhԽ@sBddlmZddlmZmZmZmZmZddlZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZddlmZmZmZm Z ddl!m"Z"ddl#m$Z$m%Z%e &e'Z(eGdddeZ)Gddde j*Z+ d>de j*dej,dej,dej,deej,de-de-fdd Z.Gd!d"d"e j*Z/Gd#d$d$e j*Z0Gd%d&d&e j*Z1Gd'd(d(e j*Z2ed)d*Gd+d,d,e)Z3eGd-d.d.eZ4Gd/d0d0e j*Z5Gd1d2d2e j*Z6ed3d*Gd4d5d5e)Z7eGd6d7d7eZ8Gd8d9d9eeZ9ed:d*Gd;d<dr?r'r@rAZnum_attention_heads num_headshead_dim ValueErrorscaleZattention_dropoutrkrr)k_projv_projq_projout_proj is_causalrHrIr3r4r?s$    zSmolVLMVisionAttention.__init__NF hidden_statesrioutput_attentionsrMc Cs|j\}}}||}||}||} ||||j|jdd}||||j|jdd}| |||j|jdd} t} |j j dkr[|j j dkrU|rUt dnt |j j } | |||| ||j|j|jsjdn|jd\} } | |||} || } |sd} | | fS) z#Input shape: Batch x Time x ChannelrreagerZsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r")rrjrkN)rQr~r|r}rXrxryrSrwr'_attn_implementationlogger warning_oncerrr{rprkreshapersr) r1rrirr\ seq_lengthrAZquerieskeysvaluesZattention_interfacervrur3r3r4r`s:        zSmolVLMVisionAttention.forward)NF) r6r7r8rar?rTrdrboolrr`rer3r3rIr4rs rcs2eZdZfddZdejdejfddZZS)SmolVLMVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) r>r?r'r Z hidden_act activation_fnrr)r@Zintermediate_sizefc1fc2rHrIr3r4r?s  zSmolVLMVisionMLP.__init__rrMcCs"||}||}||}|Sr)rrr)r1rr3r3r4r`s   zSmolVLMVisionMLP.forward)r6r7r8r?rTrdr`rer3r3rIr4rs rc sNeZdZdeffdd Z d dejdejdeede ej fd d Z Z S) SmolVLMEncoderLayerr'csRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dS)NZeps) r>r?r@rAr self_attnrr0layer_norm_eps layer_norm1rmlp layer_norm2rHrIr3r4r?s   zSmolVLMEncoderLayer.__init__FrrirrMcCsb|}||}|j|||d\}}||}|}||}||}||}|f}|r/||f7}|S)a= Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrir)rrrr)r1rrirZresidualruoutputsr3r3r4r`s      zSmolVLMEncoderLayer.forward)F) r6r7r8rr?rTrdrrrrbr`rer3r3rIr4rs rc sheZdZdZdeffdd Z    d deejdee dee d ee d e e e ff d d Z ZS)SmolVLMEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`SmolVLMEncoderLayer`]. Args: config: SmolVLMConfig r'cs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr3)r).0r]r'r3r4 .sz+SmolVLMEncoder.__init__..F) r>r?r'rZ ModuleListrangeZnum_hidden_layerslayersgradient_checkpointingrHrIrr4r?+s   zSmolVLMEncoder.__init__Nriroutput_hidden_states return_dictrMc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|}|jD]-} |r8||f}|jrH|jrH|| j|||} n| |||d} | d}|r\|| df}q/|rd||f}|srt dd|||fDSt |||dS) ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr3)rrrcss|] }|dur|VqdSrr3)rvr3r3r4 tsz)SmolVLMEncoder.forward..last_hidden_stater attentions) r'rruse_return_dictrrrpZ_gradient_checkpointing_func__call__tupler) r1 inputs_embedsrirrrZencoder_statesZall_attentionsrZ encoder_layerZ layer_outputsr3r3r4r`2sB      zSmolVLMEncoder.forwardNNNN)r6r7r8rarr?rrTrdrrrrr`rer3r3rIr4r"s$  rzN The SmolVLM Vision Transformer Model outputting raw image embedding. Z custom_introc seZdZeZdZdZdZdeffdd ZddZ ddZ dd e e j d e ed e ed e edeeeff ddZZS)SmolVLMVisionTransformerTr'csPt||j}t||_t||_|j|_tj ||j d|_ |j dk|_ dS)Nrflash_attention_2)r>r?r@r<r^rencoderrCrr0rpost_layernormr_use_flash_attention_2)r1r'rArIr3r4r?s   z!SmolVLMVisionTransformer.__init__cC|jSrr^r1r3r3r4get_input_embeddingsz-SmolVLMVisionTransformer.get_input_embeddingscC ||_dSrrr1rhr3r3r4set_input_embeddings z-SmolVLMVisionTransformer.set_input_embeddingsNrLrrrrMc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|d}|durE|j}t||d||d|f}|jtj |j d}|j ||d}| |d}t |s[d}n |jsdt||j}|j|||||d} | d} || } |s| f| ddSt| | j| jd S) Nrrr rnr[rKrLrP)rrirrrrr)r'rrrrNrCrTonesrZrr[r^rXanyrr rnrrrrr) r1rKrLrrrr\rCrZencoder_outputsrr3r3r4r`sH       z SmolVLMVisionTransformer.forwardr)r6r7r8rr9r:Z_supports_flash_attention_2r;r?rrrrTrcrrrrr`rer3r3rIr4rzs.  rc@seZdZUdZdZeejed<dZ ee e ejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<dS)SmolVLMBaseModelOutputWithPasta Base class for SmolVLM model's outputs that may also contain a past key/values (to speed up sequential decoding). Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and optionally if `config.is_encoder_decoder=True` 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder Nrr rrimage_hidden_states)r6r7r8rarrrTrb__annotations__r rrrrr3r3r3r4rs rcs$eZdZfddZddZZS)SmolVLMSimpleMLPcs:t|jj|jd}|jj}tj||dd|_dS)NrFr-) r>r? vision_configr@ scale_factor text_configrr)proj)r1r'Z input_sizeZ output_sizerIr3r4r?s zSmolVLMSimpleMLP.__init__cCs ||Sr)r)r1xr3r3r4r`rzSmolVLMSimpleMLP.forward)r6r7r8r?r`rer3r3rIr4rs rcs.eZdZfddZdddZddZZS) SmolVLMConnectorcs t|j|_t||_dSr)r>r?rrmodality_projectionrHrIr3r4r?s zSmolVLMConnector.__init__rcCs|\}}}t|d}}|||||}|||t||||}|dddd}||t||t||||d}|dddd}||t||d||d}|S)Ng?rrrr )rNintrXZpermuter)r1rrZbszseqrAheightwidthr3r3r4 pixel_shuffles("zSmolVLMConnector.pixel_shufflecCs|||j}||}|Sr)rrr)r1rr3r3r4r`s zSmolVLMConnector.forward)r)r6r7r8r?rr`rer3r3rIr4rs   rzY SmolVLM model consisting of a SIGLIP vision encoder and Llama3 language decoder c#s@eZdZdZdeffdd ZddZddZd d Zd d Z d e j de j de j fddZ d%de jde j fddZeedd             d&d ee j dee j dee j deee jdee jdee jdee jdee jdeedeedeedeed ee j d!eed"eeeffd#d$ZZS)' SmolVLMModelz A subclass of Idefics3Model. We do *not* remove or block the call to inputs_merger in forward. Instead, we override inputs_merger here with custom logic. r'cst||jjj|_|jjj|_t|j |_ t ||_ t |j|_t|j j|j jd|jd|_|jj|_|jjdk|_|dS)Nrr)r>r?r'rZ pad_token_idr/ vocab_sizerZ _from_configr vision_modelr connectorrZ from_config text_modelrrBrCrZ image_seq_lenimage_token_idrr post_initrHrIr3r4r?#s      zSmolVLMModel.__init__cs:fdddd}|||_|j||_dS)aE Enables the gradients for the input embeddings. This is useful for lora when using gradient checkpointing. c.f. https://github.com/huggingface/peft/issues/1402#issuecomment-1913675032 Override to set output.requires_grad = True for both the decoder's and vision model's embeddings. cs,tt|dkr |St|dS)Nr)lenlistchildren)r2get_lowest_moduler3r4r?szBSmolVLMModel.enable_input_require_grads..get_lowest_modulecS|ddSNTZrequires_grad_r2inputoutputr3r3r4make_inputs_require_gradsGzJSmolVLMModel.enable_input_require_grads..make_inputs_require_gradsN)rregister_forward_hook_text_require_grads_hookr_vision_require_grads_hookr1rr3rr4enable_input_require_grads5s   z'SmolVLMModel.enable_input_require_gradscC|j|jdSrrremoverrr3r3r4disable_input_require_gradsO z(SmolVLMModel.disable_input_require_gradscCs |jSr)rrrr3r3r4rSrz!SmolVLMModel.get_input_embeddingscCs|j|dSr)rrrr3r3r4rVsz!SmolVLMModel.set_input_embeddings input_idsrrcCs|j\}}}||jk}|jdd}t||dkstd||}tjjj|j ddddd} | dd} |j dd} | d|} | d|} | d| }t |}|||| |ddf||<t | d||}|S) ar This method aims at merging the token embeddings with the image hidden states into one single sequence of vectors that are fed to the transformer LM. The merging happens as follows: - The text token sequence is: `tok_1 tok_2 tok_3 ... tok_4`. - We get the image hidden states for the image through the vision encoder and that hidden state, after a pixel shuffle operation, is then projected into the text embedding space. We thus have a sequence of image hidden states of size (1, image_seq_len, hidden_dim), where 1 is for batch_size of 1 image and hidden_dim is the hidden_dim of the LM transformer. - The merging happens so that we obtain the following sequence: `vector_tok_1 vector_tok_2 vector_tok_3 vector_fake_tok_around_image {sequence of image_seq_len image hidden states} vector_fake_toke_around_image vector_tok_4`. That sequence is fed to the LM. - To fit the format of that sequence, `input_ids`, `input_embeds`, `attention_mask` are all 3 adapted to insert the image hidden states. rrmrzCAt least one sample has tokens not divisible by patch_size.)rr)rhNrP) rQrrWrTallrzrrrpadZcumsumZ unsqueezeZ zeros_likewhere)r1rrrr]rCZ image_maskZnum_image_tokensZblocks_per_sampleoffsetsZ block_offsetZrow_cumZ chunk_idxZ local_idxZ block_idxZ image_embedsZ merged_embedsr3r3r4 inputs_mergerYs       zSmolVLMModel.inputs_mergerNrKpixel_attention_maskcs*j\}}}}}j||gjddRjdd}dkjdd|k} t| s3d| d<| |durOtjfd d d Dtjj d }n|j||g|jddR}|| }|j j j } |j d| | d } | j d| | d } | jdddk} |j| d} | j} || } | S)a Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. pixel_attention_mask (`torch.LongTensor`, *optional*): The attention mask indicating padded regions in the image. rNrr")rPrlrTrcsg|]}j|qSr3)rQ)rirKr3r4rsz3SmolVLMModel.get_image_features..)rrr )rNrnr[) dimensionrNstep)rPrlr)rQrXZnumelrWrrsrTrrr[r'rrCZunfoldrrr)r1rKrr\Z num_imagesrDrrZnb_values_per_imageZreal_images_indsrCZpatches_subgridrLrr3rr4get_image_features{s.      zSmolVLMModel.get_image_featuresa Inputs fed to the model can have an arbitrary number of images. To account for this, pixel_values fed to the model have image padding -> (batch_size, max_num_images, 3, max_heights, max_widths) where max_num_images is the maximum number of images among the batch_size samples in the batch. Padding images are not needed beyond padding the pixel_values at the entrance of the model. For efficiency, we only pass through the vision_model's forward the real images by discarding the padding images i.e. pixel_values of size (image_batch_size, 3, height, width) where image_batch_size would be 7 when num_images_per_sample=[1, 3, 1, 2] and max_num_images would be 3. rrir_r use_cacherrrcache_positionrtrMc Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} | dur$| n|jj} |jr8|jjr8| r8t dd} |durB|j \}}n|durM|j \}}}nt dd}| r`|dur\t }| }|durp|durp|dkrpt d|dur|j||j}|dur|durt d|dur|||}n |dur|j|j|jd}|dur|dur|j|||d }|jd ||||| | | d | d |}t|j|j|j|j|d S)a| pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. image_hidden_states (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The hidden states of the image encoder after modality projection. NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fz5You have to specify either input_ids or inputs_embedsrzWWhen first calling the model, if input_embeds are passed, input_ids should not be None.zMYou cannot specify both pixel_values and image_hidden_states at the same timer)rrrT) rrir_r rrrrr)rr rrrr3)r'rrrrrprrrrrQrzr Zget_seq_lengthrrZr[rrnrrrr rr)r1rrir_r rrKrrrrrrrrtr\rr]Zpast_seen_tokensrr3r3r4r`sp#   zSmolVLMModel.forwardr) NNNNNNNNNNNNN)r6r7r8rarr?rrrrrT LongTensorrdrrbrrrrrrcrrrrrrr`rer3r3rIr4rs~ ".        rc@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZee ejed<dS) SmolVLMCausalLMOutputWithPasta Base class for Idefics causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder Nlosslogitsr rrr)r6r7r8rarrrTrbrrr rrrrrr3r3r3r4rs rc@s eZdZdS)KwargsForCausalLMN)r6r7r8r3r3r3r4r:s rz The SmolVLM Model with a language modeling head. It is made up a SigLIP vision encoder, with a language modeling head on top. c'sXeZdZdgZfddZddZddZdd Zd d Zd d Z ddZ e e               d)de ejde ejde ejde eejde ejde ejde ejde ejde ejde ede ede ede ejde ed eeejfd!eed"eeeff"d#d$Z        d*fd%d& Zfd'd(ZZS)+SmolVLMForConditionalGenerationzlm_head.weightcsPt|t||_|jj|_tj|jj |jj dd|_ |jj |_ | dS)NFr) r>r?rrr'rrr)rr@rlm_headrrHrIr3r4r?Es     z(SmolVLMForConditionalGeneration.__init__cCs0dd}|||_|jj||_dS)z Enables the gradients for the input embeddings. This is useful for fine-tuning adapter weights while keeping the model weights fixed. cSrrrrr3r3r4rUrz]SmolVLMForConditionalGeneration.enable_input_require_grads..make_inputs_require_gradsN)rrrrrrrr3r3r4rOs   z:SmolVLMForConditionalGeneration.enable_input_require_gradscCrrrrr3r3r4r]rz;SmolVLMForConditionalGeneration.disable_input_require_gradscCs |jjSr)rrrrr3r3r4ras z4SmolVLMForConditionalGeneration.get_input_embeddingscCs|jj|dSr)rrrrr3r3r4rdsz4SmolVLMForConditionalGeneration.set_input_embeddingscCrrrrr3r3r4get_output_embeddingsgrz5SmolVLMForConditionalGeneration.get_output_embeddingscCrrr)r1Znew_embeddingsr3r3r4set_output_embeddingsjrz5SmolVLMForConditionalGeneration.set_output_embeddingsNrrrir_r rrKrrlabelsrrrrrlogits_to_keeprtrMcKs| dur| n|jj} | dur| n|jj} |dur|n|jj}|jd||||||||| | | | dd |}|d}t|trCt| dn|}||dd|ddf}d}| durh|j d|| |jj j d|}t |||j |j|j|jdS)a pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. image_hidden_states (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The hidden states of the image encoder after modality projection. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or `model.image_token_id` (where `model` is your instance of `SmolVLMForConditionalGeneration`). Tokens with indices set to `model.image_token_id` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> import requests >>> import torch >>> from PIL import Image >>> from io import BytesIO >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> from transformers.image_utils import load_image >>> # Note that passing the image urls (instead of the actual pil images) to the processor is also possible >>> image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg") >>> image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg") >>> image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg") >>> processor = AutoProcessor.from_pretrained("HuggingFaceTB/SmolVLM2-2.2B-Instruct") >>> model = AutoModelForImageTextToText.from_pretrained("HuggingFaceTB/SmolVLM2-2.2B-Instruct", torch_dtype=torch.bfloat16, device_map="auto") >>> # Create inputs >>> messages = [ ... { ... "role": "user", ... "content": [ ... {"type": "video", "path": path/to/video}, ... {"type": "text", "text": "What is happening in this video?"}, ... ] ... } ... ] >>> inputs = processor.apply_chat_template([messages], add_generation_prompt=True) >>> # Generate >>> generated_ids = model.generate(**inputs, max_new_tokens=256) >>> generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True) >>> print(generated_texts) ```NT) rrir_r rrKrrrrrrrr)rrr)rrr rrrr3)r'rrrrr(rslicerZ loss_functionrrrr rrr)r1rrir_r rrKrrrrrrrrr rtrrZ slice_indicesrrr3r3r4r`msNF z'SmolVLMForConditionalGeneration.forwardc s^tj|f|||||||| d| } |dur!|ddkr!|| d<|dur-d| d<d| d<| S)N)r rirrrKrrr rrrKr)r>prepare_inputs_for_generation) r1rr rirrrKrrr rtZ model_inputsrIr3r4r s( z=SmolVLMForConditionalGeneration.prepare_inputs_for_generationc s(tjd|||d|}|j|d<|S)N)r model_kwargsis_encoder_decoderrr3)r>#_update_model_kwargs_for_generationr)r1rr r rtrIr3r4rs zCSmolVLMForConditionalGeneration._update_model_kwargs_for_generation)NNNNNNNNNNNNNNr)NNNNNNNN)r6r7r8Z_tied_weights_keysr?rrrrrrrrrrTrrdrrbrcrrrrrrrr`r rrer3r3rIr4r=s          s(r)rrrr)r")< dataclassesrtypingrrrrrrTrZ activationsr Z cache_utilsr Z generationr Zmodeling_attn_mask_utilsr Zmodeling_flash_attention_utilsrZmodeling_outputsrrZmodeling_utilsrrZprocessing_utilsrutilsrrrrautorZconfiguration_smolvlmrrZ get_loggerr6rrModuler<rdfloatrwrrrrrrrrrrrr__all__r3r3r3r4sz          A I0XP& x% R