o ZhN(@sdZddlmZddlmZmZmZmZmZm Z m Z ddl Z ddl m mZddlZ ddl m Z ddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZm Z m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1m2Z2e*rddl3m4Z4ddl5m6Z6e+7e8Z9eGdddeZ:eGdddeZ;    dGddZZ?Gd"d#d#e j@ZAGd$d%d%e jBZCe%DeCGd&d'd'e j jBZEd(d)ZFdHd*d+ZGGd,d-d-e jBZH .dId/e jBd0e jId1e jId2e jId3ee jId4eJd5eJfd6d7ZKGd8d9d9e jBZLGd:d;d;e jBZMGdd?d?e!ZOGd@dAdAee'ZPe(GdBdCdCeOZQGdDdEdEeOeZRgdFZSdS)JzPyTorch Idefics model.) dataclass)AnyCallableDictListOptionalTupleUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs) ModelOutput)ALL_ATTENTION_FUNCTIONSPretrainedConfigPreTrainedModel)Unpack)ALL_LAYERNORM_LAYERS) LossKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging) IdeficsConfig)IdeficsPerceiverResampler)IdeficsVisionEmbeddingsIdeficsVisionTransformer) BlockMask)make_flex_block_causal_maskc@seZdZUdZdZeejed<dZ ee e ejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<dS)IdeficsBaseModelOutputWithPasta Base class for Idefics model's outputs that may also contain a past key/values (to speed up sequential decoding). Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and optionally if `config.is_encoder_decoder=True` 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver Nlast_hidden_statepast_key_values hidden_states attentionsimage_hidden_states)__name__ __module__ __qualname____doc__r%rtorch FloatTensor__annotations__r&rr'r(r)r1r1[/var/www/auris/lib/python3.10/site-packages/transformers/models/idefics/modeling_idefics.pyr$6s $r$c@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZee ejed<dS) IdeficsCausalLMOutputWithPasta Base class for Idefics causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver Nlosslogitsr&r'r(r))r*r+r,r-r4rr.r/r0r5r&rr'rr(r)r1r1r1r2r3cs  r3FcKsJt|jdddd|d|j}|d|}|dd|d<|dd|d<|dd|d<|dd|d<d|vrM|d}|d||d<|durY|d||d <|dduri|dd||d<|ddur}|dd||d<||fS|ddur|dd||d<||fS|ddur|dd||d<||fS) Nrr pixel_valuesimage_encoder_embeddingsperceiver_embeddingsimage_attention_masktoken_type_idsattention_mask) r.arangeshapeviewrepeattodevice index_selectget) input_idsZ expand_sizeis_encoder_decoderr<Zencoder_outputs model_kwargsZexpanded_return_idxr;r1r1r2expand_inputs_for_generations:,      rHcsftjtjtjdfdd|D}|D]|r+tfdd|Dr+dqdq|S)N) LayerNormLinear Embeddingcsg|]}|qSr1r1).0m)mappingr1r2 z freeze_model..c3s|]}t|VqdSN) isinstance)rLt)moduler1r2 szfreeze_model..TF)r rIrJrKmodulesanyrequires_grad_)modelmodule_exceptionsZmodule_exceptions_mappedr1)rNrTr2 freeze_models   r[csNeZdZdZ    d deeddffdd Zdd Zdefd d Z Z S) IdeficsDecoupledEmbeddinga Implements a decoupling of parameters to allow freezing (or not) a subset of the embeddings. In practise, the regular `weight` can be trained or frozen (i.e. `partially_freeze=True`), and if `num_additional_embeddings` > 0, then it will create `num_additional_embeddings` additional parameters that are always trained. If `num_additional_embeddings=0`, then the module defaults back to the regular behavior of `nn.Embedding`. FNpartially_freezereturnc  s|dur||krtd|d|tjd|||||d|||_||_||_||_|r5|jd|jdkrGt j |j|||d|_ dSdS) a) Args: num_embeddings (`int`): Size of the dictionary of embeddings num_additional_embeddings (`int`): Number of additional embeddings. Only useful when you `partially_freeze=True`. embedding_dim (`int`): The size of each embedding vector partially_freeze: (`bool`, *optional*, defaults to `False`): If `True`, the regular `weight` will be frozen. `additional_weight` is never frozen. padding_idx (`int`, *optional*): The padding index (needs to be less than num_embeddings) Note: there are a lot of other parameters to initialize a standard `nn.Embedding` such as `padding_idx`, `max_norm` or `norm_type`. We are not supporting these. Nz/padding_idx must be within num_embeddings. Got z and )num_embeddings embedding_dimrBdtype padding_idxFr)r_r`rBrar1) ValueErrorsuper__init__r_rbnum_additional_embeddingsr]weightrXr rKadditional_embedding) selfr_rfr`r]rBrarbkwargs __class__r1r2res2   z"IdeficsDecoupledEmbedding.__init__cCsj|jdkr t||jS|}t||jk}||}|||j}d||<t||j}|||<|S)a we have 2 embeddings, with different indices - one pretrained self.weight and another self.additional_embedding.weight that is being trained. in order to make a lookup of the input ids, we: 1. find out the indices of the entries belonging to the 2nd embedding 2. extract those values while subtracting the size of the first embedding (num_embeddings), since the 2nd embedding starts from 0 and not num_embeddings 3. perform the 2nd embedding lookup 4. now we handle the 1st embedding, we overwrite indices belonging to the 2nd embedding with a padding index 5. perform the 1st embedding lookup 6. now we overwrite the values in the 1st embedding lookup with the values of the 2nd embedding lookup note: for the 1st embedding lookup we could have looked up only the low indices and not do the padding, but then we have to create a new tensor and populate it with 2 tensors that are spread out across various indices - i.e. not a simple concat - I haven't benchmarked the complex case if it's any faster, given that seqlens are usually relatively short it's probably not faster or if faster not by much - but might be a good idea to measure. r) rfFZ embeddingrgcloner.wherer_rh)rirEZadditional_vocab_indicesZinput_ids_additional_vocabZadditional_embeddingsZ full_vectorr1r1r2forwards z!IdeficsDecoupledEmbedding.forwardcCsd|j|j|j|jS)NzVnum_embeddings={}, num_additional_embeddings={}, embedding_dim={}, partially_freeze={})formatr_rfr`r]rir1r1r2 extra_repr.s z$IdeficsDecoupledEmbedding.extra_repr)FNNN) r*r+r,r-rboolrerpstrrs __classcell__r1r1rkr2r\s  5'r\c sjeZdZdZ     ddedededed ed df fd d Zd ejd ejfddZ d e fddZ Z S)IdeficsDecoupledLineara Implements a decoupling of parameters to allow freezing (or not) a subset of the parameters. In practise, the regular `weight` can be trained or frozen (i.e. `partially_freeze=True`), and if `out_additional_features` > 0, then it will create `out_additional_features * in_features` additional parameters that are always trained. If `out_additional_features=0`, then the module defaults back to the regular behavior of `nn.Linear`. rTN in_features out_featuresout_additional_featuresbiasr]r^csrt|||||||_||_||_||_|r&|jd|r&|jd|dkr7t j |||||d|_ dSdS)aG out_additional_features: int. Number of additional trainable dimensions. Only makes sense when `partially_freeze=True`. partially_freeze: bool. If True, the regular `weight` will be frozen and extra parameters (if any) will be trainable. If False, default to the regular behavior of nn.Linear. Fr)rxryr{rBraN) rdrerzr]rxryrgrXr{r rJ additional_fc)rirxryrzr{r]rBrarkr1r2re@s$   zIdeficsDecoupledLinear.__init__inputcCs:t||j|j}|jdkr||}t||fd}|S)Nrr6)rmZlinearrgr{rzr|r.cat)rir}outputZadditional_featuresr1r1r2rpds   zIdeficsDecoupledLinear.forwardcCs d|j|j|j|jdu|jS)z=Overwriting `nn.Linear.extra_repr` to include new parameters.zYin_features={}, out_features={}, out_additional_features={}, bias={}, partially_freeze={}N)rqrxryrzr{r]rrr1r1r2rsmsz!IdeficsDecoupledLinear.extra_repr)rTTNN) r*r+r,r-intrtrer.Tensorrprursrvr1r1rkr2rw7s,  $ rwcs.eZdZdfdd ZddZddZZS) IdeficsRMSNormư>cs&ttt||_||_dS)z= IdeficsRMSNorm is equivalent to T5LayerNorm N)rdrer Parameterr.onesrgvariance_epsilon)ri hidden_sizeepsrkr1r2rezs  zIdeficsRMSNorm.__init__cCs\|tjdjddd}|t||j}|jjtj tj fvr)||jj}|j|S)Nr6T)Zkeepdim) rAr.float32powmeanZrsqrtrrgraZfloat16Zbfloat16)rir'Zvariancer1r1r2rps  zIdeficsRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tuplergr>rrrr1r1r2rsszIdeficsRMSNorm.extra_repr)r)r*r+r,rerprsrvr1r1rkr2rys rcs0eZdZd fdd ZddZd dd ZZS) IdeficsEmbedding'Ncszt||_||_||_d|jtjd|jdtjdj|tj d|j}|j d|dd|j ||j j td dS) N?rrrarBrainv_freqF persistentseq_lenrBra)rdredimmax_position_embeddingsbaser.r=int64rAfloatregister_buffer_set_cos_sin_cacherrBZget_default_dtype)rirrrrBrrkr1r2res & zIdeficsEmbedding.__init__cCs|||_tj|j|tjd|j}td||j}tj||fdd}|jd| |dd|jd| |dddS) Nrzi,j->ijr6r cos_cachedFr sin_cached) max_seq_len_cachedr.r=rZtype_asrZeinsumr~rcosrAsin)rirrBrarSZfreqsZembr1r1r2rs z#IdeficsEmbedding._set_cos_sin_cachecCsN||jkr|j||j|jd|jd|j|jd|jd|j|jdfS)Nrr)rrrBrarrAr)rixrr1r1r2rps zIdeficsEmbedding.forward)rrNrQ)r*r+r,rerrprvr1r1rkr2rs rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr6rr)r>r.r~)rx1Zx2r1r1r2 rotate_halfsrcCsL|||}|||}||t||}||t||}||fS)anApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`): The position indices of the tokens corresponding to the query and key tensors. For example, this can be used to pass offsetted position ids when working with a KV-cache. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkrr position_idsZ unsqueeze_dimZq_embedZk_embedr1r1r2apply_rotary_pos_embs rcs2eZdZdededeffdd ZddZZS) IdeficsMLPrintermediate_size hidden_actcsNttj||dd|_tj||dd|_tj||dd|_t||_dS)NFr{) rdrer rJ gate_proj down_projup_projr act_fn)rirrrrkr1r2res zIdeficsMLP.__init__cCs ||||||SrQ)rrrr)rirr1r1r2rp zIdeficsMLP.forward)r*r+r,rrurerprvr1r1rkr2rs rrTquerykeyvaluer<scalingdropoutc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr6)rraptrainingrr) r.matmul transposer functionalZsoftmaxrrArarr contiguous) rTrrrr<rrrj attn_weights attn_outputr1r1r2eager_attention_forwards  rcseZdZdZ     ddedededed ed ed eeffd d Z de j dedefddZ       dde j dee j dee j dee j deee j dededee j dee j ee j eee j ffddZZS) IdeficsAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFNr num_headsris_cross_attentionconfigqk_layer_norms layer_idxc st||_||_||_|||_||_d|_|jd|_||_ |dur1t d|j j d|j||jkrEtd|jd|d||_ttjdsRtd |jrt|jd s^|jn|jj}tj|j||jd d |_tj|||jd d |_tj|||jd d |_n'tj|j||jd d |_tj|j||jd d |_tj|j||jd d |_tj||j|d d |_t|j|_||_|jrt|j|jd |_t|j|jd |_ dSdS)NTgzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.z?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).Zscaled_dot_product_attentionz)this model requires pytorch 2.0 or higher embed_dimFrr)!rdrerrrhead_dimrZ is_causalrrlogger warning_oncerlr*rcrhasattrr r vision_configrrJq_projk_projv_projo_projr rotary_embrr rms_norm_eps q_layer_norm k_layer_norm) rirrrrrrrZ kv_input_dimrkr1r2re s|       zIdeficsAttention.__init__tensorrbszcCs ||||j|jddS)Nrr)r?rrrr)rirrrr1r1r2_shape]rzIdeficsAttention._shaper'key_value_statesr<rpast_key_valueoutput_attentions use_cachecache_positionr^c Ks|jp|du} |\} } } ||| | |j|jdd}| sD||| | |j|jdd}||| | |j|jdd}n)|\} }} ||| ||j|jdd}||| ||j|jdd}|j d}|dur|||d7}| s|j |t || d\}}t |||||\}}|durd|i}| |||j|\}}|jr||}||}t}|jjdkr|jjdkr|rtd nt|jj}||||||f|jsd n|j|jd | \}}|| | d }||}|rd}|||fS) Nrrrr)rreagersdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)rrr6)rsizerr?rrrrrr>rmaxrupdaterrrrrr_attn_implementationrrrrrrreshaperr)rir'rr<rrrrrrjrrZq_len_Z query_statesZ key_statesZ value_statesZkv_lenZ kv_seq_lenrrZ cache_kwargsZattention_interfacerrr1r1r2rp`s\ ""$"          zIdeficsAttention.forward)rFNFNNNNNFFN)r*r+r,r-rrrtrrrer.rr LongTensorrrprvr1r1rkr2r sbQ   rcseZdZddedeeffdd Z      ddejdeejd eej d ee ejd ee d ee d eej de ej ee ej ej fffddZ ZS)IdeficsDecoderLayerNrrcsrt|j|_t|j|j|j||d|_t|j|j|j d|_ t |j|j d|_ t |j|j d|_|j|_dS)N)rrrrrrrrr)rdrerrnum_attention_headsr self_attnrrrmlprrinput_layernormpost_attention_layernormrirrrkr1r2res"  zIdeficsDecoderLayer.__init__Fr'r<rrrrrr^c Ks|} ||}|jd|||||||d|\}} } tjj||j|jd}| |}|} ||}||}tjj||j|jd}| |}|f} |rN| | f7} |rU| | f7} | S)a] Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states )r'r<rrrrrrNr1)rrr rrrrr) rir'r<rrrrrrjresidualself_attn_weightspresent_key_valueoutputsr1r1r2rps4       zIdeficsDecoderLayer.forwardrQ)NNNFFN)r*r+r,rrrrer.rrrrtr/rprvr1r1rkr2rs2  rcseZdZddedeeffdd Z       ddejdeejd eejd eejd eejd ee d ee dee ejde ej ee ej ej fffddZ Z S)IdeficsGatedCrossAttentionLayerNrrc s~t|j|_t|j|jd|j||j|d|_t|j|j |j d|_ t |j|j d|_t |j|j d|_|j|_t|_t|_|jdkr|jdkrgttdd|j|_ttdd|j|_n|jdkrttd|_ttd|_ntd |jd |jd kr|jdkrttdd|j|_ttdd|j|_n|jdkrttd|_ttd|_njtd |jd |jd vr$|jdkrttjd |jdd|jfd|_ttjd |jdd|jfd|_n3|jdkrttjd |jdd|_ttjd |jdd|_ntd |jd t d|jdt!|dr9t!|ds=tddS)NT)rrrrrrrrrzerosZvectorrrz Unknown value for `alpha_type` ()r>normalrandomgaussianr)rstdrzAlpha initialization scheme z not yet implemented!alpha_cross_attn alpha_densez+Alpha parameters not initialized correctly!)"rdrerrrrr cross_attnrrrrrrrrrr ZTanhact_cross_attn act_densealpha_initializerZ alpha_typerr.rrrrcrralphas_initializer_rangeNotImplementedErrorrrrkr1r2resn             z(IdeficsGatedCrossAttentionLayer.__init__Fr'r<r)r:cross_attention_gaterrrr^c Ks|durtd|durtd|durtd|} ||}|jd ||||d| \}} } tjj||j|jd}| |dkdddddfd}| | |j |}|} | |}| |}tjj||j|jd}| ||j|}|f} |r| | f7} |r| | f7} | S) a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. image_attention_mask (`torch.FloatTensor`, *optional*): image attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. cross_attention_gate (`torch.FloatTensor`, *optional*): gate of size `(batch, seq_len)` used to zero-out cross-attention output for tokens attending no images. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states Nzt`image_hidden_states` is required for Idefics cross attention module which are visual features to be conditioned on.z`cross_attention_gate` is required for Idefics cross attention module to zero-out the cross-attention hidden_states attending to no images.zMPast key value states are not implemented for Idefics cross attention module.)r'rr<rrrrr1)rcr rrr rrrr masked_fillrrrrrr)rir'r<r)r:r rrrrjrrrrr1r1r2rpBsD  "    z'IdeficsGatedCrossAttentionLayer.forwardrQr)r*r+r,rrrrer.rrtrr/rprvr1r1rkr2rs8E  rc@s<eZdZeZdZdZddgZdZdZ dZ dZ dZ ddZ dS) IdeficsPreTrainedModelrYTrrFcCs|jj}t|tjtjfr%|jjjd|d|j dur#|j j dSdSt|tj rF|jjjd|d|j durD|jj|j  dSdSt|tj r[|jjd|j j dSt|tri|jjddSt|trv|jjdSt|tr|jjdkr|jj |jj dS|jjdkr|jjd|jjddS|jjdvr|jjjd|jjd|jjjd|jjddSdSt|tr|jjdSdS)Nr)rrrrr>rrr)rZinitializer_rangerRr rJZConv2drgdataZnormal_r{Zzero_rKrbrIZfill_rr Zclass_embeddingrrrrrrZlatents)rirTrr1r1r2 _init_weightss@            z$IdeficsPreTrainedModel._init_weightsN)r*r+r,rZ config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_sdpaZ_supports_cache_classZ_supports_flash_attn_2Z_supports_static_cacheZ_supports_attention_backendrr1r1r1r2r s r c@s eZdZdS)KwargsForCausalLMN)r*r+r,r1r1r1r2rs rc'seZdZdZdeffdd Zd.ddZgfdd Zgfd d Zd d Z ddZ e e               d/de ejde ejde ejde eejde ejde ejde ejde ejde ejde ede ede ede ede ede ejd eed!eeeff"d"d#Z d0deejd$fd%ejdejdedef d&d'Zedejd(ed)ed*ejdejd+ef d,d-ZZ S)1 IdeficsModelz Transformer decoder consisting of `config.num_hidden_layers` layers. Each layer is a [`IdeficsDecoderLayer`] Args: config: IdeficsConfig rcs t|_j|_j|_tjjjj |jd|_ j j |_ j |_ t j |_jrEj}tj j|j|j|j|j|_tfddtjD|_j|_j|j}tfddt|D|_d|_t jj!d|_"|#|$dS)N)r_rfr`r]rbcg|]}t|dqS)r)rrLirr1r2rOz)IdeficsModel.__init__..crr)rrrr1r2rOrFr)%rdrerZ pad_token_idrb vocab_sizer\additional_vocab_sizerfreeze_text_layers embed_tokensrZ image_sizer! vision_model use_resamplerperceiver_configrrZresampler_depthZresampler_n_headsZresampler_head_dimZresampler_n_latentsperceiver_resamplerr Z ModuleListrangeZnum_hidden_layerslayerscross_layer_intervalgated_cross_attn_layersgradient_checkpointingrrnorm post_initfreeze_relevant_params)rirrZnum_cross_layersrkrr2resH     zIdeficsModel.__init__NcCs>|dur|j}|jr||j|jrt|j|jddSdSN)rZ)rrZfreeze_text_module_exceptionsfreeze_vision_layersr[rZfreeze_vision_module_exceptions)rirr1r1r2r&s z#IdeficsModel.freeze_relevant_paramscCs"|j|jfD]}t||dqdSr')r r$r[)rirZrTr1r1r2rszIdeficsModel.freeze_text_layerscCst|j|ddSr')r[r)rirZr1r1r2r(sz!IdeficsModel.freeze_vision_layerscC|jSrQrrrr1r1r2get_input_embeddings z!IdeficsModel.get_input_embeddingscC ||_dSrQr*rirr1r1r2set_input_embeddings z!IdeficsModel.set_input_embeddingsFrEr<rr& inputs_embedsr7r8r9r:rroutput_hidden_statesinterpolate_pos_encoding return_dictrrjr^c* s|dur|jn|j}| dur| n|jj} | dur| n|jj} | dur$| n|jj} |dur.|n|jj}|du|duAr>td|jrM|jrM| rMt dd} |durV| |}d}| rst |t ssd}|durit}n t|}t d|j\}}}|dur|nd}||}|durtj|||jd|jd }|dur|dur|d d}||dkd|dd| df}n |dur|d}td d |||fDd krtd|dur |j|j|d}|jdd \}}|j||g|jd dR}|j|| dj}n|dur(|\}}}}|j|j|d}|||||}|jj rN|durC|!|}|d|d }}n|\}}}}|}n|dur_|d|d }}ntd|||||}| d}| d } | "ddd|} | ||||} |dur|\}}}||f}| durtj#||d } |$| } nd} | dkj%d dj|jdj&dd|} |durtj#||ftj'|jd}|(||||| }|}!| rdnd}"| rdnd}#d}$t)|j*D]o\}%}&| r|"|!f7}"fdd}'|jr,|jr,d}| rt dd} |+|'|&|!||||| | | | |%|j,|j-|}(n|'|&|!f||||| | | | |%|j,|j-|d }(|(d}!| rT|(| rQd nd}$| r^|#|(df7}#q|.|!}!| rm|"|!f7}"| rr|$nd})|r{|)/})|||||}t0|!|)|"|#|dS)ab image_encoder_embeddings (`torch.FloatTensor`, *optional*): The output of the image encoder. perceiver_embeddings (`torch.FloatTensor`, *optional*): The output of the perceiver resampler. image_attention_mask (`torch.LongTensor`, *optional*): The attention mask for the image encoder. Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FTzWe detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)rrrBr6cSsg|]}|duqSrQr1)rLrr1r1r2rOcrPz(IdeficsModel.forward..rz_Exactly 1 of pixel_values, image_encoder_embeddings or perceiver_embeddings has to be not-None.)rarB)r7r3zBIf `perceiver_embeddings` are passed, use_resampler should be Truerrrr1c sb| | dkr | | | }||f|||||| dd}|d}||f||||| | d}|S)Nr)r<r)r:r rrr)r<rrrrrr1)Z main_blockr'r<rrr)r:r rrrr!r"rZxblockr layer_outputsrjr1r2vblocks:     z$IdeficsModel.forward..vblockzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...) r<rrr)r:r rrrr!r"r)r%r&r'r(r))1rBrrr2ruse_return_dictrcr#rrrrrRr rZfrom_legacy_cacher>get_seq_lengthr.r=longZcumsumZ masked_fill_rsumrArarr?rr%rrrr@rZinvert_attention_maskrWZsqueezert_update_causal_mask enumerater Z_gradient_checkpointing_funcr!r"r$Zto_legacy_cacher$)*rirEr<rr&r1r7r8r9r:rrr2r3r4rrjrBZreturn_legacy_cache batch_size seq_lengthrpast_key_values_lengthZseq_length_with_pastZ num_imagesr)Z image_seq_lenZimage_hidden_sizeZ text_seq_lenZimage_batch_sizeZimage_sequence_lengthZimage_hidden_shaper r'Zall_hidden_statesZall_self_attnsZnext_decoder_cacheidxZ decoder_layerr8r6Z next_cacher1r7r2rps8     $          "    -  zIdeficsModel.forwardr" input_tensorc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) NZflash_attention_2rZflex_attentionrFr)r1rAZ is_trainingrr6)sequence_length target_lengthrarr?)cudaZxpuZnpu)rrrWrRr.rr#r:Zis_compileablerZ_ignore_causal_mask_sdparrar>Zget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrBtypefinfominZ_unmask_unattended) rir<rCrr&rZpast_seen_tokensZusing_compilable_cacherarDrE causal_mask min_dtyper1r1r2r=(sT           z IdeficsModel._update_causal_maskrDrErar?c KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valuerarBr)Zdiagonalr5r6r)rr.rIrJfullrBZtriur=rexpandrnr>rAr ) r<rDrErarr?rjrKrLZ mask_lengthZ padding_maskr1r1r2rGls,  $ 6  zBIdeficsModel._prepare_4d_causal_attention_mask_with_cache_positionrQ)NNNNNNNNNNNNFNNF)!r*r+r,r-rrer&rr(r+r/rrrr.rrrr/rtrrr rr$rpr r= staticmethodrrarGrvr1r1rkr2rs 0          Drc)seZdZddgZd2fdd ZddZdd Zd d Zd d ZddZ ddZ ddZ e e                d3deejdeejdeejdeeejdeejdeejdeejdeejdeejdeejdeed eed!eed"eed#eed$eejd%eed&eeeff$d'd(Z         d4fd)d* Z d5d+ed,eeefd-ed&eeefffd.d/ Z e!d0d1Z"Z#S)6IdeficsForVisionText2Textzmodel.embed_tokens.weightzlm_head.weightNcs>t|t||_t|j|j|jd|jd|_ | dS)NF)rxryrzr{r]) rdrerrYrwrrrZfreeze_lm_headlm_headr%)rirrrkr1r2res   z"IdeficsForVisionText2Text.__init__cCs|jjSrQrYrrrr1r1r2r+sz.IdeficsForVisionText2Text.get_input_embeddingscCs ||j_dSrQrTr.r1r1r2r/s z.IdeficsForVisionText2Text.set_input_embeddingscCr)rQrSrrr1r1r2get_output_embeddingsr,z/IdeficsForVisionText2Text.get_output_embeddingscCr-rQrU)riZnew_embeddingsr1r1r2set_output_embeddingsr0z/IdeficsForVisionText2Text.set_output_embeddingscCr-rQrY)ridecoderr1r1r2 set_decoderr0z%IdeficsForVisionText2Text.set_decodercCr)rQrXrrr1r1r2 get_decoderr,z%IdeficsForVisionText2Text.get_decodercCs|}|}t|jddr&|j|_|jdkr&|j|jks J|jj|j_t |drDt |drF|j |_ t |drHt |drJ|j|_dSdSdSdSdS) z Overwrite `transformers.modeling_utils.PreTrainedModel.tie_weights` to handle the case of IdeficsDecoupledLinear and IdeficsDecoupledEmbedding. Ztie_word_embeddingsTrryr_rzrfN) rVr+getattrrrgrfrzrhr|rr_ry)riZoutput_embeddingsZinput_embeddingsr1r1r2 tie_weightss    z%IdeficsForVisionText2Text.tie_weightsFrEr<rr&r1r7r8r9r:labelsrrr2r3r4rrjr^cKs| dur| n|jj} | dur| n|jj} |dur|n|jj}|jd||||||||| | | | |d|d|}|d}||}d}| durS|jd|| |jjd|}t|||j |j |j |j dS)aC labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. image_encoder_embeddings (`torch.FloatTensor`, *optional*): The output of the image encoder. perceiver_embeddings (`torch.FloatTensor`, *optional*): The output of the perceiver resampler. image_attention_mask (`torch.LongTensor`, *optional*): The attention mask for the image encoder. Example: ```python >>> from transformers import AutoProcessor, IdeficsForVisionText2Text >>> model = IdeficsForVisionText2Text.from_pretrained("HuggingFaceM4/idefics-9b") >>> processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics-9b") >>> dogs_image_url_1 = "https://huggingface.co/datasets/hf-internal-testing/fixtures_nlvr2/raw/main/image1.jpeg" >>> dogs_image_url_2 = "https://huggingface.co/datasets/hf-internal-testing/fixtures_nlvr2/raw/main/image2.jpeg" >>> prompts = [ ... [ ... "User:", ... dogs_image_url_1, ... "Describe this image.\nAssistant: An image of two dogs.\n", ... "User:", ... dogs_image_url_2, ... "Describe this image.\nAssistant:", ... ] ... ] >>> inputs = processor(prompts, return_tensors="pt") >>> generate_ids = model.generate(**inputs, max_new_tokens=6) >>> processor.batch_decode(generate_ids, skip_special_tokens=True) ```NT)rEr<rr&r1r7r8r9r:rrr2r3r4rr)r5r^r)r4r5r&r'r(r)r1) rrr2r9rYrSZ loss_functionrr3r&r'r(r))rirEr<rr&r1r7r8r9r:r^rrr2r3r4rrjrr'r5r4r1r1r2rpsH< z!IdeficsForVisionText2Text.forwardc  si} |dur|jjr|| d<n || d<n|| d<| dd| d<tj|f|||||| | d| | } | durP|durP| djd}| dd| df| d <| S) Nr9r8r7r3F)r&r<r1rrrr:rErr:)rrpoprdprepare_inputs_for_generationr>)rirEr<rr1r&rr7r)r:rrjZ images_kwargsZ model_inputsr@rkr1r2r`Ds4     z7IdeficsForVisionText2Text.prepare_inputs_for_generationrrGrFc s~tj|||fi|}d|vr8|d}|dddddfd}|ddr-||d<n tj||gdd|d<|j|d<|S)Nr:r6rrTrr))rd#_update_model_kwargs_for_generationrrDr.r~r))rirrGrFrjr:Z last_maskrkr1r2raqs   z=IdeficsForVisionText2Text._update_model_kwargs_for_generationcs.d}|D]}|tfdd|Df7}q|S)Nr1c3s|] }|dVqdS)rN)rC)rLZ past_statebeam_idxr1r2rUsz;IdeficsForVisionText2Text._reorder_cache..)r)ZpastrcZreordered_pastZ layer_pastr1rbr2_reorder_cachesz(IdeficsForVisionText2Text._reorder_cacherQ)NNNNNNNNNNNNNFNN) NNNNNNNNNrP)$r*r+r,Z_tied_weights_keysrer+r/rVrWrZr[r]rrrr.rrrr/rtrrr rr3rpr`rrrurrarQrdrvr1r1rkr2rRs       g1  rR)rRrr )rFNN)r)r)Tr- dataclassesrtypingrrrrrrr r.Ztorch.nn.functionalr rrmZtorch.utils.checkpointZ activationsr Z cache_utilsr rZ generationrZmodeling_attn_mask_utilsrZmodeling_flash_attention_utilsrZmodeling_outputsrZmodeling_utilsrrrZprocessing_utilsrZ pytorch_utilsrutilsrrrrrZconfiguration_ideficsrZ perceiverrZvisionr r!Z!torch.nn.attention.flex_attentionr"Zintegrations.flex_attentionr#Z get_loggerr*rr$r3rHr[rKr\rJrwModulerappendrrrrrrrrrrr rrrR__all__r1r1r1r2s $             ,+ -nB '  &Q-f o