o Zhd^@sdZddlZddlmZmZmZmZmZddlZddl m m Z ddl Zddlm Z ddlmZddlmZmZddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd l m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,m-Z-m.Z.e)rddl/m0Z0ddl1m2Z2e*3e4Z5dej6de7de8deej6ej6ffddZ9dej6de7de7dej:dej6f ddZ;Gdd d e j<Z=Gd!d"d"e j<Z>Gd#d$d$e j<Z?d%ej6d&e7dej6fd'd(Z@ )d`d*e jZHdad?d@ZIGdAdBdBe j<ZJGdCdDdDe j<ZKGdEdFdFe j<ZLGdGdHdHej j<ZMGdIdJdJe j<ZNe'GdKdLdLe"ZOe'dMdNGdOdPdPeOZPe'dQdNGdRdSdSeOZQGdTdUdUee&ZRe'dVdNGdWdXdXeOeZSe'dYdNGdZd[d[eOZTe'd\dNGd]d^d^eOeZUgd_ZVdS)bzPyTorch Mllama model.N)CallableListOptionalTupleUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)BaseModelOutputBaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging) MllamaConfigMllamaTextConfigMllamaVisionConfig) BlockMask)make_flex_block_causal_maskcross_attention_masknum_vision_tokensdtypereturnc Cs|j^}}}|j|dd}|||d}|d}d||}||tjt|j }t|j }||kj dd |d}||9}||fS)Nrdimr?).N) shapeZrepeat_interleaveview unsqueezeto masked_filltorchboolfinfominanyZtype_as) r"r#r$ batch_sizeZtext_total_length_Zinverted_cross_attn_maskZnegative_inf_valuefull_text_row_masked_out_maskr7Y/var/www/auris/lib/python3.10/site-packages/transformers/models/mllama/modeling_mllama.py_prepare_cross_attention_mask/s   r9aspect_ratio_mask num_patches target_lengthcCs|j\}}|||dd|}|dd|d}||}d|dddd| df<d|}||||d}||ddt|j}| d}|S)Nrrr() r*r+r-repeatreshape transposer/r1r2r,)r:r;r<r$r4 max_num_tilesattention_maskZ pad_patchesr7r7r8$_prepare_aspect_ratio_attention_maskKs  rCcsDeZdZd dedeffdd Zdejdejdejfd d ZZ S) %MllamaPrecomputedAspectRatioEmbeddingTconfigis_gatedcsbt|j|_|j|_|j|_||_t|jd|j|j|_|r/t t d|_ dSdSNr) super__init__rA hidden_sizemax_aspect_ratio_idrFr Embedding embedding Parameterr/zerosgateselfrErF __class__r7r8rIgs z.MllamaPrecomputedAspectRatioEmbedding.__init__ hidden_stateaspect_ratio_idsr%cCs>||}|d|jd|j}|jr||j}||}|S)Nr(r)rMr?rArJrFrPtanh)rRrUrVZ embeddingsr7r7r8forwardrs z-MllamaPrecomputedAspectRatioEmbedding.forward)T) __name__ __module__ __qualname__rr0rIr/TensorrX __classcell__r7r7rSr8rDfs$ rDcs>eZdZdeffdd ZdejdejdejfddZZS) "MllamaPrecomputedPositionEmbeddingrEcst|j|_|j|_|j|jdd|_|j|_|jd|_t t d|_ t |j|j}t |j||_t |jd|j|j|j|_dS)Nr)rHrIrArK image_size patch_sizer;rJscalerrNr/rOrPrandnrMrLtile_embedding)rRrEZposition_embeddingrSr7r8rI~s   z+MllamaPrecomputedPositionEmbedding.__init__rUrVr%cCspd|j|j}||dd|j|j}||}|jd}|||j |j|j}|j|}||}|S)Nrr) rPrWrMr+r;rJrer*r?rA)rRrUrVZgated_position_embeddingZtile_position_embeddingr4Zgated_tile_position_embeddingr7r7r8rXs  z*MllamaPrecomputedPositionEmbedding.forward) rYrZr[rrIr/r\rXr]r7r7rSr8r^}s$r^cs2eZdZfddZdejdejfddZZS)MllamaVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) rHrIrEr hidden_act activation_fnrLinearrJintermediate_sizefc1fc2rRrErSr7r8rIs  zMllamaVisionMLP.__init__ hidden_statesr%cCs"||}||}||}|Srg)rlrirm)rRror7r7r8rXs   zMllamaVisionMLP.forward)rYrZr[rIr/r\rXr]r7r7rSr8rfs rfron_repcCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r*expandr?)rorpbatchnum_key_value_headsslenhead_dimr7r7r8 repeat_kvs 0rvmodulequerykeyvaluerBscalingdropoutc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr_rr=r()r'r$)ptrainingr)rvnum_key_value_groupsr/matmulr@r*r functionalZsoftmaxfloat32r-r$r}r contiguous) rxryrzr{rBr|r}kwargs key_states value_states attn_weights causal_mask attn_outputr7r7r8eager_attention_forwards & rc s^eZdZdeffdd Z  d dejdeejdeede ejeejffd d Z Z S) MllamaVisionAttentionrEcst||_|j|_|j|_|j|j|_|jd|_d|_ t j |j|j|jdd|_ t j |j|j|jdd|_ t j |j|j|jdd|_t j |j|j|jdd|_dS)Nr`rFbias)rHrIrErJZ embed_dimattention_heads num_headsrur|rrrjq_projk_projv_projo_projrnrSr7r8rIs   zMllamaVisionAttention.__init__NrUrBoutput_attentionsr%cKs||}||}||}|j\}} } |j\} } } ||| |j|jdd}||| |j|jdd}||| |j|jdd}t} |j j dkra|j j dkr[|r[t dnt |j j } | |||||fd|jd|\} }| || d} || } |sd}| |fS) Nrr_eagersdpa`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rwr}r|r()rrrr*r+rrur@rrE_attn_implementationlogger warning_oncerr|r?rr)rRrUrBrrryrzr{r4Z q_seq_lenr5Z kv_seq_lenattention_interfacerrr7r7r8rXs@         zMllamaVisionAttention.forwardNN) rYrZr[rrIr/r\rr0rrXr]r7r7rSr8rsrcsPeZdZd dedeffdd Z  d dejdeejd eefd d Z Z S)MllamaVisionEncoderLayerFrErFcst|j|_|j|_||_|j|_t||_t ||_ t j |j|j d|_t j |j|j d|_|rRt tdtjd|_t tdtjd|_dSdS)Nepsr)rHrIrJrnum_attention_headsrFrkr self_attnrfmlpr LayerNormZnorm_epsinput_layernormpost_attention_layernormrNr/onesmathpi gate_attngate_ffnrQrSr7r8rIs    z!MllamaVisionEncoderLayer.__init__NrUrBrcCs|}||}|j||d\}}|jr|j|}||}|}||}||}|jr4|j|}||}|f}|rB||f7}|S)N)rB)rrrFrrWrrr)rRrUrBrresidualroutputsr7r7r8rX.s     z MllamaVisionEncoderLayer.forwardFr) rYrZr[rr0rIr/r\rrXr]r7r7rSr8rsrcspeZdZdZddeffdd Z    ddejd eejd ee d ee d ee d e e e ff ddZ ZS)MllamaVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MllamaEncoderLayer`]. Args: config: MllamaConfig FrEcs@t|_tfddt|D|_d|_|_dS)Ncsg|]}tqSr7)r).0r5rErFr7r8 Xsz0MllamaVisionEncoder.__init__..F)rHrIrEr ModuleListrangelayersgradient_checkpointing)rRrEZ num_layersrFrSrr8rIUs   zMllamaVisionEncoder.__init__NrorBroutput_hidden_states return_dictr%c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|jD]-}|r6||f}|jrF|jrF||j|||} n||||d} |rV|| df}| d}q-|rb||f}|spt dd|||fDSt |||dS) ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr7)rUrBrrrcs|] }|dur|VqdSrgr7rvr7r7r8 z.MllamaVisionEncoder.forward..last_hidden_statero attentions) rErruse_return_dictrrr_gradient_checkpointing_func__call__tupler) rRrorBrrrZencoder_statesZall_attentionsZ encoder_layer layer_outputsr7r7r8rX\s>       zMllamaVisionEncoder.forward)rF)NNNN)rYrZr[__doc__rrIr/r\rr0rrrrXr]r7r7rSr8rLs(  rcs.eZdZdfdd ZddZddZZS) MllamaTextRMSNormư>cs&ttt||_||_dS)z@ MllamaTextRMSNorm is equivalent to T5LayerNorm N)rHrIrrNr/rweightvariance_epsilon)rRrJrrSr7r8rIs  zMllamaTextRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)Nr_r(T)Zkeepdim) r$r-r/rpowmeanZrsqrtrr)rRroZ input_dtypeZvariancer7r7r8rXs  zMllamaTextRMSNorm.forwardcCst|jjd|jS)Nz, eps=)rrr*rrRr7r7r8 extra_reprszMllamaTextRMSNorm.extra_repr)r)rYrZr[rIrXrr]r7r7rSr8rsrcseZdZdZ  ddeedeeffdd Z      ddej d eej d ee d eej d e d ee deej de ej eej ee ej ffddZZS)MllamaTextCrossAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNrE layer_idxcst||_|jj|_|jj|_|j|_|j|_|j|j|_||_ |j|j|_ |jd|_ t j |j|j|jdd|_t j |j|j|jdd|_t j |j|j|jdd|_t j |j|j|jdd|_t|j|jd|_t|j|jd|_dS)Nr`Frr)rHrIrErrrsr}rJrurrr|rrjrrrrr rms_norm_epsq_normk_normrRrErrSr7r8rIs    z!MllamaTextCrossAttention.__init__Frocross_attention_statespast_key_valuerBr use_cachecache_positionr%cKs|\} } } ||} | | | |j|jdd} || } |dur`||} ||}| | d|j |jdd} || d|j |jdd}| | } |dur_| | ||j d|i\} }n|ddkrt|j |j |j|j } }ntdt}|jjdkr|jjd kr|rtd nt|jj}||| | ||f|jsd n|j|jd |\}}|| | d}||}|sd}|||fS) z#Input shape: Batch x Time x Channelrr_Nr(rrz^Cross attention layer can't find neither `cross_attn_states` nor cached values for key/values!rrrrwr)sizerr+rrur@rrrrsrupdaterZ key_cacheZ value_cache ValueErrorrrErrrrrr}r|r?rr)rRrorrrBrrrrbszq_lenr5 query_statesrrrrrr7r7r8rXs\             z MllamaTextCrossAttention.forwardr)NNNFNN)rYrZr[rrrintrIr/r\r r0 LongTensorrrXr]r7r7rSr8rs@ rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr(r_r&)r*r/cat)xx1Zx2r7r7r8 rotate_halfsrcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )r,r)qkcossin position_idsZ unsqueeze_dimZq_embedZk_embedr7r7r8apply_rotary_pos_emb#s  rc sTeZdZdedeffdd Z    ddejdejd ejd ed ef d d Z Z S)MllamaTextSelfAttentionrErcst||_|j|_|j|_|j|_|j|_|j|j|_|j|j|_ |jd|_ |j |_ ||_ t j|j|j|jdd|_t j|j|j|jdd|_t j|j|j|jdd|_t j|j|j|jdd|_dS)Nr`Fr)rHrIrErrr}rJrsrurr|Z rope_thetarrrjrrrrrrSr7r8rI?s   z MllamaTextSelfAttention.__init__FNrorBposition_embeddingsrrcKsZ|\} } } ||} ||} ||}| | | |j|jdd} | | | |j|jdd} || | |j|jdd}|\}}t | | ||\} } |durb|||d}| | ||j |\} }t }|j jdkr~|j jdkrx|rxtdnt|j j}||| | ||f|jsdn|j|jd|\}}|| | d }||}|sd}|||fS) Nrr_)rrrrrrrwrr()rrrrr+rrur@rsrrrrrErrrrrr}r|r?rr)rRrorBrrrrrrrrr5rrrrrZ cache_kwargsrrrr7r7r8rXQsH         zMllamaTextSelfAttention.forward)FFNN) rYrZr[rrrIr/r\r0rXr]r7r7rSr8r>s"rcs$eZdZfddZddZZS) MllamaTextMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFr) rHrIrErJrkrrj gate_projup_proj down_projr rhact_fnrnrSr7r8rIs zMllamaTextMLP.__init__cCs$||||||}|Srg)rrrr)rRrrr7r7r8rXs zMllamaTextMLP.forward)rYrZr[rIrXr]r7r7rSr8rs  rc seZdZdedeffdd Z          ddejdeejd eejd eejd ee ejejfd eej d ee dee dee deej dee ejejfde ede ejee ejejfffddZZS)MllamaSelfAttentionDecoderLayerrErcsXt|j|_t||d|_t||_t|j|jd|_ t|j|jd|_ ||_ dS)N)rErr) rHrIrJrrrrrrrrrrrSr7r8rIs   z(MllamaSelfAttentionDecoderLayer.__init__NFrorr"rBr6rrrrrrrr%c  Ks|} ||}|jd|||||| | | d| \}}}| |}|} ||}||}| |}|f}|r9||f7}| r@||f7}|S)a. Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`Tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )rorBrrrrrrNr7)rrrr)rRrorr"rBr6rrrrrrrrZself_attn_weightsZpresent_key_valuerr7r7r8rXs2%       z'MllamaSelfAttentionDecoderLayer.forward) NNNNNNFFNN)rYrZr[rrrIr/r\rrrr r0rr FloatTensorrXr]r7r7rSr8rsN     rcseZdZdZdededdffdd Z      dd ejd ejd ejd ejd e ejejfde ej de e de e de e de ej de ejdeede ejfddZZS) MllamaCrossAttentionDecoderLayerzLCross-attention transformer block with tanh-gated attention and feedforward.rErr%Ncsxt||_t||d|_t|j|jd|_t j t d|_ t||_t|j|jd|_t j t d|_dS)N)rrr)rHrIrr cross_attnrrJrrr/rrNrOcross_attn_attn_gaterrrcross_attn_mlp_gaterrSr7r8rIs  z)MllamaCrossAttentionDecoderLayer.__init__Frorr"rBr6rrrrrrrc  Ks|} ||}|jd|||||| d| \}}}| |j|}|} ||}||}|dur<|dddf|}| |j|}|f}|rO||f7}| rV||f7}|S)N)rorBrrrrrr7)rrrrWrrr)rRrorr"rBr6rrrrrrrrrrr7r7r8rXs2       z(MllamaCrossAttentionDecoderLayer.forward)NNFFNN)rYrZr[rrrrIr/r\rrrr r0rrrXr]r7r7rSr8rsH     rcs8eZdZddeffdd ZeeddZZ S)MllamaRotaryEmbeddingNrEcsht|jd|_|j|_|j|_||_t|j|_ | |j|\}|_ |j d|dd|j |_ dS)N rope_typeinv_freqF) persistent)rHrIZ rope_scalingrZmax_position_embeddingsZmax_seq_len_cachedZoriginal_max_seq_lenrErZ rope_init_fnattention_scalingZregister_bufferrZoriginal_inv_freq)rRrEdevicerrSr7r8rI4s    zMllamaRotaryEmbedding.__init__c Cs|jddddf|jddd}|dddddf}t|jjtr2|jjdkr2|jjnd}tj |dd+|| dd}tj ||fdd }| |j }||j } Wdn1siwY|j|jd | j|jd fS) Nrr(rZmpscpuF) device_typeenabledr_r&)r$)rfloatrqr* isinstancertypestrr/Zautocastr@rrrrr-r$) rRrrZinv_freq_expandedZposition_ids_expandedrZfreqsZembrrr7r7r8rX@s(&zMllamaRotaryEmbedding.forwardrg) rYrZr[rrIr/Zno_gradrrXr]r7r7rSr8r3s  rc @seZdZeZdZdZgdZdZdZ dZ dZ dZ dZ ddZ ddeejdfd ejd ejd ed ef d dZedejdededejd ejdef ddZdS)MllamaPreTrainedModelT)rrrFcCst|jd|jj}t|tjtjfr,|jj j d|d|j dur*|j j dSdSt|tj rM|jj j d|d|jdurK|jj |j dSdSt|tjrb|jj d|j j dSt|trp|jj ddSt|trtjj |jj |ddSt|trtjj |jj |dtj|jj dSt|tr|jrtjj |jj |dtjj |jj |ddSt|tr|jj |jj dSt|t r|jr|jj dSdSdS)Ninitializer_rangerw)rstdr))r )!getattrrEget_text_configr rrrjConv2drdataZnormal_rZzero_rL padding_idxrZfill_rMllamaVisionModelinitclass_embeddingr^rMZzeros_rPrrFrrrrrrD)rRrxr r7r7r8 _init_weightsas@          z#MllamaPreTrainedModel._init_weightsrBr input_tensorrpast_key_valuesrc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) NZflash_attention_2rwZflex_attentionrFr) inputs_embedsZpast_key_values_lengthZ is_trainingrr()sequence_lengthr<r$rr4)cudaZxpuZnpu)rErr3rr/r\r!get_seq_lengthZis_compileabler Z_ignore_causal_mask_sdparr$r*Zget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrrr1r2Z_unmask_unattended) rRrBrrrrpast_seen_tokensZusing_compilable_cacher$rr<r min_dtyper7r7r8_update_causal_masksT           z)MllamaPreTrainedModel._update_causal_maskrr<r$r4c KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr)Z fill_valuer$rr)Zdiagonalrr(r)r'r/r1r2fullrZtriuaranger?rqcloner*r-r.) rBrr<r$rr4rrrZ mask_lengthZ padding_maskr7r7r8rs,  $ 6  zKMllamaPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_positionNr)rYrZr[r config_classbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_cache_class_supports_static_cacheZ_supports_sdpaZ_supports_flash_attn_2_supports_quantized_cacheZ_supports_attention_backendrrr/r\r r0r staticmethodrr$rr7r7r7r8r PsL&  Dr zH The Mllama Vision Model which consists of two vision encoders. )Z custom_introcseZdZeZdZdeffdd ZddZdej dej fd d Z e dd ej d ej dej de e de e de e deeeej dfffddZZS)r vision_modelrEcs t||j|_|j|_|j|_|j|_|j|_|j|_|j|jdd|_|jd|_ t j |j|j|j|jddd|_ t |j t|j|_t||_t|dd|_t|dd|_t |j|_t |j|_t||jdd|_t||jdd|_|dS) Nr_rr`ZvalidF)Z in_channelsZ out_channelsZ kernel_sizeZstridepaddingrT)rF)rHrIrarbrArJ num_channelsintermediate_layers_indicesr;rcrrpatch_embeddingrNr/rdrr^gated_positional_embeddingrDpre_tile_positional_embeddingpost_tile_positional_embeddingr layernorm_prelayernorm_postrnum_hidden_layers transformerZnum_global_layersglobal_transformer post_initrnrSr7r8rIs4    zMllamaVisionModel.__init__cC|jS)zg This function is used to fetch the first embedding layer to activate grads on inputs. )r-rr7r7r8get_input_embeddings,sz&MllamaVisionModel.get_input_embeddingsrUr%cCs2|j\}}}|j|d|}tj||gdd}|S)Nrr&)r*rrqr/r)rRrUr4r5rJrr7r7r8apply_class_embedding2s z'MllamaVisionModel.apply_class_embeddingN pixel_valuesrVr:rrr.csl|dur|n|jj}|dur|n|jj}|dur|n|jj}|j\}}} } } } |||| | | | }|||d}|jjj} |jjj }|| || }| d dd}|j\}}}|||| d|}| ||}|||| ||}||}|d7}|||| ||}|||}||}d|jddd}ddd|f}tj||ddd }|dkr| nd}|||d}t||j|jd|jd }|||d|}|j||d |d d}||}|||| |||}|||}|||| |||}|j||||d }|d}|||| |||}|ddddd|f}|||| ||}fd d|jD}tj|dd}|||| ||d}|ddddd|f}|||| |d}tj||gdd}|rt|t|d}nd}|r|rt|dnt|d}td|}nd}|stdd|||fDSt|||dS)a5 aspect_ratio_ids (`torch.Tensor` of shape `(batch_size, max_num_images)`, *optional*): Aspect ratio ids used to select the appropriate precomputed tile embeddings based on the aspect ratio of each input image. These ids correspond to indices in the model's list of supported aspect ratios, offset by 1. For example, if the model supports aspect ratios [[1, 1], [1, 2], [2, 1]]: - An image with aspect ratio [1, 1] would have ID 1 - An image with aspect ratio [1, 2] would have ID 2 - An image with aspect ratio [2, 1] would have ID 3 The id 0 is reserved for padding (i.e., no image). If an image has aspect ratio [1, 2], that means it was split into 2 tiles horizontally, and its `aspect_ratio_id` would be 2. aspect_ratio_mask (`torch.Tensor` of shape `(batch_size, max_num_images, max_num_tiles)`, *optional*): Mask to avoid performing attention on padding tiles. Mask values selected in `[0, 1]`: - 1 for tiles that are **not masked**, - 0 for tiles that are **masked**. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, MllamaVisionModel >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaVisionModel.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> output = model(**inputs) >>> print(output.last_hidden_state.shape) torch.Size([1, 1, 4, 1025, 7680]) ``` Nr(r_rr=rZconstant)moder{)r:r;r<r$T)rBrrcsg|]}d|qS)rr7)rioutputr7r8rsz-MllamaVisionModel.forward..r&csrrgr7rr7r7r8rrz,MllamaVisionModel.forward..r) rErrrr*r?r-rr$rr-flattenr@r/r9r.r1FpadrCr;r+r4r2r0r5r,r/stackrrr)rRr:rVr:rrrr4Znum_concurrent_mediaZ num_tilesr+heightwidthZ target_dtypeZ target_deviceZ patch_embedsrUr5r;r'Znum_padding_patchesr*Z slice_indexrBZ global_outputZall_intermediate_hidden_statesZintermediate_hidden_statesroZ global_attnrr7r>r8rX8s2           zMllamaVisionModel.forward)NNN)rYrZr[rr$r%rIr8r/r\r9rrr0rrrrXr]r7r7rSr8rs2%rzc The Mllama Text Model which consists of transformer with self and cross attention layers. c"seZdZeZdZdeffdd ZddZddZe dd e e j d e e j d e e j d e e jde e j de ee j e j fde eeee jfde e jde ede ede ede ede e j deedeeeffddZZS)MllamaTextModellanguage_model.modelrEcst||j|_|j|_t|jd|j|j|_|j |_ g}t |j D]}||j vr6| t ||q&| t||q&t||_t|j|jd|_t|d|_d|_|dS)Nr;r)rEF)rHrI pad_token_idr vocab_sizerrLrJ embed_tokenscross_attention_layersrr3appendrrrrrrnormr rotary_embrr6)rRrErrrSr7r8rIs     zMllamaTextModel.__init__cCr7rgrJrr7r7r8r8z$MllamaTextModel.get_input_embeddingscC ||_dSrgrOrRr{r7r7r8set_input_embeddings z$MllamaTextModel.set_input_embeddingsN input_idsrBrrr"r6rrrrrrrrr%cKsl| dur| n|jj} | dur| n|jj} | dur| n|jj} | dur$| n|jj} |du|duAr4td|jrC|jrC| rCt dd} |durL| |}|}| rW|durWt }| durs|durc| nd}t j|||jd|jd} |dur|| d}|||| || }|||}| rdnd}| rdnd}d}t|jD]k\}}| r||f7}||jv}|dup|duo| |dk}|r|dur|rq|jr|jr||j|||||||| | | | }n||f||||||| | | |d |}|d}| r|| rd nd}| r ||df7}q||}| r||f7}| r|nd}| s.td d ||||fDSt||||d S)aQ cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile full_text_row_masked_out_mask (`Tuple[torch.Tensor, torch.Tensor]`, *optional*): A tuple containing two tensors that mask out rows in the cross-attention mechanism: - The first tensor has shape `(batch_size, 1, seq_length, 1)` and contains values of 0 or 1. A value of 0 indicates that the corresponding text token's entire row in the cross-attention matrix should be masked out (all image tokens ignored). - The second tensor has the same shape and is used internally to apply the masking during the forward pass of cross-attention layers. This mask is derived from the cross_attention_mask and is used to handle cases where a text token should not attend to any image token. Example: ```python >>> from transformers import AutoProcessor, MllamaTextModel >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaTextModel.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> text = "<|image|>If I had to write a haiku for this one" >>> inputs = processor(text=text, return_tensors="pt") >>> output = model(**inputs) >>> print(output.last_hidden_state.shape) torch.Size([1, 13, 4096]) ``` N:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Frrr r7) rr"rBr6rrrrrrr_csrrgr7rr7r7r8rrz*MllamaTextModel.forward..rrror)rErrrrrrrrrrJr rr/r"r*rr,rrN enumeraterrKrrrMrr)rRrUrBrrr"r6rrrrrrrrrorrrZall_hidden_statesZall_self_attnsZnext_decoder_cacheidxZ decoder_layerZis_cross_attention_layerZis_cross_attention_cache_emptyrZ next_cacher7r7r8rXs9             zMllamaTextModel.forward) NNNNNNNNNNNNN)rYrZr[rr$r%rIr8rSrrr/rr\rrrr rr0rrrrXr]r7r7rSr8rFsf      rFc@s eZdZdS)KwargsForCausalLMN)rYrZr[r7r7r7r8rZs rZzE The Mllama Text Model with a language modeling head on top. c&sHeZdZeZdZdZdgZfddZddZ dd Z d d Z d d Z ddZ ddZe               d'deejdeejdeejdeejdeejdeeejejfdeeeeejfdeejdeejdeedeedeed eed!eejd"eeejfd#eed$eeeff"d%d&ZZS)(MllamaForCausalLMTlanguage_modellm_head.weightcsVt|||_|jj|_t|j|_tj |jj |jdd|_ | dSr) rHrIr text_configrIrF _from_configmodelrrjrJlm_headr6rnrSr7r8rIs    zMllamaForCausalLM.__init__cC|jjSrgr`rJrr7r7r8r8sz&MllamaForCausalLM.get_input_embeddingscCs ||j_dSrgrcrRr7r7r8rSs z&MllamaForCausalLM.set_input_embeddingscCr7rgrarr7r7r8get_output_embeddingsrPz'MllamaForCausalLM.get_output_embeddingscCrQrgrdrRZnew_embeddingsr7r7r8set_output_embeddingsrTz'MllamaForCausalLM.set_output_embeddingscCrQrgr`)rRdecoderr7r7r8 set_decoderrTzMllamaForCausalLM.set_decodercCr7rgrhrr7r7r8 get_decoderrPzMllamaForCausalLM.get_decoderNrrUrBrrr"r6rrlabelsrrrrrlogits_to_keeprr%cKs| dur| n|jj} | dur| n|jj} | dur| n|jj} |jd||||||||| | | | |d |}|d}t|trCt| dn|}||dd|ddf }d}| durg|j || |j fi|}| s}|f|dd}|dur{|f|S|St |||j |j|jdS)a cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. full_text_row_masked_out_mask (`Tuple[torch.Tensor, torch.Tensor]`, *optional*): A tuple containing two tensors that mask out rows in the cross-attention mechanism: - The first tensor has shape `(batch_size, 1, seq_length, 1)` and contains values of 0 or 1. A value of 0 indicates that the corresponding text token's entire row in the cross-attention matrix should be masked out (all image tokens ignored). - The second tensor has the same shape and is used internally to apply the masking during the forward pass of cross-attention layers. This mask is derived from the cross_attention_mask and is used to handle cases where a text token should not attend to any image token. Example: ```python >>> from transformers import AutoTokenizer, MllamaForCausalLM >>> model = MllamaForCausalLM.from_pretrained("Llama-3.2-11B-Vision") >>> tokenizer = AutoTokenizer.from_pretrained("Llama-3.2-11B-Vision") >>> prompt = "If I had to write a haiku, it would be:" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=40, do_sample=True, temperature=0.6) >>> result = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] >>> print(result) If I had to write a haiku, it would be: "Snowflakes gently fall" - simple, yet peaceful. I love the idea of snowflakes gently falling, each one ``` N) rUrrBrr"r6rrrrrrrrrlosslogitsrrorr7)rErrrr`rrslicerar loss_functionrIrrror)rRrUrBrrr"r6rrrlrrrrrrmrrro slice_indicesrpror?r7r7r8rXsJ@ zMllamaForCausalLM.forward)NNNNNNNNNNNNNNr) rYrZr[rr$r&r%_tied_weights_keysrIr8rSrergrjrkrrr/rr\rrr rrr0rrrZrrXr]r7r7rSr8r[s~         r[zr The Mllama model which consists of a vision encoder and a language model without language modeling head. c'seZdZddiZdZdeffdd ZddZd d Ze e dd e e j d e e jde e jde e jde e jde e jde e jde e j de ee jde e jde ede ede ede ede e j deedeeeff"ddZZS) MllamaModelrGr\FrEcst||jj|_|jj|_|jj|_|jj|_|jj dur$|jj nd|_ t |j|_ t |j|_tj|jj|jjdd|_|dS)Nr(Tr)rHrIr^rIrJZ vision_configrAZvision_output_dimrErHrr_r)rFr\rrjmulti_modal_projectorr6rnrSr7r8rITs      zMllamaModel.__init__cC |jSrg)r\r8rr7r7r8r8erTz MllamaModel.get_input_embeddingscC|j|dSrg)r\rSrRr7r7r8rShz MllamaModel.set_input_embeddingsNrUr:r:rVrBr"rrrrrrrrrrr%cKs|| dur| n|jj} | dur| n|jj} |dur|n|jj}|du| duAr*td|dur6| dur6td|durB|durBtd|durk|durNtd|j|||| | |d}|d}||d|jd |j }|dur|t ||jj |j d \}}nd}|dur|dur|dddd|f}|dddd|f}|j d||||||| | | | | d |d |}t|j|j|j|jd S)ar aspect_ratio_mask (`torch.Tensor` of shape `(batch_size, max_num_images, max_num_tiles)`, *optional*): Mask to avoid performing attention on padding tiles. Mask values selected in `[0, 1]`: - 1 for tiles that are **not masked**, - 0 for tiles that are **masked**. aspect_ratio_ids (`torch.Tensor` of shape `(batch_size, max_num_images)`, *optional*): Aspect ratio ids used to select the appropriate precomputed tile embeddings based on the aspect ratio of each input image. These ids correspond to indices in the model's list of supported aspect ratios, offset by 1. For example, if the model supports aspect ratios [[1, 1], [1, 2], [2, 1]]: - An image with aspect ratio [1, 1] would have ID 1 - An image with aspect ratio [1, 2] would have ID 2 - An image with aspect ratio [2, 1] would have ID 3 The id 0 is reserved for padding (i.e., no image). If an image has aspect ratio [1, 2], that means it was split into 2 tiles horizontally, and its `aspect_ratio_id` would be 2. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. NrVzdYou cannot specify both pixel_values and inputs_embeds at the same time, and must specify either onezM`pixel_values` and `cross_attention_states` cannot be provided simultaneouslyzA`aspect_ratio_ids` must be provided if `pixel_values` is provided)r:rVr:rrrrr(r=)r#r$T) rUrBrrr"r6rrrrrrrrWr7)rErrrrr)rvr?r*rJr9r;r$r\rrrror)rRrUr:r:rVrBr"rrrrrrrrrrZvision_outputsr6rr7r7r8rXksx2  zMllamaModel.forward)NNNNNNNNNNNNNNN)rYrZr[_checkpoint_conversion_mappingr'rrIr8rSrrrr/rrr\rr0rrrrrrXr]r7r7rSr8ruKst      ruzS The Mllama model which consists of a vision encoder and a language model. c+seZdZdddddZdZdgZdeffd d Zd d Zd dZ ddZ ddZ e ddZ e ddZee                 d2deejdeejdeejdeejdeejdeejdeejd eejd!eeeeejfd"eejd#eejd$eed%eed&eed'eed(eejd)eeejfd*eed+eeeff&d,d-Z            d3fd.d/ Z fd0d1Z!Z"S)4MllamaForConditionalGenerationzmodel.language_modelzmodel.vision_modelzmodel.multi_modal_projectorra)z^language_model.modelz ^vision_modelz^multi_modal_projectorz^language_model.lm_headFr]rEcs<t|t||_tj|jj|jjdd|_ | dSr) rHrIrur`rrjr^rJrIrar6rnrSr7r8rIs   z'MllamaForConditionalGeneration.__init__cCrwrg)r`r8rr7r7r8r8rTz3MllamaForConditionalGeneration.get_input_embeddingscCrxrg)r`rSrRr7r7r8rSryz3MllamaForConditionalGeneration.set_input_embeddingscCr7rgrdrr7r7r8rerPz4MllamaForConditionalGeneration.get_output_embeddingscCrQrgrdrfr7r7r8rgrTz4MllamaForConditionalGeneration.set_output_embeddingscCrbrg)r`r\rr7r7r8r\z-MllamaForConditionalGeneration.language_modelcCrbrg)r`r)rr7r7r8r) r|z+MllamaForConditionalGeneration.vision_modelNrrUr:r:rVrBr"rrrrrlrrrrrrmrr%cKs| dur| n|jj} |dur|n|jj}|dur|n|jj}|jd||||||||| | | | |d|d|}|d}t|trEt| dn|}||dd|ddf}d}| duri|j || |jj j fi|}t |||j |j|jdS)af aspect_ratio_mask (`torch.Tensor` of shape `(batch_size, max_num_images, max_num_tiles)`, *optional*): Mask to avoid performing attention on padding tiles. Mask values selected in `[0, 1]`: - 1 for tiles that are **not masked**, - 0 for tiles that are **masked**. aspect_ratio_ids (`torch.Tensor` of shape `(batch_size, max_num_images)`, *optional*): Aspect ratio ids used to select the appropriate precomputed tile embeddings based on the aspect ratio of each input image. These ids correspond to indices in the model's list of supported aspect ratios, offset by 1. For example, if the model supports aspect ratios [[1, 1], [1, 2], [2, 1]]: - An image with aspect ratio [1, 1] would have ID 1 - An image with aspect ratio [1, 2] would have ID 2 - An image with aspect ratio [2, 1] would have ID 3 The id 0 is reserved for padding (i.e., no image). If an image has aspect ratio [1, 2], that means it was split into 2 tiles horizontally, and its `aspect_ratio_id` would be 2. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, MllamaForConditionalGeneration >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaForConditionalGeneration.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> prompt = "<|image|>If I had to write a haiku for this one" >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=prompt, images=image, return_tensors="pt") >>> # Generate >>> output = model.generate(**inputs, max_new_tokens=15) >>> prompt_len = inputs.input_ids.shape[-1] >>> generated_ids = output[:, prompt_len:] >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False) >>> print(generated_text) [', it would be:.\\nA stop sign in Chinatown.\\n'] ``` NT)rUr:r:rVr"rrBrrrrrrrrrrnr7)rErrrr`rrrqrarrr^rIrrror)rRrUr:r:rVrBr"rrrrrlrrrrrrmrrrorsrpror7r7r8rXsHSz&MllamaForConditionalGeneration.forwardc  sTtj|f| | |||||||| | d | }| ddkr(d|d<d|d<d|d<|S)N) rrrrrBr:rVr:r"rrmrr:rVr:)rHprepare_inputs_for_generation)rRrUrrBrr:rVr:r"rrrrmrZ model_inputsrSr7r8r}s,  zs           & A1Xc PU>.cL _