fThd^>SrSSKrSSKJrJrJrJrJr SSKrSSK J s J r SSK rSSKJ r SSKJr SSKJrJr SSKJr SS KJr SS KJr SS KJrJrJr SS KJrJr SS K J!r!J"r" SSK#J$r$ SSK%J&r&J'r'J(r(J)r)J*r* SSK+J,r,J-r-J.r. \)"5(a SSK/J0r0 SSK1J2r2 \*Rf"\45r5S\RlS\7S\8S\\Rl\Rl44Sjr9S\RlS\7S\7S\RtS\Rl4 Sjr;"SS\ Rx5r="SS \ Rx5r>"S!S"\ Rx5r?S#\RlS$\7S\Rl4S%jr@SYS&\ RxS'\RlS(\RlS)\RlS*\\RlS+\AS,\A4S-jjrB"S.S/\ Rx5rC"S0S1\ Rx5rD"S2S3\ Rx5rE"S4S5\ Rx5rF"S6S7\ Rx5rGS8rHSZS9jrI"S:S;\ Rx5rJ"S<S=\ Rx5rK"S>S?\ Rx5rL"S@SA\RRx5rM"SBSC\ Rx5rN\'"SDSE\"55rO\'"SFSG9"SHSI\O55rP\'"SJSG9"SKSL\O55rQ"SMSN\\&5rR\'"SOSG9"SPSQ\O\55rS\'"SRSG9"SSST\O55rT\'"SUSG9"SVSW\O\55rU/SXQrVg)[zPyTorch Mllama model.N)CallableListOptionalTupleUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)BaseModelOutputBaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging) MllamaConfigMllamaTextConfigMllamaVisionConfig) BlockMask)make_flex_block_causal_maskcross_attention_masknum_vision_tokensdtypereturncURtp4nURUSS9nURX4S5nURS5nSU- R U5nUR UR [ R5[ R"U5R5n[ R"U5RnX:gRSS9RU5SnX-nX4$)Nr dimr?).N) shaperepeat_interleaveview unsqueezeto masked_filltorchboolfinfominanytype_as) r#r$r% batch_sizetext_total_length_inverted_cross_attn_masknegative_inf_valuefull_text_row_masked_out_masks b/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/mllama/modeling_mllama.py_prepare_cross_attention_maskr?/s )=(B(B%JA/AABSYZA[/44ZTVW/99!<!$&: :>>uE3?? ##EJJ/U1C1G1G U+//  388R8@HHI]^_hi"9  >>aspect_ratio_mask num_patches target_lengthctURupEURXESS5RU5nURSSUS5nX!- nSUSS2SS2U*S24'SU- nUR XEU-S5nXfR SS5-[ R"U5R-nURS5nU$)Nrrr*) r,r.r0repeatreshape transposer2r4r5r/)rArBrCr%r8 max_num_tilesattention_mask pad_patchess r>$_prepare_aspect_ratio_attention_maskrLKs!2 7 7J&++Jq!LOOPUVN#**1aBN -K*+N1a+&''N$++J 8UWXYN#&>&>r2&FFUZI[I_I__N#--a0N r@c^\rSrSrS S\S\4U4SjjjrS\RS\RS\R4Sjr S r U=r $) %MllamaPrecomputedAspectRatioEmbeddingfconfigis_gatedc>[TU]5 URUlURUlURUlX l[ R"URS-URUR-5UlU(a0[ R"[R"S55Ul ggNr) super__init__rI hidden_sizemax_aspect_ratio_idrQr Embedding embedding Parameterr2zerosgateselfrPrQ __class__s r>rU.MllamaPrecomputedAspectRatioEmbedding.__init__gs #11!--#)#=#=  d&>&>&BDDVDVY]YiYiDij  U[[^4DI r@ hidden_stateaspect_ratio_idsr&cURU5nURSURSUR5nUR(aX0R R 5-nX-nU$)Nr*r)rYrGrIrVrQr\tanh)r^rarb embeddingss r>forward-MllamaPrecomputedAspectRatioEmbedding.forwardrs[^^$45 ''D,>,>4CSCST ==#iinn&66J#0 r@)rYr\rVrQrWrI)T) __name__ __module__ __qualname____firstlineno__r r3rUr2Tensorrf__static_attributes__ __classcell__r_s@r>rNrNfsJ 51 5T 5 5ELLELLUZUaUar@rNc^\rSrSrS\4U4SjjrS\RS\RS\R4SjrSr U=r $) "MllamaPrecomputedPositionEmbedding}rPc>[TU]5 URUlURUlURUR -S-S-UlURUlURS-Ul[R"[R"S55Ul [R"UR UR5n[R"URU-5Ul[R "URS-URUR -UR-5Ulg)Nr)rTrUrIrW image_size patch_sizerBrVscalerrZr2r[r\randnrYrXtile_embedding)r^rPposition_embeddingr_s r>rU+MllamaPrecomputedPositionEmbedding.__init__~s #11#)#=#= "--1B1BBqH1L!--''- LLQ0 #[[)9)94;K;KLdjj3E&EF!ll  $ $q ($*<*rf*MllamaPrecomputedPositionEmbedding.forwards$% (8$8DNN#J #&C&CAq$JZJZ\`\l\l&mm #'"5"56F"G!''* "9"A"A **D,<,N>N# )- (8;R(R%#C r@)rYr\rVrWrIrBrxrz) rhrirjrkr rUr2rlrfrmrnros@r>rqrq}s> 1 &ELLELLUZUaUar@rqcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)MllamaVisionMLPc>[TU]5 Xl[URUl[ R"URUR5Ul [ R"URUR5Ul gN) rTrUrPr hidden_act activation_fnrLinearrVintermediate_sizefc1fc2r^rPr_s r>rUMllamaVisionMLP.__init__sb  #F$5$5699V//1I1IJ99V55v7I7IJr@ hidden_statesr&clURU5nURU5nURU5nU$r)rrr)r^rs r>rfMllamaVisionMLP.forwards4/ **=9 / r@)rrPrr) rhrirjrkrUr2rlrfrmrnros@r>rrs)KU\\ellr@rrn_repcURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r,expandrG)rrbatchnum_key_value_headsslenhead_dims r> repeat_kvrs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr@modulequerykeyvaluerJscalingdropoutc @[X R5n[X0R5n [R"XR SS55U-n Ub"USS2SS2SS2SUR S24n X-n [ RRU S[RS9RUR5n [ RRXURS9n [R"X5n U R SS5R5n X4$)Nrtr rEr*)r)r%)ptrainingr)rnum_key_value_groupsr2matmulrHr,r functionalsoftmaxfloat32r0r%rr contiguous) rrrrrJrrkwargs key_states value_states attn_weights causal_mask attn_outputs r>eager_attention_forwardrs3 ; ;[TU]5 XlURUlUR UlURUR -UlURS-UlSUl [R"URUR UR-SS9Ul [R"URUR UR-SS9Ul [R"URUR UR-SS9Ul[R"UR UR-URSS9Ulg)NrurFbias)rTrUrPrV embed_dimattention_heads num_headsrrrrrq_projk_projv_projo_projrs r>rUMllamaVisionAttention.__init__s  ++//**f.D.DD }}d* $%!ii0NUZ[ ii0NUZ[ ii0NUZ[ ii >UZ[ r@rarJoutput_attentionsr&c URU5nURU5nURU5nURupn URupn UR XUR UR 5RSS5nUR XUR UR 5RSS5nUR XUR UR 5RSS5n[n URRS:waTURRS:XaU(a[RS5 O[URRn U "UUUUU4SURS.UD6upU RXS5R!5n UR#U 5n U(dSnX4$) Nrrteagersdpa`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rrr*)rrrr,r.rrrHrrP_attn_implementationlogger warning_oncerrrGrr)r^rarJrrrrrr8 q_seq_lenr: kv_seq_lenattention_interfacerrs r>rfMllamaVisionAttention.forwards L)kk,' L)#(;; q99q :$..$--PZZ[\^_`hhzt~~t}}MWWXY[\] :4>>4==Q[[\]_`a(? ;; + +w 6{{//69>O##L '>dkk>^>^&_#$7      % LL %  % ! "))*DOOQ kk+.  L((r@) rPrrrrrrrrrNN)rhrirjrkr rUr2rlrr3rrfrmrnros@r>rrsr \1 \$26,0 .)ll.)!..)$D> .) u||Xell33 4 .).)r@rc^\rSrSrS S\S\4U4SjjjrS S\RS\ \RS\ \4Sjjr S r U=r $) MllamaVisionEncoderLayerirPrQc>[TU]5 URUlURUlX lUR Ul[U5Ul[U5Ul [R"URURS9Ul[R"URURS9UlU(a[R "["R$"S5[&R(-S- 5Ul[R "["R$"S5[&R(-S- 5Ulgg)Nepsr)rTrUrVrnum_attention_headsrQrr self_attnrmlpr LayerNormnorm_epsinput_layernormpost_attention_layernormrZr2onesmathpi gate_attngate_ffnr]s r>rU!MllamaVisionEncoderLayer.__init__s !--#)#9#9  !'!9!9.v6"6*!||D,<,<&//R(* T5E5E6??([% \\%**Q-$''*AA*EFDNLLA)@1)DEDM r@rarJrc|UnURU5nURXS9upUR(aURR 5U-nXA-nUnUR U5nUR U5nUR(aURR 5U-nXA-nU4nU(aXe4- nU$)N)rJ)rrrQrrdrrr)r^rarJrresidualroutputss r>rf MllamaVisionEncoderLayer.forward.s ++L9 %)^^L^%`" ==>>..0<?L.  44\B xx - ====--/,>L. /   &Gr@) rrrVrrrQrrrrFr) rhrirjrkr r3rUr2rlrrfrmrnros@r>rrs_F1FTFF*26,0 ll!.$D> r@rc^\rSrSrSrS S\4U4SjjjrSS\RS\ \RS\ \ S\ \ S \ \ S \ \ \ 44 S jjrS rU=r$)MllamaVisionEncoderiLz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MllamaEncoderLayer`]. Args: config: MllamaConfig rPc >[TU]5 Xl[R"[ U5Vs/sHn[ X5PM sn5UlSUlXlgs snf)NF) rTrUrPr ModuleListrangerlayersgradient_checkpointing)r^rP num_layersrQr:r_s r>rUMllamaVisionEncoder.__init__UsU  mmY^_iYj$kYjTU%=f%OYj$kl &+# %lsA"rrJroutput_hidden_states return_dictr&c*UbUOURRnUbUOURRnUbUOURRnU(aSOSnU(aSOSnURHlnU(aXa4-nUR (a0UR (aURURUUU5n OU"UUUS9n U(aXyS4-nU SnMn U(aXa4-nU(d[SXU455$[XUS9$)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. N)rarJrrrc3.# UH ocMUv M g7frr.0vs r> .MllamaVisionEncoder.forward..se$Sq$S last_hidden_stater attentions) rPrruse_return_dictrrr_gradient_checkpointing_func__call__tupler) r^rrJrrrencoder_statesall_attentions encoder_layer layer_outputss r>rfMllamaVisionEncoder.forward\s!<2C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]N$See e+Vd  r@)rPrr) F)NNNN)rhrirjrk__doc__r rUr2rlrr3rrrrfrmrnros@r>rrLs126,0/3&* D ||D !.D $D> D 'tn D d^ D  uo% &D D r@rc8^\rSrSrSU4SjjrSrSrSrU=r$)MllamaTextRMSNormic>[TU]5 [R"[R "U55UlX lg)z0 MllamaTextRMSNorm is equivalent to T5LayerNorm N)rTrUrrZr2rweightvariance_epsilon)r^rVrr_s r>rUMllamaTextRMSNorm.__init__s/ ll5::k#:; #r@cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)Nrtr*T)keepdim) r%r0r2rpowmeanrsqrtr r )r^r input_dtypevariances r>rfMllamaTextRMSNorm.forwardsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r@c^[URR5SUR3$)Nz, eps=)rr r,r r^s r> extra_reprMllamaTextRMSNorm.extra_reprs* ))*+6$2G2G1HIIr@)r r )gư>) rhrirjrkrUrfrrmrnros@r>rrs$;JJr@rcN^\rSrSrSrSS\\S\\4U4SjjjrSS\ RS\\ RS\\ S \\ RS \ S \\ S \\ RS \\ R\\ R\\\ R44SjjrSrU=r$)MllamaTextCrossAttentioniz=Multi-headed attention from 'Attention Is All You Need' paperrP layer_idxc>[TU]5 XlURRUlURR UlUR UlURUlURUR-UlX l URUR -Ul URS-Ul [R"URURUR-SS9Ul[R"URUR UR-SS9Ul[R"URUR UR-SS9Ul[R"URUR-URSS9Ul[%URUR&S9Ul[%URUR&S9Ulg)NruFrr)rTrUrPrrrrrVrrrrrrrrrrr rms_norm_epsq_normk_normr^rPrr_s r>rU!MllamaTextCrossAttention.__init__sg  88#';;#B#B ~~ !--**dnn< "$(NNd6N6N$N!}}d* ii 0 0$..4==2PW\] ii 0 0$2J2JT]]2Zafg ii 0 0$2J2JT]]2Zafg ii >@P@PW\] ' 6;N;NO ' 6;N;NO r@rcross_attention_statespast_key_valuerJr use_cachecache_positionr&c UR5upn URU5n U RXURUR5R SS5n UR U 5n UbURU5n URU5nU RU SURUR5R SS5n URU SURUR5R SS5nURU 5n Ub!URXURSU05upOFUSS:wa2URURURURpO [S5e[ nUR"R$S:waTUR"R$S :XaU(a[&R)S 5 O[*UR"R$nU"UU U UU4UR,(dS O UR.UR0S .UD6unnUR3XS5R55nUR7U5nU(dSnUUU4$) z#Input shape: Batch x Time x ChannelrrtNr*r%rz^Cross attention layer can't find neither `cross_attn_states` nor cached values for key/values!rrrrr)sizerr.rrrHrrrrrupdater key_cache value_cache ValueErrorrrPrrrrrrrrGrr)r^rr"r#rJrr$r%rbszq_lenr: query_statesrrrrrs r>rf MllamaTextCrossAttention.forwardsF&**, A{{=1 #((T^^T]]S]]^_abc {{<0 ! -%;L#b$2J2JDMMZddefhijJ',,S"d6N6NPTP]P]^hhijlmnLZ0J),:+@+@dnn?OQ_>`,( A ! #((8**4>>:% p )@ ;; + +w 6{{//69>O##L '>dkk>^>^&_#$7      % $}}C$,,LL %  % ! \"))#b9DDF kk+.  LL.88r@)rPrrrVrrrrrrrrrrrr)NNNFNN)rhrirjrkrrrintrUr2rlr r3 LongTensorrrfrmrnros@r>rrsG.2#'P)*PC=PP6:>*.15"'$(59D9||D9!) 6D9! D9 !. D9  D9D>D9!!1!12D9 u||Xell3XeELL>Q5RR SD9D9r@rcUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..Nr*rtr()r,r2cat)xx1x2s r> rotate_halfr7sZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r@cURU5nURU5nX-[U5U--nX-[U5U--nXg4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )r/r7)qkcossin position_ids unsqueeze_dimq_embedk_embeds r>apply_rotary_pos_embrA#sS( -- &C -- &Cw;q>C/0Gw;q>C/0G  r@c ^\rSrSrS\S\4U4SjjrS S\RS\RS\RS\ S \ 4 S jjr S r U=r $) MllamaTextSelfAttentioni>rPrcd>[TU]5 XlURUlUR UlUR UlURUlUR UR-UlURUR-Ul URS-Ul URUl X l [R"UR URUR-SS9Ul[R"UR URUR-SS9Ul[R"UR URUR-SS9Ul[R"URUR-UR SS9Ulg)NruFr)rTrUrPrrrrVrrrr rope_thetarrrrrrrr s r>rU MllamaTextSelfAttention.__init__?s:  33~~ !--#)#=#= **dnn< $(NNd6N6N$N!}}d*  ++"ii 0 0$..4==2PW\] ii 0 0$2J2JT]]2Zafg ii 0 0$2J2JT]]2Zafg ii >@P@PW\] r@rrJposition_embeddingsrr$c (UR5upn URU5n URU5n URU5nU R XUR UR 5RSS5n U R XURUR 5RSS5n UR XURUR 5RSS5nUunn[XUU5upUb$UXS.nURXURU5up[nURRS:waTURRS:XaU(a[R!S5 O["URRnU"UU U UU4UR$(dSO UR&UR(S.UD6unnUR+XS 5R-5nUR/U5nU(dSnUUU4$) Nrrt)r<r;r%rrrrrr*)r'rrrr.rrrHrrAr(rrrPrrrrrrrrGrr)r^rrJrGrr$r#r%rr,r-r:r.rrr;r< cache_kwargsrrrs r>rfMllamaTextSelfAttention.forwardQs&**, A{{=1 [[/ {{=1 #((T^^T]]S]]^_abc __S1I1I4==Yccdeghi #((T5M5Mt}}]gghiklm &S#7 RUWZ#[  %#&sUL'5'<'O##L '>dkk>^>^&_#$7      % $}}C$,,LL %  % ! \"))#b9DDF kk+.  LL.88r@)rPrrrVrrrrrrrrErr)FFNN) rhrirjrkrr0rUr2rlr3rfrmrnros@r>rCrC>sn^/^C^.#(99||99 99#\\ 99  99  9999r@rCc.^\rSrSrU4SjrSrSrU=r$) MllamaTextMLPic>[TU]5 XlURUlURUl[ R "URURSS9Ul[ R "URURSS9Ul[ R "URURSS9Ul [URUl gNFr) rTrUrPrVrrr gate_projup_proj down_projr ract_fnrs r>rUMllamaTextMLP.__init__s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ r@cURURURU55URU5-5nU$r)rQrRrOrP)r^r4rQs r>rfMllamaTextMLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r@)rRrPrQrOrVrrP)rhrirjrkrUrfrmrnros@r>rLrLs 0r@rLc ^\rSrSrS\S\4U4SjjrSS\RS\ \RS\ \RS\ \RS \ \ \R\R4S \ \RS \ \ S \ \ S \ \ S\ \RS\ \ \R\R4S\\S\ \R \ \ \R \R 444SjjrSrU=r$)MllamaSelfAttentionDecoderLayerirPrc>[TU]5 URUl[XS9Ul[ U5Ul[URURS9Ul [URURS9Ul X l g)N)rPrr) rTrUrVrCrrLrrrrrrr s r>rU(MllamaSelfAttentionDecoderLayer.__init__sp !--0T (01C1CI\I\](9&:L:LRXReRe(f%"r@rr"r#rJr=r=r#rr$r%rGrr&c Un URU5nUR"SUUUUUU U U S.U D6upnX-nUn URU5nURU5nX-nU4nU(aUU4- nU (aUU4- nU$)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`Tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )rrJr=r#rr$r%rGr)rrrr)r^rr"r#rJr=r=r#rr$r%rGrrself_attn_weightspresent_key_valuers r>rf'MllamaSelfAttentionDecoderLayer.forwardsJ!,,]; ?Cnn ? ')%)/) 3 ?  ? ; *;!0 !55mD /  0  "  )+ +G  )+ +Gr@)rVrrrrr) NNNNNNFFNN)rhrirjrkrr0rUr2rlrrr1r r3rr FloatTensorrfrmrnros@r>rWrWsp #/ #C #:>7;15UY37*.,1$)59KOE||E!) 6E'u||4 E !. E (0ellELL6P0Q'R Eu//0E!E$D>ED>E!!1!12E&eELL%,,,F&GHE-.E u  (51B1BEDUDU1U+V"WW XEEr@rWc^\rSrSrSrS\S\SS4U4SjjrSS\RS \RS \RS \RS \ \R\R4S \ \RS\ \ S\ \S\ \S\ \RS\ \RS\\S\ \R4SjjrSrU=r$) MllamaCrossAttentionDecoderLayerizLCross-attention transformer block with tanh-gated attention and feedforward.rPrr&Nc>[TU]5 X l[XS9Ul[ UR URS9Ul[RR[R"S55Ul [U5Ul[ UR URS9Ul[RR[R"S55Ulg)N)rrr)rTrUrr cross_attnrrVrrr2rrZr[cross_attn_attn_gaterLrrcross_attn_mlp_gater s r>rU)MllamaCrossAttentionDecoderLayer.__init__s "26O01C1CI\I\]$)HH$6$6u{{1~$F! ((9&:L:LRXReRe(f%#(88#5#5ekk!n#E r@rr"r#rJr=r=r#rr$r%rGrc |Un URU5nUR"SUUUUUU S.U D6upnXRR5U--nUn UR U5nUR U5nUb USS2S4U-nXR R5U--nU4nU(aX4- nU (aX4- nU$)N)rrJr"r#rr%rr)rrbrcrdrrrd)r^rr"r#rJr=r=r#rr$r%rGrrrrs r>rf(MllamaCrossAttentionDecoderLayer.forwards!,,]; 6:oo7 '/#9)/) 7 7 3 ^!#<#<#A#A#Cm#SS  55mD / ( 49!Q$?-OM #;#;#@#@#B]#RR  "   &G  ( (Gr@)rbrcrdrrrr)NNFFNN)rhrirjrkrrr0rUr2rlrrr1r r3rrrfrmrnros@r>r`r`sV F/ FC FD F&48*.,1$)596:,||,!& ,$ll ,  , (-U\\5<<-G'H ,u//0,!,$D>,D>,!!1!12,&ell3,-., u|| ,,r@r`cl^\rSrSrSS\4U4Sjjjr\R"5\S55r Sr U=r $)MllamaRotaryEmbeddingi3rPcT>[TU]5 URSUlURUlURUlXl[URUl URURU5uo0l URSUSS9 URUl g)N rope_typeinv_freqF) persistent)rTrU rope_scalingrkmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrPr rope_init_fnattention_scalingregister_bufferrloriginal_inv_freq)r^rPdevicerlr_s r>rUMllamaRotaryEmbedding.__init__4s ,,[9"("@"@$*$B$B! /?+/+<+rfMllamaRotaryEmbedding.forward@s@!MM$4-8>>@GG HZHZ[\H]_acde ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk ^^ U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs BF F)rsrPrprurqrrrkr) rhrirjrkrrUr2no_gradrrfrmrnros@r>riri3s7 // / / ]]_ < RR5 g[ U[@5(a7UR4(a%UR0RR5 ggg)Ninitializer_ranger)rstdr+)r)!getattrrPget_text_configrr~rrConv2dr datanormal_rzero_rX padding_idxrfill_rMllamaVisionModelinitclass_embeddingrqrYzeros_r\rrQrrr`rcrdrN)r^rrs r> _init_weights#MllamaPreTrainedModel._init_weightsasdkk#6 8S8S8U8g8gh fryy"))4 5 5 MM   & &CS & 9{{&   &&('  - - MM   & &CS & 9!!- ""6#5#56<<>.  - - MM   $ $S ) KK   " " $  1 2 2 MM   $ $S )  1 2 2 GGOOF2277SO A  B C C GGOOF,,11sO ; GGNN6;;++ ,  8 9 9foo GGOOF,,11sO ; GGOOFOO00cO :  @ A A  ' ' , , 2 2 4  & & + + 1 1 3  E F F   &&(Gr@rJr! input_tensorr%past_key_valuesrc URRS:XaUbUS:HR5(aU$gURRS:Xa,[U[R 5(a [ U5nU$UbUR5OSnUb UROSnURRS:Xa5U(d.U(d'[R"UUUURS9(agURnURSn U(aUR5n O5[U[R 5(aURS OXi-S-n URUU U UUURSS 9n URRS:XaZUbWUR R"S ;a=U(d6[R$"U5R&n [R("X5n U $) Nflash_attention_2rflex_attentionrFr) inputs_embedspast_key_values_length is_trainingrr*)sequence_lengthrCr%r%r8)cudaxpunpu)rPrr6r~r2rlr"get_seq_lengthis_compileabler_ignore_causal_mask_sdparr%r,get_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrvrr4r5_unmask_unattended) r^rJrr%rrpast_seen_tokensusing_compilable_cacher%rrCr min_dtypes r>_update_causal_mask)MllamaPreTrainedModel._update_causal_masks ;; + +/B B)~/D.I.I.K.K%% ;; + +/? ?.%,,77!<^!L! ! @O?Z?99;`aCRC^!?!?di ;; + +v 5>T]n%>>*'7 MM  ""&,,Q/ !+??AMnell;;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCK[Kr@rrCr%r8c UbUR5S:XaUnU$[R"U5Rn[R"X4XUR S9nUS:wa[R "USS9nU[R"X$R S9URSS5:-nUSSSS2SS24RUSSS5nUbUR5nURSn USS2SS2SS2SU 24USS2SSSS24RUR 5-n U S:Hn USS2SS2SS2SU 24RX5USS2SS2SS2SU 24'U$) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr) fill_valuer%rvr)diagonalrvr*r)r)r2r4r5fullrvtriuarangerGrcloner,r0r1) rJrrCr%r%r8rrr mask_length padding_masks r>rKMllamaPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_positions}>  %.*<*<*>!*C(K*' E*..I** 0Y\j\q\qK!##jjqA 5<< >S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12r@rNr)rhrirjrkr config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_cache_class_supports_static_cache_supports_sdpa_supports_flash_attn_2_supports_quantized_cache_supports_attention_backendrrr2rlr r3r staticmethodr0r%rrmrr@r>rrPsL&*# !"N! $"&)L#( BellK78BllB B  B  BH4 444{{ 4  4  44r@rzH The Mllama Vision Model which consists of two vision encoders. ) custom_introc(^\rSrSr\rSrS\4U4SjjrSrS\ RS\ R4Sjr \ SS \ RS \ RS \ RS \ \S \ \S\ \S\\\\ RS444Sjj5rSrU=r$)ri vision_modelrPc >[TU]U5 URUlURUlURUlUR UlUR UlURUlURUR-S-S-UlUR S-Ul [R"UR UR URURSSS9Ul [R"UR[R"UR 5-5Ul[#U5Ul['USS9Ul['USS9Ul[R,"UR 5Ul[R,"UR 5Ul[3XR4SS9UlR8SS9UlUR=5 g) NrtrruvalidF) in_channels out_channels kernel_sizestridepaddingrT)rQ)rTrUrvrwrIrV num_channelsintermediate_layers_indicesrBrxrrpatch_embeddingrZr2ryrrqgated_positional_embeddingrNpre_tile_positional_embeddingpost_tile_positional_embeddingr layernorm_prelayernorm_postrnum_hidden_layers transformernum_global_layersglobal_transformer post_initrs r>rUMllamaVisionModel.__init__s   ++ ++#11!--"//+1+M+M( OOt>1DqH''- !yy++))??   "||DJJTEUEU9V,VW*LV*T'-RSYdh-i*.STZei.j+ \\$*:*:; ll4+;+;</v7O7OZ_`"5f>V>Vae"f r@cUR$)zW This function is used to fetch the first embedding layer to activate grads on inputs. )rrs r>get_input_embeddings&MllamaVisionModel.get_input_embeddings,s###r@rar&cURup#nURRUSU5n[R"XQ/SS9nU$)Nrr()r,rrr2r3)r^rar8r:rVrs r>apply_class_embedding'MllamaVisionModel.apply_class_embedding2sE%1%7%7" {..55j![Qyy/!@aH r@ pixel_valuesrbrArrr.c6UbUOURRnUbUOURRnUbUOURRnURupxppUR Xx-U -XU 5nUR Xx-S5nUR RRn UR RRnUR URX55nURS5RSS5nURunnnUR Xx-U SU5nURUU5nUR Xx-U -UU5nURU5nUS- nUR Xx-U UU5nURUU5nUR!U5nSURSS-- S-nSSSU4n["R$"UUSSS 9nUS:aU*OSnUR Xx-S5n['UUR(URSURS 9nUR+Xx-SU5nUR-UUS US 9nUSnUR/U5nUR Xx-U UU-U5nUR1UU5nUR Xx-U UU--U5nUR3UUUUS 9nUSnUR Xx-U UU-U5nUSS2SS2SU24nUR XxU UU5nUR4Vs/sH nUSUPM nn[6R8"USS 9nUR Xx-U UU-S5nUSS2SS2SU24nUR XxU US5n[6R:"UU/SS 9nU(a[=U5[=US5-nOSnU(a5U(a[=US5O [=US5n[=US5U-nOSnU(d[=SUUU455$[?UUUS9$s snf)aM aspect_ratio_ids (`torch.Tensor` of shape `(batch_size, max_num_images)`, *optional*): Aspect ratio ids used to select the appropriate precomputed tile embeddings based on the aspect ratio of each input image. These ids correspond to indices in the model's list of supported aspect ratios, offset by 1. For example, if the model supports aspect ratios [[1, 1], [1, 2], [2, 1]]: - An image with aspect ratio [1, 1] would have ID 1 - An image with aspect ratio [1, 2] would have ID 2 - An image with aspect ratio [2, 1] would have ID 3 The id 0 is reserved for padding (i.e., no image). If an image has aspect ratio [1, 2], that means it was split into 2 tiles horizontally, and its `aspect_ratio_id` would be 2. aspect_ratio_mask (`torch.Tensor` of shape `(batch_size, max_num_images, max_num_tiles)`, *optional*): Mask to avoid performing attention on padding tiles. Mask values selected in `[0, 1]`: - 1 for tiles that are **not masked**, - 0 for tiles that are **masked**. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, MllamaVisionModel >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaVisionModel.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> output = model(**inputs) >>> print(output.last_hidden_state.shape) torch.Size([1, 1, 4, 1025, 7680]) ``` Nr*rtrrErconstant)moder)rArBrCr%T)rJrrr(c3.# UH ocMUv M g7frrrs r>r,MllamaVisionModel.forward..s_$Mq$Mrr) rPrrrr,rGrr r%rvr0flattenrHrrrrFpadrLrBr.rrrrrr2stackr3rr) r^rrbrArrrr8num_concurrent_media num_tilesrheightwidth target_dtype target_device patch_embedsrar:rBr)num_padding_patchesr slice_indexrJoutput global_outputiall_intermediate_hidden_statesintermediate_hidden_statesr global_attnrs r>rfMllamaVisionModel.forward8sd2C1N-TXT_T_TqTq$8$D $++JjJj &1% +00;#++J,MyZ\^ab 99,HXY $++J,MPY,Y[fhkl 11,? q $++J,MyZegjk 66|EUV )),7  !L$6$6r$:Q$>?1Da/0uu\71M .AA.E**4 +22:3TVXY=,((&,,Q/**  $(()JBPST !! )!%/ "  ay **<8 $++  -y+H[:[]` ::rrs &L&#1#J$ %,,5<<  -1/3&*i lli  ,,i !<< i $D> i 'tn i d^i  ellC&7 88 9i i r@rzc The Mllama Text Model which consists of transformer with self and cross attention layers. c"^\rSrSr\rSrS\4U4SjjrSrSr \ SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \\ R\ R4S \ \\\\ R4S\ \ RS\ \S\ \S\ \S\ \S\ \ RS\\S\\\44Sjj5rSrU=r$)MllamaTextModelilanguage_model.modelrPc>[TU]U5 URUlURUl[ R "URS-URUR5UlURUl /n[UR5HHnX0R;aUR[X55 M.UR[X55 MJ [ R"U5Ul[#URUR$S9Ul[)US9UlSUlUR/5 g)Nrr)rPF)rTrU pad_token_idr vocab_sizerrXrV embed_tokenscross_attention_layersrrappendr`rWrrrrnormri rotary_embrr)r^rPrrr_s r>rUMllamaTextModel.__init__s  !.. ++LL):):Q)>@R@RTXTdTde&,&C&C#v778I777 >vQR =fPQ 9 mmF+ %f&8&8f>Q>QR /v>&+# r@cUR$rrrs r>r$MllamaTextModel.get_input_embeddingss   r@cXlgrrr^rs r>set_input_embeddings$MllamaTextModel.set_input_embeddingss!r@ input_idsrJr=r"r#r=rrr$rrrr%rr&c U bU OURRn U bU OURRn U bU OURRn U bU OURRn USLUSL- (a [ S5eUR (a/UR(aU (a[RS5 Sn UcURU5nUnU (a Uc [5nU cEUbUR5OSn[R"UUURS-UR S9n UcU R#S5nUR%X(XU 5nUR'X5nU (aSOSnU (aSOSnSn[)UR*5HunnU (aUU4- nUUR,;nUSL=(d USL=(a URU5S:HnU(a Uc U(aM_UR (a8UR(a'UR/UR0UUUUUUUU U U U5 nOU"U4UUUUUUU U U US . UD6nUSnU (aUU (aS OSnU (dMUUS4- nM UR3U5nU (aUU4- nU (aUOSnU (d[5S UUUU455$[7UUUUS 9$) aY cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile full_text_row_masked_out_mask (`Tuple[torch.Tensor, torch.Tensor]`, *optional*): A tuple containing two tensors that mask out rows in the cross-attention mechanism: - The first tensor has shape `(batch_size, 1, seq_length, 1)` and contains values of 0 or 1. A value of 0 indicates that the corresponding text token's entire row in the cross-attention matrix should be masked out (all image tokens ignored). - The second tensor has the same shape and is used internally to apply the masking during the forward pass of cross-attention layers. This mask is derived from the cross_attention_mask and is used to handle cases where a text token should not attend to any image token. Example: ```python >>> from transformers import AutoProcessor, MllamaTextModel >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaTextModel.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> text = "<|image|>If I had to write a haiku for this one" >>> inputs = processor(text=text, return_tensors="pt") >>> output = model(**inputs) >>> print(output.last_hidden_state.shape) torch.Size([1, 13, 4096]) ``` N:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Frrrr) r"r#rJr=r=r#rr$r%rGrtc3.# UH ocMUv M g7frrrs r>r*MllamaTextModel.forward..st$bq$brrrrr)rPrrr$rr+rrrrrr rr2rr,rvr/rr  enumeraterr rrr rr)r^rrJr=r"r#r=rrr$rrrr%rrrrrGall_hidden_statesall_self_attnsnext_decoder_cacheidx decoder_layeris_cross_attention_layeris_cross_attention_cache_emptyr next_caches r>rfMllamaTextModel.forwards r2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKL] #oomJ#7BD0d!"+DKK"8 C#!m%55! (+d.I.I'I $-<-D.t+X0N0Ns0SWX0X +(,B,JOm**t}} $ A A!**!*(1 #%"' ! !.! !+A)=#.2O!-#2&7'#1(; ! ! *!,M%28I1q%Q"  =#3"55i#9l -0  -!1 1 +4'$ t]J@QSa$btt t&+&+%   r@)r rrrr rr r) NNNNNNNNNNNNN)rhrirjrkrrrrUrrrrr2r1rlr^rrr rr3rrrrfrmrnros@r>rrs $L./(!"151537>B7;UYKO59$(,0/3&*59i E,,-i !.i u//0 i !)):): ; i 'u||4 i (0ellELL6P0Q'Ri "%tE4E4E/F(F"GHi   1 12i D>i $D>i 'tni d^i !!1!12i -.i u-- .!i i r@rc\rSrSrSrg)KwargsForCausalLMirN)rhrirjrkrmrr@r>r&r&s3r@r&zE The Mllama Text Model with a language modeling head on top. c&P^\rSrSr\rSrSrS/rU4Sjr Sr Sr Sr S r S rS r\SS \\R&S \\R(S\\R&S\\R&S\\R&S\\\R(\R(4S\\\\\R24S\\R2S\\R&S\\S\\S\\S\\S\\R&S\\\R(4S\\S\\\44"Sjj5rSr U=r!$) MllamaForCausalLMiTlanguage_modellm_head.weightcz>[TU]UR55 UR5UlURRUl[ R UR5Ul[R"URRURSS9Ul UR5 grN) rTrUr text_configrr _from_configmodelrrrVlm_headrrs r>rUMllamaForCausalLM.__init__s //12!113**55$11$2B2BC yy!1!1!=!=tUZ[  r@c.URR$rr.rrs r>r&MllamaForCausalLM.get_input_embeddingsszz&&&r@c$XRlgrr2rs r>r&MllamaForCausalLM.set_input_embeddingss "' r@cUR$rr/rs r>get_output_embeddings'MllamaForCausalLM.get_output_embeddings ||r@cXlgrr7r^new_embeddingss r>set_output_embeddings'MllamaForCausalLM.set_output_embeddings% r@cXlgrr.)r^decoders r> set_decoderMllamaForCausalLM.set_decoders r@cUR$rrBrs r> get_decoderMllamaForCausalLM.get_decoders zzr@rrJr=r"r#r=rrlabelsr$rrrr%logits_to_keeprr&c ZU bU OURRn U bU OURRn U bU OURRn UR"SUUUUUUUUU U U U US. UD6nUSn[ U[ 5(a [U*S5OUnURUSS2USS245R5nSnU bUR"UXR40UD6nU (dU4USS-nUbU4U-$U$[UUURURURS9$)a cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. full_text_row_masked_out_mask (`Tuple[torch.Tensor, torch.Tensor]`, *optional*): A tuple containing two tensors that mask out rows in the cross-attention mechanism: - The first tensor has shape `(batch_size, 1, seq_length, 1)` and contains values of 0 or 1. A value of 0 indicates that the corresponding text token's entire row in the cross-attention matrix should be masked out (all image tokens ignored). - The second tensor has the same shape and is used internally to apply the masking during the forward pass of cross-attention layers. This mask is derived from the cross_attention_mask and is used to handle cases where a text token should not attend to any image token. Example: ```python >>> from transformers import AutoTokenizer, MllamaForCausalLM >>> model = MllamaForCausalLM.from_pretrained("Llama-3.2-11B-Vision") >>> tokenizer = AutoTokenizer.from_pretrained("Llama-3.2-11B-Vision") >>> prompt = "If I had to write a haiku, it would be:" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=40, do_sample=True, temperature=0.6) >>> result = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] >>> print(result) If I had to write a haiku, it would be: "Snowflakes gently fall" - simple, yet peaceful. I love the idea of snowflakes gently falling, each one ``` N) rr"rJr=r#r=rrr$rrrr%rrlosslogitsrrrr)rPrrrr.r~r0slicer/r} loss_functionrrrrr)r^rrJr=r"r#r=rrrIr$rrrr%rJrrr slice_indicesrNrMrs r>rfMllamaForCausalLM.forwardse@2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W8W~ot4]k mA}a,?@AGGI  %%ffooPPDY,F'+'7D7V# CV C%#33!//))   r@)r/r.r,r)NNNNNNNNNNNNNNr)"rhrirjrkrrrr_tied_weights_keysrUrrr8r>rDrGrrr2r1rlrrr rr^r3r0rr&rrfrmrnros@r>r(r(s $L!(*+'(&151537=A;?UYKO59-1$(,0/3&*5934!i E,,-i !.i u//0 i !))9)9 : i 'u'7'78 i (0ellELL6P0Q'Ri "%tE4E4E/F(F"GHi   1 12i ))*i D>i $D>i 'tni d^i !!1!12i c5<</0!i "*+#i $ u,, -%i i r@r(zr The Mllama model which consists of a vision encoder and a language model without language modeling head. c'^\rSrSrSS0rSrS\4U4SjjrSrSr \ \ SS \ \ RS \ \ RS \ \ R S \ \ R S \ \ R S\ \ R S\ \ R S\ \ RS\ \\ RS\ \ RS\ \S\ \S\ \S\ \S\ \ RS\\S\\\44"Sjj55rSrU=r$) MllamaModeliKrr)FrPc>[TU]U5 URRUlURRUlUR R UlUR RUlURRbURROSUl [RUR 5Ul [RUR5Ul[R "UR RURRSS9UlUR%5 g)Nr*Tr)rTrUr,rrV vision_configrIvision_output_dimrPrrr-rrr)rrmulti_modal_projectorrrs r>rUMllamaModel.__init__Ts   ,,77!--99#11??!'!5!5!G!G8< 8P8P8\DKK44bd-::6;O;OP-::6;M;MN%'YY  2 2    * *& " r@c6URR5$r)r)rrs r>r MllamaModel.get_input_embeddingses""7799r@c:URRU5 gr)r)rrs r>r MllamaModel.set_input_embeddingshs 007r@rrrArbrJr#r"r=rrr$rrrr%rr&c (U bU OURRn U bU OURRn UbUOURRnUSLU SL- (a [ S5eUbU b [ S5eUbUb [ S5eUb`Uc [ S5eUR UUUU U US9nUSnUR U5RSURS UR5nUb-[UUR RURS 9unnOSnUbUbUSS2SS2U4nUSS2SS2U4nUR"SUUUUUUU U U U U S US . UD6n[URUR UR"UR$S 9$)a aspect_ratio_mask (`torch.Tensor` of shape `(batch_size, max_num_images, max_num_tiles)`, *optional*): Mask to avoid performing attention on padding tiles. Mask values selected in `[0, 1]`: - 1 for tiles that are **not masked**, - 0 for tiles that are **masked**. aspect_ratio_ids (`torch.Tensor` of shape `(batch_size, max_num_images)`, *optional*): Aspect ratio ids used to select the appropriate precomputed tile embeddings based on the aspect ratio of each input image. These ids correspond to indices in the model's list of supported aspect ratios, offset by 1. For example, if the model supports aspect ratios [[1, 1], [1, 2], [2, 1]]: - An image with aspect ratio [1, 1] would have ID 1 - An image with aspect ratio [1, 2] would have ID 2 - An image with aspect ratio [2, 1] would have ID 3 The id 0 is reserved for padding (i.e., no image). If an image has aspect ratio [1, 2], that means it was split into 2 tiles horizontally, and its `aspect_ratio_id` would be 2. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. NrzdYou cannot specify both pixel_values and inputs_embeds at the same time, and must specify either onezM`pixel_values` and `cross_attention_states` cannot be provided simultaneouslyzA`aspect_ratio_ids` must be provided if `pixel_values` is provided)rrbrArrrrr*rE)r$r%T) rrJr=r"r#r=rr$rrrrr%rr)rPrrrr+rrYrGr,rVr?rBr%r)rrrrr)r^rrrArbrJr#r"r=rrr$rrrr%rvision_outputsr=rs r>rfMllamaModel.forwardks d2C1N-TXT_T_TqTq$8$D $++JjJj &1%(Jlm m  #' !dee!..)!1"3%9"3' /N&4A%6 "%)%?%?@V%W%_%_*004d6F6F& " +B_$"&"3"3"?"?jjC ? "? -1 )  +0J#71n8L#M ,I!QP^J^,_ )%% )%#9!5*G+'!5/)  "'%77#33!//))   r@)rVr)rIrYrrrXr)NNNNNNNNNNNNNNN)rhrirjrk_checkpoint_conversion_mappingrrrUrrrrrr2r1r^rlrr3rrrrrrfrmrnros@r>rUrUKs '=>N%O" %|":815484837157;9=37=A59$(,0/3&*59!u E,,-u u001u $ELL1 u #5<<0 u !. u 'u||4u !) 6u u//0u "$u'8'8"9:u   1 12u D>u $D>u 'tnu d^u !!1!12!u "-.#u $ u,, -%u u r@rUzS The Mllama model which consists of a vision encoder and a language model. c+^\rSrSrSSSSS.rSrS/rS \4U4S jjrS r S r S r Sr \ S5r\ S5r\\S(S\\R(S\\R*S\\R,S\\R,S\\R,S\\R,S\\R,S\\R(S\\\\\R*4S\\R*S\\R(S\\S\\S\\S\\S \\R(S!\\\R,4S"\\S#\\\44&S$jj55r S)U4S%jjr!U4S&jr"S'r#U=r$$)*MllamaForConditionalGenerationizmodel.language_modelzmodel.vision_modelzmodel.multi_modal_projectorr/)z^language_model.modelz ^vision_modelz^multi_modal_projectorz^language_model.lm_headFr*rPc>[TU]U5 [U5Ul[R "UR RUR RSS9Ul UR5 grN) rTrUrUr.rrr,rVrr/rrs r>rU'MllamaForConditionalGeneration.__init__sS   ( yy!3!3!?!?ASASA^A^ejk  r@c6URR5$r)r.rrs r>r3MllamaForConditionalGeneration.get_input_embeddingsszz..00r@c:URRU5 gr)r.rrs r>r3MllamaForConditionalGeneration.set_input_embeddingss ''.r@cUR$rr7rs r>r84MllamaForConditionalGeneration.get_output_embeddingsr:r@cXlgrr7r<s r>r>4MllamaForConditionalGeneration.set_output_embeddingsr@r@c.URR$r)r.r)rs r>r)-MllamaForConditionalGeneration.language_modelszz(((r@c.URR$r)r.rrs r>r+MllamaForConditionalGeneration.vision_model szz&&&r@rrrArbrJr#r"r=rrrIr$rrrr%rJrr&c 4U bU OURRn UbUOURRnUbUOURRnUR"SUUUUUUUUU U U U USUS.UD6nUSn[ U[ 5(a [U*S5OUnURUSS2USS245nSnU b2UR"UXRRR40UD6n[UUURURURS9$)a aspect_ratio_mask (`torch.Tensor` of shape `(batch_size, max_num_images, max_num_tiles)`, *optional*): Mask to avoid performing attention on padding tiles. Mask values selected in `[0, 1]`: - 1 for tiles that are **not masked**, - 0 for tiles that are **masked**. aspect_ratio_ids (`torch.Tensor` of shape `(batch_size, max_num_images)`, *optional*): Aspect ratio ids used to select the appropriate precomputed tile embeddings based on the aspect ratio of each input image. These ids correspond to indices in the model's list of supported aspect ratios, offset by 1. For example, if the model supports aspect ratios [[1, 1], [1, 2], [2, 1]]: - An image with aspect ratio [1, 1] would have ID 1 - An image with aspect ratio [1, 2] would have ID 2 - An image with aspect ratio [2, 1] would have ID 3 The id 0 is reserved for padding (i.e., no image). If an image has aspect ratio [1, 2], that means it was split into 2 tiles horizontally, and its `aspect_ratio_id` would be 2. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, MllamaForConditionalGeneration >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaForConditionalGeneration.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> prompt = "<|image|>If I had to write a haiku for this one" >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=prompt, images=image, return_tensors="pt") >>> # Generate >>> output = model.generate(**inputs, max_new_tokens=15) >>> prompt_len = inputs.input_ids.shape[-1] >>> generated_ids = output[:, prompt_len:] >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False) >>> print(generated_text) [', it would be:.\\nA stop sign in Chinatown.\\n'] ``` NT)rrrArbr#r"rJr=rrr$rrrr%rrLr)rPrrrr.r~r0rOr/rPr,rrrrr)r^rrrArbrJr#r"r=rrrIr$rrrr%rJrrrrQrNrMs r>rf&MllamaForConditionalGeneration.forwardsAf2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W8W~ot4]k mA}a,?@A  %%ffkk6M6M6X6Xc\bcD%#33!//))   r@c  r>[TU]"U4U U UUUUUUUU U S. U D6nU SS:waSUS'SUS'SUS'U$)N) rr$rr=rJrrbrAr#r%rJrrrbrA)rTprepare_inputs_for_generation)r^rrrJr=rrbrAr#rr$r%rJr model_inputsr_s r>rvURSS5n[TU]"SUUUS.UD6nUb#[R"XUSS2SS2S4/SS9US'U$)Nr#)r model_kwargsis_encoder_decoderr*.rr(r)getrT#_update_model_kwargs_for_generationr2r3)r^rrzr{rcross_attention_mask_prevr_s r>r}BMllamaForConditionalGeneration._update_model_kwargs_for_generationsx$0$4$45KT$R!wB %1    % 03899*ack,RSYZ4L/ 0r@)r/r.)NNNNNNNNNNNNNNNNr) NNNNNNNNNFNN)%rhrirjrkrbrrSrrUrrr8r>propertyr)rrrrr2r1r^rlrr rr3r0rr&rrrfrvr}rmrnros@r>rdrds"8-"?#, &" !&*+| 1/&))''15484837157;9=37KO59-1$(,0/3&*5934%y E,,-y u001y $ELL1 y #5<<0 y !. y 'u||4y !) 6y u//0y "%tE4E4E/F(F"GHy   1 12y ))*y D>y $D>y 'tny d^!y "!!1!12#y $c5<</0%y &*+'y ( u,, -)y y z!)Vr@rd)rdr(rrrrU)rrS)Wrrtypingrrrrrr2torch.nn.functionalrrrtorch.utils.checkpoint activationsr cache_utilsr r generationr modeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_outputsrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrrconfiguration_mllamarrr !torch.nn.attention.flex_attentionr!integrations.flex_attentionr" get_loggerrhrrlr0rr?r%rLModulerNrqrrr}rrrrrrr7rArCrLrWr`rirrrr&r(rUrd__all__rr@r>rsq 99 !.)>B``KF&hhTT !!;J   H %?,,?? ? 5<< %& ?8|| ;;   \\ 6BII.""L bii  UU\\ U# U%,, U(% II% <<% % << % U\\* %  %%4>)BII>)B.ryy.bT "))T pJ J(_9ryy_9F(6L9biiL9`BII$RbiiRj;uxx;| K -K  K \ R 'R  R j [%:O[  [| r@