fThsSSKJr SSKJrJrJrJrJr SSKrSSK J s J r SSKJ r SSK Jr SSKJrJrJrJr SSKJr SS KJr SS KJr SS KJr SS KJrJrJrJ r J!r!J"r" SS K#J$r$J%r% SSK&J'r'J(r( SSK)J*r* SSK+J,r,J-r-J.r.J/r/J0r0 SSK1J2r2 \/"5(a SSK3J4r4 SSK5J6r6 \0Rn"\85r9"SS\ Rt5r;"SS\ Rt5r<\"S5"SS\ Rt55r=Sr>SDSjr?S\RS\AS \R4S!jrBSES"\ RtS#\RS$\RS%\RS&\\RS'\CS(\C4S)jjrD"S*S+\ Rt5rE"S,S-\ Rt5rF"S.S/\ Rt5rG\-"S0S1\(55rH\-"S2S3\H55rI"S4S5\\,5rJSFS6\\R\\RS4S7\\AS&\\RS \\R\A44S8jjrK\-"S9S:\H\55rL\-"S;S<9"S=S>\H55rM\-"S?S@\H55rN\-"SASB\H55rO/SCQrPg)G)partial)CallableListOptionalTupleUnionN)nn)ACT2FN)Cache DynamicCacheSlidingWindowCache StaticCache)GenerationMixin)use_kernel_forward_from_hub)AttentionMaskConverter)FlashAttentionKwargs)BaseModelOutputWithPastMoeCausalLMOutputWithPastMoeModelOutputWithPastQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging) MixtralConfig) BlockMask)make_flex_block_causal_maskc6^\rSrSrS\4U4SjjrSrSrU=r$)MixtralBlockSparseTop2MLP@configc>[TU]5 URUlURUl[ R"UR URSS9Ul[ R"URUR SS9Ul [ R"UR URSS9Ul [URUl gNFbias)super__init__intermediate_sizeffn_dim hidden_size hidden_dimr Linearw1w2w3r hidden_actact_fnselfr+ __class__s d/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/mixtral/modeling_mixtral.pyr1"MixtralBlockSparseTop2MLP.__init__As //  ,,))DOOT\\F))DLL$//F))DOOT\\FV../ cURURU55URU5-nURU5nU$N)r;r7r9r8)r= hidden_statescurrent_hidden_statess r?forward!MixtralBlockSparseTop2MLP.forwardLs> $ DGGM,B CdggmF\ \ $(= >$$rA)r;r3r5r7r8r9) __name__ __module__ __qualname____firstlineno__r%r1rF__static_attributes__ __classcell__r>s@r?r)r)@s 0} 0%%rAr)cf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)MixtralSparseMoeBlockRa This implementation is strictly equivalent to standard MoE with full capacity (no dropped tokens). It's faster since it formulates MoE operations in terms of block-sparse operations to accommodate imbalanced assignments of tokens to experts, whereas standard MoE either (1) drop tokens at the cost of reduced performance or (2) set capacity factor to number of experts and thus waste computation and memory on padding. c>[TU]5 URUlURUlUR UlURUl [R"URURSS9Ul [R"[UR5Vs/sHn[U5PM sn5UlUR"Ulgs snfr-)r0r1r4r5r2r3num_local_experts num_expertsnum_experts_per_toktop_kr r6gate ModuleListranger)expertsrouter_jitter_noise jitter_noise)r=r+_r>s r?r1MixtralSparseMoeBlock.__init__^s  ,,// !33// IIdoot/?/?eL }}QVW[WgWgQh%iQhA&?&GQh%ij #66&js*CrDreturncURup#nUR(aSURS:aCU[R"U5R SUR- SUR-5-nUR SU5nURU5n[R"US[RS9n[R"X`RSS9upgXfRSSS9-nURUR5n[R "X#-U4URUR"S 9n[R$R&R)XpR*S 9R-S SS5n U RS S9S:R/SS 9SR15n U Hn UR2U n [R4"X5upUSU4R7SU5nU "U5XnU S4-nUR9SUURUR55 M UR7X#U5nX4$)r?r$dimdtypereT)rekeepdim)rfdevice) num_classes)rc)as_tupleN)shapetrainingr\torch empty_likeuniform_viewrWFsoftmaxfloattopkrVsumtorfzerosrir functionalone_hotrTpermutenonzerotolistrZwherereshape index_add_)r=rD batch_sizesequence_lengthr5 router_logitsrouting_weightsselected_expertsfinal_hidden_states expert_mask expert_hitted expert_idx expert_layeridxtop_x current_staterEs r?rFMixtralSparseMoeBlock.forwardms2?2E2E/ Z ==T..2 U--m<EEcDL]L]F]_beievev_vw wM%**2z:  -0 ))Mq L,1JJ XZ,[)..2t.DD),,]-@-@A#kk  ): 6m>Q>QZgZnZn  hh))112BP`P`1aiijkmnpqr $X6:CCTCRSTU\\^ 'J<< 3L[%<=JC*$+6>>r:NM$0$?/Y\^bRbBc$c ! * *1e5J5M5MmNaNa5b c(299*Wab"11rA)rZr3rWr5r\rTrV) rHrIrJrK__doc__r1rpTensorrFrLrMrNs@r?rPrPRs-  7%2U\\%2ell%2%2rArPRMSNormc8^\rSrSrSU4SjjrSrSrSrU=r$)MixtralRMSNormc>[TU]5 [R"[R "U55UlX lg)z- MixtralRMSNorm is equivalent to T5LayerNorm N)r0r1r Parameterrponesweightvariance_epsilon)r=r4epsr>s r?r1MixtralRMSNorm.__init__s/ ll5::k#:; #rAcURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)NrkrcT)rh) rfryrpfloat32powmeanrsqrtrr)r=rD input_dtypevariances r?rFMixtralRMSNorm.forwardsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::rAc^[URR5SUR3$)Nz, eps=)tuplerrnrr=s r? extra_reprMixtralRMSNorm.extra_reprs* ))*+6$2G2G1HIIrA)rr)gư>) rHrIrJrKr1rFrrLrMrNs@r?rrs$;JJrArcUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..Nrcrkrg)rnrpcat)xx1x2s r? rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''rAcURU5nURU5nX-[U5U--nX-[U5U--nXg4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkcossin position_ids unsqueeze_dimq_embedk_embeds r?apply_rotary_pos_embrsS( -- &C -- &Cw;q>C/0Gw;q>C/0G  rArDn_repr_cURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r$N)rnexpandr)rDrbatchnum_key_value_headsslenhead_dims r? repeat_kvrs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTrAmodulequerykeyvalueattention_maskscalingdropoutc @[X R5n[X0R5n [R"XR SS55U-n Ub"USS2SS2SS2SUR S24n X-n [ RRU S[RS9RUR5n [ RRXURS9n [R"X5n U R SS5R5n X4$)Nrkr rlrcrd)pror$)rnum_key_value_groupsrpmatmul transposernr r{rurryrfrro contiguous) rrrrrrrkwargs key_states value_states attn_weights causal_mask attn_outputs r?eager_attention_forwardrs3 ; ;[TU]5 XlX l[ USS5=(d UR UR -UlUR UR-Ul URS-Ul URUl SUl [R"UR UR UR-SS9Ul[R"UR URUR-SS9Ul[R"UR URUR-SS9Ul[R"UR UR-UR SS9Ulg)NrgTFr.)r0r1r+rgetattrr4num_attention_headsrrrrattention_dropout is_causalr r6q_projk_projv_projo_projr=r+rr>s r?r1MixtralAttention.__init__s.  " D9mV=O=OSYSmSm=m $*$>$>&B\B\$\!}}d* !'!9!9ii 2 2F4N4NQUQ^Q^4^ejk ii 2 2F4N4NQUQ^Q^4^ejk ii 2 2F4N4NQUQ^Q^4^ejk ii : :T]] JFL^L^ejk rArDposition_embeddingsrpast_key_valuecache_positionrr_c URSSn/UQSPURP7nURU5RU5R SS5n UR U5RU5R SS5n UR U5RU5R SS5n Uup[XX5upUb$XUS.nURXURU5up[nURRS:wadURRS:Xa-URSS5(a[RS 5 O[ URRnU"UU U U U4UR"(dS O UR$UR&[)URS S5S .UD6unnUR*"/UQSP76R-5nUR/U5nUU4$) Nrcr$rk)rrreagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.sliding_window)rrr)rnrrrsrrrrupdaterrr+_attn_implementationgetlogger warning_oncerrorrrrrr)r=rDrrrrr input_shape hidden_shape query_statesrrrr cache_kwargsattention_interfacerrs r?rFMixtralAttention.forwards$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &#7 RU#[  %#&nUL'5'<'d>d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL"4;;0@$G %  % ! \"));;;;FFH kk+. L((rA) rr+rrrrrrrrr)NN)rHrIrJrKrr%intr1rprrrr LongTensorrrrFrLrMrNs@r?rrsG l} l l&+/59 0)||0)#5<<#=>0)!. 0) ! 0) !!1!12 0)-.0) u||Xell3XeELL>Q5RR S0)0)rArc^\rSrSrS\S\4U4SjjrSS\RS\ \RS\ \RS\ \ \RS \ \ S \ \ S \ \ S \ \RS \ \ \R\R4S\ \S\ \R\ \ \R\R444SjjrSrU=r$)MixtralDecoderLayeri6r+rc>[TU]5 URUl[X5Ul[ U5Ul[URURS9Ul [URURS9Ul g)Nr) r0r1r4r self_attnrPblock_sparse_moer rms_norm_epsinput_layernormpost_attention_layernormrs r?r1MixtralDecoderLayer.__init__7sk !--)&< 5f =-f.@.@fFYFYZ(6v7I7IvObOb(c%rArDrrrroutput_router_logits use_cacherrrr_c Un URU5nUR"SUU UUUUUUS.U D6upX-nUn URU5nURU5upX-nU4nU(aX4- nU(aX4- nU$)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )rDrrrrrrr)rrrr)r=rDrrrrrrrrrresidualself_attn_weightsroutputss r?rFMixtralDecoderLayer.forwardAsF!,,]; ,0>> , ' 3)%)/) ,  , ( !0 !55mD '+'<'<]'K$  0  "  + +G  ' 'GrA)rr4rrr)NNNFFFNN)rHrIrJrKr%rr1rprrrrboolrr FloatTensorrFrLrMrNs@r?rr6s;d}dd26378<,1/4$)59KOC||C!.Cu//0 C !u||!45 C $D> C'tnCD>C!!1!12C&eELL%,,,F&GHC-.C u  (51B1BEDUDU1U+V"WW XCCrArcl^\rSrSrSS\4U4Sjjjr\R"5\S55r Sr U=r $)MixtralRotaryEmbeddingir+c>[TU]5 [US5(aHURb;URR SURR S55UlOSUlUR UlUR UlXl [UR Ul URURU5uo0l URSUSS9 URUlg)N rope_scaling rope_typetypedefaultinv_freqF) persistent)r0r1hasattrrrrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr+r rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)r=r+rirr>s r?r1MixtralRotaryEmbedding.__init__s  6> * *v/B/B/N#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk ^^ U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs $BF  F.)rr+rr!rrrrC) rHrIrJrKr%r1rpno_gradrrFrLrMrNs@r?rrs6/}//" ]]_ < .  / / MM   $ $S )0rAr N)rHrIrJrKr% config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_flex_attn_supports_cache_class_supports_quantized_cache_supports_static_cache_supports_attention_backendr>rLr rAr?r2r2sS L&*#./#4"5!N  $""& *rAr2c<^\rSrSrS\4U4SjjrSrSr\\ SS\ \ RS\ \ RS\ \ RS \ \\ RS \ \ RS \ \S \ \S \ \S\ \S\ \ RS\\S\4Sjj55rSS\\ RS4S\ RS\ RS \S \4 Sjjr\S\ RS\S\S\ R4S\ RS\S\S \4Sj5rSrU=r$) MixtralModelir+c >[TU]U5 URUlURUl[ R "URURUR5Ul[ R"[UR5Vs/sHn[X5PM sn5Ul [URURS9Ul[#US9UlSUlUR)5 gs snf)Nr)r+F)r0r1 pad_token_idr< vocab_sizer r;r4 embed_tokensrXrYnum_hidden_layersrlayersrrnormr rotary_embgradient_checkpointing post_initrs r?r1MixtralModel.__init__s  !.. ++LL):):FL] & #oomJ#7BD0d"6BD![[M#!m%55!**t}} $ A AM22H6GH! #%("' ! !.! !#.!-#2&7)='#1(; !( ! *!,M =#3"55##!mB&7%99!K)N -0  -!1 1 %+++%+   rAr& input_tensorc URRS:Xa]UbNUbKUSS2S4R5R5UR 5S:gnU(a [ S5eUbSU;aU$gURRS:Xa,[ U[R5(a [U5nU$UbUR5OSn[ U[5n[ U[5n URRS:XaQU(dJU (dCU(d<[R"UUUURRUR S9(agUR"n [R$"U 5R&n UR(S n U (dU(aUR+5n O5[ U[R5(aUR(SOX|-S -n UR-UU U U UUR(SURUS 9nURRS:Xa:Ub7UR.R0S ;aU(d[R2"X5nU$) Nflash_attention_2rcrzYou are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mixtral. Make sure to call `tokenizer.padding_side = 'left'` before tokenizing the input. rflex_attentionr)rbpast_key_values_lengthr is_trainingr$)r target_lengthrfrrr+r4)cudaxpunpu)r+rrxitemsizerir(rprr'rjrrr_ignore_causal_mask_sdparrorffinfominrnget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrir_unmask_unattended)r=rrvrr4ris_padding_rightrousing_static_cacheusing_sliding_window_cacherf min_dtyperr|rs r?rl MixtralModel._update_causal_maskRs; ;; + +/B B)o.I#1!R%#8#<#<#>#C#C#EIZIZI\]^I_#_ #$a )c^.C%% ;; + +/? ?.%,,77!<^!L! ! @O?Z?99;`a'E%/AS%T" KK , , 6'+E%%>>*'7#{{99 MM ""KK&** &,,Q/ %);+??AM nell;;$$R(%7!; PP +')#))!,;;+Q   KK , , 6*%%**.DD% 1CCK[KrArr|rfrcUbUR5S:XaUnU$[R"U5Rn [R"X4XUR S9n[R "X$R S9URSS5:n UR5n [U SS5(avU Rbi[U[5(aX:aO[R "X$R S9URSS5U R- :*n U RU 5 X-nUSSSS2SS24RUSSS5nUbUR5nUR SU:a USS2SU24nUR Sn USS2SS2SS2SU 24USS2SSSS24R#UR 5-nUS :HnUSS2SS2SS2SU 24R%X5USS2SS2SS2SU 24'U$) aS Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. config (`MixtralConfig`): The model's configuration class past_key_values (`Cache`): The cache class that is being used currently to generate N) fill_valuerfrirfrcr$use_sliding_windowTr)rerprrfullrirkrget_text_configrrr(r bitwise_or_rclonernry masked_fill)rrr|rfrrr+r4rrdiagonal_attend_mask text_configsliding_attend_mask mask_length padding_masks r?rBMixtralModel._prepare_4d_causal_attention_mask_with_cache_positionsB  %.*<*<*>!*C(K@= E*..I** 0Y\j\q\qK$)<< F[F[#\_m_u_uA`$ !002K{$8$??KD^D^Dj"/3EFF/Ji*/,,}MbMb*c&..r15 8R8RR+')445HI  /K%dD!Q&67>>z1bRTUK))//1 !''+m;%3A~ ~4E%FN,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12rA)rQrVrSrTr<rUrP) NNNNNNNNNN)F)rHrIrJrKr%r1r[r_r!r rrprrrrrrrrrFrr rl staticmethodrrfrrLrMrNs@r?rMrMs} !"151537=A59$(,0/3/359p E,,-p !.p u//0 p "$u'8'8"9: p   1 12 p D>p $D>p 'tnp 'tnp !!1!12p $$89p  p p p#( TellK78TllT T  T  TlB BBB{{ B  B  BBBBrArMc\rSrSrSrg)KwargsForCausalLMir N)rHrIrJrKrLr rAr?rrs3rAr gate_logitsrTcUb[U[5(dg[U[5(aCUSRn[R"UVs/sHoUR U5PM snSS9n[R RRWSS9n[R"XrSS9up[R RRX5n Uc:[R"U R5SS9n [R"USS9n GOURupURSX--nUSSS2SS2SS4RXXU45RSX!5R W5n[R "U R5U-SS9[R "USS9- n USSS2SS2S4RXX45RSU5R U5n[R "UU-SS9[R "USS9- n [R "XR#S5-5nUU-$s snf)ap Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://arxiv.org/abs/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrrgrc)r(rrirprryr r{rurwr|rrvrnrrrxr)rrTrVrcompute_device layer_gateconcatenated_gate_logitsrr]rrtokens_per_expertrouter_prob_per_expertrrrRexpert_attention_mask router_per_expert_attention_mask overall_losss r?load_balancing_loss_funcrs+:*[%"@"@+u%%$Q..#(99^i-j^iPZmmN.K^i-jpq#r hh))112JPR1SO**_DA((%%--.>LK!JJ{'8'8':B"'O!C&4&:&:# 4::1=*B^_ 4AtT1 2 V&OKX Y WR , R  "IIk&7&7&9[TU]U5 [U5UlURUl[ R "URURSS9UlURUl URUl URUl UR5 gr-)r0r1rMr3rPr r6r4rrouter_aux_loss_coefrSrTrUrWr<s r?r1MixtralForCausalLM.__init__Is  !&)  ++yy!3!3V5F5FUS $*$?$?!!33#)#=#=  rAc.URR$rCr3rQrs r?r['MixtralForCausalLM.get_input_embeddingsUzz&&&rAc$XRlgrCrr^s r?r_'MixtralForCausalLM.set_input_embeddingsX "' rAcUR$rCrrs r?get_output_embeddings(MixtralForCausalLM.get_output_embeddings[s ||rAcXlgrCr)r=new_embeddingss r?set_output_embeddings(MixtralForCausalLM.set_output_embeddings^s% rAcXlgrCr3)r=decoders r? set_decoderMixtralForCausalLM.set_decoderas rAcUR$rCrrs r? get_decoderMixtralForCausalLM.get_decoderds zzrArarrr4rblabelsrrrcrrlogits_to_keeprr_c  UbUOURRnU bU OURRn U bU OURRn UR"SUUUUUUUU U U S. U D6nUR n[ U [5(a [U *S5OU nURUSS2USS245nSnUbUR"UX`R40U D6nSnU (aZ[URURURU5nUb+UUR UR#UR$5-- n['UUUUR(UR*UR,URS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MixtralForCausalLM >>> model = MixtralForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-v0.1") >>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-v0.1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rarrr4rbrrrcrr)lossaux_lossrr4rDrhrr )r+rrrcr3rgr(rslicer loss_functionrPrrrTrUrryrirr4rDrh)r=rarrr4rbrrrrcrrrrr rD slice_indicesrrrs r?rFMixtralForCausalLM.forwardgsR2C1N-TXT_T_TqTq$8$D $++JjJj  %9$D $++JjJj  +/** + )%+'/!5!5) +  +  11 8B>SV8W8W~ot4]k mA}a,?@A  %%ffooPPD /%%  (( H !11HKK 4LLL(#33!//))!//  rA)rr3rTrUrrP) NNNNNNNNNNNr)rHrIrJrK_tied_weights_keys_tp_plan_pp_planr1r[r_rrrrr!r rrprrrrrrrrrrrFrLrMrNs@r?rrCs*+=)H_-z:;H '(&151537=A59-1$(,0/3/35934[ E,,-[ !.[ u//0 [ "$u'8'8"9: [   1 12 [ ))*[ D>[ $D>[ 'tn[ 'tn[ !!1!12[ c5<</0[ *+[  #[ [ rAra The Mixtral Model transformer with a sequence classification head on top (linear layer). [`MixtralForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introc*^\rSrSrU4SjrSrSr\\SS\ \ RS\ \ RS\ \ RS\ \ S \ \ RS \ \ RS \ \S \ \S \ \S\4Sjj55rSrU=r$) MixtralForSequenceClassificationic>[TU]U5 URUl[U5Ul[ R "URURSS9UlUR5 gr-) r0r1 num_labelsrMr3r r6r4scorerWr<s r?r1)MixtralForSequenceClassification.__init__sS   ++!&) YYv114??O  rAc.URR$rCrrs r?r[5MixtralForSequenceClassification.get_input_embeddingsrrAc$XRlgrCrr^s r?r_5MixtralForSequenceClassification.set_input_embeddingsrrArarrr4rbrrrrcr_c URUUUUUUUU S9n U Rn URU 5n UbURSn OURSn URR cU S:wa [ S5eURR cSnOUbXRR :gRU R[R5n[R"URSU R[RS9nUU-RS5nO.Sn[RURR S35 U [R"XRS 9U4nSnUbUR#XUURS 9n[%UUU R&U R(U R*S 9$) e labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rrr4rbrrrcNrr$z=Cannot handle batch sizes > 1 if no padding token is defined.rc)rirfz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rf)rr pooled_logitsr+)rrr4rDrh)r3rgrrnr+rOriryrirpint32rkargmaxrrr>rHrrr4rDrh)r=rarrr4rbrrrrctransformer_outputsrDrrlast_non_pad_token non_pad_mask token_indicesrrs r?rF(MixtralForSequenceClassification.forwards*8!F!Fr!J !#    >>**+,ZZ  u||J}}MOaab   %%VR_hlhshs%tD/ /??-;;*55   rA)r3rr NNNNNNNNN)rHrIrJrKr1r[r_r!r rrprrr rrrrFrLrMrNs@r?rrs'(151537+/59-1$(,0/3A E,,-A !.A u//0 A "% A   1 12 A ))*A D>A $D>A 'tnA  *A A rArc*^\rSrSrU4SjrSrSr\\SS\ \ RS\ \ RS\ \ RS\ \ S \ \ RS \ \ RS \ \S \ \S \ \S\4Sjj55rSrU=r$)MixtralForTokenClassificationi+c>[TU]U5 URUl[U5Ul[ USS5b UR nO[ USS5b URnOSn[R"U5Ul [R"URUR5Ul UR5 g)Nclassifier_dropouthidden_dropoutg?)r0r1rrMr3rrrr Dropoutrr6r4rrW)r=r+rr>s r?r1&MixtralForTokenClassification.__init__-s   ++!&) 6/ 6 B!'!:!:  V-t 4 @!'!6!6 !$ zz"45 YYv1163D3DE  rAc.URR$rCrrs r?r[2MixtralForTokenClassification.get_input_embeddings=rrAc$XRlgrCrr^s r?r_2MixtralForTokenClassification.set_input_embeddings@rrArarrr4rbrrrrcr_c  URUUUUUUUU S9n U Rn URU 5n URU 5n Sn UbUR XUR 5n [ U U U RU RS9$)rrN)rrrDrh) r3rgrrrr+rrDrh)r=rarrr4rbrrrrcr sequence_outputrrs r?rF%MixtralForTokenClassification.forwardCs*,0:: )%+'/!5,6 , "33,,7O,  %%fdkkBD$!//))   rA)rr3rrr)rHrIrJrKr1r[r_r!r rrprrr rrrrFrLrMrNs@r?rr+s '(151537+/59-1$(,0/3* E,,-* !.* u//0 * "% *   1 12 * ))** D>* $D>* 'tn*  * * rArcf^\rSrSrSrU4SjrSrSr\\ SS\ \ RS\ \ RS\ \ RS \ \\\\ R"4S \ \ R"S \ \ RS \ \ RS \ \S\ \S\4Sjj55rSrU=r$)MixtralForQuestionAnsweringirr3c>[TU]U5 [R"URS5Ul[ U5UlUR5 g)Nrk) r0r1r r6r4 qa_outputsrMr3rWr<s r?r1$MixtralForQuestionAnswering.__init__vs@  ))F$6$6:!&)  rAc.URR$rCrrs r?r[0MixtralForQuestionAnswering.get_input_embeddings~rrAc$XRlgrCrr^s r?r_0MixtralForQuestionAnswering.set_input_embeddingsrrArarrr4rbstart_positions end_positionsrrcr_c URUUUUUUU S9n U Rn URU 5n U RSSS9upUR S5R 5nUR S5R 5nSnUbUbUR "XXg40U D6n[UUUU RU RS9$)a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. )rrr4rbrrcr$rcrgN)r start_logits end_logitsrDrh) r3rgrsplitsqueezerrrrDrh)r=rarrr4rbrrrrcrr rrrrrs r?rF#MixtralForQuestionAnswering.forwards4,0:: )%+'/!5,6, "331#)<<r<#: #++B/::< ''+668   &=+D%%libhiD+%!!//))   rA)r3rr)rHrIrJrKrAr1r[r_r!r rrprrrr rrrrrFrLrMrNs@r?rrrs'(151537KO596:48,0/33 E,,-3 !.3 u//0 3 "%tE4E4E/F(F"GH 3   1 12 3 "%"2"233   0 013 $D>3 'tn3  &3 3 rAr)rrrMr2rr)Nr$)r)NrkN)Q functoolsrtypingrrrrrrptorch.nn.functionalr r{rt activationsr cache_utilsr r rr generationr integrationsrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_outputsrrrrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrr r!r"r#configuration_mixtralr%!torch.nn.attention.flex_attentionr&integrations.flex_attentionr' get_loggerrHrModuler)rPrrrrrrrvrrrrr2rMrrrrrr__all__r rAr?rs699 !OO)7>BLF&hh0 !!;J   H %% %$@2BII@2FY'JRYYJ(J((6 UU\\ U# U%,, U&% II% <<% % << % U\\* %  %%4A)ryyA)HN"))Nb "& -1 O&u||U5<<%8$>?O&#O&U\\* O&  5<<  O&d@ /@ @ F  S '=S  S lC $:C C LF "8F F R rA