fThj|SrSSKrSSKJrJrJrJrJrJrJ r SSK r SSK r SSK J r SSK JrJrJr SSKJr SSKJrJr SS KJr SS KJr SS KJr SS KJrJrJr SS K J!r!J"r" SSK#J$r$ SSK%J&r& SSK'J(r(J)r) SSK*J+r+J,r, SSK-J.r. \,"5(aSSK/J0r0J1r1 SSK2J3r3 OSur3r1r0\+"5(a SSK4J5r5J6r6 OSur6r5\7"\3\1\5\6\045r8\)Rr"\:5r;"SS\ Rx5r=\&R|"\=5 S\ R~S\@S\ R~4SjrA"SS \5rBS@S!\ RxS"\ R~S#\ R~S$\ R~S%\\ R~S&\CS'\C4S(jjrD"S)S*\ Rx5rE"S+S,\ Rx5rF"S-S.\ Rx5rG"S/S0\ Rx5rH"S1S2\ Rx5rI"S3S4\ Rx5rJ\("S5S6\"55rK\("S7S8\K55rL"S9S:\K\5rM\("S;S<9"S=S>\K55rN/S?QrOg)AzPyTorch Zamba model.N)AnyCallableDictListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)ALL_LAYERNORM_LAYERS)auto_docstringlogging)is_causal_conv1d_availableis_mamba_ssm_available) ZambaConfig)mamba_inner_fnselective_scan_fn)selective_state_update)NNN)causal_conv1d_fncausal_conv1d_updateNNc8^\rSrSrSU4SjjrSrSrSrU=r$) ZambaRMSNorm@c>[TU]5 [R"[R "U55UlX lg)z+ ZambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s `/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/zamba/modeling_zamba.pyr-ZambaRMSNorm.__init__As/ ll5::k#:; #cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)NT)keepdim) dtypetor/float32powmeanrsqrtr2r1)r3 hidden_states input_dtypevariances r7forwardZambaRMSNorm.forwardIsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r9c^[URR5SUR3$)Nz, eps=)tupler1shaper2r3s r7 extra_reprZambaRMSNorm.extra_reprPs* ))*+6$2G2G1HIIr9)r2r1)gư>) __name__ __module__ __qualname____firstlineno__r-rGrM__static_attributes__ __classcell__r6s@r7r)r)@s$;JJr9r)rDn_repreturncURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r N)rKexpandreshape)rDrVbatchnum_key_value_headsslenhead_dims r7 repeat_kvr_Xs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr9c \rSrSrSr\R S4SjrSS\RS\RS\ S\ \ \ \ 4S \\R\R44 S jjrS \R 4S jrSS\ \ S \ 4S jjrS \\\R\\R44Sjr\SS\ \\\R*S S4Sjj5rSrg)ZambaHybridDynamicCacheda| A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. Nc X0lURUlSUlURUR-UlUR UlURUl URUl /Ul /Ul /Ul 0Ul0Ul0Ul[#UR$5HnU=R[&R("X R URXCS9/- sl UURUR UR-UR4nU=R[&R("XdUS9/- sl URUS:XdMURR+U5 M [#UR$5Vs/sHn[&R,"//U-US9PM snUl[#UR$5Vs/sHn[&R,"//U-US9PM snUlgs snfs snf)NFdevicer>hybridre)r>layers_block_typehas_previous_state mamba_expandr4intermediate_size mamba_d_statessm_state_size mamba_d_convconv_kernel_size n_mamba_heads conv_states ssm_statestransformer_layers_modules _parameters_buffersrangenum_hidden_layersr/zerosappendtensor key_cache value_cache)r3config batch_sizer>rei cache_shape_s r7r- ZambaHybridDynamicCache.__init__rs !'!9!9"'!'!4!4v7I7I!I$22 & 3 3#11"$  v//0A    J(>(>@U@U^dr!  ""&&$*<*<<## K OO Ke TU UO%%a(H4''..q11SXX^XpXpRqrRqQ%,,tj'8HRqrTYZ`ZrZrTstTsqELL" ):6JTststs #H#H key_states value_states layer_idx cache_kwargsrWc|URURSS:XaXRU'X RU'Ob[R"URUU/SS9URU'[R"URUU/SS9URU'URUURU4$)Nr<rr;dim)r|rKr}r/cat)r3rrrrs r7updateZambaHybridDynamicCache.updates >>) $ * *2 .! 3(2NN9 %*6  Y '(- 4>>)3Lj2Y_`(aDNN9 %*/))T5E5Ei5PR^4_ef*gD  Y '~~i($*:*:9*EEEr9beam_idxc[[UR55GHTnURURnURUR SUR U55URU'UR URnUR UR SUR U55UR U'URURnURUR SUR U55URU'URURnURUR SUR U55URU'GMW g)zDReorders the cache for beam search, given the selected beam indices.rN) rwlenr|re index_selectr?r}rqrr)r3rrres r7 reorder_cache%ZambaHybridDynamicCache.reorder_caches=s4>>23I^^I.55F(,y(A(N(NqRZR]R]^dRe(fDNN9 %%%i077F*.*:*:9*E*R*RSTV^VaVabhVi*jD  Y '%%i077F*.*:*:9*E*R*RSTV^VaVabhVi*jD  Y '__Y/66F)-)C)P)PQRT\T_T_`fTg)hDOOI &4r9cXR;aURSOUn[UR5U::agURURS$)zYReturns the sequence length of the cached states. A layer index can be optionally passed.r)rsrr|rK)r3rs r7get_seq_length&ZambaHybridDynamicCache.get_seq_lengthsP3rirkr|rhrprmrrrsr}N)r)rOrPrQrR__doc__r/float16r-Tensorintrrstrrrr LongTensorrrr classmethod FloatTensorrrSr9r7rarads 27tuJ26 FLLFllF F tCH~. F u||U\\) * F$ ie&6&6 i3 3c3fuU\\':E%,,)ptrainingr )r_num_key_value_groupsr/matmul transposerKr functionalsoftmaxr@r?r>rr contiguous) rrrrrrrkwargsrr attn_weights causal_mask attn_outputs r7eager_attention_forwardrs3 ; ;[TU]5 XlX lURUlUR UlURUR-Ul URUl UR S- S-Ul SUl URUl [R"URURUR -SS9Ul[R"URURUR -SS9Ul[R"URURUR -SS9Ul[R"URUR -UR&SS9Ulg)Nr;TFbias)r,r-r~rattention_hidden_sizeattention_head_dimr^num_attention_headsr\rmax_position_embeddingsr is_causalattention_dropoutr Linearq_projk_projv_projr4o_projr3r~rr6s r7r-ZambaAttention.__init__s5  "%+%A%A"11 $*$>$>&B\B\$\!'-'E'E$ )d2 !'!9!9ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii : :T]] JFL^L^ejk r9rDrpast_key_valuerrWc zURSSn/UQSPURP7nURU5RU5R SS5nUR U5RU5R SS5n UR U5RU5R SS5n UbURXU5up[n URRS:wadURRS:Xa-URSS5(a[RS5 O[URRn U "UUU U U4UR(dS O UR UR"S .UD6upU R$"/UQSP76R'5n UR)U 5n X4$) Nr<r r;eagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rr)rKr^rviewrrrrrr~_attn_implementationgetlogger warning_oncerrrrrZrr)r3rDrrrr input_shape hidden_shape query_statesrrattention_interfacerrs r7rGZambaAttention.forwards$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST  %'5'<'d>d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL %  % ! "));;;;FFH kk+. ((r9) rrr~r^rrrrrrrrrr)rOrPrQrRrr!rr-r/rrrarrrrGrSrTrUs@r7rrs l{lsl.=A ))||))))!. )) !!89 )) -. )) u||Xell3XeELL>Q5RR S))))r9rc^\rSrSrSrS\4U4SjjrS S\RS\ 4Sjjr S S\ 4Sjjr S S\ 4S jjr S r U=r$) ZambaMambaMixeri(u! Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) This module differs from `transformers.models.mamba.modeling_mamba.MambaMixer` in two ways: - Added multi-head: the output of `self.in_proj` is split into `self.n_mamba_heads` heads, and each head undergoes an independent forward pass, identical to the original `MambaMixer`, up until the pre-activations of `self.out_proj`. The pre-activations, coming from different mamba heads, are then concatenated and fed into `self.out_proj`. r~c >[TU]5 XlX lURUlUR UlURUlURUR-Ul URUl URUl URUR-UlURUlUR"Ul[&R("URURUR URURURS- S9UlUR,Ul[0UR,UlUR4Ul[&R8"URURS-UR$S9Ul[&R<"[>R@"URURUR S--UR55Ul![&R<"[>R@"URURUR5S- S-URS-- 5Ul"[&R<"[>R@"URUR55Ul#[>RH"SUR S-[>RJS9SSS24nURMURS5RO5n[&R<"[>RP"U5RSURURS55Ul*[&R<"[>RV"URUR55Ul,[&R8"URURUR$S9Ul-[\(d[^RaS5 gg) Nr ) in_channels out_channelsr kernel_sizegroupspaddingr;rg?r>r<apThe fast path is not available because on of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)1r,r-r~rr4rlrmrnrorjrk mamba_dt_ranktime_step_rankrpmamba_head_dimmamba_conv_bias use_conv_biasmamba_proj_biasuse_biasr Conv1dconv1dhidden_mamba_act activationractuse_mamba_kernelsuse_fast_kernelsrin_projr.r/ry x_proj_weightdt_proj_weight dt_proj_biasaranger@rYrlogrZA_logr0Dout_projis_fast_path_availablerr)r3r~rAr6s r7r-ZambaMambaMixer.__init__5s   "!--$22 & 3 3!'!4!4v7I7I!I$22#11"448J8JJ#33.. ii..//##--))))A-   !11&112 & 8 8yy!1!143I3IA3MTXTaTab  \\ &&''$*=*=*AA'' !ll [[++T-@-@$BUBU VY\ \ !!3& '  LLT5G5GI\I\)]^ LLD//!35== I$PQ' R HHT++R 0 ; ; =\\%))A,"6"6t7I7I4K^K^`b"cd ejj););T=P=PQR $"8"8$:J:JQUQ^Q^_ %%   ^ &r9rD cache_paramsc URupEnUSL=(a UR=(a US:HnURU5RSS5nUR USSU5R SSS9upUR S5R5nU R S5n U RX@RSU5RSS5n URRR URRRS5URRRS55n U(ae[UR S5URURU URR UR"5nUR%S5nOUb1[&R("US:H5(dXR%S5-nUbc[*R,R/XR0URS- S45n URURR3U 5 [5XURR UR"S9nUb1[&R("US:H5(dXR%S5-nURSURUR6U5RSS5nUR8SS2SSS2SS24U-RSS5n [&R:"XR<UR>UR>/SS9upnUR@SS2S4U RSS5-n[&RB"URDRG55*nURHbURHRG5OSn[&RJ"USU4URLURNS9nU(a[QUR5Hn[SURTURSS2U4UUS S4UUS S4UUUUSS2S4UUSS2S4URVUU US S4UUS S 9 R%S5n[&RX"UU4SS9nM GO<[&RJ"USUR6UR>4URLURNS9n[QUR5Hn[[UUUUUUUURSS5UURSS5URVURG5U UUUS S S 9 unn[&RX"UU4SS9R5n[&RX"UUR%S54SS9nM Ub+Ub(URTURR3U5 UR]URSS55nU$) Nr r;r<rr)rrrd.T) dt_softplus)delta_softplusreturn_last_state)/rKrirrrchunksqueezerrZrprr1sizer&rqrrr unsqueezer/allr rpadrocopy_r%rrsplitrrmrexprfloatremptyrer>rwr$rrrrr#r)r3rDr rrseq_lenruse_precomputed_statesprojected_statesgate conv_weightsrqssm_parameters time_stepBCdiscrete_time_steprtime_proj_bias scan_outputsn scan_outputs_ ssm_state ssm_state_contextualized_statess r7cuda_kernels_forward$ZambaMambaMixer.cuda_kernels_forwardtsn"/!4!4 Q!-T!9!nl>]>]!nbimnbn << 6@@AF.33JAwOUUVW]^U_ %--a0;;= ||A||J(:(:BHRRSTVWX{{))..t{{/A/A/F/Fq/I4;;K]K]KbKbcdKef !0%%b)((8    M*33B7M)%))Na>{K,]$++JZJZgkgvgvwM)%))NaRAXRBU RS- S45n XR&UR'UR7UR/U 5SSU245n UbH[(R:"US:H5(d*XSS2U RS*S24R9S5-n O[(RD"X@RURFURH4U R"US9n Ub1[(R:"US:H5(dXR9S5-n UR7UR/U 5SSU245n Ub1[(R:"US:H5(dXR9S5-n U RSURURFU5RSS5n URJSS2SSS2SS24U -RSS 5n[(RL"XRNURHURH/SS9unnnURPSS2S4URSS 5-URRSS2SSS2S4-n[<R>RUU5n[(RV"URXR[55*n[(RV"USS2SSS2SSS24USS2SS2SS2SS2S4-5nUSS2SS2SS2SS2S4USS2SS2SSS2SS24R[5-nUU SS2SS2SS2SS2S4R[5-n/n[]U5HnUSS2SS2SS2USS24RSS5U -USS2SS2SS2USS24RSS5-n [(R^"U RSS5R!U5USS2SS2USS24R9S55nURaUSS2SS2SS2S45 M [(Rb"USS9nUXRdSS2SSS2S4--nUUR7U 5-nU (aXRUR'URgURSS5RUSU5RSS55nU$) Nr r;r<rr)shiftsdims.rdr)4rKr>rrrrrrrZrp isinstancerarrrrcloner?rerirqr/rollsumrr1rrrrrr rrroryrrmrrrrrsoftplusrrrrwrrzstackrr)r3 input_statesr rrrrr>rrDr use_cacher' conv_staterrr r!r"r discrete_A discrete_BdeltaB_ur$r scan_outputr)s r7 slow_forwardZambaMambaMixer.slow_forwards!-!3!3 Q""<< 5??1E.33JAwOUUVW]^U_ %--a0;;= ||A||J(:(:BHRRSTVWX|-DE 00@FFqIZW}}(33DNNCIIK (33DNNC ! ]%9%9:I///qL ,,T^^<BB1ES)55dnnE "ZZ 2BG '4Q1W'= 1a8$;E((8 % *{{7I7I!QPQ'7R*RXZ [ %%![[%5%55M $ 7 : :5 A K KB O !-eiiRS@S6T6T$11}GZGZ[]G^F^F`C`4a4k4klm4n$nM]]..}?T?TWdWjWjkmWn?npq>rs ;E((8 $])CC'M)R S !-eiiRS@S6T6T$11}GZGZ[]G^F^F`C`4a4k4klm4n$nM //1D1DdFYFYZ$++I )%))NaN ? E E GG wA"1aAq=1;;AqAIMPXYZ\]_`bcefYfPgPqPqrsuvPwwI,,y':':1a'@'C'CE'JAaQRTUWXjMLcLcdfLghK    Aq!QJ 7 8 kk,B7 !]VVAtQ=1.2.0' and 'pip install mamba-ssm', or set use_mamba_kernels=False in the model's config.)r)rrrretype ValueErrorr*r<)r3rDr rs r7rGZambaMambaMixer.forward2sl  ))V4;M;M;T;T;Y;Y-Y i ,,]Yg,h h  ^ \\r9)rrrrr~rrorrr4rrkrrrprrmrrrrrr')rOrPrQrRrr!r-r/rrar*r<rGrSrTrUs@r7rr(s^ ={=@im_%"\\_%9P_%B[%7N[%z ]3J ] ]r9rc.^\rSrSrU4SjrSrSrU=r$)ZambaMLPi?c>[TU]5 XlURUlURUl[ R "URURSS9Ul[ R "URURSS9Ul[ R "URURSS9Ul [URUl gNFr) r,r-r~r4rkr r gate_projup_proj down_projr hidden_actact_fnr3r~r6s r7r-ZambaMLP.__init__@s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ r9cURURURU55URU5-5nU$r)rIrKrGrH)r3xrIs r7rGZambaMLP.forwardJs6NN4;;t~~a/@#ADLLQRO#ST r9)rKr~rIrGr4rkrH)rOrPrQrRr-rGrSrTrUs@r7rDrD?s0r9rDc(^\rSrSrSS\S\\4U4SjjjrSS\RS\RS\S\\RS\\ S \\ S \\ S \ \ S \\R\\\R\R444S jjrSrU=r$)ZambaAttentionDecoderLayeriOr~rc>[TU]5 [X5Ul[ U5Ul[ URURS9Ul [ URURS9Ul g)Nr5) r,r-r self_attnrD feed_forwardr)r rms_norm_epsinput_layernormr4pre_ff_layernormrs r7r-#ZambaAttentionDecoderLayer.__init__Ps] ':$V,+F,H,HfNaNab ,V-?-?VEXEX Yr9rDoriginal_hidden_statesrrrr6rrWc [R"X/SS9nURU5nUR"SUUUUUUS.UD6upUR U5nUR U5nU4n U(aX4- n U $)aa Args: hidden_states (`torch.FloatTensor`): output of previous Mamba layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output of shape `(batch, seq_len, embed_dim)`. This is concatenated with `hidden_states` (which is the output of the previous (mamba) layer). The concatenated tensor is then used as input of the pre-attention RMSNorm (see fig. 2 in https://arxiv.org/pdf/2405.16712). layer_idx (`int`): layer_idx in the forward pass. Used to distinguish Zamba's tied transformer layers. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. r<r)rDrrrrr6r)r/ concatenaterXrUrYrV) r3rDr[rrrrr6rself_attn_weightsoutputss r7rG"ZambaAttentionDecoderLayer.forwardXs>))=*QWYZ ,,]; +/>>, '))/ , , ( --m< ))-8  "  + +Gr9)rVrXrYrUrNNFF)rOrPrQrRr!rrr-r/rraboolrrrrrGrSrTrUs@r7rRrROsZ{Zx}ZZ26<@,1$)3||3!& 3 3 !. 3 !!89 3$D>3D>3-.3 u  (51B1BEDUDU1U+V"WW X33r9rRc^\rSrSrS\S\4U4SjjrSS\RS\ \RS\ \S\ \RS\ \RS \ \ S \ \ S \ \ S \ \RS \ \RS\ \R\ \ \R\R444SjjrSrU=r$)ZambaMambaDecoderLayerir~rc>[TU]5 [XS9Ul[ UR UR S9UlX lg)N)r~rrT) r,r-rmambar)r4rWrXrrs r7r-ZambaMambaDecoderLayer.__init__s: $FH +F,>,>FDWDWX"r9rDr[rrrrr6cache_positiontransformer_hidden_statesrWc  Un U bX-OUnURU5nURUUUS9nSn X-nU4nU(aX4- nU(aX4- nU$)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. N)rDr r)rXrf)r3rDr[rrrrrr6rhrirresidualr^r_s r7rGZambaMambaDecoderLayer.forwards<! :S9^M 5dq ,,];  '')# !!0  "  + +G  ( (Gr9)rXrrf) NNNNNFFNN)rOrPrQrRr!rr-r/rrrarbrrrrGrSrTrUs@r7rdrds#{#s#:>#'15.2<@,1$)59<@:||:!) 6:C= : !. : ell+ :!!89:$D>:D>:!!1!12:$,ELL#9: u  (51B1BEDUDU1U+V"WW X::r9rdc|^\rSrSrS\S\R S\4U4SjjrSS\ RS\ \ RS\ \ S \ \ RS \ \ RS \ \ S \ \S \ \S\ \ RS\\ R"\ \\ R"\ R"444SjjrSrU=r$)ZambaHybridLayeri shared_transflinearrfcF>[TU]5 XlX lX0lgr)r,r-rorp mamba_decoder)r3rorprfr6s r7r-ZambaHybridLayer.__init__s * "r9rDr[rrrrrr6rhrWc URUUUUUUUU S9n U Sn U(aU Sn URU 5n URUU UUUUU S9n U(a U SW 4U SS-n U $)a\ Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. )r[rrrrr6rhrr )rirrrr6rhr;N)rorprr) r3rDr[rrrrrr6rh layer_outputsrir^s r7rGZambaHybridLayer.forwards>** #9&)/)+  %2!$4!  -a 0 $(KK0I$J!** &?))/)+  *1-/@AMRSRTDUUMr9)rprrro)NNNNNFFN)rOrPrQrRrRr rrdr-r/rrrrarbrrrrGrSrTrUs@r7rnrns#&@#"))#\r#:>#'15.2<@,1$)59>||>!) 6>C= > !. > ell+ >!!89>$D>>D>>!!1!12> u  (51B1BEDUDU1U+V"WW X>>r9rnc ^\rSrSr\rSrSrSS/rSr Sr Sr Sr Sr Sr\\SS \\R$S \\\\\\44S \S \4U4S jjj55rSrU=r$)ZambaPreTrainedModelimodelTrRrdrFctURRn[U[R[R 45(aWUR RRSUS9 URb%URRR5 gg[U[R5(adUR RRSUS9 URb2UR RURR5 gg[U[5(a&UR RRS5 g[U[5(GaUR RRSUS9 URR"S-n[R$R'UR(U*U5 URR*URR,-URR.-n[0R2"[0R4"URR.U5[6R8"URR:5[6R8"URR<5- -[6R8"URR<5-5R?URR@S9nU[0R8"[0RB"U*5*5-nURDRRGU5 [0RH"SURJS-[0RLS9SSS24nUROURPS5RS5nURTRRG[0R8"U5RWUR.URXS55 URZRRS5 gg) Nr)rBstdg?r)minr rr<).r~initializer_ranger/r rrr1datanormal_rzero_ Embedding padding_idxr)fill_rrrinituniform_rrjr4rpr/rrandmathr time_step_max time_step_minclamptime_step_floorexpm1rrrrmr@rYrkrrrZrr)r3rr{ dt_init_stdrdtinv_dtrs r7 _init_weights"ZambaPreTrainedModel._init_weights&skk++ fryy"))4 5 5 MM   & &CS & 9{{&   &&('  - - MM   & &CS & 9!!- ""6#5#56<<>.  - - MM   $ $S )  0 0  % % - -3C - @++33T9K GG  V22[L+ N![[55 8O8OOSWS^S^SlSllN 4;;44nE88DKK556$++B[B[9\\^((4;;4456e 33e4  %))U[["%5$566F    $ $ * *6 2 Q 5 5 9OPTVWPWXA1126AACA LL   # #EIIaL$8$89M9MvOdOdfh$i j HHMM   $%1r9 torch_dtype device_maphard_check_onlycheck_device_mapcb>[TU]XX4US9nU(dURS:XaSUlU$)z Overloads `PreTrainedModel._check_and_enable_flash_attn_2` so as to DISABLE Flash Attention 2 by default on Zamba models. Flash attention 2 is currently not supported in the HuggingFace implementation of Zamba v1. )rrflash_attention_2r)r,_check_and_enable_flash_attn_2r)rr~rrrrr6s r7r3ZambaPreTrainedModel._check_and_enable_flash_attn_2FsA7 _o8  6#>#>BU#U*1F ' r9rra)rOrPrQrRr! config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_cache_class _is_statefulrrrr/r>r rrrrbrrSrTrUs@r7rxrxsL&*#57OP"3"N L%@.2;? %!& ekk*U3S#X#678     r9rxcH^\rSrSrSrS\4U4SjjrSrSr\ SS\ \ RS\ \ RS \ \ RS \ \S \ \ RS \ \S \ \S\ \S\ \S\ \ RS\\\44Sjj5rSrSrU=r$) ZambaModeli_z Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`ZambaDecoderLayer`] Args: config: ZambaConfig r~c >[T U]U5 URUlURUl[ R "URURUR5Ul[U5n/n/nURUl [UR5HnURUS:XaUR[XS95 M0URUS:XdMEUR[ R"UR RUR RSS95 UR[XS95 M [#U5n[#U5n/n/Ul['UR5HupxUS:XabSUS3n /SQn /UR$QU V s/sHoU -PM sn QUlUR[)U[+U5[+U555 MmUR[+U55 M [ R,"U5UlUR0Ul[3URUR4S 9UlSUlUR;5 gs sn f) Nrf)rrfFrzlayers..) z%shared_transf.self_attn.q_proj.weightz%shared_transf.self_attn.k_proj.weightz%shared_transf.self_attn.v_proj.weightz%shared_transf.self_attn.o_proj.weightz+shared_transf.feed_forward.gate_proj.weightz)shared_transf.feed_forward.up_proj.weightz+shared_transf.feed_forward.down_proj.weightz$shared_transf.input_layernorm.weightz%shared_transf.pre_ff_layernorm.weightrT)r,r- pad_token_idr vocab_sizer rr4 embed_tokensrRrhrwrxrzrdrr~iter_tied_weights_keys enumeraternnext ModuleListlayersrr)rWfinal_layernormgradient_checkpointing post_init) r3r~block mamba_layers linear_layersrrlayer_id layer_type prefix_name tied_keysrr6s r7r-ZambaModel.__init__hs  !.. ++LL):):F,>FDWDWX&+# Gos+I7cUR$rrrLs r7get_input_embeddingsZambaModel.get_input_embeddingss   r9cXlgrrr3rs r7set_input_embeddingsZambaModel.set_input_embeddingss!r9 input_idsr position_idsr inputs_embedsr6routput_hidden_states return_dictrhrWc  UbUOURRnUbUOURRnUbUOURRnU bU OURRn USLUSL- (a [ S5eUR (a/UR(aU(a[RS5 SnUcURU5nUn [R"U5n U(aUc[RS5 U c,[R"U RSU RS9n UcU R!S5nUR#X%U 5n U(aSOSnU(aSOSn[%UR&5HunnU(aX4- nUR (a6UR(a%UR)UR*U U UUU UUUU 5 nOU"U U UUU UUUU S 9 nUSn U(dMuUScM}UUS4- nM UR-U 5n U(aX4- nU(aUR.(dS Ul[1U U(aUOSUUS 9nU (aU$UR35$) NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either onezX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Fz{Zamba requires an initialized `ZambaHybridDynamicCache` to return a cache. None was provided, so no cache will be returned.r rgrr)r[rrrrrr6rhT)last_hidden_staterrD attentions)r~rrr6use_return_dictrArrrrrr/r0rrKrer_update_causal_maskrr_gradient_checkpointing_func__call__rrirto_tuple)r3rrrrrr6rrrrhrDr[rall_hidden_statesall_self_attnsrlayerruoutputs r7rGZambaModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKre)diagonalrgrr;.r)r?xpunpu)r~rr>rer/finfor|rKfulltriurrZrYr0req masked_fillr@r_unmask_unattended) r3r input_tensorrhr>re min_dtypesequence_length target_lengthr mask_length padding_masks r7rZambaModel._update_causal_mask s ;; + +/B B)c^.C%%$**L,?,?vKK&** &,,Q/&r*Q. jj/!Aimst a **[1=Ku||MANDZDZ[]_`Daaa !$a"23::<;M;Ma;PRSUWY[\  %%++-K!!#q(,2226 *3  +<=@@EWXZ^`dfgWgHhHkHkloHpp 1k<$D>k<'tnk<d^k<!!1!12k< u-- .k<k<\!!r9rc^\rSrSrS\4U4SjjrSrSrSrSr Sr S r \ SS \ \RS \ \R S \ \RS \ \S\ \R$S\ \RS\ \S\ \S\ \S\ \S\ \RS\\\R 4S\\\44Sjj5rSSjrSrU=r$)ZambaForCausalLMi1r~c >[TU]U5 [U5UlS/URRQUlUR Ul[ R"URUR SS9Ul UR5 g)Nzlm_head.weightFr) r,r-rryrrr rr4lm_headrrLs r7r-ZambaForCausalLM.__init__2so  ' #3"Tdjj6S6S"T ++yy!3!3V5F5FUS  r9c.URR$rryrrLs r7r%ZambaForCausalLM.get_input_embeddings<zz&&&r9c$XRlgrrrs r7r%ZambaForCausalLM.set_input_embeddings? "' r9cUR$rrrLs r7get_output_embeddings&ZambaForCausalLM.get_output_embeddingsBs ||r9cXlgrr)r3new_embeddingss r7set_output_embeddings&ZambaForCausalLM.set_output_embeddingsEs% r9cXlgrry)r3decoders r7 set_decoderZambaForCausalLM.set_decoderHs r9cUR$rrrLs r7 get_decoderZambaForCausalLM.get_decoderKs zzr9rrrrrlabelsr6rrrrhlogits_to_keeprWc .UbUOURRnU bU OURRn U bU OURRn UR UUUUUUUU U U S9 nUSn[ U [ 5(a [U *S5OU nURUSS2USS245nSnUbUR"UX`R40U D6nU (dU4USS-nUbU4U-$U$[UUURURURS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, ZambaForCausalLM >>> model = ZambaForCausalLM.from_pretrained("Zyphra/Zamba-7B-v1") >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba-7B-v1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrrrr6rrrhrrr losslogitsrrDr)r~rrrryr/rslicer loss_functionrrrrDr)r3rrrrrrr6rrrrhr loss_kwargsr_rD slice_indicesr rrs r7rGZambaForCausalLM.forwardNsKP2C1N-TXT_T_TqTq%9$D $++JjJj &1%SV8W8W~ot4]k mA}a,?@A  %%ffooUUDY,F'+'7D7V# CV C%#33!//))   r9c USLn U (d]UcUSURS:aUSS2URS*S24nOaURSURS:wa USS2U4nO7[URURSURURS9nUbZUcWUR 5R S5S- nURUS:HS5 U (dUSS2URS*S24nUb U (aSU0n OSUR50n U RUUUUURRUS.5 U $)Nr<r r)r>rerr)rrr6rrrh) rKrar~r>relongcumsum masked_fill_rrnum_logits_to_keep) r3rrrrrhrr6r empty_past_kv model_inputss r7prepare_inputs_for_generation.ZambaForCausalLM.prepare_inputs_for_generationsd(4/ )!"%);;%a.*>*>q*A)A)C&CD #~';';A'>>%a&78 5 Y__Q/tzz$++O  %,*>)..077;a?L  % %n&91 = +A 0B/B/D,DE   $+];L')=)=)?@L ,#2&"0"&++"@"@"0   r9)rrryr) NNNNNNNNNNNr)NNNNNT)rOrPrQrRr!r-rrrrrrrrr/rrrarrbr rrrrGrrSrTrUs@r7rr1s{'(&151537=A59-1$(,0/3&*5934O E,,-O !.O u//0 O ""9: O   1 12 O ))*O D>O $D>O 'tnO d^O !!1!12O c5<</0O  u,, -O O h99r9ra The Zamba Model with a sequence classification head on top (linear layer). [`ZambaForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introcZ^\rSrSrU4SjrSrSr\SS\\ RS\\ RS\\ RS\\ \ \\ R4S \\ RS \\ RS \\S \\S \\S\\S\ \\44Sjj5rSrU=r$)ZambaForSequenceClassificationic>[TU]U5 URUl[U5UlURR Ul[ R"URURSS9Ul UR5 grF) r,r- num_labelsrryrr rr4scorerrLs r7r-'ZambaForSequenceClassification.__init__se   ++' "&**"?"?YYv114??O  r9c.URR$rrrLs r7r3ZambaForSequenceClassification.get_input_embeddingsrr9c$XRlgrrrs r7r3ZambaForSequenceClassification.set_input_embeddingsrr9rrrrrrr6rrrrWc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UbURSnOURSnURR cUS:wa [ S5eURR cSnOUbXRR :gRU R[R5n[R"URSU R[RS9nUU-RS5nO.Sn[RURR S35 U [R"XRS 9U4nSnUGbURU R5nURR"cUR$S:XaS URlOoUR$S:aNUR&[R(:XdUR&[R*:XaS URlOS URlURR"S :XaJ[-5nUR$S:Xa&U"UR/5UR/55nOU"UU5nOURR"S :Xa=[15nU"UR3SUR$5UR3S55nO-URR"S :Xa[55nU"UU5nU (dU4U SS-nUbU4U-$U$[7UUU R8U R:U R<S 9$)ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrrr6rrrrr z=Cannot handle batch sizes > 1 if no padding token is defined.r<rdz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rg regressionsingle_label_classificationmulti_label_classificationr)r~rryrrKrrAr?rer/int32rargmaxrrr6rO problem_typerr>rrr rr rr rrrDr)r3rrrrrrr6rrrtransformer_outputsrDr rlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrs r7rG&ZambaForSequenceClassification.forwards(&1%!F!Fr!J !#    >>**+,ZZ  u||J}}MOaab   YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6#%(;AB(??F)-)9TGf$ Ev E/ /??-;;*55   r9)rryrrr)rOrPrQrRr-rrrrr/rrr rrrrbrrrGrSrTrUs@r7rrs)'(151537KO59-1$(,0/3&*[ E,,-[ !.[ u//0 [ "%tE4E4E/F(F"GH [   1 12 [ ))*[ D>[ $D>[ 'tn[ d^[  u66 7[ [ r9r)rrrrx)r)Prrtypingrrrrrrr r/torch.utils.checkpointr torch.nnr r r activationsr cache_utilsrr generationrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_outputsrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrutils.import_utilsrrconfiguration_zambar!&mamba_ssm.ops.selective_scan_interfacer"r#+mamba_ssm.ops.triton.selective_state_updater$ causal_conv1dr%r&rr get_loggerrOrModuler)rzrrr_rarrrrrDrRrdrnrxrrr__all__rr9r7rFse( DDD AA!.)>BqqF&1,T,XR@P=-~DD-7**.0@BVXfg   H %J299J(L) UU\\ U# U%,, U[fl[fJ% II% <<% % << % U\\* %  %%4I)RYYI)XS]biiS]n ryy  <<~ARYYAHEryyEPA?AAHM%MMbh+_hV  m %9m  m ` gr9