fThSrSSKrSSKJr SSKJrJrJrJrJ r SSK r SSK r SSK J r SSK Jr SSKJr SS KJr SS KJr SS KJr SS KJrJrJr SS KJrJrJr SSKJ r \RB"\"5r#\"5(aSSK$J%r% OSr%\"5(aSSK&J'r'J(r( SSK)J*r* OSur*r(r'\"5(a SSK+J,r,J-r- OSur-r,\."\*\(\,\-\'45r/"SS\ R`5r1"SS\ R`5r2"SS\ R`5r3\"SS\55r4\"SS\55r5\"S S!\55r6\"S"S#\455r7\"S$S%9"S&S'\4\55r8/S(Qr9g))zPyTorch MAMBA model.N) dataclass)AnyDictOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN) MambaCache)GenerationMixin)PreTrainedModel) ModelOutputauto_docstringlogging)is_causal_conv1d_availableis_mamba_ssm_availableis_mambapy_available) MambaConfig)pscan)mamba_inner_fnselective_scan_fn)selective_state_updateNNN)causal_conv1d_fncausal_conv1d_update)NNc X^\rSrSrSrS\S\4U4SjjrSS\RS\ \ S\ \RS \ \R4S jjr SS\ \ S\ \RS \ \R4S jjrSS\ \ S\ \RS \ \R4S jjrS rU=r$) MambaMixer>uo Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) config layer_idxc >[TU]5 XlURUlURUlUR UlURUl[UR5Ul X l URUl [R"URURURUR URUR S- S9UlUR Ul[$UR UlUR(Ul[R*"URURS-UR,S9Ul[R*"URURUR S--SS9Ul[R*"URURSS9Ul[4R6"SUR S-[4R8S9SSS24nUR;URS5R=5n[R>"[4R@"U55Ul![R>"[4RD"UR55Ul#[R*"URURUR,S9Ul$UR,Ul[J(dWUR((a0[M5(a[NRQS 5 g[SS 5e[NRQS 5 gg) Nr) in_channels out_channelsbias kernel_sizegroupspaddingr'FTdtypea7The fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. Falling back to the mamba.py backend. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1dzuse_mambapy is set to True but the mambapy package is not installed. To install it follow https://github.com/alxndrTL/mamba.py.aThe fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. Falling back to the sequential implementation of Mamba, as use_mambapy is set to False. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. For the mamba.py backend, follow https://github.com/alxndrTL/mamba.py.)*super__init__r" hidden_size state_sizessm_state_size conv_kernelconv_kernel_sizeintermediate_sizeinttime_step_rankr# use_conv_biasr Conv1dconv1d hidden_act activationr act use_mambapyLinearuse_biasin_projx_projdt_projtorcharangefloat32expand contiguous ParameterlogA_logonesDout_projis_fast_path_availablerlogger warning_once ImportError)selfr"r#A __class__s `/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/mamba/modeling_mamba.pyr1MambaMixer.__init__Fs}  !--$// & 2 2!'!9!9!&"7"78"#11ii..//%%**))&&*   !++&++,!--yy!1!143I3IA3MTZTcTcd ii 6 68K8KdNaNadeNe8elqr yy!4!4d6L6LSWX  LLD//!35== I$PQ' R HHT++R 0 ; ; =\\%))A,/ ejj)?)?@A $"8"8$:J:JQWQ`Q`a  %%'))''F &Z##J& hidden_states cache_paramscache_positionattention_maskc URU5RSS5nUR(Ga.UGc*[UURR UR (aURROSURR URR URR UR(a$URRR5OS[R"URR55*SSUR R5URRR5SS9 nU$UR#SSS9upUbXR%S5-nURR R'URR R)S5URR R)S55nUbnUSS:ae[+UR-S5UR.UR0UURRUR25nUR%S5nOUbW[4R6R9XR:UR<S- S45n UR?UR0X5 [AXURRUR2S9nUbXR%S5-nURURSS55n [RB"XRDURFURF/SS9upn URR U RSS5-n[R"URR55*n[IURS 5(a$URRR5OSnUbcUSS:aZ[KURLUR0US US UU SS2S4U SS2S4UR US USS 9 R%S5nOo[OUUUU RSS5U RSS5UR R5UUSSS 9 unnUbUbURQUR0U5 URURSS55nU$) Nrr+T) delta_biasdelta_softplusdimrr/)r>r').r) dt_softplus)rareturn_last_state))rC transposetrainingrr<weightr:r'rDrErPrBfloatrFexprMrOchunk unsqueezeviewsizersqueeze conv_statesr#r>r functionalpadr6shapeupdate_conv_statersplitr9r4hasattrr ssm_statesrupdate_ssm_state)rUr[r\r]r^projected_statescontextualized_statesgate conv_weightsrpssm_parameters time_stepBCdiscrete_time_steprVtime_proj_bias scan_outputs ssm_states rXcuda_kernels_forwardMambaMixer.cuda_kernels_forwards+ << 6@@AF ===\1$2  ""$($6$6   D "" ## $$.2mm ""((*4::++-.. <<,,224#% !p%$O#3"8"8"8"B M) -0H0H0K K  ;;--224;;3E3E3J3J13Mt{{OaOaOfOfghOijL'N1,=,A 4!))"- ,,T^^< KK$$OO ! !. 7 7 ; +"$--"3"3%(=(= @S@STV@W(WYZ'[#K!224>>;_ 0!1A1Adoo! ) -0H0H0K K "[[)@)@A)FGN#kk!4!4d6I6I4K^K^ _egOI!"&!4!4y7J7J1a7P!P 4::++-..A:A$,,PV:W:WT\\..446]aN'N1,=,A5 ++DNN;!&)&v.adGadGFFL" $ )B-+<!&KK1%KK1%FFLLN"#'&* +' i(\-E 11$..)L%)MM,2H2HA2N$O !$$rZc ^ URupVnURnURU5RSS5n U R SSS9upUbXR S5-n UGbUR URR5n U RU R5n URSUR:Xa[RRU URU RS- S45n URURX5 UR!UR#U 5SSU245n GO6URURX5n U RUR"R$R5n [&R("XR"R$SS2SSS24-SS9n UR*(aXR"R,- n UR!U 5RU5R S5n O][&R."XPR0UR24U RUS9n UR!UR#U 5SSU245n UbXR S5-n UR5U RSS55n[&R6"XR8UR2UR2/SS9unnnUR;U5n[RR=U5RSS5n[&R>"UR@RC55*n[&R>"USSS2SSS24USS2SS2SS2S4-5nUSS2SS2SS2S4USS2SSS2SS24RC5-nUU SS2SS2SS2S4RC5-nURD(aURF(aUc[IURSS5URSS55nUUR S5-RKS5RSS5nUXRLSSS2S4--nUUR!U 5-nO/n[OU5HnUSS2SS2USS24U -USS2SS2USS24-n [&RP"U RU5USS2USS24R S55nURSUSS2SS2S45 M [&RT"USS9nUXRLSSS2S4--nUUR!U 5-nUb(UR URRWU 5 URYURSS55nU$) Nrr+rbrr/.devicer.r )-rsr.rCrfrkrlrwr#clonetorr6r rqrrrtr?r<rhrFsumr:r'zerosr7r4rDrur9rEsoftplusrjrMrir@rgrrorOrangematmulappendstackcopy_rP)rU input_statesr\r]r^ batch_sizeseq_len_r.ryr[r{r conv_stater}r~rrrrV discrete_A discrete_BdeltaB_uhs scan_outputrirzs rX slow_forwardMambaMixer.slow_forwards!-!3!3 Q""<< 5??1E.44QA4>  %),D,DQ,GGM  #$//?EEGI! ]%9%9:I##A&$*?*??]]..!**]-@-@-DDaH ..t~~zZ $])CC'M)R S );;DNNMj ']]4;;+=+=+D+DE % *{{7I7I!QPQ'7R*RXZ [ %%![[%5%55M $ 7 : :5 A K KB O  33T5H5HI$++5I!HHT[[%?XgX %NOM  %),D,DQ,GGM]%<%!QPQST*AUU #ll9<<+>!Q' @T@TUW@XY ##K1a$89$ ++l;K%a 9N)NOK&$7K'''7==iH!% k.C.CAq.I J$$rZc[(acSURRRR;a5[ R R5(dURXX45$URXX45$)Ncuda) rQrDrhrtyperF_dynamo is_compilingrr)rUr[r\r]r^s rXforwardMambaMixer.forward:sb " !f 0B0B0I0I0N0N&NW\WdWdWqWqWsWs,,].i i  n]]rZ)rMrOr?r>r"r<r6rEr2rCr7r#rPr4r9rBr:r@rDr)__name__ __module__ __qualname____firstlineno____doc__rr8r1rFTensorrr LongTensorrrr__static_attributes__ __classcell__rWs@rXr r >s:{:s:~.25959 c%||c%z*c%!!1!12 c% !!1!12 c%LO%x 7KO%aijojzjza{O%S[\a\l\lSmO%j.25959 ^z* ^!!1!12 ^ !!1!12 ^ ^rZr c8^\rSrSrSU4SjjrSrSrSrU=r$) MambaRMSNormiFc>[TU]5 [R"[R "U55UlX lg)z< MambaRMSNorm is equivalent to T5LayerNorm and LlamaRMSNorm N)r0r1r rKrFrNrhvariance_epsilon)rUr2epsrWs rXr1MambaRMSNorm.__init__Gs/ ll5::k#:; #rZcURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)Nr+r/T)keepdim) r.rrFrHpowmeanrsqrtrrh)rUr[ input_dtypevariances rXrMambaRMSNorm.forwardOsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::rZcRURRSSUR3$)Nrz, eps=)rhrsrrUs rX extra_reprMambaRMSNorm.extra_reprVs*++##A&'vd.C.C-DEErZ)rrh)gư>) rrrrr1rrrrrs@rXrrFs$;FFrZrc^\rSrSrU4SjrSS\\S\\RS\\R4Sjjr Sr U=r $) MambaBlockiZc>[TU]5 XlX lURUl[ UR URS9Ul[XS9Ul g)Nrr#) r0r1r"r#residual_in_fp32rr2layer_norm_epsilonnormr mixer)rUr"r#rWs rXr1MambaBlock.__init__[sL  " & 7 7 !3!39R9RS < rZr\r]r^c UnURURURRRS95nUR(aUR[ R 5nURXX4S9nXQ-nU$)Nr-r\r]r^)rrrhr.rrFrHr)rUr[r\r]r^residuals rXrMambaBlock.forwardcsx! -"2"29I9I9O9O"2"PQ  {{5==1H ^# !0 rZ)r"r#rrrr) rrrrr1rr rFrrrrrs@rXrrZsV=.25959 z*!!1!12  !!1!12 rZrc2\rSrSr\rSrSS/rSrSr Sr Sr g) MambaPreTrainedModelivbackbonerr TcB[U[5(Ga^SURlSURlUR R S-UR R-nUR RS:Xa5[RRURRU5 OPUR RS:Xa6[RRURRU*U5 [R "[R""UR R$5[&R("UR R*5[&R("UR R,5- -[&R("UR R,5-5R/UR R0S9nU[R("[R2"U*5*5-n[R4"5 URR6R9U5 SSS5 SURR6l[U[R<5(aSUR6bE[?UR6SS5(d)[RRAUR65 O[[U[RB5(a<[RREURUR RFS 9 UR RH(aURK5HupVUS ;dM [RRMU[&RN"S 5S 9 [R4"5 U[&RN"UR RP5-nSSS5 M gg!,(df  GN=f!,(df  M=f) zInitialize the weights.Tgconstantrandom)minN _no_reinitF)std)zout_proj.weight)a)) isinstancer rM_no_weight_decayrOr"r9time_step_scaletime_step_init_schemer init constant_rErhuniform_rFrjrandr7mathrL time_step_max time_step_minclamptime_step_floorexpm1no_gradr'rrrAgetattrzeros_ Embeddingnormal_initializer_rangerescale_prenorm_residualnamed_parameterskaiming_uniform_sqrtnum_hidden_layers)rUmodule dt_init_stddtinv_dtnameps rX _init_weights"MambaPreTrainedModel._init_weights~s fj ) ),0FLL )(,FHH %++44d:T[[=X=XXK{{00J>!!&.."7"7E22h>  !6!6 kR 4;;88988DKK556$++B[B[9\\^((4;;4456e 33e4  %))U[["%5$566F##))&1!-1FNN   * fbii ( ({{&v{{L%@@GGNN6;;/  - - GGOOFMMt{{/L/LO M ;; / /"224.. GG,,Q$))A,,?TYYt{{'D'DEE)5 0!2)s,&O=.P= P  P N) rrrrr config_classbase_model_prefix_no_split_modulessupports_gradient_checkpointing _is_statefulrrrrZrXrrvs)L"%|4&*#L-FrZrc\rSrSr%SrSr\\R\ S'Sr \\ \ S'Sr \\ \R\ S'Srg) MambaOutputia Class for the MAMBA model outputs. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. cache_params (`MambaCache`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. Includes both the State space model state matrices after the selective scan, and the Convolutional states hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. Nlast_hidden_stater\r[r)rrrrrrrrF FloatTensor__annotations__r\r r[rrrrZrXrrsH$6:x 1 129)-L(:&-8[TU]U5 [R"URUR 5Ul[R"[UR5Vs/sH n[XS9PM sn5Ul SUl [UR URS9UlUR!UR"5 UR%5 gs snf)NrFr)r0r1r r vocab_sizer2 embeddings ModuleListrrrlayersgradient_checkpointingrrnorm_f"_register_load_state_dict_pre_hook load_hook post_init)rUr"idxrWs rXr1MambaModel.__init__s  ,,v'8'8&:L:LMmmRWX^XpXpRq$rRq3Z%FRq$rs &+#"6#5#56;T;TU  //?  %ss(CclUH.nSU;dM URU5XRSS5' g g)Nz embedding.z embeddings.)popreplace)rU state_dictprefixargsks rXrMambaModel.load_hooks4Aq EO^^TUEV 99\=ABrZcUR$Nrrs rXget_input_embeddingsMambaModel.get_input_embeddingss rZcXlgr$r%rUnew_embeddingss rXset_input_embeddingsMambaModel.set_input_embeddingss(rZ input_ids inputs_embedsr\ use_cacheoutput_hidden_states return_dictr]r^returnc UbUOURRnUbUO(UR(dURROSnUbUOURRnUSLUSL- (a [ S5eUcUR U5nUR(aUR(a U(aSnU(aUcn[URURS5URURS9n[R"SURRURS9nOUc [ S5eOSnUn U(aSOSn URHZn UR(a/UR(aUR!U R"XXx5n O U "U UUUS 9n U(dMUX4-n M\ UR%U 5n U(aX4-n U(d['S XU 455$[)U U(aUU S 9$SU S 9$) ay cache_params (`MambaCache`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). use_cache (`bool`, *optional*): If set to `True`, the `cache_params` is returned and can be used to quickly generate the next logits. NFz:You must specify exactly one of input_ids or inputs_embedsrrrzYou have to specify the `cache_position` manually when `use_cache=True` and `cache_params` is passed, you don't have to pass a `cache_params` if you are in prefilling stage because in that case it will be initialized for you automaticallyrrc3.# UH ocMUv M g7fr$r).0vs rX %MambaModel.forward..Isf$Tq$Ts )rr\r[)r"r0rgr/use_return_dict ValueErrorrrr rnrr.rFrGr5r_gradient_checkpointing_func__call__rtupler) rUr-r.r\r/r0r1r]r^r[all_hidden_states mixer_blocks rXrMambaModel.forwards(%9$D $++JjJj "+!6IZ^ZgZgT[[=R=Rmr %0% Q 'tn Q d^Q !!1!12Q !!1!12Q  uk! "Q Q rZrz The MAMBA Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc^\rSrSrS/rU4SjrSrSrSrSr SS\ S \ \ \ 4S \S \ \ \ 44S jjrSS \\S\\R&S\\R&4Sjjr\SS\\R&S\\R&S\\R,S \\S\\R&S\\S\\S\\S\\R0S \\\44Sjj5rSrU=r$)MambaForCausalLMiRzlm_head.weightc>[TU]U5 [U5Ul[R "UR URSS9UlUR5 g)NFr,) r0r1rrr rAr2rlm_headr)rUr"rWs rXr1MambaForCausalLM.__init__[sF  "6* yy!3!3V5F5FUS  rZcUR$r$rGrs rXget_output_embeddings&MambaForCausalLM.get_output_embeddingsbs ||rZcXlgr$rJr)s rXset_output_embeddings&MambaForCausalLM.set_output_embeddingses% rZc6URR5$r$)rr&rs rXr&%MambaForCausalLM.get_input_embeddingshs}}1133rZc8URRU5$r$)rr+r)s rXr+%MambaForCausalLM.set_input_embeddingsks}}11.AArZoutputs model_kwargsnum_new_tokensr2c URSS5US'URSS5(aSU;aUSbUSSSU-US'SU;a<USn[R"XURURSS45/SS 9US'U$) Nr\r/Tr]r/r^rrrb)getrFcatnew_onesrs)rUrTrUrVkwargsr^s rX#_update_model_kwargs_for_generation4MambaForCausalLM._update_model_kwargs_for_generationns(/{{>4'H ^$   [$ / / L0-.:-9:J-KBC-PSa-aL) * | +)*:;N-2YY!8!8.:N:Nq:QST9U!VW]_.L) *rZr\r]r^c LU(aiUc [S5eUSS:aUSS2S4RS5nUbSnO4[R"SURR UR S9nUbUcSU0nOSUR50nURUUUUS.5 U$)Nz`cache_position` should not be None as it should have been initialized in `model.generate`, you are responsible for passing in a valid `cache_position` if you are calling `prepare_inputs_for_generation` directly with `use_cache=True`rr/r4r.r-)r\r/r]r^) r;rlrFrGr"r5rrJupdate) rUr-r.r/r\r]r^r[ model_inputss rXprepare_inputs_for_generation.MambaForCausalLM.prepare_inputs_for_generations % e a 1$%ae,66r: !-%)N"'a1H1HQZQaQa!b  $)=+];L')=)=)?@L ,&"0"0   rZr-r.labelsr0r1r/c UbUOURRnURUUUUUUU US9n U Sn URU R URR R 55R5n SnUbUR U R5nU SSS2SS24R5nUSSS24R5n[5nU"URSURS55URS55nU(dU 4U SS-nUbU4U-$U$[UU U RU RS9$)a cache_params (`MambaCache`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` use_cache (`bool`, *optional*): If set to `True`, the `cache_params` is returned and can be used to quickly generate the next logits. N)r\r.r0r1r/r]r^r.r/r)r r r\r[)r"r:rrGrrhr.rirrJr rmrnr r\r[)rUr-r^r.r\rcr0r1r/r]r[ mamba_outputsr[r r  shift_logits shift_labelsloss_fctoutputs rXrMambaForCausalLM.forwards_2&1%< !.<  u)) *< < rZrE)rErr):rr dataclassesrtypingrrrrrrFtorch.utils.checkpointr torch.nnr activationsr cache_utilsr generationrmodeling_utilsrutilsrrrutils.import_utilsrrrconfiguration_mambar get_loggerrrR mambapy.pscanr&mamba_ssm.ops.selective_scan_interfacerr+mamba_ssm.ops.triton.selective_state_updater causal_conv1drrallrQModuler rrrrr rrE__all__rrZrXrs !44 %!%)- kj,   H %# EXR@P=-~DD-7**.0@BVXfg E^E^PF299F(84F?4F4Fn =+= =0 =+= =6k %k k \ V +_V  V r ErZ