o Zh F@s`ddlmZmZmZmZmZddlZddlmm Z ddlmZddl m m mZddlmZddlmZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd l m!Z!m"Z"dd l#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,e*rddl-m.Z.ddl/m0Z0m1Z1ndZ.e)rddl2m3Z3m4Z4nd\Z4Z3e&rddl5m6Z6ddl7m8Z8e'9e:Z;ddZde?dej>fddZ@ dWd!ejAd"ej>d#ej>d$ej>d%eej>d&eBd'eBfd(d)ZCGd*d+d+ejAZDGd,d-d-ejEZEd.ej>d/e?fd0d1ZFd2d3ZGd4d5ZHeIe.e3e4fZJd6d7ZKGd8d9d9ejAZLGd:d;d;ejjAZMGdd?d?ejAZOGd@dAdAejAZPGdBdCdCejAZQGdDdEdEejAZRGdFdGdGeZSe$GdHdIdIe"ZTGdJdKdKejAZUe$GdLdMdMeTZV  N dXdOeej>eej>dfdPee?d%eej>deej>e?ffdQdRZWGdSdTdTeTeZXgdUZYdS)Y)CallableListOptionalTupleUnionN)nn)ACT2FN)Cache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutputWithPastMoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)auto_docstringcan_return_tupleis_torch_flex_attn_availablelogging)is_causal_conv1d_availableis_mamba_2_ssm_available)GraniteMoeHybridConfig)selective_state_update)mamba_chunk_scan_combined mamba_split_conv1d_scan_combined)causal_conv1d_fncausal_conv1d_updateNN) BlockMask)make_flex_block_causal_maskcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)xx1Zx2r.m/var/www/auris/lib/python3.10/site-packages/transformers/models/granitemoehybrid/modeling_granitemoehybrid.py rotate_half@sr0cCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer0)qkcossin position_idsZ unsqueeze_dimZq_embedZk_embedr.r.r/apply_rotary_pos_embGs  r7 hidden_statesn_repreturncCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r)expandreshape)r8r9batchnum_key_value_headsslenhead_dimr.r.r/ repeat_kvbs 0rAmodulequerykeyvalueattention_maskscalingdropoutc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr&r r%)r(dtype)ptrainingr)rAnum_key_value_groupsr*matmul transposer)r functionalsoftmaxfloat32torKrIrM contiguous) rCrDrErFrGrHrIkwargs key_states value_states attn_weights causal_mask attn_outputr.r.r/eager_attention_forwardns & r\cseZdZdZdedeffdd Z      ddejd e ejd e ej d e e d e d e ej de e ejejfde eje eje e ejffddZZS)GraniteMoeHybridAttentionz=Multi-headed attention from 'Attention Is All You Need' paperconfig layer_idxcs t||_||_|durtd|jjd|j|_|j |_ |j |_ |j |j |_ |j |_ |j |j |_d|_|j|_|j |j |j krUtd|j d|j dtj|j |j |j |jd|_tj|j |j |j |jd|_tj|j |j |j |jd|_tj|j |j |jd|_dS)NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).bias)super__init__r^r_logger warning_once __class____name__attention_dropout hidden_sizenum_attention_heads num_headsr@r>rNZ is_causalZattention_multiplierrH ValueErrorrLinearZattention_biasq_projk_projv_projo_projselfr^r_rfr.r/rcs2  z"GraniteMoeHybridAttention.__init__NFr8rGr6past_key_value use_cachecache_positionposition_embeddingsr:cKsj|\} } } ||} ||} ||}| | | |j|jdd} | | | |j|jdd} || | |j|jdd}|durF|nd\}}|durWt | | ||\} } |durl|||d}| | ||j |\} }t }|j jdkr|j jdkr|ddrtd nt|j j}||| | ||f|jsd n|j|jd |\}}|| | d }||}|||fS) Nrr&r")r5r4rweagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rB)rIrHr%)sizernrorpviewrkr@rPr>r7updater_r\r^_attn_implementationgetrdrerrMrhrHrq)rsr8rGr6rurvrwrxrVbszZq_len_Z query_statesrWrXr4r5Z cache_kwargsZattention_interfacer[rYr.r.r/forwardsF         z!GraniteMoeHybridAttention.forward)NNNFNN)rg __module__ __qualname____doc__rintrcr*Tensorr LongTensorr boolrr __classcell__r.r.rtr/r]s4# r]cs.eZdZdZejdfdeffdd ZZS) HybridMambaAttentionDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. Nr^c sBt|||j|_d|_|j}|j}g|_g|_g|_t |j D]^}|j|dkr\|jt j |j |jd|j|||dg7_|jt j |j|j||dg7_q$|jt jggdg7_|jt jggdg7_|j|q$fddt |j D|_fddt |j D|_dS) NFmambar&devicerKrc g|] }tjggdqSrr*tensor.0r batch_sizerr.r/  z=HybridMambaAttentionDynamicCache.__init__..crrrrrr.r/rr)rbrclayers_block_typehas_previous_state mamba_d_conv mamba_d_state conv_states ssm_statesZtransformer_layersrangenum_hidden_layersr*zeros mamba_expandrimamba_n_groups mamba_n_heads mamba_d_headrappendZ key_cacheZ value_cache)rsr^rrKrconv_kernel_sizessm_state_sizeirtrr/rcsD   z)HybridMambaAttentionDynamicCache.__init__) rgrrrr*Zfloat16rrcrr.r.rtr/rs" r input_tensorpad_sizecCsHt|jdkrddddd|ddfnddd|ddf}tjjj||dddS)z Padding x tensor with `pad_size` on the seq_len dim (dim=1) Assumes that we only have tensors of either size 4 or 3 rZconstant)moderF)lenr)r*rrQpad)rrZ pad_shaper.r.r/pad_tensor_by_size!s2rcCsXt||}t|jdkr||jdd||jdS||jdd||jd|jdS)z Padding input_tensor with `pad_size` on the seq_len dim (dim=1) and simultaneously splitting it into chunk sequences. Assumes that we only have tensors of either size 4 or 3 r rr%r&)rrr)r<)rr chunk_sizer.r.r/reshape_into_chunks,s rcCs|d}|djg||R}tjtj|||jtjddd}||d}tj|dd}tjtj|||jtjddd}||tj }|S)zo More stable segment sum calculation. Uses cumulative sums and masking instead of direct subtractions. r%.NrZdiagonalrrJr') r|r;r*Ztrilonesrr masked_fillcumsuminf)rrmaskZ tensor_segsumr.r.r/ segment_sum@s   rcCsN|dur%|jddkr%|jddkr%|j}||dddddf|}|S)zm Tunes out the hidden states for padding tokens, see https://github.com/state-spaces/mamba/issues/66 Nrr)r)rKrT)r8rGrKr.r.r/apply_mask_to_padding_statesWs$ rc seZdZdZdedeffdd Z    ddejde e d e ej d e ejd e ej f d d Z   dde e d e ej d e ejfddZ    dde e d e ej d e ejd e ej fddZZS)GraniteMoeHybridMambaLayeruO Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) The are a few differences between this and Mamba2Mixer: - The variable use_precomputed_states is slightly different due to the HybridCache structure - There's a few non-obvious bugs fixed with batching in the slow path that exist in main - Some extra variables that our layer doesn't need have been removed - We ported most of the refactors in https://github.com/huggingface/transformers/pull/35154, which is (as of Dec 18, 2024) unmerged r^r_cst|j|_|j|_|j|_|j|_t |j |j|_ ||_ |j |_|j|_t|j|_|j|_|j|_|j|_|j|_|j|_dtdf|_d|_d|_ |j d|j|j|_!t"j#|j!|j!|j |j|j!|jdd|_$|j |j!|j}t"j%|j||jd|_&t"'t()|j|_*t(+d|jd}t"'t(,||_-d |j-_.t/|j |jd |_0t"'t()|j|_1d |j1_.t"j%|j |j|jd|_2t3st45d dSt45d dS) NrBrgMbP?g?r&r)Z in_channelsZ out_channelsraZ kernel_sizegroupspaddingr`TepsaThe fast path is not available because on of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1dzOThe fast path for GraniteMoeHybrid will be used when running the model on a GPU)6rbrcrrkrirrrrrrintermediate_sizer_Zmamba_conv_bias use_conv_bias hidden_act activationractZmamba_proj_biasZuse_bias rms_norm_epsZlayer_norm_epsilonrn_groupsrr@Zmamba_chunk_sizerfloattime_step_limitZ time_step_minZ time_step_maxconv_dimrConv1dconv1drmin_proj Parameterr*rdt_biasarangelogA_logZ_no_weight_decayGraniteMoeHybridRMSNormGatednormDout_projis_fast_path_availablerdre)rsr^r_Zprojection_sizeArtr.r/rcqs\    z#GraniteMoeHybridMambaLayer.__init__Nr8 cache_paramsrwrGseq_idxcCst||}||}|j\}}} |j|j} |duoD|joD|dkoD|j|jjd|j|jjdko8|knoD|duoD|ddk} | r)| dj |j |j |j gdd\} } }t| |j|j|jj d|jj|j} tj | |j | | gdd\}}}t|j }|ddddfdddddfd|j|jjtjd}|dddddfdd|j}|jddddfd|j}|jddddfd|j}|||j|jd|j}|||j|jd|j}|||j |j}t|j|j||||||d|dd }|||j |j}||| }| |ddddf}|St|j }|j!d td fkr>ind |j!i}|j"r||dur|t#||jj d|jj|j|f|j|j$||j|jj|jj%|j j|j j|j|jd d d |}|S|j |j |j |j gdd\} } }|dur| &dd}t'j()||j*|jddf}|j|j+||jdvr|,|| &dddd|f&dd} nt-| &dd|jj d|jj|j|d&dd} t| |} tj | |j | | gdd\}}}t.|||d|j||||||jd||||jdf|j$|jd|d|jdd|\}}|dur:|dur:|j|j+||||d}||| }| |}|S)Nrrr%r'.rKT)zr dt_softplusrBrZdt_limitF) rrrrZrmsnorm_weightZ rmsnorm_epsZoutproj_weightZ outproj_biasZheaddimZngroupsZnorm_before_gatereturn_final_statesr&)siluZswish)r,weightrarr)rrrrrrr)/rrr)rrrrr_rsqueezesplitrrrkr!rrrarr*exprrr;r@rTrSrrr}rrrrrMrrvariance_epsilonrPrrQrrcopy_rr r)rsr8rrwrGrprojected_statesrseq_lenrZgroups_time_state_sizeuse_precomputed_statesgatehidden_states_B_CdtBCrrrZhidden_states_reshapedoutZdt_limit_kwargshidden_states_B_C_transposedr scan_output ssm_stater.r.r/cuda_kernels_forwards          <"  ^" V   $        z/GraniteMoeHybridMambaLayer.cuda_kernels_forwardc3 s|j\}}}|j}t||}|} | jjjjgdd\} } } |duoQ|joQ|dkoQ|j j jd|j j jdkoE|knoQ|duoQ|ddk} | r|j j j ddd|j j <| dddddf |j j j|j j dddddf<|j j j jjjd}tj|jjddd} jr| jj} | } n8|dur| dd}tj|j|jddf}|j j || dddd|fdd} t| |} tj| jjjjjgdd\}}}tj ! }| r[|j j j}| dddddfddddf} | dd"|| jdj#} j$d "j$jdj#}tjj%| | | j} t&| j'dj'd} |d "jj#jj tj(d }t| d |j |d}|)|jdddddf}|"|jjj|jd*}|)|d|jd}| d |ddddf}|)|dj#}||d j |d}|j j |j j |||)|jdddddf}|"|jjj|jd*}|)|d|jd}|j j j |j|jd }|+|jj#j}|+|jjd}t,||}|+|jj#}j-d "j-jdj#}||| |j}|)|dddddf}ntj%| j$} t&| j'dj'd} |)||dj#!}|)||dj!}|)||dj!}|j.jjdjd }|j.jjdjd }j/|j/j/j-d t0|}|| d }| |j| }fdd||||fD\}}}}|1dddd}tj2|dd}tt3|} |dddddddddddf|dddddddddddf}!|!jdd}"|"d | 1dddddd }#|#jdd}$|$d |dddddfjdd}%t|ddddddddf|}&||&1ddddd }'|'ddddf|d jdd}(| r|j j ddddfj |(jd})n t4|(ddddf})tj5|)|(gdd}(tt3tj|dddddddfd}*|*dd}*|*d |(ddddddfjdd}+|+ddddf|+dddf}(},t|}-|ddddf|(ddddddf}.|-1dddd}/|.d|/d }0|%|0}|)|djj#}||}dkrB|ddd|ddddf}|)||d}|,dur_|dur_|j j |,d|_6|| }17|1 |}2|2S)Nr%r'rr)Zshiftsdimsrr&.r).NNrr)r( output_sizecsg|] }t|jqSr.)rr)rtrrsr.r/rsz.r rrJ)rrT)8r)rKrrrrrrkrrr_rZrollrTrrrr*sumrrrarrPrrQrrrrrrrrr;r@rZsoftplusclamprrSr<rUr}ZbmmrZrepeat_interleaverrZpermuterrZ zeros_liker+rr)3rsZ input_statesrrwrGrrrrKrrrrrrrr8rrrZ cache_devicerZdAZdBZdBxrZssm_states_reshapedZ C_reshapedyrZ D_residualZA_cumsumLZG_intermediateGZM_intermediateMZY_diagZ decay_statesZB_decayZstatesZprevious_statesZ decay_chunkZ new_statesrZstate_decay_outZC_times_statesZstate_decay_out_permutedZY_offrZcontextualized_statesr.rr/ torch_forward^s      @   ,  $"$ $  $P  &*"&0 (& *   z(GraniteMoeHybridMambaLayer.torch_forwardcKstrd|jjjjvr||||||S|durtd|j}|dur@|jddkr@|jddkr@||dddddf |}| ||||S)Ncudaz\`seq_idx` support requires fast path support. Please install `mamba_ssm` and `causal_conv1d`rr) rrrrtyperNotImplementedErrorrKr)rTr)rsr8rrwrGrrVrKr.r.r/r.s $ z"GraniteMoeHybridMambaLayer.forward)NNNN)NNN)rgrrrrrrcr*rrrrZ IntTensorrrrrr.r.rtr/rcsV F . Trcs(eZdZdfdd ZdddZZS) rư>cs&ttt||_||_dSNrbrcrrr*rrrrsrirrtr.r/rcFs  z%GraniteMoeHybridRMSNormGated.__init__NcCsj|j}|tj}|dur|tj|tj}|djddd}|t ||j }|j ||SNr&r%T)Zkeepdim) rKrTr*rSrrQrpowmeanrsqrtrr)rsr8r input_dtypevariancer.r.r/rKs z$GraniteMoeHybridRMSNormGated.forwardrr)rgrrrcrrr.r.rtr/rEsrcs<eZdZdZdeffdd ZdejdejfddZZ S) GraniteMoeHybridMLPz~ MLP layer for shared experts Args: config: Configuration object with model hyperparameters. r^cs^tt||j|_|j|_t|j|_t j |j|jddd|_ t j |j|jdd|_ dS)Nr&Fr`) rbrrcri input_sizeZshared_intermediate_sizerrrrrm input_linear output_linearrsr^rtr.r/rc`s  zGraniteMoeHybridMLP.__init__r8r:cCs<||}|jddd}||d|d}||}|S)Nr&r%r'rr)rchunkrr)rsr8chunked_hidden_statesr.r.r/ris  zGraniteMoeHybridMLP.forward) rgrrrrrcr*rrrr.r.rtr/rWs rcs.eZdZdfdd ZddZddZZS) GraniteMoeHybridRMSNormrcs&ttt||_||_dS)zF GraniteMoeHybridRMSNorm is equivalent to T5LayerNorm Nrrrtr.r/rcrs  z GraniteMoeHybridRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||Sr) rKrTr*rSrr r rr)rsr8r r r.r.r/rzs  zGraniteMoeHybridRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tuplerr)rrsr.r.r/ extra_reprsz"GraniteMoeHybridRMSNorm.extra_reprr )rgrrrcrrrr.r.rtr/rqsrcs6eZdZdedededdffdd Zdd ZZS) GraniteMoeHybridParallelExperts num_expertsrrr:Ncs6ttt||||_||_||_||_ dS)a Initialize the GraniteMoeHybridParallelExperts module. The experts weights are stored in [num_experts, output_size, input_size] format. Such that it's compatible with many MoE libraries, such as [Megablock](https://github.com/databricks/megablocks) and [ScatterMoE](https://github.com/shawntan/scattermoe), as well as the [MoE kernel](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_moe.py) used in vllm. Args: num_experts (int): Number of experts. input_size (int): Size of the input. output_size (int): Size of the output. N) rbrcrrr*emptyrrrr)rsrrrrtr.r/rcs  z(GraniteMoeHybridParallelExperts.__init__cCsP|j|dd}g}t|jD]}|t|||j|qtj|dd}|S)a Forward pass of the GraniteMoeHybridParallelExperts module. Args: inputs (Tensor): Input tensor. expert_size: Expert size information. Returns: Tensor: Output tensor. rr') rrrrFZlinearrr*r+)rsZinputs expert_sizeZ input_listZ output_listrresultsr.r.r/rs  z'GraniteMoeHybridParallelExperts.forwardrgrrrrcrrr.r.rtr/rsrcs2eZdZdededeffdd ZddZZS)GraniteMoeHybridTopKGatingrrtop_kcs2t||_||_||_tj||dd|_dS)a Initialize the top-k gating mechanism. Args: input_size (`int`): Size of the input. num_experts (`int`): Number of experts. top_k (`int`): Number of top experts to select. Fr`N)rbrcrrr!rrmlayer)rsrrr!rtr.r/rcs z#GraniteMoeHybridTopKGating.__init__cCs||}|j|jdd\}}tj|dd|}tj|d|j g|j |j d}| d|d}| d}|}|} | d\} } | j|jdd} |}|| } | | | ||fS)Nrr'rrKrtrunc)Z rounding_mode)r"rtopkr!r*rRZtype_asrr|rrKrZscatterlongrtolistflattensortdiv)rsr8logitsZ top_k_logitsZ top_k_indicesZ top_k_gatesrZgatesrZ top_k_expertsrZindex_sorted_experts batch_index batch_gatesr.r.r/rsz"GraniteMoeHybridTopKGating.forwardrr.r.rtr/r sr cs.eZdZdZdeffdd ZddZZS)GraniteMoeHybridMoEz A Sparsely gated mixture of experts layer with 1-layer Feed-Forward networks as experts. Args: config: Configuration object with model hyperparameters. r^csptt||j|_|j|_t|j|_t |j |j|jd|_ t |j |j|j|_ t |j|j |jd|_dS)Nr&)rrr!)rbr.rcrirrrrrrnum_local_expertsrrr num_experts_per_tokrouterrrtr.r/rcs   zGraniteMoeHybridMoE.__init__cCs|\}}}|d|}||\}}}}} ||} || |} | jddd} || d| d} || |} | |dddf} tj|||j f| j | j d}| d|| }| |||j }|| fS)a Forward pass of the mixture of experts layer. Args: layer_input (Tensor): Input tensor. Returns: Tensor: Output tensor. Tensor: Router logits. r%r&r'rrNr#)r|r<r1rrrrr*rrrKrZ index_addr})rsZ layer_inputrlengthZemb_sizerr,r-r router_logitsZ expert_inputsr8rZexpert_outputsrZ layer_outputr.r.r/rs   zGraniteMoeHybridMoE.forward)rgrrrrrcrrr.r.rtr/r.sr.cseZdZdedeffdd Z       ddejdeejd ee d ee d ee d eej d ee dee ejejfde ej ee ej ej fffddZZS)GraniteMoeHybridDecoderLayerr^r_cst|j|_d|_t||_t|j|jd|_t|j|jd|_ |j |_ t ||_ d|_ |j|dkr=t|||_ nt|||_|j||_dS)Nrr)rbrcri self_attnr.block_sparse_moerrinput_layernormpost_attention_layernormresidual_multiplierr shared_mlprrrr] layer_typerrrtr.r/rcs    z%GraniteMoeHybridDecoderLayer.__init__NFr8rGrur{rvrwoutput_router_logitsrxr:c  Ks|} ||}|jdur|j||||d}d} n|jd|||||||d| \}} } | ||j}|} ||}||\} }| ||}| ||j}|f}|rX|| f7}|r_||f7}|rf||f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`Tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model N)r8rwrrG)r8rGrur{rvrwrxr.)r7rr5r9r8r6r:)rsr8rGrur{rvrwr<rxrVZresidualZself_attn_weightsrZmoe_hidden_statesr3outputsr.r.r/r2sF%        z$GraniteMoeHybridDecoderLayer.forward)NNFFNFN)rgrrrrrcr*rrr rrr FloatTensorrrr.r.rtr/r4s8  r4c@sDeZdZeZdZdZdgZdgZdZ dZ dZ dZ dZ dZddZdS) GraniteMoeHybridPreTrainedModelmodelTr4past_key_valuesFcCsZt|tjr|jjjd|jjd|jdur|jj n=t|tj r=|jjjd|jjd|j dur<|jj|j  nt|t rJ|jj dnt|trZ|jjjd|jjdt|tjrz|jjjd|jjd|jdurx|jj dSdSt|tr|jj dttd|jd|j_|jj ddSt|tr|jj ddSdS)NrB)r Zstdg?r) isinstancerrmrdataZnormal_r^Zinitializer_rangeraZzero_ Embedding padding_idxrZfill_rrrrr*rrrkrrr)rsrCr.r.r/ _init_weightss4           z-GraniteMoeHybridPreTrainedModel._init_weightsN)rgrrrZ config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attn_2Z_supports_sdpaZ_supports_cache_classZ_supports_quantized_cacheZ_supports_static_cacheZ _is_statefulrFr.r.r.r/r?s r?cs8eZdZddeffdd ZeeddZZ S)GraniteMoeHybridRotaryEmbeddingNr^cstt|dr|jdur|jd|jd|_nd|_|j|_|j|_||_ t |j|_ | |j |\}|_ |j d|dd|j|_dS)N rope_scaling rope_typerdefaultinv_freqF) persistent)rbrchasattrrHrrImax_position_embeddingsZmax_seq_len_cachedZoriginal_max_seq_lenr^rZ rope_init_fnattention_scalingZregister_bufferrKZoriginal_inv_freq)rsr^rrKrtr.r/rcs   z(GraniteMoeHybridRotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t j |dd+|| dd}t j ||fdd }| |j}||j} Wdn1smwY|j|jd | j|jd fS) Nrr%rZmpscpuF) device_typeenabledr&r'r)rKrr;r)rTrrBrstrr*ZautocastrPr+r4rOr5rK) rsr,r6Zinv_freq_expandedZposition_ids_expandedrQZfreqsZembr4r5r.r.r/rs0&z'GraniteMoeHybridRotaryEmbedding.forwardr) rgrrrrcr*Zno_gradrrrr.r.rtr/rGs rGcs8eZdZdeffdd ZddZddZee           d$d e j d e e j d e e j d e e eee jfd e e jde ede ede ede ede ede e j de eeffddZ d%d e e j dfde j de j d edef ddZed e j dedede jde j def d d!Zd"d#ZZS)&GraniteMoeHybridModelr^cstj|_j|_tjj|j|_t fddt j D|_ t jjd|_d|_j|_j|_j|_|j|j|_j|_j|_j|_|jdkr]tnd|_|dS)Ncsg|]}t|qSr.)r4)rr_r^r.r/rz2GraniteMoeHybridModel.__init__..rFZrope)rbrcZ pad_token_idrE vocab_sizerrDri embed_tokensZ ModuleListrrlayersrrrgradient_checkpointingembedding_multiplierrjrkr@rNZ rope_thetaZposition_embedding_typerG rotary_emb post_initrrtrUr/rcs$  zGraniteMoeHybridModel.__init__cC|jSrrXrr.r.r/get_input_embeddingsz*GraniteMoeHybridModel.get_input_embeddingscC ||_dSrr_rsrFr.r.r/set_input_embeddings z*GraniteMoeHybridModel.set_input_embeddingsN input_idsrGr6rA inputs_embedsrvr{output_hidden_statesr< return_dictrwr:c  Cs@|dur|n|jj}|dur|n|jj}|dur|n|jj}| dur$| n|jj} |du|duAr4td|jrC|jrC|rCt dd}|durL| |}||j }|r\|dur\t d| durx|durh| nd} t j| | |jd|jd} |dur| d}|||| ||} ||| }|}d}|jdur|||}|rdnd}|rdnd}| rdnd}d}|jD]L}|jd kr|n| }|r||f7}||||||| | |d }|d}|r||rd nd}|r|ddur||df7}| r|d dur||d f7}q||}|r||f7}|r|nd}t|||||d S)Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FzGraniteMoeHybrid requires an initialized `HybridMambaAttentionDynamicCache` to return a cache. Because one was not provided, no cache will be returned.rrrr.r)rGrur{rvrwr<rxr&r%)Zlast_hidden_staterAr8 attentionsr3)r^r{rhrvuse_return_dictrlrZrMrdrerXr[get_seq_lengthr*rr)rr1_update_causal_mask_update_mamba_maskr\rYr;rr)rsrfrGr6rArgrvr{rhr<rirwpast_seen_tokensrZ mamba_maskr8rxZall_hidden_statesZall_self_attnsZall_router_logitsZnext_decoder_cacheZ decoder_layerZ layer_maskZ layer_outputsZ next_cacher.r.r/rs                 zGraniteMoeHybridModel.forwardFr#rc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) NZflash_attention_2rBZflex_attentionrFrz)rgZpast_key_values_lengthZ is_trainingrr%)sequence_length target_lengthrKrwr)rZxpuZnpu)r^ranyrBr*rr$rlZis_compileabler Z_ignore_causal_mask_sdparMrKr)Zget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrrfinfominZ_unmask_unattended) rsrGrrwrAr{roZusing_compilable_cacherKrqrrrZ min_dtyper.r.r/rmjsT           z)GraniteMoeHybridModel._update_causal_maskrqrrrKrc KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr)Z fill_valuerKrrrrr%r)r(r*rurvfullrZtriurr<r;cloner)rTr) rGrqrrrKrwrrVrZrwZ mask_lengthZ padding_maskr.r.r/rts,  $ 6  zKGraniteMoeHybridModel._prepare_4d_causal_attention_mask_with_cache_positioncCs.|}|ddks|durt|dkrd}|S)zv No need for zeroing states when 1. Cached forward 2. Attending to all inputs rNr)r*all)rsrGrwrpr.r.r/rns"z(GraniteMoeHybridModel._update_mamba_mask) NNNNNNNNNNN)F)rgrrrrcr`rdrrr*rrrrr rr>rrrrrm staticmethodrrKrtrnrr.r.rtr/rTs     |  D 6rTr& gate_logitsrcs|dus t|ts dSt|tr#|djtjfdd|Ddd}tjjj|dd}tj||dd\}}tjj ||}|durStj | dd} tj |dd} ng|j \} } |j d| | } |dddddddf | | | ||fd||}tj| |ddtj|dd} |ddddddf | | | |fd|}tj||ddtj|dd} t| | d}||S)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://arxiv.org/abs/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrcsg|]}|qSr.)rT)rZ layer_gateZcompute_devicer.r/rrVz,load_balancing_loss_func..r'r%)rBrrr*r+rrQrRr%Zone_hotr rr)r;r<rTrr1)r|rr!rGZconcatenated_gate_logitsZrouting_weightsrZselected_expertsZ expert_maskZtokens_per_expertZrouter_prob_per_expertrrqrZexpert_attention_maskZ router_per_expert_attention_maskZ overall_lossr.r}r/load_balancing_loss_funcs>    r~c sFeZdZdgZdeffdd ZddZddZd d Zd d Z d dZ ddZ e             d*de ejde ejde ejde eeeejfde ejde ejde ede ede ede ede ede ejdeeejfd eeeffd!d"Zed#d$Z      %d+d&d'Zd efd(d)ZZS),GraniteMoeHybridForCausalLMzlm_head.weightr^csXt|t||_|j|_tj|j|jdd|_|j |_ |j |_ |j |_ | dS)NFr`)rbrcrTr@rWrrmrilm_headrouter_aux_loss_coefr/rr0r]rrtr.r/rcFs   z$GraniteMoeHybridForCausalLM.__init__cCs|jjSrr@rXrr.r.r/r`Ssz0GraniteMoeHybridForCausalLM.get_input_embeddingscCs ||j_dSrrrcr.r.r/rdVs z0GraniteMoeHybridForCausalLM.set_input_embeddingscCr^rrrr.r.r/get_output_embeddingsYraz1GraniteMoeHybridForCausalLM.get_output_embeddingscCrbrr)rsZnew_embeddingsr.r.r/set_output_embeddings\rez1GraniteMoeHybridForCausalLM.set_output_embeddingscCrbrr@)rsdecoderr.r.r/ set_decoder_rez'GraniteMoeHybridForCausalLM.set_decodercCr^rrrr.r.r/ get_decoderbraz'GraniteMoeHybridForCausalLM.get_decoderNrrfrGr6rArglabelsrvr{rhr<rirwlogits_to_keepr:c Ks|dur|n|jj}| dur| n|jj} | dur| n|jj} | dur$| n|jj} |j|||||||| | | | d }|d}t| trGt| dn| }| |dd|ddf}||jj }d}|duru| }|j ||fd|jj i|}d}| rt| r|jn|d|j|j|}|dur||j||j7}| s|f|dd}| r|f|}|dur|f|S|St||||j|j|j|jdS)ax labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, GraniteMoeHybridForCausalLM >>> model = GraniteMoeHybridForCausalLM.from_pretrained("ibm/PowerMoE-3b") >>> tokenizer = AutoTokenizer.from_pretrained("ibm/PowerMoE-3b") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rfrGr6rArgrvr{rhr<rirwrrWr%r)lossaux_lossr+rAr8rjr3)r^r{r<rhrkr@rBrslicerZlogits_scalingrZ loss_functionrWr~r3rr0rrTrrrAr8rj)rsrfrGr6rArgrrvr{rhr<rirwrrVr=r8Z slice_indicesr+rroutputr.r.r/rest(  z#GraniteMoeHybridForCausalLM.forwardcs.d}|D]}|tfdd|Df7}q|S)Nr.c3s$|] }|d|jVqdS)rN)Z index_selectrTr)rZ past_statebeam_idxr.r/ s"z=GraniteMoeHybridForCausalLM._reorder_cache..)r)rArZreordered_pastZ layer_pastr.rr/_reorder_caches z*GraniteMoeHybridForCausalLM._reorder_cacheTc Ks|du} | s5|dus|d|jdkr"|dd|jd df}n!|jd|jdkr4|dd|f}nt|j|jd|j|jd}|durl|durl|dd}||dkd| sl|dd|jd df}|durw| rwd|i} nd|i} | |||||d| S)Nr%rrrrgrf)r6rArvrGrw) r)rr^rKrr&rZ masked_fill_rUr~) rsrfrArGrgrwr6rvrVZ empty_past_kvZ model_inputsr.r.r/prepare_inputs_for_generations8     z9GraniteMoeHybridForCausalLM.prepare_inputs_for_generationcCsdS)aG Function overwritten as this class uses its own `HybridMambaAttentionDynamicCache` and do not need to initialize the Cache in advance in order to save memory (because no back and forth `to_legacy_cache` and `from_legacy_cache` will be performed for `HybridMambaAttentionDynamicCache`). Fr.rr.r.r/_supports_default_dynamic_cachesz;GraniteMoeHybridForCausalLM._supports_default_dynamic_cache) NNNNNNNNNNNNr)NNNNNT)rgrrZ_tied_weights_keysrrcr`rdrrrrrrr*rrrr rr>rrrrrr{rrrrr.r.rtr/rCs|         l   9r)rrTr?)Nr)rB)Nr&N)Ztypingrrrrrr*Ztorch.nn.functionalrrQrZ(transformers.models.jamba.modeling_jambamodelsZjambaZmodeling_jambaZtransformers.activationsrZ cache_utilsr Z generationr Zmodeling_attn_mask_utilsr Zmodeling_layersr Zmodeling_outputsrrrZmodeling_rope_utilsrrZmodeling_utilsrrutilsrrrrZutils.import_utilsrrZconfiguration_granitemoehybridrZ+mamba_ssm.ops.triton.selective_state_updaterZ!mamba_ssm.ops.triton.ssd_combinedrrZ causal_conv1dr r!Z!torch.nn.attention.flex_attentionr#Zintegrations.flex_attentionr$Z get_loggerrgrdr0r7rrrAModulerr\r]rrrrrzrrrrrrrr r.r4r?rGrTr~r__all__r.r.r.r/s             ]9  e-0<i'"#  R \