o Zhj@sdZddlZddlmZmZmZmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZmZmZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZmZmZdd l m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(m)Z)ddl*m+Z+m,Z,ddl-m.Z.e,rddl/m0Z0m1Z1ddl2m3Z3nd\Z3Z1Z0e+rddl4m5Z5m6Z6nd\Z6Z5e7e3e1e5e6e0fZ8e)9e:Z;Gddde j<Z=e&>e=de j?de@de j?fddZAGd d!d!eZB "dCd#e jd?Gd@dAdAeKZNgdBZOdS)DzPyTorch Zamba model.N)AnyCallableDictListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)ALL_LAYERNORM_LAYERS)auto_docstringlogging)is_causal_conv1d_availableis_mamba_ssm_available) ZambaConfig)mamba_inner_fnselective_scan_fn)selective_state_update)NNN)causal_conv1d_fncausal_conv1d_updateNNcs.eZdZdfdd ZddZddZZS) ZambaRMSNormư>cs&ttt||_||_dS)z; ZambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__W/var/www/auris/lib/python3.10/site-packages/transformers/models/zamba/modeling_zamba.pyr*As  zZambaRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)NT)Zkeepdim) dtypetor,float32powmeanZrsqrtr/r.)r0 hidden_statesZ input_dtypeZvariancer5r5r6forwardIs  zZambaRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler.shaper/r0r5r5r6 extra_reprPszZambaRMSNorm.extra_repr)r()__name__ __module__ __qualname__r*r?rC __classcell__r5r5r3r6r'@sr'r>n_repreturncCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rAexpandreshape)r>rHbatchnum_key_value_headsslenhead_dimr5r5r6 repeat_kvXs 0rPc @seZdZdZejdfddZ ddejdejdede e e e fd e ejejff d d Zd ejfd dZdde ed efddZd e e eje ejffddZedde e e ejd dfddZdS)ZambaHybridDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. Nc s||_|j|_d|_|j|j|_|j|_|j|_ |j |_ g|_ g|_ g|_ i|_i|_i|_t|jD];}|j tj|j|j |dg7_ |j |j|j |jf}|j tj||dg7_ |j|dkro|j |q4fddt|jD|_fddt|jD|_dS)NFdevicer9hybridc g|] }tjggdqSrSr,Ztensor.0_ batch_sizerSr5r6  z4ZambaHybridDynamicCache.__init__..crUrVrXrYr\r5r6r^r_)r9layers_block_typehas_previous_state mamba_expandr1intermediate_size mamba_d_statessm_state_size mamba_d_convconv_kernel_size n_mamba_heads conv_states ssm_statestransformer_layersZ_modules _parameters_buffersrangenum_hidden_layersr,zerosappend key_cache value_cache)r0configr]r9rSiZ cache_shaper5r\r6r*rs8   z ZambaHybridDynamicCache.__init__ key_states value_states layer_idx cache_kwargsrIcCsz|j|jddkr||j|<||j|<ntj|j||gdd|j|<tj|j||gdd|j|<|j||j|fS)Nr8rr7dim)rrrArsr,cat)r0rvrwrxryr5r5r6updates   zZambaHybridDynamicCache.updatebeam_idxcCstt|jD]V}|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<qdS)zDReorders the cache for beam search, given the selected beam indices.rN) rnlenrrrSZ index_selectr:rsrirj)r0r~rxrSr5r5r6 reorder_caches     z%ZambaHybridDynamicCache.reorder_cachercCs:||jvr |jdn|}t|j|krdS|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.r)rkrrrrA)r0rxr5r5r6get_seq_lengthsz&ZambaHybridDynamicCache.get_seq_lengthcCtdNz@ZambaHybridDynamicCache does not have a legacy cache equivalent.NotImplementedErrorrBr5r5r6to_legacy_cachez'ZambaHybridDynamicCache.to_legacy_cachepast_key_valuesrcCrrr)clsrr5r5r6from_legacy_cachesz)ZambaHybridDynamicCache.from_legacy_cacheN)r)rDrErF__doc__r,Zfloat16r*Tensorintrrstrrrr} LongTensorrrr classmethod FloatTensorrr5r5r5r6rQds( % "(rQmodulequerykeyvalueattention_maskscalingdropoutc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr7r rr8)r{r9)ptrainingr)rPnum_key_value_groupsr,matmul transposerAr functionalZsoftmaxr;r:r9rr contiguous) rrrrrrrkwargsrvrw attn_weights causal_mask attn_outputr5r5r6eager_attention_forwards & rcs|eZdZdZdedeffdd Z ddejdede ejd e e d e e d e eje eje e ejff d d ZZS)ZambaAttentiona9 Multi-headed attention from 'Attention Is All You Need' paper. Modified to use sliding window attention: Longformer and "Generating Long Sequences with Sparse Transformers". Adapted from transformers.models.mistral.modeling_mistral.MistralAttention: The input dimension here is attention_hidden_size = 2 * hidden_size, and head_dim = attention_hidden_size // num_heads. The extra factor of 2 comes from the input being the concatenation of original_hidden_states with the output of the previous (mamba) layer (see fig. 2 in https://arxiv.org/pdf/2405.16712). Additionally, replaced attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim) with attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim/2) rtrxcst||_||_|j|_|j|_|j|j|_ |j |_ |jdd|_ d|_ |j |_ tj|j|j|jdd|_tj|j|j|jdd|_tj|j|j|jdd|_tj|j|j|jdd|_dS)Nr7TFbias)r)r*rtrxattention_hidden_sizeZattention_head_dimrOZnum_attention_headsrMrZmax_position_embeddingsrZ is_causalattention_dropoutr Linearq_projk_projv_projr1o_projr0rtrxr3r5r6r*s  zZambaAttention.__init__Nr>rpast_key_valuerrIcKs|jdd}g|d|jR}|||dd}|||dd} |||dd} |durB|| | |\} } t} |j j dkrb|j j dkr\| ddr\t dnt|j j } | ||| | |f|jsnd n|j|jd |\} } | jg|dR} || } | | fS) Nr8rr7eagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)rr)rArOrviewrrrr}rrt_attn_implementationgetlogger warning_oncerrrrrKrr)r0r>rxrrrZ input_shapeZ hidden_shapeZ query_statesrvrwZattention_interfacerrr5r5r6r?s:    zZambaAttention.forwardr)rDrErFrr rr*r,rrrQrrrr?rGr5r5r3r6rs" rcs^eZdZdZdeffdd Z ddejdefdd Z ddefd d Z ddefd d Z Z S)ZambaMambaMixeruE Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) This module differs from `transformers.models.mamba.modeling_mamba.MambaMixer` in two ways: - Added multi-head: the output of `self.in_proj` is split into `self.n_mamba_heads` heads, and each head undergoes an independent forward pass, identical to the original `MambaMixer`, up until the pre-activations of `self.out_proj`. The pre-activations, coming from different mamba heads, are then concatenated and fed into `self.out_proj`. rtcst||_||_|j|_|j|_|j|_|j |j|_ |j |_ |j |_ |j |j |_|j|_|j|_tj|j |j |j|j|j |jdd|_|j|_t|j|_|j|_tj|j|j d|jd|_tt |j |j |jd|j|_!tt |j |j|j dd|j d|_"tt |j |j|_#tj$d|jdtj%ddddf}|&|j d'}tt(|)|j |jd|_*tt+|j |j|_,tj|j |j|jd|_-t.st/0ddSdS) Nr)Z in_channelsZ out_channelsrZ kernel_sizegroupspaddingr7rg?r9r8apThe fast path is not available because on of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)1r)r*rtrxr1rdrerfrgrbrc mamba_dt_ranktime_step_rankrhmamba_head_dimZmamba_conv_bias use_conv_biasZmamba_proj_biasZuse_biasr Conv1dconv1dZhidden_mamba_act activationractZuse_mamba_kernelsuse_fast_kernelsrin_projr+r,rp x_proj_weightdt_proj_weight dt_proj_biasaranger;rJrlogrKA_logr-Dout_projis_fast_path_availablerr)r0rtrxAr3r5r6r*5sb   $ zZambaMambaMixer.__init__Nr> cache_paramsc Cs|j\}}}|duo|jo|dk}||dd}||dd|jddd\}} |d}| d} | ||j d|dd} |j j |j j d|j j d} |rnt |d|j|j| |j j|j}|d}nK|durt|dks||d}|durtj||j|jddf} |j|j| t|| |j j|jd}|durt|dks||d}|d|j |j|dd}|jdddddddf|dd} tj| |j|j|jgdd\} }}|j dddf| dd}t!|j"# }|j$dur|j$#nd}tj%|d|f|j&|j'd}|rtt(|j D]K}t)|j*|jdd|f||d df||d df||||dddf||dddf|j+|| |d df||d d d}tj,||fdd}q'nntj%|d|j|jf|j&|j'd}t(|j D]E}t-||||||||dd||dd|j+|#| |||d d d \}}tj,||fdd}tj,||dfdd}q|dur|dur|j*|j||.|dd}|S) Nrr7r8rzr)rrrR.T)Z dt_softplus)Zdelta_softplusZreturn_last_state)/rArarrrchunksqueezerrKrhrr.sizer%rirxrr unsqueezer,allr rpadrgcopy_r$rrsplitrrerexprfloatremptyrSr9rnr#rjrr|r"r)r0r>rrr]seq_lenr[Zuse_precomputed_statesprojected_statesgateZ conv_weightsrissm_parameters time_stepBCdiscrete_time_steprZtime_proj_bias scan_outputsnZ scan_outputs_ ssm_stateZ ssm_state_contextualized_statesr5r5r6cuda_kernels_forwardts  $  *         z$ZambaMambaMixer.cuda_kernels_forwardc Cs|j\}}}|j}||dd}||dd|jddd\} } | d} | d} | ||j d|dd} t |t } | r+|j |j jd|kr+|jrZ|j |j } n|j |j } | | j} |jr|dkr|j|j jd|kr|j|j } tj| ddd} | dddddf| dddddf<| |j|j <tj| |jjdddddfdd} |jr| |jj7} || |d} n|durt|dks| |dd| jd dfd} tj | |j!| jddf} | |j|j <||| dd|f} |dur*t|dks*| |dd| jd dfd} nFtj"||j |j#|j$f| j|d} |durOt|dksO| |d} ||| dd|f} |durqt|dksq| |d} | d|j |j#|dd} |j%dddddddf| dd }tj&||j'|j$|j$gdd\}}}|j(dddf|dd |j)ddddddf}tj*|}t+|j,- }t+|ddddddddf|dddddddddf}|dddddddddf|dddddddddf-}|| dddddddddf-}g}t.|D]\}|dddddd|ddfdd| |dddddd|ddfdd} t/| dd||dddd|ddfd}|0|dddddddfq2tj1|dd}|| |j2ddddddf}||| }| r| |j |j <|3|dd|d|dd}|S) Nrr7r8rzr)Zshiftsdims.rRr)4rAr9rrrrrrrKrh isinstancerQrjrxrcloner:rSrarir,Zrollsumrr.rrrrrr rrrgrprrerrrrrZsoftplusrrrrnrrqstackrr)r0Z input_statesrrr]rr[r9rr>r use_cacherZ conv_staterrrrrrZ discrete_AZ discrete_BZdeltaB_urruZ scan_outputrr5r5r6 slow_forwards      ( ( & &*  FH* X8&" zZambaMambaMixer.slow_forwardcCs@|jrtr d|jjjvrtd|j|||dS|j|||dS)NcudazFast Mamba kernels are not available. Make sure to they are installed and that the mamba module is on a CUDA device. lease run 'pip install causal-conv1d>=1.2.0' and 'pip install mamba-ssm', or set use_mamba_kernels=False in the model's config.)r)rrrrStype ValueErrorrr)r0r>rrr5r5r6r?2szZambaMambaMixer.forwardr&) rDrErFrr r*r,rrQrrr?rGr5r5r3r6r(s @ a]rcs$eZdZfddZddZZS)ZambaMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFr) r)r*rtr1rcr r gate_projup_proj down_projrZ hidden_actact_fnr0rtr3r5r6r*@s zZambaMLP.__init__cCs$||||||}|Sr)rrrr)r0xrr5r5r6r?Js zZambaMLP.forward)rDrErFr*r?rGr5r5r3r6r?s  rcseZdZddedeeffdd Z    ddejdejded eejd ee d ee d ee d e e de ejee ejejfffddZZS)ZambaAttentionDecoderLayerNrtrxcsHtt|||_t||_t|j|jd|_ t|j |jd|_ dS)Nr2) r)r*r self_attnr feed_forwardr'r rms_norm_epsinput_layernormr1pre_ff_layernormrr3r5r6r*Ps   z#ZambaAttentionDecoderLayer.__init__Fr>original_hidden_statesrrrrrrIc Ksjtj||gdd}||}|jd||||||d|\}} ||}||}|f} |r3| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): output of previous Mamba layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output of shape `(batch, seq_len, embed_dim)`. This is concatenated with `hidden_states` (which is the output of the previous (mamba) layer). The concatenated tensor is then used as input of the pre-attention RMSNorm (see fig. 2 in https://arxiv.org/pdf/2405.16712). layer_idx (`int`): layer_idx in the forward pass. Used to distinguish Zamba's tied transformer layers. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. r8rz)r>rxrrrrNr5)r,Z concatenaterrrr) r0r>rrxrrrrrself_attn_weightsoutputsr5r5r6r?Xs$   z"ZambaAttentionDecoderLayer.forwardrNNFF)rDrErFr rrr*r,rrQboolrrrrr?rGr5r5r3r6rOs2   rcseZdZdedeffdd Z         ddejdeejdeed eejd eejd ee d ee d ee deej deejde ej ee ej ej fffddZZS)ZambaMambaDecoderLayerrtrxcs4tt||d|_t|j|jd|_||_dS)N)rtrxr) r)r*rmambar'r1rrrxrr3r5r6r*s  zZambaMambaDecoderLayer.__init__NFr>rrrrrrcache_positiontransformer_hidden_statesrIc Ksd|} | dur || n|}||}|j|||d}d} | |}|f}|r)|| f7}|r0||f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. N)r>rr)rr )r0r>rrxrrrrrrrrZresidualrr r5r5r6r?s"   zZambaMambaDecoderLayer.forward) NNNNNFFNN)rDrErFr rr*r,rrrQr rrrr?rGr5r5r3r6r sD     r cseZdZdedejdeffdd Z        ddej d e ej d e e d e ej d e ej d e e de e de e de ejdeeje eejejfffddZZS)ZambaHybridLayer shared_transflinearr cs t||_||_||_dSr)r)r*rr mamba_decoder)r0rrr r3r5r6r*s  zZambaHybridLayer.__init__NFr>rrxrrrrrrrIc Csp|j|||||||| d} | d} |r| d} || } |j|| ||||| d} |r6| d| f| dd} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. )rrxrrrrrrr)rrrrrrr7N)rrr) r0r>rrxrrrrrr layer_outputsrrr5r5r6r?s4   zZambaHybridLayer.forward)NNNNNFFN)rDrErFrr rr r*r,rrrrQr rrrr?rGr5r5r3r6rs>    rc seZdZeZdZdZddgZdZdZ dZ dZ dZ ddZ ee  dd eejd eeeeeeffd ed effdd ZZS)ZambaPreTrainedModelmodelTrr rFcCs|jj}t|tjtjfr%|jjjd|d|j dur#|j j dSdSt|tj rF|jjjd|d|j durD|jj|j  dSdSt|t rT|jjddSt|tr|jjjd|d|jjd}tj|j| ||jj|jj|jj}tt|jj|t|jjt|jjt|jjj|jj d}|tt!|  }|j"j#|tj$d|j%dtj&ddddf}|'|j(d)}|j*j#t|+|j|j,d|j-jddSdS) Nr)r=stdg?r)minrrr8).rtZinitializer_rangerr rrr.dataZnormal_rZzero_ Embedding padding_idxr'Zfill_rrrinitZuniform_rrbr1rhr,rZrandmathrZ time_step_maxZ time_step_minclampZtime_step_floorexpm1rrrrer;rJrcrrrKrr)r0rrZ dt_init_stdrdtZinv_dtrr5r5r6 _init_weights&sD       $"z"ZambaPreTrainedModel._init_weightsN torch_dtype device_maphard_check_onlycheck_device_mapcs.tj|||||d}|s|jdkrd|_|S)z Overloads `PreTrainedModel._check_and_enable_flash_attn_2` so as to DISABLE Flash Attention 2 by default on Zamba models. Flash attention 2 is currently not supported in the HuggingFace implementation of Zamba v1. )r$r%flash_attention_2r)r)_check_and_enable_flash_attn_2r)rrtr"r#r$r%r3r5r6r'Fs  z3ZambaPreTrainedModel._check_and_enable_flash_attn_2r )rDrErFr Z config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attn_2Z_supports_sdpaZ_supports_cache_classZ _is_statefulr!rrr,r9rrrrr r'rGr5r5r3r6rs4 rcseZdZdZdeffdd ZddZddZe dd e e j d e e j d e e j d e e de e jde ede ede ede ede e j deeeffddZddZZS) ZambaModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`ZambaDecoderLayer`] Args: config: ZambaConfig rtc st||j|_|j|_t|j|j|j|_t |}g}g}|j |_ t |j D]2}|j |dkr=| t||dq*|j |dkr\| tj|jj|jjdd| t||dq*t|}t|}g}g|_t|j D]6\}}|dkrd|dgd} g|jfd d | D|_| t|t|t|qo| t|qot||_|j|_t|j|jd |_d|_|dS) Nr )rxrTFrzlayers..) z%shared_transf.self_attn.q_proj.weightz%shared_transf.self_attn.k_proj.weightz%shared_transf.self_attn.v_proj.weightz%shared_transf.self_attn.o_proj.weightz+shared_transf.feed_forward.gate_proj.weightz)shared_transf.feed_forward.up_proj.weightz+shared_transf.feed_forward.down_proj.weightz$shared_transf.input_layernorm.weightz%shared_transf.pre_ff_layernorm.weightcsg|]}|qSr5r5)rZrZ prefix_namer5r6r^sz'ZambaModel.__init__..r)r)r* pad_token_idr vocab_sizer rr1 embed_tokensrr`rnrorqr rrtiter_tied_weights_keys enumeraternextZ ModuleListlayersrr'rfinal_layernormgradient_checkpointing post_init) r0rtblockZ mamba_layersZ linear_layersrur2Zlayer_idZ layer_typeZ tied_keysr3r*r6r*hs>     zZambaModel.__init__cC|jSrr-rBr5r5r6get_input_embeddingszZambaModel.get_input_embeddingscC ||_dSrr8r0rr5r5r6set_input_embeddings zZambaModel.set_input_embeddingsN input_idsr position_idsr inputs_embedsrroutput_hidden_states return_dictrrIc  Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}| dur$| n|jj} |du|duAr4td|jrC|jrC|rCt dd}|durL| |}|} t |} |r^|dur^t d| durmt j | jd| jd} |durv| d}|||| } |rdnd}|rdnd}t|jD]A\}}|r|| f7}|jr|jr||j| | ||| |||| }n || | ||| |||| d }|d} |r|ddur||df7}q|| } |r|| f7}|r|jsd |_t| |r|nd||d }| r|S|S) NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either onezX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Fz{Zamba requires an initialized `ZambaHybridDynamicCache` to return a cache. None was provided, so no cache will be returned.rrWrr5)rrxrrrrrrT)Zlast_hidden_staterr> attentions)rtrrBruse_return_dictrr4rrrr-r,rrrArSr_update_causal_maskr0r2Z_gradient_checkpointing_func__call__r3rarZto_tuple)r0r?rr@rrArrrBrCrr>rrZall_hidden_statesZall_self_attnsrxlayerroutputr5r5r6r?s               zZambaModel.forwardc Csv|jjdkr|durd|vr|SdS|j|j}}t|j}|jd}|dd}tj||f|||d} |dkr@tj | dd} | tj ||d| ddk9} | ddddddf |jdddd} |dur| } |d kr|jd} | d d| fd|ddddddfd} | d d| f| || d d| f<|jjd kr|dur|jjd vrt| |} | S) Nr&rrr8)Z fill_valuer9rS)ZdiagonalrWrr7.r)rZxpuZnpu)rtrr9rSr,ZfinforrAfullZtriurrKrJrr{eqZ masked_fillrrZ_unmask_unattended) r0rZ input_tensorrr9rSZ min_dtypeZsequence_lengthZ target_lengthrZ mask_lengthZ padding_maskr5r5r6rF s0    *  4$   zZambaModel._update_causal_mask NNNNNNNNNN)rDrErFrr r*r9r=rrr,rrrQrr rrrr?rFrGr5r5r3r6r(_sP/    nr(cseZdZdeffdd ZddZddZdd Zd d Zd d Z ddZ e            d$de e jde e jde e jde ede e jde e jde ede ede ede ede e jdeee jfdeeeffdd Z      !d%d"d#ZZS)&ZambaForCausalLMrtcsPt|t||_dg|jj|_|j|_tj|j|jdd|_ | dS)Nzlm_head.weightFr) r)r*r(rr/r,r rr1lm_headr5rr3r5r6r*2s   zZambaForCausalLM.__init__cC|jjSrrr-rBr5r5r6r9<rz%ZambaForCausalLM.get_input_embeddingscC ||j_dSrrPr<r5r5r6r=? z%ZambaForCausalLM.set_input_embeddingscCr7rrNrBr5r5r6get_output_embeddingsBr:z&ZambaForCausalLM.get_output_embeddingscCr;rrS)r0Znew_embeddingsr5r5r6set_output_embeddingsEr>z&ZambaForCausalLM.set_output_embeddingscCr;rr)r0decoderr5r5r6 set_decoderHr>zZambaForCausalLM.set_decodercCr7rrVrBr5r5r6 get_decoderKr:zZambaForCausalLM.get_decoderNrr?rr@rrAlabelsrrrBrCrlogits_to_keeprIc  Ks|dur|n|jj}| dur| n|jj} | dur| n|jj} |j|||||||| | | d }|d}t| tr>> from transformers import AutoTokenizer, ZambaForCausalLM >>> model = ZambaForCausalLM.from_pretrained("Zyphra/Zamba-7B-v1") >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba-7B-v1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) r?rr@rrArrrBrrCrrlosslogitsrr>rD)rtrrBrErrrslicerNZ loss_functionr,rrr>rD)r0r?rr@rrArZrrrBrCrr[Z loss_kwargsr r>Z slice_indicesr^r]rIr5r5r6r?Ns@( zZambaForCausalLM.forwardTc Ks|du} | s5|dus|d|jdkr"|dd|jd df}n!|jd|jdkr4|dd|f}nt|j|jd|j|jd}|durl|durl|dd}||dkd| sl|dd|jd df}|durw| rwd|i} nd|i} | |||||jj |d| S)Nr8rr)r9rSrAr?)r@rrrr[r) rArQrtr9rSlongZcumsumZ masked_fill_rr}Znum_logits_to_keep) r0r?rrrArr@rrZ empty_past_kvZ model_inputsr5r5r6prepare_inputs_for_generations:     z.ZambaForCausalLM.prepare_inputs_for_generation) NNNNNNNNNNNr)NNNNNT)rDrErFr r*r9r=rTrUrXrYrrr,rrrQrr rrrrr?rarGr5r5r3r6rM1sn        TrMa The Zamba Model with a sequence classification head on top (linear layer). [`ZambaForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). )Z custom_introcseZdZfddZddZddZe          ddeej d eej d eej d ee e e ejfd eejd eej deedeedeedeede eeffddZZS)ZambaForSequenceClassificationcsJt||j|_t||_|jj|_tj|j|jdd|_ | dSr) r)r* num_labelsr(rr/r rr1scorer5rr3r5r6r*s    z'ZambaForSequenceClassification.__init__cCrOrrPrBr5r5r6r9rz3ZambaForSequenceClassification.get_input_embeddingscCrQrrPr<r5r5r6r=rRz3ZambaForSequenceClassification.set_input_embeddingsNr?rr@rrArZrrrBrCrIc  CsB| dur| n|jj} |j|||||||| | d } | d} || } |dur+|jd}n|jd}|jjdur>|dkr>td|jjdurGd}n1|durl||jjk| jt j }t j |jd| jt j d}|| d}n d}t |jjd| t j || jd |f}d}|dur|| j}|jjdur|jdkrd |j_n|jdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jdkr|||}n+|||}n%|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jd S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rr@rrArrrBrCrrz=Cannot handle batch sizes > 1 if no padding token is defined.r8rRz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rWZ regressionZsingle_label_classificationZmulti_label_classificationr\)rtrErrdrAr+rr:rSr,Zint32rZargmaxrrr4rDZ problem_typercr9r`rr rr rr rrr>rD)r0r?rr@rrArZrrrBrCZtransformer_outputsr>r^r]Zlast_non_pad_tokenZ non_pad_maskZ token_indicesZ pooled_logitsr]Zloss_fctrIr5r5r6r?sx         "       z&ZambaForSequenceClassification.forwardrL)rDrErFr*r9r=rrr,rrrrrrr rrr?rGr5r5r3r6rbsL      rb)rMrbr(r)r)Prrtypingrrrrrrrr,Ztorch.utils.checkpointr Ztorch.nnr r r Z activationsrZ cache_utilsrrZ generationrZmodeling_attn_mask_utilsrZmodeling_flash_attention_utilsrZmodeling_outputsrrrZmodeling_utilsrrZprocessing_utilsrZ pytorch_utilsrutilsrrZutils.import_utilsrrZconfiguration_zambar Z&mamba_ssm.ops.selective_scan_interfacer!r"Z+mamba_ssm.ops.triton.selective_state_updater#Z causal_conv1dr$r%rrZ get_loggerrDrModuler'rqrrrPrQrrrrrrr rrr(rMrb__all__r5r5r5r6s$             e L?DHDR, p