o Zh5@s<dZddlZddlmZmZmZmZddlZddlZddlm Z ddl m Z ddl m Z mZddlmZdd lmZdd lmZmZdd lmZmZmZmZdd lmZmZdd lmZddl m!Z!m"Z"m#Z#m$Z$ddl%m&Z&e#r}ddl'm(Z(ddl)m*Z*erddlm+Z+e$,e-Z.Gddde j/Z0ddZ1d9ddZ2Gddde j/Z3Gddde j/Z4dej5de6d ej5fd!d"Z7Gd#d$d$e j/Z8Gd%d&d&e8Z9Gd'd(d(e8Z:e8e9e:d)Z;Gd*d+d+e j/ZGd0d1d1e=eZ?e!d2d3Gd4d5d5e=Z@e!Gd6d7d7e=ZAgd8ZBdS):zPyTorch StableLM model.N)ListOptionalTupleUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)!flash_attn_supports_top_left_maskis_flash_attn_available)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPastTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)PreTrainedModel)auto_docstringcan_return_tupleis_torch_flex_attn_availablelogging)StableLmConfig) BlockMask)make_flex_block_causal_mask)_flash_attention_forwardcs8eZdZddeffdd ZeeddZZ S)StableLmRotaryEmbeddingNconfigcstt|dr|jdur|jd|jd|_nd|_|j|_|j|_||_ t |j|_ | |j |\}|_ |j d|dd|j|_dS)N rope_scaling rope_typetypedefaultinv_freqF) persistent)super__init__hasattrr!getr"Zmax_position_embeddingsZmax_seq_len_cachedZoriginal_max_seq_lenr rZ rope_init_fnattention_scalingZregister_bufferr%Zoriginal_inv_freq)selfr devicer% __class__]/var/www/auris/lib/python3.10/site-packages/transformers/models/stablelm/modeling_stablelm.pyr(=s   z StableLmRotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t j |dd+|| dd}t j ||fdd }| |j}||j} Wdn1smwY|j|jd | j|jd fS) NrrZmpscpuF) device_typeenableddim)dtype)r%floatexpandshapetor- isinstancer#strtorchZautocast transposecatcosr+sinr9) r,x position_idsZinv_freq_expandedZposition_ids_expandedr4ZfreqsZembrCrDr0r0r1forwardNs0&zStableLmRotaryEmbedding.forwardN) __name__ __module__ __qualname__rr(r@Zno_gradrrG __classcell__r0r0r.r1r<s rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr2r6r7)r<r@rB)rEx1Zx2r0r0r1 rotate_half_srNcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezerN)qkrCrDrFZ unsqueeze_dimZq_embedZk_embedr0r0r1apply_rotary_pos_embgs  rRcs$eZdZfddZddZZS) StableLmMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFbias) r'r(r hidden_sizeZintermediate_sizerLinear gate_projup_proj down_projrZ hidden_actact_fnr,r r.r0r1r(s zStableLmMLP.__init__cCs$||||||}|SrH)r[r\rYrZ)r,rEr[r0r0r1rGs zStableLmMLP.forward)rIrJrKr(rGrLr0r0r.r1rSs  rScs.eZdZdfdd ZdejfddZZS) StableLmLayerNormPerHeadh㈵>Fcs>t|_||_tfddt|jD|_dS)Ncsg|] }tjdqS))epsrV)r LayerNorm).0_rVr8r`r0r1 sz5StableLmLayerNormPerHead.__init__..)r'r(r8 num_headsr ModuleListrangenorms)r,r8rfr`rVr.rdr1r(s (z!StableLmLayerNormPerHead.__init__ hidden_statescCs0tj|ddd}tjddt|j|DddS)Nrr7cSsg|]\}}||qSr0r0)rbnormrjr0r0r1resz4StableLmLayerNormPerHead.forward..)r@splitrBzipri)r,rjZstates_per_headsr0r0r1rGs z StableLmLayerNormPerHead.forward)r_F)rIrJrKr(r@TensorrGrLr0r0r.r1r^sr^rjn_repreturncCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r<r;reshape)rjrobatchnum_key_value_headsslenhead_dimr0r0r1 repeat_kvs 0rvcseZdZdZddedeeffdd Z       ddej d eej d eej d ee d e d e deej dee ej ej fde ej eej ee ej ffddZZS)StableLmAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNr layer_idxcs~t||_||_|durtd|jjd|j|_|j |_ |j|j |_ |j |_ |j |j |_ |j|_t|j |j|_d|_|j |j |jkrZtd|jd|j dtj|j|j |j |jd|_tj|j|j |j |jd|_tj|j|j |j |jd|_tj|j|jdd|_|j|_|jrt|j |j |jd |_t|j |j |jd |_t |j!|_!t"|jd |_#dS) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).rUFr`r )$r'r(r rxlogger warning_oncer/rIrWZnum_attention_headsrfrursnum_key_value_groupsZ rope_thetaintZpartial_rotary_factor rotary_ndims is_causal ValueErrorrrXZ use_qkv_biasq_projk_projv_projo_proj qk_layernormr^layer_norm_eps q_layernorm k_layernormDropoutattention_dropoutr rotary_embr,r rxr.r0r1r(sB   zStableLmAttention.__init__Frjattention_maskrFpast_key_valueoutput_attentions use_cachecache_positionposition_embeddingsrpc Csn|\} } } ||} ||} ||}| | | |j|jdd} | | | |j|jdd} || | |j|jdd}|j rM| | } | | } |\}}| dd|j f| d|j df}}| dd|j f| d|j df}}t ||||\}}tj||fdd} tj||fdd} |dur|||j |d}|| ||j|\} }t| |j} t||j}t| | ddt|j}|dur|ddddddd| jdf}||7}tjj|tjdd | j}||}t||}|| |j| |jfkrtd | |j| |jfd ||dd }|!| | |j"}|#|}|s2d}|||fS) Nrr6.r2r7rDrCZpartial_rotation_sizerr)r9r8z `attn_output` should be of size z , but is )$sizerrrviewrfrurArsrrrrrRr@rBupdaterxrvr}matmulmathsqrtr<r functionalZsoftmaxZfloat32r=r9rr contiguousrqrWr)r,rjrrFrrrrrbszq_lenrc query_states key_states value_statesrCrD query_rot query_passkey_rotkey_pass cache_kwargs attn_weights causal_mask attn_outputr0r0r1rGs`        &    zStableLmAttention.forwardrHNNNFFNN)rIrJrK__doc__rrr~r(r@rn LongTensorr boolrrGrLr0r0r.r1rws:+  rwcseZdZ       ddejdeejdeejdeededed eejd ee ejejfd e ejeejee ejfffd d Z Z S)StableLmSdpaAttentionNFrjrrFrrrrrrpc  sl|rtdtj||||||||dS|\} } } ||} ||} ||}| | | |j |j  dd} | | | |j |j  dd} || | |j |j  dd}|j rb|| } || } |\}}| dd|jf| d|jdf}}| dd|jf| d|jdf}}t||||\}}tj||fdd} tj||fdd} |dur|||j|d}|| ||j|\} }t| |j} t||j}|}|dur|ddddddd| jd f}| jjd kr|dur| } | } |}|dur| dkrd nd }tjjj| | |||jr|j j!nd |d}| dd}|| | |j"}|#|}|d|fS)NaStableLmModel is using StableLmSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rjrrFrrrrrrr6.r2r7rrcudaTF)Z attn_maskZ dropout_pr)$r{r|r'rGrrrrrrfrurArsrrrrrRr@rBrrxrvr}r<r-r#rrrZscaled_dot_product_attentiontrainingrprWr)r,rjrrFrrrrrrrrcrrrrCrDrrrrrrrrr.r0r1rG1s~        &   zStableLmSdpaAttention.forwardr) rIrJrKr@rnrrr rrrGrLr0r0r.r1r0s6  rcseZdZdZfddZ       ddejdeejdeejd ee d e d e d eejd ee ejejfde ejeejee ejffddZ Z S)StableLmFlashAttention2aL StableLM flash attention module. This module inherits from `StableLmAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cstj|i|t|_dSrH)r'r(r _flash_attn_uses_top_left_mask)r,argskwargsr.r0r1r(s z StableLmFlashAttention2.__init__NFrjrrFrrrrrrpc  Ksd}|\} } } ||} ||}||}| | | |j|jdd} || | |j|jdd}|| | |j|jdd}|j rO| | } | |}|\}}| dd|j f| d|j df}}|dd|j f|d|j df}}t ||||\}}tj||fdd} tj||fdd}|dur|||j |d}||||j|\}}| dd} |dd}|dd}|jr|jjnd}t| |||| |||j|jd }|| | |j}||}|sd}|||fS) NFrr6.r2r7rr)rFdropoutZuse_top_left_maskr)rrrrrrfrurArsrrrrrRr@rBrrxrrrrrrrqrWrr)r,rjrrFrrrrrrrrrcrrrrCrDrrrrrZ dropout_raterrr0r0r1rGs`          zStableLmFlashAttention2.forwardr)rIrJrKrr(r@rnrrr rrrGrLr0r0r.r1rs:    r)eagersdpaflash_attention_2cseZdZdedeffdd Z       ddejdeejd eej d ee ejd ee d ee d eej dee ejejfde ej ee ej ej fffddZ ZS)StableLmDecoderLayerr rxcs~t|j|_|j|_t|j||d|_t||_t j |j|j d|_ d|_ |js6t j |j|j d|_ t |j|_dS)N)rxry)r'r(use_parallel_residualrWATTENTION_CLASSES_attn_implementation self_attnrSmlprrarinput_layernormpost_attention_layernormrhidden_dropoutrrr.r0r1r( s  zStableLmDecoderLayer.__init__NFrjrrFrrrrrrpc  Cs|} ||}|j||||||||d\} } } |jr+||} || } | | | }n| | } ||| } || } | | }|f}|rJ|| f7}|rQ|| f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`Tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. r)rrrrrr)r,rjrrFrrrrrZresidualZself_attn_outputZself_attn_weightsZpresent_key_valueZ mlp_outputoutputsr0r0r1rGs4$       zStableLmDecoderLayer.forwardr)rIrJrKrr~r(r@rnrrrr FloatTensorrGrLr0r0r.r1r s8   rc@s>eZdZeZdZdZdgZdZdZ dZ dZ dZ dZ ddZdS)StableLmPreTrainedModelmodelTrpast_key_valuescCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|tj rC|jjjd|d|j durA|jj|j  dSdSt|tj rX|jj d|jj dSdS)Nr)meanstdg?)r Zinitializer_ranger>rrXweightdataZnormal_rVZzero_ Embedding padding_idxraZfill_)r,modulerr0r0r1 _init_weightsss     z%StableLmPreTrainedModel._init_weightsN)rIrJrKrZ config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attn_2Z_supports_cache_classZ_supports_sdpaZ_supports_quantized_cacheZ_supports_static_cacherr0r0r0r1rfs rcseZdZdZdeffdd ZddZddZee d!d e e j d e e j d e e j d e ee jde e jde ede ede ede e j defddZ d"d ee j dfde j de j d edef ddZed e j dedede jde j def dd ZZS)# StableLmModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`StableLmDecoderLayer`] Args: config: StableLmConfig r cstj|_j|_tjj|j|_t fddt j D|_ tj jjd|_td|_j|_d|_|dS)Ncsg|]}t|qSr0)r)rbrxrzr0r1resz*StableLmModel.__init__..ryrzF)r'r( pad_token_idr vocab_sizerrrW embed_tokensrgrhZnum_hidden_layerslayersrarrkrrrgradient_checkpointing post_initr]r.rzr1r(s   zStableLmModel.__init__cC|jSrHrr,r0r0r1get_input_embeddingsz"StableLmModel.get_input_embeddingscC ||_dSrHrr,valuer0r0r1set_input_embeddings z"StableLmModel.set_input_embeddingsN input_idsrrFr inputs_embedsrroutput_hidden_statesrrpc  Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|du|duAr*td|jr9|jr9|r9tdd}d} |rVt |t sVd} |durLt }n t |}td|dur_| |}| dur{|durk|nd} tj| | |jd|jd} |dur| d}|||| ||} |} || |}|rd nd}|rd nd}d}|jD]A}|r|| f7}|jr|jr||j| | ||||| | }n || | ||||| |d }|d} |r||rd nd}|r||df7}q|| } |r|| f7}|r|nd}| r|}t| |||d S) Nz:You must specify exactly one of input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FTzWe detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)rrr-r0)rrFrrrrrr6)last_hidden_staterrj attentions)r rrrrrrr{r|r>r r Zfrom_legacy_cacherget_seq_lengthr@aranger<r-rO_update_causal_maskrrZ_gradient_checkpointing_func__call__rkZto_legacy_cacher)r,rrrFrrrrrrZreturn_legacy_cachepast_seen_tokensrrjrZall_hidden_statesZall_self_attnsZnext_decoder_cacheZ decoder_layerZ layer_outputsZ next_cacher0r0r1rGs                zStableLmModel.forwardFr input_tensorc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) NrrZflex_attentionrFr)rZpast_key_values_lengthZ is_trainingrr2)sequence_length target_lengthr9r batch_size)rZxpuZnpu)r ranyr>r@rnrrZis_compileabler Z_ignore_causal_mask_sdparr9r<Zget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionr-r#finfominZ_unmask_unattended) r,rrrrrrZusing_compilable_cacher9rrr min_dtyper0r0r1rsT           z!StableLmModel._update_causal_maskrrr9rc KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valuer9r-r)Zdiagonalrr2r)r8r@rrfullr-Ztriurrqr;cloner<r=Z masked_fill) rrrr9rrrrrZ mask_lengthZ padding_maskr0r0r1ras,  $ 6  zCStableLmModel._prepare_4d_causal_attention_mask_with_cache_position NNNNNNNNN)F)rIrJrKrrr(rrrrrr@rrnrrrrrGrr r staticmethodr~r9rrLr0r0r.r1rs~      DrcseZdZdgZfddZddZddZdd Zd d Zd d Z ddZ e e           d de ejde ejde ejde eejde ejde ejde ede ede ede ejdeeejfdefddZZS)!StableLmForCausalLMzlm_head.weightcs@t|t||_|j|_tj|j|jdd|_| dSrT) r'r(rrrrrXrWlm_headrr]r.r0r1r(s   zStableLmForCausalLM.__init__cC|jjSrHrrrr0r0r1rz(StableLmForCausalLM.get_input_embeddingscC ||j_dSrHrrr0r0r1r z(StableLmForCausalLM.set_input_embeddingscCrrHrrr0r0r1get_output_embeddingsrz)StableLmForCausalLM.get_output_embeddingscCrrHr)r,Znew_embeddingsr0r0r1set_output_embeddingsrz)StableLmForCausalLM.set_output_embeddingscCrrHr)r,decoderr0r0r1 set_decoderrzStableLmForCausalLM.set_decodercCrrHrrr0r0r1 get_decoderrzStableLmForCausalLM.get_decoderNrrrrFrrlabelsrrrrlogits_to_keeprpc  Ks|dur|n|jj}| dur| n|jj} |j|||||||| | d } | j}t| tr0t| dn| }||dd|ddf}d}|durT|j ||fd|jj i| }t ||| j | j | jdS)ui labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, StableLmForCausalLM >>> model = StableLmForCausalLM.from_pretrained("adept/persimmon-8b-base") >>> tokenizer = AutoTokenizer.from_pretrained("adept/persimmon-8b-base") >>> prompt = "human: Hey, what should I eat for dinner?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] 'human: Hey, what should I eat for dinner?\n\ncat: 🐱\n\nhuman: 😐\n\n' ```N) rrrFrrrrrrrlosslogitsrrjr)r rrrrr>r~slicer loss_functionrrrrjr)r,rrrFrrr rrrrr rrrjZ slice_indicesrrr0r0r1rGsD) zStableLmForCausalLM.forward) NNNNNNNNNNr)rIrJrKZ_tied_weights_keysr(rrrrr r rrrr@rrnrrrrr~rrGrLr0r0r.r1rs^       ra The StableLm transformer with a sequence classification head on top (linear layer). [`StableLmForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). )Z custom_introceZdZfddZddZddZee         ddee j d ee j d ee j d ee d ee j d ee j deedeedeedefddZZS)!StableLmForSequenceClassificationcs@t||j|_t||_tj|j|jdd|_| dSrT) r'r( num_labelsrrrrXrWscorerr]r.r0r1r("s   z*StableLmForSequenceClassification.__init__cCrrHrrr0r0r1r+rz6StableLmForSequenceClassification.get_input_embeddingscCrrHrrr0r0r1r.rz6StableLmForSequenceClassification.set_input_embeddingsNrrrFrrr rrrrpc  Cs(|j|||||||| d} | j} || } |dur|jd} n|jd} |jjdur2| dkr2td|jjdur;d}n1|dur`||jjk| jt j }t j |jd| jt j d}|| d}n d}t |jjd| t j | | jd |f}d}|dur|j| |||jd }t||| j| j| jd S)  labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rrFrrrrrNrrz=Cannot handle batch sizes > 1 if no padding token is defined.r2)r-r9z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r)rr  pooled_logitsr r )rrrr<r rrr=r-r@Zint32rZargmaxr{r|r/rIrrrrjr)r,rrrFrrr rrrZtransformer_outputsrjrrZlast_non_pad_tokenZ non_pad_maskZ token_indicesrrr0r0r1rG1sL     z)StableLmForSequenceClassification.forwardr)rIrJrKr(rrrrrr@rrnr rrrrGrLr0r0r.r1rsH     rcr)StableLmForTokenClassificationcs|t||j|_t||_t|dddur|j}nt|dddur'|j}nd}t ||_ t |j |j|_ |dS)Nclassifier_dropoutrg?)r'r(rrrgetattrrrrrrrXrWrr)r,r rr.r0r1r(zs    z'StableLmForTokenClassification.__init__cCrrHrrr0r0r1rrz3StableLmForTokenClassification.get_input_embeddingscCrrHrrr0r0r1rrz3StableLmForTokenClassification.set_input_embeddingsNrrrFrrr rrrrpc  Csd|j|||||||| d} | j} || } || } d} |dur(|| ||j} t| | | j| jdS)rrN)rrrjr) rrrrrr rrjr)r,rrrFrrr rrrrZsequence_outputrrr0r0r1rGs,  z&StableLmForTokenClassification.forwardr)rIrJrKr(rrrrrr@rrnr rrrrGrLr0r0r.r1rwsH    r)rrrrr)Nr)Crrtypingrrrrr@Ztorch.utils.checkpointrZ activationsrZ cache_utilsr r Z generationr Zmodeling_attn_mask_utilsr Zmodeling_flash_attention_utilsr rZmodeling_outputsrrrrZmodeling_rope_utilsrrZmodeling_utilsrutilsrrrrZconfiguration_stablelmrZ!torch.nn.attention.flex_attentionrZintegrations.flex_attentionrrZ get_loggerrIr{ModulerrNrRrSr^rnr~rvrwrrrrrrrrr__all__r0r0r0r1sf          #  lh\wV F