o Zh @sfdZddlZddlmZmZmZmZddlZddlm m Z ddl Zddlm Z ddl mZddlmZmZmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZmZdd l m!Z!ddl"m#Z#ddl$m%Z%m&Z&ddl'm(Z(erddlm)Z)e&*e+Z,   d:deej-eej-dfdee.deej-deej-e.ffddZ/Gddde j0Z1e#2e1Gddde j0Z3ddZ4d;d d!Z5Gd"d#d#e j0Z6d$ej-d%e.dej-fd&d'Z7Gd(d)d)e j0Z8Gd*d+d+e8Z9Gd,d-d-e8Z:e8e9e:d.Z;Gd/d0d0e j0Ze%Gd5d6d6e>Z?Gd7d8d8e>eZ@gd9ZAdS)Mz,load_balancing_loss_func..dim) isinstancetupledevicetorchcatr functionalsoftmaxtopkone_hotmeanfloatshapeexpandreshaper!sum unsqueeze)rrtop_krZconcatenated_gate_logitsrouting_weights_selected_experts expert_maskZtokens_per_expertZrouter_prob_per_expert batch_sizesequence_lengthnum_hidden_layersZexpert_attention_maskZ router_per_expert_attention_maskZ overall_lossr r#r$load_balancing_loss_func+s>    rBcs.eZdZdfdd ZddZddZZS) OlmoeRMSNormh㈵>cs&ttt||_||_dS)z; OlmoeRMSNorm is equivalent to T5LayerNorm N)super__init__r Parameterr-Zonesweightvariance_epsilon)self hidden_sizeeps __class__r r$rF~s  zOlmoeRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)Nrr)T)keepdim) dtyper!r-float32powr3ZrsqrtrIrH)rJ hidden_states input_dtypeZvariancer r r$forwards  zOlmoeRMSNorm.forwardcCst|jjd|jS)Nz, eps=)r+rHr5rIrJr r r$ extra_reprszOlmoeRMSNorm.extra_repr)rD)__name__ __module__ __qualname__rFrUrW __classcell__r r rMr$rC}srCcs8eZdZddeffdd ZeeddZZ S)OlmoeRotaryEmbeddingNconfigcstt|dr|jdur|jd|jd|_nd|_|j|_|j|_||_ t |j|_ | |j |\}|_ |j d|dd|j|_dS)N rope_scaling rope_typetypedefaultinv_freqF) persistent)rErFhasattrr^getr_max_position_embeddingsZmax_seq_len_cachedZoriginal_max_seq_lenr]rZ rope_init_fnattention_scalingZregister_bufferrbZoriginal_inv_freq)rJr]r,rbrMr r$rFs   zOlmoeRotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t j |dd+|| dd}t j ||fdd }| |j}||j} Wdn1smwY|j|jd | j|jd fS) Nrr)rZmpscpuF) device_typeenabledrr')rP)rbr4r6r5r!r,r*r`strr-Zautocast transposer.cosrgsinrP) rJx position_idsZinv_freq_expandedZposition_ids_expandedriZfreqsZembrmrnr r r$rUs0&zOlmoeRotaryEmbedding.forwardN) rXrYrZrrFr-Zno_gradrrUr[r r rMr$r\s r\cCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr)rr')r5r-r.)rox1Zx2r r r$ rotate_halfsrscCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )r9rs)qkrmrnrpZ unsqueeze_dimZq_embedZk_embedr r r$apply_rotary_pos_embs  rvcs$eZdZfddZddZZS)OlmoeMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFbias) rErFr]rKZintermediate_sizerLinear gate_projup_proj down_projrZ hidden_actact_fnrJr]rMr r$rFs zOlmoeMLP.__init__cCs$||||||}|Srq)r~rr|r})rJror~r r r$rUs zOlmoeMLP.forward)rXrYrZrFrUr[r r rMr$rws  rwrSn_repcCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r5r6r7)rSrbatchnum_key_value_headsslenhead_dimr r r$ repeat_kvs 0rcseZdZdZddedeeffdd Z       ddej d eej d eej d ee d e d e deej dee ej ej fde ej eej ee ej ffddZZS)OlmoeAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNr] layer_idxcsXt||_||_|durtd|jjd|j|_|j |_ |j |_ |j |j |_ |j |_ |j |j |_|j|_|j|_d|_|j |j |j krYtd|j d|j dtj|j |j |j |jd|_tj|j |j |j |jd|_tj|j |j |j |jd|_tj|j |j |jd|_t|j |jd|_t|j |j |j |jd|_dS) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).ryrL)rErFr]rlogger warning_oncerNrXattention_dropoutrKZnum_attention_heads num_headsrrnum_key_value_groupsrfZ rope_theta is_causal ValueErrorrr{Zattention_biasq_projk_projv_projo_projrC rms_norm_epsq_normk_normrJr]rrMr r$rFs<   zOlmoeAttention.__init__FrSrrppast_key_valueoutput_attentions use_cachecache_positionposition_embeddingsrc KsL|\} } } |||} |||}||}|jjdurF| j|jj |jjd|j|jj |jjd|j|jj |jjd| | | |j |j  dd} | | | |j |j  dd}| | | |j |j  dd}|\}}t| |||\} }|dur|||d}||||j|\}}t||j}t||j}t| | ddt|j }|dur|ddddddd|jdf}||}tjj|dtjd| j}tjj||j|j d }t||}|| |j | |j fkrt!d | |j | |j fd || dd"}|#| | |j$}|%|}|s!d}|||fS) Nminmaxrrrnrmrrr)r(rP)ptrainingz `attn_output` should be of size z , but is )&sizerrrrrr]clip_qkvclamp_viewrrrlrrvupdaterrrr-matmulmathsqrtr5rr/r0rQr!rPdropoutrrr contiguousr7rKr)rJrSrrprrrrrkwargsbszq_lenr< query_states key_states value_statesrmrn cache_kwargs attn_weights causal_mask attn_outputr r r$rU sJ       &   zOlmoeAttention.forwardrqNNNFFNN)rXrYrZ__doc__rrintrFr-Tensor LongTensorr boolrrUr[r r rMr$rs:'  rcseZdZdZfddZ       ddejdeejdeejd ee d e d e d eejd ee ejejfde ejeejee ejffddZ Z S)OlmoeFlashAttention2aF OLMoE flash attention module. This module inherits from `OlmoeAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cstj|i|t|_dSrq)rErFr_flash_attn_uses_top_left_mask)rJargsrrMr r$rFjs zOlmoeFlashAttention2.__init__NFrSrrprrrrrrc  Ksd}|\} } } |||} |||}||}|jjdurH| j|jj |jjd|j|jj |jjd|j|jj |jjd| | | |j |j  dd} | | | |j |j  dd}| | | |j |j  dd}|\}}t| |||\} }|dur|||d}||||j|\}}| dd} | dd}| dd}|jr|jnd}| j}|tjkrtrt}nt|jdr|jj}n|jjj}td|d | |} ||}||}t| |||| ||j|j d }|!| | |j"#}|$|}|s d}|||fS) NFrrrr_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)rZuse_top_left_maskr)%rrrrrrr]rrrrrrlrrvrrrrrPr-rQZis_autocast_enabledZget_autocast_gpu_dtyperdrrHrrr!rrrr7rKrr)rJrSrrprrrrrrrrr<rrrrmrnrZ dropout_raterTZ target_dtyperrr r r$rUrsf                 zOlmoeFlashAttention2.forwardr)rXrYrZrrFr-rrrr rrrUr[r r rMr$rcs:    rcseZdZdZ       ddejdeejdeejdeede d e d eejd ee ejejfd e ejeejee ejfffd d Z Z S)OlmoeSdpaAttentionz OLMoE attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from `OlmoeAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to SDPA API. NFrSrrprrrrrrc  s6|rtdtj||||||||dS|\} } } |||} |||} | |}|j j dur[| j |j j |j j d| j |j j |j j d|j |j j |j j d| | | |j|jdd} | | | |j|jdd} | | | |j|jdd}|\}}t| | ||\} } |dur|||d}|| ||j|\} }t| |j} t||j}|}|dur|ddddddd| jdf}| jjdkr|dur| } | } |}|dur| dkrd nd }tjjj| | |||jr|j nd |d }|dd}| | | |j!}|"|}|d|fS) NaOlmoeModel is using OlmoeSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rSrrprrrrrrrrrrcudaTFr)Z attn_maskZ dropout_pr)#rrrErUrrrrrrr]rrrrrrlrrvrrrrr5r,r`rr-rr/Zscaled_dot_product_attentionrrrKr)rJrSrrprrrrrrrr<rrrrmrnrrrrrMr r$rUsh       &   zOlmoeSdpaAttention.forwardr) rXrYrZrr-rrrr rrrUr[r r rMr$rs8   r)eagerflash_attention_2sdpacs2eZdZfddZdejdejfddZZS)OlmoeSparseMoeBlockcs\tj|_j|_j|_tjj|jdd|_ t fddt |jD|_ dS)NFrycsg|]}tqSr )rw)r"r<r]r r$r%<sz0OlmoeSparseMoeBlock.__init__..) rErFrnum_experts_per_tokr:norm_topk_probrr{rKgate ModuleListrangeexpertsrrMrr$rF6s $zOlmoeSparseMoeBlock.__init__rSrcCs*|j\}}}|d|}||}tj|dtjd}tj||jdd\}}|j r1||j ddd}| |j }tj |||f|j |jd}tjjj||jdd dd } t|jD]0} |j| } t| | \} } |d| fd|}| ||| | df}|d | | |j qY||||}||fS) Nr)rrr'T)r(rO)rPr,)Z num_classesrr)r5rrFr0r-r4r1r:rr8r!rPZzerosr,rr/r2rZpermuterrwherer7Z index_add_)rJrSr?r@Z hidden_dim router_logitsr;r=Zfinal_hidden_statesr>Z expert_idxZ expert_layeridxZtop_xZ current_stateZcurrent_hidden_statesr r r$rU>s(     zOlmoeSparseMoeBlock.forward)rXrYrZrFr-rrUr[r r rMr$r5s rcseZdZdedeffdd Z        ddejdeejd eej d ee d ee d ee d ee deej dee ejejfde ej ee ej ej fffddZZS)OlmoeDecoderLayerr]rcsXt|j|_t|j||d|_t||_t|j|j d|_ t|j|j d|_ dS)N)r]rr) rErFrKOLMOE_ATTENTION_CLASSES_attn_implementation self_attnrmlprCrinput_layernormpost_attention_layernormrrMr r$rFfs  zOlmoeDecoderLayer.__init__NFrSrrprroutput_router_logitsrrrrc  Ks|} ||}|jd|||||||| d| \}} } | |}|} ||}||\}}| |}|f}|r;|| f7}|rB|| f7}|rI||f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`Tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model rNr )rrrr)rJrSrrprrrrrrrZresidualZself_attn_weightsZpresent_key_valueroutputsr r r$rUps6&       zOlmoeDecoderLayer.forward)NNNFFFNN)rXrYrZrrrFr-rrrr rr FloatTensorrUr[r r rMr$res>    rc@s@eZdZeZdZdZdgZdgZdZ dZ dZ dZ dZ ddZdS) OlmoePreTrainedModelmodelTrpast_key_valuesFcCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|t r0|jj ddSt|tj rO|jjjd|d|j durQ|jj|j  dSdSdS)Nr)r3stdg?)r]Zinitializer_ranger*rr{rHdataZnormal_rzZzero_rCZfill_ Embedding padding_idx)rJmodulerr r r$ _init_weightss     z"OlmoePreTrainedModel._init_weightsN)rXrYrZrZ config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attn_2Z_supports_sdpaZ_supports_cache_classZ_supports_quantized_cacheZ_supports_static_cacherr r r r$rs rcs*eZdZdeffdd ZddZddZe           d!d ee j d ee j d ee j d ee e ee jfd ee jdeedeedeedeedeedee j de eeffddZd e j de j de j d e def ddZed e j dedede jde jde j defdd ZZS)" OlmoeModelr]cstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr )r)r"rrr r$r%r&z'OlmoeModel.__init__..rrF)rErFZ pad_token_idr vocab_sizerrrK embed_tokensrrrAlayersrCrnormr\ rotary_embgradient_checkpointing post_initrrMrr$rFs   zOlmoeModel.__init__cC|jSrqrrVr r r$get_input_embeddingszOlmoeModel.get_input_embeddingscC ||_dSrqrrJvaluer r r$set_input_embeddings zOlmoeModel.set_input_embeddingsN input_idsrrpr inputs_embedsrroutput_hidden_statesr return_dictrrc  Cs|dur|n|jj}| dur| n|jj} |dur|n|jj}|dur$|n|jj}| dur.| n|jj} |du|duAr>td|jrM|jrM|rMt dd}|durV| |}d} |rst |t ssd} |durit}n t|}t d| dur|dur|nd} tj| | |jd|jd} |dur| d}|||| ||}|}|||}|rd nd}|rd nd}| rd nd}d}|jd|jjD]V}|r||f7}|jr|jr||j|||||| || | }n ||||||| || |d }|d}|r||rd nd}|r ||df7}| r|d dur||d f7}q||}|r*||f7}|r/|nd}| r8|}| sHtd d||||fDSt|||||dS)Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FTzWe detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)rrr,r )rrprrrrrrrr)css|] }|dur|VqdSrqr )r"vr r r$ nsz%OlmoeModel.forward..)Zlast_hidden_staterrS attentionsr) r]rrrruse_return_dictrrrrrrr*r r Zfrom_legacy_cacheget_seq_lengthr-aranger5r,r9_update_causal_maskrrrAZ_gradient_checkpointing_func__call__rZto_legacy_cacher+r)rJrrrprrrrrrrrZreturn_legacy_cachepast_seen_tokensrrSrZall_hidden_statesZall_self_attnsZall_router_logitsZnext_decoder_cacheZ decoder_layerZ layer_outputsZ next_cacher r r$rUs              zOlmoeModel.forward input_tensorc Cs|jjdkr|durd|vr|SdS|dur|nd}t|t}|jjdkr7|s7|s7tj||||jdr7dS|j|j }} |j d} |rJ| } nt|t j rU|j dn|| d} |j|| | || ||j dd} |jjdkr|dur|j jd vr|st |j} t| | } | S) Nrrrr)rZpast_key_values_lengthZ is_trainingrr))r@ target_lengthrPr,rr?)rZxpuZnpu)r]rrr*r r Z_ignore_causal_mask_sdparrPr,r5Zget_max_cache_shaper-r5_prepare_4d_causal_attention_mask_with_cache_positionr`finforZ_unmask_unattended)rJrr rrrr Zusing_static_cacherPr,r@r r min_dtyper r r$rwsN          zOlmoeModel._update_causal_maskr@r rPr,r?c Ks8|dur|dkr|}|St|j} tj||f| ||d}|dkr*tj|dd}|tj||d|ddk9}|ddddddf|ddd}|dur| }|j d} |ddddddd| f|ddddddf} | dk} |ddddddd| f | | |ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. device (`torch.device`): The device to place the 4D attention mask on. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valuerPr,r)Zdiagonalrr)r) r(r-r rfullZtriurr7r6cloner5Z masked_fill) rr@r rPr,rr?rrrZ mask_lengthZ padding_maskr r r$r s(!  $ 8  z@OlmoeModel._prepare_4d_causal_attention_mask_with_cache_position) NNNNNNNNNNN)rXrYrZrrFrrrrr-rrrr rrrrrrUr staticmethodrrPr,r r[r r rMr$rs      Arc seZdZdgZfddZddZddZdd Zd d Zd d Z ddZ e             d"de e jde e jde e jde ee jde e jde e jde ede ede ede ede ede e jdeee jfdeeeffd d!ZZS)#OlmoeForCausalLMzlm_head.weightcsXt|t||_|j|_tj|j|jdd|_|j |_ |j |_ |j |_ | dSrx) rErFrrrrr{rKlm_headrouter_aux_loss_coefrrrrrMr r$rFs   zOlmoeForCausalLM.__init__cCs|jjSrqrrrVr r r$rsz%OlmoeForCausalLM.get_input_embeddingscCs ||j_dSrqrrr r r$rs z%OlmoeForCausalLM.set_input_embeddingscCrrqrrVr r r$get_output_embeddingsrz&OlmoeForCausalLM.get_output_embeddingscCrrqr)rJZnew_embeddingsr r r$set_output_embeddings rz&OlmoeForCausalLM.set_output_embeddingscCrrqr)rJdecoderr r r$ set_decoder rzOlmoeForCausalLM.set_decodercCrrqrrVr r r$ get_decoderrzOlmoeForCausalLM.get_decoderNrrrrprrlabelsrrrrrrlogits_to_keeprc Ksp|dur|n|jj}| dur| n|jj} | dur| n|jj} | dur$| n|jj} |j|||||||| | | | d }|d}t| trGt| dn| }| |dd|ddf}d}|duri|j |||j fi|}d}| rt | rs|j n|d|j|j|}|dur||j||j7}| s|f|dd}| r|f|}|dur|f|S|St||||j|j|j|j dS)u labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, OlmoeForCausalLM >>> model = OlmoeForCausalLM.from_pretrained("allenai/OLMoE-1B-7B-0924") >>> tokenizer = AutoTokenizer.from_pretrained("allenai/OLMoE-1B-7B-0924") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] 'Hey, are you conscious? Can you talk to me?\nI’m not sure if you’re conscious of this, but I’m' ``` N) rrrprrrrrrrrrr)r)lossaux_losslogitsrrSrr)r]rrrrrr*rslicerZ loss_functionrrBrrrrr!r,rrrSr)rJrrrprrrrrrrrrrZ loss_kwargsrrSZ slice_indicesr"r r!outputr r r$rUsb) zOlmoeForCausalLM.forward) NNNNNNNNNNNNr)rXrYrZZ_tied_weights_keysrFrrrrrrrrr-rrrrrrrrrrUr[r r rMr$rsh          r)rrr)NrN)Nr)Brrtypingrrrrr-Ztorch.nn.functionalrr/rZtorch.utils.checkpointZ activationsrZ cache_utilsr r r Z generationr Zmodeling_attn_mask_utilsr Zmodeling_flash_attention_utilsrrZmodeling_outputsrrZmodeling_rope_utilsrrZmodeling_utilsrZ pytorch_utilsrutilsrrZconfiguration_olmoerrZ get_loggerrXrrrrBModulerCappendr\rsrvrwrrrrrrrrrr__all__r r r r$st            R #  jl`0W