o Zh@sddlmZmZddlZddlmZddlmZddlmZddl m Z dd l m Z m Z mZmZd d lmZe eZGd d d ejZGddde ZGdddeZGdddeZGddde ZgdZdS))OptionalTupleN)nn)ACT2FN)Cache)logging)GraniteMoeDecoderLayerGraniteMoeForCausalLMGraniteMoeModelGraniteMoePreTrainedModel)GraniteMoeSharedConfigcs<eZdZdZdeffdd ZdejdejfddZZ S) GraniteMoeSharedMLPz~ MLP layer for shared experts Args: config: Configuration object with model hyperparameters. configcs^tt||j|_|j|_t|j|_t j |j|jddd|_ t j |j|jdd|_ dS)Nr F)Zbias) superr__init__Z hidden_sizeZ input_sizeshared_intermediate_sizerZ hidden_act activationrZLinear input_linear output_linearselfr __class__l/var/www/auris/lib/python3.10/site-packages/transformers/models/granitemoeshared/modular_granitemoeshared.pyr-s  zGraniteMoeSharedMLP.__init__ hidden_statesreturncCs<||}|jddd}||d|d}||}|S)Nr )dimrr)rchunkrr)rrZchunked_hidden_statesrrrforward6s  zGraniteMoeSharedMLP.forward) __name__ __module__ __qualname____doc__rrtorchTensorr# __classcell__rrrrr$s rcseZdZdedeffdd Z        ddejdeejd eej d ee d ee d ee d eej dee dee ejejfde ej ee ej ej fffddZZS)GraniteMoeSharedDecoderLayerr layer_idxcs0t|||jdkrd|_dSt||_dS)Nr)rrrr shared_mlp)rrr,rrrr?s"z%GraniteMoeSharedDecoderLayer.__init__NFrattention_mask position_idspast_key_valueoutput_attentions use_cachecache_positionoutput_router_logitsposition_embeddingsrc  Ks|} ||}|jd|||||||| d| \}} } | ||j}|} ||}||\}}|jdur8|}n|||}~| ||j}|f}|rQ|| f7}|rX|| f7}|r_||f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`Tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )rr.r/r0r1r2r3r5Nr)Zinput_layernormZ self_attnZresidual_multiplierZpost_attention_layernormZblock_sparse_moer-)rrr.r/r0r1r2r3r4r5kwargsZresidualZself_attn_weightsZpresent_key_valueZmoe_hidden_statesZ router_logitsZoutputsrrrr#Cs>&        z$GraniteMoeSharedDecoderLayer.forward)NNNFFNFN)r$r%r&rintrr(r)rZ LongTensorrboolrZ FloatTensorr#r*rrrrr+>s>   r+c@seZdZeZdgZdS)GraniteMoeSharedPreTrainedModelr+N)r$r%r&rZ config_classZ_no_split_modulesrrrrr9s r9cs"eZdZdeffdd ZZS)GraniteMoeSharedModelrcs0ttfddtjD|_dS)Ncsg|]}t|qSr)r+).0r,rrr sz2GraniteMoeSharedModel.__init__..)rrrZ ModuleListrangeZnum_hidden_layersZlayersrrr<rrs  zGraniteMoeSharedModel.__init__)r$r%r&rrr*rrrrr:sr:cs(eZdZdgZdeffdd ZZS)GraniteMoeSharedForCausalLMzlm_head.weightrcs"t|t||_|dS)N)rrr:modelZ post_initrrrrrs   z$GraniteMoeSharedForCausalLM.__init__)r$r%r&Z_tied_weights_keysrrr*rrrrr?sr?)r?r:r9)typingrrr(rZ activationsrZ cache_utilsrutilsrZgranitemoe.modeling_granitemoer r r r Zconfiguration_granitemoesharedrZ get_loggerr$loggerModulerr+r9r:r?__all__rrrrs      Z