fThSSKJrJr SSKrSSKJr SSKJr SSKJr SSK J r SS K J r J r JrJr S S KJr \ R$"\5r"S S \R*5r"SS\ 5r"SS\5r"SS\5r"SS\ 5r/SQrg))OptionalTupleN)nn)ACT2FN)Cache)logging)GraniteMoeDecoderLayerGraniteMoeForCausalLMGraniteMoeModelGraniteMoePreTrainedModel)GraniteMoeSharedConfigcn^\rSrSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $) GraniteMoeSharedMLP$zj MLP layer for shared experts Args: config: Configuration object with model hyperparameters. configc`>[[U] 5 URUlUR Ul[ URUl[R"URURS-SS9Ul [R"URURSS9Ul g)Nr F)bias) superr__init__ hidden_size input_sizeshared_intermediate_sizer hidden_act activationrLinear input_linear output_linearselfr __class__s u/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/granitemoeshared/modular_granitemoeshared.pyrGraniteMoeSharedMLP.__init__-s !413 ,,!:: !2!23IIdoot7G7G!7KRWXYYt'7'7uU hidden_statesreturncURU5nURSSS9nURUS5US-nURU5nU$)Nr )dimrr)rchunkrr )r"r'chunked_hidden_statess r$forwardGraniteMoeSharedMLP.forward6s^))-8 - 3 3A2 3 >(=a(@ADYZ[D\\ **=9 r&)rrrrr ) __name__ __module__ __qualname____firstlineno____doc__rrtorchTensorr.__static_attributes__ __classcell__r#s@r$rr$s7V5VU\\ellr&rc^\rSrSrS\S\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \ S \ \RS \ \ S \ \ \R\R4S\ \R\ \ \R\R444SjjrSrU=r$)GraniteMoeSharedDecoderLayer>r layer_idxct>[TU]X5 URS:XaSUlg[U5Ulg)Nr)rrrr shared_mlpr"rr=r#s r$r%GraniteMoeSharedDecoderLayer.__init__?s1 +"("A"AQ"F$L_`fLgr&r'attention_mask position_idspast_key_valueoutput_attentions use_cachecache_positionoutput_router_logitsposition_embeddingsr(c Un URU5nUR"SUUUUUUUU S.U D6upn XUR--nUn URU5nUR U5upUR cUnOXR U5-nAXUR--nU4nU(aUU 4- nU(aUU 4- nU(aUU4- nU$)aY Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`Tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )r'rBrCrDrErFrGrI)input_layernorm self_attnresidual_multiplierpost_attention_layernormblock_sparse_moer?)r"r'rBrCrDrErFrGrHrIkwargsresidualself_attn_weightspresent_key_valuemoe_hidden_states router_logitsoutputss r$r.$GraniteMoeSharedDecoderLayer.forwardCs L!,,]; ?Cnn ? ')%)/) 3 ?  ? ; *;!43K3K#KK !55mD +/+@+@+O( ?? "-M- 0NNM  43K3K#KK  "  )+ +G  )+ +G   ' 'Gr&)r?)NNNFFNFN)r0r1r2r3rintrr5r6r LongTensorrboolr FloatTensorr.r7r8r9s@r$r;r;>s!h5h#h2637*.,1$)59/4KOR||R!.Ru//0 R ! R $D> RD>R!!1!12R'tnR&eELL%,,,F&GHR u  (51B1BEDUDU1U+V"WW XRRr&r;c\rSrSr\rS/rSrg)GraniteMoeSharedPreTrainedModelr;rKN)r0r1r2r3r config_class_no_split_modulesr7rKr&r$r^r^s)L78r&r^c0^\rSrSrS\4U4SjjrSrU=r$)GraniteMoeSharedModelrc >[TU]U5 [R"[ UR 5Vs/sHn[ X5PM sn5Ulgs snfN)rrr ModuleListrangenum_hidden_layersr;layersr@s r$rGraniteMoeSharedModel.__init__sI  mmNSTZTlTlNm nNm )& [TU]U5 [U5UlUR 5 grf)rrrcmodel post_initr!s r$r$GraniteMoeSharedForCausalLM.__init__s&  *62  r&)rp) r0r1r2r3_tied_weights_keysrrr7r8r9s@r$rmrms*+5r&rm)rmrcr^)typingrrr5r activationsr cache_utilsrutilsr granitemoe.modeling_granitemoer r r rconfiguration_granitemoesharedr get_loggerr0loggerModulerr;r^rcrm__all__rKr&r$r~s # !  C   H %"))4W#9Wt9&?9  O "7 fr&