fThySrSSKrSSKJrJrJrJrJrJr SSK r SSK r SSK J r SSK JrJrJr SSKJr SSKJr SS KJr SS KJr SS KJrJr SS KJrJrJrJ r J!r!J"r" SS K#J$r$ SSK%J&r&J'r'J(r(J)r) SSK*J+r+J,r, SSK-J.r. \"5(aSSKJ/r/ \,R`"\15r2S\3S\3S\ Rh4Sjr5S\3S\3S\ Rh4Sjr6"SS\ Rn5r8"SS\ Rn5r9"SS\95r:"SS\95r;"S S!\ Rn5r<\9\:\;S".r="S#S$\ Rn5r>"S%S&\ Rn5r?\+"S'S(\$55r@\+"S)S*\@55rA\+"S+S,9"S-S.\@55rB\+"S/S,9"S0S1\@55rC\+"S2S3\@55rD\+"S4S5\@55rE\+"S6S7\@55rF/S8QrGg)9z PyTorch DistilBERT model adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) and in part from HuggingFace PyTorch version of Google AI Bert model (https://github.com/google-research/bert) N)DictListOptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)get_activation)PretrainedConfig)is_deepspeed_zero3_enabled)#_prepare_4d_attention_mask_for_sdpa)!flash_attn_supports_top_left_maskis_flash_attn_available)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indices"is_torch_greater_or_equal_than_2_2prune_linear_layer)auto_docstringlogging)DistilBertConfig)_flash_attention_forwardn_posdimoutc[5(aTSSKnURRUSS9 [R R 5S:Xa [XUS9 SSS5 g[XUS9 g!,(df  g=f)Nr) modifier_rankr$r%r&)r deepspeedzeroGatheredParameterstorch distributedget_rank_create_sinusoidal_embeddings)r$r%r&r*s j/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/distilbert/modeling_distilbert.pycreate_sinusoidal_embeddingsr2Bse!## ^^ . .s! . D  ))+q0-ELE D &ED E Ds -A// A=c[R"[U5VVs/sH?n[U5Vs/sH%oC[R"SSUS--U- 5- PM' snPMA snn5nSUl[ R "[R"USS2SSS2455USS2SSS24'[ R "[R"USS2SSS2455USS2SSS24'UR5 gs snfs snnf)Ni'Frr!) nparrayrangepower requires_gradr- FloatTensorsincosdetach_)r$r%r&posj position_encs r1r0r0Ms88hmnshtuhtadQVWZQ[\Q[ABHHUAaL34F$GGQ[\htuvLC$$RVVLADqD,A%BCC14a4L$$RVVLADqD,A%BCC14a4LKKM ]usD ,DD D c^\rSrSrS\4U4SjjrS S\RS\\RS\R4Sjjr Sr U=r $) EmbeddingsUconfigc>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"UR SS9Ul [R"UR5Ul URS[R"UR5R!S5SS9 g)N) padding_idx-q=eps position_ids)r!F) persistent)super__init__r Embedding vocab_sizer% pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormDropoutdropoutregister_bufferr-arangeexpandselfrD __class__s r1rNEmbeddings.__init__Vs !||F,=,=vzzW]WjWjk#%<<0N0NPVPZPZ#[ fjje<zz&..1   ELL)G)GHOOPWXej  input_ids input_embedsreturncUbURU5nURS5n[US5(aURSS2SU24nON[R "U[R URS9nURS5RU5nURU5nX%-nURU5nURU5nU$)am Parameters: input_ids (torch.Tensor): torch.tensor(bs, max_seq_length) The token ids to embed. input_embeds (*optional*, torch.Tensor): The pre-computed word embeddings. Can only be passed if the input ids are `None`. Returns: torch.tensor(bs, max_seq_length, dim) The embedded tokens (plus position embeddings, no token_type embeddings) Nr!rJ)dtypedevicer) rRsizehasattrrJr-rYlongre unsqueeze expand_asrTrUrW)r\r`ra seq_lengthrJrT embeddingss r1forwardEmbeddings.forwardas  // :L!&&q) 4 ( (,,Q  ^>yIL"66|D!7 ^^J/ \\*- r_)rUrWrTrRN) __name__ __module__ __qualname____firstlineno__rrNr-Tensorrrm__static_attributes__ __classcell__r]s@r1rBrBUsC  /  Xell=S_d_k_kr_rBc^\rSrSrS\4U4SjjrS\\4SjrSS\ RS\ RS\ RS \ RS \ \ RS \ S \ \ RS 44SjjrSrU=r$)MultiHeadSelfAttentionrDc>[TU]5 XlURUlURUl[ R "URS9UlSUl URUR-S:wa&[SURSURS35e[ R"URURS9Ul [ R"URURS9Ul [ R"URURS9Ul[ R"URURS9Ul[!5UlURUR-Ulg)NpFrzself.n_heads: z must divide self.dim:  evenly in_features out_features)rMrNrDn_headsr%r rVattention_dropoutrW is_causal ValueErrorLinearq_link_linv_linout_linset pruned_headsattention_head_sizer[s r1rNMultiHeadSelfAttention.__init__s   ~~ ::zzF$<$<=  88dll "a '~dll^;RSWS[S[R\\cde eYY6::FJJO YY6::FJJO YY6::FJJO yyVZZfjjQ &)e#'88t||#; r_headsc [U5S:Xag[XRURUR5up[ UR U5Ul[ URU5Ul[ URU5Ul[ URUSS9Ul UR[U5- UlURUR-Ul URRU5Ulg)Nrr!r%) lenrrrrrrrrrr%union)r\rindexs r1 prune_heads"MultiHeadSelfAttention.prune_headss u:? 7 <<!9!94;L;L ( E: ' E: ' E: )$,,1E ||c%j0 ++dll: --33E:r_querykeyvaluemask head_maskoutput_attentionsrb.c ^^^UR5umpxURS5n TRTR-mTSSU 4n S[RS[R4UUU4Sjjn S[RS[R4UUU4Sjjn U "TR U55n U "TR U55nU "TRU55nU [R"T5- n [R"XRSS55nUS:HRU 5RU5nURU[R"[R "UR"5R$55n[&R(R+US S 9nTR-U5nUbUU-n[R"UU5nU "U5nTR/U5nU(aUU4$U4$)  Parameters: query: torch.tensor(bs, seq_length, dim) key: torch.tensor(bs, seq_length, dim) value: torch.tensor(bs, seq_length, dim) mask: torch.tensor(bs, seq_length) Returns: weights: torch.tensor(bs, n_heads, seq_length, seq_length) Attention weights context: torch.tensor(bs, seq_length, dim) Contextualized layer. Optional: only if `output_attentions=True` r!xrbc`>URTSTRT5RSS5$separate headsrKr!r4viewr transposerbs dim_per_headr\s r1shape-MultiHeadSelfAttention.forward..shapes)66"b$,, =GG1M Mr_c>URSS5R5RTSTRT-5$z group headsr!r4rKr contiguousrrrs r1unshape/MultiHeadSelfAttention.forward..unshapes5;;q!$//166r2t||l?Z[ [r_r4r rrKr)rfr%rr-rtrrrmathsqrtmatmulrrrj masked_filltensorfinfordminr functionalsoftmaxrWr)r\rrrrrrq_lengthr%k_length mask_reshprrqkvscoresweightscontextrrs` @@r1rmMultiHeadSelfAttention.forwards("JJLH88A;xx4<</ !Q)  NU\\ Nell N N \u|| \  \ \ $**U# $ $**S/ " $**U# $  ,' 'aQ!23  +55f=## %,,u{{6<<8<<= --''B'7,,w'   )G,,w*'",,w' W% %: r_) rrDr%rWrrrrrrrNF)rprqrrrsrrNrintrr-rtrboolrrmrurvrws@r1ryrys</<,;c;,-1"'>||>\\>|| > ll > ELL) > > u||S !>>r_ryc^\rSrSrSrU4SjrSS\RS\RS\RS\RS\\RS \ S \ \RS 44S jjr S r U=r $)DistilBertFlashAttention2aC DistilBert flash attention module. This module inherits from `MultiHeadSelfAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cD>[TU]"U0UD6 [5Ulgro)rMrNr_flash_attn_uses_top_left_mask)r\argskwargsr]s r1rN"DistilBertFlashAttention2.__init__s# $)&) /P.Q+r_rrrrrrrb.c .^^^UR5umpxTRTR-mS[RS[R4UUU4Sjjn U "TR U55n U "TR U55n U "TRU55n TR(aTRROSn U R[R:Xa[R"5(a[R"5nOR[TRS5(aTRR nO TR R"Rn[$R'SUS35 U R)U5n U R)U5n U R)U5n [+U U U UUU TR,TR.S9nUR1TUTRT-5nTR3U5nU(aUU4$U4$) rrrbc@>URTSTRT5$)rrK)rrr batch_sizerr\s r1reshape2DistilBertFlashAttention2.forward..reshapes66*b$,, E Er__pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)rWuse_top_left_maskr)rfr%rr-rtrrrtrainingrDrrdfloat32is_autocast_enabledget_autocast_gpu_dtypergrweightlogger warning_oncetor#rrrr)r\rrrrrrrr%r query_states key_states value_states attn_dropout target_dtype attn_weightsattn_weights_reshaped attn_outputrrs` @@r1rm!DistilBertFlashAttention2.forwards(%*JJL! Hxx4<</  Fu|| F  F F tzz%01 TZZ_- tzz%01 8< t{{443     .((**$;;= &?@@#{{BB #zz0066    >$  (??<8L#|4J'??<8L/      "AAnn  !- 4 4Z4< !r_)rr)rprqrrrs__doc__rNr-rtrrrrmrurvrws@r1rrs R-1"'N"||N"\\N"|| N" ll N" ELL) N" N" u||S !N"N"r_rc^\rSrSrS\4U4SjjrSS\RS\RS\RS\RS\\RS \ S \ \RS 44U4S jjjr S r U=r $)DistilBertSdpaAttentioniKrDcb>[TU]US9 URUl[(+Ulg)N)rD)rMrNr dropout_probrrequire_contiguous_qkvr[s r1rN DistilBertSdpaAttention.__init__Ls, '"44*L&L#r_rrrrrrrb.c >^^^U(dUb)[RS5 [TT] UUUUUU5$UR 5um nTR TR -mS[RS[R4UUU4SjjnS[RS[R4UUU4Sjjn U"TRU55n U"TRU55n U"TRU55n TR(aMU RRS:Xa3Ub0U R5n U R5n U R5n [R R"R%U U U UTR&(a TR(OSSS 9n U "U 5n TR+U 5n U 4$) raDistilBertSdpaAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rrbc`>URTSTRT5RSS5$rrrs r1r.DistilBertSdpaAttention.forward..shapexs+66*b$,, EOOPQSTU Ur_c>URSS5R5RTSTRT-5$rrrs r1r0DistilBertSdpaAttention.forward..unshape|s6;;q!$//166z2t||VbGbc cr_cudarF) attn_mask dropout_pr)rrrMrmrfr%rr-rtrrrrretyperr rscaled_dot_product_attentionrrr)r\rrrrrr_rrrrrrrrr]s` @@r1rmDistilBertSdpaAttention.forwardQs(  5   d  7?!  !::< Aqxx4<</  VU\\ Vell V V du|| d  d d $**U# $ $**S/ " $**U# $  & &188==F+BtGW A A Ahh))FF +/==d''c G k* ll;/ ~r_)rrrrprqrrrsrrNr-rtrrrrmrurvrws@r1rrKsM/M-1"'G||G\\G|| G ll G ELL) G G u||S !GGr_rc^\rSrSrS\4U4SjjrS\RS\R4SjrS\RS\R4Sjr Sr U=r $) FFNirDc>[TU]5 [R"URS9UlUR UlSUl[R"URURS9Ul [R"URURS9Ul [UR5Ul g)Nr|r!r)rMrNr rVrWchunk_size_feed_forward seq_len_dimrr% hidden_dimlin1lin2r activationr[s r1rN FFN.__init__s zzFNN3 '-'E'E$II&**6CTCTU II&*;*;&**U ():):;r_inputrbcZ[URURURU5$ro)rff_chunkrr)r\rs r1rm FFN.forwards%(8T8TVZVfVfhmnnr_cURU5nURU5nURU5nURU5nU$ro)rrrrW)r\rrs r1r  FFN.ff_chunks= IIe  OOA  IIaL LLOr_)rrrWrrr) rprqrrrsrrNr-rtrmr rurvrws@r1rrsN</<oU\\oelloellu||r_r)eagerflash_attention_2sdpac ^\rSrSrS\4U4SjjrS S\RS\\RS\\RS\ S\ \RS 44 S jjr S r U=r $) TransformerBlockirDc>[TU]5 URUR-S:wa&[ SURSURS35e[ UR "U5Ul[R"URSS9Ul [U5Ul [R"URSS9Ul g)Nrzconfig.n_heads z must divide config.dim r~rG)normalized_shaperI)rMrNr%rrDISTILBERT_ATTENTION_CLASSES_attn_implementation attentionr rU sa_layer_normrffnoutput_layer_normr[s r1rNTransformerBlock.__init__s  :: &! +v~~.>>VW]WaWaVbbijk k5f6Q6QRSYZ\\6::5Qv;!#vzzu!Ur_rrrrrb.c 2URUUUUUUS9nU(aUupVO/[U5[La[S[U5S35eUSnUR XQ-5nUR U5nUR Xu-5nU4nU(aW4U-nU$)a- Parameters: x: torch.tensor(bs, seq_length, dim) attn_mask: torch.tensor(bs, seq_length) Returns: sa_weights: torch.tensor(bs, n_heads, seq_length, seq_length) The attention weights ffn_output: torch.tensor(bs, seq_length, dim) The output of the transformer block contextualization. )rrrrrrz$sa_output must be a tuple but it is z typer)rrtuple TypeErrorrrr) r\rrrr sa_output sa_weights ffn_outputoutputs r1rmTransformerBlock.forwards"NN/ #  $- !IzIe+"FtIFWW\ ]^^!! I&&y}5 XXi( #'#9#9*:P#Q   ]V+F r_)rrrr)NNFrrws@r1rrs{ V/ V -1,0"' ) <<)ELL))ELL) )  ) u||S ! ))r_rc^\rSrSrS\4U4SjjrSS\RS\\RS\\RS\ S\ S \\ S \ \ \ \RS 444S jjr S rU=r$) TransformerirDc>[TU]5 URUl[R"[ UR5Vs/sHn[ U5PM sn5UlSUlgs snfr) rMrNn_layersr ModuleListr7rlayergradient_checkpointing)r\rDrr]s r1rNTransformer.__init__sW  ]]eFOOF\#]F\$4V$