fThWSrSSKrSSKrSSKJr SSKJrJrJrJ r J r SSK r SSK r SSK J r SSKJrJrJr SSKJrJr SS KJr SS KJrJrJrJrJrJrJrJr SS K J!r! SS K"J#r#J$r$J%r% SS K&J'r'J(r(J)r) SSK*J+r+ \)RX"\-5r.SESjr/"SS\ R`5r1"SS\ R`5r2"SS\ R`5r3S\20r4"SS\ R`5r5"SS\ R`5r6"SS\ R`5r7"SS\ R`5r8"S S!\ R`5r9"S"S#\ R`5r:"S$S%\ R`5r;\("S&S'\!55r<\"S(S)\'55r=\("S*S+\<55r>"S,S-\ R`5r?"S.S/\ R`5r@\("S0S19"S2S3\<55rA\("S4S19"S5S6\<55rB\("S7S19"S8S9\<55rC\("S:S19"S;S<\<55rD\("S=S>\<55rE\("S?S@\<55rF\("SAS19"SBSC\<\55rG/SDQrHg)FzPyTorch ELECTRA model.N) dataclass)CallableListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNget_activation)GenerationMixin)"BaseModelOutputWithCrossAttentions)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging) ElectraConfigc>SSKnSSKnSSKn[ RRU5n[RSU35 URRU5n/n /n UH]up[RSU SU 35 URRX{5n U RU 5 U RU 5 M_ [X5GH;upU n[U[ 5(aU R#SS5n US :Xa$U R#S S 5n U R#S S 5n U R#S S5n U R#SS5n U R%S5n ['SU 55(a[RSU35 MUnU HnUR)SU5(aUR%SU5nOU/nUSS:Xd USS:Xa [+US5nOZUSS:Xd USS:Xa [+US5nO;USS:Xa [+US5nO%USS:Xa [+US5nO[+UUS5n[-U5S:dM[/US 5nUUnM WR1S!5(a [+US5nOUS:XaUR3U 5n UR4U R4:wa&[7S"UR4S#U R4S$35e[;S%U 3U5 [<R>"U 5Ul GM> U$![a [R S5 ef=f![6a1nU=R8UR4U R44- sleSnAff=f![Ban[;SU3U U5 SnAGMSnAff=f)&z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape zelectra/embeddings/zgenerator/embeddings/ generatorzelectra/zdiscriminator/z generator/dense_1dense_predictionz!generator_predictions/output_biaszgenerator_lm_head/bias/c3*# UH oS;v M g7f)) global_step temperatureN).0ns d/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/electra/modeling_electra.py -load_tf_weights_in_electra..^sE166sz Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifierr _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )"renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendzip isinstanceElectraForMaskedLMreplacesplitany fullmatchgetattrlenintendswith transposeshape ValueErrorargsprinttorch from_numpydataAttributeError)modelconfigtf_checkpoint_pathdiscriminator_or_generatorr;nptftf_path init_varsnamesarraysnamerUarray original_namepointerm_name scope_namesnumes r-load_tf_weights_in_electraro4so ggoo01G KK8 BC''0I E F  (l5'BC&&w5 T e ! 5) ! 3 %!344||$9;RS)[8||J0@A||L*=<< +=>D<< CE]^D::c?DEEEE i 78G<< 0&99"$((9f"=K#)(Kq>X-Q71J%gx8G ^}4 A&8P%gv6G ^'77%gx8G ^w.%g|G{#q(k!n-C%clG#$}--!'848# U+ ==EKK/$~gmm_DUV[VaVaUbbm%noo0 .tf5} E ++E2GLg*n LQ   Q   @ 7==%++66    Im_-tQ 7  sW LB5M6B:M6AM6"AL8"+M6!L58 M3,M..M33M66 NNNc ^\rSrSrSrU4SjrS S\\RS\\RS\\RS\\RS\ S \R4 S jjr S r U=r$) ElectraEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.c.>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR#S[$R&"UR5R)S5SS9 [+USS5UlUR#S [$R."UR0R35[$R4S 9SS9 g) N) padding_idxeps position_ids)r F) persistentposition_embedding_typeabsolutetoken_type_idsdtype)super__init__r Embedding vocab_sizeembedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferrYarangeexpandrPrzzerosrwsizelongselfr^ __class__s r-rElectraEmbeddings.__init__s1 !||F,=,=v?T?Tbhbubuv#%<<0N0NPVPePe#f %'\\&2H2H&J_J_%`"f&;&;AVAVWzz&"<"<=   ELL)G)GHOOPWXej  (/v7PR\']$  ekk$*;*;*@*@*B%**Ubg  input_idsr|rw inputs_embedspast_key_values_lengthreturncdUbUR5nOUR5SSnUSnUcURSS2XWU-24nUcv[US5(a-URSS2SU24nUR USU5n U nO8[ R "U[ RURRS9nUcURU5nURU5n XJ-n URS:XaURU5n X- n URU 5n URU 5n U $)Nrxr r|rr~devicer{)rrwhasattrr|rrYrrrrrrzrrr) rrr|rwrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr embeddingsrs r-forwardElectraEmbeddings.forwardsC  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  !t-..*.*=*=a*n*M'3J3Q3QR]^_R`bl3m0!A!&[ SWSdSdSkSk!l   00;M $ : :> J":  ' ': 5"&":":<"H   -J^^J/ \\*- r)rrrzrrr)NNNNr)__name__ __module__ __qualname____firstlineno____doc__rrrY LongTensor FloatTensorrRTensorr__static_attributes__ __classcell__rs@r-rqrqsQ .15593759&' 'E,,-'!!1!12'u//0 '   1 12 ' !$ ' ''rrqcb^\rSrSrSU4SjjrS\R S\R 4SjrSS\R S\\RS\\RS \\RS \\RS \\ \ \RS \\ S\ \R 4S jjr Sr U=r$)ElectraSelfAttentionc>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eURUl[ URUR- 5UlURUR-Ul[R"URUR5Ul [R"URUR5Ul [R"URUR5Ul [R"UR5UlU=(d [#USS5UlUR$S:XdUR$S :XaGUR&Ul[R("S UR&-S - UR5UlUR,Ulg) NrrzThe hidden size (z6) is not a multiple of the number of attention heads ()rzr{ relative_keyrelative_key_queryr9r )rr hidden_sizenum_attention_headsrrVrRattention_head_size all_head_sizer Linearquerykeyvaluerattention_probs_dropout_probrrPrzrrdistance_embedding is_decoderrr^rzrs r-rElectraSelfAttention.__init__s     : : :a ?PVXhHiHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD # ++rxrcUR5SSURUR4-nURU5nUR SSSS5$)Nrxrr9r r )rrrviewpermute)rr new_x_shapes r-transpose_for_scores)ElectraSelfAttention.transpose_for_scoressLffhsmt'?'?AYAY&ZZ FF; yyAq!$$r hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionscVURU5nUSLn U (aUbUSn USn UnGOU (aCURURU55n URURU55n UnOUbuURURU55n URURU55n [R "USU /SS9n [R "USU /SS9n O@URURU55n URURU55n URU5n USLn UR (aX4n[R"XRSS55nURS:XdURS:XGaU RSU RSnnU (aB[R"US- [RURS 9RSS5nO>[R"U[RURS 9RSS5n[R"U[RURS 9RSS5nUU- nUR!UUR"-S- 5nUR%U R&S 9nURS:Xa[R("S U U5nUU-nOHURS:Xa8[R("S U U5n[R("S U U5nUU-U-nU[*R,"UR.5- nUbX-n[0R2R5USS9nUR7U5nUbUU-n[R"UU 5nUR9SSSS 5R;5nUR=5SSUR>4-nURU5nU(aUU4OU4nUR (aUU4-nU$)Nrr r9dimrxrrrr}zbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rrrrrYcatrmatmulrTrzrUtensorrrrrrrtor~einsummathsqrtrr functionalsoftmaxrr contiguousrr)rrrrrrrrmixed_query_layeris_cross_attention key_layer value_layer query_layer use_cacheattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputss r-rElectraSelfAttention.forwards!JJ}5 3$> ."<&q)I(+K3N 11$((;P2QRI33DJJ?T4UVK3N  '11$((=2IJI33DJJ}4MNK >!#4i"@aHI))^A%6 $D!LK11$((=2IJI33DJJ}4MNK//0AB "$. ??(5N!<< 5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC 6G=/2mM] ?? 11Gr) rrrrrrrrrzrrNNNNNNF)rrrrrrYrrrrrboolrrrrs@r-rrs,4%ell%u||%7;15=A>BDH,1c||c!!2!23cE--. c ((9(9: c !)):): ; c!uU->->'?!@Ac$D>c u|| ccrrcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)ElectraSelfOutputiPc(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNru) rrr rrdenserrrrrrs r-rElectraSelfOutput.__init__Qs` YYv1163E3EF f&8&8f>S>STzz&"<"<= rr input_tensorrcpURU5nURU5nURX-5nU$rrrrrrrs r-rElectraSelfOutput.forwardW5 =1  ]3 }'CD rrrr rrrrrrYrrrrrs@r-rrP6> U\\RWR^R^rreagerc.^\rSrSrSU4SjjrSrSS\RS\\RS\\RS\\RS\\RS \\ \ \RS \\ S \ \R4S jjr S r U=r$)ElectraAttentionidc>[TU]5 [UR"XS9Ul[ U5Ul[5Ulg)Nrz) rrELECTRA_SELF_ATTENTION_CLASSES_attn_implementationrroutputset pruned_headsrs r-rElectraAttention.__init__es@ 263N3NO   (/ Erc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrr r)rQrrrrrrrrrrrrunion)rheadsindexs r- prune_headsElectraAttention.prune_headsms  u:? 7 9900$))2O2OQUQbQb  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rrrrrrrrrc pURUUUUUUU5nURUSU5n U 4USS-n U $)Nrr )rr) rrrrrrrr self_outputsattention_outputrs r-rElectraAttention.forwardsWyy    ! "    ;;|A F#% QR(88r)rrrrr)rrrrrrrYrrrrrrrrrs@r-r r ds";*7;15=A>BDH,1||!!2!23E--.  ((9(9:  !)):): ; !uU->->'?!@A$D> u|| rr cb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)ElectraIntermediateic>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) rrr rrintermediate_sizerrJ hidden_actstrrintermediate_act_fnrs r-rElectraIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $rrrcJURU5nURU5nU$rrr&)rrs r-rElectraIntermediate.forwards& =1 00? rr)r rs@r-r!r!s(9U\\ellrr!cz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) ElectraOutputic(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) rrr rr#rrrrrrrrs r-rElectraOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rrrrcpURU5nURU5nURX-5nU$rrrs r-rElectraOutput.forwardrrrr rs@r-r,r,r rr,c*^\rSrSrU4SjrSS\R S\\RS\\RS\\RS\\RS\\ \ \RS \\ S \ \R 4S jjr S r S r U=r$) ElectraLayerict>[TU]5 URUlSUl[ U5UlUR UlURUlUR(a.UR (d[US35e[ USS9Ul [U5Ul [U5Ul g)Nr z> should be used as a decoder model if cross attention is addedr{r)rrchunk_size_feed_forward seq_len_dimr  attentionradd_cross_attentionrVcrossattentionr! intermediater,rrs r-rElectraLayer.__init__s '-'E'E$)&1 ++#)#=#=  # #?? D6)g!hii"26S]"^D /7#F+ rrrrrrrrrc UbUSSOSnURUUUUUS9n U Sn UR(a U SSn U Sn OU SSn Sn UR(aZUbW[US5(d[SUS35eUbUS SOSnUR U UUUUUU5nUSn XSS-n USn W U -n [ UR URURU 5nU4U -n UR(aU W 4-n U $) Nr9)rrrr rxr8z'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) r6rrrVr8rfeed_forward_chunkr4r5)rrrrrrrrself_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputs r-rElectraLayer.forwards}:H9S>"1#5Y] !%   /3 "0" 2!4 ??,Qr2G 6r : ,QR0G'+$ ??4@4!122 =dVDDD @N?Yrs(;_c %&*&9&9 %&)!' # 7q9 " ==G,C2+F ( 14P P 0  # #T%A%A4CSCSUe  /G+ ??!2 44GrcJURU5nURX!5nU$r)r9r)rrintermediate_outputrCs r-r<ElectraLayer.feed_forward_chunks)"//0@A{{#6I r)r7r6r4r8r9rrr5r)rrrrrrYrrrrrrr<rrrs@r-r2r2s ,"7;15=A>BDH,1?||?!!2!23?E--. ? ((9(9: ? !)):): ; ?!uU->->'?!@A?$D>? u|| ?Brr2cR^\rSrSrU4SjrSS\R S\\RS\\RS\\RS\\RS\\ \ \RS \\ S \\ S \\ S \\ S \ \ \R \ 44Sjjr SrU=r$)ElectraEncoderic>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snf)NF) rrr^r ModuleListrangenum_hidden_layersr2layergradient_checkpointing)rr^_rs r-rElectraEncoder.__init__sR  ]]%H`H`Ba#bBaQL$8Ba#bc &+#$csA&rrrrrpast_key_valuesrroutput_hidden_states return_dictrc 8U (aSOSn U(aSOSn U(aURR(aSOSn UR(a/UR(aU(a[R S5 SnU(aSOSn[ UR5HunnU (aX4-n UbX?OSnUbXoOSnUR(a4UR(a#URURUUUUUUU5nOU"UUUUUUU5nUSnU(a UUS4- nU(dMU US4-n URR(dMU US4-n M U (aX4-n U (d[SUUU U U 455$[UUU U U S 9$) Nr*zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frrxr r9c30# UH nUcMUv M g7frr*)r+vs r-r.)ElectraEncoder.forward..Ws" As )last_hidden_staterRr attentionscross_attentions) r^r7rOtrainingr? warning_once enumeraterN_gradient_checkpointing_func__call__tupler)rrrrrrrRrrrSrTall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_maskr layer_outputss r-rElectraEncoder.forwards#7BD$5b4%64;;;Z;Zr`d  & &4==##p" #,R$(4OA|#$58H$H!.7.CilO3B3N_/TXN**t}} $ A A ))!"#)*"% ! !-!"#)*"%! *!,M"}R'8&::"  &9]1=MBEI$(,1/4&*S ||S !!2!23S E--. S ((9(9: S !)):): ; S "%e.?.?(@"ABS D>S $D>S 'tnS d^S  uU\\"$MM NS S rrIc2^\rSrSrSrU4SjrSrSrU=r$)ElectraDiscriminatorPredictionsikzEPrediction module for the discriminator, made up of two dense layers.c>[TU]5 [R"URUR5Ul[ UR5Ul[R"URS5Ul Xl gNr ) rrr rrrrr$ activationr%r^rs r-r(ElectraDiscriminatorPredictions.__init__ns\ YYv1163E3EF ():):; " &*<*?  6 &&}5==bA r)ror^rr% rrrrrrrrrrs@r-rlrlksOrrlc2^\rSrSrSrU4SjrSrSrU=r$)ElectraGeneratorPredictionsi~zAPrediction module for the generator, made up of two dense layers.c>[TU]5 [S5Ul[R "UR URS9Ul[R"URUR 5Ul g)Ngeluru) rrrror rrrrrrrs r-r$ElectraGeneratorPredictions.__init__sV (0f&;&;AVAVWYYv1163H3HI rclURU5nURU5nURU5nU$r)rror)rgenerator_hidden_statesrs r-r#ElectraGeneratorPredictions.forwards3 #:;  6 }5 r)rrorrvrs@r-rxrx~sKJrrxc*\rSrSr\r\rSrSr Sr Sr g)ElectraPreTrainedModelielectraTc[U[R5(akURRR SUR RS9 URb%URRR5 gg[U[R5(axURRR SUR RS9 URb2URRURR5 gg[U[R5(aJURRR5 URRRS5 gg)zInitialize the weightsg)meanstdNg?)rJr rr2r[normal_r^initializer_ranger5zero_rrtrfill_)rmodules r- _init_weights$ElectraPreTrainedModel._init_weightss fbii ( ( MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).rr*N) rrrrr! config_classroload_tf_weightsbase_model_prefixsupports_gradient_checkpointingrrr*rr-rrs L0O!&*#*rrc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \R\ S'Sr \\ \R\ S'Srg) ElectraForPreTrainingOutputia Output type of [`ElectraForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss of the ELECTRA objective. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Prediction scores of the head (scores for each token before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. NlossrtrrZr*)rrrrrrrrYr__annotations__rtrrrZrr*rr-rrsg*)-D(5$$ %,*.FHU&& '.8[TU]U5 [U5UlURUR :wa0[ R"URUR 5Ul[U5Ul Xl UR5 gr) rrrqrrrr rembeddings_projectrIencoderr^ post_initrs r-rElectraModel.__init__sj  +F3  F$6$6 6&(ii0E0EvGYGY&ZD #%f-   rc.URR$rrrrs r-get_input_embeddings!ElectraModel.get_input_embeddingss...rc$XRlgrr)rrs r-set_input_embeddings!ElectraModel.set_input_embeddingss */'rcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrNr6r)rheads_to_prunerNrs r- _prune_headsElectraModel._prune_headss< +002LE LL  u % / / ; ;E B3rrrr|rwrrrrrRrrrSrTrc U bU OURRn U bU OURRn U bU OURRn UbUb [ S5eUb"UR X5 UR 5nO"UbUR 5SSnO [ S5eUunnUb URO URnU bU SSRSOSnUc[R"UUS9nUcs[URS5(a4URRSS2SU24nURUU5nUnO$[R"U[R US9nUR#X.5nURR$(aEUbBUR 5unnnUU4nUc[R"UUS9nUR'U5nOSnUR)XPRR*5nURUUUUUS 9n[US 5(aUR-U5nUR/UUUUUU U U U U S 9 nU$) NzDYou cannot specify both input_ids and inputs_embeds at the same timerxz5You have to specify either input_ids or inputs_embedsrr9)rr|r)rrwr|rrr) rrrrrRrrrSrT)r^rrSuse_return_dictrV%warn_if_padding_and_no_attention_maskrrrUrYonesrrr|rrrget_extended_attention_maskrinvert_attention_mask get_head_maskrMrr)rrrr|rwrrrrrRrrrSrTr batch_sizerrrrrextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrPencoder_hidden_shapeencoder_extended_attention_maskrs r-rElectraModel.forwards"2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, J%.%:!! @T@TDSC^!3A!6!"_ ;; ! !&;&G=R=W=W=Y :  7$68O#P %-).4HQW)X&.2.H.HI_.` +.2 +&&y++2O2OP %)'#9 (  4- . . 33MBM 2"7#B+/!5#%  r)r^rrr) NNNNNNNNNNNNN)rrrrrrrrrrrYrrrrrrrrrrrs@r-rrsk /0C-11515/3,0048<9==A$(,0/3&*WELL)W!.W!. W u||, W ELL) W  -W ( 5W!) 6W"$u'8'8"9:WD>W$D>W'tnWd^W uU\\"$FF GWWrrc2^\rSrSrSrU4SjrSrSrU=r$)ElectraClassificationHeadi<z-Head for sentence-level classification tasks.c>[TU]5 [R"URUR5UlUR b UR O URn[S5Ul [R"U5Ul [R"URUR5Ul g)Nrz)rrr rrrclassifier_dropoutrrrorr num_labelsout_projrr^rrs r-r"ElectraClassificationHead.__init__?s YYv1163E3EF )/)B)B)NF % %TZTnTn )0zz"45  &"4"4f6G6GH rc USS2SSS24nURU5nURU5nURU5nURU5nURU5nU$)Nr)rrror)rfeatureskwargsrs r-r!ElectraClassificationHead.forwardIsZ Q1W  LLO JJqM OOA  LLO MM! r)rorrrrvrs@r-rr<s7Irrc^\rSrSrSrS\4U4SjjrS S\RS\ \RS\R4Sjjr S r U=r $) ElectraSequenceSummaryiTa Compute a single vector summary of a sequence hidden states. Args: config ([`ElectraConfig`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. r^c>[TU]5 [USS5UlURS:Xa[e[ R "5Ul[US5(aUR(aq[US5(a.UR(aURS:a URnO URn[ R"URU5Ul[USS5nU(a [U5O[ R "5Ul[ R "5Ul[US5(a5UR"S:a%[ R$"UR"5Ul[ R "5Ul[US 5(a7UR(S:a&[ R$"UR(5Ulggg) N summary_typelastattnsummary_use_projsummary_proj_to_labelsrsummary_activationsummary_first_dropoutsummary_last_dropout)rrrPrNotImplementedErrorr Identitysummaryrrrrrrrro first_dropoutrr last_dropoutr)rr^ num_classesactivation_stringrs r-rElectraSequenceSummary.__init__nsa #FNFC    && %{{} 6- . .63J3Jv788V=Z=Z_e_p_pst_t$// $00 99V%7%7EDL#F,@$GIZN3D$E`b`k`k`m[[] 62 3 38T8TWX8X!#F,H,H!ID KKM 61 2 2v7R7RUV7V " 6+F+F GD 8W 2rr cls_indexrcURS:Xa USS2S4nGOURS:Xa USS2S4nGOURS:XaURSS9nOURS :XaUc?[R"US SS2SS24URS S- [R S 9nOXUR S5R S5nURS UR5S- -URS54-5nURS U5RS 5nOURS:Xa[eURW5nURU5nURU5nUR!U5nU$)a# Compute a single vector summary of a sequence hidden states. Args: hidden_states (`torch.FloatTensor` of shape `[batch_size, seq_len, hidden_size]`): The hidden states of the last layer. cls_index (`torch.LongTensor` of shape `[batch_size]` or `[batch_size, ...]` where ... are optional leading dimensions of `hidden_states`, *optional*): Used if `summary_type == "cls_index"` and takes the last token of the sequence as classification token. Returns: `torch.FloatTensor`: The summary of the sequence hidden states. rNrxfirstrrr rr.rr})rxr)rrrY full_likerUr unsqueezerrrgatherrrrrrror)rrrrs r-rElectraSequenceSummary.forwardsn    &"1b5)F   ' )"1a4(F   & ("''A'.F   + - !OO!#rr1*-!''+a/** &//3==bA %,,Uimmo6I-JmN`N`acNdMf-fg "))"i8@@DF   & (% %##F+f%(""6* r)rorrrrr)rrrrrr!rrYrrrrrrrs@r-rrTsV2H}H<Y])"..);CEDTDT;U)   ))rrz ELECTRA Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introcl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \R\ 44Sjj5r SrU=r$) ElectraForSequenceClassificationic>[TU]U5 URUlXl[ U5Ul[ U5UlUR5 gr) rrrr^rrrr8rrs r-r)ElectraForSequenceClassification.__init__sF   ++ #F+ 3F; rrrr|rwrrlabelsrrSrTrc 0U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n SnUGbURRcUR S:XaSURlOoUR S:aNUR [R:XdUR [R:XaSURlOSURlURRS:XaI[5nUR S:Xa&U"U R5UR55nOU"X5nOURRS:Xa=[5nU"U RSUR 5URS55nO,URRS:Xa[5nU"X5nU (dU 4U SS-nUbU4U-$U$[UU U R U R"S 9$) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr|rwrrrrSrTrr regressionsingle_label_classificationmulti_label_classificationrxrrtrrZ)r^rrr8 problem_typerr~rYrrRr rrr rr rrrZ)rrrr|rwrrrrrSrTrssequence_outputrtrloss_fctrs r-r(ElectraForSequenceClassification.forwards(&1%D 'tnD d^D  uU\\"$<< =D D rrz Electra model with a binary classification head on top as used during pretraining for identifying generated tokens. It is recommended to load the discriminator checkpoint into that model. cl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \R\ 44Sjj5r SrU=r$)ElectraForPreTrainingic>[TU]U5 [U5Ul[ U5UlUR 5 gr)rrrrrldiscriminator_predictionsrrs r-rElectraForPreTraining.__init__s3  #F+ )H)P& rrrr|rwrrrrrSrTrc lU bU OURRn URUUUUUUUU U S9 n U Sn URU 5n SnUb[R "5nUbaUR SU RS5S:HnU R SU RS5UnUUnU"UUR55nO4U"U R SU RS5UR55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the ELECTRA loss. Input should be a sequence of tokens (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates the token is an original token, - 1 indicates the token was replaced. Examples: ```python >>> from transformers import ElectraForPreTraining, AutoTokenizer >>> import torch >>> discriminator = ElectraForPreTraining.from_pretrained("google/electra-base-discriminator") >>> tokenizer = AutoTokenizer.from_pretrained("google/electra-base-discriminator") >>> sentence = "The quick brown fox jumps over the lazy dog" >>> fake_sentence = "The quick brown fox fake over the lazy dog" >>> fake_tokens = tokenizer.tokenize(fake_sentence, add_special_tokens=True) >>> fake_inputs = tokenizer.encode(fake_sentence, return_tensors="pt") >>> discriminator_outputs = discriminator(fake_inputs) >>> predictions = torch.round((torch.sign(discriminator_outputs[0]) + 1) / 2) >>> fake_tokens ['[CLS]', 'the', 'quick', 'brown', 'fox', 'fake', 'over', 'the', 'lazy', 'dog', '[SEP]'] >>> predictions.squeeze().tolist() [0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0] ```Nrrrxr r) r^rrrr r rrUfloatrrrZ)rrrr|rwrrrrrSrTrsdiscriminator_sequence_outputrtrr active_loss active_logits active_labelsrs r-rElectraForPreTraining.forward spZ&1%Q 'tnQ d^Q  uU\\"$?? @Q Q rrz Electra model with a language modeling head on top. Even though both the discriminator and generator may be loaded into this model, the generator is the only model of the two to have been trained for the masked language modeling task. c~^\rSrSrS/rU4SjrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ S\ \ S\ \ S\ \\ R\44Sjj5rSrU=r$)rKiugenerator_lm_head.weightc>[TU]U5 [U5Ul[ U5Ul[ R"URUR5Ul UR5 gr) rrrrrxgenerator_predictionsr rrrgenerator_lm_headrrs r-rElectraForMaskedLM.__init__sR  #F+ %@%H"!#6+@+@&BSBS!T rcUR$rrrs r-get_output_embeddings(ElectraForMaskedLM.get_output_embeddings%%%rcXlgrr)rrs r-set_output_embeddings(ElectraForMaskedLM.set_output_embeddingss!0rrrr|rwrrrrrSrTrc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUbQ[ R "5nU"U RSURR5URS55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrrrxr r) r^rrrrr r rrrrrZ)rrrr|rwrrrrrSrTr}generator_sequence_outputprediction_scoresrrrs r-rElectraForMaskedLM.forwards(&1%! 667PQ 223DE  **,H-222t{{7M7MNPVP[P[\^P_`D'),CAB,GGF)-)9TGf$ Ev E$1??.99   rrrrr)rrrr_tied_weights_keysrrrrrrYrrrrrrrrrs@r-rKrKus55&1-11515/3,004)-,0/3&*4 ELL)4 !.4 !. 4 u||, 4 ELL) 4   -4 &4 $D>4 'tn4 d^4  uU\\"N2 34 4 rrKz Electra model with a token classification head on top. Both the discriminator and generator may be loaded into this model. cl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \R\ 44Sjj5r SrU=r$)ElectraForTokenClassificationicf>[TU]U5 URUl[U5UlUR b UR O UR n[R"U5Ul [R"URUR5Ul UR5 gr)rrrrrrrr rrrrr8rrs r-r&ElectraForTokenClassification.__init__s   ++#F+ )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rrrr|rwrrrrrSrTrc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUb<[ 5nU"U R SUR5UR S55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrrxr r) r^rrrr8r rrrrrZ)rrrr|rwrrrrrSrTrsrrtrrrs r-r%ElectraForTokenClassification.forwards$&1%?  ')HFKKDOO1 'tn1 d^1  uU\\"$99 :1 1 rrc^\rSrSr\rSrU4Sjr\SS\ \ RS\ \ RS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ S \ \ S\ \ S\ \\ R\44Sjj5rSrU=r$)ElectraForQuestionAnsweringirc>[TU]U5 URUl[U5Ul[ R "URUR5UlUR5 gr) rrrrrr rr qa_outputsrrs r-r$ElectraForQuestionAnswering.__init__sS   ++#F+ ))F$6$68I8IJ rrrr|rwrrstart_positions end_positionsrrSrTrc "U bU OURRn URUUUUUUU U S9n U Sn URU 5nUR SSS9unnUR S5R 5nUR S5R 5nSnUbUb[UR55S:aUR S5n[UR55S:aUR S5nURS5nURSU5nURSU5n[US9nU"X5nU"UU5nUU-S- nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$) N)rr|rwrrrrSrr rxr) ignore_indexr9)r start_logits end_logitsrrZ)r^rrrrMrrrrQrclampr rrrZ)rrrr|rwrrrrrrSrTrsrrtrr total_loss ignored_indexr start_lossend_lossrs r-r#ElectraForQuestionAnswering.forward!s&1%@ 'tn@ d^@  uU\\"$@@ A@ @ rrcl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \R\ 44Sjj5r SrU=r$)ElectraForMultipleChoiceiec>[TU]U5 [U5Ul[ U5Ul[ R"URS5Ul UR5 grn) rrrrrsequence_summaryr rrr8rrs r-r!ElectraForMultipleChoice.__init__gsM  #F+ 6v >))F$6$6: rrrr|rwrrrrrSrTrc \U bU OURRn UbURSOURSn Ub!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb1URSUR S5UR S55OSnUR UUUUUUUU U S9 n U Sn UR U 5nURU5nURSU 5nSnUb[5nU"UU5nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr rxrrrr) r^rrUrrrr'r8r rrrZ)rrrr|rwrrrrrSrT num_choicesrsr pooled_outputrtreshaped_logitsrrrs r-r ElectraForMultipleChoice.forwardqsX&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  '+ll ))%'/!5#'3 ' #6a8--o> / ++b+6  ')HOV4D%'*Eab*IIF)-)9TGf$ Ev E("5CC2==   r)r8rr'r)rrrrrrrrYrrrrrrrrrs@r-r%r%es-11515/3,004)-,0/3&*X ELL)X !.X !. X u||, X ELL) X   -X &X $D>X 'tnX d^X  uU\\"$== >X X rr%zS ELECTRA Model with a `language modeling` head on top for CLM fine-tuning. c"^\rSrSrS/rU4SjrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \ \ RS\ \ S\ \ S\ \ S\ \ S\\\ R\44Sjj5rSrSrU=r$)ElectraForCausalLMirc0>[TU]U5 UR(d[R S5 [ U5Ul[U5Ul[R"URUR5Ul UR5 g)NzOIf you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.`)rrrr?warningrrrxrr rrrr init_weightsrs r-rElectraForCausalLM.__init__sj    NNl m#F+ %@%H"!#6+@+@&BSBS!T rcUR$rrrs r-r(ElectraForCausalLM.get_output_embeddingsrrcXlgrr)rnew_embeddingss r-r(ElectraForCausalLM.set_output_embeddingss!/rrrr|rwrrrrrrRrrrSrTrc UbUOURRnU bSn URUUUUUUUUU U U U US9 nUSnURUR U55nSnU b*UR "UU 4SURR 0UD6nU(dU4USS-nUbU4U-$U$[UUURURURURS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, ElectraForCausalLM, ElectraConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/electra-base-generator") >>> config = ElectraConfig.from_pretrained("google/electra-base-generator") >>> config.is_decoder = True >>> model = ElectraForCausalLM.from_pretrained("google/electra-base-generator", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NF) rr|rwrrrrrRrrrSrTrrr )rrtrRrrZr[) r^rrrr loss_functionrrrRrrZr[)rrrr|rwrrrrrrRrrrSrTrrrr lm_lossrs r-rElectraForCausalLM.forwards,R&1%# UH/oRSTRUR55v M1 g7f)rN) index_selectrr)r+ past_statebeam_idxs r-r.4ElectraForCausalLM._reorder_cache..Bs1ncmU_--aZ=N=N1OPPcms7:)ra)rrRrAreordered_past layer_pasts ` r-_reorder_cache!ElectraForCausalLM._reorder_cache>s8)J ncmnn N*rr )NNNNNNNNNNNNNN)rrrrr rrrrrrYrrrrrrrrErrrs@r-r/r/s 55 &0-11515/3,0048<9=)-8<$(,0/3&*S ELL)S !.S !. S u||, S ELL) S   -S  ( 5S !) 6S &S "$u||"45S D>S $D>S 'tnS d^S " uU\\"$EE F#S S lrr/) r/rKr%rrrrrrro) discriminator)IrrrA dataclassesrtypingrrrrrrYtorch.utils.checkpointr torch.nnr r r activationsrr generationrmodeling_outputsrrrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrrconfiguration_electrar! get_loggerrr?roModulerqrrrr r!r,r2rIrlrxrrrrrrrrKrrr%r/__all__r*rr-rVs !99 AA1)   .ll 1   H %Od? ?FC299CN   !" 0ryy0h "))  BII S299SnZ RYYZ zbii&"))$*_**. :+: :8s)ssl 0`RYY`F P '=P  P f[ 2[ [ |H /H H V@ $:@ @ FO "8O O dd 5d d N r/r  rj r