fTh}1SrSSKrSSKrSSKJr SSKJrJrJrJ r SSK r SSK r SSK J r SSK JrJrJr SSKJr SS KJr SS KJrJrJrJrJrJrJrJrJr SS KJ r SS K!J"r"J#r#J$r$ SS K%J&r&J'r'J(r( SSK)J*r* \(RV"\,5r-"SS\ R\5r/"SS\ R\5r0"SS\ R\5r1S\00r2"SS\ R\5r3"SS\ R\5r4"SS\ R\5r5"SS\ R\5r6"SS \ R\5r7"S!S"\ R\5r8"S#S$\ R\5r9"S%S&\ R\5r:"S'S(\ R\5r;"S)S*\ R\5r<"S+S,\ R\5r=\'"S-S.\ 55r>\"S/S0\&55r?\'"S1S29"S3S4\>55r@\'"S5S29"S6S7\>55rA\'"S8S29"S9S:\>\55rB\'"S;S<\>55rC\'"S=S29"S>S?\>55rD\'"S@S29"SASB\>55rE\'"SCSD\>55rF\'"SESF\>55rG\'"SGSH\>55rH/SIQrIg)JzPyTorch ERNIE model.N) dataclass)ListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GenerationMixin) )BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging) ErnieConfigc^\rSrSrSrU4SjrS S\\RS\\RS\\RS\\RS\\RS \ S \R4S jjr S r U=r$)ErnieEmbeddings1zGConstruct the embeddings from word, position and token_type embeddings.c>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul URUl UR(a0[R"URUR 5Ul[R"UR UR S9Ul[R""UR$5Ul[)USS5UlUR-S[.R0"UR5R3S5SS9 UR-S [.R4"UR6R95[.R:S 9SS9 g) N) padding_idxepsposition_embedding_typeabsolute position_ids)rF) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings use_task_idtask_type_vocab_sizetask_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr(register_buffertorcharangeexpandzerosr*sizelongselfconfig __class__s `/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/ernie/modeling_ernie.pyr1ErnieEmbeddings.__init__4sf !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"!--   (* V5P5PRXRdRd(eD %f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  input_idsr- task_type_idsr* inputs_embedspast_key_values_lengthreturnc&UbUR5nOUR5SSnUSnUcURSS2XhU-24nUcv[US5(a-URSS2SU24n U R USU5n U nO8[ R "U[ RURRS9nUcURU5nURU5n X[-n URS:XaURU5n X- n UR(aPUc8[ R "U[ RURRS9nURU5nX- n URU 5n UR!U 5n U $)Nr+rr-rr/devicer))rIr*hasattrr-rGrErHrJrYr6r:r(r8r;r=r>rB)rLrRr-rSr*rTrU input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr: embeddingsr8r=s rOforwardErnieEmbeddings.forwardJs  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  !t-..*.*=*=a*n*M'3J3Q3QR]^_R`bl3m0!A!&[ SWSdSdSkSk!l   00;M $ : :> J":  ' ': 5"&":":<"H   -J   $ % KuzzRVRcRcRjRj k #'#<#<]#K  .J^^J/ \\*- rQ)r>rBr(r8r=r:r;r6)NNNNNr)__name__ __module__ __qualname____firstlineno____doc__r1rrE LongTensor FloatTensorintTensorr`__static_attributes__ __classcell__rNs@rOr"r"1sQ 01559483759&'0E,,-0!!1!120  0 01 0 u//0 0   1 12 0!$0 00rQr"cb^\rSrSrSU4SjjrS\R S\R 4SjrSS\R S\\RS\\RS \\RS \\RS \\ \ \RS \\ S\ \R 4S jjr Sr U=r$)ErnieSelfAttention~c>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eURUl[ URUR- 5UlURUR-Ul[R"URUR5Ul [R"URUR5Ul [R"URUR5Ul [R"UR5UlU=(d [#USS5UlUR$S:XdUR$S :XaGUR&Ul[R("S UR&-S - UR5UlUR,Ulg) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r(r) relative_keyrelative_key_queryr)r0r1r4num_attention_headsrZ ValueErrorriattention_head_size all_head_sizerLinearquerykeyvaluer@attention_probs_dropout_probrBrCr(r7r2distance_embedding is_decoderrLrMr(rNs rOr1ErnieSelfAttention.__init__s     : : :a ?PVXhHiHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD # ++rQxrVcUR5SSURUR4-nURU5nUR SSSS5$)Nr+rrvrr )rIrwryviewpermute)rLr new_x_shapes rOtranspose_for_scores'ErnieSelfAttention.transpose_for_scoressLffhsmt'?'?AYAY&ZZ FF; yyAq!$$rQ hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionscVURU5nUSLn U (aUbUSn USn UnGOU (aCURURU55n URURU55n UnOUbuURURU55n URURU55n [R "USU /SS9n [R "USU /SS9n O@URURU55n URURU55n URU5n USLn UR (aX4n[R"XRSS55nURS:XdURS:XGaU RSU RSnnU (aB[R"US- [RURS 9RSS5nO>[R"U[RURS 9RSS5n[R"U[RURS 9RSS5nUU- nUR!UUR"-S- 5nUR%U R&S 9nURS:Xa[R("S U U5nUU-nOHURS:Xa8[R("S U U5n[R("S U U5nUU-U-nU[*R,"UR.5- nUbX-n[0R2R5USS9nUR7U5nUbUU-n[R"UU 5nUR9SSSS 5R;5nUR=5SSUR>4-nURU5nU(aUU4OU4nUR (aUU4-nU$)Nrrrvdimr+rtrurXr.zbhld,lrd->bhlrzbhrd,lrd->bhlrr ) r|rr}r~rEcatrmatmul transposer(shapetensorrJrYrrFrr7tor/einsummathsqrtryr functionalsoftmaxrBr contiguousrIrz)rLrrrrrrrmixed_query_layeris_cross_attention key_layer value_layer query_layer use_cacheattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputss rOr`ErnieSelfAttention.forwards!JJ}5 3$> ."<&q)I(+K3N 11$((;P2QRI33DJJ?T4UVK3N  '11$((=2IJI33DJJ}4MNK >!#4i"@aHI))^A%6 $D!LK11$((=2IJI33DJJ}4MNK//0AB "$. ??(5N!<< 5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC 6G=/2mM] ?? 11GrQ) rzryrrBrr}r7rwr(r|r~NNNNNNF)rbrcrdrer1rErjrrrhrboolr`rkrlrms@rOroro~s,4%ell%u||%7;15=A>BDH,1c||c!!2!23cE--. c ((9(9: c !)):): ; c!uU->->'?!@Ac$D>c u|| ccrQrocz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)ErnieSelfOutputic(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNr&) r0r1rr{r4denser>r?r@rArBrKs rOr1ErnieSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rQr input_tensorrVcpURU5nURU5nURX-5nU$rrrBr>rLrrs rOr`ErnieSelfOutput.forward 5 =1  ]3 }'CD rQr>rrB rbrcrdrer1rErjr`rkrlrms@rOrr6> U\\RWR^R^rQreagerc.^\rSrSrSU4SjjrSrSS\RS\\RS\\RS\\RS\\RS \\ \ \RS \\ S \ \R4S jjr S r U=r$)ErnieAttentionic>[TU]5 [UR"XS9Ul[ U5Ul[5Ulg)Nr() r0r1ERNIE_SELF_ATTENTION_CLASSES_attn_implementationrLroutputset pruned_headsrs rOr1ErnieAttention.__init__s@ 01L1LM   &f- ErQc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrrr)lenrrLrwryrrr|r}r~rrrzunion)rLheadsindexs rO prune_headsErnieAttention.prune_heads"s  u:? 7 9900$))2O2OQUQbQb  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rQrrrrrrrrVc pURUUUUUUU5nURUSU5n U 4USS-n U $)Nrr)rLr) rLrrrrrrr self_outputsattention_outputrs rOr`ErnieAttention.forward4sWyy    ! "    ;;|A F#% QR(88rQ)rrrLrr)rbrcrdrer1rrErjrrhrrr`rkrlrms@rOrrs";*7;15=A>BDH,1||!!2!23E--.  ((9(9:  !)):): ; !uU->->'?!@A$D> u|| rQrcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)ErnieIntermediateiMc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) r0r1rr{r4intermediate_sizer isinstance hidden_actstrr intermediate_act_fnrKs rOr1ErnieIntermediate.__init__Ns` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $rQrrVcJURU5nURU5nU$rrrrLrs rOr`ErnieIntermediate.forwardVs& =1 00? rQrrrms@rOrrMs(9U\\ellrQrcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) ErnieOutputi]c(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) r0r1rr{rr4rr>r?r@rArBrKs rOr1ErnieOutput.__init__^s` YYv779K9KL f&8&8f>S>STzz&"<"<= rQrrrVcpURU5nURU5nURX-5nU$rrrs rOr`ErnieOutput.forwarddrrQrrrms@rOrr]rrQrc*^\rSrSrU4SjrSS\R S\\RS\\RS\\RS\\RS\\ \ \RS \\ S \ \R 4S jjr S r S r U=r$) ErnieLayerilct>[TU]5 URUlSUl[ U5UlUR UlURUlUR(a.UR (d[US35e[ USS9Ul [U5Ul [U5Ul g)Nrz> should be used as a decoder model if cross attention is addedr)r)r0r1chunk_size_feed_forward seq_len_dimr attentionradd_cross_attentionrxcrossattentionr intermediaterrrKs rOr1ErnieLayer.__init__ms '-'E'E$'/ ++#)#=#=  # #?? D6)g!hii"0Q["\D -f5!&) rQrrrrrrrrVc UbUSSOSnURUUUUUS9n U Sn UR(a U SSn U Sn OU SSn Sn UR(aZUbW[US5(d[SUS35eUbUS SOSnUR U UUUUUU5nUSn XSS-n USn W U -n [ UR URURU 5nU4U -n UR(aU W 4-n U $) Nrv)rrrrr+rz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrZrxrrfeed_forward_chunkrr)rLrrrrrrrself_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputs rOr`ErnieLayer.forward{s}:H9S>"1#5Y] !%   /3 "0" 2!4 ??,Qr2G 6r : ,QR0G'+$ ??4@4!122 =dVDDD @N?Yrs(;_c %&*&9&9 %&)!' # 7q9 " ==G,C2+F ( 14P P 0  # #T%A%A4CSCSUe  /G+ ??!2 44GrQcJURU5nURX!5nU$r)rr)rLrintermediate_outputrs rOrErnieLayer.feed_forward_chunks)"//0@A{{#6I rQ)rrrrrrrrr)rbrcrdrer1rErjrrhrrr`rrkrlrms@rOrrls *"7;15=A>BDH,1?||?!!2!23?E--. ? ((9(9: ? !)):): ; ?!uU->->'?!@A?$D>? u|| ?BrQrcR^\rSrSrU4SjrSS\R S\\RS\\RS\\RS\\RS\\ \ \RS \\ S \\ S \\ S \\ S \ \ \R \ 44Sjjr SrU=r$) ErnieEncoderic>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snf)NF) r0r1rMr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)rLrM_rNs rOr1ErnieEncoder.__init__sR  ]]fF^F^@_#`@_1Jv$6@_#`a &+#$asA&rrrrrpast_key_valuesrroutput_hidden_states return_dictrVc 8U (aSOSn U(aSOSn U(aURR(aSOSn UR(a/UR(aU(a[R S5 SnU(aSOSn[ UR5HunnU (aX4-n UbX?OSnUbXoOSnUR(a4UR(a#URURUUUUUUU5nOU"UUUUUUU5nUSnU(a UUS4- nU(dMU US4-n URR(dMU US4-n M U (aX4-n U (d[SUUU U U 455$[UUU U U S 9$) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frr+rrvc30# UH nUcMUv M g7frr).0vs rO 'ErnieEncoder.forward.. s" As )last_hidden_staterr attentionscross_attentions) rMrrtraininglogger warning_once enumerater_gradient_checkpointing_func__call__tupler)rLrrrrrrrrrrall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_maskr layer_outputss rOr`ErnieEncoder.forwards#7BD$5b4%64;;;Z;Zr`d  & &4==##p" #,R$(4OA|#$58H$H!.7.CilO3B3N_/TXN**t}} $ A A ))!"#)*"% ! !-!"#)*"%! *!,M"}R'8&::"  &9]1=MBEI$(,1/4&*S ||S !!2!23S E--. S ((9(9: S !)):): ; S "%e.?.?(@"ABS D>S $D>S 'tnS d^S  uU\\"$MM NS S rQr cb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $) ErniePooleri!c>[TU]5 [R"URUR5Ul[R "5Ulgr)r0r1rr{r4rTanh activationrKs rOr1ErniePooler.__init__"s9 YYv1163E3EF '')rQrrVc\USS2S4nURU5nURU5nU$)Nr)rr4)rLrfirst_token_tensor pooled_outputs rOr`ErniePooler.forward's6+1a40 #56  6 rQ)r4rrrms@rOr1r1!s($ U\\ellrQr1cb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)ErniePredictionHeadTransformi1cp>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURS9Ul gr) r0r1rr{r4rrrrr transform_act_fnr>r?rKs rOr1%ErniePredictionHeadTransform.__init__2s~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrQrrVclURU5nURU5nURU5nU$r)rr=r>rs rOr`$ErniePredictionHeadTransform.forward;s4 =1 --m< }5 rQ)r>rr=rrms@rOr;r;1s)UU\\ellrQr;c4^\rSrSrU4SjrSrSrSrU=r$)ErnieLMPredictionHeadiCcH>[TU]5 [U5Ul[R "UR URSS9Ul[R"[R"UR55Ul URURl g)NF)bias) r0r1r; transformrr{r4r3decoder ParameterrErHrDrKs rOr1ErnieLMPredictionHead.__init__Dsm 5f=yy!3!3V5F5FUS LLV->->!?@ !II rQc:URURlgr)rDrFrLs rO _tie_weights"ErnieLMPredictionHead._tie_weightsQs II rQcJURU5nURU5nU$r)rErFrs rOr`ErnieLMPredictionHead.forwardTs$}5  ]3 rQ)rDrFrE) rbrcrdrer1rKr`rkrlrms@rOrBrBCs &&rQrBcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)ErnieOnlyMLMHeadi[cB>[TU]5 [U5Ulgr)r0r1rB predictionsrKs rOr1ErnieOnlyMLMHead.__init__\s 08rQsequence_outputrVc(URU5nU$rrR)rLrTprediction_scoress rOr`ErnieOnlyMLMHead.forward`s ,,_=  rQrVrrms@rOrPrP[s(9!u||! !!rQrPc.^\rSrSrU4SjrSrSrU=r$)ErnieOnlyNSPHeadifcn>[TU]5 [R"URS5UlgNrv)r0r1rr{r4seq_relationshiprKs rOr1ErnieOnlyNSPHead.__init__gs'  " &*<*[TU]5 [U5Ul[R "UR S5Ulgr\)r0r1rBrRrr{r4r]rKs rOr1ErniePreTrainingHeads.__init__rs4 08 " &*<*fill_)rLmodules rO _init_weights"ErniePreTrainedModel._init_weightss fbii ( ( MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).rQrN) rbrcrdrer config_classbase_model_prefixsupports_gradient_checkpointingrxrkrrQrOrlrl}sL&*#*rQrlc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\ \R\ S'Sr\\ \R\ S'S rg) ErnieForPreTrainingOutputia  Output type of [`ErnieForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossprediction_logitsseq_relationship_logitsrrr)rbrcrdrerfrrrErh__annotations__rrrrrrkrrQrOr~r~s~2)-D(5$$ %,59x 1 129;?Xe&7&78?8[TU]U5 Xl[U5Ul[ U5UlU(a [U5OSUlUR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r0r1rMr"r_r encoderr1pooler post_init)rLrMadd_pooling_layerrNs rOr1ErnieModel.__init__sK   )&1#F+ ->k&)D  rQc.URR$rr_r6rJs rOget_input_embeddingsErnieModel.get_input_embeddingss...rQc$XRlgrr)rLr~s rOset_input_embeddingsErnieModel.set_input_embeddingss */'rQcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rLheads_to_prunerrs rO _prune_headsErnieModel._prune_headss< +002LE LL  u % / / ; ;E B3rQrRrr-rSr*rrTrrrrrrrrVc U bU OURRn U bU OURRn UbUOURRnURR(aU bU OURR n OSn UbUb [ S5eUb"URX5 UR5nO"UbUR5SSnO [ S5eUunnUb URO URnU bU SSRSOSnUc[R"UUU-4US9nUcs[URS 5(a4URRSS2SU24nUR!UU5nUnO$[R""U[R$US 9nUR'X/5nURR(aEUbBUR5unnnUU4nU c[R"UUS9n UR)U 5nOSnUR+X`RR,5nURUUUUUUS 9nUR/UUUUUU U U U US 9 nUSnUR0bUR1U5OSnU(d UU4US S-$[3UUUR4UR6UR8UR:S9$) task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] NFzDYou cannot specify both input_ids and inputs_embeds at the same timer+z5You have to specify either input_ids or inputs_embedsrrv)rYr-rX)rRr*r-rSrTrU) rrrrrrrrrr)r pooler_outputrrrr)rMrruse_return_dictrrrx%warn_if_padding_and_no_attention_maskrIrYrrEonesrZr_r-rGrHrJget_extended_attention_maskinvert_attention_mask get_head_maskrrrrrrrr) rLrRrr-rSr*rrTrrrrrrrr[ batch_sizer\rYrUr]r^extended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputsrTr8s rOr`ErnieModel.forwards22C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, J%.%:!! @T@TDSC^!3A!6!u $D>u 'tnu d^u uU\\"$PP Q!u u rQrz Ernie Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. c^\rSrSrSS/rU4SjrSrSr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \ RS\ \ S\ \ S\ \ S\ \\ R\44Sjj5rSrU=r$)ErnieForPreTrainingi^cls.predictions.decoder.biascls.predictions.decoder.weightc>[TU]U5 [U5Ul[ U5UlUR 5 gr)r0r1rrmreclsrrKs rOr1ErnieForPreTraining.__init__hs4  ' (0 rQcBURRR$rrrRrFrJs rOget_output_embeddings)ErnieForPreTraining.get_output_embeddingsrxx##+++rQcXRRlURURRlgrrrRrFrDrLnew_embeddingss rOset_output_embeddings)ErnieForPreTraining.set_output_embeddingsv*'5$$2$7$7!rQrRrr-rSr*rrTlabelsnext_sentence_labelrrrrVc U bU OURRn URUUUUUUUU U U S9 n U SSupURX5unnSnUbvU bs[ 5nU"UR SURR 5UR S55nU"UR SS5U R S55nUU-nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$)aj task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, ErnieForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForPreTraining.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` N rr-rSr*rrTrrrrvr+)rrrrr) rMrrmrr rr3r~rr)rLrRrr-rSr*rrTrrrrrrrTr8rWra total_lossloss_fctmasked_lm_lossnext_sentence_lossrs rOr`ErnieForPreTraining.forwardzsFb&1%S 'tnS d^S  uU\\"$== >S S rQrzQ Ernie Model with a `language modeling` head on top for CLM fine-tuning. c$^\rSrSrSS/rU4SjrSrSr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \ RS\ \ RS\ \ \ RS\ \ S\ \ S\ \ S\ \ S\\\ R\44 Sjj5rSrSrU=r$)ErnieForCausalLMirrc>[TU]U5 UR(d[R S5 [ USS9Ul[U5UlUR5 g)NzMIf you want to use `ErnieForCausalLM` as a standalone, add `is_decoder=True.`Fr r0r1rr!warningrrmrPrrrKs rOr1ErnieForCausalLM.__init__sL    NNj k%@ #F+ rQcBURRR$rrrJs rOr&ErnieForCausalLM.get_output_embeddingsrrQcXRRlURURRlgrrrs rOr&ErnieForCausalLM.set_output_embeddingsrrQrRrr-rSr*rrTrrrrrrrrrVc UbUOURRnU bSn URUUUUUUUUU U U U UUS9nUSnURU5nSnU b*UR"UU 4SURR 0UD6nU(dU4USS-nUbU4U-$U$[ UUURURURURS9$)a@ task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` NF) rr-rSr*rrTrrrrrrrrr3rv)rlogitsrrrr) rMrrmr loss_functionr3rrrrr)rLrRrr-rSr*rrTrrrrrrrrkwargsrrTrWlm_lossrs rOr`ErnieForCausalLM.forwards!>&1%# UH/oRSTRUR55v M1 g7f)rN) index_selectrrY)r past_statebeam_idxs rOr2ErnieForCausalLM._reorder_cache..As1ncmU_--aZ=N=N1OPPcms7:)r&)rLrrreordered_past layer_pasts ` rO_reorder_cacheErnieForCausalLM._reorder_cache=s8)J ncmnn N*rQr)NNNNNNNNNNNNNNN)rbrcrdrerr1rrrrrErjrrrrrr`rrkrlrms@rOrrs 9:Z[ ,8-1151504/3,0048<9=)-8<$(,0/3&*!J ELL)J !.J !. J   - J u||, J ELL)J   -J  ( 5J !) 6J &J "$u||"45J D>J $D>J 'tnJ d^!J $ uU\\"$EE F%J J ZrQrc ^\rSrSrSS/rU4SjrSrSr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \ RS\ \ RS\ \ S\ \ S\ \ S\ \\ R\44Sjj5rSSjr\S\ 4Sj5rSrU=r$)ErnieForMaskedLMiFrrc>[TU]U5 UR(a[R S5 [ USS9Ul[U5UlUR5 g)NzlIf you want to use `ErnieForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FrrrKs rOr1ErnieForMaskedLM.__init__KsR     NN1   %@ #F+ rQcBURRR$rrrJs rOr&ErnieForMaskedLM.get_output_embeddings[rrQcXRRlURURRlgrrrs rOr&ErnieForMaskedLM.set_output_embeddings_rrQrRrr-rSr*rrTrrrrrrrVcU bU OURRn URUUUUUUUUU U U U S9 nUSnURU5nSnU bF[ 5nU"UR SURR 5U R S55nU (dU4USS-nUbU4U-$U$[UUURURS9$)a# task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` N) rr-rSr*rrTrrrrrrr+rvrrrr) rMrrmrr rr3rrr)rLrRrr-rSr*rrTrrrrrrrrTrWrrrs rOr`ErnieForMaskedLM.forwardcs:&1% &IIrQcg)z Legacy correction: ErnieForMaskedLM can't call `generate()` from `GenerationMixin`, even though it has a `prepare_inputs_for_generation` method. Fr)rs rO can_generateErnieForMaskedLM.can_generates rQr) NNNNNNNNNNNNNr)rbrcrdrerr1rrrrrErjrrrrr`r classmethodrrkrlrms@rOrrFs8:Z[  ,8-1151504/3,0048<9=)-,0/3&*> ELL)> !.> !. >   - > u||, > ELL)>   ->  ( 5> !) 6> &> $D>> 'tn> d^>  uU\\"N2 3> > BJ TrQrzU Ernie Model with a `next sentence prediction (classification)` head on top. c^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \R\ 44Sjj5r SrU=r$)ErnieForNextSentencePredictionic>[TU]U5 [U5Ul[ U5UlUR 5 gr)r0r1rrmrZrrrKs rOr1'ErnieForNextSentencePrediction.__init__s4  ' #F+ rQrRrr-rSr*rrTrrrrrVc SU ;a,[R"S[5 U RS5nU bU OURR n UR UUUUUUUU U U S9 n U SnURU5nSnUb2[5nU"URSS5URS55nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$) a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` rzoThe `next_sentence_label` argument is deprecated and will be removed in a future version, use `labels` instead.Nrrr+rvr) warningswarn FutureWarningpoprMrrmrr rrrr)rLrRrr-rSr*rrTrrrrrrr8seq_relationship_scoresrrrs rOr`&ErnieForNextSentencePrediction.forwards ` !F * MM%  ZZ 56F%0%X 'tnX d^X  uU\\"$?? @X X rQrz Ernie Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \R\ 44Sjj5r SrU=r$)ErnieForSequenceClassificationi*cr>[TU]U5 URUlXl[ U5UlUR b UR O URn[R"U5Ul [R"URUR5Ul UR5 gr)r0r1 num_labelsrMrrmclassifier_dropoutrArr@rBr{r4 classifierrrLrMr rNs rOr1'ErnieForSequenceClassification.__init__2s   ++ ' )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rQrRrr-rSr*rrTrrrrrVc TU bU OURRn URUUUUUUUU U U S9 n U Sn URU 5n UR U 5nSnUGbURR cUR S:XaSURlOoUR S:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5nUR S:Xa&U"UR5UR55nOU"X5nOURR S:Xa=[5nU"URSUR 5URS55nO,URR S:Xa[5nU"X5nU (dU4U SS-nUbU4U-$U$[!UUU R"U R$S 9$) a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr regressionsingle_label_classificationmulti_label_classificationr+rvr)rMrrmrBr  problem_typer r/rErJrir squeezer rr rrr)rLrRrr-rSr*rrTrrrrrr8rrrrs rOr`&ErnieForSequenceClassification.forwardAs4&1%L 'tnL d^L  uU\\"$<< =L L rQrc^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \R\ 44Sjj5r SrU=r$)ErnieForMultipleChoiceic0>[TU]U5 [U5UlURb URO UR n[ R"U5Ul[ R"URS5Ul UR5 g)Nr) r0r1rrmr rArr@rBr{r4r rr s rOr1ErnieForMultipleChoice.__init__su  ' )/)B)B)NF % %TZTnTn zz"45 ))F$6$6: rQrRrr-rSr*rrTrrrrrVc ^U bU OURRn UbURSOURSn Ub!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb1URSUR S5UR S55OSnUR UUUUUUUU U U S9 n U SnUR U5nURU5nURSU 5nSnUb[5nU"UU5nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)aQ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) task_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr+rrrvr) rMrrrrIrmrBr r rrr)rLrRrr-rSr*rrTrrrr num_choicesrr8rreshaped_logitsrrrs rOr`ErnieForMultipleChoice.forwardsd&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  ** ))'%'/!5#     ]3 / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   rQ)r rBrmr)rbrcrdrer1rrrErjrrrrr`rkrlrms@rOrrs+ -1151504/3,004)-,0/3&*_ ELL)_ !._ !. _   - _ u||, _ ELL)_   -_ &_ $D>_ 'tn_ d^_  uU\\"$== >_ _ rQrc^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \R\ 44Sjj5r SrU=r$)ErnieForTokenClassificationicd>[TU]U5 URUl[USS9UlUR b UR O UR n[R"U5Ul [R"URUR5Ul UR5 gNFr)r0r1r rrmr rArr@rBr{r4r rr s rOr1$ErnieForTokenClassification.__init__s   ++%@ )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rQrRrr-rSr*rrTrrrrrVc U bU OURRn URUUUUUUUU U U S9 n U Sn URU 5n UR U 5nSnUb<[ 5nU"UR SUR5UR S55nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)al task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr+rvr) rMrrmrBr r rr rrr)rLrRrr-rSr*rrTrrrrrrTrrrrs rOr`#ErnieForTokenClassification.forwards0&1%9 'tn9 d^9  uU\\"$99 :9 9 rQrc^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\RS \\ S \\ S\\ S\ \ \R\ 44Sjj5r SrU=r$)ErnieForQuestionAnsweringiRc>[TU]U5 URUl[USS9Ul[ R "URUR5UlUR5 gr!) r0r1r rrmrr{r4 qa_outputsrrKs rOr1"ErnieForQuestionAnswering.__init__UsU   ++%@ ))F$6$68I8IJ rQrRrr-rSr*rrTstart_positions end_positionsrrrrVc (U bU OURRn URUUUUUUUU U U S9 n U SnURU5nUR SSS9unnUR S5R 5nUR S5R 5nSnUbU b[UR55S:aUR S5n[U R55S:aU R S5n URS5nURSU5nU RSU5n [US9nU"UU5nU"UU 5nUU-S- nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS 9$) rNrrrr+r) ignore_indexrv)r start_logits end_logitsrr)rMrrmr(splitrrrrIclampr rrr)rLrRrr-rSr*rrTr*r+rrrrrTrr.r/r ignored_indexr start_lossend_lossrs rOr`!ErnieForQuestionAnswering.forward_s.&1%G 'tnG d^G  uU\\"$@@ AG G rQr&) rrrrrr&rrrrl)Jrfrr dataclassesrtypingrrrrrEtorch.utils.checkpointrtorch.nnr r r activationsr generationrmodeling_outputsrrrrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrrconfiguration_ernier get_loggerrbr!Moduler"rorrrrrrr r1r;rBrPrZrerlr~rrrrrrrrr&__all__rrQrOrDs !// AA!)   .ll99,   H %IbiiIZCCN bii    0RYY0h   ")) SSnZ 299Z | "))  299$BII0!ryy!&ryy& 9BII 9*?**, : : :B Y %Y  Y x j .j  j Z m+_m  m`t+ttn d %9d  d N ^ %9^  ^ Bo 1o o dJ "6J J ZT 4T T n rQ