fTh7@SrSSKrSSKrSSKrSSKJr SSKJrJrJ r J r SSK r SSK r SSK Jr SSK Jr SSKJrJrJr SS KJr SS KJr SS KJrJr SS KJrJrJrJrJ r J!r!J"r"J#r#J$r$ SS K%J&r& SSK'J(r(J)r)J*r* SSK+J,r,J-r-J.r.J/r/ SSK0J1r1 \/Rd"\35r4Sr5"SS\Rl5r7"SS\Rl5r8"SS\85r9"SS\Rl5r:\8\9S.r;"SS\Rl5r<"SS\Rl5r="S S!\Rl5r>"S"S#\Rl5r?"S$S%\Rl5r@"S&S'\Rl5rA"S(S)\Rl5rB"S*S+\Rl5rC"S,S-\Rl5rD"S.S/\Rl5rE"S0S1\Rl5rF\-"S2S3\&55rG\"S4S5\,55rH\-"S6S79"S8S9\G55rI\-"S:S79"S;S<\G55rJ\-"S=S79"S>S?\G\55rK\-"S@SA\G55rL\-"SBS79"SCSD\G55rM\-"SES79"SFSG\G55rN\-"SHSI\G55rO\-"SJSK\G55rP\-"SLSM\G55rQ/SNQrRg)OzPyTorch BERT model.N) dataclass)ListOptionalTupleUnion)version)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GenerationMixin)#_prepare_4d_attention_mask_for_sdpa*_prepare_4d_causal_attention_mask_for_sdpa) )BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringget_torch_versionlogging) BertConfigc SSKnSSKnSSKn[ RRU5n[RSU35 URRU5n/n/n UH]up[RSU SU 35 URRXj5n URU 5 U RU 5 M_ [X5GHupU RS5n [!SU 55(a)[RS SR#U 535 MWUn U HnUR%S U5(aURS U5nOU/nUSS :Xd USS :Xa ['U S5n OZUSS:Xd USS:Xa ['U S5n O;USS:Xa ['U S5n O%USS:Xa ['U S5n O['XS5n [+U5S:dM[-US5nU Un M WSSS:Xa ['U S5n OUS :XaUR/U 5n U R0U R0:wa&[3SU R0SU R0S35e[RSU 35 [6R8"U 5U lGM U$![a [R S5 ef=f![(a, [RS SR#U 535 GMf=f![2a1nU=R4U R0U R04- sleSnAff=f)z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape /c3,# UH nUS;v M g7f))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0ns ^/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/bert/modeling_bert.py *load_tf_weights_in_bert..Rs  n nsz Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifierr#i _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendzipsplitanyjoin fullmatchgetattrAttributeErrorlenint transposeshape ValueErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathr>nptftf_path init_varsnamesarraysnamerVarraypointerm_name scope_namesnumes r0load_tf_weights_in_bertrl5s ggoo01G KK8 BC''0I E F  (l5'BC&&w5 T e ! 5) zz#      KK)CHHTN#34 5 F||,f55 hhy&9 %h 1~)[^w-F!'84Q=0KNf4L!'62Q#33!'84Q7*!'<8%g1~>G;1$+a.)!#,+, #$<= (gx0G x LL'E }} + >'--@QRWR]R]Q^^i!jkk,  078''. Y*Z L   Q   Z&KK)CHHTN+; <=  FFw}}ekk2 2F  s6 J'K ,AL'!K 1LL L?,L::L?c ^\rSrSrSrU4SjrS S\\RS\\RS\\RS\\RS\ S \R4 S jjr S r U=r$) BertEmbeddings~zGConstruct the embeddings from word, position and token_type embeddings.c.>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5Ul[#USS5UlUR'S[(R*"UR5R-S5SS9 UR'S [(R."UR0R35[(R4S 9SS9 g) N) padding_idxepsposition_embedding_typeabsolute position_ids)r#F) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutrQrtregister_bufferrYarangeexpandzerosrvsizelongselfr] __class__s r0r}BertEmbeddings.__init__s/ !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  input_idsryrv inputs_embedspast_key_values_lengthreturncdUbUR5nOUR5SSnUSnUcURSS2XWU-24nUcv[US5(a-URSS2SU24nUR USU5n U nO8[ R "U[ RURRS9nUcURU5nURU5n XJ-n URS:XaURU5n X- n URU 5n URU 5n U $)Nrwr#ryrr{deviceru)rrvhasattrryrrYrrrrrrtrrr) rrryrvrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr embeddingsrs r0forwardBertEmbeddings.forwardsC  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  !t-..*.*=*=a*n*M'3J3Q3QR]^_R`bl3m0!A!&[ SWSdSdSkSk!l   00;M $ : :> J":  ' ': 5"&":":<"H   -J^^J/ \\*- r)rrrtrrr)NNNNr)__name__ __module__ __qualname____firstlineno____doc__r}rrY LongTensor FloatTensorrTTensorr__static_attributes__ __classcell__rs@r0rnrn~sQ *15593759&' 'E,,-'!!1!12'u//0 '   1 12 ' !$ ' ''rrncb^\rSrSrSU4SjjrS\R S\R 4SjrSS\R S\\RS\\RS \\RS \\RS \\ \ \RS \\ S\ \R 4S jjr Sr U=r$)BertSelfAttentionc>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eURUl[ URUR- 5UlURUR-Ul[R"URUR5Ul [R"URUR5Ul [R"URUR5Ul [R"UR5UlU=(d [#USS5UlUR$S:XdUR$S :XaGUR&Ul[R("S UR&-S - UR5UlUR,Ulg) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rtru relative_keyrelative_key_queryr<r#)r|r}rnum_attention_headsrrWrTattention_head_size all_head_sizer Linearquerykeyvaluerattention_probs_dropout_probrrQrtrr~distance_embedding is_decoderrr]rtrs r0r}BertSelfAttention.__init__s     : : :a ?PVXhHiHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD # ++rxrcUR5SSURUR4-nURU5nUR SSSS5$)Nrwrr<r#r )rrrviewpermute)rr new_x_shapes r0transpose_for_scores&BertSelfAttention.transpose_for_scoressLffhsmt'?'?AYAY&ZZ FF; yyAq!$$r hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionscVURU5nUSLn U (aUbUSn USn UnGOU (aCURURU55n URURU55n UnOUbuURURU55n URURU55n [R "USU /SS9n [R "USU /SS9n O@URURU55n URURU55n URU5n USLn UR (aX4n[R"XRSS55nURS:XdURS:XGaU RSU RSnnU (aB[R"US- [RURS 9RSS5nO>[R"U[RURS 9RSS5n[R"U[RURS 9RSS5nUU- nUR!UUR"-S- 5nUR%U R&S 9nURS:Xa[R("S U U5nUU-nOHURS:Xa8[R("S U U5n[R("S U U5nUU-U-nU[*R,"UR.5- nUbX-n[0R2R5USS9nUR7U5nUbUU-n[R"UU 5nUR9SSSS 5R;5nUR=5SSUR>4-nURU5nU(aUU4OU4nUR (aUU4-nU$)Nrr#r<dimrwrrrrzzbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rrrrrYcatrmatmulrUrtrVtensorrrrrrrtor{einsummathsqrtrr functionalsoftmaxrr contiguousrr)rrrrrrrrmixed_query_layeris_cross_attention key_layer value_layer query_layer use_cacheattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputss r0rBertSelfAttention.forwards!JJ}5 3$> ."<&q)I(+K3N 11$((;P2QRI33DJJ?T4UVK3N  '11$((=2IJI33DJJ}4MNK >!#4i"@aHI))^A%6 $D!LK11$((=2IJI33DJJ}4MNK//0AB "$. ??(5N!<< 5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC 6G=/2mM] ?? 11Gr) rrrrrrrrrtrrNNNNNNF)rrrrr}rYrrrrrboolrrrrs@r0rrs,4%ell%u||%7;15=A>BDH,1c||c!!2!23cE--. c ((9(9: c !)):): ; c!uU->->'?!@Ac$D>c u|| ccrrc.^\rSrSrS U4SjjrSS\R S\\R S\\RS\\RS\\RS\\ \ \RS \\ S \ \R 4U4S jjjr S r U=r $)BertSdpaSelfAttentioniDc>[TU]XS9 URUl[R "[ 55[R "S5:Ulg)Nrtz2.2.0)r|r}r dropout_probrparser!require_contiguous_qkvrs r0r}BertSdpaSelfAttention.__init__EsE Q"??&-mm4E4G&H7==Y`Ka&a#rrrrrrrrrc >URS:wd U(dUb*[RS5 [TU]UUUUUUU5$UR 5upn UR URU55n USLn U (aUOUn U (aUOUnU (a/U(a(USRSU RS:XaUupO~UR URU 55nUR URU 55nUb;U (d4[R"USU/SS9n[R"USU/SS9nUR(aX4nUR(aMU RR S:Xa3Ub0U R#5n UR#5nUR#5nUR(aU (d UcU S:aSOS n[R$R&R)U UUUUR*(a UR,OS US 9nUR/SS5nUR1XUR25nU4nUR(aUU4-nU$) NruaBertSdpaSelfAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support non-absolute `position_embedding_type` or `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rr<r#rcudaTF) attn_mask dropout_p is_causal)rtrB warning_oncer|rrrrrVrrrYrrrrtyperr rscaled_dot_product_attentiontrainingrrUreshaper)rrrrrrrrbsztgt_len_rrcurrent_statesrrr attn_outputrrs r0rBertSdpaSelfAttention.forwardKsZ  ' ': 59JiNc   H 7?%&! (,,.a// =0IJ 3$>2D.-3E/> .^A5F5L5LQ5OSaSgSghiSj5j%3 "I{11$((>2JKI33DJJ~4NOK)2D!II~a'8)&D!L #ii):K(HaP ??(5N  & &;+=+=+B+Bf+LQ_Qk%002K!,,.I%002KOO,>>CY^ehi^iDot hh))FF   $+/==d''c G "++Aq1 !))#8J8JK . ?? 11Gr)rrrr)rrrrr}rYrrrrrrrrrs@r0rrDsb2615=A>BDH,1[||[!.[E--. [ ((9(9: [ !)):): ; [!uU->->'?!@A[$D>[ u|| [[rrcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)BertSelfOutputic(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNrr) r|r}r rrdenserrrrrrs r0r}BertSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rr input_tensorrcpURU5nURU5nURX-5nU$rrrrrrrs r0rBertSelfOutput.forward5 =1  ]3 }'CD rrrr rrrrr}rYrrrrrs@r0rr6> U\\RWR^R^rr)eagersdpac.^\rSrSrSU4SjjrSrSS\RS\\RS\\RS\\RS\\RS \\ \ \RS \\ S \ \R4S jjr S r U=r$) BertAttentionic>[TU]5 [UR"XS9Ul[ U5Ul[5Ulg)Nr) r|r}BERT_SELF_ATTENTION_CLASSES_attn_implementationrroutputset pruned_headsrs r0r}BertAttention.__init__s@ /0K0KL   %V, Erc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrr#r)rSrrrrr*rrrrr(rrunion)rheadsindexs r0 prune_headsBertAttention.prune_headss  u:? 7 9900$))2O2OQUQbQb  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rrrrrrrrrc pURUUUUUUU5nURUSU5n U 4USS-n U $)Nrr#)rr() rrrrrrrr self_outputsattention_outputrs r0rBertAttention.forwardsWyy    ! "    ;;|A F#% QR(88r)r(r*rrr)rrrrr}r0rYrrrrrrrrrs@r0r$r$s";*7;15=A>BDH,1||!!2!23E--.  ((9(9:  !)):): ; !uU->->'?!@A$D> u|| rr$cb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)BertIntermediateic>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) r|r}r rrintermediate_sizer isinstance hidden_actstrrintermediate_act_fnrs r0r}BertIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $rrrcJURU5nURU5nU$rrr=rrs r0rBertIntermediate.forwards& =1 00? rr@rrs@r0r7r7s(9U\\ellrr7cz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) BertOutputic(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) r|r}r rr9rrrrrrrrs r0r}BertOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rrrrcpURU5nURU5nURX-5nU$rrrs r0rBertOutput.forwardrrrrrs@r0rDrDr rrDc*^\rSrSrU4SjrSS\R S\\RS\\RS\\RS\\RS\\ \ \RS \\ S \ \R 4S jjr S r S r U=r$) BertLayeri ct>[TU]5 URUlSUl[ U5UlUR UlURUlUR(a.UR (d[US35e[ USS9Ul [U5Ul [U5Ul g)Nr#z> should be used as a decoder model if cross attention is addedrur)r|r}chunk_size_feed_forward seq_len_dimr$ attentionradd_cross_attentionrWcrossattentionr7 intermediaterDr(rs r0r}BertLayer.__init__s '-'E'E$&v. ++#)#=#=  # #?? D6)g!hii"/PZ"[D ,V4 ( rrrrrrrrrc UbUSSOSnURUUUUUS9n U Sn UR(a U SSn U Sn OU SSn Sn UR(aZUbW[US5(d[SUS35eUbUS SOSnUR U UUUUUU5nUSn XSS-n USn W U -n [ UR URURU 5nU4U -n UR(aU W 4-n U $) Nr<)rrrr#rwrPz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rNrrrWrPrfeed_forward_chunkrLrM)rrrrrrrrself_attn_past_key_valueself_attention_outputsr4rpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputs r0rBertLayer.forwards}:H9S>"1#5Y] !%   /3 "0" 2!4 ??,Qr2G 6r : ,QR0G'+$ ??4@4!122 =dVDDD @N?Yrs(;_c %&*&9&9 %&)!' # 7q9 " ==G,C2+F ( 14P P 0  # #T%A%A4CSCSUe  /G+ ??!2 44GrcJURU5nURX!5nU$r)rQr()rr4intermediate_outputr[s r0rTBertLayer.feed_forward_chunk]s)"//0@A{{#6I r)rOrNrLrPrQrr(rMr)rrrrr}rYrrrrrrrTrrrs@r0rJrJ s )"7;15=A>BDH,1?||?!!2!23?E--. ? ((9(9: ? !)):): ; ?!uU->->'?!@A?$D>? u|| ?BrrJcR^\rSrSrU4SjrSS\R S\\RS\\RS\\RS\\RS\\ \ \RS \\ S \\ S \\ S \\ S \ \ \R \ 44Sjjr SrU=r$) BertEncodericc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snf)NF) r|r}r]r ModuleListrangenum_hidden_layersrJlayergradient_checkpointing)rr]rrs r0r}BertEncoder.__init__dsR  ]]uVE]E]?^#_?^!If$5?^#_` &+#$`sA&rrrrrpast_key_valuesrroutput_hidden_states return_dictrc 8U (aSOSn U(aSOSn U(aURR(aSOSn UR(a/UR(aU(a[R S5 SnU(aSOSn[ UR5HunnU (aX4-n UbX?OSnUbXoOSnUR(a4UR(a#URURUUUUUUU5nOU"UUUUUUU5nUSnU(a UUS4- nU(dMU US4-n URR(dMU US4-n M U (aX4-n U (d[SUUU U U 455$[UUU U U S 9$) Nr-zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frrwr#r<c30# UH nUcMUv M g7frr-)r.vs r0r1&BertEncoder.forward..s" As )last_hidden_staterir attentionscross_attentions) r]rOrgr rBr enumeraterf_gradient_checkpointing_func__call__tupler)rrrrrrrirrrjrkall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_maskr layer_outputss r0rBertEncoder.forwardjs#7BD$5b4%64;;;Z;Zr`d  & &4==##p" #,R$(4OA|#$58H$H!.7.CilO3B3N_/TXN**t}} $ A A ))!"#)*"% ! !-!"#)*"%! *!,M"}R'8&::"  &9]1=MBEI$(,1/4&*S ||S !!2!23S E--. S ((9(9: S !)):): ; S "%e.?.?(@"ABS D>S $D>S 'tnS d^S  uU\\"$MM NS S rracb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $) BertPooleric>[TU]5 [R"URUR5Ul[R "5Ulgr)r|r}r rrrTanh activationrs r0r}BertPooler.__init__s9 YYv1163E3EF '')rrrc\USS2S4nURU5nURU5nU$)Nr)rr)rrfirst_token_tensor pooled_outputs r0rBertPooler.forwards6+1a40 #56  6 r)rrrrs@r0rrs($ U\\ellrrcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)BertPredictionHeadTransformicp>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURS9Ul gr) r|r}r rrrr:r;r<rtransform_act_fnrrrs r0r}$BertPredictionHeadTransform.__init__s~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrrrclURU5nURU5nURU5nU$r)rrrrAs r0r#BertPredictionHeadTransform.forwards4 =1 --m< }5 r)rrrrrs@r0rrs)UU\\ellrrc4^\rSrSrU4SjrSrSrSrU=r$)BertLMPredictionHeadicH>[TU]5 [U5Ul[R "UR URSS9Ul[R"[R"UR55Ul URURl g)NF)r8) r|r}r transformr rrrdecoder ParameterrYrr8rs r0r}BertLMPredictionHead.__init__sm 4V<yy!3!3V5F5FUS LLV->->!?@ !II rc:URURlgr)r8rrs r0 _tie_weights!BertLMPredictionHead._tie_weightss II rcJURU5nURU5nU$r)rrrAs r0rBertLMPredictionHead.forwards$}5  ]3 r)r8rr) rrrrr}rrrrrs@r0rrs &&rrcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)BertOnlyMLMHeadicB>[TU]5 [U5Ulgr)r|r}r predictionsrs r0r}BertOnlyMLMHead.__init__s /7rsequence_outputrc(URU5nU$rr)rrprediction_scoress r0rBertOnlyMLMHead.forwards ,,_=  rrrrs@r0rrs(8!u||! !!rrc.^\rSrSrU4SjrSrSrU=r$)BertOnlyNSPHeadicn>[TU]5 [R"URS5UlgNr<)r|r}r rrseq_relationshiprs r0r}BertOnlyNSPHead.__init__s'  " &*<*[TU]5 [U5Ul[R "UR S5Ulgr)r|r}rrr rrrrs r0r}BertPreTrainingHeads.__init__ s4 /7 " &*<*.  - - KK   " " $ MM   $ $S )  4 5 5 KK   " " $6rr-N) rrrrr$ config_classrlload_tf_weightsbase_model_prefixsupports_gradient_checkpointing_supports_sdparrr-rr0rrs"L-O&*#N%rrc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\ \R\ S'Sr\\ \R\ S'S rg) BertForPreTrainingOutputi2a  Output type of [`BertForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossprediction_logitsseq_relationship_logitsrrqr-)rrrrrrrrYr__annotations__rrrrrqrr-rr0rr2s~2)-D(5$$ %,59x 1 129;?Xe&7&78?8[TU]U5 Xl[U5Ul[ U5UlU(a [U5OSUlURUl URUl UR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r|r}r]rnrraencoderrpoolerr'attn_implementationrt post_init)rr]add_pooling_layerrs r0r}BertModel.__init__csg   (0"6* ,=j(4 #)#>#> '-'E'E$ rc.URR$rrrrs r0get_input_embeddingsBertModel.get_input_embeddingsvs...rc$XRlgrr)rrs r0set_input_embeddingsBertModel.set_input_embeddingsys */'rcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrfrNr0)rheads_to_prunerfr.s r0 _prune_headsBertModel._prune_heads|s< +002LE LL  u % / / ; ;E B3rrrryrvrrrrrirrrjrkrc RU bU OURRn U bU OURRn U bU OURRn URR(aU bU OURR n OSn UbUb [ S5eUb"URX5 UR5nO"UbUR5SSnO [ S5eUunnUb URO URnU bU SSRSOSnUcs[URS5(a4URRSS2SU24nURUU5nUnO$[R "U[R"US9nURUUUUUS 9nUc[R$"UUU-4US 9nUR&S :H=(a( UR(S :H=(a USL=(a U (+nU(aTUR+5S:Xa@URR(a[-UUUU5nO'[/UUR0US 9nOUR3X.5nURR(avUbsUR5unnnUU4nUc[R$"UUS 9nU(a*UR+5S:Xa[/UUR0US 9nOUR5U5nOSnUR7XPRR85nUR;UUUUUU U U U U S9 nUSnUR<bUR=U5OSnU (d UU4USS-$[?UUUR@URBURDURFS9$)NFzDYou cannot specify both input_ids and inputs_embeds at the same timerwz5You have to specify either input_ids or inputs_embedsrr<ryr)rrvryrr)rr"ru)r ) rrrrrirrrjrkr#)rp pooler_outputrirrqrr)$r]rrjuse_return_dictrrrW%warn_if_padding_and_no_attention_maskrrrVrrryrrYrronesrrtrrrr{get_extended_attention_maskinvert_attention_mask get_head_maskrerrrrirrqrr) rrrryrvrrrrrirrrjrkr batch_sizerrrrrembedding_outputuse_sdpa_attention_masksextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskencoder_outputsrrs r0rBertModel.forwards"2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, J%.%:!! @T@TDSC^!3A!6!L $D>L 'tnL d^L  uU\\"$PP QL L rrz Bert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. c^\rSrSrSS/rU4SjrSrSr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \ S\ \ S\ \ S\ \\ R\44Sjj5rSrU=r$)BertForPreTrainingipredictions.decoder.biascls.predictions.decoder.weightc>[TU]U5 [U5Ul[ U5UlUR 5 gr)r|r}rrrclsrrs r0r}BertForPreTraining.__init__s4  f% '/ rcBURRR$rrrrrs r0get_output_embeddings(BertForPreTraining.get_output_embeddings&xx##+++rcXRRlURURRlgrrrrr8rnew_embeddingss r0set_output_embeddings(BertForPreTraining.set_output_embeddings)*'5$$2$7$7!rrrryrvrrlabelsnext_sentence_labelrrjrkrc  U bU OURRn URUUUUUUU U U S9 n U SSupURX5unnSnUbvUbs[ 5nU"UR SURR 5UR S55nU"UR SS5UR S55nUU-nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, BertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> model = BertForPreTraining.from_pretrained("google-bert/bert-base-uncased") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` Nrryrvrrrrjrkr<rw)rrrrrq) r]rrrr rrrrrq)rrrryrvrrrrrrjrkrrrrr total_lossloss_fctmasked_lm_lossnext_sentence_lossr(s r0rBertForPreTraining.forward-sCV&1%L 'tnL d^L  uU\\"$<< =L L rrzP Bert Model with a `language modeling` head on top for CLM fine-tuning. c"^\rSrSrSS/rU4SjrSrSr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \ RS\ \ \ RS\ \ S\ \ S\ \ S\ \ S\\\ R\44Sjj5rSrSrU=r$)BertLMHeadModeli}zcls.predictions.decoder.biasrc>[TU]U5 UR(d[R S5 [ USS9Ul[U5UlUR5 g)NzLIf you want to use `BertLMHeadModel` as a standalone, add `is_decoder=True.`Fr r|r}rrBwarningrrrrrrs r0r}BertLMHeadModel.__init__sL    NNi jf> "6* rcBURRR$rrrs r0r%BertLMHeadModel.get_output_embeddingsrrcXRRlURURRlgrrr s r0r %BertLMHeadModel.set_output_embeddingsr rrrryrvrrrrrrirrrjrkrc UbUOURRnU bSn URUUUUUUUUU U U U US9 nUSnURU5nSnU b(UR"UXRR 40UD6nU(dU4USS-nUbU4U-$U$[ UUURURURURS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` NF) rryrvrrrrrirrrjrkrr<)rlogitsrirrqrr) r]rrr loss_functionrrrirrqrr)rrrryrvrrrrrrirrrjrk loss_kwargsrrrlm_lossr(s r0rBertLMHeadModel.forwards 2&1%# UH/oRSTRUR55v M1 g7f)rN) index_selectrr)r. past_statebeam_idxs r0r11BertLMHeadModel._reorder_cache..s1ncmU_--aZ=N=N1OPPcms7:)rv)rrir/reordered_past layer_pasts ` r0_reorder_cacheBertLMHeadModel._reorder_caches8)J ncmnn N*rr)NNNNNNNNNNNNNN)rrrrrr}rr r rrYrrrrrrrr3rrrs@r0rr}s 9:Z[ ,8-11515/3,0048<9=)-8<$(,0/3&*> ELL)> !.> !. > u||, > ELL) >   ->  ( 5> !) 6> &> "$u||"45> D>> $D>> 'tn> d^> " uU\\"$EE F#> > @rrc^\rSrSrSS/rU4SjrSrSr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \ RS\ \ S\ \ S\ \ S\ \\ R\44Sjj5rSSjr\S\ 4Sj5rSrU=r$)BertForMaskedLMirrc>[TU]U5 UR(a[R S5 [ USS9Ul[U5UlUR5 g)NzkIf you want to use `BertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Frrrs r0r}BertForMaskedLM.__init__sR     NN1  f> "6* rcBURRR$rrrs r0r%BertForMaskedLM.get_output_embeddingsrrcXRRlURURRlgrrr s r0r %BertForMaskedLM.set_output_embeddingsr rrrryrvrrrrrrrjrkrc U bU OURRn URUUUUUUUUU U U S9 n U SnURU5nSnU bF[ 5nU"UR SURR 5U R S55nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` N) rryrvrrrrrrjrkrrwr<rr&rrq) r]rrrr rrrrrq)rrrryrvrrrrrrrjrkrrrrrr(s r0rBertForMaskedLM.forwards.&1% &IIrcg)z Legacy correction: BertForMaskedLM can't call `generate()` from `GenerationMixin`, even though it has a `prepare_inputs_for_generation` method. Fr-)rs r0 can_generateBertForMaskedLM.can_generateFs rr) NNNNNNNNNNNNr)rrrrrr}rr r rrYrrrrrrrF classmethodrIrrrs@r0r6r6sj46VW ,8-11515/3,0048<9=)-,0/3&*7 ELL)7 !.7 !. 7 u||, 7 ELL) 7   -7  ( 57 !) 67 &7 $D>7 'tn7 d^7  uU\\"N2 37 7 rJ Trr6zT Bert Model with a `next sentence prediction (classification)` head on top. cl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \R\ 44Sjj5r SrU=r$)BertForNextSentencePredictioniOc>[TU]U5 [U5Ul[ U5UlUR 5 gr)r|r}rrrrrrs r0r}&BertForNextSentencePrediction.__init__Us4  f% "6* rrrryrvrrrrrjrkrc SU ;a,[R"S[5 U RS5nU bU OURR n UR UUUUUUUU U S9 n U Sn URU 5nSnUb2[5nU"URSS5URS55nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$) a" labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, BertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> model = BertForNextSentencePrediction.from_pretrained("google-bert/bert-base-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` rzoThe `next_sentence_label` argument is deprecated and will be removed in a future version, use `labels` instead.Nrr#rwr<r>) warningswarn FutureWarningpopr]rrrr rrrrq)rrrryrvrrrrrjrkkwargsrrseq_relationship_scoresrrr(s r0r%BertForNextSentencePrediction.forward^sT !F * MM%  ZZ 56F%0%Q 'tnQ d^Q  uU\\"$?? @Q Q rrMz Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \R\ 44Sjj5r SrU=r$)BertForSequenceClassificationicr>[TU]U5 URUlXl[ U5UlUR b UR O URn[R"U5Ul [R"URUR5Ul UR5 gr)r|r} num_labelsr]rrclassifier_dropoutrr rrrrr;rrr]r]rs r0r}&BertForSequenceClassification.__init__s   ++ f% )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rrrryrvrrrrrjrkrc RU bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUGbURR cUR S:XaSURlOoUR S:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5nUR S:Xa&U"U R5UR55nOU"X5nOURR S:Xa=[5nU"U RSUR 5URS55nO,URR S:Xa[5nU"X5nU (dU 4U SS-nUbU4U-$U$[!UU U R"U R$S 9$) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr# regressionsingle_label_classificationmulti_label_classificationrwr<r>)r]rrrr; problem_typer\r{rYrrTr squeezer rr rrrq)rrrryrvrrrrrjrkrrr&rrr(s r0r%BertForSequenceClassification.forwards(&1%E 'tnE d^E  uU\\"$<< =E E rrZcl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \R\ 44Sjj5r SrU=r$)BertForMultipleChoiceic0>[TU]U5 [U5UlURb URO UR n[ R"U5Ul[ R"URS5Ul UR5 g)Nr#) r|r}rrr]rr rrrrr;rr^s r0r}BertForMultipleChoice.__init__su  f% )/)B)B)NF % %TZTnTn zz"45 ))F$6$6: rrrryrvrrrrrjrkrc ZU bU OURRn UbURSOURSn Ub!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb1URSUR S5UR S55OSnUR UUUUUUUU U S9 n U Sn UR U 5n URU 5nURSU 5nSnUb[5nU"X5nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr#rwrrr<r>) r]rrVrrrrr;r rrrq)rrrryrvrrrrrjrk num_choicesrrr&reshaped_logitsrrr(s r0rBertForMultipleChoice.forward!sX&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  )) ))%'/!5#     ]3 / ++b+6  ')HO4D%''!"+5F)-)9TGf$ Ev E("!//))   r)rr;rrX)rrrrr}r rrYrrrrrrrrrs@r0rhrhs -11515/3,004)-,0/3&*X ELL)X !.X !. X u||, X ELL) X   -X &X $D>X 'tnX d^X  uU\\"$== >X X rrhcl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \R\ 44Sjj5r SrU=r$)BertForTokenClassificationi}cd>[TU]U5 URUl[USS9UlUR b UR O UR n[R"U5Ul [R"URUR5Ul UR5 gNFr)r|r}r\rrr]rr rrrrr;rr^s r0r}#BertForTokenClassification.__init__s   ++f> )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rrrryrvrrrrrjrkrc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUb<[ 5nU"U R SUR5UR S55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrrwr<r>) r]rrrr;r rr\rrrq)rrrryrvrrrrrjrkrrr&rrr(s r0r"BertForTokenClassification.forwards$&1%2 'tn2 d^2  uU\\"$99 :2 2 rrpc^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \R\ 44Sjj5r SrU=r$)BertForQuestionAnsweringic>[TU]U5 URUl[USS9Ul[ R "URUR5UlUR5 grr) r|r}r\rrr rr qa_outputsrrs r0r}!BertForQuestionAnswering.__init__sU   ++f> ))F$6$68I8IJ rrrryrvrrstart_positions end_positionsrrjrkrc $U bU OURRn URUUUUUUU U U S9 n U Sn URU 5nUR SSS9unnUR S5R 5nUR S5R 5nSnUbUb[UR55S:aUR S5n[UR55S:aUR S5nURS5nURSU5nURSU5n[US9nU"X5nU"UU5nUU-S- nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$) Nrrr#rwr) ignore_indexr<)r start_logits end_logitsrrq)r]rrryrMrerrSrclampr rrrq)rrrryrvrrr{r|rrjrkrrr&rrr ignored_indexr start_lossend_lossr(s r0r BertForQuestionAnswering.forwards&1% ELL)> !.> !. > u||, > ELL) >   -> "%,,/>   -> $D>> 'tn> d^>  uU\\"$@@ A> > rrw) r6rhrMrrwrZrprJrrrrl)SrrrDrQ dataclassesrtypingrrrrrYtorch.utils.checkpoint packagingrr torch.nnr r r activationsr generationrmodeling_attn_mask_utilsrrmodeling_outputsrrrrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrr r!r"configuration_bertr$ get_loggerrrBrlModulernrrrr&r$r7rDrJrarrrrrrrrrrrr6rMrZrhrprw__all__r-rr0rs  !// AA!)w   .llLL*   H %FR=RYY=@C CLb-bJ RYY  ! 0BII0f ryy   S SlZ "))Z z  "))"299.!bii!&bii& 9299 9%/%%4 :{: :B  q #q  q h ` ,`  ` F ])?]  ]@i)iiX \ $7\  \ ~ V $7V  V rg /g g TB !4B B JJ 2J J Z r