fThl>SrSSKrSSKrSSKJr SSKJrJrJrJ r SSK r SSK r SSK J r SSK JrJrJr SSKJrJr SS KJrJrJrJrJrJr SS KJr SS KJrJrJ r SS K!J"r"J#r# S SK$J%r% \#RL"\'5r(Sr)"SS\ RT5r+\""SS\55r,"SS\ RT5r-"SS\ RT5r."SS\ RT5r/"SS\ RT5r0"SS\ RT5r1"SS\ RT5r2"S S!\ RT5r3"S"S#\ RT5r4"S$S%\ RT5r5"S&S'\ RT5r6"S(S)\ RT5r7\""S*S+\,55r8"S,S-\ RT5r9\""S.S/\,55r:"S0S1\ RT5r;\""S2S39"S4S5\,55r<\""S6S7\,55r=\""S8S9\,55r>\""S:S;\,55r?/S3U SU S?3'SU S@3U SU SA3'GM> UR5GH#n U Sn[U5nU"U5nXn[ R""UU5n[RSBUSCUSD35 UR%S75(a8UR%SE5(d"UR%SF5(d UR&nUR%SG5(aUR)SSHS5nUR%SI5(aUR)SHSS5nUR%SJ5(aUR+SK5nUUlGM& U$![a [RS5 ef=f)Lz'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape z"electra/embeddings/word_embeddingsz&electra/embeddings/position_embeddingsz(electra/embeddings/token_type_embeddingsz"electra/embeddings/LayerNorm/gammaz!electra/embeddings/LayerNorm/betaz!electra/embeddings_project/kernelzelectra/embeddings_project/bias)z!embeddings.word_embeddings.weightz%embeddings.position_embeddings.weightz'embeddings.token_type_embeddings.weightzembeddings.LayerNorm.weightzembeddings.LayerNorm.biaszembeddings_project.weightzembeddings_project.biasrg_densedensezelectra/encoder/layer_z/attention/self/query/kernelzencoder.layer.z.attention.self.query.weightz/attention/self/query/biasz.attention.self.query.biasz/attention/self/key/kernelz.attention.self.key.weightz/attention/self/key/biasz.attention.self.key.biasz/attention/self/value/kernelz.attention.self.value.weightz/attention/self/value/biasz.attention.self.value.biasz./attention/self/conv_attn_key/depthwise_kernelz4.attention.self.key_conv_attn_layer.depthwise.weightz./attention/self/conv_attn_key/pointwise_kernelz4.attention.self.key_conv_attn_layer.pointwise.weightz"/attention/self/conv_attn_key/biasz(.attention.self.key_conv_attn_layer.biasz'/attention/self/conv_attn_kernel/kernelz(.attention.self.conv_kernel_layer.weightz%/attention/self/conv_attn_kernel/biasz&.attention.self.conv_kernel_layer.biasz&/attention/self/conv_attn_point/kernelz%.attention.self.conv_out_layer.weightz$/attention/self/conv_attn_point/biasz#.attention.self.conv_out_layer.biasz/attention/output/dense/kernelz.attention.output.dense.weightz!/attention/output/LayerNorm/gammaz".attention.output.LayerNorm.weightz/attention/output/dense/biasz.attention.output.dense.biasz /attention/output/LayerNorm/betaz .attention.output.LayerNorm.biasz/intermediate/z/kernelz.intermediate.dense.weightz/biasz.intermediate.dense.biasz/output/z.output.dense.weightz.output.dense.biasz/output/LayerNorm/gammaz.output.LayerNorm.weightz/output/LayerNorm/betaz.output.LayerNorm.biaszTF: z, PT:  z/intermediate/g_dense/kernelz/output/g_dense/kernelz/depthwise_kernelz/pointwise_kernelz/conv_attn_key/bias) tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variable num_groupsrangenum_hidden_layersnamed_parametersrtorch from_numpyendswithTpermute unsqueezedata)modelconfigtf_checkpoint_pathtftf_path init_varstf_datanameshapearray param_mappinggroup_dense_namejparam param_name retrieverresulttf_namevalues f/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/convbert/modeling_convbert.pyload_tf_weights_in_convbertrM0sfggoo01G KK8 BC''0IG  (l5'BC&&w5 ! .R1Y3]'K%H%H#DM1$" 6++ ,$QC'C D qc)EFG%QC'A B qc)CDE%QC'A B qc)CDE%QC'? @ qc)ABC%QC'C D qc)EFG%QC'A B qc)CDE%QC'U V qc)]^_%QC'U V qc)]^_%QC'I J qc)QRS%QC'N O qc)QRS%QC'L M qc)OPQ%QC'M N qc)NOP%QC'K L qc)LMN%QC'E F qc)GHI%QC'H I qc)KLM%QC'C D qc)EFG%QC'G H qc)IJK%QC~6F5Gw O qc)CDE%QC~6F5Gu M qc)ABC%QCx0@/A I qc)=>?%QCx0@/A G qc);<=%QC'> ? qc)ABCG]]^\__uDv qc)?@AG-J'')1X z* 5!+  !12 d7)6*Q78   I & &##$BCC''(@AA!GGE   / 0 0MM!Q*E   / 0 0MM!Q*E   1 2 2OOB'E #*$ Lk   Q   s M !M.c ^\rSrSrSrU4SjrS S\\RS\\RS\\RS\\RS\R4 S jjr S r U=r $) ConvBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.c >[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR#S[$R&"UR5R)S5SS9 UR#S[$R*"UR,R/5[$R0S9SS9 g) N) padding_idxeps position_ids)rr"F) persistenttoken_type_idsdtype)super__init__r Embedding vocab_sizeembedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferr2arangeexpandzerosrUsizelongselfr: __class__s rLr[ConvBertEmbeddings.__init__s !||F,=,=v?T?Tbhbubuv#%<<0N0NPVPePe#f %'\\&2H2H&J_J_%`"f&;&;AVAVWzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  input_idsrWrU inputs_embedsreturnc>UbUR5nOUR5SSnUSnUcURSS2SU24nUcv[US5(a-URSS2SU24nUR USU5nUnO8[ R "U[ RURRS9nUcURU5nURU5n URU5n XI-U -n URU 5n URU 5n U $)Nr"rrWrrYdevice)rnrUhasattrrWrlr2rmrorzr`rbrdreri) rqrurWrUrv input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedrbrd embeddingss rLforwardConvBertEmbeddings.forwards,  #..*K',,.s3K ^  ,,Q  ^ J"8;PP ^^J/ \\*- rt)rerirbrdr`)NNNN) __name__ __module__ __qualname____firstlineno____doc__r[rr2 LongTensor FloatTensorr__static_attributes__ __classcell__rrs@rLrOrOsQ (15593759 $E,,-$!!1!12$u//0 $   1 12 $    $$rtrOc*\rSrSr\r\rSrSr Sr Sr g)ConvBertPreTrainedModelconvbertTc[U[R[R45(akURR R SURRS9 URb%URR R5 gg[U[R5(axURR R SURRS9 URb2URR URR5 gg[U[R5(aJURR R5 URR RS5 g[U[5(a%URR R5 g[U[ 5(a]URR R SURRS9 URR R5 gg)zInitialize the weightsmeanstdNg?) isinstancerLinearConv1dweightr8normal_r:initializer_rangebiaszero_r\rRrefill_SeparableConv1DGroupedLinearLayer)rqmodules rL _init_weights%ConvBertPreTrainedModel._init_weightss fryy"))4 5 5 MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S )  0 0 KK   " " $  2 3 3 MM   & &CT[[5R5R & S KK   " " $4rtN) rrrrr config_classrMload_tf_weightsbase_model_prefixsupports_gradient_checkpointingrrrrtrLrrs!L1O"&*#%rtrcf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)rizSThis class implements separable convolution, i.e. a depthwise and a pointwise layerc >[TU]5 [R"UUUUUS-SS9Ul[R"X#SSS9Ul[R "[R"US55Ul URRRRSURS9 UR RRRSURS9 g)Nr!F) kernel_sizegroupspaddingrr)rrrr)rZr[rr depthwise pointwise Parameterr2rmrrr8rr)rqr: input_filtersoutput_filtersrkwargsrrs rLr[SeparableConv1D.__init__s   # 1$  =aV[\LL^Q!?@  ""**9Q9Q*R ""**9Q9Q*Rrt hidden_statesrwcfURU5nURU5nX R- nU$N)rrr)rqrxs rLrSeparableConv1D.forwards. NN= ) NN1  YYrt)rrr rrrrrr[r2Tensorrrrrs@rLrrs,]S U\\ellrtrc^\rSrSrU4SjrSrS S\RS\\RS\\RS\\RS\\ S \ \R\\R44 S jjr S r U=r$) ConvBertSelfAttentionic~>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eURUR -nUS:aURUlSUlOX lUR UlURUlURUR-S:wa [ S5eURUR-S-UlURUR-Ul [R"URUR5Ul [R"URUR5Ul [R"URUR5Ul[XRURUR5Ul[R"URURUR-5Ul[R"URUR5Ul[R&"URS/[)URS- S- 5S/S 9Ul[R,"UR.5Ulg) Nrr^zThe hidden size (z6) is not a multiple of the number of attention heads ()rz6hidden_size should be divisible by num_attention_headsr!)rr)rZr[ hidden_sizenum_attention_headsr{ ValueError head_ratioconv_kernel_sizeattention_head_size all_head_sizerrquerykeyrKrkey_conv_attn_layerconv_kernel_layerconv_out_layerUnfoldintunfoldrgattention_probs_dropout_probri)rqr:new_num_attention_headsrrs rLr[ConvBertSelfAttention.__init__ s=     : : :a ?PVXhHiHi#F$6$6#78 445Q8  #)"<"<@Q@Q"Q "Q &$88DO'(D $'> $$//DO & 7 7    8 8 8A =UV V$*$6$6$:R:R$RWX#X !558P8PPYYv1143E3EF 99V//1C1CDYYv1143E3EF #2 &&(:(:D //0AB ..)BV 22?C!MM*;b$BWBWYZ=[\!MM*;C,,];~ BHZHZ7[\'11!Q7BBDNNrR-- ..2++a/A5q9 . (11!Q7?? D..0E0E ~D P u||Xell33 4PPrtrcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)ConvBertSelfOutputic(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNrS) rZr[rrrrrerfrgrhrirps rLr[ConvBertSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rtr input_tensorrwcpURU5nURU5nURX-5nU$rrrirerqrrs rLrConvBertSelfOutput.forward5 =1  ]3 }'CD rtrerri rrrrr[r2rrrrrs@rLrrs6> U\\RWR^R^rtrc^\rSrSrU4SjrSrS S\RS\\RS\\RS\\RS\\ S \ \R\\R44 S jjr S r U=r$) ConvBertAttentionic>[TU]5 [U5Ul[ U5Ul[ 5Ulgr)rZr[rrqroutputset pruned_headsrps rLr[ConvBertAttention.__init__s0 )&1 (0 Ertc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrrr)lenrrqrrrrrrrKr rrunion)rqheadsindexs rL prune_headsConvBertAttention.prune_headss  u:? 7 9900$))2O2OQUQbQb  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rtrrrrrrwclURUUUUU5nURUSU5nU4USS-nU$)Nrr)rqr ) rqrrrrr self_outputsattention_outputrs rLrConvBertAttention.forwardsQyy    !    ;;|A F#% QR(88rt)r rrqr)rrrrr[rr2rrrrrrrrrs@rLr r s" ;*7;158<,1 ||!!2!23E--.  ( 5  $D>  u||Xe&7&788 9rtr cb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)ric>[TU]5 XlX lX0lURUR-UlURUR-Ul[R"[R"URUR UR 55Ul [R"[R"U55Ul gr) rZr[ input_size output_sizer. group_in_dim group_out_dimrrr2emptyrr)rqrrr.rrs rLr[GroupedLinearLayer.__init__s $&$ OOt>!--@ll5;;t@Q@QSWSeSe#fg LL[!9: rtrrwc[UR55Sn[R"USURUR /5nUR SSS5n[R"X0R5nUR SSS5n[R"X2SUR/5nX0R-nU$)Nrr"rr!) listrnr2rr.r r6rrrr)rqrrrs rLrGroupedLinearLayer.forwards-,,./2 MM-"doot?P?P)Q R IIaA  LLKK ( IIaA  MM!"d.>.>? @ Mrt)rr r!rr.rrr rs@rLrrs(;U\\ellrtrcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)ConvBertIntermediateic>[TU]5 URS:Xa1[R"UR UR 5UlO.[UR UR URS9Ul[UR[5(a[URUl gURUl g)Nrrrr.)rZr[r.rrrintermediate_sizerrr hidden_actstrr intermediate_act_fnrps rLr[ConvBertIntermediate.__init__s     !6#5#5v7O7OPDJ+!--6;S;S`f`q`qDJ f'' - -'-f.?.?'@D $'-'8'8D $rtrrwcJURU5nURU5nU$rrr.rqrs rLrConvBertIntermediate.forwards& =1 00? rtr1r rs@rLr(r(s( 9U\\ellrtr(cz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)ConvBertOutputic>[TU]5 URS:Xa1[R"UR UR 5UlO.[UR UR URS9Ul[R"UR URS9Ul [R"UR5Ul g)Nrr*rS)rZr[r.rrr+rrrrerfrgrhrirps rLr[ConvBertOutput.__init__s     !6#;#;V=O=OPDJ+!33ASAS`f`q`qDJf&8&8f>S>STzz&"<"<= rtrrrwcpURU5nURU5nURX-5nU$rrrs rLrConvBertOutput.forwardrrtrr rs@rLr5r5s6 >U\\RWR^R^rtr5c^\rSrSrU4SjrS S\R S\\RS\\RS\\R S\\R S\\ S \ \R \\R44S jjr S r S r U=r$) ConvBertLayericv>[TU]5 URUlSUl[ U5UlUR UlURUlUR(a/UR (d[US35e[ U5Ul [U5Ul [U5Ul g)Nrz> should be used as a decoder model if cross attention is added)rZr[chunk_size_feed_forward seq_len_dimr  attention is_decoderadd_cross_attention TypeErrorcrossattentionr( intermediater5r rps rLr[ConvBertLayer.__init__s '-'E'E$*62 ++#)#=#=  # #??4&(f ghh"3F";D 08$V, rtrrrrencoder_attention_maskrrwcLURUUUUS9nUSnUSSn UR(aDUbA[US5(d[SUS35eUR UUUUU5n U SnXSS-n [ UR URURU5n U 4U -n U $)N)rrrrCz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`) r?r@r{AttributeErrorrCrfeed_forward_chunkr=r>) rqrrrrrFrself_attention_outputsrrcross_attention_outputs layer_outputs rLrConvBertLayer.forward&s"&   / "0"  2!4(, ??4@4!122$=dVDDD'+&9&9 &%! ' # 7q9  ;;G0  # #T%A%A4CSCSUe  /G+rtcJURU5nURX!5nU$r)rDr )rqrintermediate_outputrLs rLrI ConvBertLayer.feed_forward_chunkNs)"//0@A{{#6I rt)rAr?r=rCrDr@r r>)NNNNF)rrrrr[r2rrrrrrrIrrrs@rLr;r;s -"7;158<9=,1&||&!!2!23&E--. & ( 5 & !) 6 &$D>& u||Xe&7&788 9&Prtr;c^\rSrSrU4SjrSS\R S\\RS\\RS\\R S\\R S\\ S \\ S \\ S \ \ \ 44S jjr S rU=r$)ConvBertEncoderiTc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snf)NF) rZr[r:r ModuleListr/r0r;layergradient_checkpointing)rqr:_rrs rLr[ConvBertEncoder.__init__UsR  ]]5IaIaCb#cCbaM&$9Cb#cd &+#$dsA&rrrrrFroutput_hidden_states return_dictrwc ^U(aSOSn U(aSOSn U(aURR(aSOSn [UR5HupU(aX4-n UbX<OSnUR(a3UR (a"UR U RUUUUUU5nO U "UUUUUU5nUSnU(dMyXS4-n URR(dMXS4-n M U(aX4-n U(d[SXX455$[UU U U S9$)Nrrrr!c30# UH nUcMUv M g7frr).0vs rL *ConvBertEncoder.forward..sfAfs )last_hidden_stater attentionscross_attentions) r:rA enumeraterUrVtraining_gradient_checkpointing_func__call__tupler)rqrrrrrFrrYrZall_hidden_statesall_self_attentionsall_cross_attentionsi layer_modulelayer_head_mask layer_outputss rLrConvBertEncoder.forward[sM#7BD$5b4%64;;;Z;Zr`d(4OA#$58H$H!.7.CilO**t}} $ A A ))!"#)*%! !-!"#)*% ! *!,M  &91=M 14D D '; 'tn; d^;  u88 9; ; rtrRcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)ConvBertPredictionHeadTransformicp>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURS9Ul gr) rZr[rrrrrr,r-r transform_act_fnrerfrps rLr[(ConvBertPredictionHeadTransform.__init__s~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrtrrwclURU5nURU5nURU5nU$r)rrtrer2s rLr'ConvBertPredictionHeadTransform.forwards4 =1 --m< }5 rt)rerrtr rs@rLrrrrs)UU\\ellrtrrc^\rSrSrSrS\4U4SjjrS S\RS\ \RS\R4Sjjr S r U=r $) ConvBertSequenceSummaryia Compute a single vector summary of a sequence hidden states. Args: config ([`ConvBertConfig`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. r:c>[TU]5 [USS5UlURS:Xa[e[ R "5Ul[US5(aUR(aq[US5(a.UR(aURS:a URnO URn[ R"URU5Ul[USS5nU(a [U5O[ R "5Ul[ R "5Ul[US5(a5UR"S:a%[ R$"UR"5Ul[ R "5Ul[US 5(a7UR(S:a&[ R$"UR(5Ulggg) N summary_typelastattnsummary_use_projsummary_proj_to_labelsrsummary_activationsummary_first_dropoutsummary_last_dropout)rZr[getattrr{NotImplementedErrorrIdentitysummaryr{r~r num_labelsrrr activation first_dropoutrrg last_dropoutr)rqr: num_classesactivation_stringrrs rLr[ ConvBertSequenceSummary.__init__sa #FNFC    && %{{} 6- . .63J3Jv788V=Z=Z_e_p_pst_t$// $00 99V%7%7EDL#F,@$GIZN3D$E`b`k`k`m[[] 62 3 38T8TWX8X!#F,H,H!ID KKM 61 2 2v7R7RUV7V " 6+F+F GD 8W 2rtr cls_indexrwcURS:Xa USS2S4nGOURS:Xa USS2S4nGOURS:XaURSS9nOURS :XaUc?[R"US SS2SS24URS S- [R S 9nOXUR S5R S5nURS UR5S- -URS54-5nURS U5RS 5nOURS:Xa[eURW5nURU5nURU5nUR!U5nU$)a# Compute a single vector summary of a sequence hidden states. Args: hidden_states (`torch.FloatTensor` of shape `[batch_size, seq_len, hidden_size]`): The hidden states of the last layer. cls_index (`torch.LongTensor` of shape `[batch_size]` or `[batch_size, ...]` where ... are optional leading dimensions of `hidden_states`, *optional*): Used if `summary_type == "cls_index"` and takes the last token of the sequence as classification token. Returns: `torch.FloatTensor`: The summary of the sequence hidden states. r|Nr"firstrrrrr.rrX)r"r})r{rr2 full_likerAror7rlrrngathersqueezerrrrr)rqrrr s rLrConvBertSequenceSummary.forwardsn    &"1b5)F   ' )"1a4(F   & ("''A'.F   + - !OO!#rr1*-!''+a/** &//3==bA %,,Uimmo6I-JmN`N`acNdMf-fg "))"i8@@DF   & (% %##F+f%(""6* rt)rrrrr{r)rrrrrrr[r2rrrrrrrs@rLryrysV2H~H<Y])"..);CEDTDT;U)   ))rtrycD^\rSrSrU4SjrSrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ S \ \ S\ \ S\\\44Sjj5rSrU=r$) ConvBertModelic$>[TU]U5 [U5UlURUR :wa0[ R"URUR 5Ul[U5Ul Xl UR5 gr) rZr[rOrr^rrrembeddings_projectrRencoderr: post_initrps rLr[ConvBertModel.__init__sj  ,V4  F$6$6 6&(ii0E0EvGYGY&ZD #&v.   rtc.URR$rrr`rqs rLget_input_embeddings"ConvBertModel.get_input_embeddingss...rtc$XRlgrr)rqrKs rLset_input_embeddings"ConvBertModel.set_input_embeddingss */'rtcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrUr?r)rqheads_to_prunerUrs rL _prune_headsConvBertModel._prune_heads"s< +002LE LL  u % / / ; ;E B3rtrurrWrUrrvrrYrZrwc UbUOURRnUbUOURRnU bU OURRn UbUb [ S5eUb"UR X5 UR 5n O"UbUR 5SSn O [ S5eU upUb URO URn Uc[R"XS9nUcr[URS5(a3URRSS2SU 24nURX5nUnO$[R"U [RU S9nUR!X*5nUR#XPRR$5nURXX6S9n[US5(aUR'U5nUR)UUUUUU S 9nU$) NzDYou cannot specify both input_ids and inputs_embeds at the same timer"z5You have to specify either input_ids or inputs_embeds)rzrWry)rurUrWrvr)rrrrYrZ)r:rrYuse_return_dictr%warn_if_padding_and_no_attention_maskrnrzr2onesr{rrWrlrmroget_extended_attention_mask get_head_maskr0rr)rqrurrWrUrrvrrYrZr|rr}rzr~rextended_attention_maskrs rLrConvBertModel.forward*s2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, %.%:!! @T@T  !"ZZ CN  !t(899*.//*H*HKZK*X'3J3Q3QR\3i0!A!&[ SY!Z"&"B"B>"_&&y++2O2OP >(  4- . . 33MBM 2/!5# % rt)r:rrr) NNNNNNNNN)rrrrr[rrrrrr2rrrrrrrrrrs@rLrrs /0C156:59371559,0/3&*<E,,-<!!2!23<!!1!12 < u//0 < E--. <  1 12<$D><'tn<d^< u88 9<<rtrcf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)ConvBertGeneratorPredictionsijzAPrediction module for the generator, made up of two dense layers.c>[TU]5 [S5Ul[R "UR URS9Ul[R"URUR 5Ul g)NgelurS) rZr[rrrrer^rfrrrrps rLr[%ConvBertGeneratorPredictions.__init__msV (0f&;&;AVAVWYYv1163H3HI rtgenerator_hidden_statesrwclURU5nURU5nURU5nU$r)rrre)rqrrs rLr$ConvBertGeneratorPredictions.forwardts3 #:;  6 }5 rt)rerr) rrrrrr[r2rrrrrs@rLrrjs0KJu/@/@UEVEVrtrcd^\rSrSrS/rU4SjrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ S\ \ S\ \ S\\\44Sjj5rSrU=r$)ConvBertForMaskedLMi|zgenerator.lm_head.weightc>[TU]U5 [U5Ul[ U5Ul[ R"URUR5Ul UR5 gr) rZr[rrrgenerator_predictionsrrr^r]generator_lm_headrrps rLr[ConvBertForMaskedLM.__init__sR  %f- %A&%I"!#6+@+@&BSBS!T rtcUR$rrrs rLget_output_embeddings)ConvBertForMaskedLM.get_output_embeddingss%%%rtcXlgrr)rqr`s rLset_output_embeddings)ConvBertForMaskedLM.set_output_embeddingss!0rtrurrWrUrrvlabelsrrYrZrwc U bU OURRn URUUUUUUUU U 5 n U Sn URU 5n UR U 5n SnUbQ[ R "5nU"U RSURR5URS55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrr"rlosslogitsrrb) r:rrrrrr rr]rrrb)rqrurrWrUrrvrrrYrZrgenerator_sequence_outputprediction_scoresrloss_fctr s rLrConvBertForMaskedLM.forwards(&1%! 667PQ 223DE  **,H-222t{{7M7MNPVP[P[\^P_`D'),CAB,GGF)-)9TGf$ Ev E$1??.99   rt)rrr NNNNNNNNNN)rrrr_tied_weights_keysr[rrrrr2rrrrrrrrrrs@rLrr|s45&1156:59371559-1,0/3&*4 E,,-4 !!2!234 !!1!12 4 u//0 4 E--. 4   1 124 ))*4 $D>4 'tn4 d^4  un$ %4 4 rtrcf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)ConvBertClassificationHeadiz-Head for sentence-level classification tasks.cn>[TU]5 [R"URUR5UlUR b UR O URn[R"U5Ul [R"URUR5Ul Xl gr) rZr[rrrrclassifier_dropoutrhrgrirout_projr:rqr:rrrs rLr[#ConvBertClassificationHead.__init__s YYv1163E3EF )/)B)B)NF % %TZTnTn zz"45  &"4"4f6G6GH  rtrrwc USS2SSS24nURU5nURU5n[URR"U5nURU5nUR U5nU$)Nr)rirr r:r,r)rqrrrs rLr"ConvBertClassificationHead.forwardse !Q' " LLO JJqM 4;;)) *1 - LLO MM! rt)r:rrirrrs@rLrrs+7 U\\ rtrz ConvBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introcR^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \ 44Sjj5rSrU=r$)!ConvBertForSequenceClassificationic>[TU]U5 URUlXl[ U5Ul[ U5UlUR5 gr) rZr[rr:rrr classifierrrps rLr[*ConvBertForSequenceClassification.__init__sF   ++ %f- 4V< rtrurrWrUrrvrrrYrZrwc 0U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n SnUGbURRcUR S:XaSURlOoUR S:aNUR [R:XdUR [R:XaSURlOSURlURRS:XaI[5nUR S:Xa&U"U R5UR55nOU"X5nOURRS:Xa=[5nU"U RSUR 5URS55nO,URRS:Xa[5nU"X5nU (dU 4U SS-nUbU4U-$U$[UU U R U R"S 9$) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrWrUrrvrrYrZrr regressionsingle_label_classificationmulti_label_classificationr"r)r:rrr problem_typerrYr2rorr rr rr rrrbrqrurrWrUrrvrrrYrZrsequence_outputrrrr s rLr)ConvBertForSequenceClassification.forwards(&1%D 'tnD d^D  u.. /D D rtrcR^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \ 44Sjj5rSrU=r$)ConvBertForMultipleChoicei9c>[TU]U5 [U5Ul[ U5Ul[ R"URS5Ul UR5 g)Nr) rZr[rrrysequence_summaryrrrrrrps rLr["ConvBertForMultipleChoice.__init__;sM  %f- 7 ?))F$6$6: rtrurrWrUrrvrrrYrZrwc \U bU OURRn UbURSOURSn Ub!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb1URSUR S5UR S55OSnUR UUUUUUUU U S9 n U Sn UR U 5nURU5nURSU 5nSnUb[5nU"UU5nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr"rrrr) r:rrArrnrrrr rrrb)rqrurrWrUrrvrrrYrZ num_choicesrr pooled_outputrreshaped_logitsrrr s rLr!ConvBertForMultipleChoice.forwardEsZ&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  -- ))%'/!5#  "!*--o> / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   rt)rrrr)rrrrr[rrr2rrrrrrrrrrs@rLrr9s156:59371559-1,0/3&*Y E,,-Y !!2!23Y !!1!12 Y u//0 Y E--. Y   1 12Y ))*Y $D>Y 'tnY d^Y  u// 0Y Y rtrcR^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \ 44Sjj5rSrU=r$)ConvBertForTokenClassificationicf>[TU]U5 URUl[U5UlUR b UR O UR n[R"U5Ul [R"URUR5Ul UR5 gr)rZr[rrrrrhrrgrirrrrrs rLr['ConvBertForTokenClassification.__init__s   ++%f- )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rtrurrWrUrrvrrrYrZrwc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUb<[ 5nU"U R SUR5UR S55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr"rr) r:rrrirr rrrrrbrs rLr&ConvBertForTokenClassification.forwards$&1%2 'tn2 d^2  u++ ,2 2 rtrcr^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \ 44Sjj5rSrU=r$)ConvBertForQuestionAnsweringic>[TU]U5 URUl[U5Ul[ R "URUR5UlUR5 gr) rZr[rrrrrr qa_outputsrrps rLr[%ConvBertForQuestionAnswering.__init__sS   ++%f- ))F$6$68I8IJ rtrurrWrUrrvstart_positions end_positionsrrYrZrwc $U bU OURRn URUUUUUUU U U S9 n U Sn URU 5nUR SSS9unnUR S5R 5nUR S5R 5nSnUbUb[UR55S:aUR S5n[UR55S:aUR S5nURS5nURSU5nURSU5n[US9nU"X5nU"UU5nUU-S- nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$) Nrrrr"r) ignore_indexr!)r start_logits end_logitsrrb)r:rrrsplitrrrrnclampr rrrb)rqrurrWrUrrvrrrrYrZrrrrr total_loss ignored_indexr start_lossend_lossr s rLr$ConvBertForQuestionAnswering.forwards&1% E,,-> !!2!23> !!1!12 > u//0 > E--. >   1 12> "%"2"23>   0 01> $D>> 'tn> d^>  u22 3> > rtr) rrrrrr;rrrM)Arrr'operatorrtypingrrrrr2torch.utils.checkpointrtorch.nnr r r activationsr rmodeling_outputsrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_convbertr get_loggerrr%rMModulerOrrrrr rr(r5r;rRrrryrrrrrrrr__all__rrtrLrs; 33 AA1.ll3   H %yx99x%o%%8bii4}BII}@  * *Z,299(RYY&:BII:zB biiB Jbii$`bii`FX+XXv299$H 1H H V0 P (?P  P fe 7e e PB %<B B JJ #:J J Z rt