eThjSrSSKrSSKrSSKJr SSKJrJrJrJ r J r SSK r SSK J r SSK JrJrJr SSKJr SS KJr SS KJrJrJrJrJrJrJr SS KJr SS KJ r J!r!J"r"J#r# SS K$J%r%J&r&J'r' SSK(J)r) \'RT"\+5r,Sr-"SS\ R\5r/"SS\ R\5r0"SS\05r1\0\1S.r2"SS\ R\5r3"SS\ R\5r4"SS\ R\5r5\&"SS\55r6\"S S!\%55r7\&"S"S#\655r8\&"S$S%9"S&S'\655r9"S(S)\ R\5r:"S*S+\ R\5r;\&"S,S-\655r<\&"S.S%9"S/S0\655r=\&"S1S2\655r>\&"S3S4\655r?\&"S5S6\655r@/S7QrAg)8zPyTorch ALBERT model.N) dataclass)DictListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)#_prepare_4d_attention_mask_for_sdpa)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indices"is_torch_greater_or_equal_than_2_2prune_linear_layer) ModelOutputauto_docstringlogging) AlbertConfigc  SSKnSSKnSSKn[ RRU5n[RSU35 URRU5n/n/n UH]up[RSU SU 35 URRXj5n URU 5 U RU 5 M_ [X5Hup[U 5 M [X5GHoupU n U R!SS5n U R!S S 5n U R!S S 5n U R!S S5n U R!SS5n U R!SS5n U R!SS5n U R!SS5n U R!SS5n U R!SS5n U R!SS5n U R!SS5n U R!SS5n U R!SS5n U R!SS5n U R!S S!5n U R!S"S#5n U R!S$S%5n [#U R%S55S&:XaS'U ;dS(U ;aS)U -n S*U ;a$U R!S+S,5n U R!S-S.5n U R%S5n S/U ;dS0U ;dS1U ;d S2U ;dS3U ;a*[RS4SR'U 535 GMUnU HnUR)S5U5(aUR%S6U5nOU/nUSS7:Xd USS8:Xa [+US.5nO[USS':Xd USS9:Xa [+US:5nOSS?:Xa [+US.5nOUS7:XaUR1U 5n UR2U R2:wa&[5S@UR2SAU R2SB35e[SCU SDU 35 [8R:"U 5UlGMr U$![a [R S5 ef=f![,a, [RS4SR'U 535 GMf=f![4a1nU=R6UR2U R24- sleSnAff=f)Ez'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape zmodule/ffn_1ffnzbert/zalbert/ attention_1 attentionz transform/ LayerNorm_1full_layer_layer_norm LayerNormzattention/LayerNormz transformer/zintermediate/dense/zffn/intermediate/output/dense/z ffn_output/z/output//z/self/z pooler/densepoolerzcls/predictions predictionszpredictions/attentionzembeddings/attention embeddings inner_group_zalbert_layers/group_zalbert_layer_groups/r output_biasoutput_weightsz classifier/seq_relationshipzseq_relationship/output_zsop_classifier/classifier/weightsweightadam_madam_vAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepz Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammabetabiassquad classifieri _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight z from )renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendzipprintreplacelensplitjoin fullmatchgetattrAttributeErrorint transposeshape ValueErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathrBnptftf_path init_varsnamesarraysnamer[array original_namepointerm_name scope_namesnumes b/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/albert/modeling_albert.pyload_tf_weights_in_albertrs3s ggoo01G KK8 BC''0I E F  (l5'BC&&w5 T e ! 5)  d *5)  ||Ir*||GU+||GY/||M;7||L"-||M+BC||K)>?||NB/||126||G;1$+a.)!#,-0 #$<= (gx0G x LL'E }} + >'--@QRWR]R]Q^^i!jkk, *4&}oFG''. {*~ Li   Q   @&KK)CHHTN+; <=  FFw}}ekk2 2F  s6 Q'R 3AS'!R 1SS S?,S::S?c ^\rSrSrSrS\4U4SjjrS S\\RS\\RS\\RS\\RS \ S \R4 S jjr S rU=r$)AlbertEmbeddingszI Construct the embeddings from word, position and token_type embeddings. rbc.>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR#S[$R&"UR5R)S5SS9 [+USS5UlUR#S [$R."UR0R35[$R4S 9SS9 g) N) padding_idxeps position_ids)rF) persistentposition_embedding_typeabsolutetoken_type_idsdtype)super__init__r Embedding vocab_sizeembedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddingsr)layer_norm_epsDropouthidden_dropout_probdropoutregister_bufferr^arangeexpandrWr~zerosr{sizelongselfrb __class__s rrrAlbertEmbeddings.__init__s1 !||F,=,=v?T?Tbhbubuv#%<<0N0NPVPePe#f %'\\&2H2H&J_J_%`"f&;&;AVAVWzz&"<"<=   ELL)G)GHOOPWXej  (/v7PR\']$  ekk$*;*;*@*@*B%**Ubg  input_idsrr{ inputs_embedspast_key_values_lengthreturncdUbUR5nOUR5SSnUSnUcURSS2XWU-24nUcv[US5(a-URSS2SU24nUR USU5n U nO8[ R "U[ RURRS9nUcURU5nURU5n XJ-n URS:XaURU5n X- n URU 5n URU 5n U $)Nr|rrrrdevicer)rr{hasattrrrr^rrrrrr~rr)r) rrrr{rr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedrr-rs rrforwardAlbertEmbeddings.forwardsC  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  !t-..*.*=*=a*n*M'3J3Q3QR]^_R`bl3m0!A!&[ SWSdSdSkSk!l   00;M $ : :> J":  ' ': 5"&":":<"H   -J^^J/ \\*- r)r)rr~rrr)NNNNr)__name__ __module__ __qualname____firstlineno____doc__r rrr^ LongTensor FloatTensorrYTensorr__static_attributes__ __classcell__rs@rrrurus | .15593759&' 'E,,-'!!1!12'u//0 '   1 12 ' !$ ' ''rrucH^\rSrSrS\4U4SjjrS\RS\R4SjrS\ \ SS4S jr SS \RS \ \RS \ \RS \S\\\R\\R\R444 SjjrSrU=r$)AlbertAttentionrbc>[TU]5 URUR-S:wa6[ US5(d%[ SURSUR35eURUlURUlURUR-UlURUR -Ul[R"URUR5Ul [R"URUR5Ul [R"URUR5Ul [R"UR5Ul[R"UR 5Ul[R"URUR5Ul[R&"URUR(S9Ul[+5Ul[/USS5UlUR0S:XdUR0S :XaHUR2Ul[R4"S UR2-S - UR 5Ulgg) NrrzThe hidden size (z6) is not a multiple of the number of attention heads (ryr~r relative_keyrelative_key_queryr@r)rr hidden_sizenum_attention_headsrr\attention_head_size all_head_sizer Linearquerykeyvaluerattention_probs_dropout_probattention_dropoutroutput_dropoutdenser)rset pruned_headsrWr~rrdistance_embeddingrs rrrAlbertAttention.__init__s     : : :a ?PVXhHiHi#F$6$6#78 4457  $*#=#= !--#)#5#59S9S#S !558P8PPYYv1143E3EF 99V//1C1CDYYv1143E3EF !#F,O,O!P jj)C)CDYYv1163E3EF f&8&8f>S>STE'.v7PR\']$  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD #>rrxrcUR5SSURUR4-nURU5nUR SSSS5$)Nr|rr@rr )rrrviewpermute)rr new_x_shapes rrtranspose_for_scores$AlbertAttention.transpose_for_scoressLffhsmt'?'?AYAY&ZZ FF; yyAq!$$rheadsNc [U5S:Xag[XRURUR5up[ UR U5Ul[ URU5Ul[ URU5Ul[ URUSS9Ul UR[U5- UlURUR-Ul URRU5Ulg)Nrrdim) rSrrrrrrrrrrunion)rrindexs rr prune_headsAlbertAttention.prune_headss u:? 7 ++T-E-EtGXGX  ( E: %dhh6' E: ' EqA $(#;#;c%j#H !558P8PP --33E:r hidden_statesattention_mask head_maskoutput_attentionscURU5nURU5nURU5nURU5nURU5n URU5n [R "XR SS55n U [R"UR5- n UbX-n URS:XdURS:XGaBUR5Sn [R"U [RURS9RSS5n [R"U [RURS9RSS5nX- nUR!XR"-S- 5nUR%UR&S9nURS:Xa[R("SUU5nU U-n OHURS:Xa8[R("SUU5n[R("S U U5nU U-U-n [*R,R/U SS 9nUR1U5nUbUU-n[R "UU 5nUR S S5R3S 5nUR5U5nUR7U5nUR9UU-5nU(aUU4$U4$) Nr|rrrrrzbhld,lrd->bhlrzbhrd,lrd->bhlrrr@)rrrrr^matmulrZmathsqrtrr~rrrrrrrtoreinsumr functionalsoftmaxrflattenrrr))rrrrrmixed_query_layermixed_key_layermixed_value_layer query_layer key_layer value_layerattention_scoresrposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layerprojected_context_layerprojected_context_layer_dropoutlayernormed_context_layers rrrAlbertAttention.forward*s!JJ}5((=1 JJ}5//0AB --o> //0AB !<< 5H5HR5PQ+dii8P8P.QQ  %/@   ' '> 9T=Y=Y]q=q&++-a0J"\\*EJJ}OcOcdiijlnopN"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8FbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||U*V'$(NN=Cb3b$c!?P)?;rWpVrrr)r)rrrrrrrrrrr~rrrNNF)rrrrr rr^rrrrYrrrboolrrrrrrs@rrrrsu|u:%ell%u||% ;c;t;*7;15"' 8s||8s!!2!238sE--. 8s  8s uU\\"E%,, *D$EE F 8s8srrc^\rSrSrU4SjrS S\R S\\RS\\RS\ S\ \ \R \ \R \R 444 U4Sjjjr S r U=r$) AlbertSdpaAttentioniecf>[TU]U5 URUl[(+UlgN)rrr dropout_probrrequire_contiguous_qkvrs rrrAlbertSdpaAttention.__init__fs)  "??*L&L#rrrrrrc x>URS:wdU(a#[RS5 [TU]XUS9$UR 5upVnUR URU55nUR URU55n UR URU55n UR(aMURRS:Xa3Ub0UR5nU R5n U R5n [RR R#UU U UUR$(a UR&OSSS9n U R)SS 5n U R+XVUR,5n UR/U 5n UR1U 5n UR3X-5nU4$) NraAlbertSdpaAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support non-absolute `position_embedding_type` or `output_attentions=True` . Falling back to the eager attention implementation, but specifying the eager implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rcudaF)rrr attn_mask dropout_p is_causalrr@)r~rFwarningrrrrrrrrrtype contiguousr^r rscaled_dot_product_attentiontrainingrrZreshaperrrr))rrrrr batch_sizeseq_len_rrrattention_outputrrrrs rrrAlbertSdpaAttention.forwardks  ' ': 59J NNH 7?=Te?f f!.!3!3!5 Q// =0IJ --dhh}.EF // =0IJ  & &;+=+=+B+Bf+LQ_Qk%002K!,,.I%002K 88..KK$+/==d''c L ,55a;+33JI[I[\"&**-=">*.*=*=>U*V'$(NN=3b$c!)++r)rrr)rrrrrr^rrrrrrrrrrs@rrrresM7;15"' -,||-,!!2!23-,E--. -,  -, uU\\"E%,, *D$EE F -,-,rr)eagersdpac^\rSrSrS\4U4SjjrSS\RS\\RS\\RS\ S\ S \ \R\R44 S jjr S \RS \R4S jr S rU=r$) AlbertLayerirbc0>[TU]5 XlURUlSUl[ R "URURS9Ul [UR"U5Ul [ R"URUR5Ul[ R"URUR5Ul["UR$Ul[ R("UR*5Ulg)Nrry)rrrbchunk_size_feed_forward seq_len_dimr r)rrr(ALBERT_ATTENTION_CLASSES_attn_implementationr&rintermediate_sizer$ ffn_outputr hidden_act activationrrrrs rrrAlbertLayer.__init__s  '-'E'E$%'\\&2D2D&J_J_%`"1&2M2MNvV99V//1I1IJ))F$<$P>PQ !2!23zz&"<"<= rrrrroutput_hidden_statesrcURXX45n[URURURUS5nUR XvS-5nU4USS-$)Nrr)r&rff_chunkrrr()rrrrrr%rr!s rrrAlbertLayer.forwardsq >>-f. MM  ( (    Q   22:QR@S3ST "212"666rrclURU5nURU5nURU5nU$r)r$r#r!)rrr!s rrr'AlbertLayer.ff_chunks3XX./ __Z0 __Z0 r) r#r&rrbrr$r!r(rNNFF)rrrrr rr^rrrrrrr'rrrs@rrrrs >| > 7;15"'%* 7||7!!2!237E--. 7  7 # 7 u||U\\) *7(%,,rrc^\rSrSrS\4U4SjjrS S\RS\\RS\\RS\ S\ S \ \ \R\ \R4S 44 S jjr S rU=r$)AlbertLayerGroupirbc>[TU]5 [R"[ UR 5Vs/sHn[ U5PM sn5Ulgs snfr)rrr ModuleListrangeinner_group_numr albert_layersrrbrrs rrrAlbertLayerGroup.__init__sC ]]vOeOeIf+gIfAK,?If+gh+gsArrrrr%r.cSnSn[UR5H3upU "XX8U5n U SnU(aXzS4-nU(dM.Xa4-nM5 U4n U(aX4-n U(aX4-n U $)Nrr) enumerater2) rrrrrr%layer_hidden_stateslayer_attentions layer_index albert_layer layer_outputoutputss rrrAlbertLayerGroup.forwards!)243E3E)F %K' yG]_pqL(OM #3A6H#H ##&9[TU]5 Xl[R"UR UR 5Ul[R"[UR5Vs/sHn[U5PM sn5Ul gs snfr) rrrbr rrrembedding_hidden_mapping_inr/r0num_hidden_groupsr-albert_layer_groupsr3s rrrAlbertTransformer.__init__sj  +-99V5J5JFL^L^+_(#%==TYZ`ZrZrTs1tTsq2B62JTs1t#u 1ts-Brrrrr% return_dictrc URU5nU(aU4OSnU(aSOSnUcS/URR-OUn[URR5Hn [ URRURR - 5n [ XRRURR - - 5n UR U "UUX;U -U S-U -UU5n U SnU(aXS-nU(dMXq4-nM U(d[SXU455$[XUS9$)Nr6rrr|c3.# UH ocMUv M g7frr6).0vs rr ,AlbertTransformer.forward..sh$Vq$Vs )last_hidden_stater attentions) rBrbnum_hidden_layersr0rYrCrDtupler) rrrrrr%rFall_hidden_statesall_attentionsilayers_per_group group_idxlayer_group_outputs rrrAlbertTransformer.forwardsC88G 0D],$0d>G>OTFT[[:::U^ t{{445A"4;;#@#@4;;C`C`#`a A!>!>A^A^!^_`I!%!9!9)!D&66)a-K[9[\!$ " /q1M !/R2H!H##$58H$H!)6,h]~$Vhh h+Yg  r)rDrbrB)NNFFT)rrrrr rr^rrrrrrrrrrrs@rrr@r@sv|v7;15"'%* * ||* !!2!23* E--. *  * # * *  % &* * rr@c*\rSrSr\r\rSrSr Sr Sr g)AlbertPreTrainedModeli"albertTc[U[R5(akURRR SUR RS9 URb%URRR5 gg[U[R5(axURRR SUR RS9 URb2URRURR5 gg[U[R5(aJURRR5 URRRS5 g[U[5(a%URRR5 gg)zInitialize the weights.r)meanstdN?) isinstancer rr4r`normal_rbinitializer_ranger=zero_rrxr)fill_ AlbertMLMHead)rmodules rr _init_weights#AlbertPreTrainedModel._init_weights)s2 fbii ( ( MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S )  . . KK   " " $/rr6N) rrrrr config_classrsload_tf_weightsbase_model_prefix_supports_sdparfrr6rrrrYrY"sL/O N%rrYc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\ \R\ S'Sr\\ \R\ S'S rg) AlbertForPreTrainingOutputi<a Output type of [`AlbertForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). sop_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossprediction_logits sop_logitsrrNr6)rrrrrrnrr^r__annotations__rorprrrNrr6rrrrmrm<s}2)-D(5$$ %,59x 1 129.2J**+28[TU]U5 Xl[U5Ul[ U5UlU(aK[R"URUR5Ul [R"5Ul OSUl SUl URUlURUlUR!5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N)rrrbrur-r@encoderr rrr+Tanhpooler_activationrattn_implementationr~ post_init)rrbrtrs rrrAlbertModel.__init__cs   *62(0 ))F$6$68J8JKDK%'WWYD "DK%)D "#)#>#> '-'E'E$ rrc.URR$rr-rrs rrget_input_embeddings AlbertModel.get_input_embeddingszs...rrNc$XRlgrr})rrs rrset_input_embeddings AlbertModel.set_input_embeddings}s */'rheads_to_prunec>UR5Hup#[X RR- 5n[X$URR-- 5nURR UR URRU5 M g)a Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} ALBERT has a different architecture in that its layers are shared across groups, which then has inner groups. If an ALBERT model has 12 hidden layers and 2 hidden groups, with two inner groups, there is a total of 4 different layers. These layers are flattened: the indices [0,1] correspond to the two inner groups of the first hidden layer, while [2,3] correspond to the two inner groups of the second hidden layer. Any layer with in index other than [0,1,2,3] will result in an error. See base class PreTrainedModel for more information about head pruning N) itemsrYrbr1rvrDr2r&r)rrlayerrrUinner_group_idxs rr _prune_headsAlbertModel._prune_headss{+002LEEKK$?$??@I!%dkk6Q6Q*Q"QRO LL , ,Y 7 E Eo V ` ` l lmr s3rrrrr{rrrr%rFc UbUOURRnUbUOURRnU bU OURRn UbUb [ S5eUb"UR X5 UR 5n O"UbUR 5SSn O [ S5eU upUb URO URn Uc[R"XS9nUcr[URS5(a3URRSS2SU 24nURX5nUnO$[R"U [RU S9nURXX6S9nUR S:H=(a( UR"S :H=(a USL=(a U(+nU(a[%UUR&U S 9nOiUR)S 5R)S 5nUR+UR&S 9nSU- [R,"UR&5R.-nUR1XPRR25nUR5UUUUUU S9nUSnUR6b'UR9UR7USS2S455OSnU (d UU4US S-$[;UUUR<UR>S9$)NzDYou cannot specify both input_ids and inputs_embeds at the same timer|z5You have to specify either input_ids or inputs_embeds)rrr)r{rrrr)tgt_lenrr@rr^)rrr%rFr)rM pooler_outputrrN) rbrr%use_return_dictr\%warn_if_padding_and_no_attention_maskrrr^onesrr-rrrrryr~rr unsqueezerfinfomin get_head_maskrOrvr+rxrrrN)rrrrr{rrrr%rFrrrrrrembedding_outputuse_sdpa_attention_maskextended_attention_maskencoder_outputssequence_output pooled_outputs rrrAlbertModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, %.%:!! @T@T  !"ZZ CN  !t(899*.//*H*HKZK*X'3J3Q3QR\3i0!A!&[ SY!Z?? +   $ $ . &,, : &T! &&%  #&I 0 6 6 ' #'5&>&>q&A&K&KA&N #&=&@&@tzz&@&R #'*-D'D TXT^T^H_HcHc&c #&&y++2O2OP ,,  #/!5# ' *!,VZVaVaVm..t{{?1a4;P/QRsw #]3oab6II I)-')77&11   r)ryrbr-rvr+rxr~)T) NNNNNNNNN)rrrrr rhrjrrr rrrrrYrrrrr^rrrrrrrrrs@rrrsrs^shL |./bll/0",,040t4T#Y+?tDt"156:59371559,0/3&*T E,,-T !!2!23T !!1!12 T u//0 T E--. T   1 12T $D>T 'tnT d^T  )50 1T T rrsz Albert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `sentence order prediction (classification)` head. ) custom_introc^\rSrSrSS/rS\4U4SjjrS\R4Sjr S\RSS 4S jr S\R4S jr \ SS \\R S \\R"S\\R S\\R S\\R"S\\R"S\\R S\\R S\\S\\S\\S\\\44Sjj5rSrU=r$)AlbertForPreTrainingipredictions.decoder.biaspredictions.decoder.weightrbc>[TU]U5 [U5Ul[ U5Ul[ U5UlUR5 gr) rrrsrZrdr, AlbertSOPHeadsop_classifierrzrs rrrAlbertForPreTraining.__init__sB  !&) (0+F3 rrc.URR$rr,decoderr~s rrget_output_embeddings*AlbertForPreTraining.get_output_embeddings'''rnew_embeddingsNc$XRlgrrrrs rrset_output_embeddings*AlbertForPreTraining.set_output_embeddingss#1 rcBURRR$rrZr-rr~s rrr)AlbertForPreTraining.get_input_embeddings{{%%555rrrrr{rrlabelssentence_order_labelrr%rFc (U bU OURRn URUUUUUUU U U S9 n U SSupURU 5nUR U5nSnUbvUbs[ 5nU"UR SURR5UR S55nU"UR SS5UR S55nUU-nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` sentence_order_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`. `0` indicates original order (sequence A, then sequence B), `1` indicates switched order (sequence B, then sequence A). Example: ```python >>> from transformers import AutoTokenizer, AlbertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("albert/albert-base-v2") >>> model = AlbertForPreTraining.from_pretrained("albert/albert-base-v2") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) >>> # Batch size 1 >>> outputs = model(input_ids) >>> prediction_logits = outputs.prediction_logits >>> sop_logits = outputs.sop_logits ```Nrrr{rrrr%rFr@r|)rnrorprrN) rbrrZr,rr rrrmrrN)rrrrr{rrrrrr%rFr=rrprediction_scores sop_scores total_lossloss_fctmasked_lm_losssentence_order_lossoutputs rrrAlbertForPreTraining.forwardsKR&1%L 'tnL d^L  )50 1L L rrct^\rSrSrS\4U4SjjrS\RS\R4SjrS Sjr Sr U=r $) rdiUrbc>[TU]5 [R"URUR S9Ul[R "[R"UR55Ul [R"URUR5Ul [R"URUR5Ul[UR UlURURl g)Nry)rrr r)rr Parameterr^rrr=rrrrrr"r#rs rrrAlbertMLMHead.__init__Vs f&;&;AVAVWLLV->->!?@ YYv1163H3HI yy!6!68I8IJ  !2!23 II rrrcURU5nURU5nURU5nURU5nUnU$r)rr#r)r)rrrs rrrAlbertMLMHead.forward`sF =1  6 }5  ]3 )  rcURRRRS:XaURURlgURRUlg)Nmeta)rr=rr r~s rr _tie_weightsAlbertMLMHead._tie_weightsjsC <<   # # ( (F 2 $ DLL  ))DIr)r)r#r=rr)rN) rrrrr rr^rrrrrrs@rrrdrdUs4&|&!U\\!ell!**rrdcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $)risrbc>[TU]5 [R"UR5Ul[R "URUR5Ul gr) rrr rclassifier_dropout_probrrr num_labelsr?rs rrrAlbertSOPHead.__init__tsB zz&"@"@A ))F$6$68I8IJrrrcJURU5nURU5nU$r)rr?)rrdropout_pooled_outputlogitss rrrAlbertSOPHead.forwardzs% $ ] ;!67 r)r?r) rrrrr rr^rrrrrs@rrrrss1K|K U\\ellrrc^\rSrSrSS/rU4SjrS\R4SjrS\RSS4S jr S\R4S jr \ SS \ \RS \ \R S \ \RS\ \RS\ \R S\ \R S\ \RS\ \S\ \S\ \S\\\44Sjj5rSrU=r$)AlbertForMaskedLMirrc>[TU]U5 [USS9Ul[ U5UlUR 5 gNF)rt)rrrsrZrdr,rzrs rrrAlbertForMaskedLM.__init__s7  !&EB (0 rrc.URR$rrr~s rrr'AlbertForMaskedLM.get_output_embeddingsrrrNcZXRlURURlgr)r,rr=rs rrr'AlbertForMaskedLM.set_output_embeddingss"#1 . 3 3rcBURRR$rrr~s rrr&AlbertForMaskedLM.get_input_embeddingsrrrrrr{rrrrr%rFc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n SnUbF[ 5nU"U R SURR 5UR S55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)a$ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, AlbertForMaskedLM >>> tokenizer = AutoTokenizer.from_pretrained("albert/albert-base-v2") >>> model = AlbertForMaskedLM.from_pretrained("albert/albert-base-v2") >>> # add mask_token >>> inputs = tokenizer("The capital of [MASK] is Paris.", return_tensors="pt") >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> # retrieve index of [MASK] >>> mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0] >>> predicted_token_id = logits[0, mask_token_index].argmax(axis=-1) >>> tokenizer.decode(predicted_token_id) 'france' ``` ```python >>> labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"] >>> labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100) >>> outputs = model(**inputs, labels=labels) >>> round(outputs.loss.item(), 2) 0.81 ``` N rrrr{rrrr%rFrr|r@rnrrrN) rbrrZr,r rrrrrN)rrrrr{rrrrr%rFr=sequence_outputsrrrrs rrrAlbertForMaskedLM.forwardsb&1%  ')H%&7&<&O 'tnO d^O  ~u$ %O O rrz Albert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cZ^\rSrSrS\4U4Sjjr\SS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \44Sjj5rSrU=r$)AlbertForSequenceClassificationirbcP>[TU]U5 URUlXl[ U5Ul[ R"UR5Ul [ R"URURR5Ul UR5 gr)rrrrbrsrZr rrrrrr?rzrs rrr(AlbertForSequenceClassification.__init__sr   ++ !&) zz&"@"@A ))F$6$6 8N8NO rrrrr{rrrrr%rFrc RU bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUGbURR cUR S:XaSURlOoUR S:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5nUR S:Xa&U"U R5UR55nOU"X5nOURR S:Xa=[5nU"U RSUR 5URS55nO,URR S:Xa[5nU"X5nU (dU 4U SS-nUbU4U-$U$[!UU U R"U R$S 9$) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr regressionsingle_label_classificationmulti_label_classificationr|r@r)rbrrZrr? problem_typerrr^rrYr squeezer rr rrrN)rrrrr{rrrrr%rFr=rrrnrrs rrr'AlbertForSequenceClassification.forwards(&1%F 'tnF d^F  '. /F F rrcZ^\rSrSrS\4U4Sjjr\SS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \44Sjj5rSrU=r$)AlbertForTokenClassificationiGrbcx>[TU]U5 URUl[USS9UlUR b UR O UR n[R"U5Ul [R"URURR5Ul UR5 gr)rrrrsrZrrr rrrrrbr?rz)rrbrrs rrr%AlbertForTokenClassification.__init__Is   ++!&EB --9  * *++ zz"9: ))F$6$6 8N8NO rrrrr{rrrrr%rFrc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUb<[ 5nU"U R SUR5UR S55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr|r@r) rbrrZrr?r rrrrrN)rrrrr{rrrrr%rFr=rrrnrrs rrr$AlbertForTokenClassification.forwardYs$&1%2 'tn2 d^2  $e+ ,2 2 rrcz^\rSrSrS\4U4Sjjr\SS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\RS \\ S \\ S\\ S\ \ \44Sjj5rSrU=r$)AlbertForQuestionAnsweringirbc>[TU]U5 URUl[USS9Ul[ R "URUR5UlUR5 gr) rrrrsrZr rr qa_outputsrzrs rrr#AlbertForQuestionAnswering.__init__sU   ++!&EB ))F$6$68I8IJ rrrrr{rrstart_positions end_positionsrr%rFrc $U bU OURRn URUUUUUUU U U S9 n U Sn URU 5nUR SSS9unnUR S5R 5nUR S5R 5nSnUbUb[UR55S:aUR S5n[UR55S:aUR S5nURS5nURSU5nURSU5n[US9nU"X5nU"UU5nUU-S- nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$) Nrrrr|r) ignore_indexr@)rn start_logits end_logitsrrN)rbrrZrrTrrrSrclampr rrrN)rrrrr{rrrrrr%rFr=rrrrr ignored_indexr start_lossend_lossrs rrr"AlbertForQuestionAnswering.forwards&1% E,,-> !!2!23> !!1!12 > u//0 > E--. >   1 12> "%"2"23>   0 01> $D>> 'tn> d^>  )50 1> > rrcZ^\rSrSrS\4U4Sjjr\SS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \44Sjj5rSrU=r$)AlbertForMultipleChoiceirbc>[TU]U5 [U5Ul[R "UR 5Ul[R"URS5Ul UR5 g)Nr) rrrsrZr rrrrrr?rzrs rrr AlbertForMultipleChoice.__init__sV  !&) zz&"@"@A ))F$6$6: rrrrr{rrrrr%rFrc ZU bU OURRn UbURSOURSn Ub!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb1URSUR S5UR S55OSnUR UUUUUUUU U S9 n U Sn UR U 5n URU 5nURSU 5nSnUb[5nU"X5nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where *num_choices* is the size of the second dimension of the input tensors. (see *input_ids* above) Nrr|rrr@r) rbrr[rrrZrr?r rrrN)rrrrr{rrrrr%rF num_choicesr=rrreshaped_logitsrnrrs rrrAlbertForMultipleChoice.forwardsX&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  ++ ))%'/!5#     ]3 #}= ++b+6  ')HO4D%''!"+5F)-)9TGf$ Ev E("!//))   r)rZr?rrrrs@rrr r s |156:59371559-1,0/3&*W E,,-W !!2!23W !!1!12 W u//0 W E--. W   1 12W ))*W $D>W 'tnW d^W  )50 1W W rr ) rsrYrsrrrrrr )BrrrH dataclassesrtypingrrrrrr^r torch.nnr r r activationsrmodeling_attn_mask_utilsrmodeling_outputsrrrrrrrmodeling_utilsr pytorch_utilsrrrrutilsrrrconfiguration_albertr get_loggerrrFrsModulerurrrrr-r@rYrmrsrrdrrrrrr __all__r6rrrrs7 !55 AA!K. :9.   H %{|AryyAHmsbiims`3,/3,n  &"))&R ryy F2  2 j%O%%2 :: :BG 'G G T c 0c  c L*BII*< BII f -f f R T &;T  T nD #8D D NJ !6J J Zc 3c c L r