o Zh@s<ddlZddlZddlZddlmZddlmZmZmZddl Z ddl m Z ddl m Z m Z mZddlmZddlmZmZmZmZmZmZmZmZdd lmZdd lmZmZdd lm Z m!Z!m"Z"d d l#m$Z$e"%e&Z'ddZ(Gddde j)Z*e j+e*dZ,Gddde j)Z-Gddde j)Z.Gddde j)Z/Gddde j)Z0Gddde j)Z1Gddde j)Z2Gdd d e j)Z3Gd!d"d"e j)Z4Gd#d$d$e j)Z5Gd%d&d&e j)Z6Gd'd(d(e j)Z7Gd)d*d*e j)Z8Gd+d,d,e j)Z9Gd-d.d.e j)Z:Gd/d0d0e j)Z;Gd1d2d2e j)Ze!Gd7d8d8eZ?eGd9d:d:e Z@e!Gd;d<dGd?d@d@e?ZBe!GdAdBdBe?ZCGdCdDdDe j)ZDe!dEd>GdFdGdGe?ZEe!dHd>GdIdJdJe?ZFe!GdKdLdLe?ZGe!GdMdNdNe?ZHe!GdOdPdPe?ZIgdQZJdS)RN) dataclass)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging)MobileBertConfigc Cszddl}ddl}ddl}Wn tytdwtj|}t d||j |}g}g} |D] \} } t d| d| |j || } | | | | q6t|| D]\} } | dd} | d d } | d d } | d d} | d} tdd| Drt dd| q\|} | D]~}|d|r|d|}n|g}|ddks|ddkrt| d} nI|ddks|ddkrt| d} n7|ddkrt| d} n+|ddkrt| d} nz t| |d} Wntyt dd| Yqwt|dkrt|d}| |} q|d dd!kr%t| d} n |dkr/|| } z| j| jksDJd"| jd#| jd$Wnty^}z |j| j| jf7_d}~wwt d%| t| | _q\|S)&z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape Z ffn_layerffnZ FakeLayerNorm LayerNormZextra_output_weightsz dense/kernelZbert mobilebert/css|]}|dvVqdS))Zadam_vZadam_mZAdamWeightDecayOptimizerZAdamWeightDecayOptimizer_1Z global_stepN).0nr r a/var/www/auris/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py Vs  z0load_tf_weights_in_mobilebert..z Skipping z [A-Za-z]+_\d+z_(\d+)ZkernelgammaweightZ output_biasbetabiasZoutput_weightsZsquad classifierriZ _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )renumpyZ tensorflow ImportErrorloggererrorospathabspathinfotrainZlist_variablesZ load_variableappendzipreplacesplitanyjoin fullmatchgetattrAttributeErrorlenint transposeshapeAssertionErrorargstorchZ from_numpydata)modelconfigZtf_checkpoint_pathr+nptfZtf_pathZ init_varsnamesZarraysnamerAarrayZpointerZm_nameZ scope_namesnumer r r#load_tf_weights_in_mobilebert5s                       rOcs4eZdZdfdd ZdejdejfddZZS) NoNormNcs2ttt||_tt||_dSN) super__init__r ParameterrDzerosr(onesr&)selfZ feat_sizeeps __class__r r#rSs zNoNorm.__init__ input_tensorreturncCs||j|jSrQ)r&r()rWr[r r r#forwardszNoNorm.forwardrQ__name__ __module__ __qualname__rSrDTensorr] __classcell__r r rYr#rPsrP) layer_normZno_normc sbeZdZdZfddZ    d deejdeejdeejdeejd ej f d d Z Z S) MobileBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cst|j|_|j|_|j|_tj|j|j|jd|_ t|j |j|_ t|j |j|_ |jr4dnd}|j|}t||j|_t|j|j|_t|j|_|jdt|j ddddS)N) padding_idxr r position_ids)rF) persistent)rRrS trigram_inputembedding_size hidden_sizer Embedding vocab_sizeZ pad_token_idword_embeddingsZmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddingsLinearembedding_transformationNORM2FNnormalization_typerDropouthidden_dropout_probdropoutZregister_bufferrDZarangeexpand)rWrGZembed_dim_multiplierZembedded_input_sizerYr r#rSs   zMobileBertEmbeddings.__init__N input_idstoken_type_idsrg inputs_embedsr\c Cs*|dur |}n|dd}|d}|dur$|jddd|f}|dur3tj|tj|jjd}|dur<||}|jrktjt j j |ddddfgddd|t j j |ddddfgdddgdd }|jst|j |j kry||}||}||}|||} || } || } | S) Nrhrdtypedevice)rrrrrr)value)rrrrrrr*dim)sizergrDrUlongrrorjcatr functionalpadrkrlrsrprqrrx) rWrzr{rgr| input_shapeZ seq_lengthrprq embeddingsr r r#r]s4  $$      zMobileBertEmbeddings.forward)NNNN) r_r`ra__doc__rSrrD LongTensor FloatTensorrbr]rcr r rYr#res$ recsneZdZfddZddZ   ddejdejdejd eejd eejd ee d e ejfd dZ Z S)MobileBertSelfAttentioncst|j|_t|j|j|_|j|j|_t|j|j|_ t|j|j|_ t|j r3|jn|j |j|_ t|j|_dSrQ)rRrSnum_attention_headsr?true_hidden_sizeattention_head_size all_head_sizerrrquerykeyuse_bottleneck_attentionrlrrvZattention_probs_dropout_probrxrWrGrYr r#rSs z MobileBertSelfAttention.__init__cCs6|dd|j|jf}||}|ddddS)Nrhrr*rr )rrrviewpermute)rWxZ new_x_shaper r r#transpose_for_scoress z,MobileBertSelfAttention.transpose_for_scoresN query_tensor key_tensor value_tensorattention_mask head_maskoutput_attentionsr\cCs||}||}||} ||} ||} || } t| | dd} | t|j } |dur8| |} t j j | dd}| |}|durM||}t|| }|dddd}|dd|jf}||}|rv||f}|S|f}|S)Nrhrrr*rr )rrrrrDmatmulr@mathsqrtrrrZsoftmaxrxr contiguousrrr)rWrrrrrrZmixed_query_layerZmixed_key_layerZmixed_value_layerZ query_layerZ key_layerZ value_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr r r#r]s,         zMobileBertSelfAttention.forwardNNN) r_r`rarSrrDrbrrboolrr]rcr r rYr#rs*   rc8eZdZfddZdejdejdejfddZZS)MobileBertSelfOutputcsXt|j|_t|j|j|_t|j|j|j d|_ |js*t |j |_ dSdSNrX)rRrSuse_bottleneckrrrrdensertrulayer_norm_epsrrvrwrxrrYr r#rSs zMobileBertSelfOutput.__init__ hidden_statesresidual_tensorr\cCs,||}|js ||}|||}|SrQ)rrrxrrWrr layer_outputsr r r#r]s  zMobileBertSelfOutput.forwardr^r r rYr#rs $rcsteZdZfddZddZ   ddejdejdejd ejd eejd eejd ee d e ejfddZ Z S)MobileBertAttentioncs*tt||_t||_t|_dSrQ)rRrSrrWroutputset pruned_headsrrYr r#rS(s    zMobileBertAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)r>rrWrrrrrrrrrrunion)rWheadsindexr r r# prune_heads.s zMobileBertAttention.prune_headsNrrr layer_inputrrrr\c Cs:|||||||}||d|} | f|dd} | S)Nrr)rWr) rWrrrrrrrZ self_outputsattention_outputrr r r#r]@s  zMobileBertAttention.forwardr) r_r`rarSrrDrbrrrrr]rcr r rYr#r's.  rc2eZdZfddZdejdejfddZZS)MobileBertIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSrQ) rRrSrrrrintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrrYr r#rSZs   zMobileBertIntermediate.__init__rr\cCs||}||}|SrQ)rrrWrr r r#r]b  zMobileBertIntermediate.forwardr^r r rYr#rYs rcr)OutputBottleneckcsFtt|j|j|_t|j|j|j d|_ t |j |_ dSr)rRrSrrrrrlrrtrurrrvrwrxrrYr r#rSis zOutputBottleneck.__init__rrr\cCs&||}||}|||}|SrQ)rrxrrr r r#r]os  zOutputBottleneck.forwardr^r r rYr#rhs $rcs>eZdZfddZdejdejdejdejfddZZS) MobileBertOutputcs\t|j|_t|j|j|_t|j |j|_ |js't |j |_ dSt||_dSrQ)rRrSrrrrrrrrtrurrvrwrxr bottleneckrrYr r#rSws zMobileBertOutput.__init__intermediate_statesresidual_tensor_1residual_tensor_2r\cCsJ||}|js||}|||}|S|||}|||}|SrQ)rrrxrr)rWrrr layer_outputr r r#r]s   zMobileBertOutput.forwardr^r r rYr#rvs  rcr)BottleneckLayerc8tt|j|j|_t|j|j|j d|_ dSr) rRrSrrrrlZintra_bottleneck_sizerrtrurrrrYr r#rS zBottleneckLayer.__init__rr\cCs||}||}|SrQrr)rWrrr r r#r]rzBottleneckLayer.forwardr^r r rYr#r rcs6eZdZfddZdejdeejfddZZS) Bottleneckcs<t|j|_|j|_t||_|jrt||_dSdSrQ)rRrSkey_query_shared_bottleneckrrinput attentionrrYr r#rSs  zBottleneck.__init__rr\cCsB||}|jr |fdS|jr||}||||fS||||fS)N)rrrr)rWrZbottlenecked_hidden_statesZshared_attention_inputr r r#r]s     zBottleneck.forward r_r`rarSrDrbrr]rcr r rYr#rs "rcr) FFNOutputcrr) rRrSrrrrrrrtrurrrrYr r#rSrzFFNOutput.__init__rrr\cCs||}|||}|SrQrrr r r#r]s zFFNOutput.forwardr^r r rYr#rs $rcr)FFNLayercs"tt||_t||_dSrQ)rRrSr intermediaterrrrYr r#rSs  zFFNLayer.__init__rr\cCs||}|||}|SrQ)rr)rWrintermediate_outputrr r r#r]s  zFFNLayer.forwardr^r r rYr#rrrc sZeZdZfddZ   d dejdeejdeejdeede ejf d d Z Z S) MobileBertLayercs~tj|_j|_t|_t|_t|_ |jr$t |_ jdkr=t fddtjdD|_dSdS)Nrcg|]}tqSr )rr!_rGr r# z,MobileBertLayer.__init__..)rRrSrnum_feedforward_networksrrrrrrrrr ModuleListrangerrrYrr#rSs      (zMobileBertLayer.__init__Nrrrrr\c Cs|jr ||\}}}}n |gd\}}}}|j|||||||d} | d} | f} | dd} |jdkrGt|jD] \} }|| } | | f7} q9|| }||| |}|f| t d||||| |f| } | S)Nr)rrri) rrrr enumeraterrrrDZtensor)rWrrrrrrrrZself_attention_outputsrsriZ ffn_modulerrr r r#r]sJ      zMobileBertLayer.forwardr) r_r`rarSrDrbrrrrr]rcr r rYr#rs  rcspeZdZfddZ     ddejdeejdeejd eed eed eed e e e ffd dZ Z S)MobileBertEncodercs.ttfddtjD|_dS)Ncrr )rrrr r#rrz.MobileBertEncoder.__init__..)rRrSrrrnum_hidden_layerslayerrrYrr#rSs $zMobileBertEncoder.__init__NFTrrrroutput_hidden_states return_dictr\c Cs|rdnd}|r dnd}t|jD]!\} } |r||f}| |||| |} | d}|r2|| df}q|r:||f}|sHtdd|||fDSt|||dS)Nr rrcss|] }|dur|VqdSrQr )r!vr r r#r$9sz,MobileBertEncoder.forward..)last_hidden_stater attentions)rrtupler ) rWrrrrrrZall_hidden_statesZall_attentionsrZ layer_modulerr r r#r]s,   zMobileBertEncoder.forward)NNFFT)r_r`rarSrDrbrrrrrr r]rcr r rYr#rs,  rcr)MobileBertPoolercs2t|j|_|jrt|j|j|_dSdSrQ)rRrSZclassifier_activation do_activaterrrrlrrrYr r#rS@s zMobileBertPooler.__init__rr\cCs2|dddf}|js |S||}t|}|S)Nr)rrrDtanh)rWrZfirst_token_tensor pooled_outputr r r#r]Fs   zMobileBertPooler.forwardr^r r rYr#r?s rcr)!MobileBertPredictionHeadTransformcsXtt|j|j|_t|jtrt |j|_ n|j|_ t d|j|j d|_ dS)Nrdr)rRrSrrrrlrrrrr transform_act_fnrtrrrrYr r#rSSs  z*MobileBertPredictionHeadTransform.__init__rr\cCs"||}||}||}|SrQ)rrrrr r r#r]\s   z)MobileBertPredictionHeadTransform.forwardr^r r rYr#rRs  rcs<eZdZfddZd ddZdejdejfdd ZZS) MobileBertLMPredictionHeadcshtt||_tj|j|j|jdd|_ tj|j|jdd|_ t t |j|_|j|j _dS)NF)r()rRrSr transformrrrrnrlrkrdecoderrTrDrUr(rrYr r#rSds  z#MobileBertLMPredictionHead.__init__r\NcCs|j|j_dSrQ)r(rrWr r r# _tie_weightsosz'MobileBertLMPredictionHead._tie_weightsrcCs>||}|tj|jj|jjgdd}||jj7}|S)Nrr) rrrDrrr&trr(rr r r#r]rs $ z"MobileBertLMPredictionHead.forward)r\N) r_r`rarSrrDrbr]rcr r rYr#rcs  rcr)MobileBertOnlyMLMHeadcstt||_dSrQ)rRrSr predictionsrrYr r#rSzs zMobileBertOnlyMLMHead.__init__sequence_outputr\cC||}|SrQ)r)rWrprediction_scoresr r r#r]~ zMobileBertOnlyMLMHead.forwardr^r r rYr#ry rcs<eZdZfddZdejdejdeejfddZZS)MobileBertPreTrainingHeadscs(tt||_t|jd|_dSNr*)rRrSrrrrrrlseq_relationshiprrYr r#rSs  z#MobileBertPreTrainingHeads.__init__rrr\cCs||}||}||fSrQ)rr)rWrrrseq_relationship_scorer r r#r]s  z"MobileBertPreTrainingHeads.forwardrr r rYr#rs (rc@s eZdZeZeZdZddZdS)MobileBertPreTrainedModelrcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj t frZ|jj |jjddSt|trg|jj dSdS)zInitialize the weightsr)meanZstdNg?)rrrrr&rEZnormal_rGZinitializer_ranger(Zzero_rmrfrrPZfill_r)rWmoduler r r# _init_weightss      z'MobileBertPreTrainedModel._init_weightsN) r_r`rarZ config_classrOZload_tf_weightsZbase_model_prefixr r r r r#r s  r c@steZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ ee ejed<dS)MobileBertForPreTrainingOutputab Output type of [`MobileBertForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossprediction_logitsseq_relationship_logitsrr)r_r`rarrrrDr__annotations__rrrrrr r r r#rs rcseZdZdZdfdd ZddZddZd d Ze dd e e j d e e j de e j de e j de e j de e j de e de e de e deeeffddZZS)MobileBertModelz. https://arxiv.org/pdf/2004.02984.pdf TcsDt|||_t||_t||_|rt|nd|_| dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rRrSrGrerrencoderrpooler post_init)rWrGadd_pooling_layerrYr r#rSs    zMobileBertModel.__init__cCs|jjSrQrrorr r r#get_input_embeddingssz$MobileBertModel.get_input_embeddingscCs ||j_dSrQr)rWrr r r#set_input_embeddingss z$MobileBertModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rWZheads_to_prunerrr r r# _prune_headsszMobileBertModel._prune_headsNrzrr{rgrr|rrrr\c Csh|dur|n|jj}|dur|n|jj}| dur| n|jj} |dur*|dur*td|dur9||||} n|durF|dd} ntd|durQ|jn|j} |dur_tj | | d}|durltj | tj | d}| || } | ||jj}|j||||d} |j| | |||| d}|d}|jdur||nd}| s||f|d dSt|||j|jd S) NzDYou cannot specify both input_ids and inputs_embeds at the same timerhz5You have to specify either input_ids or inputs_embeds)rr})rzrgr{r|)rrrrrrr)rZ pooler_outputrr)rGrruse_return_dict ValueErrorZ%warn_if_padding_and_no_attention_maskrrrDrVrUrZget_extended_attention_maskZ get_head_maskrrrrr rr)rWrzrr{rgrr|rrrrrZextended_attention_maskZembedding_outputZencoder_outputsrrr r r#r]sP    zMobileBertModel.forward)T) NNNNNNNNN)r_r`rarrSrrrrrrDrrrrrr r]rcr r rYr#rsJ   rz MobileBert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. )Z custom_introcseZdZddgZfddZddZddZdd eed e j ffd d Z e ddee jdee jdee jdee jdee jdee jdee jdee jdee jdee jdee jd eeeffddZZS)MobileBertForPreTrainingcls.predictions.decoder.weightcls.predictions.decoder.biasc,t|t||_t||_|dSrQ)rRrSrrrclsrrrYr r#rS>s    z!MobileBertForPreTraining.__init__cC |jjjSrQr#rrrr r r#get_output_embeddingsF z.MobileBertForPreTraining.get_output_embeddingscC||jj_|j|jj_dSrQr#rrr(rWZnew_embeddingsr r r#set_output_embeddingsI z.MobileBertForPreTraining.set_output_embeddingsNnew_num_tokensr\c*|j|jjj|dd|jj_tj|dSNT)r-Z transposed)r-Z_get_resized_lm_headr#rrrRresize_token_embeddingsrWr-rYr r#r1Ms  z0MobileBertForPreTraining.resize_token_embeddingsrzrr{rgrr|labelsnext_sentence_labelrrrc  Cs| dur| n|jj} |j||||||| | | d } | dd\} }|| |\}}d}|durS|durSt}||d|jj|d}||dd|d}||}| sj||f| dd}|durh|f|S|St|||| j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Examples: ```python >>> from transformers import AutoTokenizer, MobileBertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForPreTraining.from_pretrained("google/mobilebert-uncased") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) >>> # Batch size 1 >>> outputs = model(input_ids) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ```Nrr{rgrr|rrrr*rh)rrrrr) rGrrr#rrrnrrr)rWrzrr{rgrr|r3r4rrrrrrrr  total_lossloss_fctmasked_lm_lossnext_sentence_lossrr r r#r]Us<+ z MobileBertForPreTraining.forwardrQ NNNNNNNNNNN)r_r`ra_tied_weights_keysrSr&r+rr?rrmr1rrDrrrrrr]rcr r rYr#r5sV      rcseZdZddgZfddZddZddZdd eed e j ffd d Z e ddee jdee jdee jdee jdee jdee jdee jdeedeedeed eeeffddZZS)MobileBertForMaskedLMr r!cs6t|t|dd|_t||_||_|dSNF)r)rRrSrrrr#rGrrrYr r#rSs   zMobileBertForMaskedLM.__init__cCr$rQr%rr r r#r&r'z+MobileBertForMaskedLM.get_output_embeddingscCr(rQr)r*r r r#r+r,z+MobileBertForMaskedLM.set_output_embeddingsNr-r\cr.r/r0r2rYr r#r1s  z-MobileBertForMaskedLM.resize_token_embeddingsrzrr{rgrr|r3rrrc  Cs| dur| n|jj} |j|||||||| | d } | d} || } d}|dur8t}|| d|jj|d}| sN| f| dd}|durL|f|S|St|| | j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nr5rrhr*rlogitsrr) rGrrr#rrrnrrr)rWrzrr{rgrr|r3rrrrrrr8r7rr r r#r]s6 zMobileBertForMaskedLM.forwardrQ NNNNNNNNNN)r_r`rar;rSr&r+rr?rrmr1rrDrrrrrrr]rcr r rYr#r<sP      r<cr)MobileBertOnlyNSPHeadcstt|jd|_dSr)rRrSrrrrlrrrYr r#rSs zMobileBertOnlyNSPHead.__init__rr\cCrrQ)r)rWrr r r r#r]rzMobileBertOnlyNSPHead.forwardr^r r rYr#rArrAzZ MobileBert Model with a `next sentence prediction (classification)` head on top. cseZdZfddZe          ddeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e e ffddZ ZS)#MobileBertForNextSentencePredictioncr"rQ)rRrSrrrAr#rrrYr r#rSs    z,MobileBertForNextSentencePrediction.__init__Nrzrr{rgrr|r3rrrr\c  Ksd| vrtdt| d}| dur| n|jj} |j|||||||| | d } | d} || }d}|durEt}|| dd| d}| s[|f| dd}|durY|f|S|St ||| j | j dS) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`. - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Examples: ```python >>> from transformers import AutoTokenizer, MobileBertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForNextSentencePrediction.from_pretrained("google/mobilebert-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> loss = outputs.loss >>> logits = outputs.logits ```r4zoThe `next_sentence_label` argument is deprecated and will be removed in a future version, use `labels` instead.Nr5rrhr*r>) warningswarn FutureWarningpoprGrrr#rrrrr)rWrzrr{rgrr|r3rrrkwargsrrr r9r7rr r r#r]sB)  z+MobileBertForNextSentencePrediction.forwardr@)r_r`rarSrrrDrrrrrrr]rcr r rYr#rBsH      rBz MobileBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfddZe          ddeejdeejdeejdeejdeejd eejd eejd eed eed eede e eje ffddZ Z S)#MobileBertForSequenceClassificationcsdt||j|_||_t||_|jdur|jn|j}t ||_ t |j |j|_ |dSrQ)rRrS num_labelsrGrrclassifier_dropoutrwrrvrxrrrlr)rrWrGrKrYr r#rSis    z,MobileBertForSequenceClassification.__init__Nrzrr{rgrr|r3rrrr\c  Csr| dur| n|jj} |j|||||||| | d } | d} || } || } d}|dur|jjdurV|jdkr 1` a classification loss is computed (Cross-Entropy). Nr5rZ regressionZsingle_label_classificationZmulti_label_classificationrhr*r>)rGrrrxr)Z problem_typerJr~rDrr?r squeezerrrrrr)rWrzrr{rgrr|r3rrrrrr?rr7rr r r#r]xsV     "       z+MobileBertForSequenceClassification.forwardr@)r_r`rarSrrrDrbrrrrr]rcr r rYr#rIasH     rIcseZdZfddZe           ddeejdeejdeejdeejdeejd eejd eejd eejd eed eedeede e eje ffddZ Z S)MobileBertForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSr=) rRrSrJrrrrrrl qa_outputsrrrYr r#rSs  z'MobileBertForQuestionAnswering.__init__Nrzrr{rgrr|start_positions end_positionsrrrr\c  CsH| dur| n|jj} |j||||||| | | d } | d} || }|jddd\}}|d}|d}d}|dur|durt|dkrO|d}t|dkr\|d}|d}| d|}| d|}t |d}|||}|||}||d}| s||f| dd}|dur|f|S|St |||| j | j dS) Nr5rrrhr)Z ignore_indexr*)r start_logits end_logitsrr)rGrrrOr8rMrr>rclamprrrr)rWrzrr{rgrr|rPrQrrrrrr?rRrSr6Z ignored_indexr7Z start_lossZend_lossrr r r#r]sP          z&MobileBertForQuestionAnswering.forwardr:)r_r`rarSrrrDrbrrrrr]rcr r rYr#rNsN       rNcrH)MobileBertForMultipleChoicecsTt|t||_|jdur|jn|j}t||_t |j d|_ | dS)Nr) rRrSrrrKrwrrvrxrrrlr)rrLrYr r#rSs    z$MobileBertForMultipleChoice.__init__Nrzrr{rgrr|r3rrrr\c  Csn| dur| n|jj} |dur|jdn|jd} |dur%|d|dnd}|dur4|d|dnd}|durC|d|dnd}|durR|d|dnd}|dure|d|d|dnd}|j|||||||| | d } | d} || } || }|d| }d}|durt}|||}| s|f| dd}|dur|f|S|St ||| j | j dS)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrrhrr5r*r>) rGrrArrrrxr)rrrr)rWrzrr{rgrr|r3rrrZ num_choicesrrr?Zreshaped_logitsrr7rr r r#r] sL,    z#MobileBertForMultipleChoice.forwardr@)r_r`rarSrrrDrbrrrrr]rcr r rYr#rUsH      rUcrH) MobileBertForTokenClassificationcsbt||j|_t|dd|_|jdur|jn|j}t||_ t |j |j|_ | dSr=)rRrSrJrrrKrwrrvrxrrrlr)rrLrYr r#rSs   z)MobileBertForTokenClassification.__init__Nrzrr{rgrr|r3rrrr\c  Cs| dur| n|jj} |j|||||||| | d } | d} || } || } d}|dur) rGrrrxr)rrrJrrr)rWrzrr{rgrr|r3rrrrrr?rr7rr r r#r]s8  z(MobileBertForTokenClassification.forwardr@)r_r`rarSrrrDrbrrrrr]rcr r rYr#rV|sH     rV) r<rUrBrrNrIrVrrr rO)Krr0rC dataclassesrtypingrrrrDrZtorch.nnrrr Z activationsr Zmodeling_outputsr r rrrrrrZmodeling_utilsrZ pytorch_utilsrrutilsrrrZconfiguration_mobilebertrZ get_loggerr_r.rOModulerPrrtrerrrrrrrrrrrrrrrrrr rrrr<rArBrIrNrUrV__all__r r r r#s   (   N L:2 $  ?*  !jiP ]YMj E