o Zhy@srdZddlZddlmZmZmZmZmZmZddl Z ddl Z ddl m Z ddl mZmZmZddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZmZmZm Z m!Z!m"Z"dd l#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+m,Z,ddl-m.Z.erddlm/Z/e,0e1Z2de3de3de j4fddZ5de3de3de j4fddZ6Gddde j7Z8Gddde j7Z9Gddde9Z:Gd d!d!e9Z;Gd"d#d#e j7ZGd'd(d(e j7Z?e+Gd)d*d*e$Z@e+Gd+d,d,e@ZAe+d-d.Gd/d0d0e@ZBe+d1d.Gd2d3d3e@ZCe+Gd4d5d5e@ZDe+Gd6d7d7e@ZEe+Gd8d9d9e@ZFgd:ZGdS);z PyTorch DistilBERT model adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) and in part from HuggingFace PyTorch version of Google AI Bert model (https://github.com/google-research/bert) N)DictListOptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)get_activation)PretrainedConfig)is_deepspeed_zero3_enabled)#_prepare_4d_attention_mask_for_sdpa)!flash_attn_supports_top_left_maskis_flash_attn_available)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indices"is_torch_greater_or_equal_than_2_2prune_linear_layer)auto_docstringlogging)DistilBertConfig)_flash_attention_forwardn_posdimoutcCstr8ddl}|jj|ddtjdkr&t|||dWddSWddS1s1wYdSt|||ddS)Nr)Z modifier_rankr#r$r%)r deepspeedzeroZGatheredParameterstorch distributedZget_rank_create_sinusoidal_embeddings)r#r$r%r'r,a/var/www/auris/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.pycreate_sinusoidal_embeddingsBs"r.c stfddt|D}d|_tt|dddddf|dddddf<tt|dddddf|dddddf<|dS)Ncs$g|]fddtDqS)c s(g|]}tdd|dqS)i')nppower).0j)r$posr,r- Ns(z<_create_sinusoidal_embeddings...)range)r2r$)r4r-r5Ns$z1_create_sinusoidal_embeddings..Frr/r ) r0arrayr6Z requires_gradr)Z FloatTensorsincosZdetach_)r#r$r%Z position_encr,r7r-r+Ms 44 r+csDeZdZdeffdd Zd dejdeejdejfdd ZZ S) Embeddingsconfigcsvttj|j|j|jd|_t|j|j|_ tj |jdd|_ t |j |_ |j dt|jddddS)N) padding_idx-q=eps position_ids)r F) persistent)super__init__r Embedding vocab_sizer$Z pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormDropoutdropoutZregister_bufferr)arangeexpandselfr< __class__r,r-rEVs  zEmbeddings.__init__N input_ids input_embedsreturncCs|dur ||}|d}t|dr|jddd|f}ntj|tj|jd}|d |}| |}||}| |}| |}|S)a Parameters: input_ids (torch.Tensor): torch.tensor(bs, max_seq_length) The token ids to embed. input_embeds (*optional*, torch.Tensor): The pre-computed word embeddings. Can only be passed if the input ids are `None`. Returns: torch.tensor(bs, max_seq_length, dim) The embedded tokens (plus position embeddings, no token_type embeddings) Nr rA)dtypedevicer) rHsizehasattrrAr)rNlongrXZ unsqueeze expand_asrJrKrM)rQrTrUZ seq_lengthrArJ embeddingsr,r,r-forwardas      zEmbeddings.forwardN) __name__ __module__ __qualname__rrEr)Tensorrr^ __classcell__r,r,rRr-r;Us* r;csxeZdZdeffdd ZdeefddZ  dd ej d ej d ej d ej d e ej de de ej dffddZ ZS)MultiHeadSelfAttentionr<cst||_|j|_|j|_tj|jd|_d|_ |j|jdkr0t d|jd|jdtj |j|jd|_ tj |j|jd|_ tj |j|jd|_tj |j|jd|_t|_|j|j|_dS)NpFrzself.n_heads: z must divide self.dim:  evenlyZ in_featuresZ out_features)rDrEr<n_headsr$rrLattention_dropoutrM is_causal ValueErrorLinearq_link_linv_linout_linset pruned_headsattention_head_sizerPrRr,r-rEs zMultiHeadSelfAttention.__init__headscCst|dkrdSt||j|j|j\}}t|j||_t|j||_t|j||_t|j |dd|_ |jt||_|j|j|_ |j ||_dS)Nrr r7) lenrrjrurtrrorprqrrr$union)rQrvindexr,r,r- prune_headss z"MultiHeadSelfAttention.prune_headsNFquerykeyvaluemask head_maskoutput_attentionsrV.csB|\}}|d} jjdd| f} dtjdtjffdd } dtjdtjffdd } | |} | |}| |}| t } t | | dd }|d k |  |}||tt|jj}tjj|d d }|}|d ur||}t ||}| |}|}|r||fS|fS) Parameters: query: torch.tensor(bs, seq_length, dim) key: torch.tensor(bs, seq_length, dim) value: torch.tensor(bs, seq_length, dim) mask: torch.tensor(bs, seq_length) Returns: weights: torch.tensor(bs, n_heads, seq_length, seq_length) Attention weights context: torch.tensor(bs, seq_length, dim) Contextualized layer. Optional: only if `output_attentions=True` r xrVc|djddSseparate headsrBr r/viewrj transposerbs dim_per_headrQr,r-shapez-MultiHeadSelfAttention.forward..shapec |dddjSz group headsr r/rBr contiguousrrjrrr,r-unshape z/MultiHeadSelfAttention.forward..unshaper/r rrBr7N)rYr$rjr)rcrorprqmathsqrtmatmulrrr\Z masked_fillZtensorZfinforWminr functionalZsoftmaxrMrr)rQr{r|r}r~rrq_lengthr$Zk_lengthZ mask_reshprrqkvZscoresweightscontextr,rr-r^s2      zMultiHeadSelfAttention.forwardNF)r`rarbrrErintrzr)rcrboolrr^rdr,r,rRr-res( recsdeZdZdZfddZ  ddejdejdejd ejd eejd ed e ejd ffddZ Z S)DistilBertFlashAttention2aS DistilBert flash attention module. This module inherits from `MultiHeadSelfAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cstj|i|t|_dSr_)rDrEr_flash_attn_uses_top_left_mask)rQargskwargsrRr,r-rEs z"DistilBertFlashAttention2.__init__NFr{r|r}r~rrrV.c s.|\}}jjdtjdtjffdd } | |} | |} | |} jr8j j nd} | j tj krqt rIt}ntj drTj j}njjj }td|d| |} | |} | |} t| | | ||| jjd }||j}|}|r||fS|fS) rrrVcs|djS)rrB)rrjrZ batch_sizerrQr,r-reshapesz2DistilBertFlashAttention2.forward..reshape_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)rMZuse_top_left_maskrl)rYr$rjr)rcrorprqtrainingr<rkrWZfloat32Zis_autocast_enabledZget_autocast_gpu_dtyperZrweightlogger warning_oncetor"rrlrrr)rQr{r|r}r~rrrr$rZ query_statesZ key_statesZ value_statesZ attn_dropoutZ target_dtypeZ attn_weightsZattn_weights_reshaped attn_outputr,rr-r^sH          z!DistilBertFlashAttention2.forwardr) r`rarb__doc__rEr)rcrrrr^rdr,r,rRr-rs(  rcsjeZdZdeffdd Z  ddejdejdejd ejd eejd ed e ejd fffdd Z Z S)DistilBertSdpaAttentionr<cs"tj|d|j|_t |_dS)Nr<)rDrErk dropout_probrrequire_contiguous_qkvrPrRr,r-rELs z DistilBertSdpaAttention.__init__NFr{r|r}r~rrrV.cs|s|durtdt||||||S|\}}jjdtjdtjffdd }dtjdtjffdd } | |} | |} | |} j ro| j jd kro|duro| } | } | } tjjj| | | |jr}jnd d d } | | } | } | fS) rNaDistilBertSdpaAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rrVcrrrrrr,r-rxrz.DistilBertSdpaAttention.forward..shapecrrrrrr,r-r|rz0DistilBertSdpaAttention.forward..unshapecudarF) attn_maskZ dropout_prl)rrrDr^rYr$rjr)rcrorprqrrXtyperrrZscaled_dot_product_attentionrrrr)rQr{r|r}r~rr_rrrrrrrRrr-r^QsD   zDistilBertSdpaAttention.forwardr r`rarbrrEr)rcrrrr^rdr,r,rRr-rKs&  rcsNeZdZdeffdd ZdejdejfddZdejdejfdd ZZ S) FFNr<cs`ttj|jd|_|j|_d|_tj|j|j d|_ tj|j |jd|_ t |j |_ dS)Nrfr ri)rDrErrLrMchunk_size_feed_forward seq_len_dimrnr$Z hidden_dimlin1lin2r activationrPrRr,r-rEs z FFN.__init__inputrVcCst|j|j|j|Sr_)rff_chunkrr)rQrr,r,r-r^sz FFN.forwardcCs,||}||}||}||}|Sr_)rrrrM)rQrrr,r,r-rs    z FFN.ff_chunk) r`rarbrrEr)rcr^rrdr,r,rRr-rs r)eagerflash_attention_2sdpac s`eZdZdeffdd Z   ddejdeejdeejd ed e ejd ff d d Z Z S)TransformerBlockr<csvt|j|jdkrtd|jd|jdt|j||_tj |jdd|_ t ||_ tj |jdd|_ dS)Nrzconfig.n_heads z must divide config.dim rhr>)Znormalized_shaper@)rDrEr$rjrmDISTILBERT_ATTENTION_CLASSES_attn_implementation attentionrrK sa_layer_normrffnoutput_layer_normrPrRr,r-rEs  zTransformerBlock.__init__NFrrrrrV.c Cs|j||||||d}|r|\}}nt|tur"tdt|d|d}|||}||}|||}|f}|rC|f|}|S)ae Parameters: x: torch.tensor(bs, seq_length, dim) attn_mask: torch.tensor(bs, seq_length) Returns: sa_weights: torch.tensor(bs, n_heads, seq_length, seq_length) The attention weights ffn_output: torch.tensor(bs, seq_length, dim) The output of the transformer block contextualization. )r{r|r}r~rrz$sa_output must be a tuple but it is z typer)rrtuple TypeErrorrrr) rQrrrrZ sa_outputZ sa_weightsZ ffn_outputoutputr,r,r-r^s(    zTransformerBlock.forward)NNFrr,r,rRr-rs  rcsxeZdZdeffdd Z     ddejdeejdeejd ed ed eed e e e ejd fffddZ Z S) Transformerr<cs<tj|_tfddtjD|_d|_dS)Ncsg|]}tqSr,)r)r2rrr,r-r5sz(Transformer.__init__..F)rDrEZn_layersrZ ModuleListr6layergradient_checkpointingrPrRrr-rEs   zTransformer.__init__NFrrrroutput_hidden_states return_dictrV.cCs |rdnd}|r dnd}|} t|jD]U\} } |r|| f}|jr1|jr1|| j| ||| |} n | | ||| |} | d} |rYt| dkrOtdt| | d} || f}qt| dkrhtdt| q|rp|| f}|s~td d | ||fDSt | ||d S) a Parameters: x: torch.tensor(bs, seq_length, dim) Input sequence embedded. attn_mask: torch.tensor(bs, seq_length) Attention mask on the sequence. Returns: hidden_state: torch.tensor(bs, seq_length, dim) Sequence of hidden states in the last (top) layer all_hidden_states: Tuple[torch.tensor(bs, seq_length, dim)] Tuple of length n_layers with the hidden states from each layer. Optional: only if output_hidden_states=True all_attentions: Tuple[torch.tensor(bs, n_heads, seq_length, seq_length)] Tuple of length n_layers with the attention weights from each layer Optional: only if output_attentions=True r,NrBr/z7The length of the layer_outputs should be 2, but it is rr z7The length of the layer_outputs should be 1, but it is css|] }|dur|VqdSr_r,)r2rr,r,r- 8sz&Transformer.forward..)Zlast_hidden_state hidden_states attentions) enumeraterrrZ_gradient_checkpointing_func__call__rwrmrr)rQrrrrrrZall_hidden_statesZall_attentions hidden_stateiZ layer_moduleZ layer_outputsrr,r,r-r^sH        zTransformer.forward)NNFFN)r`rarbrrEr)rcrrrrrr^rdr,r,rRr-rs, rc@s4eZdZeZdZdZdZdZdZ de j fddZ dS)DistilBertPreTrainedModelN distilbertTmodulecCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSt|tro|jjrqt|jj|jj|jjdSdSdS)zInitialize the weights.r)meanZstdNg?) isinstancerrnrdataZnormal_r<Zinitializer_rangeZbiasZzero_rFr=rKZfill_r;sinusoidal_pos_embdsr.rIr$rJ)rQrr,r,r- _init_weightsHs$      z'DistilBertPreTrainedModel._init_weights) r`rarbr!Z config_classZload_tf_weightsZbase_model_prefixZsupports_gradient_checkpointingZ_supports_flash_attn_2Z_supports_sdparModulerr,r,r,r-r?srcseZdZdeffdd ZdejfddZdefdd Z dejfd d Z d ejfd dZ de ee e effddZe       ddeejdeejdeejdeejdeedeedeedeeeejdfffddZZS)DistilBertModelr<csDt|t||_t||_|jdk|_|jdk|_| dS)Nrr) rDrEr;r]r transformerr_use_flash_attention_2 _use_sdpa post_initrPrRr,r-rE_s      zDistilBertModel.__init__rVcC|jjSz1 Returns the position embeddings )r]rJrQr,r,r-get_position_embeddingsjsz'DistilBertModel.get_position_embeddingsnew_num_position_embeddingscCs||jj}|dkr dStd|d||j_|jjj}t |jj|jj |j_|jj r>t |jj|jj |jjdn2t &|dkrUt||jjjd| <n t|d||jj_Wdn1skwY|jj|jdS) Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`. Arguments: new_num_position_embeddings (`int`): The number of new position embedding matrix. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. rNz(Setting `config.max_position_embeddings=z`...r&)r<rIrinfor]rJrclonerrFr$rr.r)Zno_grad ParameterrrX)rQrZnum_position_embeds_diffZold_position_embeddings_weightr,r,r-resize_position_embeddingsps,     z*DistilBertModel.resize_position_embeddingscCrr_r]rHrr,r,r-get_input_embeddingssz$DistilBertModel.get_input_embeddingsnew_embeddingscCs ||j_dSr_rrQrr,r,r-set_input_embeddingss z$DistilBertModel.set_input_embeddingsheads_to_prunecCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrrz)rQrrrvr,r,r- _prune_headsszDistilBertModel._prune_headsNrTattention_maskr inputs_embedsrrr.c Cs:|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur9||||}n|durF|dd}ntd|durQ|jn|j} |du} |||jj }| ||} |j rv|dursd|vrs|nd}n|durt j || d}|jr| r|st|| j|dd}|j| |||||d S)  input_ids (`torch.LongTensor` of shape `(batch_size, num_choices)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. NzDYou cannot specify both input_ids and inputs_embeds at the same timerBz5You have to specify either input_ids or inputs_embedsr)rXr )Ztgt_len)rrrrrr)r<rruse_return_dictrmZ%warn_if_padding_and_no_attention_maskrYrXZ get_head_maskZnum_hidden_layersr]rr)ZonesrrrWr) rQrTrrrrrrZ input_shaperXZhead_mask_is_noner]r,r,r-r^s@    zDistilBertModel.forward)NNNNNNN)r`rarbrrErrFrrrrrrrrrrr)rcrrrrr^rdr,r,rRr-r]s@ * rzI DistilBert Model with a `masked language modeling` head on top. )Z custom_introcseZdZdgZdeffdd ZdejfddZde fd d Z dej fd d Z d ej fddZ e        ddeejdeejdeejdeejdeejdeedeedeedeeeejdfffddZZS)DistilBertForMaskedLMzvocab_projector.weightr<csnt|t|j|_t||_t|j|j|_ tj |jdd|_ t|j|j |_ |t|_dS)Nr>r?)rDrEr rrrrrnr$vocab_transformrKvocab_layer_normrGvocab_projectorrr mlm_loss_fctrPrRr,r-rEs   zDistilBertForMaskedLM.__init__rVcC |jSrrrrr,r,r-r z-DistilBertForMaskedLM.get_position_embeddingsrcC|j|dSrNrrrQrr,r,r-r  z0DistilBertForMaskedLM.resize_position_embeddingscCs|jSr_rrr,r,r-get_output_embeddingssz+DistilBertForMaskedLM.get_output_embeddingsrcCs ||_dSr_rrr,r,r-set_output_embeddingss z+DistilBertForMaskedLM.set_output_embeddingsNrTrrrlabelsrrr.c  Cs|dur|n|jj}|j|||||||d} | d} || } || } || } || } d} |durD|| d| d|d} |sZ| f| dd} | durX| f| S| St | | | j | j dS)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. NrTrrrrrrrrBr losslogitsrr) r<rrrrrrrrrYrrr)rQrTrrrrrrrZ dlbrt_outputrZprediction_logitsZmlm_lossrr,r,r-r^s6     zDistilBertForMaskedLM.forwardNNNNNNNN)r`rarbZ_tied_weights_keysrrErrFrrrrrrrrr)rc LongTensorrrrrr^rdr,r,rRr-rsF  rz DistilBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZdeffdd ZdejfddZdefdd Z e dd e e j d e e j d e e j de e j de e jde ede ede edeeee j dfffddZZS)#DistilBertForSequenceClassificationr<csbt||j|_||_t||_t|j|j|_ t|j|j|_ t |j |_ |dSr_)rDrE num_labelsr<rrrrnr$pre_classifier classifierrLseq_classif_dropoutrMrrPrRr,r-rEcs   z,DistilBertForSequenceClassification.__init__rVcCrrrrr,r,r-rprz;DistilBertForSequenceClassification.get_position_embeddingsrcCr r r r r,r,r-rvr z>DistilBertForSequenceClassification.resize_position_embeddingsNrTrrrrrrr.c  Cs|dur|n|jj}|j|||||||d} | d} | dddf} || } t| } || } || } d} |dur|jjdurg|j dkrMd|j_n|j dkrc|j t j ks^|j t j krcd|j_nd|j_|jjdkrt}|j dkr|| |} n+|| |} n%|jjdkrt}|| d|j |d} n|jjdkrt}|| |} |s| f| dd}| dur| f|S|St| | | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrr Z regressionZsingle_label_classificationZmulti_label_classificationrBr)r<rrrrReLUrMrZ problem_typerrWr)r[rr squeezer rr rrr)rQrTrrrrrrrdistilbert_outputr pooled_outputrrloss_fctrr,r,r-r^sX        "       z+DistilBertForSequenceClassification.forwardr)r`rarbrrErrFrrrrrr)rcrrrrrr^rdr,r,rRr-r\s@   rcseZdZdeffdd ZdejfddZdefdd Z e dd e e j d e e j d e e j de e j de e j de e j de ede ede edeeee j dfffddZZS)DistilBertForQuestionAnsweringr<cs\t|t||_t|j|j|_|jdkr!t d|jt |j |_ | dS)Nr/z)config.num_labels should be 2, but it is )rDrErrrrnr$r qa_outputsrmrLZ qa_dropoutrMrrPrRr,r-rEs    z'DistilBertForQuestionAnswering.__init__rVcCrrrrr,r,r-rrz6DistilBertForQuestionAnswering.get_position_embeddingsrcCr r r r r,r,r-rr z9DistilBertForQuestionAnswering.resize_position_embeddingsNrTrrrstart_positions end_positionsrrr.c  CsP| dur| n|jj} |j||||||| d} | d} || } || } | jddd\} }| d} |d}d}|dur|durt| dkrR|d}t| dkr_|d}| d}| d|}| d|}t j |d}|| |}|||}||d}| s| |f| dd}|dur|f|S|St || || j| jd S) rNrrr rBr7)Z ignore_indexr/)r start_logits end_logitsrr)r<rrrMr$splitrrrwrYclamprr rrr)rQrTrrrr%r&rrrr rrr'r(Z total_lossZ ignored_indexr"Z start_lossZend_lossrr,r,r-r^sN           z&DistilBertForQuestionAnswering.forward) NNNNNNNNN)r`rarbrrErrFrrrrrr)rcrrrrr^rdr,r,rRr-r#sF    r#cr) DistilBertForTokenClassificationr<csJt||j|_t||_t|j|_t|j |j|_ | dSr_) rDrErrrrrLrMrnZ hidden_sizerrrPrRr,r-rE;s   z)DistilBertForTokenClassification.__init__rVcCrrrrr,r,r-rFrz8DistilBertForTokenClassification.get_position_embeddingsrcCr r r r r,r,r-rLr z;DistilBertForTokenClassification.resize_position_embeddingsNrTrrrrrrr.c  Cs|dur|n|jj}|j|||||||d} | d} || } || } d} |dur:t} | | d|j|d} |sP| f| dd}| durN| f|S|St| | | j | j dS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. NrrrrrrrrBr r) r<rrrMrr rrrrr)rQrTrrrrrrroutputsZsequence_outputrrr"rr,r,r-r^Zs4  z(DistilBertForTokenClassification.forwardr)r`rarbrrErrFrrrrrr)rcrrrrrr^rdr,r,rRr-r+9@   r+cr)DistilBertForMultipleChoicer<csRt|t||_t|j|j|_t|jd|_t |j |_ | dS)Nr ) rDrErrrrnr$rrrLrrMrrPrRr,r-rEs   z$DistilBertForMultipleChoice.__init__rVcCrrrrr,r,r-rrz3DistilBertForMultipleChoice.get_position_embeddingsrcCr )a Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`. Arguments: new_num_position_embeddings (`int`) The number of new position embeddings. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. Nr r r,r,r-rr z6DistilBertForMultipleChoice.resize_position_embeddingsNrTrrrrrrr.c  CsT|dur|n|jj}|dur|jdn|jd} |dur%|d|dnd}|dur4|d|dnd}|durG|d|d|dnd}|j|||||||d} | d} | dddf} || } t| } | | } | | } | d| }d}|durt }|||}|s|f| dd}|dur|f|S|St ||| j | jdS)a^ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Examples: ```python >>> from transformers import AutoTokenizer, DistilBertForMultipleChoice >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased") >>> model = DistilBertForMultipleChoice.from_pretrained("distilbert-base-cased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> choice0 = "It is eaten with a fork and a knife." >>> choice1 = "It is eaten while held in the hand." >>> labels = torch.tensor(0).unsqueeze(0) # choice0 is correct (according to Wikipedia ;)), batch size 1 >>> encoding = tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors="pt", padding=True) >>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels) # batch size is 1 >>> # the linear classifier still needs to be trained >>> loss = outputs.loss >>> logits = outputs.logits ```Nr rBr,rr)r<rrrrYrrrrrMrr rrr)rQrTrrrrrrrZ num_choicesr-rr!rZreshaped_logitsrr"rr,r,r-r^sJ2       z#DistilBertForMultipleChoice.forwardr)r`rarbrrErrFrrrrrr)rcrrrrrr^rdr,r,rRr-r/r.r/)rr/r#rr+rr)Hrrtypingrrrrrrnumpyr0r)rZtorch.nnr r r Z activationsr Zconfiguration_utilsrZintegrations.deepspeedrZmodeling_attn_mask_utilsrZmodeling_flash_attention_utilsrrZmodeling_outputsrrrrrrZmodeling_utilsrZ pytorch_utilsrrrrutilsrrZconfiguration_distilbertr!r"Z get_loggerr`rrrcr.r+rr;rerrrrrrrrrrr#r+r/__all__r,r,r,r-sl            .h`P:NjimR