o Zh@sPdZddlZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZmZmZmZmZmZmZdd lmZdd lmZmZdd lmZmZd d lm Z d dl!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'e(e)Z*Gdddej+Z,Gdddej+Z-Gdddej+Z.Gdddej+Z/Gdddej+Z0Gdddej+Z1Gdddej+Z2Gdddej+Z3Gdd d ej+Z4eGd!d"d"eZ5eGd#d$d$e5Z6eGd%d&d&e5Z7Gd'd(d(ej+Z8ed)d*Gd+d,d,e5Z9eGd-d.d.e5Z:eGd/d0d0e5Z;Gd1d2d2ej+Zgd7Z?dS)9zPyTorch I-BERT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)gelu))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) IBertConfig)IntGELU IntLayerNorm IntSoftmaxQuantActQuantEmbedding QuantLinearcs4eZdZdZfddZ d ddZdd ZZS) IBertEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. cs t|j|_d|_d|_d|_d|_d|_t|j |j |j |j|jd|_ t|j |j |j|jd|_|jdt|jdd d t|d d |_|j |_t|j|j |j|j|jd|_t|j|jd |_t|j|jd |_t|j |j|j|j|jd|_t|j|jd |_t !|j"|_#dS)N ) padding_idx weight_bit quant_mode)r%r& position_ids)rF) persistentposition_embedding_typeabsoluter&epsZ output_bitr& force_dequant)$super__init__r&Z embedding_bitZembedding_act_bitact_bit ln_input_bit ln_output_bitr vocab_size hidden_sizeZ pad_token_idword_embeddingsZtype_vocab_sizetoken_type_embeddingsZregister_buffertorcharangeZmax_position_embeddingsexpandgetattrr*r$position_embeddingsrembeddings_act1Zembeddings_act2rlayer_norm_epsr/ LayerNormoutput_activationrDropouthidden_dropout_probdropoutselfconfig __class__W/var/www/auris/lib/python3.10/site-packages/transformers/models/ibert/modeling_ibert.pyr15sP  zIBertEmbeddings.__init__NrcCs|dur|durt||j||j}n||}|dur"|}n|dd}|dur9tj|tj|j jd}|durE| |\}}nd}| |\}} |j |||| d\} } |j dkrp||\} } |j | | | | d\} } || | \} } || } || | \} } | | fS)Nr(dtypedeviceidentityZidentity_scaling_factorr+)"create_position_ids_from_input_idsr$torN&create_position_ids_from_inputs_embedssizer9zeroslongr'r7r8r>r*r=r@rDrA)rF input_idstoken_type_idsr' inputs_embedspast_key_values_length input_shapeZinputs_embeds_scaling_factorr8Z$token_type_embeddings_scaling_factor embeddingsZembeddings_scaling_factorr=Z"position_embeddings_scaling_factorrJrJrKforwardisF     zIBertEmbeddings.forwardcCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr(rrLr)rTr9r:r$rVrNZ unsqueezer;)rFrYr[Zsequence_lengthr'rJrJrKrSs  z6IBertEmbeddings.create_position_ids_from_inputs_embeds)NNNNr)__name__ __module__ __qualname____doc__r1r]rS __classcell__rJrJrHrKr0s  5 /rc4eZdZfddZddZ   d ddZZS) IBertSelfAttentionc svt|j|jdkrt|dstd|jd|jd|j|_d|_d|_d|_ |j|_t |j|j|_ |j|j |_ t |j|j d|j|j|jdd |_t |j|j d|j|j|jdd |_t |j|j d|j|j|jdd |_t|j |jd |_t|j |jd |_t|j |jd |_t|j |jd |_t|j|_t|d d |_|jd krtd t|j |j|jd|_dS)NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r r#Tbiasr%bias_bitr&Z per_channelr,r*r+zDI-BERT only supports 'absolute' for `config.position_embedding_type`r&r/)r0r1r6num_attention_headshasattr ValueErrorr&r%rhr2intattention_head_size all_head_sizerquerykeyvaluerquery_activationkey_activationvalue_activationrArrBZattention_probs_dropout_probrDr<r*rr/softmaxrErHrJrKr1sd      zIBertSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)Nr(rrr )rTrjrnviewpermute)rFxZ new_x_shaperJrJrKtranspose_for_scoress z'IBertSelfAttention.transpose_for_scoresNFcCs|||\}}|||\}} |||\} } |||\} } ||| \}}|| | \}}|| } ||}||}t| | dd}t |j }||}|j r]| ||}nd}|durg||}|||\}}||}|dur|||}t||}|dur||}nd}|dddd}|dd|jf}|j|}|||\}}|r||fn|f}|r||fn|f}||fS)Nr(rrwrr )rprqrrrsrtrur{r9matmulZ transposemathsqrtrnr&rvrDry contiguousrTrorxrA)rF hidden_stateshidden_states_scaling_factorattention_mask head_maskoutput_attentionsZmixed_query_layerZ mixed_query_layer_scaling_factorZmixed_key_layerZmixed_key_layer_scaling_factorZmixed_value_layerZ mixed_value_layer_scaling_factorZ query_layerZquery_layer_scaling_factorZ key_layerZkey_layer_scaling_factorZ value_layerZvalue_layer_scaling_factorZattention_scoresscaleZattention_scores_scaling_factorZattention_probsZattention_probs_scaling_factorZ context_layerZcontext_layer_scaling_factorZnew_context_layer_shapeoutputsZoutput_scaling_factorrJrJrKr]sV          zIBertSelfAttention.forwardNNF)r^r_r`r1r{r]rbrJrJrHrKrds : rdc$eZdZfddZddZZS)IBertSelfOutputc st|j|_d|_d|_d|_d|_d|_t|j |j d|j|j|jdd|_ t |j|jd|_ t |j |j|j|j|jd|_t |j|jd|_t|j|_dSNr r#r"Trfr,r-)r0r1r&r2r%rhr3r4rr6denser ln_input_actrr?r/r@rArrBrCrDrErHrJrKr194  zIBertSelfOutput.__init__cCX|||\}}||}|j||||d\}}|||\}}|||\}}||fSNrOrrDrr@rArFrrZ input_tensorZinput_tensor_scaling_factorrJrJrKr]V  zIBertSelfOutput.forwardr^r_r`r1r]rbrJrJrHrKr8 rcrc) IBertAttentioncs2t|j|_t||_t||_t|_dSN) r0r1r&rdrFroutputset pruned_headsrErHrJrKr1hs    zIBertAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrdim)lenrrFrjrnrrrprqrrrrrounion)rFheadsindexrJrJrK prune_headsos zIBertAttention.prune_headsNFc Cs^||||||\}}||d|d||\}} |f|dd} | f|dd} | | fS)Nrr)rFr) rFrrrrrZ self_outputsZself_outputs_scaling_factorattention_outputattention_output_scaling_factorrZoutputs_scaling_factorrJrJrKr]szIBertAttention.forwardr)r^r_r`r1rr]rbrJrJrHrKrgs rcr)IBertIntermediatec st|j|_d|_d|_d|_t|j|jd|j|j|jdd|_ |j dkr,t dt |j|j d|_t|j|jd|_dS) Nr r#Trfr z3I-BERT only supports 'gelu' for `config.hidden_act`rir,)r0r1r&r2r%rhrr6intermediate_sizerZ hidden_actrlrr/intermediate_act_fnrrArErHrJrKr1s$  zIBertIntermediate.__init__cCs8|||\}}|||\}}|||\}}||fSr)rrrA)rFrrrJrJrKr]szIBertIntermediate.forwardrrJrJrHrKrs rcr) IBertOutputc st|j|_d|_d|_d|_d|_d|_t|j |j d|j|j|jdd|_ t |j|jd|_ t|j |j|j|j|jd|_t |j|jd|_t|j|_dSr)r0r1r&r2r%rhr3r4rrr6rrrrr?r/r@rArrBrCrDrErHrJrKr1rzIBertOutput.__init__cCrrrrrJrJrKr]rzIBertOutput.forwardrrJrJrHrKrrrcs4eZdZfddZ   d ddZddZZS) IBertLayercsdt|j|_d|_d|_t||_t||_t ||_ t |j|jd|_ t |j|jd|_ dS)Nr rr,)r0r1r&r2Z seq_len_dimr attentionr intermediaterrrpre_intermediate_actpre_output_actrErHrJrKr1s    zIBertLayer.__init__NFc CsR|j|||||d\}}|d}|d} |dd} ||| \} } | f| } | S)N)rrr)rfeed_forward_chunk) rFrrrrrZself_attention_outputsZ%self_attention_outputs_scaling_factorrrr layer_outputlayer_output_scaling_factorrJrJrKr]s   zIBertLayer.forwardcCsL|||\}}|||\}}|||\}}|||||\}}||fSr)rrrr)rFrrZintermediate_outputZ"intermediate_output_scaling_factorrrrJrJrKrszIBertLayer.feed_forward_chunkr)r^r_r`r1r]rrbrJrJrHrKrs  rcs0eZdZfddZ     dddZZS) IBertEncodercs<t|_j|_tfddtjD|_dS)Ncsg|]}tqSrJ)r).0_rGrJrK (sz)IBertEncoder.__init__..) r0r1rGr&rZ ModuleListrangenum_hidden_layerslayerrErHrrKr1$s $zIBertEncoder.__init__NFTcCs|rdnd}|r dnd} d} d} t|jD]*\} } |r ||f}|dur(|| nd}| |||||}|d}|r?| |df} q|rG||f}|sWtdd|| || | fDSt|| || | dS)NrJrrcss|] }|dur|VqdSrrJ)rvrJrJrK Osz'IBertEncoder.forward..)last_hidden_statepast_key_valuesr attentionscross_attentions) enumeratertupler )rFrrrrroutput_hidden_states return_dictZall_hidden_statesZall_self_attentionsZall_cross_attentionsZnext_decoder_cacheiZ layer_moduleZlayer_head_maskZ layer_outputsrJrJrKr]*sJ    zIBertEncoder.forward)NNFFTrrJrJrHrKr#s  rcr) IBertPoolercs2t|j|_t|j|j|_t|_dSr) r0r1r&rLinearr6rZTanh activationrErHrJrKr1ds zIBertPooler.__init__cCs(|dddf}||}||}|SNr)rr)rFrZfirst_token_tensor pooled_outputrJrJrKr]js  zIBertPooler.forwardrrJrJrHrKrcs rc@s&eZdZeZdZddZdddZdS)IBertPreTrainedModelibertcCst|ttjfr"|jjjd|jjd|j dur |j j dSdSt|t tj frG|jjjd|jjd|j durE|jj|j  dSdSt|ttjfr^|j j |jjddSt|trk|j j dSdS)zInitialize the weightsg)meanZstdNg?) isinstancerrrweightdataZnormal_rGZinitializer_rangergZzero_rZ Embeddingr$rr@Zfill_ IBertLMHead)rFmodulerJrJrK _init_weightsxs     z"IBertPreTrainedModel._init_weightsNcCstd)Nz6`resize_token_embeddings` is not supported for I-BERT.)NotImplementedError)rFZnew_num_tokensrJrJrKresize_token_embeddingsz,IBertPreTrainedModel.resize_token_embeddingsr)r^r_r`rZ config_classZbase_model_prefixrrrJrJrJrKrss rcseZdZdZdfdd ZddZddZd d Ze dd e e j d e e j de e j de e j de e j de e j de e de e de e deeee j ffddZZS) IBertModela The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. TcsLt|||_|j|_t||_t||_|rt|nd|_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r0r1rGr&rr\rencoderrpooler post_init)rFrGadd_pooling_layerrHrJrKr1s    zIBertModel.__init__cC|jjSrr\r7rFrJrJrKget_input_embeddingsrzIBertModel.get_input_embeddingscCs ||j_dSrr)rFrrrJrJrKset_input_embeddingss zIBertModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rFZheads_to_prunerrrJrJrK _prune_headsszIBertModel._prune_headsNrWrrXr'rrYrrrreturnc  Cs|dur|n|jj}|dur|n|jj}| dur| n|jj} |dur*|dur*td|dur9||||} n|durF|dd} ntd| \} } |durU|jn|j} |duretj | | f| d}|durrtj | tj | d}| || }| ||jj}|j||||d\}}|j||||||| d}|d}|jdur||nd}| s||f|d dSt|||j|j|j|jd S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer(z5You have to specify either input_ids or inputs_embeds)rNrL)rWr'rXrY)rrrrrrr)rZ pooler_outputrrrr)rGrruse_return_dictrlZ%warn_if_padding_and_no_attention_maskrTrNr9ZonesrUrVZget_extended_attention_maskZ get_head_maskrr\rrr rrrr)rFrWrrXr'rrYrrrr[Z batch_sizeZ seq_lengthrNZextended_attention_maskZembedding_outputZembedding_output_scaling_factorZencoder_outputssequence_outputrrJrJrKr]s^      zIBertModel.forward)T) NNNNNNNNN)r^r_r`rar1rrrrrr9 LongTensor FloatTensorboolrr rr]rbrJrJrHrKrsJ    rcseZdZddgZfddZddZddZe dd ee j d ee j d ee j d ee j dee j dee j dee j dee dee dee de eee j ffddZZS)IBertForMaskedLMzlm_head.decoder.biaszlm_head.decoder.weightcs0t|t|dd|_t||_|dSNF)r)r0r1rrrlm_headrrErHrJrKr1 s   zIBertForMaskedLM.__init__cCrr)rdecoderrrJrJrKget_output_embeddingsrz&IBertForMaskedLM.get_output_embeddingscCs||j_|j|j_dSr)rrrg)rFZnew_embeddingsrJrJrKset_output_embeddingssz&IBertForMaskedLM.set_output_embeddingsNrWrrXr'rrYlabelsrrrrc  Cs| dur| n|jj} |j|||||||| | d } | d} || } d}|dur8t}|| d|jj|d}| sN| f| dd}|durL|f|S|St|| | j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` NrrXr'rrYrrrrr(rwlosslogitsrr) rGrrrrrxr5r rr)rFrWrrXr'rrYrrrrrrZprediction_scoresZmasked_lm_lossloss_fctrrJrJrKr]s6 zIBertForMaskedLM.forward NNNNNNNNNN)r^r_r`Z_tied_weights_keysr1rrrrr9rrrrr rr]rbrJrJrHrKrsN      rcs2eZdZdZfddZddZd dd ZZS) rz)I-BERT Head for masked language modeling.csdtt|j|j|_tj|j|jd|_t|j|j |_ t t |j |_|j|j _dS)N)r.)r0r1rrr6rr@r? layer_normr5r Parameterr9rUrgrErHrJrKr1Ss zIBertLMHead.__init__cKs*||}t|}||}||}|Sr)rr rr)rFfeatureskwargsrzrJrJrKr]\s   zIBertLMHead.forwardrNcCs,|jjjjdkr|j|j_dS|jj|_dS)Nmeta)rrgrNtyperrJrJrK _tie_weightsfszIBertLMHead._tie_weights)rN)r^r_r`rar1r]rrbrJrJrHrKrPs    rz I-BERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. )Z custom_introceZdZfddZe          ddeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e e ejffddZ ZS)IBertForSequenceClassificationcs8t||j|_t|dd|_t||_|dSr)r0r1 num_labelsrrIBertClassificationHead classifierrrErHrJrKr1vs   z'IBertForSequenceClassification.__init__NrWrrXr'rrYrrrrrc  Csh| dur| n|jj} |j|||||||| | d } | d} || } d}|dur|jjdurQ|jdkr7d|j_n|jdkrM|jtjksH|jtj krMd|j_nd|j_|jjdkrot }|jdkri|| | }n+|| |}n%|jjdkrt }|| d|j| d}n|jjdkrt}|| |}| s| f| d d}|dur|f|S|St|| | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrrZ regressionZsingle_label_classificationZmulti_label_classificationr(rwr)rGrrrZ problem_typerrMr9rVrmrsqueezerrxrrrrrFrWrrXr'rrYrrrrrrrrrrrJrJrKr]sT    "       z&IBertForSequenceClassification.forwardr)r^r_r`r1rrr9rrrrrrr]rbrJrJrHrKrosH      rcseZdZfddZe          ddeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e e ejffddZ ZS)IBertForMultipleChoicecs@t|t||_t|j|_t|j d|_ | dS)Nr) r0r1rrrrBrCrDrr6rrrErHrJrKr1s   zIBertForMultipleChoice.__init__NrWrXrrr'rrYrrrrc  Csn| dur| n|jj} |dur|jdn|jd} |dur%|d|dnd} |dur4|d|dnd} |durC|d|dnd}|durR|d|dnd}|dure|d|d|dnd}|j| | |||||| | d }|d}||}||}|d| }d}|durt}|||}| s|f|dd}|dur|f|S|St |||j |j dS)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nrr(r|)r'rXrrrYrrrrwr) rGrshaperxrTrrDrrrrr)rFrWrXrrr'rrYrrrZ num_choicesZflat_input_idsZflat_position_idsZflat_token_type_idsZflat_attention_maskZflat_inputs_embedsrrrZreshaped_logitsrrrrJrJrKr]sL,    zIBertForMultipleChoice.forwardr)r^r_r`r1rrr9rrrrrrr]rbrJrJrHrKrsH      rcr)IBertForTokenClassificationcsNt||j|_t|dd|_t|j|_t |j |j|_ | dSr) r0r1rrrrrBrCrDrr6rrrErHrJrKr1/s  z$IBertForTokenClassification.__init__NrWrrXr'rrYrrrrrc  Cs| dur| n|jj} |j|||||||| | d } | d} || } || } d}|dursZ  $    z/1"/:@xHQfB M