o ZŽh¹ã@s–dZddlZddlmZddlmZmZmZmZm Z ddl Z ddl m Z ddl m Z mZmZddlmZmZdd lmZdd lmZdd lmZdd lmZmZmZd dlmZe e ¡Z!dDdd„Z"dd„Z#Gdd„de j$ƒZ%Gdd„de j$ƒZ&Gdd„de j$ƒZ'Gdd„de j$ƒZ(Gdd„de j$ƒZ)Gdd„de j$ƒZ*Gdd „d e j$ƒZ+eGd!d"„d"eƒƒZ,eGd#d$„d$eƒƒZ-eGd%d&„d&eƒƒZ.eGd'd(„d(eƒƒZ/eGd)d*„d*eƒƒZ0eGd+d,„d,eƒƒZ1eGd-d.„d.eƒƒZ2eGd/d0„d0eƒƒZ3eGd1d2„d2e,ƒƒZ4ed3d4Gd5d6„d6e,eƒƒZ5ed7d4Gd8d9„d9e,ƒƒZ6eGd:d;„d;e,ƒƒZ7eGdd4Gd?d@„d@e,ƒƒZ9eGdAdB„dBe,ƒƒZ:gdC¢Z;dS)Ez PyTorch XLNet model. éN)Ú dataclass)ÚCallableÚListÚOptionalÚTupleÚUnion)Únn)ÚBCEWithLogitsLossÚCrossEntropyLossÚMSELossé)ÚACT2FNÚget_activation)ÚGenerationMixin)ÚPreTrainedModel)Úapply_chunking_to_forward)Ú ModelOutputÚauto_docstringÚloggingé)Ú XLNetConfigc Cs*i}t|dƒrUt|dƒr|jj|d<t|dƒr)d|vr)|jjj|d<|jjj|d<t|dƒrR|jdurRd |j›d |vrR|jj|d |j›d <|jj|d |j›d <|j}|  |j j|j d œ¡t |j ƒD]e\}}d |›d}|  |d|jjj|d|jjj|d|jj|d|jj|d|jj|d|jj|d|jj|d|jjj|d|jjj|d|jjj|d|jjj|d|jjj|d|jjji ¡qe|jrùg}g}g} g} |j D]}| |jj¡| |jj¡|  |jj¡|  |jj¡qÙn|jg}|jg}|jg} |jg} |  ||| | dœ¡|S)z A map of modules from TF to PyTorch. I use a map to keep the PyTorch model as identical to the original PyTorch model as possible. Ú transformerÚlm_losszmodel/lm_loss/biasÚsequence_summaryz%model/sequnece_summary/summary/kernelz#model/sequnece_summary/summary/biasÚ logits_projNzmodel/regression_z /logit/kernelz /logit/bias)z-model/transformer/word_embedding/lookup_tablez#model/transformer/mask_emb/mask_embzmodel/transformer/layer_ú/zrel_attn/LayerNorm/gammazrel_attn/LayerNorm/betazrel_attn/o/kernelzrel_attn/q/kernelzrel_attn/k/kernelzrel_attn/r/kernelzrel_attn/v/kernelzff/LayerNorm/gammazff/LayerNorm/betazff/layer_1/kernelzff/layer_1/biaszff/layer_2/kernelzff/layer_2/bias)zmodel/transformer/r_r_biaszmodel/transformer/r_w_biaszmodel/transformer/r_s_biaszmodel/transformer/seg_embed)ÚhasattrrÚbiasrÚsummaryÚweightZfinetuning_taskrrÚupdateÚword_embeddingÚmask_embÚ enumerateÚlayerÚrel_attnÚ layer_normÚoÚqÚkÚrÚvÚffÚlayer_1Úlayer_2Zuntie_rÚappendÚr_r_biasÚr_w_biasÚr_s_biasÚ seg_embed) ÚmodelÚconfigÚ tf_weightsÚ tf_to_pt_mapÚiÚbZ layer_strZr_r_listZr_w_listZr_s_listZseg_embed_list©r:úW/var/www/auris/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.pyÚbuild_tf_xlnet_to_pytorch_map'sx   ÿ þÿ      óÿ üüÿr<c Csœz ddl}ddl}Wn tyt d¡‚w|j |¡}i}|D]\}}t d|›d|›¡|j ||¡} | ||<q"t |||ƒ} |   ¡D]÷\}} t d|›¡||vr`t |›d¡qG||} d|vr~d |vstd |vstd |vr~t d ¡|  | ¡} t | t ƒrët| ƒ| jdksœJd t| ƒ›d| jd›dƒ‚t| ƒD]I\} } | | df}z| j|jks¾Jd| j›d|j›dƒ‚Wnty×}z |j| j|jf7_‚d}~wwt d|›d| ›¡t |¡| _q n=z| j| jksÿJd| j›d| j›dƒ‚Wnty}z |j| j| jf7_‚d}~wwt d|›¡t | ¡| _| |d¡| |dd¡| |dd¡qGt dd | ¡¡›¡|S)z&Load tf checkpoints in a pytorch modelrNzšLoading a TensorFlow models in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.zLoading TF weight z with shape z Importing z( not in tf pre-trained weights, skippingZkernelr,rZlogitZ TransposingzPointer length z and array length z mismatched.zPointer shape z and array shape zInitialize PyTorch weight z for layer z/Adamz/Adam_1z%Weights not copied to PyTorch model: z, )ÚnumpyZ tensorflowÚ ImportErrorÚloggerÚerrorÚtrainZlist_variablesÚinfoZ load_variabler<ÚitemsÚ transposeÚ isinstanceÚlistÚlenÚshaper#ÚAssertionErrorÚargsÚtorchZ from_numpyÚdataÚpopÚjoinÚkeys)r4r5Ztf_pathÚnpÚtfZ init_varsr6ÚnamerHÚarrayr7Zpointerr8Zp_iZarr_iÚer:r:r;Úload_tf_weights_in_xlnetzsv  ÿû        ÿ ÿ€þö ÿ€þ  rUcsneZdZ‡fdd„Zdd„Zeddd„ƒZeddd „ƒZ dd d „Zddd„Z ddd„Z ‡Z S)ÚXLNetRelativeAttentioncsntƒ ¡|j|jdkrtd|j›d|j›ƒ‚|j|_|j|_|j|_d|jd|_t t   |j|j|j¡¡|_ t t   |j|j|j¡¡|_ t t   |j|j|j¡¡|_ t t   |j|j|j¡¡|_t t   |j|j|j¡¡|_t t   |j|j¡¡|_t t   |j|j¡¡|_t t   |j|j¡¡|_t t   d|j|j¡¡|_tj|j|jd|_t |j¡|_dS)NrzThe hidden size (z6) is not a multiple of the number of attention heads (rçà?é©Zeps)ÚsuperÚ__init__Úd_modelZn_headÚ ValueErrorZd_headÚscalerÚ ParameterrKÚ FloatTensorr(r)r+r'r*r0r2r1r3Ú LayerNormÚlayer_norm_epsr&ÚDropoutÚdropout©Úselfr5©Ú __class__r:r;r[¾s,  ÿÿzXLNetRelativeAttention.__init__cCót‚©N©ÚNotImplementedError)rfZheadsr:r:r;Ú prune_headsÚóz"XLNetRelativeAttention.prune_headséÿÿÿÿc Cs||j}| |d|d|d|d¡}|dd…df}| |d|dd|d|d¡}t |dtj||jtjd¡}|S)zbnijr )ryNrzibnd,snd->ibnszijbs,ibns->bnijéÜÿz ijbn->bnijçꌠ9Y>)F©Údimzbnij,jbnd->ibndz bnij->ijbn)rKÚeinsumr1r0r{rHr2r3r^rrÚfloat16rÚ functionalÚsoftmaxrd)rfZq_headÚk_head_hÚv_head_hÚk_head_rÚseg_matÚ attn_maskÚ head_maskÚoutput_attentionsÚacZbdZefZ attn_scoreÚ attn_probÚattn_vecr:r:r;Ú rel_attn_coreùs(  z$XLNetRelativeAttention.rel_attn_coreTcCs4t d||j¡}| |¡}|r||}| |¡}|S)zPost-attention processing.z ibnd,hnd->ibh)rKr€r'rdr&)rfÚhrZresidualZattn_outÚoutputr:r:r;Úpost_attention-s   z%XLNetRelativeAttention.post_attentionc  Cs |dur¡|dur| ¡dkrtj||gdd} n|} t d| |j¡} t d| |j¡} t d||j¡}t d||j¡}|j|| | |||| | d}| rM|\}}|  ||¡}t d||j¡}|durt d||¡}|j|| | |||| | d}| ry|\}}t d||¡}n|j|| | |||| | d}| r”|\}}|  ||¡}| r ||f}nV|durµ| ¡dkrµtj||gdd} n|} t d||j¡}t d| |j¡} t d| |j¡} t d|  |jj ¡|j¡}|j|| | |||| | d}| rï|\}}|  ||¡}d}||f}| r||f}|S)Nrrr~z ibh,hnd->ibnd)r‡rˆr‰rŠzmbnd,mlb->lbndzlbnd,mlb->mbnd) rrKÚcatr€r)r+r*r(rŽr‘Útyperr)rfrÚgÚ attn_mask_hÚ attn_mask_gr*r‡ÚmemsÚtarget_mappingr‰rŠr’r„r…r†Zq_head_hZ attn_vec_hZ attn_prob_hÚoutput_hZq_head_gZ attn_vec_gZ attn_prob_gÚoutput_grŒrÚoutputsr:r:r;Úforward9sœ ø  ø ø  €ø   zXLNetRelativeAttention.forward©ro©NNNF)T) Ú__name__Ú __module__Ú __qualname__r[rmÚ staticmethodrzr{rŽr‘rœÚ __classcell__r:r:rgr;rV½s$    ÷ 4õrVcs$eZdZ‡fdd„Zdd„Z‡ZS)ÚXLNetFeedForwardcsxtƒ ¡tj|j|jd|_t |j|j¡|_ t |j|j¡|_ t  |j ¡|_ t |jtƒr6t|j|_dS|j|_dS)NrY)rZr[rrar\rbr&ÚLinearZd_innerr-r.rcrdrEZ ff_activationÚstrr Úactivation_functionrergr:r;r[Âs   zXLNetFeedForward.__init__cCsH|}| |¡}| |¡}| |¡}| |¡}| |¡}| ||¡}|Srj)r-r§rdr.r&)rfÚinprr:r:r;rœÍs     zXLNetFeedForward.forward)rŸr r¡r[rœr£r:r:rgr;r¤Ás  r¤cs6eZdZ‡fdd„Z    d dd„Zdd„Z‡ZS) Ú XLNetLayercs>tƒ ¡t|ƒ|_t|ƒ|_t |j¡|_|j |_ d|_ dS©Nr) rZr[rVr%r¤r,rrcrdÚchunk_size_feed_forwardÚ seq_len_dimrergr:r;r[Ùs    zXLNetLayer.__init__NFc Csv|j||||||||| | d } | dd…\}}|dur%t|j|j|j|ƒ}t|j|j|j|ƒ}||f| dd…} | S)N)r—r˜r‰rŠrX)r%rÚff_chunkr«r¬) rfr™ršr•r–r*r‡r—r˜r‰rŠr›r:r:r;rœás( ö ÿzXLNetLayer.forwardcCs| |¡}|Srj)r,)rfZoutput_xr:r:r;r­s zXLNetLayer.ff_chunkrž)rŸr r¡r[rœr­r£r:r:rgr;r©Øs  õ$r©csJeZdZdZdef‡fdd„ Z d dejdeejdejfd d „Z ‡Z S) ÚXLNetPoolerStartLogitszÍ Compute SQuAD start logits from sequence hidden states. Args: config ([`XLNetConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model. r5cstƒ ¡t |jd¡|_dSrª)rZr[rr¥Ú hidden_sizeÚdenserergr:r;r[s zXLNetPoolerStartLogits.__init__NÚ hidden_statesÚp_maskÚreturncCsT| |¡ d¡}|dur(|jtjkr|d|d|}|S|d|d|}|S)aì Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. p_mask (`torch.FloatTensor` of shape `(batch_size, seq_len)`, *optional*): Mask for tokens at invalid position, such as query and special symbols (PAD, SEP, CLS). 1.0 means token should be masked. Returns: `torch.FloatTensor`: The start logits for SQuAD. roNrr|r})r°ÚsqueezerrrKr)rfr±r²rxr:r:r;rœs þzXLNetPoolerStartLogits.forwardrj) rŸr r¡Ú__doc__rr[rKr`rrœr£r:r:rgr;r® sÿÿÿþr®c sbeZdZdZdef‡fdd„ Z   d dejdeejdeej d eejd ejf d d „Z ‡Z S)ÚXLNetPoolerEndLogitsz÷ Compute SQuAD end logits from sequence hidden states. Args: config ([`XLNetConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model and the `layer_norm_eps` to use. r5csRtƒ ¡t |jd|j¡|_t ¡|_tj|j|j d|_t |jd¡|_ dS)NrXrYr) rZr[rr¥r¯Údense_0ÚTanhÚ activationrarbÚdense_1rergr:r;r[<s  zXLNetPoolerEndLogits.__init__Nr±Ú start_statesÚstart_positionsr²r³cCsè|dus |dus Jdƒ‚|dur4|jdd…\}}|dd…ddf dd|¡}| d|¡}| d|d¡}| tj||gdd¡}| |¡}| |¡}| |¡  d¡}|durr|j tj krh|d|d|}|S|d|d|}|S) aë Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. start_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`, *optional*): The hidden states of the first tokens for the labeled span. start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): The position of the first token for the labeled span. p_mask (`torch.FloatTensor` of shape `(batch_size, seq_len)`, *optional*): Mask for tokens at invalid position, such as query and special symbols (PAD, SEP, CLS). 1.0 means token should be masked. One of `start_states` or `start_positions` should be not `None`. If both are set, `start_positions` overrides `start_states`. Returns: `torch.FloatTensor`: The end logits for SQuAD. Nú7One of start_states, start_positions should be not Noneéþÿÿÿror~rr|r}) rHÚexpandÚgatherr·rKr’r¹rarºr´rrr)rfr±r»r¼r²ÚslenÚhszrxr:r:r;rœCs$ÿ    þzXLNetPoolerEndLogits.forward©NNN© rŸr r¡rµrr[rKr`rÚ LongTensorrœr£r:r:rgr;r¶2s"  ûþýüûúr¶c sbeZdZdZdef‡fdd„ Z   d dejdeejdeej d eej d ejf d d „Z ‡Z S)ÚXLNetPoolerAnswerClasszè Compute SQuAD 2.0 answer class from classification and start tokens hidden states. Args: config ([`XLNetConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model. r5csBtƒ ¡t |jd|j¡|_t ¡|_tj|jddd|_dS)NrXrF©r) rZr[rr¥r¯r·r¸r¹rºrergr:r;r[s  zXLNetPoolerAnswerClass.__init__Nr±r»r¼Ú cls_indexr³cCsÜ|jd}|dus|dusJdƒ‚|dur,|dd…ddf dd|¡}| d|¡ d¡}|durH|dd…ddf dd|¡}| d|¡ d¡}n |dd…ddd…f}| tj||gdd¡}| |¡}| |¡ d¡}|S)a¸ Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. start_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`, *optional*): The hidden states of the first tokens for the labeled span. start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): The position of the first token for the labeled span. cls_index (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Position of the CLS token for each sentence in the batch. If `None`, takes the last token. One of `start_states` or `start_positions` should be not `None`. If both are set, `start_positions` overrides `start_states`. Returns: `torch.FloatTensor`: The SQuAD 2.0 answer class. roNr½r¾r~) rHr¿rÀr´r·rKr’r¹rº)rfr±r»r¼rÈrÂZcls_token_staterxr:r:r;rœ‡s ÿ zXLNetPoolerAnswerClass.forwardrÃrÄr:r:rgr;rÆxs" ûþýüûúrÆcsJeZdZdZdef‡fdd„ Z d dejdeej dejfd d „Z ‡Z S) ÚXLNetSequenceSummaryaÍ Compute a single vector summary of a sequence hidden states. Args: config ([`XLNetConfig`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. r5csþtƒ ¡t|ddƒ|_|jdkrt‚t ¡|_t|dƒr<|j r