o Zh0@s|dZddlZddlZddlmZddlmZmZmZm Z m Z ddl Z ddl Z ddl mZddlmZmZmZddlmZmZdd lmZdd lmZmZmZmZmZmZdd lm Z dd l!m"Z"m#Z#m$Z$dd l%m&Z&m'Z'm(Z(ddl)m*Z*e(+e,Z-ddZ.d=ddZ/eGddde&Z0Gdddej1Z2Gdddej1Z3Gdddej1Z4Gdddej1Z5Gdddej1Z6Gd d!d!ej1Z7Gd"d#d#ej1Z8e'Gd$d%d%e Z9eGd&d'd'e&Z:e'Gd(d)d)e9Z;Gd*d+d+ej1Ze'd3d-Gd4d5d5e9Z?e'Gd6d7d7e9Z@e'Gd8d9d9e9ZAe'Gd:d;d;e9ZBgd<ZCdS)>z PyTorch XLM model. N) dataclass)CallableDictOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)geluget_activation)GenerationMixin)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging) XLMConfigc stfddt|D}d|_tt|dddddf|dddddf<tt|dddddf|dddddf<|dS)Ncs$g|]fddtDqS)c s(g|]}tdd|dqS)i')nppower).0j)dimposS/var/www/auris/lib/python3.10/site-packages/transformers/models/xlm/modeling_xlm.py 1s(z;create_sinusoidal_embeddings...)range)r"r$)r%r'r(1s$z0create_sinusoidal_embeddings..Frrr) r arrayr)Z requires_gradtorch FloatTensorsincosZdetach_)Zn_posr$outZ position_encr&r*r'create_sinusoidal_embeddings0s 44 r1cCstj|tj|jd}|dur|}n||ksJ||dddfk}|d}|rD|ddddf||d|ddddfk}n|}|||fksPJ|dus_||||fks_J||fS)zH Generate hidden states mask, and optionally an attention mask. dtypedeviceNrrF)r,arangelongr4maxitemsizerepeat)slenlengthscausal padding_maskalenmaskbs attn_maskr&r&r' get_masks8s 0rCc@s~eZdZUdZdZeejed<dZ eejed<dZ eej ed<dZ eejed<dZ eej ed<dZeejed<dS) XLMSquadHeadOutputa Base class for outputs of question answering models using a [`~modeling_utils.XLMSQuADHead`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned if both `start_positions` and `end_positions` are provided): Classification loss as the sum of start token, end token (and is_impossible if provided) classification losses. start_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top config.start_n_top start token possibilities (beam-search). start_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top config.start_n_top start token possibilities (beam-search). end_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). end_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). cls_logits (`torch.FloatTensor` of shape `(batch_size,)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the `is_impossible` label of the answers. Nlossstart_top_log_probsstart_top_indexend_top_log_probs end_top_index cls_logits)__name__ __module__ __qualname____doc__rErr,r-__annotations__rFrG LongTensorrHrIrJr&r&r&r'rDQs rDcsJeZdZdZdeffdd Z d dejdeejdejfd d Z Z S) XLMPoolerStartLogitsz Compute SQuAD start logits from sequence hidden states. Args: config ([`XLMConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model. configcstt|jd|_dSNr)super__init__rLinear hidden_sizedenseselfrR __class__r&r'rUys zXLMPoolerStartLogits.__init__N hidden_statesp_maskreturncCsT||d}|dur(|jtjkr|d|d|}|S|d|d|}|S)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. p_mask (`torch.FloatTensor` of shape `(batch_size, seq_len)`, *optional*): Mask for tokens at invalid position, such as query and special symbols (PAD, SEP, CLS). 1.0 means token should be masked. Returns: `torch.FloatTensor`: The start logits for SQuAD. Nrꌠ9Y>)F)rXsqueezer3r,float16)rZr]r^xr&r&r'forward}s zXLMPoolerStartLogits.forwardN) rKrLrMrNrrUr,r-rrf __classcell__r&r&r[r'rQpsrQc sbeZdZdZdeffdd Z   d dejdeejdeej d eejd ejf d d Z Z S)XLMPoolerEndLogitsz Compute SQuAD end logits from sequence hidden states. Args: config ([`XLMConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model and the `layer_norm_eps` to use. rRcsRtt|jd|j|_t|_tj|j|j d|_t|jd|_ dS)NrZepsr) rTrUrrVrWdense_0Tanh activation LayerNormlayer_norm_epsdense_1rYr[r&r'rUs  zXLMPoolerEndLogits.__init__Nr] start_statesstart_positionsr^r_cCs|dus |dus Jd|dur4|jdd\}}|ddddfdd|}|d|}|d|d}|tj||gdd}||}||}|| d}|durr|j tj krh|d|d|}|S|d|d|}|S) a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. start_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`, *optional*): The hidden states of the first tokens for the labeled span. start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): The position of the first token for the labeled span. p_mask (`torch.FloatTensor` of shape `(batch_size, seq_len)`, *optional*): Mask for tokens at invalid position, such as query and special symbols (PAD, SEP, CLS). 1.0 means token should be masked. One of `start_states` or `start_positions` should be not `None`. If both are set, `start_positions` overrides `start_states`. Returns: `torch.FloatTensor`: The end logits for SQuAD. N7One of start_states, start_positions should be not Noner`r*rrarb) shapeexpandgatherrkr,catrmrnrprcr3rd)rZr]rqrrr^r;hszrer&r&r'rfs$    zXLMPoolerEndLogits.forwardNNN rKrLrMrNrrUr,r-rrPrfrhr&r&r[r'ris"  ric sbeZdZdZdeffdd Z   d dejdeejdeej d eej d ejf d d Z Z S)XLMPoolerAnswerClassz Compute SQuAD 2.0 answer class from classification and start tokens hidden states. Args: config ([`XLMConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model. rRcsBtt|jd|j|_t|_tj|jddd|_dS)NrrFbias) rTrUrrVrWrkrlrmrprYr[r&r'rUs  zXLMPoolerAnswerClass.__init__Nr]rqrr cls_indexr_cCs|jd}|dus|dusJd|dur,|ddddfdd|}|d|d}|durH|ddddfdd|}|d|d}n |dddddf}|tj||gdd}||}||d}|S)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. start_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`, *optional*): The hidden states of the first tokens for the labeled span. start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): The position of the first token for the labeled span. cls_index (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Position of the CLS token for each sentence in the batch. If `None`, takes the last token. One of `start_states` or `start_positions` should be not `None`. If both are set, `start_positions` overrides `start_states`. Returns: `torch.FloatTensor`: The SQuAD 2.0 answer class. r`Nrsrtr*) rurvrwrcrkr,rxrmrp)rZr]rqrrrryZcls_token_staterer&r&r'rfs  zXLMPoolerAnswerClass.forwardrzr{r&r&r[r'r|s" r|cseZdZdZdeffdd Ze      ddejde ej d e ej d e ej d e ej d e ejd e de e eejffddZZS) XLMSQuADHeadz A SQuAD head inspired by XLNet. Args: config ([`XLMConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model and the `layer_norm_eps` to use. rRcs<t|j|_|j|_t||_t||_t||_ dSrg) rTrU start_n_top end_n_toprQ start_logitsri end_logitsr| answer_classrYr[r&r'rU&s   zXLMSQuADHead.__init__NFr]rr end_positionsr is_impossibler^ return_dictr_cCs|j||d}|durk|durk||||fD]} | dur&| dkr&| dq|j|||d} t} | ||} | | |} | | d}|dura|dura|j|||d}t}|||}||d7}|rht|d S|fS| \}}}tj j |dd }t j ||jdd \}}|ddd|}t |d |}|dd|dd}|d|}|dur|dnd}|j|||d } tj j | dd }t j ||jdd \}}|d|j|j}|d|j|j}t d ||}|j|||d}|s|||||fSt|||||dS)a hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): Final hidden states of the model on the sequence tokens. start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Positions of the first token for the labeled span. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Positions of the last token for the labeled span. cls_index (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Position of the CLS token for each sentence in the batch. If `None`, takes the last token. is_impossible (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Whether the question has a possible answer in the paragraph or not. p_mask (`torch.FloatTensor` of shape `(batch_size, seq_len)`, *optional*): Mask for tokens at invalid position, such as query and special symbols (PAD, SEP, CLS). 1.0 means token should be masked. )r^Nrr`)rrr^r)rrrg?)rEr*rt)rqr^z blh,bl->bh)rqr)rFrGrHrIrJ)rr$Zsqueeze_rr rrr rDr9 functionalsoftmaxr,Ztopkr unsqueezervrw expand_asrviewZeinsum)rZr]rrrrrr^rrrerloss_fct start_lossend_loss total_lossrJZ loss_fct_clsZcls_lossZbszr;ryZstart_log_probsrFrGZstart_top_index_exprqZhidden_states_expandedZ end_log_probsrHrIr&r&r'rf/s^         zXLMSQuADHead.forward)NNNNNF)rKrLrMrNrrUrr,r-rrPboolrrDrrfrhr&r&r[r'rs6   rcsJeZdZdZdeffdd Z d dejdeej dejfd d Z Z S) XLMSequenceSummarya Compute a single vector summary of a sequence hidden states. Args: config ([`XLMConfig`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. rRcstt|dd|_|jdkrtt|_t|dr<|j r