o Zh`c@sdZddlmZddlmZmZmZddlZddlmZm Z ddl m Z ddl m Z dd lmZmZmZd d lmZd d lmZeeZeGdddeZeGdddeZeGdddeZeGddde ZGdddeZGdddeZGdddeZGdddeZ GdddeZ!ed d!Gd"d#d#eZ"ed$d!Gd%d&d&e Z#ed'd!Gd(d)d)e!Z$gd*Z%dS)+z5PyTorch DPR model for Open Domain Question Answering.) dataclass)OptionalTupleUnionN)Tensornn)BaseModelOutputWithPooling)PreTrainedModel) ModelOutputauto_docstringlogging) BertModel) DPRConfigc@PeZdZUdZejed<dZee ejdfed<dZ ee ejdfed<dS)DPRContextEncoderOutputac Class for outputs of [`DPRQuestionEncoder`]. Args: pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the context representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed contexts for nearest neighbors queries with questions embeddings. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. pooler_outputN. hidden_states attentions __name__ __module__ __qualname____doc__torch FloatTensor__annotations__rrrrrrS/var/www/auris/lib/python3.10/site-packages/transformers/models/dpr/modeling_dpr.pyr*  rc@r)DPRQuestionEncoderOutputac Class for outputs of [`DPRQuestionEncoder`]. Args: pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the question representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed questions for nearest neighbors queries with context embeddings. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. rN.rrrrrrr r"Fr!r"c@steZdZUdZejed<dZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS) DPRReaderOutputa Class for outputs of [`DPRQuestionEncoder`]. Args: start_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the start index of the span for each passage. end_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the end index of the span for each passage. relevance_logits (`torch.FloatTensor` of shape `(n_passages, )`): Outputs of the QA classifier of the DPRReader that corresponds to the scores of each passage to answer the question, compared to all the other passages. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. start_logitsN end_logitsrelevance_logits.rr) rrrrrrrr%rr&rrrrrrr r#bs  r#c@seZdZdZddZdS)DPRPreTrainedModelTcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsg)meanZstdNg?) isinstancerLinearweightdataZnormal_configZinitializer_rangeZbiasZzero_Z EmbeddingZ padding_idxZ LayerNormZfill_)selfmodulerrr _init_weightss      z DPRPreTrainedModel._init_weightsN)rrrZ_supports_sdpar0rrrr r's r'cseZdZdZdeffdd Z      ddedeed eed eed ed ed ede e e edfffddZ e defddZZS) DPREncoder bert_modelr-csdt|t|dd|_|jjjdkrtd|j|_|jdkr,t |jjj|j|_ | dS)NF)Zadd_pooling_layerrz!Encoder hidden_size can't be zero) super__init__rr2r- hidden_size ValueErrorprojection_dimrr* encode_proj post_initr.r- __class__rr r4s   zDPREncoder.__init__NF input_idsattention_masktoken_type_ids inputs_embedsoutput_attentionsoutput_hidden_states return_dictreturn.c Csv|j|||||||d}|d} | dddddf} |jdkr%|| } |s1| | f|ddSt| | |j|jdS)Nr=r>r?r@rArBrCrr)Zlast_hidden_staterrr)r2r7r8r rr) r.r=r>r?r@rArBrCoutputssequence_outputZ pooled_outputrrr forwards*    zDPREncoder.forwardcCs|jdkr |jjS|jjjS)Nr)r7r8Z out_featuresr2r-r5)r.rrr embeddings_sizes  zDPREncoder.embeddings_size)NNNFFF)rrrbase_model_prefixrr4rrboolrr rrHpropertyintrI __classcell__rrr;r r1s8 #r1csjeZdZdZdeffdd Z    ddeded eed ed ed ed e e e edfffddZ Z S)DPRSpanPredictorencoderr-csFt|t||_t|jjd|_t|jjd|_| dS)Nrr) r3r4r1rPrr*rI qa_outputs qa_classifierr9r:r;rr r4s   zDPRSpanPredictor.__init__NFr=r>r@rArBrCrD.cCs|dur|n|dd\}}|j||||||d} | d} || } | jddd\} } | d} | d} || dddddf}| ||} | ||} ||}|si| | |f| ddSt| | || j | j dS)Nr)r>r@rArBrCrr)dim)r$r%r&rr) sizerPrQsplitZsqueeze contiguousrRviewr#rr)r.r=r>r@rArBrCZ n_passagesZsequence_lengthrFrGZlogitsr$r%r&rrr rHs6$     zDPRSpanPredictor.forward)NFFF)rrrrJrr4rrrKrr#rrHrNrrr;r rOs, rOc@eZdZdZeZdZdZdS)DPRPretrainedContextEncoder An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. N ctx_encoderrrrrrZ config_classZload_tf_weightsrJrrrr rZ rZc@rY)DPRPretrainedQuestionEncoderr[Nquestion_encoderr]rrrr r_r^r_c@rY)DPRPretrainedReaderr[Nspan_predictorr]rrrr ra$r^razf The bare DPRContextEncoder transformer outputting pooler outputs as context representations. )Z custom_introceZdZdeffdd Ze       ddeedeedeedeed eed eed eed e e e ed fffddZ Z S)DPRContextEncoderr-c(t|||_t||_|dSN)r3r4r-r1r\r9r:r;rr r4:   zDPRContextEncoder.__init__Nr=r>r?r@rArBrCrD.c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur3|}n|dur@|dd}ntd|durK|jn|j} |durc|dur]tj|| dn||jj k}|durptj |tj | d}|j |||||||d} |s| ddSt | j| j| jd S) aS input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be formatted with [CLS] and [SEP] tokens as follows: (a) For sequence pairs (for a pair title+text for example): ``` tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] token_type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 ``` (b) For single sequences (for a question for example): ``` tokens: [CLS] the dog is hairy . [SEP] token_type_ids: 0 0 0 0 0 0 0 ``` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import DPRContextEncoder, DPRContextEncoderTokenizer >>> tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> model = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ```NDYou cannot specify both input_ids and inputs_embeds at the same timerS5You have to specify either input_ids or inputs_embedsdeviceZdtyperkrErrrr)r-rArBuse_return_dictr6rUrkrones pad_token_idzeroslongr\rrrr r.r=r>r?r@rArBrC input_shaperkrFrrr rHAsB1    zDPRContextEncoder.forwardNNNNNNN)rrrrr4r rrrKrrrrHrNrrr;r rd46 rdzh The bare DPRQuestionEncoder transformer outputting pooler outputs as question representations. crc)DPRQuestionEncoderr-crerf)r3r4r-r1r`r9r:r;rr r4rgzDPRQuestionEncoder.__init__Nr=r>r?r@rArBrCrD.c Cs(|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur9||||}n|durF|dd}ntd|durQ|jn|j} |duri|durctj || dn||jj k}|durvtj |tj | d}|j |||||||d} |s| ddSt| j| j| jd S) aj input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be formatted with [CLS] and [SEP] tokens as follows: (a) For sequence pairs (for a pair title+text for example): ``` tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] token_type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 ``` (b) For single sequences (for a question for example): ``` tokens: [CLS] the dog is hairy . [SEP] token_type_ids: 0 0 0 0 0 0 0 ``` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer >>> tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> model = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ``` NrhrSrirjrlrErrm)r-rArBrnr6%warn_if_padding_and_no_attention_maskrUrkrrorprqrrr`r"rrrrsrrr rHsD1     zDPRQuestionEncoder.forwardru)rrrrr4r rrrKrr"rrHrNrrr;r rwrvrwzE The bare DPRReader transformer outputting span predictions. cseZdZdeffdd Ze      ddeedeedeedeed eed eed e e e ed fffd dZ Z S) DPRReaderr-crerf)r3r4r-rOrbr9r:r;rr r4 rgzDPRReader.__init__Nr=r>r@rArBrCrD.c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur9||||}n|durF|dd}ntd|durQ|jn|j}|dur_tj ||d}|j ||||||dS)a input_ids (`Tuple[torch.LongTensor]` of shapes `(n_passages, sequence_length)`): Indices of input sequence tokens in the vocabulary. It has to be a sequence triplet with 1) the question and 2) the passages titles and 3) the passages texts To match pretraining, DPR `input_ids` sequence should be formatted with [CLS] and [SEP] with the format: `[CLS] [SEP] [SEP] ` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`DPRReaderTokenizer`]. See this class documentation for more details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(n_passages, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Examples: ```python >>> from transformers import DPRReader, DPRReaderTokenizer >>> tokenizer = DPRReaderTokenizer.from_pretrained("facebook/dpr-reader-single-nq-base") >>> model = DPRReader.from_pretrained("facebook/dpr-reader-single-nq-base") >>> encoded_inputs = tokenizer( ... questions=["What is love ?"], ... titles=["Haddaway"], ... texts=["'What Is Love' is a song recorded by the artist Haddaway"], ... return_tensors="pt", ... ) >>> outputs = model(**encoded_inputs) >>> start_logits = outputs.start_logits >>> end_logits = outputs.end_logits >>> relevance_logits = outputs.relevance_logits ``` NrhrSrirj)r@rArBrC) r-rArBrnr6rxrUrkrrorb) r.r=r>r@rArBrCrtrkrrr rHs.0  zDPRReader.forward)NNNNNN)rrrrr4r rrrKrr#rrHrNrrr;r rys0ry)rdrZr'r_rarwry)&r dataclassesrtypingrrrrrrZmodeling_outputsr Zmodeling_utilsr utilsr r r Zbert.modeling_bertrZconfiguration_dprrZ get_loggerrloggerrr"r#r'r1rOrZr_rardrwry__all__rrrr sJ       8>  de W