fTh`SrSSKrSSKJr SSKJrJrJr SSKrSSK rSSKJ r SSK J r J r Jr SSKJrJr SS KJrJr SS KJr SS KJr SS KJrJrJr S SKJr \R>"\ 5r!\"SS\55r"\"SS\55r#\"SS\55r$\"SS\55r%\"SS\55r&\"SS\55r'\"SS\55r(\"SS\55r)\"SS \55r*\"S!S"\55r+"S#S$\ RX5r-"S%S&\ RX5r."S'S(\ RX5r/"S)S*\ RX5r0"S+S,\ RX5r1"S-S.\ RX5r2"S/S0\ RX5r3"S1S2\ RX5r4"S3S4\ RX5r5"S5S6\ RX5r6"S7S8\ RX5r7"S9S:\ RX5r8\"S;S<\55r9\"S=S>9"S?S@\955r:SAr;"SBSC\ RX5r<\"SDS>9"SESF\955r=\"SGS>9"SHSI\955r>\"SJS>9"SKSL\955r?\"SMS>9"SNSO\955r@\"SPS>9"SQSR\955rA\"SSS>9"STSU\955rB\"SVSW\955rC\"SXSY\955rD/SZQrEg)[zPyTorch LUKE model.N) dataclass)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNgelu)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel)apply_chunking_to_forward) ModelOutputauto_docstringlogging) LukeConfigct\rSrSr%SrSr\\R\ S'Sr \\ \RS4\ S'Sr g)BaseLukeModelOutputWithPooling%a Base class for outputs of the LUKE model. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. entity_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, entity_length, hidden_size)`): Sequence of entity hidden-states at the output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length + entity_length, sequence_length + entity_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nentity_last_hidden_state.entity_hidden_states __name__ __module__ __qualname____firstlineno____doc__rrtorch FloatTensor__annotations__rr__static_attributes__r^/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/luke/modeling_luke.pyrr%@2=Ahu'8'89@DH(5):):C)?#@AHr'rct\rSrSr%SrSr\\R\ S'Sr \\ \RS4\ S'Sr g)BaseLukeModelOutputDa Base class for model's outputs, with potential hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. entity_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, entity_length, hidden_size)`): Sequence of entity hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr.rrrrr'r(r+r+Dr)r'r+ct\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\R\ S'Sr\\\R\ S 'Sr\\\RS 4\ S 'Sr\\\RS 4\ S 'S rg)LukeMaskedLMOutputca Base class for model's outputs, with potential hidden states and attentions. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): The sum of masked language modeling (MLM) loss and entity prediction loss. mlm_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Masked language modeling (MLM) loss. mep_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Masked entity prediction (MEP) loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). entity_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the entity prediction head (scores for each entity vocabulary token before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossmlm_lossmep_losslogits entity_logits hidden_states.r attentionsr)rrr r!r"r0rr#r$r%r1r2r3r4r5rrr6r&rr'r(r.r.cs>)-D(5$$ %,,0Hhu(()0,0Hhu(()0*.FHU&& '.15M8E--.58Ju00#567>r'r.c\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \RS4\ S'Sr \\ \RS4\ S'Sr\\ \RS4\ S 'S rg) EntityClassificationOutputa- Outputs of entity classification models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr0r3.r5rr6rrrr r!r"r0rr#r$r%r3r5rrr6r&rr'r(r8r8,)-D(5$$ %,*.FHU&& '.=AM8E%"3"3S"89:ADH(5):):C)?#@AH:>Ju00#567>r'r8c\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \RS4\ S'Sr \\ \RS4\ S'Sr\\ \RS4\ S 'S rg) EntityPairClassificationOutputa2 Outputs of entity pair classification models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr0r3.r5rr6rr:rr'r(r=r=r;r'r=c\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \RS4\ S'Sr \\ \RS4\ S'Sr\\ \RS4\ S 'S rg) EntitySpanClassificationOutputaA Outputs of entity span classification models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, entity_length, config.num_labels)`): Classification scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr0r3.r5rr6rr:rr'r(r@r@r;r'r@c\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \RS4\ S'Sr \\ \RS4\ S'Sr\\ \RS4\ S 'S rg) LukeSequenceClassifierOutputa Outputs of sentence classification models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr0r3.r5rr6rr:rr'r(rCrC2)-D(5$$ %,*.FHU&& '.=AM8E%"3"3S"89:ADH(5):):C)?#@AH:>Ju00#567>r'rCc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \RS4\ S'Sr \\ \RS4\ S'Sr\\ \RS4\ S 'S rg) LukeTokenClassifierOutputi a Base class for outputs of token classification models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) : Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_labels)`): Classification scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr0r3.r5rr6rr:rr'r(rGrG rEr'rGc(\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\ \RS4\ S'Sr\\ \RS4\ S 'Sr\\ \RS4\ S 'S rg) LukeQuestionAnsweringModelOutputi/a! Outputs of question answering models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Span-start scores (before SoftMax). end_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Span-end scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr0 start_logits end_logits.r5rr6r)rrr r!r"r0rr#r$r%rJrKr5rrr6r&rr'r(rIrI/s6)-D(5$$ %,04L(5,,-4.2J**+2=AM8E%"3"3S"89:ADH(5):):C)?#@AH:>Ju00#567>r'rIc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \RS4\ S'Sr \\ \RS4\ S'Sr\\ \RS4\ S 'S rg) LukeMultipleChoiceModelOutputiTa Outputs of multiple choice models. Args: loss (`torch.FloatTensor` of shape *(1,)*, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, num_choices)`): *num_choices* is the second dimension of the input tensors. (see *input_ids* above). Classification scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr0r3.r5rr6rr:rr'r(rMrMTs6)-D(5$$ %,*.FHU&& '.=AM8E%"3"3S"89:ADH(5):):C)?#@AH:>Ju00#567>r'rMcD^\rSrSrSrU4SjrSSjrSrSrU=r $)LukeEmbeddingsixzN Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. c>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR Ul[R"URUR UR"S9Ul g)N padding_idxeps)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutrRselfconfig __class__s r(rVLukeEmbeddings.__init__}s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= "..#%<<  * *F,>,>DL\L\$  r'cUcCUb/[XR5RUR5nOUR U5nUbUR 5nOUR 5SSnUc8[ R"U[ RURRS9nUcURU5nURU5nURU5nXF-U-nURU5nURU5nU$)Ndtypedevice)"create_position_ids_from_input_idsrRtorn&create_position_ids_from_inputs_embedssizer#zeroslong position_idsr[r]r_r`rd) rf input_idstoken_type_idsru inputs_embeds input_shaper]r_ embeddingss r(forwardLukeEmbeddings.forwards  $A)M]M]^aabkbrbrs #JJ=Y  #..*K',,.s3K  !"[[EJJtO`O`OgOghN   00;M"66|D $ : :> J"8;PP ^^J/ \\*- r'cUR5SSnUSn[R"URS-X0R-S-[RUR S9nUR S5RU5$)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nrkrrlr)rrr#arangerRrtrn unsqueezeexpand)rfrxrysequence_lengthrus r(rq5LukeEmbeddings.create_position_ids_from_inputs_embedss~$((*3B/ %a.||   q /4D4D"Dq"HPUPZPZcpcwcw %%a(// <[TU]5 Xl[R"UR UR SS9UlUR UR:wa/[R"UR URSS9Ul [R"URUR5Ul [R"URUR5Ul[R"URUR S9Ul[R""UR$5Ulg)NrrQFbiasrS)rUrVrgrrWentity_vocab_sizeentity_emb_sizeentity_embeddingsrYLinearentity_embedding_denser\r]r^r_r`rarbrcrdres r(rVLukeEntityEmbeddings.__init__s  !#f.F.FH^H^lm!n  ! !V%7%7 7*,))F4J4JFL^L^ej*kD '#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= r' entity_idsrurwc<Uc[R"U5nURU5nURRURR :waUR U5nURURSS95nUS:gRU5RS5nXV-n[R"USS9nXVRSS9RSS9- nURU5nXE-U-nURU5nURU5nU$)Nr)minrkdimgHz>)r# zeros_likerrgrrYrr]clamptype_asrsumr_r`rd) rfrrurwrr]position_embedding_maskr_rzs r(r{LukeEntityEmbeddings.forwards  !"--j9N 22:> ;; & &$++*A*A A $ ; ;">?R"S"]"]^`"a1K#ii(;D14O4OTV4O4W4]4]bf4]4gg $ : :> J&z >$6: $$&&!!1!12 r'rc>^\rSrSrU4SjrSrSSjrSrU=r$)LukeSelfAttentionicD>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eURUl[ URUR- 5UlURUR-UlURUl [R"URUR5Ul [R"URUR5Ul [R"URUR5UlUR(a[R"URUR5Ul[R"URUR5Ul[R"URUR5Ul[R$"UR&5Ulg)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .)rUrVrYnum_attention_headshasattr ValueErrorintattention_head_size all_head_sizeuse_entity_aware_attentionrrquerykeyvalue w2e_query e2w_query e2e_queryrbattention_probs_dropout_probrdres r(rVLukeSelfAttention.__init__s     : : :a ?PVXhHiHi"6#5#5"67334A7  $*#=#= #&v'9'9FS"S%**,CD $1!ZiZ2B$C!  '*. '*79:q8H*I ' 02M_G12MNGr') rrrdrrrrrrrrNNF) rrr r!rVrr{r&rrs@r(rrs%G0% PPr'rcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)LukeSelfOutputi^c(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNrS) rUrVrrrYdenser`rarbrcrdres r(rVLukeSelfOutput.__init___s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r'r5 input_tensorreturncpURU5nURU5nURX-5nU$rrrdr`rfr5rs r(r{LukeSelfOutput.forwarde5 =1  ]3 }'CD r'r`rrd rrr r!rVr#Tensorr{r&rrs@r(rr^6> U\\RWR^R^r'rc>^\rSrSrU4SjrSrSSjrSrU=r$) LukeAttentionilc>[TU]5 [U5Ul[ U5Ul[ 5Ulgr)rUrVrrfroutputset pruned_headsres r(rVLukeAttention.__init__ms0 %f- $V, Er'c[S5eNz4LUKE does not support the pruning of attention headsNotImplementedError)rfheadss r( prune_headsLukeAttention.prune_headss!"XYYr'c<URS5nURUUUUU5nUcUSnUn O.[R"USSSS9n[R"X/SS9n UR X5n U SS2SU2SS24n UcSn OU SS2US2SS24n X4USS-n U $)Nrrrr)rrrfr#rr)rfrrrrrr self_outputsconcat_self_outputsrattention_outputword_attention_outputentity_attention_outputrs r(r{LukeAttention.forwardvs'++A. yy       '".q/ #5 "'))L!,[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) rUrVrrrYintermediate_sizer isinstance hidden_actstrr intermediate_act_fnres r(rVLukeIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r'r5rcJURU5nURU5nU$rrrrfr5s r(r{LukeIntermediate.forwards& =1 00? r'rrrs@r(r r s(9U\\ellr'r cz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) LukeOutputic(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) rUrVrrr rYrr`rarbrcrdres r(rVLukeOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= r'r5rrcpURU5nURU5nURX-5nU$rrrs r(r{LukeOutput.forwardrr'rrrs@r(rrrr'rc>^\rSrSrU4SjrSSjrSrSrU=r$) LukeLayeric>[TU]5 URUlSUl[ U5Ul[ U5Ul[U5Ul gNr) rUrVchunk_size_feed_forward seq_len_dimr attentionr  intermediaterrres r(rVLukeLayer.__init__sI '-'E'E$&v.,V4 ( r'cBURS5nURUUUUUS9nUcUSnO[R"USSSS9nUSSn [ UR UR URU5n U SS2SU2SS24n UcSn OU SS2US2SS24n X4U -n U $)Nr)rrrr)rrr#r#rrfeed_forward_chunkr!r") rfrrrrrrself_attention_outputsconcat_attention_outputr layer_outputword_layer_outputentity_layer_outputs r(r{LukeLayer.forwards'++A. !%   / "0"  '&[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snf)NF) rUrVrgr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)rfrg_rhs r(rVLukeEncoder.__init__sR  ]]uVE]E]?^#_?^!If$5?^#_` &+#$`sA&c U(aSOSnU(aSOSn U(aSOSn [UR5HupU(a X4-nX4-n UbXKOSn UR(a2UR(a!UR U R UUUU U5nO U "UUUU U5nUSnUbUSnU(dMXS4-n M U(a X4-nX4-n U(d[ SUUU UU 455$[UUU UU S9$)Nrrrrc30# UH nUcMUv M g7frr.0vs r( &LukeEncoder.forward..(" A )last_hidden_stater5r6rr) enumerater7r8training_gradient_checkpointing_func__call__tupler+)rfrrrrroutput_hidden_states return_dictall_word_hidden_statesall_entity_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r(r{LukeEncoder.forwardsS(<)=24 $5b4(4OA#)?BW)W&+CF]+](.7.CilO**t}} $ A A ))&("#% ! !-&("#% ! "/q!1 #/'4Q'7$  &91=MS%S "'?BY'Y $ '*'(,    #00*%9!9   r')rgr8r7)NNFFTrrr r!rVr{r&rrs@r(r2r2s%,"D D r'r2cb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $) LukePooleri=c>[TU]5 [R"URUR5Ul[R "5Ulgr)rUrVrrrYrTanh activationres r(rVLukePooler.__init__>s9 YYv1163E3EF '')r'r5rc\USS2S4nURU5nURU5nU$)Nr)rrY)rfr5first_token_tensor pooled_outputs r(r{LukePooler.forwardCs6+1a40 #56  6 r')rYrrrs@r(rVrV=s($ U\\ellr'rVc.^\rSrSrU4SjrSrSrU=r$)EntityPredictionHeadTransformiLcp>[TU]5 [R"URUR 5Ul[UR[5(a[URUl OURUl [R"UR URS9Ul gr)rUrVrrrYrrrrrr transform_act_fnr`rares r(rV&EntityPredictionHeadTransform.__init__Ms~ YYv1163I3IJ f'' - -$*6+<+<$=D !$*$5$5D !f&<&<&BWBWXr'clURU5nURU5nURU5nU$r)rrbr`rs r(r{%EntityPredictionHeadTransform.forwardVs4 =1 --m< }5 r')r`rrbrTrs@r(r`r`LsYr'r`c.^\rSrSrU4SjrSrSrU=r$)EntityPredictionHeadi]c>[TU]5 Xl[U5Ul[ R "URURSS9Ul [ R"[R"UR55Ul g)NFr)rUrVrgr` transformrrrrdecoder Parameterr#rsrres r(rVEntityPredictionHead.__init__^s_  6v>yy!7!79Q9QX]^ LLV-E-E!FG r'cdURU5nURU5UR-nU$r)rirjrrs r(r{EntityPredictionHead.forwardes-}5  ]3dii? r')rrgrjrirTrs@r(rgrg]sHr'rgcJ\rSrSr\rSrSrSS/rS\ R4Sjr Sr g ) LukePreTrainedModelillukeTrrmodulec[U[R5(akURRR SUR RS9 URb%URRR5 gg[U[R5(aURS:Xa%URRR5 O8URRR SUR RS9 URb2URRURR5 gg[U[R5(aJURRR5 URRRS5 gg)zInitialize the weightsg)meanstdNr?)rrrweightdatanormal_rginitializer_rangerzero_rW embedding_dimrRr`fill_)rfrrs r( _init_weights!LukePreTrainedModel._init_weightsss1 fbii ( ( MM   & &CT[[5R5R & S{{&   &&('  - -##q( ""((* ""**9V9V*W!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).r'rN) rrr r!r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modulesrModuler~r&rr'r(rprpls/L&*#(*@A*BII*r'rpzt The bare LUKE model transformer outputting raw hidden-states for both word tokens and entities without any ) custom_introc ^\rSrSrSS\S\4U4SjjjrSrSrSr Sr S r \ SS \ \RS \ \R S \ \RS \ \RS\ \RS\ \R S\ \RS\ \RS\ \R S\ \R S\ \S\ \S\ \S\\\44Sjj5rS\RS\ \R4SjrSrU=r$) LukeModelirgadd_pooling_layerc>[TU]U5 Xl[U5Ul[ U5Ul[U5UlU(a [U5OSUl UR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rUrVrgrOrzrrr2encoderrVpooler post_init)rfrgrrhs r(rVLukeModel.__init__sX   (0!5f!="6* ,=j(4  r'c.URR$rrzr[rfs r(get_input_embeddingsLukeModel.get_input_embeddingss...r'c$XRlgrrrfrs r(set_input_embeddingsLukeModel.set_input_embeddingss */'r'c.URR$rrrs r(get_entity_embeddingsLukeModel.get_entity_embeddingss%%777r'c$XRlgrrrs r(set_entity_embeddingsLukeModel.set_entity_embeddingss380r'c[S5err)rfheads_to_prunes r( _prune_headsLukeModel._prune_headsrr'rvrrwrurentity_attention_maskentity_token_type_idsentity_position_idsrrxrrJrKrc U bU OURRn U bU OURRn U bU OURRn UbU b [ S5eUb"UR X5 UR 5nO"U bU R 5SSnO [ S5eUunnUb URO U RnUc[R"UU4US9nUc$[R"U[RUS9nUbTUR S5nUc[R"UU4US9nUc&[R"UU4[RUS9nURXRR5n URUUUU S9nURX&5nUcSnOUR!XXU5nUR#UUUU U U U S 9nUS nUR$bUR%U5OSnU (d UU4USS-$['UUUR(UR*UR,UR.S 9$) u entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. Examples: ```python >>> from transformers import AutoTokenizer, LukeModel >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-base") >>> model = LukeModel.from_pretrained("studio-ousia/luke-base") # Compute the contextualized entity representation corresponding to the entity mention "Beyoncé" >>> text = "Beyoncé lives in Los Angeles." >>> entity_spans = [(0, 7)] # character-based entity span corresponding to "Beyoncé" >>> encoding = tokenizer(text, entity_spans=entity_spans, add_prefix_space=True, return_tensors="pt") >>> outputs = model(**encoding) >>> word_last_hidden_state = outputs.last_hidden_state >>> entity_last_hidden_state = outputs.entity_last_hidden_state # Input Wikipedia entities to obtain enriched contextualized representations of word tokens >>> text = "Beyoncé lives in Los Angeles." >>> entities = [ ... "Beyoncé", ... "Los Angeles", ... ] # Wikipedia entity titles corresponding to the entity mentions "Beyoncé" and "Los Angeles" >>> entity_spans = [ ... (0, 7), ... (17, 28), ... ] # character-based entity spans corresponding to "Beyoncé" and "Los Angeles" >>> encoding = tokenizer( ... text, entities=entities, entity_spans=entity_spans, add_prefix_space=True, return_tensors="pt" ... ) >>> outputs = model(**encoding) >>> word_last_hidden_state = outputs.last_hidden_state >>> entity_last_hidden_state = outputs.entity_last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerkz5You have to specify either input_ids or inputs_embeds)rnrlr)rvrurwrx)rrrrJrKr)rD pooler_outputr5r6rr)rgrrJuse_return_dictr%warn_if_padding_and_no_attention_maskrrrnr#onesrsrt get_head_maskr6rzget_extended_attention_maskrrrrr5r6rr)rfrvrrwrurrrrrrxrrJrKry batch_size seq_lengthrnentity_seq_lengthword_embedding_outputextended_attention_maskentity_embedding_outputencoder_outputssequence_outputr]s r(r{LukeModel.forwardszR2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T  !"ZZZ(@PN  !"[[EJJvVN  ! * 2 $,(- J@Q3R[a(b%$,(- ZAR4S[`[e[ent(u%&&y++2O2OP !%%)' !0! #'"B"B>"i  &* #&*&<&1559,0/3&*Y E,,-Y !!2!23Y !!1!12 Y u//0 Y U--. Y  ((9(9:Y  ((8(89Y &e&6&67Y E--.Y   1 12Y $D>Y 'tnY d^Y  u44 5Y Y v'#(#3#3'LTUZUeUeLf''r'rcURU5R5n[R"USS9R U5U-nUR 5U-$)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)nerr#cumsumrrt)rvrRmaskincremental_indicess r(rorofsP << $ ( ( *D <<!4<[TU]5 [R"URUR5Ul[R "URURS9Ul[R"URUR5Ul [R"[R"UR55UlURURlgr)rUrVrrrYrr`ra layer_normrXrjrkr#rsrres r(rVLukeLMHead.__init__zs YYv1163E3EF ,,v'9'9v?T?TUyy!3!3V5F5FG LLV->->!?@  II r'c URU5n[U5nURU5nURU5nU$r)rr rrj)rffeatureskwargsrs r(r{LukeLMHead.forwards; JJx  G OOA  LLOr'cURRRRS:XaURURlgURRUlg)Nmeta)rjrrntypers r( _tie_weightsLukeLMHead._tie_weightssC <<   # # ( (F 2 $ DLL  ))DIr')rrjrr) rrr r!r"rVr{rr&rrs@r(rrws4&**r'rz The LUKE model with a language modeling head and entity prediction head on top for masked language modeling and masked entity prediction. c$^\rSrSr/SQrU4SjrU4SjrSrSr\ SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \ RS\ \ RS\ \ RS\ \ RS\ \S\ \S\ \S\\\44 Sjj5rSrU=r$)LukeForMaskedLMi)zlm_head.decoder.weightzlm_head.decoder.biasz!entity_predictions.decoder.weightc>[TU]U5 [U5Ul[ U5Ul[ U5Ul[R"5Ul UR5 gr) rUrVrrqrlm_headrgentity_predictionsrr loss_fnrres r(rVLukeForMaskedLM.__init__sQ  f% !&) "6v">**,  r'c>[TU]5 URURRUR R R 5 gr)rU tie_weights_tie_or_clone_weightsrrjrqr)rfrhs r(rLukeForMaskedLM.tie_weightss:  ""4#:#:#B#BDIID_D_DqDqrr'c.URR$rrrjrs r(get_output_embeddings%LukeForMaskedLM.get_output_embeddingss||###r'c$XRlgrr)rfnew_embeddingss r(set_output_embeddings%LukeForMaskedLM.set_output_embeddingss - r'rvrrwrurrrrlabels entity_labelsrrxrrJrKrcTUbUOURRnURUUUUUUUUU U U USS9 nSnSnURUR5nU beU R UR 5n URURSURR5U RS55nUcUnSnSnURbnURUR5nU bPURURSURR5U RS55nUcUnOUU-nU(d8[SUUUUUURURUR 455$[#UUUUUURURUR S9$)a{ entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` entity_labels (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` NT rvrrwrurrrrrrxrrJrKrkc30# UH nUcMUv M g7frrr=s r(r@*LukeForMaskedLM.forward..s"  A rC)r0r1r2r3r4r5rr6)rgrrqrrDrprnrrrXrrrrIr5rr6r.)rfrvrrwrurrrrrrrrxrrJrKrr0r1r3r2r4s r(r{LukeForMaskedLM.forwardsb&1%-1481559,0/3&*!q E,,-q !!2!23q !!1!12 q u//0 q U--. q  ((8(89q  ((8(89q &e&6&67q ))*q   0 01q E--.q   1 12q $D>q 'tnq d^!q " u(( )#q q r'rz The LUKE model with a classification head on top (a linear layer on top of the hidden state of the first entity token) for entity classification tasks, such as Open Entity. c"^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\RS \\RS \\RS\\ S\\ S\\ S\ \ \ 44Sjj5rSrU=r$)LukeForEntityClassificationi+c0>[TU]U5 [U5UlURUl[ R "UR5Ul[ R"URUR5Ul UR5 gr rUrVrrq num_labelsrrbrcrdrrY classifierrres r(rV$LukeForEntityClassification.__init__2si  f%  ++zz&"<"<= ))F$6$68I8IJ r'rvrrwrurrrrrrxrrrJrKrcUbUOURRnURUUUUUUUUU U U U SS9 nURSS2SSS24nUR U5nUR U5nSnU bU R UR5n U RS:Xa![RRUU 5nOM[RRURS5U RS5RU55nU(d5[SUUUR UR"UR$455$['UUUR UR"UR$S9$) u entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)` or `(batch_size, num_labels)`, *optional*): Labels for computing the classification loss. If the shape is `(batch_size,)`, the cross entropy loss is used for the single-label classification. In this case, labels should contain the indices that should be in `[0, ..., config.num_labels - 1]`. If the shape is `(batch_size, num_labels)`, the binary cross entropy loss is used for the multi-label classification. In this case, labels should only contain `[0, 1]`, where 0 and 1 indicate false and true, respectively. Examples: ```python >>> from transformers import AutoTokenizer, LukeForEntityClassification >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-open-entity") >>> model = LukeForEntityClassification.from_pretrained("studio-ousia/luke-large-finetuned-open-entity") >>> text = "Beyoncé lives in Los Angeles." >>> entity_spans = [(0, 7)] # character-based entity span corresponding to "Beyoncé" >>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: person ```NTrrrrkc30# UH nUcMUv M g7frrr=s r(r@6LukeForEntityClassification.forward..pAprCr0r3r5rr6)rgrrqrrdrrprnndimrr cross_entropy binary_cross_entropy_with_logitsrrrIr5rr6r8rfrvrrwrurrrrrrxrrrJrKrfeature_vectorr3r0s r(r{#LukeForEntityClassification.forward>sm|&1%1559.2,0/3&*k E,,-k !!2!23k !!1!12 k u//0 k U--. k  ((9(9:k  ((8(89k &e&6&67k E--.k   1 12k **+k $D>k 'tnk d^k u00 1!k k r'rz The LUKE model with a classification head on top (a linear layer on top of the hidden states of the two entity tokens) for entity pair classification tasks, such as TACRED. c"^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\RS \\RS \\RS\\ S\\ S\\ S\ \ \ 44Sjj5rSrU=r$)LukeForEntityPairClassificationic8>[TU]U5 [U5UlURUl[ R "UR5Ul[ R"URS-URS5Ul UR5 g)NrFrres r(rV(LukeForEntityPairClassification.__init__sp  f%  ++zz&"<"<= ))F$6$6$:F>> from transformers import AutoTokenizer, LukeForEntityPairClassification >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-tacred") >>> model = LukeForEntityPairClassification.from_pretrained("studio-ousia/luke-large-finetuned-tacred") >>> text = "Beyoncé lives in Los Angeles." >>> entity_spans = [ ... (0, 7), ... (17, 28), ... ] # character-based entity spans corresponding to "Beyoncé" and "Los Angeles" >>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: per:cities_of_residence ```NTrrrrrkc30# UH nUcMUv M g7frrr=s r(r@:LukeForEntityPairClassification.forward..%rrCr)rgrrqr#rrrdrrprnrrrrrrrrIr5rr6r=rs r(r{'LukeForEntityPairClassification.forwardsB&1%1559-1,0/3&*p E,,-p !!2!23p !!1!12 p u//0 p U--. p  ((9(9:p  ((8(89p &e&6&67p E--.p   1 12p ))*p $D>p 'tnp d^p u44 5!p p r'rz The LUKE model with a span classification head on top (a linear layer on top of the hidden states output) for tasks such as named entity recognition. c&^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\RS \\RS \\RS\\RS\\RS\\ S\\ S\\ S\ \ \ 44"Sjj5rSrU=r$)LukeForEntitySpanClassificationi4c6>[TU]U5 [U5UlURUl[ R "UR5Ul[ R"URS-UR5Ul UR5 g)Nr rres r(rV(LukeForEntitySpanClassification.__init__;sn  f%  ++zz&"<"<= ))F$6$6$:F>> from transformers import AutoTokenizer, LukeForEntitySpanClassification >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-conll-2003") >>> model = LukeForEntitySpanClassification.from_pretrained("studio-ousia/luke-large-finetuned-conll-2003") >>> text = "Beyoncé lives in Los Angeles" # List all possible entity spans in the text >>> word_start_positions = [0, 8, 14, 17, 21] # character-based start positions of word tokens >>> word_end_positions = [7, 13, 16, 20, 28] # character-based end positions of word tokens >>> entity_spans = [] >>> for i, start_pos in enumerate(word_start_positions): ... for end_pos in word_end_positions[i:]: ... entity_spans.append((start_pos, end_pos)) >>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> predicted_class_indices = logits.argmax(-1).squeeze().tolist() >>> for span, predicted_class_idx in zip(entity_spans, predicted_class_indices): ... if predicted_class_idx != 0: ... print(text[span[0] : span[1]], model.config.id2label[predicted_class_idx]) Beyoncé PER Los Angeles LOC ```NTrrkrrrc30# UH nUcMUv M g7frrr=s r(r@:LukeForEntitySpanClassification.forward..rrCr)rgrrqrDrrrrrnrpr#gatherrrrdrrrrrrrrrrIr5rr6r@)rfrvrrwrurrrrr r rrxrrrJrKrrY start_states end_statesrr3r0s r(r{'LukeForEntitySpanClassification.forwardGs^&1%%>w?X?X?_?_%` "||G$=$=rCYZ 3==bAHHRQ\]  & &'*C*C*J*J J#7#:#:7;T;T;[;[#\ \\'";";RAUV L*g>^>^#_efgn50  YYv}}-F{{a}}226;;r4??3SU[U`U`acUde}}EEfkkRToW]WbWbceWfWnWnouWvw(=(=w?[?[]d]o]op  .!//!(!=!=))   r'r)NNNNNNNNNNNNNNNN)rrr r!rVrrr#rr$rrrr@r{r&rrs@r(rr4s 156:593715<@<@:>=A;?1559-1,0/3&*#H E,,-H !!2!23H !!1!12 H u//0 H U--. H  ((8(89H  ((8(89H &e&6&67H !))9)9 :H 'u'7'78H E--.H   1 12H ))*H $D>H 'tn!H "d^#H $ u44 5%H H r'rz The LUKE Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c"^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\RS \\RS \\RS\\ S\\ S\\ S\ \ \ 44Sjj5rSrU=r$)LukeForSequenceClassificationicb>[TU]U5 URUl[U5Ul[ R "URb URO UR5Ul [ R"URUR5Ul UR5 grrUrVrrrqrrbclassifier_dropoutrcrdrrYrrres r(rV&LukeForSequenceClassification.__init__s   ++f% zz)/)B)B)NF % %TZTnTn  ))F$6$68I8IJ r'rvrrwrurrrrrrxrrrJrKrcUbUOURRnURUUUUUUUUU U U U SS9 nURnUR U5nUR U5nSnU GbU R UR5n URRcURS:XaSURlOoURS:aNU R[R:XdU R[R:XaSURlOSURlURRS:XaJ[5nURS:Xa&U"UR5U R55nOU"UU 5nOURRS:Xa=[!5nU"UR#SUR5U R#S55nO-URRS:Xa[%5nU"UU 5nU(d5['S UUUR(UR*UR,455$[/UUUR(UR*UR,S 9$) a entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrr regressionsingle_label_classificationmulti_label_classificationrkc30# UH nUcMUv M g7frrr=s r(r@8LukeForSequenceClassification.forward..BrrCr)rgrrqrrdrrprn problem_typerrmr#rtrr squeezer rrrIr5rr6rC)rfrvrrwrurrrrrrxrrrJrKrr]r3r0loss_fcts r(r{%LukeForSequenceClassification.forwards#V&1%1559.2,0/3&*g E,,-g !!2!23g !!1!12 g u//0 g U--. g  ((9(9:g  ((8(89g &e&6&67g E--.g   1 12g **+g $D>g 'tng d^g u22 3!g g r'rz The LUKE Model with a token classification head on top (a linear layer on top of the hidden-states output). To solve Named-Entity Recognition (NER) task using LUKE, `LukeForEntitySpanClassification` is more suitable than this class. c"^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\RS \\RS \\RS\\ S\\ S\\ S\ \ \ 44Sjj5rSrU=r$)LukeForTokenClassificationiQc`>[TU]U5 URUl[USS9Ul[ R "URb URO UR5Ul [ R"URUR5Ul UR5 gNF)rrres r(rV#LukeForTokenClassification.__init__Ys   ++f> zz)/)B)B)NF % %TZTnTn  ))F$6$68I8IJ r'rvrrwrurrrrrrxrrrJrKrcPUbUOURRnURUUUUUUUUU U U U SS9 nURnUR U5nUR U5nSnU bWU R UR5n [5nU"URSUR5U RS55nU(d5[SUUURURUR455$[UUURURURS9$)a entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) NTrrkc30# UH nUcMUv M g7frrr=s r(r@5LukeForTokenClassification.forward..rrCr)rgrrqrDrdrrprnr rrrIr5rr6rG)rfrvrrwrurrrrrrxrrrJrKrrr3r0r"s r(r{"LukeForTokenClassification.forwardfs1V&1%1559.2,0/3&*U E,,-U !!2!23U !!1!12 U u//0 U U--. U  ((9(9:U  ((8(89U &e&6&67U E--.U   1 12U **+U $D>U 'tnU d^U u// 0!U U r'r%c$^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\RS \\RS \\RS\\RS\\ S\\ S\\ S\ \ \ 44 Sjj5rSrU=r$)LukeForQuestionAnsweringic>[TU]U5 URUl[USS9Ul[ R "URUR5UlUR5 gr') rUrVrrrqrrrY qa_outputsrres r(rV!LukeForQuestionAnswering.__init__sU   ++f> ))F$6$68I8IJ r'rvrrwrurrrrrrxstart_positions end_positionsrrJrKrc\UbUOURRnURUUUUUUUUU U U USS9 nURnUR U5nUR SSS9unnUR S5nUR S5nSnU bU b[U R55S:aU R S5n [U R55S:aU R S5n URS5nU RSU5 U RSU5 [US9nU"UU 5nU"UU 5nUU-S - nU(d6[S UUUURURUR455$[UUUURURURS 9$) a entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. NTrrrkrr) ignore_indexrc30# UH nUcMUv M g7frrr=s r(r@3LukeForQuestionAnswering.forward..s" ArC)r0rJrKr5rr6)rgrrqrDr0splitr!lenrrclamp_r rIr5rr6rI)rfrvrrwrurrrrrrxr2r3rrJrKrrr3rJrK total_loss ignored_indexr" start_lossend_losss r(r{ LukeForQuestionAnswering.forwardsP&1%15596:48,0/3&*!f E,,-f !!2!23f !!1!12 f u001 f U--. f  ((9(9:f  ((8(89f &e&6&67f E--.f   1 12f "%"2"23f   0 01f $D>f 'tnf d^!f " u66 7#f f r'r.c"^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\RS \\RS \\RS\\ S\\ S\\ S\ \ \ 44Sjj5rSrU=r$)LukeForMultipleChoicei6c,>[TU]U5 [U5Ul[R "UR b UR O UR5Ul[R"URS5Ul UR5 gr ) rUrVrrqrrbrrcrdrrYrrres r(rVLukeForMultipleChoice.__init__8so  f% zz)/)B)B)NF % %TZTnTn  ))F$6$6: r'rvrrwrurrrrrrxrrrJrKrcPUbUOURRnUbURSOU RSnUb!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnU b1U RSU R S5U R S55OSn Ub!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb1URSUR S5UR S55OSnUR UUUUUUUUU U U U SS9 nUR nURU5nURU5nURSU5nSnU b.U RUR5n [5nU"UU 5nU(d5[SUUURURUR455$[!UUURURURS9$) a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) NrrkrTrc30# UH nUcMUv M g7frrr=s r(r@0LukeForMultipleChoice.forward..rBrCr)rgrrrrrrqrrdrrprnr rIr5rr6rM)rfrvrrwrurrrrrrxrrrJrK num_choicesrr]r3reshaped_logitsr0r"s r(r{LukeForMultipleChoice.forwardDsF&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  BLAWZ__R)<=]a %0 " & &r+@+E+Eb+I J %0 " & &r+@+E+Eb+I J #. $ $R)<)A)A")EGZG_G_`bGc d  ))))%!"7"7 3'/!5   --  ]3 / ++b+6  YY556F')HOV4D #))00&&    -"!//!(!=!=))   r')rrdrqr)rrr r!rVrrr#rr$rrrrMr{r&rrs@r(rArA6s} 156:593715=A<@:>1559.2,0/3&*P E,,-P !!2!23P !!1!12 P u//0 P U--. P  ((9(9:P  ((8(89P &e&6&67P E--.P   1 12P **+P $D>P 'tnP d^P u33 4!P P r'rA) rrrrAr.rr%rrrp)Fr"r dataclassesrtypingrrrr#torch.utils.checkpointrtorch.nnrr r activationsr r modeling_outputsrrmodeling_utilsr pytorch_utilsrutilsrrrconfiguration_luker get_loggerrloggerrr+r.r8r=r@rCrGrIrMrrOrrrrr rrr2rVr`rgrprrorrrrrrr%r.rA__all__rr'r(rWs !)) AA'K-699*   H % I%?I I< I/I I< '?'? '?T ?? ?< ?[? ?< ?[? ?< ?;? ?B ? ? ?B !?{!? !?H  ?K ?  ?FF=RYYF=R(299(Vn nd RYY ,BII,` ryy   1 1hK "))K ^  BII" 299 */**0 Y'#Y'  Y'x 4"**> L )L  L ^ y "5y  y x ~ &9~  ~ B V &9V  V r u $7u  u pd !4d d Ns 2s s l^ /^ ^ B r'