o Zh@s:dZddlZddlZddlmZmZmZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZmZmZmZmZmZdd lmZmZmZmZdd lmZmZd d lm Z e!e"Z#Gd dde j$Z%d8ddZ&Gddde j$Z'Gddde j$Z(Gddde j$Z)Gddde j$Z*Gddde j$Z+Gddde j$Z,Gddde j$Z-Gdd d e j$Z.Gd!d"d"e j$Z/d#e/iZ0Gd$d%d%e j$Z1Gd&d'd'e j$Z2Gd(d)d)e j$Z3eGd*d+d+eZ4eGd,d-d-e4Z5eGd.d/d/e4Z6ed0d1Gd2d3d3e4Z7ed4d1Gd5d6d6e4Z8gd7Z9dS)9zPyTorch MarkupLM model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModelapply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging)MarkupLMConfigcs*eZdZdZfddZdddZZS)XPathEmbeddingszConstruct the embeddings from xpath tags and subscripts. We drop tree-id in this version, as its info can be covered by xpath. cstt|j|_tj|jj|_t j |_ t |_ tj|jdj|_tdjj|_tfddt|jD|_tfddt|jD|_dS)Ncg|] }tjjqS)r EmbeddingZmax_xpath_tag_unit_embeddingsxpath_unit_hidden_size.0_configr]/var/www/auris/lib/python3.10/site-packages/transformers/models/markuplm/modeling_markuplm.py Cz,XPathEmbeddings.__init__..crr)rrZmax_xpath_subs_unit_embeddingsrrr"rr$r%Jr&)superr__init__ max_depthrLinearr hidden_sizeZxpath_unitseq2_embeddingsDropouthidden_dropout_probdropoutZReLU activationxpath_unitseq2_inner inner2emb ModuleListrangexpath_tag_sub_embeddingsxpath_subs_sub_embeddingsselfr# __class__r"r$r(6s"    zXPathEmbeddings.__init__Nc Csg}g}t|jD](}||j||dddd|f||j||dddd|fq tj|dd}tj|dd}||}||| | |}|S)Ndim) r3r)appendr4r5torchcatr1r.r/r0)r7xpath_tags_seqxpath_subs_seqZxpath_tags_embeddingsZxpath_subs_embeddingsixpath_embeddingsrrr$forwardPs&(zXPathEmbeddings.forward)NN)__name__ __module__ __qualname____doc__r(rD __classcell__rrr8r$r0s rcCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr;)neintr>ZcumsumZtype_aslong) input_ids padding_idxpast_key_values_lengthmaskZincremental_indicesrrr$"create_position_ids_from_input_idscs  rQcs@eZdZdZfddZddZ       d dd ZZS) MarkupLMEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cstt|||_tj|j|j|jd|_ t|j |j|_ |j |_ t ||_t|j|j|_tj|j|jd|_t|j|_|jdt|j ddd|j|_tj|j |j|jd|_ dS)N)rNZeps position_ids)rr:F) persistent)r'rRr(r#rr vocab_sizer+Z pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsr)rrCZtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsr,r-r.Zregister_bufferr>arangeexpandrNr6r8rr$r(vs    zMarkupLMEmbeddings.__init__cCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr:rdtypedevicer)sizer>r]rNrLra unsqueezer^)r7 inputs_embeds input_shapeZsequence_lengthrTrrr$&create_position_ids_from_inputs_embedss  z9MarkupLMEmbeddings.create_position_ids_from_inputs_embedsNrcCs<|dur |}n|dd}|dur|jn|j} |dur0|dur+t||j|}n||}|dur=tj|tj| d}|durF||}|dur_|j j tj t t ||jgtj| d}|durx|j jtj t t ||jgtj| d}|} ||} ||} |||} | | | | }||}||}|S)Nr:r_)rbrarQrNrfr>zerosrLrWr#Z tag_pad_idonestuplelistr)Z subs_pad_idrYrZrCr[r.)r7rMr@rAtoken_type_idsrTrdrOreraZwords_embeddingsrYrZrC embeddingsrrr$rDs8          zMarkupLMEmbeddings.forward)NNNNNNr)rErFrGrHr(rfrDrIrrr8r$rRss rRc8eZdZfddZdejdejdejfddZZS)MarkupLMSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNrS) r'r(rr*r+denser[r\r,r-r.r6r8rr$r( zMarkupLMSelfOutput.__init__ hidden_states input_tensorreturncC&||}||}|||}|SNrpr.r[r7rrrsrrr$rD  zMarkupLMSelfOutput.forwardrErFrGr(r>TensorrDrIrrr8r$rn $rnc2eZdZfddZdejdejfddZZS)MarkupLMIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSrv) r'r(rr*r+intermediate_sizerp isinstance hidden_actstrr intermediate_act_fnr6r8rr$r(s   zMarkupLMIntermediate.__init__rrrtcC||}||}|Srv)rprr7rrrrr$rD  zMarkupLMIntermediate.forwardrzrrr8r$r~s r~crm)MarkupLMOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSro) r'r(rr*rr+rpr[r\r,r-r.r6r8rr$r(rqzMarkupLMOutput.__init__rrrsrtcCrurvrwrxrrr$rDryzMarkupLMOutput.forwardrzrrr8r$rr|rcr})MarkupLMPoolercs*tt|j|j|_t|_dSrv)r'r(rr*r+rpZTanhr/r6r8rr$r(s zMarkupLMPooler.__init__rrrtcCs(|dddf}||}||}|S)Nr)rpr/)r7rrZfirst_token_tensor pooled_outputrrr$rD s  zMarkupLMPooler.forwardrzrrr8r$rs rcr})MarkupLMPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dSro) r'r(rr*r+rprrrr transform_act_fnr[r\r6r8rr$r(s  z(MarkupLMPredictionHeadTransform.__init__rrrtcCs"||}||}||}|Srv)rprr[rrrr$rDs   z'MarkupLMPredictionHeadTransform.forwardrzrrr8r$rs  rcs,eZdZfddZddZddZZS)MarkupLMLMPredictionHeadcsLtt||_tj|j|jdd|_t t |j|_ |j |j_ dS)NF)bias) r'r(r transformrr*r+rVdecoder Parameterr>rgrr6r8rr$r('s  z!MarkupLMLMPredictionHead.__init__cCs|j|j_dSrv)rrr7rrr$ _tie_weights4sz%MarkupLMLMPredictionHead._tie_weightscCrrv)rrrrrr$rD7rz MarkupLMLMPredictionHead.forward)rErFrGr(rrDrIrrr8r$r&s  rcr})MarkupLMOnlyMLMHeadcstt||_dSrv)r'r(r predictionsr6r8rr$r(?s zMarkupLMOnlyMLMHead.__init__sequence_outputrtcCs||}|Srv)r)r7rprediction_scoresrrr$rDCs zMarkupLMOnlyMLMHead.forwardrzrrr8r$r>s rcseZdZdfdd ZdejdejfddZ      dd ejd eejd eejd eejd eejdee e ejdee de ejfddZ Z S)MarkupLMSelfAttentionNcst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|p\t|dd|_|jdksh|jd kry|j|_t d |jd |j|_|j|_dS) NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()position_embedding_typeabsolute relative_keyrelative_key_queryr)r'r(r+num_attention_headshasattr ValueErrorrKattention_head_size all_head_sizerr*querykeyvaluer,Zattention_probs_dropout_probr.getattrrrXrdistance_embedding is_decoderr7r#rr8rr$r(Js*   zMarkupLMSelfAttention.__init__xrtcCs6|dd|j|jf}||}|ddddS)Nr:rrrr )rbrrviewpermute)r7rZ new_x_shaperrr$transpose_for_scoresds z*MarkupLMSelfAttention.transpose_for_scoresFrrattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionscCs||}|du} | r|dur|d} |d} |}nP| r/|||} |||} |}n;|durZ|||} |||} tj|d| gdd} tj|d| gdd} n|||} |||} ||} |du} |jrz| | f}t| | dd}|j dks|j dkr | j d| j d}}| rtj |dtj |j d dd}ntj|tj |j d dd}tj|tj |j d dd}||}|||jd}|j| jd }|j dkrtd | |}||}n|j dkr td | |}td | |}|||}|t|j}|dur||}tjj|dd}||}|dur0||}t|| }|dddd }|dd|jf}||}|rX||fn|f}|jrd||f}|S)Nrrrr;r:rrr_r`zbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rrrrr>r?rmatmulZ transposershapeZtensorrLrarr]rrXtor`ZeinsummathsqrtrrZ functionalZsoftmaxr.r contiguousrbr)r7rrrrrrrrZmixed_query_layerZis_cross_attentionZ key_layerZ value_layerZ query_layer use_cacheZattention_scoresZ query_lengthZ key_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_probsZ context_layerZnew_context_layer_shapeoutputsrrr$rDisn             zMarkupLMSelfAttention.forwardrvNNNNNF) rErFrGr(r>r{rr FloatTensorrboolrDrIrrr8r$rIs4 reagercseZdZdfdd ZddZ      ddejdeejd eejd eejd eejd ee e ejd ee de ejfddZ Z S)MarkupLMAttentionNcs4tt|j||d|_t||_t|_dS)Nr) r'r(MARKUPLM_SELF_ATTENTION_CLASSESZ_attn_implementationr7rnoutputset pruned_headsrr8rr$r(s   zMarkupLMAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr;)lenrr7rrrrrrrrrprunion)r7headsindexrrr$ prune_headss zMarkupLMAttention.prune_headsFrrrrrrrrrtc Cs<||||||||}||d|} | f|dd} | S)Nrr)r7r) r7rrrrrrrrZ self_outputsattention_outputrrrr$rDs  zMarkupLMAttention.forwardrvr) rErFrGr(rr>r{rrrrrDrIrrr8r$rs4 rcseZdZfddZ      ddejdeejdeejdeejd eejd eeeejd ee d eejfd dZ ddZ Z S) MarkupLMLayercsrt|j|_d|_t||_|j|_|j|_|jr-|js&t|dt|dd|_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is addedrr)r'r(chunk_size_feed_forward seq_len_dimr attentionradd_cross_attentionrcrossattentionr~ intermediaterrr6r8rr$r( s   zMarkupLMLayer.__init__NFrrrrrrrrrtc Cs|dur |ddnd}|j|||||d} | d} |jr(| dd} | d} n| dd} d} |jro|durot|dsDtd|d|durN|d dnd}|| ||||||}|d} | |dd} |d} | | } t|j|j|j| }|f| } |jr| | f} | S) Nr)rrrrr:rz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrrrrfeed_forward_chunkrr)r7rrrrrrrrZself_attn_past_key_valueZself_attention_outputsrrZpresent_key_valueZcross_attn_present_key_valueZcross_attn_past_key_valueZcross_attention_outputs layer_outputrrr$rDsP         zMarkupLMLayer.forwardcCs||}|||}|Srv)rr)r7rZintermediate_outputrrrr$rYs  z MarkupLMLayer.feed_forward_chunkr) rErFrGr(r>r{rrrrrDrrIrrr8r$r s4  ArcseZdZfddZ         ddejdeejdeejd eejd eejd eeeejd ee d ee dee dee de eeje ffddZ Z S)MarkupLMEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr)rrr"rr$r%dsz,MarkupLMEncoder.__init__..F) r'r(r#rr2r3num_hidden_layerslayergradient_checkpointingr6r8r"r$r(as   zMarkupLMEncoder.__init__NFTrrrrrrpast_key_valuesrroutput_hidden_states return_dictrtc  Cs^| rdnd} |r dnd} |r|jjrdnd} |jr%|jr%|r%tdd}|r)dnd}t|jD]^\}}| r;| |f} |durC||nd}|durM||nd}|jrc|jrc||j |||||||}n ||||||||}|d}|rz||df7}|r| |df} |jjr| |df} q0| r| |f} | st dd ||| | | fDSt ||| | | d S) NrzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frr:rrcss|] }|dur|VqdSrvr)r vrrr$ sz*MarkupLMEncoder.forward..)last_hidden_staterrr attentionscross_attentions) r#rrZtrainingloggerZ warning_once enumeraterZ_gradient_checkpointing_func__call__rir )r7rrrrrrrrrrrZall_hidden_statesZall_self_attentionsZall_cross_attentionsZnext_decoder_cacherBZ layer_moduleZlayer_head_maskrZ layer_outputsrrr$rDgsz         zMarkupLMEncoder.forward) NNNNNNFFT)rErFrGr(r>r{rrrrrr rDrIrrr8r$r`sD      rcsDeZdZeZdZddZedee e e j fffdd Z ZS)MarkupLMPreTrainedModelmarkuplmcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSt|tre|jj dSdS)zInitialize the weightsg)meanZstdN?)rrr*weightdataZnormal_r#Zinitializer_rangerZzero_rrNr[Zfill_r)r7modulerrr$ _init_weightss       z%MarkupLMPreTrainedModel._init_weightspretrained_model_name_or_pathcstt|j|g|Ri|Srv)r'rfrom_pretrained)clsrZ model_argskwargsr8rr$rs z'MarkupLMPreTrainedModel.from_pretrained)rErFrGrZ config_classZbase_model_prefixr classmethodrrrosPathLikerrIrrr8r$rs *rcseZdZdfdd ZddZddZdd Ze dd ee j d ee j d ee j dee j dee j dee j dee j dee j dee dee dee de eeffddZddZZS) MarkupLMModelTcsDt|||_t||_t||_|rt|nd|_| dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r'r(r#rRrlrencoderrpooler post_init)r7r#add_pooling_layerr8rr$r(s    zMarkupLMModel.__init__cCs|jjSrvrlrWrrrr$get_input_embeddingssz"MarkupLMModel.get_input_embeddingscCs ||j_dSrvr)r7rrrr$set_input_embeddingss z"MarkupLMModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r7Zheads_to_prunerrrrr$ _prune_headsszMarkupLMModel._prune_headsNrMr@rArrkrTrrdrrrrtc Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur*|dur*td|dur9||||} n|durF|dd} ntd|durQ|jn|j} |dur_tj | | d}|durltj | tj | d}| d d}|j |jd }d |d }|dur|dkr| d  d  d d}||jjdddd}n|dkr| d d d}|j t|jd }ndg|jj}|j||||||d }|j|||| | | d}|d }|jdur||nd}| s||f|ddSt|||j|j|jdS)a xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. Examples: ```python >>> from transformers import AutoProcessor, MarkupLMModel >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = MarkupLMModel.from_pretrained("microsoft/markuplm-base") >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 4, 768] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer:z5You have to specify either input_ids or inputs_embeds)rar_rrrrgr)rMr@rArTrkrd)rrrr)rZ pooler_outputrrrr)r#rruse_return_dictrZ%warn_if_padding_and_no_attention_maskrbrar>rhrgrLrcrr`r<r^rnext parametersrlrrr rrrr)r7rMr@rArrkrTrrdrrrreraZextended_attention_maskZembedding_outputZencoder_outputsrrrrr$rDsn&     zMarkupLMModel.forwardcs.d}|D]}|tfdd|Df7}q|S)Nrc3s$|] }|d|jVqdS)rN)Z index_selectrra)r Z past_statebeam_idxrr$rms"z/MarkupLMModel._reorder_cache..)ri)r7rrZreordered_pastZ layer_pastrrr$_reorder_cacheis zMarkupLMModel._reorder_cache)T) NNNNNNNNNNN)rErFrGr(rrrrrr>Z LongTensorrrrrr rDrrIrrr8r$rsV     krc seZdZfddZe             ddeejdeejdeejdeejdeejd eejd eejd eejd eejd eejdeedeedeede e eje ffddZ Z S)MarkupLMForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSNF)r) r'r( num_labelsrrrr*r+ qa_outputsrr6r8rr$r(us  z%MarkupLMForQuestionAnswering.__init__NrMr@rArrkrTrrdstart_positions end_positionsrrrrtc CsL| dur| n|jj} |j||||||||| | | d }|d}||}|jddd\}}|d}|d}d}| dur| durt| dkrQ| d} t| dkr^| d} |d}| d|| d|t |d}||| }||| }||d}| s||f|dd}|dur|f|S|St ||||j |j d S) ae xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. Examples: ```python >>> from transformers import AutoProcessor, MarkupLMForQuestionAnswering >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> model = MarkupLMForQuestionAnswering.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> html_string = " My name is Niels " >>> question = "What's his name?" >>> encoding = processor(html_string, questions=question, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> processor.decode(predict_answer_tokens).strip() 'Niels' ```N r@rArrkrTrrdrrrrrr:r;)Z ignore_indexr)loss start_logits end_logitsrrr)r#rrrsplitsqueezerrrbZclamp_rrrrr)r7rMr@rArrkrTrrdr r rrrrrlogitsr rZ total_lossZ ignored_indexloss_fctZ start_lossZend_lossrrrr$rDsT/          z$MarkupLMForQuestionAnswering.forward) NNNNNNNNNNNNN)rErFrGr(rrr>r{rrrrrDrIrrr8r$rrsZ       rzC MarkupLM Model with a `token_classification` head on top. )Z custom_introceZdZfddZe            ddeejdeejdeejdeejdeejd eejd eejd eejd eejd eedeedeede e eje ffddZ Z S)MarkupLMForTokenClassificationcsbt||j|_t|dd|_|jdur|jn|j}t||_ t |j |j|_ | dSr)r'r(rrrclassifier_dropoutr-rr,r.r*r+ classifierrr7r#rr8rr$r(s   z'MarkupLMForTokenClassification.__init__NrMr@rArrkrTrrdlabelsrrrrtc  Cs| dur| n|jj} |j||||||||| | | d } | d}||}d}| dur:t}||d|jj| d}| sP|f| dd}|durN|f|S|St||| j| j dS)a xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import AutoProcessor, AutoModelForTokenClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> processor.parse_html = False >>> model = AutoModelForTokenClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> nodes = ["hello", "world"] >>> xpaths = ["/html/body/div/li[1]/div/span", "/html/body/div/li[1]/div/span"] >>> node_labels = [1, 2] >>> encoding = processor(nodes=nodes, xpaths=xpaths, node_labels=node_labels, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```Nr rr:rr rrrr) r#rrrrrrrrrr)r7rMr@rArrkrTrrdrrrrrrrr rrrrr$rDs@- z&MarkupLMForTokenClassification.forward NNNNNNNNNNNN)rErFrGr(rrr>r{rrrr rDrIrrr8r$rsT      rz MarkupLM Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cr)!MarkupLMForSequenceClassificationcsdt||j|_||_t||_|jdur|jn|j}t ||_ t |j |j|_ |dSrv)r'r(rr#rrrr-rr,r.r*r+rrrr8rr$r(Ts    z*MarkupLMForSequenceClassification.__init__NrMr@rArrkrTrrdrrrrrtc  Csv| dur| n|jj} |j||||||||| | | d } | d}||}||}d}| dur|jjdurX|jdkr>d|j_n|jdkrT| jtj ksO| jtj krTd|j_nd|j_|jjdkrvt }|jdkrp|| | }n+||| }n%|jjdkrt }||d|j| d}n|jjdkrt}||| }| s|f| dd}|dur|f|S|St||| j| jd S) a xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoProcessor, AutoModelForSequenceClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = AutoModelForSequenceClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```Nr rZ regressionZsingle_label_classificationZmulti_label_classificationr:rr)r#rrr.rZ problem_typerr`r>rLrKrrrrrrrrr)r7rMr@rArrkrTrrdrrrrrrrr rrrrr$rDcsZ,     "       z)MarkupLMForSequenceClassification.forwardr)rErFrGr(rrr>r{rrrrrDrIrrr8r$rLsT      r)rrrrr)r):rHrrtypingrrrr>Ztorch.utils.checkpointrZtorch.nnrrrZ activationsr Zmodeling_outputsr r r rrrZmodeling_utilsrrrrutilsrrZconfiguration_markuplmrZ get_loggerrErModulerrQrRrnr~rrrrrrrrrrrrrrr__all__rrrr$s^      3c 4W]pd t