o Zh@sdZddlZddlZddlmZmZmZddlZddlm Z ddl m m Z ddl ZddlmZmZmZddlmZddlmZmZmZmZddlmZdd lmZdd lmZmZmZd d l m!Z!e"e#Z$Gd dde j%Z&Gddde j%Z'eGdddeZ(Gddde j%Z)Gddde j%Z*Gddde j%Z+Gddde j%Z,Gddde j%Z-Gddde j%Z.Gdd d e j%Z/eGd!d"d"e(Z0Gd#d$d$e j%Z1ed%d&Gd'd(d(e(Z2eGd)d*d*e(Z3ed+d&Gd,d-d-e(Z4gd.Z5dS)/zPyTorch LayoutLMv3 model.N)OptionalTupleUnion)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)auto_docstringlogging torch_int)LayoutLMv3Configcs*eZdZdZfddZdddZZS)LayoutLMv3PatchEmbeddingszLayoutLMv3 image (patch) embeddings. This class also automatically interpolates the position embeddings for varying image sizes.cstt|jtjjr|jn|j|jf}t|jtjjr!|jn|j|jf}|d|d|d|df|_t j |j |j ||d|_ dS)Nrr)Z kernel_sizeZstride)super__init__ isinstance input_size collectionsabcIterable patch_size patch_shapennConv2dZ num_channels hidden_sizeproj)selfconfig image_sizer __class__a/var/www/auris/lib/python3.10/site-packages/transformers/models/layoutlmv3/modeling_layoutlmv3.pyr3s   "z"LayoutLMv3PatchEmbeddings.__init__NcCs||}|dur8|d|jd|jdd}|dddd}|jd|jd}}tj|||fdd}||}|ddd}|S)NrrrZbicubic)sizemode) r"viewrpermuteshapeFZ interpolateflatten transpose)r# pixel_valuesZposition_embedding embeddings patch_height patch_widthr(r(r)forwardCs z!LayoutLMv3PatchEmbeddings.forwardN__name__ __module__ __qualname____doc__rr8 __classcell__r(r(r&r)r/s rcsLeZdZdZfddZddZddZdd Z d d d ZZ S)LayoutLMv3TextEmbeddingszm LayoutLMv3 text embeddings. Same as `RobertaEmbeddings` but with added spatial (layout) embeddings. csttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j |_|jdt|jddd|j|_tj|j|j|jd|_t|j|j|_t|j|j|_t|j|j|_t|j|j|_dS)N) padding_idxZeps position_ids)rr*F) persistent)rrr EmbeddingZ vocab_sizer!Z pad_token_idword_embeddingsZtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutZregister_buffertorcharangeZmax_position_embeddingsexpandrAposition_embeddingsZmax_2d_position_embeddingsZcoordinate_sizex_position_embeddingsy_position_embeddingsZ shape_sizeh_position_embeddingsw_position_embeddingsr#r$r&r(r)rWs  z!LayoutLMv3TextEmbeddings.__init__c Cs.z:||dddddf}||dddddf}||dddddf}||dddddf}WntyK}ztd|d}~ww|t|dddddf|dddddfdd}|t|dddddf|dddddfdd}tj||||||gdd} | S) Nrrr+rz;The `bbox` coordinate values should be within 0-1000 range.ir*dim)rQrR IndexErrorrSrMZcliprTcat) r#bboxZleft_position_embeddingsZupper_position_embeddingsZright_position_embeddingsZlower_position_embeddingserSrTspatial_position_embeddingsr(r(r)%calculate_spatial_position_embeddingsns,  << z>LayoutLMv3TextEmbeddings.calculate_spatial_position_embeddingscCs2||}tj|dd||}||S)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. rrV)neintrMZcumsumZtype_aslong)r# input_idsrAmaskZincremental_indicesr(r(r)"create_position_ids_from_input_idss z;LayoutLMv3TextEmbeddings.create_position_ids_from_input_idscCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Nr*rdtypedevicer)r,rMrNrAr`rf unsqueezerO)r# inputs_embeds input_shapeZsequence_lengthrCr(r(r)&create_position_ids_from_inputs_embedss z?LayoutLMv3TextEmbeddings.create_position_ids_from_inputs_embedsNc Cs|dur|dur|||j|j}n||}|dur"|}n|dd}|dur9tj|tj|j jd}|durB| |}| |}||}| |} || 7}| |} || }||}||}|S)Nr*rd)rcrAtorfrjr,rMzerosr`rCrFrGrPr]rHrL) r#rarZtoken_type_idsrCrhrirGr5rPr\r(r(r)r8s,        z LayoutLMv3TextEmbeddings.forward)NNNNN) r;r<r=r>rr]rcrjr8r?r(r(r&r)r@Rs  r@c@seZdZeZdZddZdS)LayoutLMv3PreTrainedModel layoutlmv3cCst|tjtjfr#|jjjd|jjd|j dur!|j j dSdSt|tj rF|jjjd|jjd|j durD|jj|j  dSdSt|tj r[|j j |jjddSt|trr|jjrt|jj |jj dSdSdS)zInitialize the weights)meanZstdNg?)rrLinearr weightdataZnormal_r$Zinitializer_rangebiasZzero_rErArHZfill_LayoutLMv3Model visual_embed cls_token pos_embed)r#moduler(r(r) _init_weightss&       z'LayoutLMv3PreTrainedModel._init_weightsN)r;r<r=rZ config_classZbase_model_prefixr{r(r(r(r)rns rncsBeZdZfddZddZd ddZ    d d d ZZS)LayoutLMv3SelfAttentioncst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|j|_|j|_dS)NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())rrr!num_attention_headshasattr ValueErrorr_attention_head_size all_head_sizerrrquerykeyvaluerJZattention_probs_dropout_probrLhas_relative_attention_biashas_spatial_attention_biasrUr&r(r)rs   z LayoutLMv3SelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)Nr*rr+rr)r,r~rr.r/)r#xZ new_x_shaper(r(r)transpose_for_scoress z,LayoutLMv3SelfAttention.transpose_for_scores cCs6||}|jddd}|||}tjdd|S)a https://arxiv.org/abs/2105.13290 Section 2.4 Stabilization of training: Precision Bottleneck Relaxation (PB-Relax). A replacement of the original nn.Softmax(dim=-1)(attention_scores). Seems the new attention_probs will result in a slower speed and a little bias. Can use torch.allclose(standard_attention_probs, cogview_attention_probs, atol=1e-08) for comparison. The smaller atol (e.g., 1e-08), the better. r*rV)ZamaxrgrZSoftmax)r#attention_scoresalphaZscaled_attention_scoresZ max_valueZnew_attention_scoresr(r(r)cogview_attentions z)LayoutLMv3SelfAttention.cogview_attentionNFcCs$||}|||}|||} ||} t| t|j| dd} |j r=|j r=| ||t|j7} n |j rJ| |t|j7} |durR| |} | | } | | } |durd| |} t| | } | dddd} | dd|jf}| j|} |r| | f}|S| f}|S)Nr*rr+rr)rrrrrMmatmulmathsqrtrr3rrrrLr/ contiguousr,rr.)r# hidden_statesattention_mask head_maskoutput_attentionsrel_pos rel_2d_posZmixed_query_layerZ key_layerZ value_layerZ query_layerrZattention_probsZ context_layerZnew_context_layer_shapeoutputsr(r(r)r8s.         zLayoutLMv3SelfAttention.forward)rNNFNN)r;r<r=rrrr8r?r(r(r&r)r|s  r|c8eZdZfddZdejdejdejfddZZS)LayoutLMv3SelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNrB) rrrrrr!denserHrIrJrKrLrUr&r(r)r> zLayoutLMv3SelfOutput.__init__r input_tensorreturncC&||}||}|||}|Sr9rrLrHr#rrr(r(r)r8D  zLayoutLMv3SelfOutput.forwardr;r<r=rrMZTensorr8r?r(r(r&r)r= $rcs0eZdZfddZ     dddZZS)LayoutLMv3Attentioncs"tt||_t||_dSr9)rrr|r#routputrUr&r(r)rMs  zLayoutLMv3Attention.__init__NFc Cs<|j||||||d}||d|}|f|dd} | S)Nrrrr)r#r) r#rrrrrrZ self_outputsattention_outputrr(r(r)r8Rs zLayoutLMv3Attention.forwardr)r;r<r=rr8r?r(r(r&r)rLs rcs8eZdZfddZ     d ddZddZZS) LayoutLMv3Layercs:t|j|_d|_t||_t||_t||_ dSNr) rrchunk_size_feed_forward seq_len_dimr attentionLayoutLMv3Intermediate intermediateLayoutLMv3OutputrrUr&r(r)rjs   zLayoutLMv3Layer.__init__NFc CsL|j||||||d}|d}|dd} t|j|j|j|} | f| } | S)N)rrrrr)rrfeed_forward_chunkrr) r#rrrrrrZself_attention_outputsrr layer_outputr(r(r)r8rs   zLayoutLMv3Layer.forwardcCs||}|||}|Sr9)rr)r#rZintermediate_outputrr(r(r)rs  z"LayoutLMv3Layer.feed_forward_chunkr)r;r<r=rr8rr?r(r(r&r)ris   rcsReZdZfddZdddZdd Zd d Z  dddZZS)LayoutLMv3Encodercst|_tfddtjD|_d|_j |_ j |_ |j r9j |_ j |_ tj |j jdd|_|j r\j|_j|_tj |jjdd|_tj |jjdd|_dSdS)Ncsg|]}tqSr()r).0_r$r(r) sz.LayoutLMv3Encoder.__init__..F)ru)rrr$rZ ModuleListrangenum_hidden_layerslayergradient_checkpointingrr rel_pos_bins max_rel_posrrr~ rel_pos_biasmax_rel_2d_posrel_2d_pos_binsrel_pos_x_biasrel_pos_y_biasrUr&rr)rs  zLayoutLMv3Encoder.__init__Trc Csd}|r|d}||dk|7}t|}n t| t|}|d}||k}|t||t||||tj} t | t | |d} |t ||| 7}|S)Nrr+r) r`rMabsmaxZ zeros_likelogfloatrrkminZ full_likewhere) r#Zrelative_position bidirectional num_buckets max_distanceretnZ max_exactZis_smallZ val_if_larger(r(r)relative_position_buckets &z*LayoutLMv3Encoder.relative_position_bucketcCsx|d|d}|j||j|jd}t|jj| dddd}Wdn1s1wY| }|S)Nrr*rrrrrr+) rgrrrrMno_gradrrstr/r)r#rCZ rel_pos_matrr(r(r)_cal_1d_pos_embs z!LayoutLMv3Encoder._cal_1d_pos_embc Cs|dddddf}|dddddf}|d|d}|d|d}|j||j|jd}|j||j|jd}t$|jj| dddd}|j j| dddd}Wdn1siwY| }| }||}|S)Nrrrr*rrr+) rgrrrrMrrrsrr/rr) r#rZZposition_coord_xZposition_coord_yZrel_pos_x_2d_matZrel_pos_y_2d_matZ rel_pos_xZ rel_pos_yrr(r(r)_cal_2d_pos_embs, z!LayoutLMv3Encoder._cal_2d_pos_embNFc  Cs|rdnd} |r dnd} |jr||nd} |jr||nd}t|jD]?\}}|r0| |f} |dur8||nd}|jrM|jrM||j ||||| |}n |||||| |d}|d}|rd| |df} q%|rl| |f} |szt dd|| | fDSt || | dS)Nr(rrrcss|] }|dur|VqdSr9r()rvr(r(r) !sz,LayoutLMv3Encoder.forward..Zlast_hidden_stater attentions) rrrr enumeraterrZtrainingZ_gradient_checkpointing_func__call__tupler )r#rrZrrroutput_hidden_states return_dictrCr6r7Zall_hidden_statesZall_self_attentionsrriZ layer_moduleZlayer_head_maskZ layer_outputsr(r(r)r8sX       zLayoutLMv3Encoder.forward)Trr) NNNFFTNNN) r;r<r=rrrrr8r?r(r(r&r)rs  rcs2eZdZfddZdejdejfddZZS)rcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr9) rrrrrr!intermediate_sizerrZ hidden_actstrr intermediate_act_fnrUr&r(r)r3s   zLayoutLMv3Intermediate.__init__rrcCs||}||}|Sr9)rr)r#rr(r(r)r8;s  zLayoutLMv3Intermediate.forwardrr(r(r&r)r2s rcr)rcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rrrrrrr!rrHrIrJrKrLrUr&r(r)rCrzLayoutLMv3Output.__init__rrrcCrr9rrr(r(r)r8IrzLayoutLMv3Output.forwardrr(r(r&r)rBrrcseZdZfddZddZddZddZd d d Zd dZddZ e           d!de e j de e j de e jde e j de e j de e jde e jde e jde ede ede edeeeffddZZS)"rvcst|||_|jrt||_|jrnt||_t |j |j }t tdd|j|_t td||d|j|_t jdd|_t j|j|jd|_t |j|_|jjs]|jjre|j||fdt j|jdd|_t||_|dS)Nrrp)prB)r%gư>) rrr$Z text_embedr@r5rwr patch_embedr_rrr ParameterrMrlr!rxryrJpos_droprHrIrKrLrrinit_visual_bboxnormrencoder init_weights)r#r$r,r&r(r)rRs"      zLayoutLMv3Model.__init__cCs|jjSr9r5rF)r#r(r(r)get_input_embeddingsosz$LayoutLMv3Model.get_input_embeddingscCs ||j_dSr9r)r#rr(r(r)set_input_embeddingsrs z$LayoutLMv3Model.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrZ prune_heads)r#Zheads_to_prunerZheadsr(r(r) _prune_headsuszLayoutLMv3Model._prune_headsrcCstjtd||dd||ddd}tjtd||dd||ddd}tj|dd|dd|dd|dddd|dd|dd|dd|ddddgdddd}tdd|d|dgg}tj||gdd|_ dS) zJ Create the bounding boxes for the visual (patch) tokens. rrtrunc)Z rounding_modeNr*rV) rMdivrNstackrepeatr3r.ZtensorrY visual_bbox)r#r%max_lenZ visual_bbox_xZ visual_bbox_yrZ cls_token_boxr(r(r)r}s$   z LayoutLMv3Model.init_visual_bboxcCs$|j|dd}|||}|Sr)rrrktype)r#rfre batch_sizerr(r(r)calculate_visual_bboxsz%LayoutLMv3Model.calculate_visual_bboxcCsf||}|\}}}|j|dd}tj||fdd}|jdur'||j}||}||}|S)Nr*rrV) rr,rxrOrMrYryrr)r#r4r5rZseq_lenrZ cls_tokensr(r(r) forward_images     zLayoutLMv3Model.forward_imageNrarZrrmrCrrhr4rrrrc  Cs|| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur.|} | \} }|j}n$|durB|dd} | \} }|j}n|durNt|} |j}ntd|dusZ|dur|durgtj | |f|d}|durttj | tj |d}|durtj t t | dgtj |d}|j|||||d}d}}d}}|durYt|jd|jjt|jd |jj}}||}tj | |jd ftj |d}|durtj||gd d }n|}|jjs|jjr8|jjr|j|tj | d }|durtj||gd d }n|}tjd |jd tj |d| d }|dus|dur6tjd | d |dd }|| }tj||gd d }n|}|dusB|durLtj||gd d }n|}||}||}n+|jjsc|jjr|jjrj|}|jjr|jjddd| d f}||}|}|j|d||j d}|!||jj"}|j#|||||| | | ||d }|d }| s|f|d dSt$||j%|j&dS)a input_ids (`torch.LongTensor` of shape `(batch_size, token_sequence_length)`): Indices of input sequence tokens in the vocabulary. Note that `sequence_length = token_sequence_length + patch_sequence_length + 1` where `1` is for [CLS] token. See `pixel_values` for `patch_sequence_length`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) bbox (`torch.LongTensor` of shape `(batch_size, token_sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. Note that `sequence_length = token_sequence_length + patch_sequence_length + 1` where `1` is for [CLS] token. See `pixel_values` for `patch_sequence_length`. token_type_ids (`torch.LongTensor` of shape `(batch_size, token_sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. Note that `sequence_length = token_sequence_length + patch_sequence_length + 1` where `1` is for [CLS] token. See `pixel_values` for `patch_sequence_length`. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, token_sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. Note that `sequence_length = token_sequence_length + patch_sequence_length + 1` where `1` is for [CLS] token. See `pixel_values` for `patch_sequence_length`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, token_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. Examples: ```python >>> from transformers import AutoProcessor, AutoModel >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv3-base", apply_ocr=False) >>> model = AutoModel.from_pretrained("microsoft/layoutlmv3-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train", trust_remote_code=True) >>> example = dataset[0] >>> image = example["image"] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = processor(image, words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state ```Nr*zEYou have to specify either input_ids or inputs_embeds or pixel_values)rfrdr)rarZrCrmrhr+rrrV)rerr)re) rZrCrrrrrr6r7r)'r$rruse_return_dictr,rflenrrMZonesrlr`rlistr5rr0rrrYrrrrNrrgrOrHrLrCZ expand_asZget_extended_attention_maskreZ get_head_maskrrr rr)r#rarZrrmrCrrhr4rrrrir seq_lengthrfZembedding_outputZ final_bboxZfinal_position_idsr6r7Zvisual_embeddingsZvisual_attention_maskrZvisual_position_idsZextended_attention_maskZencoder_outputssequence_outputr(r(r)r8sN            zLayoutLMv3Model.forward)rr) NNNNNNNNNNN)r;r<r=rrrrrrrrrrM LongTensor FloatTensorboolrrr r8r?r(r(r&r)rvPsZ       rvcs*eZdZdZdfdd ZddZZS)LayoutLMv3ClassificationHeadz\ Head for sentence-level classification tasks. Reference: RobertaClassificationHead Fcsvt||_|rt|jd|j|_n t|j|j|_|jdur'|jn|j}t ||_ t|j|j |_ dS)Nr) rr pool_featurerrrr!rclassifier_dropoutrKrJrL num_labelsout_proj)r#r$r r r&r(r)rvs  z%LayoutLMv3ClassificationHead.__init__cCs6||}||}t|}||}||}|Sr9)rLrrMtanhr )r#rr(r(r)r8s     z$LayoutLMv3ClassificationHead.forward)Fr:r(r(r&r)rqs ra LayoutLMv3 Model with a token classification head on top (a linear layer on top of the final hidden states) e.g. for sequence labeling (information extraction) tasks such as [FUNSD](https://guillaumejaume.github.io/FUNSD/), [SROIE](https://rrc.cvc.uab.es/?ch=13), [CORD](https://github.com/clovaai/cord) and [Kleister-NDA](https://github.com/applicaai/kleister-nda). )Z custom_introcseZdZfddZe            ddeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee deejde e e ffddZ ZS) LayoutLMv3ForTokenClassificationcsdt||j|_t||_t|j|_|jdkr%t |j |j|_ nt |dd|_ | dS)N Fr )rrr rvrorrJrKrLrrr! classifierrrrUr&r(r)rs    z)LayoutLMv3ForTokenClassification.__init__NrarZrrmrCrrhlabelsrrrr4rc  Cs| dur| n|jj} |j|||||||| | | | d } |dur#|}n|dd}|d}| dddd|f}||}||}d}|dur[t}||d|j|d}| sq|f| dd}|duro|f|S|St ||| j | j dS)a9 bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import AutoProcessor, AutoModelForTokenClassification >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv3-base", apply_ocr=False) >>> model = AutoModelForTokenClassification.from_pretrained("microsoft/layoutlmv3-base", num_labels=7) >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train", trust_remote_code=True) >>> example = dataset[0] >>> image = example["image"] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> word_labels = example["ner_tags"] >>> encoding = processor(image, words, boxes=boxes, word_labels=word_labels, return_tensors="pt") >>> outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```N) rZrrmrCrrhrrrr4r*rrlosslogitsrr) r$rror,rLrrr.r r rr)r#rarZrrmrCrrhrrrrr4rrirrrrloss_fctrr(r(r)r8sD/   z(LayoutLMv3ForTokenClassification.forward NNNNNNNNNNNN)r;r<r=rrrrMrrrrrr r8r?r(r(r&r)rsT        rc seZdZfddZe             ddeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee deejdeejde e e ffddZ ZS)LayoutLMv3ForQuestionAnsweringcs8t||j|_t||_t|dd|_|dSNFr)rrr rvror qa_outputsrrUr&r(r)rs   z'LayoutLMv3ForQuestionAnswering.__init__NrarrmrCrrhstart_positions end_positionsrrrrZr4rc CsL| dur| n|jj} |j||||||| | | | | d }|d}||}|jddd\}}|d}|d}d}|dur|durt|dkrQ|d}t|dkr^|d}|d}| d|}| d|}t |d}|||}|||}||d}| s||f|dd}|dur|f|S|St ||||j |j d S) a/ bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. Examples: ```python >>> from transformers import AutoProcessor, AutoModelForQuestionAnswering >>> from datasets import load_dataset >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv3-base", apply_ocr=False) >>> model = AutoModelForQuestionAnswering.from_pretrained("microsoft/layoutlmv3-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train", trust_remote_code=True) >>> example = dataset[0] >>> image = example["image"] >>> question = "what's his name?" >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = processor(image, question, words, boxes=boxes, return_tensors="pt") >>> start_positions = torch.tensor([1]) >>> end_positions = torch.tensor([3]) >>> outputs = model(**encoding, start_positions=start_positions, end_positions=end_positions) >>> loss = outputs.loss >>> start_scores = outputs.start_logits >>> end_scores = outputs.end_logits ```N rrmrCrrhrrrrZr4rrr*rV)Z ignore_indexr+)r start_logits end_logitsrr)r$rrorsplitsqueezerrr,clamprr rr)r#rarrmrCrrhrrrrrrZr4rrrrrZ total_lossZ ignored_indexrZ start_lossZend_lossrr(r(r)r8sT3          z&LayoutLMv3ForQuestionAnswering.forward) NNNNNNNNNNNNN)r;r<r=rrrrMrrrrrr r8r?r(r(r&r)rsZ        ra  LayoutLMv3 Model with a sequence classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for document image classification tasks such as the [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/) dataset. cseZdZfddZe            ddeejdeejdeejdeejdeejd eejd eejd ee d ee d ee deejdeejde e e ffddZ ZS)#LayoutLMv3ForSequenceClassificationcs>t||j|_||_t||_t|dd|_|dSr) rrr r$rvrorrrrUr&r(r)rws   z,LayoutLMv3ForSequenceClassification.__init__NrarrmrCrrhrrrrrZr4rc  Cs~| dur| n|jj} |j|||||||| | | | d } | ddddddf}||}d}|dur|jjdur\|jdkrBd|j_n|jdkrX|jtjksS|jtj krXd|j_nd|j_|jjdkrzt }|jdkrt|| | }n+|||}n%|jjdkrt }|| d|j| d}n|jjdkrt}|||}| s|f| dd}|dur|f|S|St||| j| jd S) aw bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. Examples: ```python >>> from transformers import AutoProcessor, AutoModelForSequenceClassification >>> from datasets import load_dataset >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv3-base", apply_ocr=False) >>> model = AutoModelForSequenceClassification.from_pretrained("microsoft/layoutlmv3-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train", trust_remote_code=True) >>> example = dataset[0] >>> image = example["image"] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = processor(image, words, boxes=boxes, return_tensors="pt") >>> sequence_label = torch.tensor([1]) >>> outputs = model(**encoding, labels=sequence_label) >>> loss = outputs.loss >>> logits = outputs.logits ```NrrrZ regressionZsingle_label_classificationZmulti_label_classificationr*r)r$rrorZ problem_typer rerMr`r_rr!rr.rr rr)r#rarrmrCrrhrrrrrZr4rrrrrrr(r(r)r8sX.    "       z+LayoutLMv3ForSequenceClassification.forwardr)r;r<r=rrrrMrrrrrr r8r?r(r(r&r)r#osT        r#)rr#rrvrn)6r>rrtypingrrrrMZtorch.nnrZtorch.nn.functionalZ functionalr1Ztorch.utils.checkpointrrrZ activationsr Zmodeling_outputsr r r r Zmodeling_utilsrZ pytorch_utilsrutilsrrrZconfiguration_layoutlmv3rZ get_loggerr;loggerModulerr@rnr|rrrrrrrvrrrr#__all__r(r(r(r)sX      #v[+"hr n