o Zh@sdZddlZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZmZmZmZmZdd lmZdd lmZmZmZdd lmZmZd d lmZe e!Z"Gdddej#Z$Gdddej#Z%Gdddej#Z&Gdddej#Z'Gdddej#Z(Gdddej#Z)Gdddej#Z*Gdddej#Z+Gdddej#Z,Gd d!d!ej#Z-eGd"d#d#eZ.eGd$d%d%e.Z/ed&d'Gd(d)d)e.Z0eGd*d+d+e.Z1Gd,d-d-ej#Z2eGd.d/d/e.Z3gd0Z4dS)1zPyTorch LiLT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) LiltConfigcs>eZdZfddZ    d ddZddZdd ZZS) LiltTextEmbeddingscsttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|jdt|jdddt|dd|_|j|_tj|j|j|jd|_ dS) N padding_idxZeps position_ids)rF) persistentposition_embedding_typeabsolute)super__init__r EmbeddingZ vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutZregister_buffertorcharangeexpandgetattrrrselfconfig __class__U/var/www/auris/lib/python3.10/site-packages/transformers/models/lilt/modeling_lilt.pyr"+s   zLiltTextEmbeddings.__init__Nc Cs|dur|dur|||j|j}n||}|dur"|}n|dd}|dur9tj|tj|j jd}|durB| |}| |}||}|j dkrY| |}||7}||}||}||fS)Nrdtypedevicer )"create_position_ids_from_input_idsrtor<&create_position_ids_from_inputs_embedssizer/zeroslongrr&r)rr(r*r.) r4 input_idstoken_type_idsr inputs_embeds input_shaper) embeddingsr(r8r8r9forwardBs*        zLiltTextEmbeddings.forwardcCs2||}tj|dd||}||S)a Args: Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. x: torch.Tensor x: Returns: torch.Tensor rdim)neintr/ZcumsumZtype_asrB)r4rCrmaskZincremental_indicesr8r8r9r=fs  z5LiltTextEmbeddings.create_position_ids_from_input_idscCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z Args: We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids.: inputs_embeds: torch.Tensor Returns: torch.Tensor Nrrr:r)r@r/r0rrBr<Z unsqueezer1)r4rErFZsequence_lengthrr8r8r9r?ss z9LiltTextEmbeddings.create_position_ids_from_inputs_embeds)NNNN)__name__ __module__ __qualname__r"rHr=r? __classcell__r8r8r6r9r*s  $ rcs&eZdZfddZdddZZS)LiltLayoutEmbeddingscstt|j|jd|_t|j|jd|_t|j|jd|_t|j|jd|_ |j |_ tj|j |j|j |j d|_tj|j|j|j d|_tj|j|j |jd|_t|j|_dS)Nr)Z in_featuresZ out_featuresr)r!r"rr#Zmax_2d_position_embeddingsr$x_position_embeddingsy_position_embeddingsh_position_embeddingsw_position_embeddingsr%rr'channel_shrink_ratiobox_position_embeddingsLinearbox_linear_embeddingsr*r+r,r-r.r3r6r8r9r"s  zLiltLayoutEmbeddings.__init__Nc CsJz:||dddddf}||dddddf}||dddddf}||dddddf}WntyK}ztd|d}~ww||dddddf|dddddf}||dddddf|dddddf} tj|||||| gdd} || } ||} | | } | | } | | } | S)Nrrr z;The `bbox` coordinate values should be within 0-1000 range.rrI) rTrU IndexErrorrVrWr/catr[rYr*r.) r4bboxrZleft_position_embeddingsZupper_position_embeddingsZright_position_embeddingsZlower_position_embeddingserVrWZspatial_position_embeddingsrYr8r8r9rHs6  22   zLiltLayoutEmbeddings.forward)NN)rNrOrPr"rHrQr8r8r6r9rRs rRcs8eZdZd fdd Zd ddZ   d dd ZZS) LiltSelfAttentionNcsXt|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|j|j|j|_t |j|j|j|j|_t |j|j|j|j|_t |j|_|pt|dd|_|jdks|jd kr|j|_t d |jd |j|_|j|_dS) NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rr relative_keyrelative_key_queryr\r)r!r"r$num_attention_headshasattr ValueErrorrLattention_head_size all_head_sizerrZquerykeyvaluerX layout_query layout_key layout_valuer,Zattention_probs_dropout_probr.r2rr'r#distance_embedding)r4r5rr6r8r9r"s<   zLiltSelfAttention.__init__rcCs:|dd|j|j|f}|j|}|ddddS)Nrrr\rr )r@rerhviewpermute)r4xrZ new_x_shaper8r8r9transpose_for_scoress  z&LiltSelfAttention.transpose_for_scoresFcCs|j|||jd}|j|||jd}|j|||jd}||} |||} |||} || } t | | dd} t || dd}|j dksY|j dkr| d}tj |tj|jddd}tj |tj|jddd}||}|||jd}|j| jd}|j dkrtd | |}| |} n|j dkrtd | |}td | |}| ||} | t|j}|t|j|j}||} ||}|dur||}tjdd |}||}|dur||}t ||}|d d dd}| dd|j|jf}|j|}|dur| |} tjdd | }||}|dur5||}t || }|d d dd}| dd|jf}|j|}|ra||f|f}|S||ff}|S)N)rtrrcrdrr:)r;zbhld,lrd->bhlrzbhrd,lrd->bhlrrIrr\r )rurorXrnrmrjrkrlr/matmulZ transposerr@r0rBr<rqrpr'r>r;ZeinsummathsqrtrhrZSoftmaxr.rr contiguousri)r4 hidden_states layout_inputsattention_mask head_maskoutput_attentionsZlayout_value_layerZlayout_key_layerZlayout_query_layerZmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZlayout_attention_scores seq_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZtmp_attention_scoresZtmp_layout_attention_scoresZlayout_attention_probsZlayout_context_layerZnew_context_layer_shapeZattention_probsZ context_layeroutputsr8r8r9rHsp                zLiltSelfAttention.forwardN)rNNF)rNrOrPr"rurHrQr8r8r6r9ras $ rac8eZdZfddZdejdejdejfddZZS)LiltSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr) r!r"rrZr$denser*r+r,r-r.r3r6r8r9r"F zLiltSelfOutput.__init__r{ input_tensorreturncC&||}||}|||}|Srrr.r*r4r{rr8r8r9rHL  zLiltSelfOutput.forwardrNrOrPr"r/TensorrHrQr8r8r6r9rE $rcsjeZdZdfdd ZddZ   ddejdejd eejd eejd ee d e ejf d dZ Z S) LiltAttentionNcsRtt||d|_t||_t|_|j}|j|j |_t||_ ||_dS)N)r) r!r"rar4routputset pruned_headsr$rX layout_output)r4r5rori_hidden_sizer6r8r9r"Ts    zLiltAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrI)lenrr4rerhrrrjrkrlrrriunion)r4headsindexr8r8r9 prune_heads`s zLiltAttention.prune_headsFr{r|r}r~rrc CsT||||||}||dd|}||dd|}||ff|dd} | S)Nrr)r4rr) r4r{r|r}r~rZ self_outputsattention_outputlayout_attention_outputrr8r8r9rHrszLiltAttention.forwardrr) rNrOrPr"rr/rr FloatTensorboolrrHrQr8r8r6r9rSs& rc2eZdZfddZdejdejfddZZS)LiltIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r!r"rrZr$intermediate_sizer isinstanceZ hidden_actstrr intermediate_act_fnr3r6r8r9r"s   zLiltIntermediate.__init__r{rcCs||}||}|Sr)rr)r4r{r8r8r9rHs  zLiltIntermediate.forwardrr8r8r6r9rs rcr) LiltOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r!r"rrZrr$rr*r+r,r-r.r3r6r8r9r"rzLiltOutput.__init__r{rrcCrrrrr8r8r9rHrzLiltOutput.forwardrr8r8r6r9rrrcspeZdZfddZ   ddejdejdeejdeejd eed e ejf d d Z d dZ ddZ Z S) LiltLayercst|j|_d|_t||_t||_t||_ |j }|j }|j |j |_ |j |j |_ t||_ t||_||_ ||_ dS)Nr)r!r"chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr$rrXlayout_intermediater)r4r5rZori_intermediate_sizer6r8r9r"s       zLiltLayer.__init__NFr{r|r}r~rrc Csr|j|||||d}|dd}|dd}|dd} t|j|j|j|} t|j|j|j|} | | ff| } | S)N)rrr)rrfeed_forward_chunkrrlayout_feed_forward_chunk) r4r{r|r}r~rZself_attention_outputsrrr layer_outputZlayout_layer_outputr8r8r9rHs$   zLiltLayer.forwardcC||}|||}|Sr)rrr4rZintermediate_outputrr8r8r9r  zLiltLayer.feed_forward_chunkcCrr)rrrr8r8r9rrz#LiltLayer.layout_feed_forward_chunkr)rNrOrPr"r/rrrrrrHrrrQr8r8r6r9rs(  rcs|eZdZfddZ     ddejdejdeejd eejd eed eed eed e e eje ffddZ Z S) LiltEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr8)r).0_r5r8r9 sz(LiltEncoder.__init__..F) r!r"r5rZ ModuleListrangenum_hidden_layerslayergradient_checkpointingr3r6rr9r"s   zLiltEncoder.__init__NFTr{r|r}r~routput_hidden_states return_dictrc Cs|rdnd}|r dnd} t|jD]D\} } |r||f}|dur$|| nd} |jr8|jr8|| j|||| |} n| |||| |} | dd}| dd}|rU| | df} q|r]||f}|sktdd||| fDSt||| dS)Nr8rrcss|] }|dur|VqdSrr8)rvr8r8r9 sz&LiltEncoder.forward..)last_hidden_stater{ attentions) enumeraterrZtrainingZ_gradient_checkpointing_func__call__tupler )r4r{r|r}r~rrrZall_hidden_statesZall_self_attentionsiZ layer_moduleZlayer_head_maskZ layer_outputsr8r8r9rHsR        zLiltEncoder.forward)NNFFT)rNrOrPr"r/rrrrrrr rHrQr8r8r6r9rs0   rcr) LiltPoolercs*tt|j|j|_t|_dSr)r!r"rrZr$rZTanh activationr3r6r8r9r"+s zLiltPooler.__init__r{rcCs(|dddf}||}||}|SNr)rr)r4r{Zfirst_token_tensor pooled_outputr8r8r9rH0s  zLiltPooler.forwardrr8r8r6r9r*s rc@s$eZdZeZdZdZgZddZdS)LiltPreTrainedModelliltTcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsg)meanZstdNg?)rrrZweightdataZnormal_r5Zinitializer_rangeZbiasZzero_r#rr*Zfill_)r4moduler8r8r9 _init_weights@s      z!LiltPreTrainedModel._init_weightsN) rNrOrPrZ config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesrr8r8r8r9r9s  rcseZdZdfdd ZddZddZdd Ze dd ee j d ee j d ee j dee j dee j dee j dee j dee dee dee de e e j effddZZS) LiltModelTcsNt|||_t||_t||_t||_|rt |nd|_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r!r"r5rrGrRlayout_embeddingsrencoderrpooler post_init)r4r5add_pooling_layerr6r8r9r"Ss     zLiltModel.__init__cCs|jjSrrGr&)r4r8r8r9get_input_embeddingsdszLiltModel.get_input_embeddingscCs ||j_dSrr)r4rlr8r8r9set_input_embeddingsgs zLiltModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r4Zheads_to_prunerrr8r8r9 _prune_headsjszLiltModel._prune_headsNrCr_r}rDrr~rErrrrc  Cs|dur|n|jj}| dur| n|jj} | dur| n|jj} |dur*|dur*td|dur9||||} n|durF|dd} ntd| \} } |durU|jn|j}|durgtj | dtj |d}|durttj | | f|d}|durt |j dr|j jddd| f}|| | }|}n tj | tj |d}||| }|||jj}|j ||||d \}}|j||d }|j|||||| | d }|d }|jdur||nd}| s||f|d dSt|||j|jdS)a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, AutoModel >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModel.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train", trust_remote_code=True) >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embeds)r:)r<rD)rCrrDrE)r_r)r}r~rrrrr)rZ pooler_outputr{r)r5rruse_return_dictrgZ%warn_if_padding_and_no_attention_maskr@r<r/rArBZonesrfrGrDr1Zget_extended_attention_maskZ get_head_maskrrrrr r{r)r4rCr_r}rDrr~rErrrrFZ batch_sizerr<Zbuffered_token_type_idsZ buffered_token_type_ids_expandedZextended_attention_maskZembedding_outputZlayout_embedding_outputZencoder_outputssequence_outputrr8r8r9rHrsh(       zLiltModel.forward)T) NNNNNNNNNN)rNrOrPr"rrrrrr/rrrrr rHrQr8r8r6r9rQsN    rz LiLT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. )Z custom_introcseZdZfddZe           ddeejdeejdeej deejdeejd eej d eej d eejd ee d ee dee de e eje ffddZZS)LiltForSequenceClassificationcs>t||j|_||_t|dd|_t||_|dSNF)r) r!r" num_labelsr5rrLiltClassificationHead classifierrr3r6r8r9r"s   z&LiltForSequenceClassification.__init__NrCr_r}rDrr~rElabelsrrrrc  Csv| dur| n|jj} |j|||||||| | | d } | d} || }d}|dur||j}|jjdurX|jdkr>d|j_n|jdkrT|jt j ksO|jt j krTd|j_nd|j_|jjdkrvt }|jdkrp|| | }n+|||}n%|jjdkrt}||d|j|d}n|jjdkrt}|||}| s|f| d d}|dur|f|S|St||| j| jd S) a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForSequenceClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForSequenceClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train", trust_remote_code=True) >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_idx = outputs.logits.argmax(-1).item() >>> predicted_class = model.config.id2label[predicted_class_idx] ```N r_r}rDrr~rErrrrrZ regressionZsingle_label_classificationZmulti_label_classificationrr\losslogitsr{r)r5rrrr>r<Z problem_typerr;r/rBrLrsqueezerrqrrr{rr4rCr_r}rDrr~rErrrrrrrrloss_fctrr8r8r9rHsX.     "       z%LiltForSequenceClassification.forward NNNNNNNNNNN)rNrOrPr"rrr/ LongTensorrrrrrrrHrQr8r8r6r9rsN       rcseZdZfddZe           ddeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee de e ej e ffddZZS)LiltForTokenClassificationcsbt||j|_t|dd|_|jdur|jn|j}t||_ t |j |j|_ | dSr)r!r"rrrclassifier_dropoutr-rr,r.rZr$rrr4r5rr6r8r9r"`s   z#LiltForTokenClassification.__init__NrCr_r}rDrr~rErrrrrc  Cs| dur| n|jj} |j|||||||| | | d } | d} || } || }d}|durC||j}t}||d|j |d}| sY|f| dd}|durW|f|S|St ||| j | j dS)a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForTokenClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForTokenClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train", trust_remote_code=True) >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_indices = outputs.logits.argmax(-1) ```Nrrrr\r) r5rrr.rr>r<rrqrrr{rrr8r8r9rHns<+   z"LiltForTokenClassification.forwardr)rNrOrPr"rrr/rrrrrrrrHrQr8r8r6r9r]sN      rcs(eZdZdZfddZddZZS)rz-Head for sentence-level classification tasks.csTtt|j|j|_|jdur|jn|j}t||_ t|j|j |_ dSr) r!r"rrZr$rrr-r,r.rout_projrr6r8r9r"s  zLiltClassificationHead.__init__cKsL|dddddf}||}||}t|}||}||}|Sr)r.rr/tanhr)r4featureskwargsrsr8r8r9rHs     zLiltClassificationHead.forward)rNrOrP__doc__r"rHrQr8r8r6r9rs  rcseZdZfddZe            ddeejdeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee de e ej e ffddZZS)LiltForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSr) r!r"rrrrrZr$ qa_outputsrr3r6r8r9r"s  z!LiltForQuestionAnswering.__init__NrCr_r}rDrr~rEstart_positions end_positionsrrrrc  CsJ| dur| n|jj} |j|||||||| | | d } | d}||}|jddd\}}|d}|d}d}|dur| durt|dkrP|d}t| dkr]| d} |d}| d|}| d|} t |d}|||}||| }||d}| s||f| dd}|dur|f|S|St |||| j | j d S) a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForQuestionAnswering >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForQuestionAnswering.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train", trust_remote_code=True) >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> predicted_answer = tokenizer.decode(predict_answer_tokens) ```NrrrrrI)Z ignore_indexr\)r start_logits end_logitsr{r)r5rrrsplitrrzrr@clamprr r{r)r4rCr_r}rDrr~rErrrrrrrrrrZ total_lossZ ignored_indexrZ start_lossZend_lossrr8r8r9rHsR/          z LiltForQuestionAnswering.forward) NNNNNNNNNNNN)rNrOrPr"rrr/rrrrrrr rHrQr8r8r6r9rsT       r)rrrrr)5rrxtypingrrrr/Ztorch.utils.checkpointrZtorch.nnrrrZ activationsr Zmodeling_outputsr r r rrZmodeling_utilsrZ pytorch_utilsrrrutilsrrZconfiguration_liltrZ get_loggerrNloggerModulerrRrarrrrrrrrrrrrr__all__r8r8r8r9sP     Y8 5<Hqc o