eTh  SSKrSSKrSSKrSSKJr SSKJr SSKJr SSK J r J r J r J r JrJrJrJr SSKrSSKJrJr SSKJr SS KJr \"S \ 5r\"S \ \ \/\ \\ 445r"S S 5rSrS.S\ \S\ \\ 44Sjjr \"SS\55r!S\ \S\ \\ 44Sjr"S\ \S\ \\ 44Sjr#S\ \S\ \\ 44Sjr$\"SS55r%\"SS\55r&S/S\\'4Sjjr(S/S\\'4Sjjr)S/S\\'4Sjjr*\"SS \55r+\"S!S"55r,\"S#S$\55r-\"S%S&\-55r.S'r/\"S(S)\-55r0\"S*S+\55r1\"S,S-\!55r2g)0N)Mapping) dataclass)randint)AnyCallableDictListNewTypeOptionalTupleUnion) BertTokenizerBertTokenizerFast)PreTrainedTokenizerBase)PaddingStrategyInputDataClass DataCollatorc\rSrSrSSjrSrg)DataCollatorMixin'NcUc URnUS:XaURU5$US:XaURU5$US:XaURU5$[ SUS35e)Ntfptnpz Framework 'z' not recognized!)return_tensorstf_call torch_call numpy_call ValueErrorselffeaturesrs W/var/www/auris/envauris/lib/python3.13/site-packages/transformers/data/data_collator.py__call__DataCollatorMixin.__call__(sn  !!00N T !<<) ) t #??8, , t #??8, ,{>*::KLM MN)__name__ __module__ __qualname____firstlineno__r%__static_attributes__r(r'r$rr's Nr'rc [US5(dUR"U0UD6$URRSS5nSURS'UR"U0UD6nX0RS'U$!X0RS'f=f)zr Pads without triggering the warning about how using the pad function is sub-optimal when using a fast tokenizer. deprecation_warningszAsking-to-pad-a-fast-tokenizerFT)hasattrpadr0get) tokenizerpad_args pad_kwargs warning_statepaddeds r$"pad_without_fast_tokenizer_warningr95s 94 5 5}}h5*5522667WY^_MGKI""#CDY7J7LY&&'GH MLY&&'GHs A22Br#returncjUS:Xa [U5$US:Xa [U5$US:Xa [U5$g)a Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named: - `label`: handles a single value (int or float) per object - `label_ids`: handles a list of values per object Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it's useful. rrrN)torch_default_data_collatortf_default_data_collatornumpy_default_data_collator)r#rs r$default_data_collatorr?KsB"*844 4 '11 4 *844 r'cX\rSrSr%SrSr\\S'S S\\ \\ 4S\ \\ 44Sjjr S r g) DefaultDataCollatorda Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named: - `label`: handles a single value (int or float) per object - `label_ids`: handles a list of values per object Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it's useful. This is an object (like other data collators) rather than a pure function like default_data_collator. This can be helpful if you need to set a return_tensors value at initialization. Args: return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". rrNr#r:c6Uc URn[X5$r))rr?r!s r$r%DefaultDataCollator.__call__zs  !!00N$X>>r'r(r)) r*r+r,r-__doc__rstr__annotations__r rrr%r.r(r'r$rArAdsC$NC?d38n!5?tTWY\T\~??r'rAc JSSKn[US[5(dUVs/sHn[U5PM nnUSn0nSU;aUSb[USUR5(aUSR 5OUSn[U[ 5(a URO URnURUVs/sHo"SPM snUS9US'OSU;aUSb[USUR5(a(URUVs/sHo"SPM sn5US'OY[USS[ 5(a URO URnURUVs/sHo"SPM snUS9US'UR5HupxUS;dM UcM[U[5(aM)[XR5(a(URUVs/sHo"UPM sn5XG'Mk[U[R5(a<UR[R"UVs/sHo"UPM sn55XG'MURUVs/sHo"UPM sn5XG'M U$s snfs snfs snfs snfs snfs snfs snfNrlabeldtypelabels label_ids)rJrN)torch isinstancervarsTensoritemintlongfloattensorstackitemsrFrndarray from_numpy) r#rOffirstbatchrJrLkvs r$r<r<s? hqk7 + +%-.XDGX. QKE E %E'N6)3E'NELL)Q)Qg##%W\]dWe(44 %++,,H'EHq' H'EU,Sh  % "4"@ eK(%,, 7 7#kk8*L8a[>8*LME(O",U;-?-BC"H"HEJJekkE#llH+MHqkNH+MUZl[E(O  * *q}ZPQSVEWEW!\\** ;;h'?h!h'?@Arzz** ++BHHH5MHqdH5M,NO <k>k " e.C.I.I2:: V VBHH\^\f\fE -t} = = *5+@+CS I IBHHrzzE *5+@# F FBHHBJJE..8/T8a.0A8/T\a.bh  4 4zZ[]`OaOa!ii455888$<8aqT8$<=//x0Hx!1x0HI  LE/00U %=0HsJ3%J8.J= K c[US[5(dUVs/sHn[U5PM nnUSn0nSU;aUSb[US[R5(aUSR 5OUSn[U[ 5(a[RO[Rn[R"UVs/sHoSPM snUS9US'OSU;aUSb[US[R5(a-[R"UVs/sHoSPM sn5US'Of[USS[ 5(a[RO[Rn[R"UVs/sHoSPM snUS9US'UR5HupgUS;dM UcM[U[5(aM)[U[R5(a-[R"UVs/sHoUPM sn5X6'Mu[R"UVs/sHoUPM sn5X6'M U$s snfs snfs snfs snfs snfs snfrI) rPrrQrrZrSrTrdrearrayrXrYrF)r#r\r]r^rJrLr_r`s r$r>r>s hqk7 + +%-.XDGX. QKE E %E'N6)3E'NBJJ)O)Og##%UZ[bUc&uc22 ((#A1gJ#AOh  % "4"@ eK("** 5 5 hh'I1+'IJE(O *5+=a+@# F FBHHBJJE hh'I1+'IQVWE(O  * *q}ZPQSVEWEW!RZZ((888$<8aqT8$<=888$<8aqT8$<=  L7/$B(J(J%=$= 7.0 (Volta). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". r4TpaddingN max_lengthpad_to_multiple_ofrrr#r:c [URUURURURUR S9nSU;a USUS'US SU;a USUS'US U$)NrrrsrtrrJrMrN)r9r4rrrsrtr)r"r#r^s r$r% DataCollatorWithPadding.__call__sw2 NN LL#66..   e #GnE(Og % #K0E(Ok" r'r()r*r+r,r-rErrGrrr boolrFrrsr rTrtrr rrr%r.r(r'r$rprpst4'&15GU4o- .5 $J $(, ,NCd38n!5$sCx.r'rpc\rSrSr%Sr\\S'Sr\\ \ \ 4\S'Sr \ \\S'Sr\ \\S'S r\\S 'S r\ \S 'S rSrSrSrg)"DataCollatorForTokenClassificationi!a' Data collator that will dynamically pad the inputs received, as well as the labels. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.0 (Volta). label_pad_token_id (`int`, *optional*, defaults to -100): The id to use when padding the labels (-100 will be automatically ignore by PyTorch loss functions). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". r4TrrNrsrtlabel_pad_token_idrrc N^ SSKm SUSR5;aSOSnX!SR5;aUVs/sHo3UPM snOSnUVVVs/sH.o3R5VVs0sHupVXR:wdM XV_M snnPM0 nnnn[URUUR UR URSS9nUcU$USRSn URRn U 4Sjn U S :Xa8UV s/sH'o"U 5UR/U [U 5- --PM) sn X'O7UV s/sH'oR/U [U 5- -U "U 5-PM) sn X'T RXT RS 9X'U$s snfs snnfs snnnfs sn fs sn f) NrrJrMrrv input_idscp>[UTR5(aUR5$[U5$r))rPrRtolistrj)tensor_or_iterablerOs r$to_list>DataCollatorForTokenClassification.torch_call..to_list]s0,ell;;)0022*+ +r'rightrK)rOkeysrYr9r4rrrsrtshape padding_sider|lenrWrd)r"r# label_namefeaturerMr_r`no_labels_featuresr^sequence_lengthrrrJrOs @r$r-DataCollatorForTokenClassification.torch_callFs '8A;+;+;+= =W8 BLYZP[P`P`PbBbX>X'*%X>hlcklckX_ S1?tqtSckl2 NN LL#66   >L ,2215~~22  , 7 "io!io`e$"9"9!:oPSTYPZ>Z![[io!E  jp!io`e(()_s5z-IJWUZ^[io!E "LL):%++LN E?Sl.!!s/F F, F;FF.F8.F"Fc (SSKnSUSR5;aSOSnX1SR5;aUVs/sHoDUPM snOSn[URUURUR UR UcSOSS9nUcU$URUS5RSnURRnUS:Xa=UV s/sH+n [U 5UR/U[U 5- --PM- sn US'O;UV s/sH*oR/U[U 5- -[U 5-PM, sn US'UR5V V s0sHupXRXRS 9_M nn n U$s snfs sn fs sn fs sn n f) NrrJrMrrvr~rrrK)rbrr9r4rrrsrtrkrrrjr|rrYrd) r"r#rrrrMr^rrrJr_r`s r$r*DataCollatorForTokenClassification.tf_callns '8A;+;+;+= =W8 BLYZP[P`P`PbBbX>X'*%X>hl2 NN LL#66#)>4t  >L..u[/ABHHK~~22 7 "flfl]bU t667?SQVZ;WXXflE(O gmfl]b(()_s5z-IJTRW[XflE(OIN V (((((;; V 5?"WsE?2F 1F %Fc @SUSR5;aSOSnX!SR5;aUVs/sHo3UPM snOSn[URUURURUR UcSOSS9nUcU$[ R"US5RSnURRnUS:Xa=UVs/sH+n[U5UR/U[U5- --PM- snUS'O;UVs/sH*oR/U[U5- -[U5-PM, snUS'UR5V V s0sH)upU [ R"U [ RS 9_M+ nn n U$s snfs snfs snfs sn n f) NrJrrMrrvr~rrrK)rr9r4rrrsrtrrnrrrjr|rrYrd) r"r#rrrMr^rrrJr_r`s r$r-DataCollatorForTokenClassification.numpy_calls '8A;+;+;+= =W8 BLYZP[P`P`PbBbX>X'*%X>hl2 NN LL#66#)>4t  >L((5#56<# UHoRS5T:Hv M g7frNsize.0xlength_of_firsts r$ '_torch_collate_batch..s!Q1&&)">s #)dimCYou are attempting to pad samples but the tokenizer you are using () does not have a pad token.c3B# UHoRS5v M g7frrrrs r$rrs11VVAYYsrr)rOrPrjrirrZrWrUrallrRrX pad_tokenr __class__r*maxnew_fullr pad_token_id enumeraterr) examplesr4rtrOeare_tensors_same_lengthrsresultiexamplers @r$_torch_collate_batchrs(1+eRZZ899?GHx!LL**L5xHqk&&q)O"!Q!QQ$6$>/TfBfjkBk(LL11;;xQ;/ /" $$--..J L  111J%:+Ja+O!71<@RR a[ ! !3x=*"=y?U?U VF)   ! !W ,,31( a((( )-41}}Q'')) * * M9Is"Fc >^ SSKn[US[[45(a(UVs/sHoCR XCR S9PM nn[ US5m [U 4SjU55nU(aUb T U-S:XaURUSS9$URc#[SURRS35e[SU55nUbXb-S:wa Xb-S-U-n/nURUS5n[R "US 4[R"S9n UH_n UR$S :XaU[ U 5- U S 'OU[ U 5- U S 'UR'UR)XUR*S 95 Ma URUSS9$s snf)NrrKc3@># UHn[U5T:Hv M g7fr)rrs r$r$_tf_collate_batch..!NX#a&O";Xaxisrrc38# UHn[U5v M g7fr)rrs r$rr.XSVVXrrr)rr)rr)constant_values)rbrPrjrirkrdrrrXrr rr*rrankrzerosint32rappendr2r) rr4rtrrrrsrrpaddingsrrs @r$_tf_collate_batchrsi(1+e}--EMNX(((((;XN(1+&O!!NX!NN$6$>/TfBfjkBkxxqx))" $$--..J L  .X..J%:+Ja+O!71<@RR F 778A; Dxxq 2H  ! !W ,'#g,6HTN'#g,6HTN bffW @V@VfWX  88F8 ##;Os"Fc^ [US[[45(a3UVs/sH&n[R"U[R S9PM( nn[ US5m [U 4SjU55nU(a!Ub T U-S:Xa[R"USS9$URc#[SURRS35e[SU55nUbXR-S:wa XR-S -U-n[R"[ U5U4URUSR S 9n[#U5HAupxUR$S :XaXUSUR&S24'M,XXxR&S*S24'MC U$s snf) rrrKc3@># UHn[U5T:Hv M g7fr)rrs r$r'_numpy_collate_batch..rrNrrrc38# UHn[U5v M g7fr)rrs r$rr rrr)r fill_valuerLr)rPrjrirrnrdrrrXrr rr*rfullrrLrrr) rr4rtrrrsrrrrs @r$_numpy_collate_batchrs(1+e}--9ABABHHQbhh/B(1+&O!!NX!NN$6$>/TfBfjkBkxxq))" $$--..J L  .X..J%:+Ja+O!71<@RR WWCM:69CYCYaijkalarar sF)   ! !W ,,31( a((( )-41}}Q'')) * * M3Cs-E?c\rSrSr%Sr\\S'Sr\\ \ \ 4\S'Sr \ \\S'Sr\ \\S'S r\ \S 'S \\\ \44S jrS rSrg)DataCollatorForMultipleChoiceia Data collator that dynamically pads a batch of nested examples for multiple choice, so that all choices of all examples have the same length. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences according to the model's padding side and padding index among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): Pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". r4TrrNrsrtrrrc ^ SSKnSUSR5;aSOSnUVs/sHoDRU5PM nn[U5n[USS5m [ U 4SjU5/S9nUR R UURURURSS9nUR5V V s0sHupXRUT S 5_M nn n URXRRS 9US'U$s snfs sn n f) NrrJrMr~c 3># UHJn[T5VVVs/sH*o!R5VVs0sH up4X4U_M snnPM, snnnv ML gs snnfs snnnf7fr))rangerY)rrrr_r` num_choicess r$r;DataCollatorForMultipleChoice.torch_call..IsN i`hU\U;=O P=O==?3?41qA$w?3=O P P`h3 Ps'A"A AA A"A A")startrrvrK)rOrpoprsumr4r2rrrsrtrYviewrWrd) r"rrOrrrM batch_size flat_examplesr^r_r`rs @r$r(DataCollatorForMultipleChoice.torch_call<s!(8A;+;+;+= =W8 9ABg++j)B] (1+k23  i`h iqs "" LL#66 # EJKKMRMDAFF:{B77MR,,v[[,Ah /C*Ss C??DcSSKnSUSR5;aSOSnUVs/sHoDRU5PM nn[U5n[USS5nUVVV V s/sHHn[ U5VV V s/sH*oR 5V V s0sH upXU_M sn n PM, sn n nPMJ n n nnn [ U /5n URRU URURURSS9n U R 5V V s0sHupXRXUS45_M n n n URXRRS9U S'U $s snfs sn n fs sn n nfs sn n nnfs sn n f) NrrJrMr~rrvrrK)rbrrrrrYrr4r2rrrsrtreshaperkrd) r"r#rrrrMrrrr_r`flattened_featuresr^s r$r%DataCollatorForMultipleChoice.tf_callZsX '8A;+;+;+= =W8 9ABg++j)B] (1+k23 _g _gT[E+= 7.0 (Volta). label_pad_token_id (`int`, *optional*, defaults to -100): The id to use when padding the labels (-100 will be automatically ignored by PyTorch loss functions). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". r4NmodelTrrrsrtr{r|rrc Uc URnSUSR5;aSOSnX1SR5;aUVs/sHoDUPM snOSnUb[SU55(aSnUVVVs/sH.oDR5VVs0sHupgXc:wdM Xg_M snnPM0 nnnn[ UR UUR URURUS9n UR SL=(d UR [R:Hn UGb8U (aY[USU[5(a[U5U S'GOUV s/sHn [R"U //5PM sn U S'GOUR [R:H=(a URSLn U (d[!SU55O URn URb+XR-S- UR-UR-n UR R"n[USU[5(aXUV s/sHFn US :XaXR$/U ['U 5- --OUR$/U ['U 5- -U -PMH sn U S'OUV s/sHn US :XaS[R"U [R("UR$/U ['U 5- -[R*S 9/5OR[R"[R("UR$/U ['U 5- -[R*S 9U /5PM sn U S'U R-SS5bUS :Xa%SSKnUR1U SUR*S 9U S'OZUS :Xa%SSKnUR5U SUR*S 9U S'O/[R("U S[R*S 9U S'OSU S'UbIUR6b<[9UR6S 5(a!UR6R;U SS9nUU S'U $s snfs snnfs snnnfs sn fs sn fs sn f)NrJrrMc3(# UHoSLv M g7fr)r()rrJs r$r2DataCollatorForSeq2Seq.__call__..s%HtmsrvFc38# UHn[U5v M g7fr)r)rls r$rrs&>v!s1vvvrrrrKrr%prepare_decoder_input_ids_from_labels)rMdecoder_input_ids)rrrrYr9r4rrrsrtr DO_NOT_PADrPrjr concatenate MAX_LENGTHrrr|rrnrdr3rOrWrbconstantrr1r)r"r#rrrrMr_r`non_labels_featuresr^ no_paddingrJ max_paddingmax_label_lengthrrOrrs r$r%DataCollatorForSeq2Seq.__call__s)  !!00N '8A;+;+;+= =W8 BLYZP[P`P`PbBbX>X'*%X>hl  #%H%H"H"HFdlmdlY`TAOTdlm3 NN LL#66)  \\U*Xdllo>X>X.X  hqk*5t<<&*6lE(OPV&WPVur~~ubk'BPV&WE(O"llo.H.HHhT__dhMh FQ3&>v&>#>W[WfWf **6),C,CCaG223112% $~~:: hqk*5t<< &, '&,E(72!8!8 9=MPSTYPZ=Z [["556:JSQVZ:WX[``a&, 'E(O*&,'&,E(72  % "$*A*A)BFVY\]bYcFc)dlnltlt u ^^ "$*A*A)BFVY\]bYcFc)dlnltlt u % &,'E(O$ 99Xt $ 0%"',,uXekk,"Rh4''"$++eHoRXX+"Nh"$((5?"(("Kh"E(O   & $KLL $ P PX]^fXg P h ):E% & k? Um&'X''s8P=:Q Q!Q'Q"QA QB6QQr(r))r*r+r,r-rErrGrr rrrr rxrFrrsrTrtr|rr%r.r(r'r$rrsslB'&E8C=15GU4o- .5 $J $(, ,""NCZr'rc \rSrSr%Sr\\S'Sr\\S'Sr \ \S'Sr \ \S 'S r \ \S 'S r \\\S 'Sr\\S'Sr\\S'S r\\\S'SrSrSr\S"Sj5rS"S\S\\S\\\44SjjrS\\\\\\\\44S\\\44SjrS\\\\\\\\44S\\\44Sjr S"S\S\\S\\\44Sjjr!S\\\\\\\\44S\\\44Sjr"S"S\S\\S\\\44S jjr#S!r$g )#DataCollatorForLanguageModelingia Data collator used for language modeling. Inputs are dynamically padded to the maximum length of a batch if they are not all of the same length. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. mlm (`bool`, *optional*, defaults to `True`): Whether or not to use masked language modeling. If set to `False`, the labels are the same as the inputs with the padding tokens ignored (by setting them to -100). Otherwise, the labels are -100 for non-masked tokens and the value to predict for the masked token. mlm_probability (`float`, *optional*, defaults to 0.15): The probability with which to (randomly) mask tokens in the input, when `mlm` is set to `True`. mask_replace_prob (`float`, *optional*, defaults to 0.8): The probability with which masked tokens are replaced by the tokenizer's mask token (e.g., `[MASK]`). Defaults to 0.8, meaning 80% of the masked tokens will be replaced with `[MASK]`. Only works when `mlm` is set to `True`. random_replace_prob (`float`, *optional*, defaults to 0.1): The probability with which masked tokens are replaced by random tokens from the tokenizer's vocabulary. Defaults to 0.1, meaning 10% of the masked tokens will be replaced with random tokens. The remaining masked tokens (1 - mask_replace_prob - random_replace_prob) are left unchanged. Only works when `mlm` is set to `True`. pad_to_multiple_of (`int`, *optional*): If set, will pad the sequence to a multiple of the provided value. return_tensors (`str`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". seed (`int`, *optional*): The seed to use for the random number generator for masking. If not provided, the global RNG will be used. For best performance, this data collator should be used with a dataset having items that are dictionaries or BatchEncoding, with the `"special_tokens_mask"` key, as returned by a [`PreTrainedTokenizer`] or a [`PreTrainedTokenizerFast`] with the argument `return_special_tokens_mask=True`. 1. Default Behavior: - `mask_replace_prob=0.8`, `random_replace_prob=0.1`. - Expect 80% of masked tokens replaced with `[MASK]`, 10% replaced with random tokens, and 10% left unchanged. 2. All masked tokens replaced by `[MASK]`: - `mask_replace_prob=1.0`, `random_replace_prob=0.0`. - Expect all masked tokens to be replaced with `[MASK]`. No tokens are left unchanged or replaced with random tokens. 3. No `[MASK]` replacement, only random tokens: - `mask_replace_prob=0.0`, `random_replace_prob=1.0`. - Expect all masked tokens to be replaced with random tokens. No `[MASK]` replacements or unchanged tokens. 4. Balanced replacement: - `mask_replace_prob=0.5`, `random_replace_prob=0.4`. - Expect 50% of masked tokens replaced with `[MASK]`, 40% replaced with random tokens, and 10% left unchanged. Note: The sum of `mask_replace_prob` and `random_replace_prob` must not exceed 1. If their sum is less than 1, the remaining proportion will consist of masked tokens left unchanged. r4Tmlmg333333?mlm_probability皙?mask_replace_probg?random_replace_probNrtFtf_experimental_compilerrseedcUR(a"URRc [S5eURS:dURS:a [S5eUR UR -S:a [S5eUR S:dUR S:a [S5eUR S:dUR S:a [S5e[UR5Ul[UR 5Ul[UR 5UlUR(a#SSK nURURSS 9Ul SUl g) NzThis tokenizer does not have a mask token which is necessary for masked language modeling. You should pass `mlm=False` to train on causal language modeling instead.rrz*mlm_probability should be between 0 and 1.zHThe sum of mask_replace_prob and random_replace_prob should not exceed 1z,mask_replace_prob should be between 0 and 1.z.random_replace_prob should be between 0 and 1.T) jit_compile) rr4 mask_tokenr rrrrVrrbfunctiontf_mask_tokens generator)r"rs r$ __post_init__-DataCollatorForLanguageModeling.__post_init__Cs. 88119\    ! #t';';a'?IJ J  ! !D$<$< !T333&*%=%=%N"   kt~~ V    >>>>11+X^XdXd1eL99,,[SYS_S_,`L.?~r'rcSSKnUR(aURcUR5 [ US[ 5(a![ URUSURS9nO S[XRURS90nURSS5nUR(aUc}USR5R5Vs/sHnURRUSS9PM nnURUR!XBR"S 9UR$5nOURXBR$5nUR'URUSUR"5UURR([+UR5S 9uUS'US 'U$USnURR,b*UR/X`RR,:HS U5nOUR1U5nXcS 'U$s snf) Nrrrrtr~rtr!Talready_has_special_tokensrK)r!r(r'rMr{)rbrrrrPrr9r4rtrrrrrget_special_tokens_maskrrkrdrxrr(rrr&identity)r"rrr^r!valrMs r$r'DataCollatorForLanguageModeling.tf_calls 99/ OO  hqk7 + +6RVRiRiE .x\`\s\stE $ii(=tD 88"* %[1779@@B'BNN::3[_:`B$' ')ggb.B.BCV^f^f.B.gikipip&q#&(gg.A77&K#262E2Ek*BHH5$7"nn::t~~. 3F3 /E+ h ;'F~~**6&NN,G,G"GvVV,$(O -'s$Hc(UR(aURcUR5 [US[5(a![ UR USURS9nO S[XR URS90nURSS5nUR(aURUSUS9uUS'US'U$USR5nUR RbS XDUR R:H'XBS'U$) Nrrr2r~r3r!r!rMr{)rrrrPrr9r4rtrrrtorch_mask_tokensclonerr"rr^r!rMs r$r*DataCollatorForLanguageModeling.torch_calls  99/ OO  hqk7 + +6RVRiRiE 1(NN_c_v_vwE $ii(=tD 88262H2Hk"8K3I3 /E+ h ;'--/F~~**6@D!>#G#GHaHa#b  ! !Q &$*B*Ba*G> !T333&*%=%=%N" OOEJJv||5OP\`\j\jO k p p r    }}S%8&,,ejjdhdrdr}s !-!=~K#s#H1c4UR(aURcUR5 [US[5(a![ UR USURS9nO S[XR URS90nURSS5nUR(aURUSUS9uUS'US'U$[R"US5nUR RbS XDUR R:H'XBS'U$) Nrrr2r~r3r!r;rMr{)rrrrPrr9r4rtrrrnumpy_mask_tokensrcopyrr>s r$r*DataCollatorForLanguageModeling.numpy_call;s  99/ OO  hqk7 + +6RVRiRiE 1(NN_c_v_vwE $ii(=tD 88262H2Hk"8K3I3 /E+ h WWU;/0F~~**6@D!d"K "5"<">!^^44Q8JQiQi4jqqrvwNYY//3ELdLd/ellmqrN"& >>''4+A+A 'U\\]abess   ""1d&<&<6<<"PWWX\]`nn $(>>#?#?  ! !Q &$*B*Ba*G> !T333&*%=%=%N" >>''+ELL'Y``aef !##$   >>22C/b6F6F~6V^`^f^f3L  ""1&@||"T[[\`a !##$  99,,C/b6F6F~6V^`^f^f-L".~~o#s#M0)rrrrrr))%r*r+r,r-rErrGrrxrrVrrrtr rTrrrFrrr r staticmethodrrr rr r rrrr<rrLr.r(r'r$rrs:x'&C!OU!"u"!$$(, ,$)T)NCD(3-4 /L0 W W\`--KSTW=- sCx-^,U49c4S>+I%J K,PTUXZ]U]P^,\4d3id38n.L(M#NSWX[]`X`Sa This collator relies on details of the implementation of subword tokenization by [`BertTokenizer`], specifically that subword tokens are prefixed with *##*. For tokenizers that do not adhere to this scheme, this collator will produce an output that is roughly equivalent to [`.DataCollatorForLanguageModeling`]. rr:cUR(aURcUR5 [US[5(aUVs/sHo"SPM nnOUnUVs/sHnSU0PM nn[ X0R URS9n/nUHn/n[US5H/nUR RU5nURU5 M1 SU;a>[US5n [US5n [U 5Hn X;dM SXk-Xk'M URURU55 M [ XPR URS9n URXL5upXS.$s snfs snfNrr~r3 chinese_ref##r~rM)rrrrPrrr4rtr_convert_id_to_tokenrrr_whole_word_maskr<r"rrr~ batch_input mask_labels ref_tokensrtokenref_poslen_seqr batch_maskr rMs r$r'DataCollatorForWholeWordMask.torch_callf 99/ OO  hqk7 + +19:A;I:I I2:;(Qa((H;*9nnY]YpYpq  AJQ{^,;;B?!!%(- ! =!12a n-wA|(,z}(< (   t44Z@ A*+~~Z^ZqZqr // H#661;< E7"E<c>SSKnUR(aURcUR5 [ US[ 5(aUVs/sHo3SPM nnOUnUVs/sHnSU0PM nn[ X@RURS9n/nUHn/n[US5H/nURRU5n URU 5 M1 SU;a>[US5n [US5n [U 5Hn X;dM SX|-X|'M URURU55 M [ X`RURS9n URUR!XRR"5U 5upXS.$s snfs snfr\)rbrrrrPrrr4rtrr`rrrrarrrd)r"rrrr~rcrdrerrfrgrhrrir rMs r$r$DataCollatorForWholeWordMask.tf_callsx 99/ OO  hqk7 + +19:A;I:I I2:;(Qa((H;' >>VZVmVmn  AJQ{^,;;B?!!%(- ! =!12a n-wA|(,z}(< (   t44Z@ A'{NNW[WnWno ,,RWW[((-KZX#661;[US5n [US5n [U 5Hn X;dM SXk-Xk'M URURU55 M [ XPR URS9n URXL5upXS.$s snfs snfr\)rrrrPrrr4rtrr`rrrrarLrbs r$r'DataCollatorForWholeWordMask.numpy_callrkrlcURc[R"U5 U$URS:Xa<SSKnUR [ U5URS9nUVs/sHoAUPM sn$URS:XaSSKnURRS5SnURRRUR[ U55US9R5R5nUVs/sHoAUPM sn$URS:XaURRU5 U$gs snfs snf)NrrrDrr)rr)rrshufflerrOrandpermrrrb make_seeds experimentalstateless_shufflerrr)r" cand_indexesrOindicesrrrs r$_shuffle%DataCollatorForWholeWordMask._shuffles 99  NN< (    $ & nnS%6$..nQG-45WOW5 5  D ( #>>,,Q/2Dii,,>>rxxLHY?Zae>fllnuuwG-45WOW5 5  D ( NN " "< 0 )66s !D?E input_tokensc ~[UR[[45(d[R "S5 /n[ U5H`upEUS:XdUS:XaM[U5S:a,URS5(aUSRU5 MNURU/5 Mb URU5n[U[S[[[U5UR-5555n/n[!5nUH\n [U5U:a OL[U5[U 5-U:aM1U H%n UR#U 5 URU 5 M' M^ [U5[U5:wa [%S5e['[U55Vs/sH oDU;aSOSPM n nU $s snf) z= Get 0/1 labels for masked tokens with whole word mask proxy zDataCollatorForWholeWordMask is only suitable for BertTokenizer-like tokenizers. Please refer to the documentation for more information.z[CLS]z[SEP]rr^rz?Length of covered_indexes is not equal to length of masked_lms.r)rPr4rrwarningswarnrr startswithrryminrrTroundrsetaddr r) r"r{max_predictionsrwrrfnum_to_predict masked_lmscovered_indexes index_setindexrds r$ra-DataCollatorForWholeWordMask._whole_word_mask(s$..=:K*LMM MMJ   !,/HA5G#3< A%%*:*:4*@*@R ''*##QC(0}}\2 _c!Ss> $ $ ,A )ekdqdqds ds]`NN 2 23SW 2 Xds   '' 5HPZPZ ([cf'g >> # # /!99T^^%@%@AL  + +L + D+002"& OOEJJv||T5K5KLX\XfXfO g l l n  $(>>#G#GHaHa#b  ! !Q &$*B*Ba*G> !T333&*%=%=%N" OOEJJv||5OP\`\j\jO k p p r    }}S%8&,,ejjdhdrdr}s !-!=~K s #Ic|SSKnURU5nURRc [ S5eUR U5nUR X#R5nUVs/sHopRRUSS9PM nnXcR XRS9)-nURRbXRR:Hn Xi)-nURXaS5nURX@RUR5U-n URXRRU5nURS:XdUR S:XaX4$SUR- n UR U - n URXLUR5U-U )-n UR(a9URR#U[%UR5UR&S 9nO8UR(R#U[%UR5UR&S 9nURXU5nX4$s snf) rrNrTr4rKr{rr$)rbrr4rr r7rrxr6rrr&rrrr(rrrrdr)r"r rdrr)rMr*r8r!rr+r,r-r.r/s r$r+DataCollatorForWholeWordMask.tf_mask_tokenss" hhv& >> $ $ ,A V$gg6ek dj]`NN 2 23SW 2 Xdj  (773Fgg7+V*VV >> # # /!^^%@%@@L+m;N.$7 ,,[:P:PRVR`R`adrr*NN,H,H&Q  ! !Q &$*B*Ba*G> !T333&*%=%=%N"   kt~~ V    >>>>11+c$..FYacaiai1jL99,,[T^^AT\^\d\d,eL.?~Q s(#H9c:URRc [S5e[R"U5nUR [ 5nUR5Vs/sHoPRRUSS9PM nnSU[R"U[ S9'URRbX0RR:HnSXG'SX4)'UR(aFURRSURURS9R [ 5U-nOI[R RSURURS9R [ 5U-nURR#URR5X'URS:XdUR$S:XaX4$SUR- n UR$U - n UR(aURRSXRS9R [ 5U-U)-n URR'S[)UR5UR[R*S 9n O[R RSXRS9R [ 5U-U)-n [R R-S[)UR5UR[R*S 9n XX'X4$s snf) rrTr4rrKr{rrrP)r4rr rrMrSrxrr6rnrrrrTrrrrHrrUrrdr) r"r rdrMr*r8r!rr+r,r-r.r/s r$rL.DataCollatorForWholeWordMask.numpy_mask_tokenss >> $ $ ,A %++D1ekdqdqds ds]`NN 2 23SW 2 Xds  EFrxx 34@A >> # # /!^^%@%@@L+,N ("& >>''4+A+A 'U\\]abess   ""1d&<&<6<<"PWWX\]`nn $(>>#G#GHaHa#b  ! !Q &$*B*Ba*G> !T333&*%=%=%N" >>''+ELL'Y``aef !##$   >>22qs4>>?RY_YeYemomumu2vL ""1&@||"T[[\`a !##$  99,,T^^9LSYS_S_gigogo,pL!-!=~a s #Lr(N)i)r*r+r,r-rEr r rTrrrFrrrryrar r<rrLr.r(r'r$rZrZs? 74d3id38n.L(M#N7SWX[]`X`Sa7B!7U49c4S>+I%J K!7PTUXZ]U]P^!7F74d3id38n.L(M#N7SWX[]`X`Sa7B 0&T#Y&P66#6%S/6p:S:s:uS#X:x??#?%S/?r'rZc[U[5(aU$[US5(aUR5nUR 5$)Nr)rPrjr1rr)rs r$rrs8!T G   GGI 88:r'ch\rSrSrSrSrS\\\\ 4S\\\ 44Sjr S\ S\ \ \ \ 44Sjr S r g ) DataCollatorForSOPiz Data collator used for sentence order prediction task. - collates batches of tensors, honoring their tokenizer's pad_token - preprocesses batches for both masked language modeling and sentence order prediction c:[R"S[5 g)NzDataCollatorForSOP is deprecated and will be removed in a future version, you can now use DataCollatorForLanguageModeling instead.)r}r~ FutureWarning)r"argskwargss r$__init__DataCollatorForSOP.__init__s  7  r'rr:chSSKnSSKJn UVs/sHoDSPM nn[XPR5nUR U5upVnUVs/sHoDSPM nnU"USURR S9nUVs/sHoDSPM n nURU 5n UUUUU S.$s snfs snfs snf) Nr) pad_sequencer~token_type_idsT) batch_first padding_valuesentence_order_label)r~rMattention_maskrr)rOtorch.nn.utils.rnnrrr4 mask_tokensrrX) r"rrOrrr~rMrrsop_label_listrs r$r%DataCollatorForSOP.__call__s39ABg[) B(NNC ,0,<,CKL8"238L%n$VZVdVdVqVqrIQRg"89R${{>:#,,$8   CMSsB% B*<B/r cSSKnURRc [S5eUR 5nUR UR UR5nUR5Vs/sHoPRRUSS9PM nnURURXbRS9SS9 URRb5URURR5nURUSS9 UR!U5R5nU)R#5n URRb5URURR5n U RU S S9 S X8)'UR!UR UR S 55R5U-n URR%URR5X'UR!UR UR S 55R5U-U )-n UR'[)UR5UR UR*S9n XX'XU 4$s snf) z Prepare masked tokens inputs/labels/attention_mask for masked language modeling: 80% MASK, 10% random, 10% original. N-gram not applied yet. rNrTr4rKrArBg?r{rg?)rOr4rr r=rrrrr6rFrWrxrrrrGrVrHrrrU)r"r rOrMrIr8r!rr*rattention_padding_maskr+r.r/s r$rDataCollatorForSOP.mask_tokens5s'  >> $ $ ,A  "ZZ d6J6JKdjdqdqds ds]`NN 2 23SW 2 Xds   '' 5HPZPZ ([cf'g >> # # /!99T^^%@%@AL  + +L + D);<AAC)/002 >> # # /%+YYt~~/J/J%K "  ' '(>c ' J"&!??5::fllC+HINNPSaa#'>>#G#GHaHa#b FLL#)FGLLNQ__csbss}}S%8&,,ejj}Y !-!=~--3 s/#I7r(N)r*r+r,r-rErr rrFrr%r rr.r(r'r$rrsR  d38n!5 $sCx. .).#).%S# *>).r'rch\rSrSr%Sr\\S'Sr\\S'Sr \ \S'Sr \ \S 'S \ \\ \ \\\ \44S \\ \44S jrS \ \\ \ \\\ \44S \\ \44S jrS \ \\ \ \\\ \44S \\ \44SjrS\S \\\\\44SjrS\S \\\\\44SjrS\S \\\\\44SjrSrg)*DataCollatorForPermutationLanguageModelingiaz Data collator used for permutation language modeling. - collates batches of tensors, honoring their tokenizer's pad_token - preprocesses batches for permutation language modeling with procedures specific to XLNet r4gUUUUUU?plm_probabilitymax_span_lengthrrrr:c[US[5(aUVs/sHo"SPM nn[XR5nUR U5upEpgXEXgS.$s snfNrr~)r~ perm_masktarget_mappingrM)rPrrr4r<r"rrr^r rrrMs r$r5DataCollatorForPermutationLanguageModeling.torch_callo\ hqk7 + +0891+H9$X~~>484J4J54Q1>#~pp:Ac[US[5(aUVs/sHo"SPM nn[XR5nUR U5upEpgXEXgS.$s snfr)rPrrr4rrs r$r2DataCollatorForPermutationLanguageModeling.tf_callvs\ hqk7 + +0891+H9!(NN;484G4G4N1>#~pp:rc[US[5(aUVs/sHo"SPM nn[XR5nUR U5upEpgXEXgS.$s snfr)rPrrr4rLrs r$r5DataCollatorForPermutationLanguageModeling.numpy_call}rrr c SSKnURRc [S5eUR S5S-S:wa [S5eUR 5nUR URSURS9nURUR S5UR S5UR S54URS9n[UR S55HnSnUR S5nXx:aURSURS-S5R5n [XR - 5n XrRX- S-S5R5-n SXFXU -24'Xz- nXx:aMUR#UR S55XV'M UR%UR'5V s/sHoRR)U S S 9PM sn URS9n UR+U S S 9 URR,b5UR/URR05nUR+US S 9 WU -)nURR2X'S X4)'URUR S5UR S5UR S54URS9n[UR S55GHnUR5UR S55nUR7SUR S5S-45R9SS5nUUR;UR S5S-5nUR=UR9SS55nUR+XF)X-S5 UR7UR S5S45UR7SUR S545:*XF-UU'GM UR?5UXSR?54$s sn f) The masked tokens to be predicted for a particular sequence are determined by the following algorithm: 0. Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far). 1. Sample a `span_length` from the interval `[1, max_span_length]` (length of span of tokens to be masked) 2. Reserve a context of length `context_length = span_length / plm_probability` to surround span to be masked 3. Sample a starting point `start_index` from the interval `[cur_len, cur_len + context_length - span_length]` and mask tokens `start_index:start_index + span_length` 4. Set `cur_len = cur_len + context_length`. If `cur_len < max_len` (i.e. there are tokens remaining in the sequence to be processed), repeat from Step 1. rNThis tokenizer does not have a mask token which is necessary for permutation language modeling. Please add a mask token if you want to use this tokenizer.rrThis collator requires that sequence lengths be even to create a leakage-free perm_mask. Please see relevant comments in source code for details.rK)rTr4rArBr{r) rOr4rr rr=rrrxrrerrrrSrTreyerWrr6rFrrrr(aranger transposersflattenrU)r"r rOrMr*rrcur_lenmax_len span_lengthcontext_length start_indexr8r!r non_func_maskr perm_indexs r$r<> $ $ ,N  ;;q>A  "A  FLL!5::Ffkk!nfkk!nfkkRSn%U]b]j]jkv{{1~&AGkk!nG##mmAt/C/Ca/GNSSU !$[3G3G%G!H% n6RUV6VX\(](b(b(dd MN+k0I"IIJ)#!& &++a. 9N %'($llekereret uet^a^^ 3 3CTX 3 Yet u**+  ##$7s#C >> # # /!99T^^%@%@AL  ' ' C ' @')<<= !%!=!="&KKQQQ PX]XeXeKf v{{1~&Afkk!n5J#++RQ11D,EFPPQRTUVJ#ENN6;;q>Q3F$GHJz';';Aq'ABJ  # #^%6$69I$I2 N ""FKKNA#67:;M;MqRXR]R]^_R`Na;bb!"IaL1'8{{}iFFY vs#Pc SSKnURRc [S5eUR U5SS-S:wa [S5eUR U5n[ R"URR5S[S9nUR U5n[ R"USUSUS4[ RS9n[[U55HnSnUR U5Sn X:aY[SURS-5n [!XR"- 5n U[SX- S-5-n SXGXU -24'X- nX:aMY[ R$"US5Xg'M UR'UR)U5URS9nUR)U5nUR)UR+5R-5V s/sHn URR/U SS 9PM sn 5nUR'XRS9nXN)-nURR0bX0RR2:HnXO)-nWU-)nUR5X@RR6U5nUR5XCS 5n/n[[U55HnURUS5nUR9UR;US USS-455nUR<R?U5nUR;UR9U5S 5nUR5XG)UU-S U5nURAUR;UUSS45UR;USUS45:*XG-5 M URCUSS 9nUR'XRD5UR'UUR5XbR'X2RD54$s sn f)rrNrrrrrKTr4r{r)rr)#rbr4rr rr7rras_listrxrrerrrrrTrrrrkrrr6rrr&r(rrrrrrrXrd)r"r rrMr* labels_shaperrrrrrrr8r!rrrrs r$r9DataCollatorForPermutationLanguageModeling.tf_mask_tokenss >> $ $ ,N  88F A  "a 'A  V$!5!5!7$Gxx' <?LO\RS_"U]_]g]ghs6{#AGhhv&q)G#%a)=)=)AB !$[3G3G%G!H%>3ORS3S(TT MN+k0I"IIJ)#!#|A 7N %$&!5!5n!ERWWU--n= 22"<<>002 2C66sW[6\2   !gg&9gI'*>> >> # # /!^^%@%@@L+m;N')<<= ...*F*FO.$7 s6{#A,q/2Jbjjb,q/UVBV=W&XYJ**:6JBLL$#4"4}Q7G"GZXJ   Ja!(<=JYZ\hij\kXlAmm #$ 3$:HHYQH/ wwvxx("'')RZZ*H.ZaZabhjrjrZssse s$O0c jURRc [S5eURSS-S:wa [S5e[R "U5n[R "URS[S9n[R"URSURSURS4[RS9n[URS5HnSnURSnXg:aY[SURS-5n[XR- 5n U[SX- S-5-n SX5XU-24'Xi- nXg:aMY[R"URS5XE'M [R "UR#5V s/sHoRR%U SS9PM sn [S9n SX<'URR&bX RR(:Hn S X='W U -)nURR*X'S X#)'[R"URSURSURS4[RS9n[URS5Hn[R,"URS5nUR/S URSS-45R0n[R2R5U5 UR0R75nS UX5)X-'UR/URSS45UR/SURS45:*X5-X'M UR9[R:5XUR9[R:54$s sn f) rrrrrrrKTr4rAr{r)r4rr rrrMrrxrrerrrrTrrrnrr6rrr(rrTrrrrrSrd)r"r rMr*rrrrrrrr8r!rrrrs r$rL> $ $ ,N  <<?Q ! #A  q=6<<?FLLOV\\RS_"U]_]g]ghv||A'AGll1oG#%a)=)=)AB !$[3G3G%G!H%>3ORS3S(TT MN+k0I"IIJ)#!#v||A 7N %((!hhekereret uet^a^^ 3 3CTX 3 Yet u /0+ >> # # /!^^%@%@@L+.N (')<<= !%!=!="&HHfll1ov||A QPXZXbXbc v||A'A6<<?3J#++RaA1E,FGIIJ II  j )#--/J ACJ))M,<< = ""FLLOQ#78J+I%J KqPTUXZ]U]P^qq4d3id38n.L(M#NqSWX[]`X`SaqaGaGc3S6H0IaGFitSitU3S#3E-FitV_[_[c3S6H0I_[r'rcD^\rSrSrSrSSSSS.U4SjjrS SjrS rU=r$) DataCollatorWithFlatteningia Data collator used for padding free approach. Does the following: - concatenates the entire mini batch into single long sequence of shape [1, total_tokens] - uses `separator_id` to separate sequences within the concatenated `labels`, default value is -100 - no padding will be added, returns `input_ids`, `labels` and `position_ids` by default - optionally returns the kwargs contained in FlashAttentionKwargs - optionally returns seq_idx indicating which sequence each token belongs to Using `DataCollatorWithFlattening` will flatten the entire mini batch into single long sequence. Make sure your attention computation is able to handle it! Tr{F)return_position_ids separator_idreturn_flash_attn_kwargsreturn_seq_idxc>[TU]"U0UD6 XlX lX0lX@l1SkUl1SkUlSS1Ulg)N>rMr~ position_ids>rMseq_idxr~r max_length_q max_length_k) superrrrrr _int_64_keys_batch_dim_keys _py_int_keys)r"rrrrrrrs r$r#DataCollatorWithFlattening.__init__sL $)&)#6 ((@%,CQ+^<r'c FUc URnUc URnSUS;n//S.nUR(aURS/05 UR(aURS/05 UR (aS/nSn[ U5GH upU Sn US==U - ss'U(aUS==U/U SSS-- ss'OUS==U/U SS-- ss'UR(a(US==[[[U 555- ss'UR(a/US==[[U 55V s/sHoPM sn - ss'UR (dMWRUS[U 5-5 [W[U 55nGM UR (aW=US 'US 'W=US 'US 'US :Xa)SSK n U Rn U RnU RnOGUS:Xa1[ R"n [ Rn[ RnO[%SU<S35eUR'5HGunnUUR(;aU/nUUR*;dM+U "UUUR,;aUOUS9UU'MI U$s sn f)NrMrr_rrr~rr cu_seq_lens_q cu_seq_lens_krrrrz;return_tensors must be one of ("pt", "np"), return_tensors=z not suportedrK)rrrupdaterrrrjrrrrrOrWrdrrrnr rYrrr)r"r#rris_labels_providedr^ cu_seq_lensrsrsampler~_rOdata_clsdtype_64dtype_32r_r`s r$r%#DataCollatorWithFlattening.__call__sy  !!00N  ,,L%!4 B/  # # LL."- .    LL)R )  ( (#KJ(2OG{+I + ) + !hL>F84DQR4H#HHhL>IabM#AA''n%eC N.C)DD%""i eC N6K$L6KW6K$LL ,,,"";r?S^#CD S^<  3  ( (>I IE/ "U?%;O>O9OXU]^a " =%Ms J )rrrrrrr)NN) r*r+r,r-rErr%r. __classcell__)rs@r$rrs*(!!& ==$66r'r)rr))3multiprocessingr rr}collections.abcr dataclassesrrtypingrrrr r r r r rr models.bertrrtokenization_utils_baserrrrrFrrr9r?rAr<r=r>rprzrTrrrrrrrZrrrrr(r'r$rsq #!MMM:=#)3/~xn1E0FSRUX0V'WX  N N,5D$85RVWZ\_W_R`52 ?+? ?6!$~*>!4S>!H&tN';&S#X&R$~*>4S>@ 00 0f H):H HV"(3-"J#$x}#$L(3-@ W$5W Wt DD DN \&7\ \~  g#Bg gT  O.8O. O.d O[1BO[ O[d  Z!4Z Zr'