o Zh @s\ddlZddlZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z mZmZmZmZddlZddlmZmZddlmZdd lmZed e Z ed e e ege ee ffZGd d d ZddZd8de ede ee ffddZ eGdddeZ!de ede ee ffddZ"de ede ee ffddZ#de ede ee ffddZ$eGdddZ%eGdd d eZ&d9d!ee'fd"d#Z(d9d!ee'fd$d%Z)d9d!ee'fd&d'Z*eGd(d)d)eZ+eGd*d+d+Z,eGd,d-d-eZ-eGd.d/d/e-Z.d0d1Z/eGd2d3d3e-Z0eGd4d5d5eZ1eGd6d7d7e!Z2dS):N)Mapping) dataclass)randint)AnyCallableDictListNewTypeOptionalTupleUnion) BertTokenizerBertTokenizerFast)PreTrainedTokenizerBase)PaddingStrategyInputDataClass DataCollatorc@seZdZdddZdS)DataCollatorMixinNcCsT|dur|j}|dkr||S|dkr||S|dkr"||Std|d)Ntfptnpz Framework 'z' not recognized!)return_tensorstf_call torch_call numpy_call ValueErrorselffeaturesrr N/var/www/auris/lib/python3.10/site-packages/transformers/data/data_collator.py__call__(s   zDataCollatorMixin.__call__N)__name__ __module__ __qualname__r"r r r r!r'src Os`t|ds |j|i|S|jdd}d|jd<z|j|i|}W||jd<|S||jd<w)zz Pads without triggering the warning about how using the pad function is sub-optimal when using a fast tokenizer. deprecation_warningszAsking-to-pad-a-fast-tokenizerFT)hasattrpadr'get) tokenizerZpad_argsZ pad_kwargsZ warning_statepaddedr r r!"pad_without_fast_tokenizer_warning5s    r-rrreturncCs4|dkrt|S|dkrt|S|dkrt|SdS)a Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named: - `label`: handles a single value (int or float) per object - `label_ids`: handles a list of values per object Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it's useful. rrrN)torch_default_data_collatortf_default_data_collatornumpy_default_data_collator)rrr r r!default_data_collatorKsr2c@sFeZdZUdZdZeed<d deeee fdeee ffddZ dS) DefaultDataCollatora* Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named: - `label`: handles a single value (int or float) per object - `label_ids`: handles a list of values per object Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it's useful. This is an object (like other data collators) rather than a pure function like default_data_collator. This can be helpful if you need to set a return_tensors value at initialization. Args: return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". rrNrr.cCs|dur|j}t||Sr#)rr2rr r r!r"zs zDefaultDataCollator.__call__r#) r$r%r&__doc__rstr__annotations__rrrr"r r r r!r3ds  ,r3csddl}t|dtsdd|D}|d}i}d|vrN|ddurNt|d|jr0|dn|d}t|tr<|jn|j}|jdd|D|d|d<nz/torch_default_data_collator..labelcSg|]}|dqSr?r r:r r r!r=r>dtypelabels label_idscSr@rEr r:r r r!r=r>cSr@rFr r:r r r!r=r>r?rEcg|]}|qSr r r:kr r!r=r>crHr r r:rIr r!r=r>crHr r r:rIr r!r=r>)torch isinstancerTensoritemintlongfloattensorstackitemsr5rndarrayZ from_numpy)rrKfirstbatchr?rCvr rIr!r/s.$  $r/csddl}t|dtsdd|D}|d}i}d|vr%|ddur%dnd|vr2|ddur2dnd|vr?|ddur?dnddurt||jrZ|jjrV|jn|j}nGt|tj sjt|tj r{t |jtj rw|jn|j}n&t|t tfrt|dtr|jn|j}n t|tr|jn|j}|jfdd|D|d|d<|D]7\}d vr|durt|tst||jtj fr|fd d|D|<q|fd d|D|<q|S) NrcSr7r r8r:r r r!r=r>z,tf_default_data_collator..r?rErDcrHr r r:)label_col_namer r!r=r>rB)r?rErDcrHr r r:rIr r!r=r>crHr r r:rIr r!r=r>) tensorflowrLrrMrC is_integerint64float32rrUZgenericZ issubdtypeintegertuplelistrOconvert_to_tensorrTr5rS)rrrVrWrCrXr )rJrYr!r0s8 "  r0cst|dtsdd|D}|d}i}d|vrJ|ddurJt|dtjr,|dn|d}t|tr8tjntj}tjdd|D|d|d<nz/numpy_default_data_collator..r?cSr@rAr r:r r r!r=r>rBrDrEcSr@rFr r:r r r!r=r>cSr@rFr r:r r r!r=r>rGcrHr r r:rIr r!r=r>crHr r r:rIr r!r=r>) rLrrrUrNrOr\r]arrayrSrTr5)rrVrWr?rCrXr rIr!r1s($ r1c@seZdZUdZeed<dZeee e fed<dZ e e ed<dZe e ed<dZe ed <d eee efd ee effd d ZdS)DataCollatorWithPaddinga Data collator that will dynamically pad the inputs received. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.0 (Volta). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". r+TpaddingN max_lengthpad_to_multiple_ofrrrr.cCsVt|j||j|j|j|jd}d|vr|d|d<|d=d|vr)|d|d<|d=|S)Nrdrerfrr?rDrE)r-r+rdrerfr)rrrWr r r!r"s  z DataCollatorWithPadding.__call__)r$r%r&r4rr6rdr boolr5rrer rOrfrrrrr"r r r r!rcs  *rcc@seZdZUdZeed<dZeee e fed<dZ e e ed<dZe e ed<dZe ed <d Ze ed <d d ZddZddZdS)"DataCollatorForTokenClassificationa Data collator that will dynamically pad the inputs received, as well as the labels. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.0 (Volta). label_pad_token_id (`int`, *optional*, defaults to -100): The id to use when padding the labels (-100 will be automatically ignore by PyTorch loss functions). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". r+TrdNrerflabel_pad_token_idrrcsddld|dvrdnd|dvr!fdd|Dnd}fdd|D}tj|jjjdd}|dur@|S|d jd jj}fd d |d krcfdd|D|<n fdd|D|<j |j d|<|S)Nrr?rDcrHr r r;feature label_namer r!r=Jr>zADataCollatorForTokenClassification.torch_call..c"g|] }fdd|DqS)ci|] \}}|kr||qSr r r;rJrXrnr r! LzLDataCollatorForTokenClassification.torch_call...rTrlrnr r!r=L"rrg input_idscst|jr |St|Sr#)rLrMtolistr`)Ztensor_or_iterablerKr r!to_list]s z>DataCollatorForTokenClassification.torch_call..to_listrightcs(g|]}|jgt|qSr rklenr;r?rsequence_lengthr{r r!r=ccs(g|]}jgt||qSr r}rrr r!r=grrB) rKkeysr-r+rdrerfshape padding_siderRr\)rrrDZno_labels_featuresrWrr )rorrr{rKr!rFs4&    z-DataCollatorForTokenClassification.torch_callcsddld|dvrdnd|dvr!fdd|Dnd}tj|jjj|dur3dndd}|dur=|S|djd jj }|d kr\fd d|D|d<n fd d|D|d<fd d| D}|S)Nrr?rDcrHr r rlrnr r!r=rr>z>DataCollatorForTokenClassification.tf_call..rrgrwrxr|c(g|]}t|jgt|qSr r`rkr~rrrr r!r=rc(g|]}jgt|t|qSr rkr~r`rrr r!r=rcs"i|] \}}|j|jdqSrBrar\rrrr r!rsrvz>DataCollatorForTokenClassification.tf_call..) rZrr-r+rdrerfrarrrTrrrDrWrr )rorrrr!rns0&    z*DataCollatorForTokenClassification.tf_callcsd|dvr dnd|dvrfdd|Dnd}tj|jjj|dur/dndd}|dur9|St|djd jj }|d krXfd d|D|d<n fd d|D|d<d d| D}|S)Nr?rrDcrHr r rlrnr r!r=r>zADataCollatorForTokenClassification.numpy_call..rrgrwrxr|crr rrrr r!r=rcrr rrrr r!r=rcSs"i|] \}}|tj|tjdqSrrrbr\rrr r r!rsrvzADataCollatorForTokenClassification.numpy_call..) rr-r+rdrerfrrbrrrTrr )rorrr!rs.&    z-DataCollatorForTokenClassification.numpy_call)r$r%r&r4rr6rdr rhr5rrer rOrfrkrrrrr r r r!ri!s   ( rirfcsDddlt|dtttjfrfdd|D}|ddtfdd|D}|rC|dus6|dkrCt|jsCj |ddS|j durRt d|j j d td d|D}|durm||dkrm||d |}|dt||g|j}t|D]!\}}|jd kr|||d|jdf<q~||||jd df<q~|S) _Collate `examples` into a batch, using the information in `tokenizer` for padding if necessary.rNcg|] }j|jdqSr)rRrPr;erzr r!r=z(_torch_collate_batch..c3s|] }|dkVqdSrNsizer;xlength_of_firstr r! sz'_torch_collate_batch..)dimCYou are attempting to pad samples but the tokenizer you are using () does not have a pad token.css|]}|dVqdSrrrr r r!rsrxr|)rKrLr`r_rrUrallrMrS pad_tokenr __class__r$maxZnew_fullr~ pad_token_id enumeraterrexamplesr+rfare_tensors_same_lengthreresultiexampler )rrKr!_torch_collate_batchs.   rc sLddl t|dttfrfdd|D}t|dtfdd|D}|r;|dus4|dkr;j|ddS|jdurJtd|j j dt d d|D}|dure||dkre||d |}g} |d}t j|d ft jd }|D]$}|jd kr|t||d<n|t||d<|j|||jdqzj|ddS)Nrcrrrrrr r!r=rz%_tf_collate_batch..c3|] }t|kVqdSr#r~rrr r!rz$_tf_collate_batch..Zaxisrrcs|]}t|VqdSr#rrr r r!rrxr rBr|)rrx)rr)Zconstant_values)rZrLr`r_r~rrSrrrr$rrankrzerosint32rappendr)r) rr+rfrrerrZpaddingsrr )rrr!_tf_collate_batchs4   rcs,t|dttfrdd|D}t|dtfdd|D}|r4|dus-|dkr4tj|ddS|jdurCtd|j j d t d d|D}|dur^||dkr^||d |}tj t||f|j |djd }t|D]!\}}|jd kr|||d|jdf<qr||||jd df<qr|S)rrcSsg|] }tj|tjdqSrrrr r r!r=rz(_numpy_collate_batch..c3rr#rrrr r!rrz'_numpy_collate_batch..Nrrrcsrr#rrr r r!r rrx)rZ fill_valuerCr|)rLr`r_r~rrrSrrrr$rfullrrCrrrrr rr!_numpy_collate_batchs*    rc@s~eZdZUdZeed<dZeee e fed<dZ e e ed<dZe e ed<dZe ed <d eee effd d Zd dZdS)DataCollatorForMultipleChoicea Data collator that dynamically pads a batch of nested examples for multiple choice, so that all choices of all examples have the same length. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences according to the model's padding side and padding index among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): Pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". r+TrdNrerfrrrcsddl}d|dvrdndfdd|D}t|t|ddtfdd|Dgd }|jj||j|j|jd d }fd d | D}|j ||j d|d<|S)Nrr?rDcg|]}|qSr popr;rrnr r!r=Az.rwc3s&|]fddtDVqdS)c"g|] fddDqS)ci|] \}}||qSr r rrrr r!rsIzQDataCollatorForMultipleChoice.torch_call....rur;rrr!r=IrvzFDataCollatorForMultipleChoice.torch_call...Nranger num_choicesrr!rIs$z;DataCollatorForMultipleChoice.torch_call..)startrrgcs i|] \}}||dqS)viewrr) batch_sizerr r!rsVs z.rB) rKrr~sumr+r)rdrerfrTrRr\)rrrKrDZ flat_examplesrWr )rrorr!r<s$ z(DataCollatorForMultipleChoice.torch_callcsddld|dvrdndfdd|D}t|t|ddfdd|D}t|g}|jj||j|j|jdd }fd d | D}j |j d |d<|S) Nrr?rDcrr rrlrnr r!r=^rz9DataCollatorForMultipleChoice.tf_call..rwcs"g|] fddtDqS)cr)crr r rrrr r!rsbrzODataCollatorForMultipleChoice.tf_call....rurrmrr!r=brvzDDataCollatorForMultipleChoice.tf_call...rrrrr!r=asrrgc s$i|]\}}||dfqSr)reshaperr)rrrr r!rsns$z9DataCollatorForMultipleChoice.tf_call..rB) rZrr~rr+r)rdrerfrTrar\)rrrDZflattened_featuresrWr )rrorrr!rZs&  z%DataCollatorForMultipleChoice.tf_call)r$r%r&r4rr6rdr rhr5rrer rOrfrrrrrrr r r r!rs   rc@seZdZUdZeed<dZeeed<dZ e e e e fed<dZeeed<dZeeed<d Zeed <d Ze ed <dd dZdS)DataCollatorForSeq2Seqa Data collator that will dynamically pad the inputs received, as well as the labels. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. model ([`PreTrainedModel`], *optional*): The model that is being trained. If set and has the *prepare_decoder_input_ids_from_labels*, use it to prepare the *decoder_input_ids* This is useful when using *label_smoothing* to avoid calculating loss twice. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.0 (Volta). label_pad_token_id (`int`, *optional*, defaults to -100): The id to use when padding the labels (-100 will be automatically ignored by PyTorch loss functions). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". r+NmodelTrdrerfrjrkrrc s^|durj}d|dvrdnd|dvr$fdd|Dnd}|dur5tdd|Dr5d}fdd|D}tj|jjj|d }jd upVjtj k}|dur|rwt |dt rmt ||d<n]d d|D|d<nSjtj kojdu}|st d d|Dnjjdurjd jjjjt |dt rfdd|D|d<n fdd|D|d<|dddur |dkrddl}|j|d|jd|d<n&|dkrddl} | j|d| jd|d<ntj|dtjd|d<nd|d<|dur-jdur-tjdr-jj|dd} | |d<|S)Nr?rrDcrHr r rlrnr r!r=r>z3DataCollatorForSeq2Seq.__call__..css|]}|duVqdSr#r rr r r!rrz2DataCollatorForSeq2Seq.__call__..crp)crqr r rrrnr r!rsrtz>DataCollatorForSeq2Seq.__call__...rurlrnr r!r=rvrgFcSsg|] }t|ggqSr )r concatenaterr r r!r=rcsrr#r)r;lr r r!rrrxcsDg|]}dkr|jgt|n jgt||qS)r|r}rmax_label_lengthrrr r!r=s c shg|]0}dkrt|tjjgt|tjdgnttjjgt|tjd|gqS)r|rB)rrrbrkr~r\rrr r!r=srrBr%prepare_decoder_input_ids_from_labels)rDdecoder_input_ids)rrrr-r+rdrerfrZ DO_NOT_PADrLr`Z MAX_LENGTHrrr*rKrRr\rZZconstantrrbrr(r) rrrrDZnon_labels_featuresrWZ no_paddingZ max_paddingrKrrr )rorrrr!r"sl&        zDataCollatorForSeq2Seq.__call__r#)r$r%r&r4rr6rr rrdr rhr5rrerOrfrkrr"r r r r!rss !  rc @seZdZUdZeed<dZeed<dZe ed<dZ e ed<d Z e ed <d Z e eed <d Zeed<dZeed<d Ze eed<ddZddZddZed*ddZ d*dede edeeeffddZdeeeeeeeeffdeeeffd d!Zdeeeeeeeeffdeeeffd"d#Zd*dede edeeeffd$d%Z deeeeeeeeffdeeeffd&d'Z!d*dede edeeeffd(d)Z"d S)+DataCollatorForLanguageModelingar Data collator used for language modeling. Inputs are dynamically padded to the maximum length of a batch if they are not all of the same length. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. mlm (`bool`, *optional*, defaults to `True`): Whether or not to use masked language modeling. If set to `False`, the labels are the same as the inputs with the padding tokens ignored (by setting them to -100). Otherwise, the labels are -100 for non-masked tokens and the value to predict for the masked token. mlm_probability (`float`, *optional*, defaults to 0.15): The probability with which to (randomly) mask tokens in the input, when `mlm` is set to `True`. mask_replace_prob (`float`, *optional*, defaults to 0.8): The probability with which masked tokens are replaced by the tokenizer's mask token (e.g., `[MASK]`). Defaults to 0.8, meaning 80% of the masked tokens will be replaced with `[MASK]`. Only works when `mlm` is set to `True`. random_replace_prob (`float`, *optional*, defaults to 0.1): The probability with which masked tokens are replaced by random tokens from the tokenizer's vocabulary. Defaults to 0.1, meaning 10% of the masked tokens will be replaced with random tokens. The remaining masked tokens (1 - mask_replace_prob - random_replace_prob) are left unchanged. Only works when `mlm` is set to `True`. pad_to_multiple_of (`int`, *optional*): If set, will pad the sequence to a multiple of the provided value. return_tensors (`str`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". seed (`int`, *optional*): The seed to use for the random number generator for masking. If not provided, the global RNG will be used. For best performance, this data collator should be used with a dataset having items that are dictionaries or BatchEncoding, with the `"special_tokens_mask"` key, as returned by a [`PreTrainedTokenizer`] or a [`PreTrainedTokenizerFast`] with the argument `return_special_tokens_mask=True`. 1. Default Behavior: - `mask_replace_prob=0.8`, `random_replace_prob=0.1`. - Expect 80% of masked tokens replaced with `[MASK]`, 10% replaced with random tokens, and 10% left unchanged. 2. All masked tokens replaced by `[MASK]`: - `mask_replace_prob=1.0`, `random_replace_prob=0.0`. - Expect all masked tokens to be replaced with `[MASK]`. No tokens are left unchanged or replaced with random tokens. 3. No `[MASK]` replacement, only random tokens: - `mask_replace_prob=0.0`, `random_replace_prob=1.0`. - Expect all masked tokens to be replaced with random tokens. No `[MASK]` replacements or unchanged tokens. 4. Balanced replacement: - `mask_replace_prob=0.5`, `random_replace_prob=0.4`. - Expect 50% of masked tokens replaced with `[MASK]`, 40% replaced with random tokens, and 10% left unchanged. Note: The sum of `mask_replace_prob` and `random_replace_prob` must not exceed 1. If their sum is less than 1, the remaining proportion will consist of masked tokens left unchanged. r+Tmlmg333333?mlm_probability皙?mask_replace_probg?random_replace_probNrfFtf_experimental_compilerrseedcCs|jr |jjdur td|jdks|jdkrtd|j|jdkr'td|jdks1|jdkr5td|jdks?|jdkrCtdt|j|_t|j|_t|j|_|jreddl }|j |j dd |_ d|_ dS) NzThis tokenizer does not have a mask token which is necessary for masked language modeling. You should pass `mlm=False` to train on causal language modeling instead.rrxz*mlm_probability should be between 0 and 1.zHThe sum of mask_replace_prob and random_replace_prob should not exceed 1z,mask_replace_prob should be between 0 and 1.z.random_replace_prob should be between 0 and 1.T)Z jit_compile) rr+ mask_tokenrrrrrQrrZfunctiontf_mask_tokens generator)rrr r r! __post_init__Cs&    z-DataCollatorForLanguageModeling.__post_init__cCsT|jdkrddl}||S|jdkr ddl}|jj|Sddl}|j|S)Nrrr) rrK GeneratorZ manual_seedrZrandomZ from_seednumpyZ default_rng)rrrKrrr r r! get_generator]s   z-DataCollatorForLanguageModeling.get_generatorcCs`tjdkr||j|_dSddl}|jj }|dur$d}t |||j|j |_dS)N MainProcessr)zZWorker process information is not available for seeding the generator. This may be becausezZyou are using multiprocessing without using a PyTorch DataLoader. The `seed` parameter canzVonly be used when using multiprocessing with a PyTorch DataLoader. Please either use azAsingle process or use a PyTorch DataLoader with multiple workers.) mpcurrent_processnamerrrrKutilsdataZget_worker_inforid)rrKZ worker_infoZ error_stringr r r! create_rngks z*DataCollatorForLanguageModeling.create_rngcCsZddl}|||}|r||||dddk|jS|||j|dddk|jS)Nrrx)rZfillcastuniformrhr)rZ probabilityrrZ prob_matrixr r r! tf_bernoullis   "z,DataCollatorForLanguageModeling.tf_bernoulliinputsspecial_tokens_maskr.cCsddl}|||j}||}|||j|j|@}|||d}|||j|j|@} || ||}|jdks?|j dkrC||fSd|j} |j | } ||| |j|@| @} |jrh|jj |||jd} n |j j |||jd} || | |}||fS)w Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. rNrjrxmaxvalrC) rZrrCrrrrwhererrrr)rr vocab_size mask_token_idrr input_shapemasked_indicesrDindices_replacedremaining_probrandom_replace_prob_scaledindices_random random_wordsr r r!rs,   z.DataCollatorForLanguageModeling.tf_mask_tokensrcs8ddl}jrjdurt|dtr"tj|djd}n dt |jjdi}| dd}j r{|durWfdd|d D}||j||jd |j}n|||j}j||d|j|jjtjd \|d<|d <|S|d}jjdur||jjkd |}n||}||d <|S) Nrrrrfrwrfrcg|] }jj|ddqST)Zalready_has_special_tokensr+Zget_special_tokens_maskr;valrr r!r=z;DataCollatorForLanguageModeling.tf_call..rB)rrrrDrj)rZrrrrLrr-r+rfrrrrryrrar\rhrrr~rridentity)rrrrWrrDr r r!rs<     z'DataCollatorForLanguageModeling.tf_callcCs|jr |jdur |t|dtrt|j|d|jd}n dt||j|jdi}| dd}|j rC|j |d|d\|d<|d<|S|d }|jj durWd |||jj k<||d<|S) NrrrrwrrrrDrj)rrrrLrr-r+rfrrrtorch_mask_tokensclonerrrrWrrDr r r!rs&   z*DataCollatorForLanguageModeling.torch_callc sFddl}|}||jj}|dur(fdd|D}|j||jd}n|}|j|dd|j |j d}d ||<|j ||jj j d|@}j j j||<j d kshjdkrl||fSd j }j|} |j ||j| j d|@|@} |jtj |j|jj d } | | || <||fS) rrNcrr r r r r r!r=zEDataCollatorForLanguageModeling.torch_mask_tokens..rBvaluerrjrxrCr)rKrrrrryrRrh masked_fill_ bernoullirrr+convert_tokens_to_idsrrrr~rP) rrrrKrDprobability_matrixrrrrrrr r r!r s<     z1DataCollatorForLanguageModeling.torch_mask_tokenscCs|jr |jdur |t|dtrt|j|d|jd}n dt||j|jdi}| dd}|j rC|j |d|d\|d<|d<|St |d}|jjdurXd |||jjk<||d<|S) NrrrrwrrrrDrj)rrrrLrr-r+rfrrrnumpy_mask_tokensrcopyrrr r r!r;s&   z*DataCollatorForLanguageModeling.numpy_callc st|}t|jj}|dur$fdd|D}tj|td}n|t}d||<j r>j j d||jdt}n tj j d||jdt}d||<j rdj j dj |jdt|@}ntj j dj |jdt|@}j j||<j dksjdkr||fSdj }j|}j rj j d||jdt|@|@} j jdtj t| tjd } n#tj j d||jdt|@|@} tj jdtj t| tjd } | || <||fS) rNcrr r r r r r!r=arzEDataCollatorForLanguageModeling.numpy_mask_tokens..rBrrxrrjlowhighrrC)rrrrrryrbrhastyperbinomialrrr+rrintegersr~Z count_nonzeror\r) rrrrDrrrrrrrr r r!rYsX       z1DataCollatorForLanguageModeling.numpy_mask_tokensr#)#r$r%r&r4rr6rrhrrQrrrfr rOrrr5rrrr staticmethodrrr rrr rrrrrrr r r r!rs: <          4/4.$41(rc@s"eZdZdZdeeeeeee effdee effddZ deeeeeee effdee effddZ deeeeeee effdee effdd Z d d Z dd ee fddZdededeeeffddZdededeeeffddZdededeeeffddZdS)DataCollatorForWholeWordMaska# Data collator used for language modeling that masks entire words. - collates batches of tensors, honoring their tokenizer's pad_token - preprocesses batches for masked language modeling This collator relies on details of the implementation of subword tokenization by [`BertTokenizer`], specifically that subword tokens are prefixed with *##*. For tokenizers that do not adhere to this scheme, this collator will produce an output that is roughly equivalent to [`.DataCollatorForLanguageModeling`]. rr.cC|jr |jdur |t|dtrdd|D}n |}dd|D}t||j|jd}g}|D]C}g}t|dD] }|j |}| |q;d|vrlt|d} t |d} t | D]} | | vrkd|| || <q]| | |q1t||j|jd} ||| \} }| |d S) NrcSr@rwr rr r r!r=r>z;DataCollatorForWholeWordMask.torch_call..cSg|]}d|iqSr)r rr r r!r=r>rrw chinese_ref##rwrD)rrrrLrrr+rfry_convert_id_to_tokenrr~r_whole_word_maskrrrrw batch_input mask_labelsr ref_tokensrtokenref_poslen_seqr batch_maskrrDr r r!r0      z'DataCollatorForWholeWordMask.torch_callcCs(ddl}|jr|jdur|t|dtrdd|D}n |}dd|D}t||j|jd}g}|D]C}g}t |dD] }|j |} | | q?d|vrpt |d} t |d} t | D]} | | vrod|| || <qa| ||q5t||j|jd} ||||j| \}}||d S) NrcSr@r)r rr r r!r=r>z8DataCollatorForWholeWordMask.tf_call..cSr*r)r rr r r!r=r>rrwr+r,r-)rZrrrrLrrr+rfryr.rr~rr/rrr\)rrrrwr1r2rr3rr4r5r6rr7rrDr r r!rs2      z$DataCollatorForWholeWordMask.tf_callcCr() NrcSr@r)r rr r r!r=r>z;DataCollatorForWholeWordMask.numpy_call..cSr*r)r rr r r!r=r>rrwr+r,r-)rrrrLrrr+rfryr.rr~rr/rr0r r r!rr8z'DataCollatorForWholeWordMask.numpy_callcs|jdur tS|jdkr(ddl}|jt|jd}fdd|DS|jdkrTddl}|j dd}|jj j | t|d }fd d|DS|jd kra|jSdS) Nrrrcg|]}|qSr r r;r cand_indexesr r!r=r>z9DataCollatorForWholeWordMask._shuffle..rr )rcr9r r r:r;r r!r="r>r)rrshufflerrKrandpermr~rrZZ make_seedsZ experimentalZstateless_shufflerrry)rr<rKindicesrrr r;r!_shuffles    $  z%DataCollatorForWholeWordMask._shuffle input_tokensc s.t|jttfs tdg}t|D]&\}}|dks|dkr qt|dkr3|dr3|d |q| |gq| |}t |t dt tt||j}g}t|D]$}t||krant|t||krlqW|D] } | | | qnqWtt|krtdfdd tt|D} | S) zM Get 0/1 labels for masked tokens with whole word mask proxy zDataCollatorForWholeWordMask is only suitable for BertTokenizer-like tokenizers. Please refer to the documentation for more information.z[CLS]z[SEP]rxr,rz?Length of covered_indexes is not equal to length of masked_lms.csg|] }|vr dndqS)rxrr r:Zcovered_indexesr r!r=MrzADataCollatorForWholeWordMask._whole_word_mask..)rLr+rrwarningswarnrr~ startswithrr@minrrOroundrsetaddrr) rrBZmax_predictionsr<rr4Znum_to_predictZ masked_lmsZ index_setindexr2r rCr!r/(s8 "   z-DataCollatorForWholeWordMask._whole_word_maskrr2csTddl}jjdurtd|}|}fdd|D}|j|j||jdddjj dur@| jj }|j|dd|}d ||<|j | |jjjd |@} jjj|| <jd ksojdkrs||fSd j} j| } |j | |j| jd |@| @} |jtj|j|jjd } | | || <||fS) Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. Set 'mask_labels' means we use whole word mask (wwm), we directly mask idxs according to it's ref. rNThis tokenizer does not have a mask token which is necessary for masked language modeling. Remove the --mlm flag if you want to use this tokenizer.crr r r r r r!r=arzBDataCollatorForWholeWordMask.torch_mask_tokens..rBrrrjrrxr)rKr+rrrryrrRrhreqrrrrrrrrrr~rP)rrr2rKrDrr padding_maskrrrrrrr r r!rPsD       z.DataCollatorForWholeWordMask.torch_mask_tokenscsZddl}||}jjdurtd||}|||j}fdd|D}||j||jd@}jjdurD|jj k}||@}| ||d} |j j |@} | | jj|}j dksijdkrm||fSdj } j| } || j |@| @} j rj j|tj|jd } n |jj|tj|jd } | | | |}||fS) rLrNrMcrr r r r r r!r=rz?DataCollatorForWholeWordMask.tf_mask_tokens..rBrjrxr)rZrr+rrrrrhrrrrrrrrrr~r\r)rrr2rrrDrrrOrrrrrr r r!rsB         z+DataCollatorForWholeWordMask.tf_mask_tokensc sjjdur tdt|}|t}fdd|D}d|tj|td<jj dur8|jj k}d||<d||<j rQj j dj |jd t|@}ntjj dj |jd t|@}jjj||<j dksujdkry||fSdj }j|} j rj j d| |jd t|@|@} j jdtj|jtjd } n!tjj d| |jd t|@|@} tjjdtj|jtjd } | | || <||fS) rLNrMcrr r r r r r!r=rzBDataCollatorForWholeWordMask.numpy_mask_tokens..rrBrjrxrr )r+rrrrr#rhryrbrrrr$rrrrrr%r~r\r) rrr2rDrrrOrrrrrr r r!rsR           z.DataCollatorForWholeWordMask.numpy_mask_tokensN)rA)r$r%r&r4rr rOrrr5rrrr@r/r rrrr r r r!r's44!4#!(8"z/DataCollatorForSOP.__call__..cSr@)token_type_idsr rr r r!r=&r>T)Z batch_firstZ padding_valuecSr@)sentence_order_labelr rr r r!r=*r>)rwrDattention_maskrVrW)rKZtorch.nn.utils.rnnrUrr+ mask_tokensrrS) rrrKrUrwrDrXrVZsop_label_listrWr r r!r"s   zDataCollatorForSOP.__call__rc sXddl}jjdurtd|}||jj}fdd|D}|j |j ||j dddjj durF| jj}|j |dd|| }|}jj durf| jj} |j | d dd ||<|||jd  |@} jjj|| <|||jd  |@| @} |jtj|j|jd} | | || <|||fS) z Prepare masked tokens inputs/labels/attention_mask for masked language modeling: 80% MASK, 10% random, 10% original. N-gram not applied yet. rNrMcrr r r r r r!r=Erz2DataCollatorForSOP.mask_tokens..rBrrg?rjrg?)rKr+rrrrrrryrrRrhrrNrrrQrrr~rP) rrrKrDrrrOrrXZattention_padding_maskrrrr r r!rY5s4      "  zDataCollatorForSOP.mask_tokensN) r$r%r&r4rTrrr5rr"r rYr r r r!rPs & rPc@s4eZdZUdZeed<dZeed<dZe ed<dZ e ed<d e e e e eee effd ee effd d Zd e e e e eee effd ee effd dZd e e e e eee effd ee effddZded eeeeeffddZded eeeeeffddZded eeeeeffddZdS)*DataCollatorForPermutationLanguageModelingz Data collator used for permutation language modeling. - collates batches of tensors, honoring their tokenizer's pad_token - preprocesses batches for permutation language modeling with procedures specific to XLNet r+gUUUUUU?plm_probabilitymax_span_lengthrrrr.cCHt|dtrdd|D}t||j}||\}}}}||||dS)NrcSr@r)r rr r r!r=qr>zIDataCollatorForPermutationLanguageModeling.torch_call..rw perm_masktarget_mappingrD)rLrrr+rrrrWrr`rarDr r r!ro  z5DataCollatorForPermutationLanguageModeling.torch_callcCr^)NrcSr@r)r rr r r!r=xr>zFDataCollatorForPermutationLanguageModeling.tf_call..r_)rLrrr+rrbr r r!rvrcz2DataCollatorForPermutationLanguageModeling.tf_callcCr^)NrcSr@r)r rr r r!r=r>zIDataCollatorForPermutationLanguageModeling.numpy_call..r_)rLrrr+rrbr r r!r}rcz5DataCollatorForPermutationLanguageModeling.numpy_callrcsddl}jjdurtd|dddkrtd|}|j|jd|jd}|j |d|d|df|j d}t |dD]J}d}|d}||kr| dj dd} t| j} || | | dd} d||| | | f<|| 7}||ksQ||d||<qD|jfd d |D|jd} |j| d d jjdur|jj} |j| d d | | B}jj||<d ||<|j |d|d|df|j d}t |dD]V}||d}|d|ddfdd}|||dd}||dd}|||||@d||ddf|d|dfk||@||<q||||fS)g The masked tokens to be predicted for a particular sequence are determined by the following algorithm: 0. Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far). 1. Sample a `span_length` from the interval `[1, max_span_length]` (length of span of tokens to be masked) 2. Reserve a context of length `context_length = span_length / plm_probability` to surround span to be masked 3. Sample a starting point `start_index` from the interval `[cur_len, cur_len + context_length - span_length]` and mask tokens `start_index:start_index + span_length` 4. Set `cur_len = cur_len + context_length`. If `cur_len < max_len` (i.e. there are tokens remaining in the sequence to be processed), repeat from Step 1. rNThis tokenizer does not have a mask token which is necessary for permutation language modeling. Please add a mask token if you want to use this tokenizer.rxr This collator requires that sequence lengths be even to create a leakage-free perm_mask. Please see relevant comments in source code for details.rB)rxcrr r r r r r!r=rzPDataCollatorForPermutationLanguageModeling.torch_mask_tokens..rrrjr)rKr+rrrrrrrhrr]rrr]rNrOr[eyerRryrrrNrraranger transposer>flattenrP)rrrKrDrrarcur_lenmax_len span_lengthcontext_length start_indexrrO non_func_maskr` perm_indexr r r!rsZ (      ( & z.rjrrr)"rZr+rrrrrras_listrhrr]rr~rr]rOr[rgrrarryrrrrrirrr=rrSr\)rrrrDrZ labels_shaperarrkrlrmrnrorrOrpr`rqr r r!rsj   "           &*z9DataCollatorForPermutationLanguageModeling.tf_mask_tokenscsRjjdur td|jdddkrtdt|}tj|jdtd}tj|jd|jd|jdftj d}t |jdD]C}d}|jd}||kryt dj d}t |j} |t d| |d} d||| | |f<|| 7}||ksMt|jd||<q@tjfdd |Dtd} d|| <jjdur|jjk} d || <| | B} jj||<d ||<tj|jd|jd|jdftj d}t |jdD]G}t|jd}|d |jddfj}tj||j}d |||| |@<||jddf|d|jdfk||@||<q|tj|||tjfS) rdNrerxr rrfrBcrr r r r r r!r=rzPDataCollatorForPermutationLanguageModeling.numpy_mask_tokens..rrjr)r+rrrrrrrhrr]rrr]rOr[rgrbryrrrrhrTrr=rjr#r\)rrrDrrarrkrlrmrnrorrOrpr`rqr r r!rRsX  (       (   & z Using `DataCollatorWithFlattening` will flatten the entire mini batch into single long sequence. Make sure your attention computation is able to handle it! TrjF)return_position_ids separator_idreturn_flash_attn_kwargsreturn_seq_idxcsLtj|i|||_||_||_||_hd|_hd|_ddh|_dS)N>rw position_idsrD>seq_idxrwryrD max_length_q max_length_k) superrTrurvrwrx _int_64_keys_batch_dim_keys _py_int_keys)rrurvrwrxrRrSrr r!rTs   z#DataCollatorWithFlattening.__init__Ncs|dur|j}|dur|j}d|dv}ggd}|jr#|dgi|jr-|dgi|jr5dg}d}t|D]o\}|d} |d| 7<|r]|d|g|ddd7<n|d|g| dd7<|jr}|dttt | 7<|jr|dfdd tt | D7<|jr| |d t | t |t | }q9|jr||d <|d <||d <|d<|dkrddl } | j } | j} | j} n|dkrtj} tj} tj} ntd|d|D]"\}}||jvr|g}||jvr | |||jvr| n| d||<q|S)NrDrr-ryrzrwrxcsg|]}qSr r )r;_rzr r!r=sz7DataCollatorWithFlattening.__call__..rZ cu_seq_lens_qZ cu_seq_lens_kr{r|rrz;return_tensors must be one of ("pt", "np"), return_tensors=z not suportedrB)rrvruupdaterxrwrr`rr~rrrKrRr\rrrbrrTrrr~)rrrrvZis_labels_providedrWZ cu_seq_lensresamplerwrKZdata_clsZdtype_64Zdtype_32rJrXr rr!r"s^  $&   z#DataCollatorWithFlattening.__call__)NN)r$r%r&r4rTr" __classcell__r r rr!rtsrt)rr#)3multiprocessingrrrDcollections.abcr dataclassesrrtypingrrrrr r r r rrZ models.bertrrZtokenization_utils_baserrrrr5rrr-r2r3r/r0r1rcrirOrrrrrrr'ryrPrZrtr r r r!sj   (     $) 3 %& Z"lRT