o ZhJ@sddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z mZddlmZddlmZeeZdZd d d Zgd gd dZddddddddZeddGdddeZdgZdS)N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)logging)requiresu▁zsentencepiece.bpe.modelztokenizer.json) vocab_filetokenizer_file)__java__ __python__ __en_XX__)rrr__javascript____php____ruby____go__)basemultirrrrrrr)javapythonen_XX javascriptphprubygo) sentencepiece)backendsc s2eZdZUdZeZddgZgZee e d<gZ ee e d<      dKde e eefffdd ZddZddZeddZedefddZejdedd fddZ dLdee de ee dedee ffd d! Z dMdee de ee dee fd"d#Z dMdee de ee dee fd$d%Zd&ed'e ed(e efd)d*Zd+d,Zd-edeefd.d/Zd0d1Zd2d3Zd4d5Z dMd6ed7e ede!efd8d9Z" : ;dNdd? Z$d@dAZ%dBdCZ&dOdDdEZ'dFedd fdGdHZ(dFedefdIdJZ)Z*S)PPLBartTokenizera Construct an PLBART tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Args: vocab_file (`str`): Path to the vocabulary file. src_lang (`str`, *optional*): A string representing the source language. tgt_lang (`str`, *optional*): A string representing the target language. bos_token (`str`, *optional*, defaults to `""`): The start of sequence token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. cls_token (`str`, *optional*, defaults to `""`): The cls token, which is a special token used as the first token for all tasks. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. mask_token(`str`, *optional*, defaults to `""`): The token used for masking values. This is the token used when training this model with masking tasks. This is only used in the `"base"` tokenizer type. For `"multi"` tokenizer, masking is never done for the downstream tasks. language_codes (`str`, *optional*, defaults to `"base"`): What language codes to use. Should be one of `"base"` or `"multi"`. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Examples: ```python >>> from transformers import PLBartTokenizer >>> tokenizer = PLBartTokenizer.from_pretrained("uclanlp/plbart-python-en_XX", src_lang="python", tgt_lang="en_XX") >>> example_python_phrase = "def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])" >>> expected_translation_english = "Returns the maximum value of a b c." >>> inputs = tokenizer(example_python_phrase, text_target=expected_translation_english, return_tensors="pt") ```Z input_idsZattention_mask prefix_tokens suffix_tokensrNTsp_model_kwargsc st|tr t|dddn|}| durin| _| } | } tjdij_jt||_ | _ t j }ddddd_ d_ tj_fd d t|D_d d jD_j d kr{tjtjj j d <j jdd j D_tj|durfdd|Dj d kr| _jdurjjnj_n| dur| nd_jj_tjd|||||||| | | | j|d|| _jdS)NTF)lstriprstriprr)r%r(r&r'cs"i|] \}}|j|jqS) sp_model_sizefairseq_offset).0icodeselfr/]/var/www/auris/lib/python3.10/site-packages/transformers/models/plbart/tokenization_plbart.py sz,PLBartTokenizer.__init__..cSi|]\}}||qSr/r/r2kvr/r/r7r8rr)cSr9r/r/r:r/r/r7r8r=csg|]}|vr|qSr/r/)r2t)_additional_special_tokensr/r7 z,PLBartTokenizer.__init__..r) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokenlanguage_codesrsrc_langtgt_langadditional_special_tokensr*clean_up_tokenization_spacesr/) isinstancestrr r*!_convert_lang_code_special_formatspmSentencePieceProcessorsp_modelLoadrrIFAIRSEQ_LANGUAGE_CODESfairseq_tokens_to_idsr1lenr0 enumeratelang_code_to_iditemsZid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysextend _src_langZcur_lang_code_idsuper__init__rKset_src_lang_special_tokens)r6rrBrCrErFrDrGrHrIrrJrKr*rLrMkwargsZfairseq_language_codes __class__)r?r6r7rbwsf         zPLBartTokenizer.__init__cCs$|j}d|d<|j|d<|S)NrSsp_model_proto)__dict__copyrSserialized_model_proto)r6stater/r/r7 __getstate__s zPLBartTokenizer.__getstate__cCs<||_t|ds i|_tjdi|j|_|j|jdS)Nr*r/)rhhasattrr*rQrRrSZLoadFromSerializedProtorg)r6dr/r/r7 __setstate__s  zPLBartTokenizer.__setstate__cCsB|jdkrt|jt|j|jdSt|jt|j|jS)Nrr-)rIrWrSrYr1r5r/r/r7 vocab_sizes zPLBartTokenizer.vocab_sizereturncCs|jSN)r`r5r/r/r7rJszPLBartTokenizer.src_lang new_src_langcCs ||}||_||jdSrr)rPr`rc)r6rsr/r/r7rJs F token_ids_0 token_ids_1already_has_special_tokenscsx|r tj||ddSdgt|j}dgt|j}|dur*|dgt||S|dgt|dgt||S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rtrurvr-Nr)raget_special_tokens_maskrWr#r$)r6rtrurvZ prefix_onesZ suffix_onesrer/r7rws$z'PLBartTokenizer.get_special_tokens_maskcCs,|dur |j||jS|j|||jS)ac Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An PLBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)r#r$)r6rtrur/r/r7 build_inputs_with_special_tokenssz0PLBartTokenizer.build_inputs_with_special_tokenscCsP|jg}|jg}|durt|||dgSt||||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. PLBart does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)Z sep_token_idZ cls_token_idrW)r6rtrusepclsr/r/r7$create_token_type_ids_from_sequences1s "z4PLBartTokenizer.create_token_type_ids_from_sequencesreturn_tensorsrJrKcKs^|dus|dur td|||_|||_||fd|d|}||j}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)Zadd_special_tokensr|Zforced_bos_token_id) ValueErrorrPrJrKZconvert_tokens_to_ids)r6Z raw_inputsr|rJrK extra_kwargsZinputsZ tgt_lang_idr/r/r7_build_translation_inputsIs   z)PLBartTokenizer._build_translation_inputscs(fddtjD}|j|S)Ncsi|]}||qSr/)Zconvert_ids_to_tokens)r2r3r5r/r7r8WrAz-PLBartTokenizer.get_vocab..)rangerpr[Zadded_tokens_encoder)r6Zvocabr/r5r7 get_vocabVs zPLBartTokenizer.get_vocabtextcCs|jj|tdS)N)Zout_type)rSencoderO)r6rr/r/r7 _tokenize[szPLBartTokenizer._tokenizecCs4||jvr |j|S|j|}|r||jS|jS)z0Converts a token (str) in an id using the vocab.)rVrSZ PieceToIdr1Z unk_token_id)r6tokenZspm_idr/r/r7_convert_token_to_id^s   z$PLBartTokenizer._convert_token_to_idcCs&||jvr |j|S|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)r\rSZ IdToPiecer1)r6indexr/r/r7_convert_id_to_tokengs  z$PLBartTokenizer._convert_id_to_tokencCsd|td}|S)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r6tokensZ out_stringr/r/r7convert_tokens_to_stringmsz(PLBartTokenizer.convert_tokens_to_stringsave_directoryfilename_prefixcCstj|std|ddStj||r|dndtd}tj|jtj|kr?tj |jr?t |j||fStj |jsgt |d}|j }||Wd|fS1sbwY|fS)NzVocabulary path (z) should be a directory-rrwb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathrisfileropenrSrjwrite)r6rrZout_vocab_filefiZcontent_spiece_modelr/r/r7save_vocabularyrs" (     zPLBartTokenizer.save_vocabularyrr src_texts tgt_textsc s.|||_|||_tj||fi|Srr)rPrJrKraprepare_seq2seq_batch)r6rrJrrKrdrer/r7rs  z%PLBartTokenizer.prepare_seq2seq_batchcC ||jSrr)rcrJr5r/r/r7_switch_to_input_mode z%PLBartTokenizer._switch_to_input_modecCrrr)set_tgt_lang_special_tokensrKr5r/r/r7_switch_to_target_moderz&PLBartTokenizer._switch_to_target_modecCR||}|dur|j|nd|_g|_|jdur"|j|jg|_dS|jg|_dS)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].NrPrYZ cur_lang_coder#Z eos_token_idr$)r6rJr/r/r7rcs  z+PLBartTokenizer.set_src_lang_special_tokenslangcCr)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].Nrr6rr/r/r7rs  z+PLBartTokenizer.set_tgt_lang_special_tokenscCs |tvr t|}|S|}|S)z;Convert Language Codes to format tokenizer uses if required)FAIRSEQ_LANGUAGE_CODES_MAPr^rr/r/r7rPsz1PLBartTokenizer._convert_lang_code_special_format)r%r&r&r%r'r(r)rNNNNNT)NFrr)rNr)rqN)+__name__ __module__ __qualname____doc__rZvocab_files_namesZmodel_input_namesr#rint__annotations__r$rrrOrrbrlropropertyrprJsetterboolrwrxr{rrrrrrrrr rrrrcrrP __classcell__r/r/rer7r"2s =^               r")rshutilrtypingrrrrrr rQZtokenization_utilsr r r utilsr Zutils.import_utilsr Z get_loggerrrrrrUrr"__all__r/r/r/r7s4