o Zh7@sddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z mZddlmZddlmZeeZdZd d iZgd Zed d GdddeZdgZdS)N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)logging)requiresu▁ vocab_filezsentencepiece.bpe.model)Zar_ARcs_CZde_DEen_XXZes_XXet_EEfi_FIZfr_XXgu_INhi_INit_ITZja_XXkk_KZko_KRlt_LTlv_LVZmy_MMne_NPZnl_XXro_ROru_RUsi_LKtr_TRvi_VNzh_CN) sentencepiece)backendsc seZdZUdZeZddgZgZee e d<gZ ee e d<      dGd e e eefffd d ZddZddZeddZedefddZejdedd fddZ dHdee de ee dedee ffdd Z dIdee de ee dee fd d!Z dIdee de ee dee fd"d#Zd$ed%e ed&e efd'd(Zd)d*Zd+edeefd,d-Zd.d/Zd0d1Zd2d3Z dId4ed5e ede!efd6d7Z" 8 9dJd:eed%ed;e eed&ede#f fdd?Z%d@dAZ&dKdBdCZ'dDedd fdEdFZ(Z)S)LMBartTokenizeruT Construct an MBART tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import MBartTokenizer >>> tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO") >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt") ```Z input_idsZattention_mask prefix_tokens suffix_tokensNsp_model_kwargsc s~t|tr t|dddn|}| durin| _tjdij_jt||_ddddd_ d_ t j_ fd d t tD_d d jD_t jt jj j d <j jd d j D_tj| durfdd| Dtjd|||||||d| | jd || dur| nd_jj_| _jdS)NTF)lstrip normalizedrr)r'r*r(r)cs"i|] \}}|j|jqS) sp_model_sizefairseq_offset).0icodeselfr1[/var/www/auris/lib/python3.10/site-packages/transformers/models/mbart/tokenization_mbart.py jsz+MBartTokenizer.__init__..cSi|]\}}||qSr1r1r4kvr1r1r9r:mr+cSr;r1r1r<r1r1r9r:qr?csg|]}|vr|qSr1r1)r4t)_additional_special_tokensr1r9 wz+MBartTokenizer.__init__..) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokentokenizer_filesrc_langtgt_langadditional_special_tokensr,rr1) isinstancestrr r,spmSentencePieceProcessorsp_modelLoadrfairseq_tokens_to_idsr3lenr2 enumerateFAIRSEQ_LANGUAGE_CODESlang_code_to_iditemsZid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysextendsuper__init__ _src_langZcur_lang_code_idrMset_src_lang_special_tokens)r8rrDrErGrHrFrIrJrKrLrMr,rNkwargs __class__)rAr8r9raAsR     zMBartTokenizer.__init__cCs$|j}d|d<|j|d<|S)NrSsp_model_proto)__dict__copyrSserialized_model_proto)r8stater1r1r9 __getstate__s zMBartTokenizer.__getstate__cCs<||_t|ds i|_tjdi|j|_|j|jdS)Nr,r1)rhhasattrr,rQrRrSZLoadFromSerializedProtorg)r8dr1r1r9 __setstate__s  zMBartTokenizer.__setstate__cCst|jt|j|jdS)Nr/)rVrSrYr3r7r1r1r9 vocab_sizeszMBartTokenizer.vocab_sizereturncCs|jSN)rbr7r1r1r9rLszMBartTokenizer.src_lang new_src_langcCs||_||jdSrr)rbrc)r8rsr1r1r9rLsF token_ids_0 token_ids_1already_has_special_tokenscsx|r tj||ddSdgt|j}dgt|j}|dur*|dgt||S|dgt|dgt||S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rtrurvr/Nr)r`get_special_tokens_maskrVr%r&)r8rtrurvZ prefix_onesZ suffix_onesrer1r9rws$z&MBartTokenizer.get_special_tokens_maskcCs,|dur |j||jS|j|||jS)ab Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An MBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)r%r&)r8rtrur1r1r9 build_inputs_with_special_tokenssz/MBartTokenizer.build_inputs_with_special_tokenscCsP|jg}|jg}|durt|||dgSt||||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. mBART does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)Z sep_token_idZ cls_token_idrV)r8rtrusepclsr1r1r9$create_token_type_ids_from_sequencess "z3MBartTokenizer.create_token_type_ids_from_sequencesreturn_tensorsrLrMcKsJ|dus|dur td||_||fd|d|}||}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)Zadd_special_tokensr|Zforced_bos_token_id) ValueErrorrLZconvert_tokens_to_ids)r8Z raw_inputsr|rLrM extra_kwargsZinputsZ tgt_lang_idr1r1r9_build_translation_inputss z(MBartTokenizer._build_translation_inputscs(fddtjD}|j|S)Ncsi|]}||qSr1)Zconvert_ids_to_tokens)r4r5r7r1r9r: rCz,MBartTokenizer.get_vocab..)rangerpr[Zadded_tokens_encoder)r8Zvocabr1r7r9 get_vocab s zMBartTokenizer.get_vocabtextcCs|jj|tdS)N)Zout_type)rSencoderP)r8rr1r1r9 _tokenizeszMBartTokenizer._tokenizecCs4||jvr |j|S|j|}|r||jS|jS)z0Converts a token (str) in an id using the vocab.)rUrSZ PieceToIdr3Z unk_token_id)r8tokenZspm_idr1r1r9_convert_token_to_ids   z#MBartTokenizer._convert_token_to_idcCs&||jvr |j|S|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)r\rSZ IdToPiecer3)r8indexr1r1r9_convert_id_to_tokens  z#MBartTokenizer._convert_id_to_tokencCsd|td}|S)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r8tokensZ out_stringr1r1r9convert_tokens_to_string!sz'MBartTokenizer.convert_tokens_to_stringsave_directoryfilename_prefixcCstj|std|ddStj||r|dndtd}tj|jtj|kr?tj |jr?t |j||fStj |jsgt |d}|j }||Wd|fS1sbwY|fS)NzVocabulary path (z) should be a directory-rrwb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathrisfileropenrSrjwrite)r8rrZout_vocab_filefiZcontent_spiece_modelr1r1r9save_vocabulary&s" (     zMBartTokenizer.save_vocabularyrr src_texts tgt_textsc s"||_||_tj||fi|Srr)rLrMr`prepare_seq2seq_batch)r8rrLrrMrdrer1r9r7sz$MBartTokenizer.prepare_seq2seq_batchcC ||jSrr)rcrLr7r1r1r9_switch_to_input_modeC z$MBartTokenizer._switch_to_input_modecCrrr)set_tgt_lang_special_tokensrMr7r1r1r9_switch_to_target_modeFrz%MBartTokenizer._switch_to_target_modecC$|j||_g|_|j|jg|_dS)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].NrYZ cur_lang_coder%Z eos_token_idr&)r8rLr1r1r9rcI z*MBartTokenizer.set_src_lang_special_tokenslangcCr)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].Nr)r8rr1r1r9rOrz*MBartTokenizer.set_tgt_lang_special_tokens) r'r(r(r'r)r*r+NNNNN)NFrr)rNr)rqN)*__name__ __module__ __qualname____doc__rZvocab_files_namesZmodel_input_namesr%rint__annotations__r&rrrPrrarlropropertyrprLsetterboolrwrxr{rrrrrrrrr rrrrcr __classcell__r1r1rer9r$%s  N             r$)rshutilrtypingrrrrrr"rQZtokenization_utilsr r r utilsr Zutils.import_utilsr Z get_loggerrrrrrXr$__all__r1r1r1r9s      2