o Zh+@sddlZddlmZddlmZmZmZddlmZddl m Z m Z ddl m Z ddlmZmZer:d d lmZndZeeZd d d ZgdZGddde ZdgZdS)N)copyfile)ListOptionalTuple) processors) AddedToken BatchEncoding)PreTrainedTokenizerFast)is_sentencepiece_availablelogging)MBartTokenizerzsentencepiece.bpe.modelztokenizer.json) vocab_filetokenizer_file)Zar_ARcs_CZde_DEen_XXZes_XXet_EEfi_FIZfr_XXgu_INhi_INit_ITZja_XXkk_KZko_KRlt_LTlv_LVZmy_MMne_NPZnl_XXro_ROru_RUsi_LKtr_TRvi_VNzh_CNc seZdZUdZeZddgZeZgZ e e e d<gZ e e e d<         d3fd d ZedefddZedefddZejdeddfddZ d4de e dee e de e fddZ d4de e dee e de e fddZdedeedeefddZ  !d5d"e eded#ee ededef fd$d% Zd&d'Zd(d)Zd6d*d+Zd,eddfd-d.Zd4d/ed0eedeefd1d2Z Z!S)7MBartTokenizerFastuO Construct a "fast" MBART tokenizer (backed by HuggingFace's *tokenizers* library). Based on [BPE](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models). This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import MBartTokenizerFast >>> tokenizer = MBartTokenizerFast.from_pretrained( ... "facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO" ... ) >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt") ```Z input_idsZattention_mask prefix_tokens suffix_tokensNc  st| tr t| dddn| } t| dur"fdd| Dtjd ||||||||| | | d | |_fddtD_ | durK| nd _  j _ | _ j dS) NTF)lstriprstripcsg|]}|vr|qSr.).0t)_additional_special_tokensr.`/var/www/auris/lib/python3.10/site-packages/transformers/models/mbart/tokenization_mbart_fast.py asz/MBartTokenizerFast.__init__..) rr bos_token eos_token sep_token cls_token unk_token pad_token mask_tokensrc_langtgt_langadditional_special_tokenscsi|]}||qSr.)convert_tokens_to_ids)r/Z lang_codeselfr.r2 us z/MBartTokenizerFast.__init__..rr.) isinstancestrrFAIRSEQ_LANGUAGE_CODEScopyextendsuper__init__rZlang_code_to_id _src_langr> cur_lang_coder<set_src_lang_special_tokens)r@rrr4r5r6r7r8r9r:r;r<r=kwargs __class__)r1r@r2rHIs<  zMBartTokenizerFast.__init__returncCs|jr tj|jSdS)NF)rospathisfiler?r.r.r2can_save_slow_tokenizer~sz*MBartTokenizerFast.can_save_slow_tokenizercCs|jSN)rIr?r.r.r2r;szMBartTokenizerFast.src_lang new_src_langcCs||_||jdSrT)rIrK)r@rUr.r.r2r;s token_ids_0 token_ids_1cCs,|dur |j||jS|j|||jS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. The special tokens depend on calling set_lang. An MBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: list of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)r%r&)r@rVrWr.r.r2 build_inputs_with_special_tokenssz3MBartTokenizerFast.build_inputs_with_special_tokenscCsP|jg}|jg}|durt|||dgSt||||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. mBART does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)Z sep_token_idZ cls_token_idlen)r@rVrWsepclsr.r.r2$create_token_type_ids_from_sequencess "z7MBartTokenizerFast.create_token_type_ids_from_sequencesreturn_tensorsr;r<cKsJ|dus|dur td||_||fd|d|}||}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)Zadd_special_tokensr]Zforced_bos_token_id) ValueErrorr;r>)r@Z raw_inputsr]r;r< extra_kwargsZinputsZ tgt_lang_idr.r.r2_build_translation_inputss z,MBartTokenizerFast._build_translation_inputsrr src_texts tgt_textsc s"||_||_tj||fi|SrT)r;r<rGprepare_seq2seq_batch)r@rar;rbr<rLrMr.r2rcsz(MBartTokenizerFast.prepare_seq2seq_batchcC ||jSrT)rKr;r?r.r.r2_switch_to_input_mode z(MBartTokenizerFast._switch_to_input_modecCrdrT)set_tgt_lang_special_tokensr<r?r.r.r2_switch_to_target_moderfz)MBartTokenizerFast._switch_to_target_modecC||||_g|_|j|jg|_||j}||j}tj|dg||ddg|tt |||j|jd|j _ dS)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].$A$BsinglepairZspecial_tokensN r>rJr%Z eos_token_idr&Zconvert_ids_to_tokensrZTemplateProcessinglistzipZ _tokenizerZpost_processor)r@r;prefix_tokens_strsuffix_tokens_strr.r.r2rK    z.MBartTokenizerFast.set_src_lang_special_tokenslangcCri)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].rjrkrlNro)r@rurrrsr.r.r2rgrtz.MBartTokenizerFast.set_tgt_lang_special_tokenssave_directoryfilename_prefixcCs~|jstdtj|std|ddStj||r"|dndtd}tj |j tj |krs       h