fTh7SSKrSSKJr SSKJrJrJrJrJr SSK r SSK J r J r Jr SSKJr SSKJr \R&"\5rSrS S 0r/S Qr\"S S 9"SS\55rS/rg)N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)logging)requiresu▁ vocab_filezsentencepiece.bpe.model)ar_ARcs_CZde_DEen_XXes_XXet_EEfi_FIfr_XXgu_INhi_INit_ITja_XXkk_KZko_KRlt_LTlv_LVmy_MMne_NPnl_XXro_ROru_RUsi_LKtr_TRvi_VNzh_CN) sentencepiece)backendsc ^\rSrSr%Sr\rSS/r/r\ \ \ S'/r \ \ \ S'S-S\ \\\44U4S jjjrS rS r\S 5r\S \4Sj5r\R.S\S S4Sj5rS.S\ \ S\ \ \ S\S \ \ 4U4SjjjrS/S\ \ S\ \ \ S \ \ 4SjjrS/S\ \ S\ \ \ S \ \ 4SjjrS\S\ \S\ \4SjrSrS\S \ \4SjrSrSr S r!S/S!\S"\ \S \"\4S#jjr#S0S$\ \S\S%\ \ \S\S \$4 U4S&jjjr%S'r&S(r'S1S)jr(S*\S S4S+jr)S,r*U=r+$)2MBartTokenizer%u  Construct an MBART tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import MBartTokenizer >>> tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO") >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt") ``` input_idsattention_mask prefix_tokens suffix_tokensNsp_model_kwargsc B>[U[5(a [USSS9OUnU c0OU Ul[R "S 0URD6UlUR R[U55 XlSSSSS.Ul SUl [UR 5Ul [[5VVs0sH#unnUURU-UR-_M% snnUlURR!5VVs0sH unnUU_M snnUl[UR 5[UR5-UR-URS 'URR%UR5 URR!5VVs0sH unnUU_M snnUl[)URR+55nU b*UR-U Vs/sH nUU;dM UPM sn5 [.TU]`"S UUUUUUUSU U UURS . UD6 U bU OS UlURUR2UlXlUR9UR25 gs snnfs snnfs snnfs snf) NTF)lstrip normalizedrr )) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokentokenizer_filesrc_langtgt_langadditional_special_tokensr2r) isinstancestrr r2spmSentencePieceProcessorsp_modelLoadrfairseq_tokens_to_idsfairseq_offsetlen sp_model_size enumerateFAIRSEQ_LANGUAGE_CODESlang_code_to_iditemsid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysextendsuper__init__ _src_langcur_lang_code_idrFset_src_lang_special_tokens)selfrr=r>r@rAr?rBrCrDrErFr2rGkwargsicodekv_additional_special_tokenst __class__s d/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/mbart/tokenization_mbart.pyr^MBartTokenizer.__init__Asz&FPPZ\_E`E`Jz$5 Afp &5%+>> >No 261E1E1K1K1MN1MA11MN/24==/ACH\H\D]/]`d`s`s/s""8, ""))$*>*>?7;7Q7Q7W7W7Y%Z7Ytq!ad7Y%Z"%)$*>*>*C*C*E%F" $ 0 & - -5]5qB\9\5]   !&@ 00  &.%9w $ 4 4T^^ D  ((8G  O&[ ^s>*J J.J< J Jc~URR5nSUS'URR5US'U$)NrMsp_model_proto)__dict__copyrMserialized_model_proto)rbstates rk __getstate__MBartTokenizer.__getstate__s; ""$ j"&--"F"F"H cXl[US5(d0Ul[R"S0URD6UlUR R UR5 g)Nr2rH)rohasattrr2rKrLrMLoadFromSerializedProtorn)rbds rk __setstate__MBartTokenizer.__setstate__sR t.//#%D 22JT5I5IJ  --d.A.ABrucx[UR5[UR5-UR-S-$)Nr6)rQrMrUrPrbs rk vocab_sizeMBartTokenizer.vocab_sizes24==!C(<(<$==@S@SSVWWWrureturncUR$N)r_r}s rkrEMBartTokenizer.src_langs ~~ru new_src_langcFXlURUR5 gr)r_ra)rbrs rkrErs% ((8ru token_ids_0 token_ids_1already_has_special_tokensc >U(a[TU]XSS9$S/[UR5-nS/[UR5-nUcUS/[U5--U-$US/[U5--S/[U5--U-$)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrr6r)r]get_special_tokens_maskrQr0r1)rbrrr prefix_ones suffix_onesrjs rkr&MBartTokenizer.get_special_tokens_masks& &72']a3 cC 2 233 cC 2 233  1#K(8"89KG GqcC $445!s;?O9OPS^^^ruc~UcURU-UR-$URU-U-UR-$)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An MBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r0r1)rbrrs rk build_inputs_with_special_tokens/MBartTokenizer.build_inputs_with_special_tokenssG,  %% 3d6H6HH H!!K/+=@R@RRRrucUR/nUR/nUc[XA-U-5S/-$[XA-U-U-U-U-5S/-$)a{ Create a mask from the two sequences passed to be used in a sequence-pair classification task. mBART does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. r) sep_token_id cls_token_idrQ)rbrrsepclss rk$create_token_type_ids_from_sequences3MBartTokenizer.create_token_type_ids_from_sequencessi$  !  !  s(3./1#5 53$s*S0;>DEKKrureturn_tensorsrErFc vUbUc [S5eX0lU"U4SUS.UD6nURU5nXvS'U$)zIUsed by translation pipeline, to prepare inputs for the generate functionzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensrforced_bos_token_id) ValueErrorrEconvert_tokens_to_ids)rb raw_inputsrrErF extra_kwargsinputs tgt_lang_ids rk_build_translation_inputs(MBartTokenizer._build_translation_inputssU  x/`a a jiT.i\hi00: (3$% ruc[UR5Vs0sHoRU5U_M nnURUR5 U$s snfr)ranger~convert_ids_to_tokensrXadded_tokens_encoder)rbrdvocabs rk get_vocabMBartTokenizer.get_vocab sL;@;QR;Qa++A.1;QR T../ SsAtextc>URRU[S9$)N)out_type)rMencoderJ)rbrs rk _tokenizeMBartTokenizer._tokenizes}}##D3#77rucXR;aURU$URRU5nU(aX R-$UR$)z0Converts a token (str) in an id using the vocab.)rOrM PieceToIdrP unk_token_id)rbtokenspm_ids rk_convert_token_to_id#MBartTokenizer._convert_token_to_idsQ .. .--e4 4((/06v+++L4;L;LLrucXR;aURU$URRXR- 5$)z=Converts an index (integer) in a token (str) using the vocab.)rYrM IdToPiecerP)rbindexs rk_convert_id_to_token#MBartTokenizer._convert_id_to_tokens= .. .--e4 4}}&&u/B/B'BCCruclSRU5R[S5R5nU$)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)rbtokens out_strings rkconvert_tokens_to_string'MBartTokenizer.convert_tokens_to_string!s,WWV_,,-=sCIIK rusave_directoryfilename_prefixc[RRU5(d[R SUS35 g[RR X(aUS-OS[ S-5n[RRUR5[RRU5:waG[RRUR5(a[URU5 U4$[RRUR5(dC[US5nURR5nURU5 SSS5 U4$U4$!,(df  U4$=f)NzVocabulary path (z) should be a directory-rrwb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathrisfileropenrMrqwrite)rbrrout_vocab_fileficontent_spiece_models rksave_vocabularyMBartTokenizer.save_vocabulary&s,ww}}^,, LL,^,<>X lX@l[TU] "X40UD6$r)rErFr]prepare_seq2seq_batch)rbrrErrFrcrjs rkr$MBartTokenizer.prepare_seq2seq_batch7s$!  w,YLVLLruc8URUR5$r)rarEr}s rk_switch_to_input_mode$MBartTokenizer._switch_to_input_modeC// >>ruc8URUR5$r)set_tgt_lang_special_tokensrFr}s rk_switch_to_target_mode%MBartTokenizer._switch_to_target_modeFrructURUUl/UlURUR/Ulg)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].NrU cur_lang_coder0 eos_token_idr1)rbrEs rkra*MBartTokenizer.set_src_lang_special_tokensIs6!11(;"//1C1CDrulangctURUUl/UlURUR/Ulg)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].Nr)rbrs rkr*MBartTokenizer.set_tgt_lang_special_tokensOs6!11$7"//1C1CDru)ror_rr`rYrPrOrWrUr0rMr2rRrEr1rFr) r8r:r:r8r;r9r<NNNNN)NFr)rNr#)rN),__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesr0rint__annotations__r1rrrJrr^rsrzpropertyr~rEsetterboolrrrrrrrrrrrr rrrrar__static_attributes__ __classcell__)rjs@rkr,r,%s (*$&67!M49!!M49! 48"&L9"$sCx.1L9L9\ CXX#__9S9T99 sx_9_3;DI3F_ko_ c__>JNS9S3;DI3FS cS8JNL9L3;DI3FL cL2 *- 9A# RZ[^R_  8c8d3i8MD  !c!HSM!]bcf]g!( )- M9 M MDI& M  M  M M??E EEEErur,)rshutilrtypingrrrrrr)rKtokenization_utilsr r r utilsr utils.import_utilsr get_loggerrrrrrTr,__all__rHrurkrs  33PP*   H %!#<={ %&mE(mE'mE`  ru