fThJSSKrSSKJr SSKJrJrJrJrJr SSK r SSK J r J r Jr SSKJr SSKJr \R&"\5rSrS S S .r/S Q/S QS.rSSSSSSSS.r\"SS9"SS\55rS/rg)N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)logging)requiresu▁zsentencepiece.bpe.modelztokenizer.json) vocab_filetokenizer_file)__java__ __python__ __en_XX__)rrr__javascript____php____ruby____go__)basemultirrrrrrr)javapythonen_XX javascriptphprubygo) sentencepiece)backendsc ^\rSrSr%Sr\rSS/r/r\ \ \ S'/r \ \ \ S'S.S\ \\\44U4S jjjrS rS r\S 5r\S \4Sj5r\R.S\S S4Sj5rS/S\ \ S\ \ \ S\S \ \ 4U4SjjjrS0S\ \ S\ \ \ S \ \ 4SjjrS0S\ \ S\ \ \ S \ \ 4SjjrS\S\ \S\ \4SjrSrS\S \ \4SjrSrSr S r!S0S!\S"\ \S \"\4S#jjr#S1S$\ \S\S%\ \ \S\S \$4 U4S&jjjr%S'r&S(r'S2S)jr(S*\S S4S+jr)S*\S \4S,jr*S-r+U=r,$)3PLBartTokenizer2a Construct an PLBART tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Args: vocab_file (`str`): Path to the vocabulary file. src_lang (`str`, *optional*): A string representing the source language. tgt_lang (`str`, *optional*): A string representing the target language. bos_token (`str`, *optional*, defaults to `""`): The start of sequence token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. cls_token (`str`, *optional*, defaults to `""`): The cls token, which is a special token used as the first token for all tasks. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. mask_token(`str`, *optional*, defaults to `""`): The token used for masking values. This is the token used when training this model with masking tasks. This is only used in the `"base"` tokenizer type. For `"multi"` tokenizer, masking is never done for the downstream tasks. language_codes (`str`, *optional*, defaults to `"base"`): What language codes to use. Should be one of `"base"` or `"multi"`. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Examples: ```python >>> from transformers import PLBartTokenizer >>> tokenizer = PLBartTokenizer.from_pretrained("uclanlp/plbart-python-en_XX", src_lang="python", tgt_lang="en_XX") >>> example_python_phrase = "def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])" >>> expected_translation_english = "Returns the maximum value of a b c." >>> inputs = tokenizer(example_python_phrase, text_target=expected_translation_english, return_tensors="pt") ``` input_idsattention_mask prefix_tokens suffix_tokensNsp_model_kwargsc p>[U[5(a [USSS9OUnU c0OU UlUR U 5n UR U 5n [ R "S 0URD6UlURR[U55 Xl Xl [URnSSSSS.Ul SUl [UR5Ul[!U5VVs0sH#unnUURU-UR-_M% snnUlUR"R%5VVs0sH unnUU_M snnUlURS :XaE[UR5[UR"5-UR-URS 'URR)UR"5 URR%5VVs0sH unnUU_M snnUl[-UR"R/55nUb*UR1UVs/sH nUU;dM UPM sn5 URS :Xa>XlUR2bUR"UR2O UR2UlO*U bU OS UlUR"UR2Ul[6TU]p"S UUUUUUUU U U U UURUS .UD6 XlUR=UR25 gs snnfs snnfs snnfs snf)NTF)lstriprstriprr )rr) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokenlanguage_codesrsrc_langtgt_langadditional_special_tokensr*clean_up_tokenization_spaces) isinstancestrr r*!_convert_lang_code_special_formatspmSentencePieceProcessorsp_modelLoadrr<FAIRSEQ_LANGUAGE_CODESfairseq_tokens_to_idsfairseq_offsetlen sp_model_size enumeratelang_code_to_iditemsid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysextend _src_langcur_lang_code_idsuper__init__r>set_src_lang_special_tokens)selfrr5r6r8r9r7r:r;r<rr=r>r*r?r@kwargsfairseq_language_codesicodekv_additional_special_tokenst __class__s f/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/plbart/tokenization_plbart.pyrZPLBartTokenizer.__init__ws(KUU_adJeJeZ 4Fku %4%+>> >No 261E1E1K1K1MN1MA11MN   & (36t}}3EDL`L`Ha3adhdwdw3wD & &x 0 ""))$*>*>?7;7Q7Q7W7W7Y%Z7Ytq!ad7Y%Z"%)$*>*>*C*C*E%F" $ 0 & - -5]5qB\9\5]    & (%N8<8R$$T^^4X\XfXf  !*2)=X;DN$($8$8$HD !  !))&@ 00)E  $!  ((8]  O &[ ^s5*L!L'5L- L3L3c~URR5nSUS'URR5US'U$)NrGsp_model_proto)__dict__copyrGserialized_model_proto)r\states rf __getstate__PLBartTokenizer.__getstate__s; ""$ j"&--"F"F"H cXl[US5(d0Ul[R"S0URD6UlUR R UR5 g)Nr*rA)rjhasattrr*rErFrGLoadFromSerializedProtori)r\ds rf __setstate__PLBartTokenizer.__setstate__sR t.//#%D 22JT5I5IJ  --d.A.ABrpcURS:Xa;[UR5[UR5-UR-S-$[UR5[UR5-UR-$)Nrr.)r<rLrGrOrKr\s rf vocab_sizePLBartTokenizer.vocab_sizesn   & (DMM"S)=)=%>>ATATTWXX t}}%D,@,@(AADDWDWW WrpreturncUR$N)rWrxs rfr=PLBartTokenizer.src_langs ~~rp new_src_langchURU5nXlURUR5 gr})rDrWr[)r\rs rfr=r~s)==lK % ((8rp token_ids_0 token_ids_1already_has_special_tokensc >U(a[TU]XSS9$S/[UR5-nS/[UR5-nUcUS/[U5--U-$US/[U5--S/[U5--U-$)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrr.r)rYget_special_tokens_maskrLr(r))r\rrr prefix_ones suffix_onesres rfr'PLBartTokenizer.get_special_tokens_masks& &72']a3 cC 2 233 cC 2 233  1#K(8"89KG GqcC $445!s;?O9OPS^^^rpc~UcURU-UR-$URU-U-UR-$)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An PLBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r(r))r\rrs rf build_inputs_with_special_tokens0PLBartTokenizer.build_inputs_with_special_tokenssG,  %% 3d6H6HH H!!K/+=@R@RRRrpcUR/nUR/nUc[XA-U-5S/-$[XA-U-U-U-U-5S/-$)a{ Create a mask from the two sequences passed to be used in a sequence-pair classification task. PLBart does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. r) sep_token_id cls_token_idrL)r\rrsepclss rf$create_token_type_ids_from_sequences4PLBartTokenizer.create_token_type_ids_from_sequences1si"  !  !  s(3./1#5 53$s*S0;>DEKKrpreturn_tensorsr=r>c UbUc [S5eURU5UlURU5UlU"U4SUS.UD6nUR UR5nXvS'U$)zIUsed by translation pipeline, to prepare inputs for the generate functionzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensrforced_bos_token_id) ValueErrorrDr=r>convert_tokens_to_ids)r\ raw_inputsrr=r> extra_kwargsinputs tgt_lang_ids rf_build_translation_inputs)PLBartTokenizer._build_translation_inputsIsx  x/`a a>>xH >>xH jiT.i\hi00? (3$% rpc[UR5Vs0sHoRU5U_M nnURUR5 U$s snfr})rangeryconvert_ids_to_tokensrRadded_tokens_encoder)r\r_vocabs rf get_vocabPLBartTokenizer.get_vocabVsL;@;QR;Qa++A.1;QR T../ SsAtextc>URRU[S9$)N)out_type)rGencoderC)r\rs rf _tokenizePLBartTokenizer._tokenize[s}}##D3#77rpcXR;aURU$URRU5nU(aX R-$UR$)z0Converts a token (str) in an id using the vocab.)rJrG PieceToIdrK unk_token_id)r\tokenspm_ids rf_convert_token_to_id$PLBartTokenizer._convert_token_to_id^sQ .. .--e4 4((/06v+++L4;L;LLrpcXR;aURU$URRXR- 5$)z=Converts an index (integer) in a token (str) using the vocab.)rSrG IdToPiecerK)r\indexs rf_convert_id_to_token$PLBartTokenizer._convert_id_to_tokengs= .. .--e4 4}}&&u/B/B'BCCrpclSRU5R[S5R5nU$)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r\tokens out_strings rfconvert_tokens_to_string(PLBartTokenizer.convert_tokens_to_stringms,WWV_,,-=sCIIK rpsave_directoryfilename_prefixc[RRU5(d[R SUS35 g[RR X(aUS-OS[ S-5n[RRUR5[RRU5:waG[RRUR5(a[URU5 U4$[RRUR5(dC[US5nURR5nURU5 SSS5 U4$U4$!,(df  U4$=f)NzVocabulary path (z) should be a directory-rrwb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathrisfileropenrGrlwrite)r\rrout_vocab_fileficontent_spiece_models rfsave_vocabularyPLBartTokenizer.save_vocabularyrs,ww}}^,, LL,^,<URU5UlURU5Ul[TU]"X40UD6$r})rDr=r>rYprepare_seq2seq_batch)r\rr=rr>r]res rfr%PLBartTokenizer.prepare_seq2seq_batchs>>>xH >>xH w,YLVLLrpc8URUR5$r})r[r=rxs rf_switch_to_input_mode%PLBartTokenizer._switch_to_input_mode// >>rpc8URUR5$r})set_tgt_lang_special_tokensr>rxs rf_switch_to_target_mode&PLBartTokenizer._switch_to_target_moderrpcURU5nUbURUOSUl/UlURbURUR/UlgUR/Ulg)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].NrDrO cur_lang_coder( eos_token_idr))r\r=s rfr[+PLBartTokenizer.set_src_lang_special_tokenssk99(C?G?ST11(;Y]    )"&"3"3T5G5G!HD "&"3"3!4D rplangcURU5nUbURUOSUl/UlURbURUR/UlgUR/Ulg)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].Nrr\rs rfr+PLBartTokenizer.set_tgt_lang_special_tokenssk55d;;?;KT11$7QU    )"&"3"3T5G5G!HD "&"3"3!4D rpcPU[R5;a [UnU$UnU$)z;Convert Language Codes to format tokenizer uses if required)FAIRSEQ_LANGUAGE_CODES_MAPrUrs rfrD1PLBartTokenizer._convert_lang_code_special_formats137;U;Z;Z;\3\)$/ cg rp)rjrWrrXrSrKrJrQrOr<r(rGr*rMr=r)r>r)r0r2r2r0r3r1r4rNNNNNT)NFr})rNr)r{N)-__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesr(rint__annotations__r)rrrCrrZrnrupropertyryr=setterboolrrrrrrrrrrrr rrrr[rrD__static_attributes__ __classcell__)res@rfr$r$2s; z*$&67!M49!!M49! 48"&%)!\9"$sCx.1\9\9| CXX#__9S9T99 sx_9_3;DI3F_ko_ c__>JNS9S3;DI3FS cS8JNL9L3;DI3FL cL0 *- 9A# RZ[^R_  8c8d3i8MD  !c!HSM!]bcf]g!( )- M9 M MDI& M  M  M M??5 5 5 5ccrpr$)rshutilrtypingrrrrrr!rEtokenization_utilsr r r utilsr utils.import_utilsr get_loggerrrrrrIrr$__all__rArprfrs  33PP*   H %#