eTh[JSSKJr SSKJr SSKJr SSKJrJr S\S\ 4Sjr g) )Path)Any)TikTokenConverter)TIKTOKEN_VOCAB_FILETOKENIZER_FILEencoding output_dirc[U5nURSS9 US- [- nU[- n[ UR 55n[ UR 55nSSKJn SSKJ n [U[5(aU"U5nU"URU5 [X@RUR S9R#5nUR%U5 g ![a [S5ef=f) a Converts given `tiktoken` encoding to `PretrainedTokenizerFast` and saves the configuration of converted tokenizer on disk. Args: encoding (`str` or `tiktoken.Encoding`): Tokenizer from `tiktoken` library. If `encoding` is `str`, the tokenizer will be loaded with `tiktoken.get_encoding(encoding)`. output_dir (`str`): Save path for converted tokenizer configuration file. T)exist_oktiktokenr) get_encoding)dump_tiktoken_bpezY`tiktoken` is required to save a `tiktoken` file. Install it with `pip install tiktoken`.) vocab_filepatternadditional_special_tokensN)rmkdirrrstrabsoluter r tiktoken.loadr isinstance_mergeable_ranks ImportError ValueErrorr_pat_str_special_tokens convertedsave) rr save_filetokenizer_filesave_file_absoluteoutput_file_absoluter r tokenizers Z/var/www/auris/envauris/lib/python3.13/site-packages/transformers/integrations/tiktoken.pyconvert_tiktoken_to_fastr$sj!Jd#Z'*==I.0NY//12~6689 v)3 h $ $#H-H(335GH"%/@/@\d\t\tikNN'( vtuuvs #r+s$AT#)s#)#)r*