fTh SrSSKrSSKJrJr SSKJr SSKJr \R"\ 5r SS0r S r "S S \5rS /rg) zTokenization classes for ESM.N)ListOptional)PreTrainedTokenizer)logging vocab_file vocab.txtc[US5nUR5R5nUVs/sHo3R5PM snsSSS5 $s snf!,(df  g=f)Nr)openread splitlinesstrip)rflinesls `/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/esm/tokenization_esm.pyload_vocab_filersL j# !##%#()5a 5)  )  s#AA AA A(c ^\rSrSrSr\rSS/rSU4SjjrS\ S\ 4Sjr S \ S\ 4S jr S r S rS \ S\ 4S jrS\ S\ 4SjrSS\\ S\\\ S\\ 4SjjrSS\S\\S\S\\ 4SjjrSr\S\ 4Sj5rSrU=r$) EsmTokenizer#z Constructs an ESM tokenizer. input_idsattention_maskc Z>[U5Ul[[UR55Ul[UR5VV s0sHupX_M sn nUl[ T U]"SUUUUUS.UD6 URUlURUR5 gs sn nf)N) unk_token cls_token pad_token mask_token eos_token) r all_tokensdict enumerate _id_to_token _token_to_idsuper__init__unique_no_split_tokens _update_trie) selfrrrrrrkwargsindtok __class__s rr'EsmTokenizer.__init__+s**5 4??!;<6?6PQ6P(#SX6PQ  !    '+oo# $556Rs B'indexreturncLURRXR5$Nr$getrr*r0s r_convert_id_to_token!EsmTokenizer._convert_id_to_tokenG  $$UNN;;tokenc~URRXRRUR55$r3r%r5rr*r;s r_convert_token_to_id!EsmTokenizer._convert_token_to_idJ.  $$U,=,=,A,A$..,QRRr:c "UR5$r3)split)r*textr+s r _tokenizeEsmTokenizer._tokenizeMszz|r:cpURR5nURUR5 U$r3)r%copyupdateadded_tokens_encoder)r* base_vocabs r get_vocabEsmTokenizer.get_vocabPs0&&++- $334r:c~URRXRRUR55$r3r=r>s r token_to_idEsmTokenizer.token_to_idUrAr:cLURRXR5$r3r4r6s r id_to_tokenEsmTokenizer.id_to_tokenXr9r: token_ids_0 token_ids_1cUR/nUR/nUcURcX1-$X1-U-$URc [S5eX1-U-U-U-$)Nz=Cannot tokenize multiple sequences when EOS token is not set!) cls_token_id eos_token_id ValueError)r*rTrUclsseps r build_inputs_with_special_tokens-EsmTokenizer.build_inputs_with_special_tokens[sy  !  !    ((((3..    &\] ] 3&4s::r:already_has_special_tokenscU(a2Ub [S5eUVs/sHoDUR;aSOSPM sn$S/S/[U5--S/-nUbUS/[U5-S/-- nU$s snf)at Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`List[int]`): List of ids of the first sequence. token_ids_1 (`List[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. zYou should not supply a second sequence if the provided sequence of ids is already formatted with special tokens for the model.r)rYall_special_idslen)r*rTrUr^r;masks rget_special_tokens_mask$EsmTokenizer.get_special_tokens_maskis$ && R LWW;%$"6"66AA=;W WsqcC ,,-3  " QC#k**aS0 0D XsA,c[RRX(aUS-OSS-5n[US5nUR SRUR 55 SSS5 U4$!,(df  U4$=f)N-r w )ospathjoinr writer!)r*save_directoryfilename_prefixrrs rsave_vocabularyEsmTokenizer.save_vocabularysiWW\\.O?S3Hacgr2rs *c "a GGDIIdoo. /#}# "}s +A11 Bc,[UR5$r3)rbr!)r*s r vocab_sizeEsmTokenizer.vocab_sizes4??##r:)r$r%r!r()zzzzzr3)NF)__name__ __module__ __qualname____firstlineno____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr'intstrr7r?rErLrOrRrrr\boolrdrqpropertyrt__static_attributes__ __classcell__)r.s@rrr#s*$&67 78<#<#<S#S#S SSS<<<JN ;9 ;3;DI3F ; c ;in.6tnae c> $C$$r:r)rzrktypingrrtokenization_utilsrutilsr get_loggerrvloggerr{rr__all__r r:rrsR$ !5   H %!;/* m$&m$`  r: