o Zh @shdZddlZddlmZmZddlmZddlmZe e Z ddiZ d d Z Gd d d eZd gZdS) zTokenization classes for ESM.N)ListOptional)PreTrainedTokenizer)logging vocab_file vocab.txtcCsHt|d}|}dd|DWdS1swYdS)NrcSsg|]}|qS)strip).0lr r W/var/www/auris/lib/python3.10/site-packages/transformers/models/esm/tokenization_esm.py sz#load_vocab_file..)openread splitlines)rflinesr r rload_vocab_files   $rc seZdZdZeZddgZ     d'fd d Zd ed e fd dZ de d efddZ ddZ ddZ de d efddZd ed e fddZ d(deedeeed eefddZ d)dedeed ed eefd!d"Zd#d$Zed efd%d&ZZS)* EsmTokenizerz& Constructs an ESM tokenizer. Z input_idsZattention_maskc sft||_tt|j|_ddt|jD|_tjd|||||d||j|_| |jdS)NcSsi|]\}}||qSr r )r indtokr r r 7sz)EsmTokenizer.__init__..) unk_token cls_token pad_token mask_token eos_tokenr ) r all_tokensdict enumerate _id_to_token _token_to_idsuper__init__Zunique_no_split_tokensZ _update_trie)selfrrr r!r"r#kwargs __class__r rr*+s  zEsmTokenizer.__init__indexreturncC|j||jSNr'getrr+r/r r r_convert_id_to_tokenGz!EsmTokenizer._convert_id_to_tokentokencC|j||j|jSr2r(r4rr+r8r r r_convert_token_to_idJz!EsmTokenizer._convert_token_to_idcKs|Sr2)split)r+textr,r r r _tokenizeMszEsmTokenizer._tokenizecCs|j}||j|Sr2)r(copyupdateZadded_tokens_encoder)r+Z base_vocabr r r get_vocabPs  zEsmTokenizer.get_vocabcCr9r2r:r;r r r token_to_idUr=zEsmTokenizer.token_to_idcCr1r2r3r5r r r id_to_tokenXr7zEsmTokenizer.id_to_tokenN token_ids_0 token_ids_1cCs\|jg}|jg}|dur|jdur||S|||S|jdur$td|||||S)Nz=Cannot tokenize multiple sequences when EOS token is not set!)Z cls_token_idZ eos_token_id ValueError)r+rFrGclssepr r r build_inputs_with_special_tokens[s   z-EsmTokenizer.build_inputs_with_special_tokensFalready_has_special_tokenscsd|r|dur tdfdd|DSdgdgt|dg}|dur0|dgt|dg7}|S)a Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`List[int]`): List of ids of the first sequence. token_ids_1 (`List[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. NzYou should not supply a second sequence if the provided sequence of ids is already formatted with special tokens for the model.csg|] }|jvr dndqS)r)Zall_special_ids)r r8r+r rrsz8EsmTokenizer.get_special_tokens_mask..rMr)rHlen)r+rFrGrLmaskr rNrget_special_tokens_maskisz$EsmTokenizer.get_special_tokens_maskcCsdtj||r |dndd}t|d}|d|jWd|fS1s*wY|fS)N-rw )ospathjoinrwriter$)r+Zsave_directoryZfilename_prefixrrr r rsave_vocabularys  zEsmTokenizer.save_vocabularycCs t|jSr2)rOr$rNr r r vocab_sizes zEsmTokenizer.vocab_size)rrrrrr2)NF)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESZvocab_files_namesZmodel_input_namesr*intstrr6r<r@rCrDrErrrKboolrQrZpropertyr[ __classcell__r r r-rr#sJ   r)r_rVtypingrrZtokenization_utilsrutilsrZ get_loggerr\loggerr`rr__all__r r r rs    p