fTh3SrSSKrSSKrSSKJrJrJr SSKJr SSK J r \ R"\ 5r SSS .rS r"S S \5rS /rg) z Tokenization classes for BioGPT.N)ListOptionalTuple)PreTrainedTokenizer)loggingz vocab.jsonz merges.txt) vocab_file merges_filecd[5nUSnUSSHnURX#45 UnM U$)zw Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rN)setadd)wordpairs prev_charchars f/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/biogpt/tokenization_biogpt.py get_pairsr!s? EEQIQR 9#$  Lc b^\rSrSrSr\rSS/rSU4Sjjr\ S5r Sr Sr S r S rSS jrS rS rSrSS\\S\\\S\\4SjjrSS\\S\\\S\S\\4U4SjjjrSS\\S\\\S\\4SjjrSS\S\\S\\4SjjrSrSrSrU=r $) BioGptTokenizer.a Construct an FAIRSEQ Transformer tokenizer. Moses tokenization followed by Byte-Pair Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Merges file. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc >SSKn SUlXl0Ul0Ul[ USS9n [R"U 5Ul SSS5 URR5V V s0sHupX_M sn n Ul [ USS9n U R5RS5SSnSSS5 WVs/sHn[UR5SS5PM! nn[[!U[#[%U5555Ul0Ul[*TU]X"S UUUUUS .UD6 g![a [S5ef=f!,(df  GN=fs sn n f!,(df  N=fs snf) NrzqYou need to install sacremoses to use BioGptTokenizer. See https://pypi.org/project/sacremoses/ for installation.enutf-8encoding ) bos_token eos_token sep_token unk_token pad_token) sacremoses ImportErrorlangsmcache_moses_tokenizercache_moses_detokenizeropenjsonloadencoderitemsdecoderreadsplittupledictziprangelen bpe_rankscachesuper__init__)selfr r r&r#r$r%r'kwargsr) vocab_handlekv merges_handlemergesmerge __class__s rr?BioGptTokenizer.__init__\sS   %'"')$ *w /<99\2DL0)-););)=>)=)=> + 0M"'')//5cr:F18>?u% bq)*?c&%F *<=>       - M  0 /> 0 0?s.D0E .E#E!>&E20E E! E/c,[UR5$)zReturns vocab size)r;r2r@s r vocab_sizeBioGptTokenizer.vocab_sizes4<<  rcB[UR40URD6$N)r8r2added_tokens_encoderrKs r get_vocabBioGptTokenizer.get_vocabsDLL>D$=$=>>rcX R;a'URRUS9nX0RU'URURUSSSS9$)Nr+TF)aggressive_dash_splits return_strescape)r-r,MosesTokenizertokenize)r@textr+moses_tokenizers rmoses_tokenizeBioGptTokenizer.moses_tokenizes_ 11 1"gg44$4?O/> & &t ,))$/88 %9  rcX R;a'URRUS9nX0RU'URURU5$)NrT)r.r,MosesDetokenizer detokenize)r@tokensr+moses_detokenizers rmoses_detokenize BioGptTokenizer.moses_detokenizesP 33 3 $ 8 8d 8 C 1B ( ( .++D1<cN>TRRU[S55$)Ninf)r<getfloat)pairr@s r%BioGptTokenizer.bpe..s1C1CD%PU,1Wrkeyrr r" z z ) r7r=rminr<r;indexextend ValueErrorappendjoin) r@tokenrrbigramfirstsecondnew_wordijs ` rbpeBioGptTokenizer.bpesU3BZ E"I$6#88 DJJ ::e$ $$6> !$WXFT^^+"MEHAc$i- 5,A OOD1I.A7e#CIM(9dq5kV>SOOEN3FAOODG,FAc$i- XHD4yA~!$9:xx~ : D  5 /"OODH-sE))F  F cU(aUR5nOURXR5n/nUHDnU(dM UR[ UR U5RS555 MF U$)zReturns a tokenized string.rp)r6r\r+rslistr~)r@rZbypass_tokenizer split_tokensrws r _tokenizeBioGptTokenizer._tokenizesf ::)>s)C$DErc~URRXRRUR55$)z0Converts a token (str) in an id using the vocab.)r2rir&)r@rws r_convert_token_to_id$BioGptTokenizer._convert_token_to_ids*||||'7'7'GHHrcLURRXR5$)z=Converts an index (integer) in a token (str) using the vocab.)r4rir&)r@rrs r_convert_id_to_token$BioGptTokenizer._convert_id_to_tokens||~~66rcUVs/sH$o"RSS5RSS5PM& nnSRU5R5nURXR5nU$s snf)z:Converts a sequence of tokens (string) in a single string.rprf)replacervr6rcr+)r@ratrZs rconvert_tokens_to_string(BioGptTokenizer.convert_tokens_to_stringseDJJ6a))C$,,VS96J&&($$VYY7 Ks+A. token_ids_0 token_ids_1returncVUcUR/U-$UR/nX1-U-U-$)a+ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BioGPT sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) sep_token_idr@rrseps r build_inputs_with_special_tokens0BioGptTokenizer.build_inputs_with_special_tokenss?&  %%&4 4  ! 3&44ralready_has_special_tokensc>U(a[TU]XSS9$Ub'S/S/[U5--S/-S/[U5--$S/S/[U5--$)ad Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrr r)r>get_special_tokens_maskr;)r@rrrrHs rr'BioGptTokenizer.get_special_tokens_masksw$ &72']a3   "31#K 001QC7A3[AQ;QR RsqcC ,,--rcUR/nUc[X-5S/-$[X-5S/-[X#-5S/--$)ah Create a mask from the two sequences passed to be used in a sequence-pair classification task. A FAIRSEQ Transformer sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). rr )rr;rs r$create_token_type_ids_from_sequences4BioGptTokenizer.create_token_type_ids_from_sequencessX.  !  {()QC/ /;$%+c+2C.Ds.JJJrsave_directoryfilename_prefixc B[RRU5(d[R SUS35 g[RR X(aUS-OS[ S-5n[RR X(aUS-OS[ S-5n[USSS 9nUR[R"URS S S S 9S-5 SSS5 Sn[USSS 9n[URR5SS9HMupXi:wa[RSUS35 U nURSR U5S-5 US- nMO SSS5 X44$!,(df  N=f!,(df  X44$=f)NzVocabulary path (z) should be a directory-rr r wrrr"TF)indent sort_keys ensure_asciir rc US$)Nr r()kvs rrl1BioGptTokenizer.save_vocabulary..Js Y[\]Y^rrnzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rpr )ospathisdirloggererrorrvVOCAB_FILES_NAMESr/writer0dumpsr2sortedr<r3warning) r@rrr merge_filefrrwriter bpe_tokens token_indexs rsave_vocabularyBioGptTokenizer.save_vocabulary:syww}}^,, LL,^,<zrz)FrO)NF)!__name__ __module__ __qualname____firstlineno____doc__rvocab_files_namesmodel_input_namesr?propertyrLrQr\rcr~rrrrrintrrboolrrstrrrrr__static_attributes__ __classcell__)rHs@rrr.sW(T*$&67 * X!!? E *X I7JN5953;DI3F5 c52sx.9.3;DI3F.ko. c..8JNK9K3;DI3FK cK<&c&HSM&]bcf]g&8  rr)rr0rtypingrrrtokenization_utilsrutilsr get_loggerrrrrr__all__r(rrrs]' ((5   H % x)xv  r