o Zh3 @sddlZddlmZddlmZmZddlmZddlm Z ddl m Z m Z e r0dd l mZndZe eZd d d ZGd dde ZdgZdS)N)copyfile)OptionalTuple) processors)PreTrainedTokenizerFast)is_sentencepiece_availablelogging)GemmaTokenizerztokenizer.modelztokenizer.json) vocab_filetokenizer_filecseZdZdZeZeZdZddgZ      dfd d Z e de fddZ ddZe ddZe ddZejddZejddZd dedeedeefddZd ddZZS)!GemmaTokenizerFastu Construct a Gemma tokenizer fast. Based on byte-level Byte-Pair-Encoding. This uses notably ByteFallback and no prefix space. Normalization is applied to replace `" "` with `"▁"` ```python >>> from transformers import GemmaTokenizerFast >>> tokenizer = GemmaTokenizerFast.from_pretrained("hf-internal-testing/dummy-gemma") >>> tokenizer.encode("Hello this is a test") [2, 4521, 736, 603, 476, 2121] ``` If you want to change the `bos_token` or the `eos_token`, make sure to specify them when initializing the model, or call `tokenizer.update_post_processor()` to make sure that the post-processing is correctly done (otherwise the values of the first token and final token of an encoded sequence will not be correct). For more details, checkout [post-processors] (https://huggingface.co/docs/tokenizers/api/post-processors) documentation. This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .model extension) that contains the vocabulary necessary to instantiate a tokenizer. tokenizer_file (`str`, *optional*): [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that contains everything needed to load the tokenizer. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like extra spaces. unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The padding token add_bos_token (`bool`, *optional*, defaults to `True`): Whether or not to add an `bos_token` at the start of sequences. add_eos_token (`bool`, *optional*, defaults to `False`): Whether or not to add an `eos_token` at the end of sequences. leftZ input_idsZattention_maskNFTc sDtjd||||||||| d | ||_| |_|||_dS)N) r r clean_up_tokenization_spaces unk_token bos_token eos_token pad_token add_bos_token add_eos_token)super__init___add_bos_token_add_eos_tokenupdate_post_processorr ) selfr r rrrrrrrkwargs __class__r`/var/www/auris/lib/python3.10/site-packages/transformers/models/gemma/tokenization_gemma_fast.pyrWs"    zGemmaTokenizerFast.__init__returncCs|jr tj|jSdS)NF)r ospathisfiler!rrr%can_save_slow_tokenizerusz*GemmaTokenizerFast.can_save_slow_tokenizercCs|j}|j}|dur|jrtd|j}|j}|dur"|jr"td|jr)|dndd|jr5d|dnd}||jrDd|d ndd |jrPd|d nd}g}|jr`|||f|jrj|||ftj |||d |j _ dS) ze Updates the underlying post processor with the current `bos_token` and `eos_token`. Nz)add_bos_token = True but bos_token = Nonez)add_eos_token = True but eos_token = Nonez:0 z$A:0 z:0z:1z $B:1)singlepairspecial_tokens) r bos_token_idr ValueErrorr eos_token_idrappendrZTemplateProcessingZ _tokenizerZpost_processor)r!Zbosr1Zeosr3r.r/r0rrr%r zs$.6z(GemmaTokenizerFast.update_post_processorcC|jSN)rr*rrr%rz GemmaTokenizerFast.add_eos_tokencCr5r6)rr*rrr%rr7z GemmaTokenizerFast.add_bos_tokencC||_|dSr6)rr r!valuerrr%r cCr8r6)rr r9rrr%rr;save_directoryfilename_prefixcCs~|jstdtj|std|ddStj||r"|dndtd}tj |j tj |krs      &