o Zh@sdZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl m Z mZe r4d d lmZndZeeZd d d ZdZGddde ZdgZdS)z$Tokenization classes for FNet model.N)copyfile)ListOptionalTuple) AddedToken)PreTrainedTokenizerFast)is_sentencepiece_availablelogging) FNetTokenizerz spiece.modelztokenizer.json) vocab_filetokenizer_fileu▁cseZdZdZeZddgZeZ        dfd d Z e de fddZ dde edee ede efddZ dde edee ede efddZddedeedeefddZZS)FNetTokenizerFastaR Construct a "fast" FNetTokenizer (backed by HuggingFace's *tokenizers* library). Adapted from [`AlbertTokenizerFast`]. Based on [Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models). This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods Args: vocab_file (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. do_lower_case (`bool`, *optional*, defaults to `False`): Whether or not to lowercase the input when tokenizing. remove_space (`bool`, *optional*, defaults to `True`): Whether or not to strip the text when tokenizing (removing excess spaces before and after the string). keep_accents (`bool`, *optional*, defaults to `True`): Whether or not to keep accents when tokenizing. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. sep_token (`str`, *optional*, defaults to `"[SEP]"`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. cls_token (`str`, *optional*, defaults to `"[CLS]"`): The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. mask_token (`str`, *optional*, defaults to `"[MASK]"`): The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. Z input_idsZtoken_type_idsNFT[SEP][CLS][MASK]c st| tr t| dddn| } t| trt| dddn| } t|tr(t|dddn|}tj|f|||||||| | d | ||_||_||_||_dS)NTF)lstriprstrip) r do_lower_case remove_space keep_accents unk_token sep_token pad_token cls_token mask_token) isinstancestrrsuper__init__rrrr ) selfr rrrrrrrrrkwargs __class__^/var/www/auris/lib/python3.10/site-packages/transformers/models/fnet/tokenization_fnet_fast.pyr"Ms,  zFNetTokenizerFast.__init__returncCs|jr tj|jSdS)NF)r ospathisfile)r#r'r'r(can_save_slow_tokenizertsz)FNetTokenizerFast.can_save_slow_tokenizer token_ids_0 token_ids_1cCs8|jg}|jg}|dur|||S|||||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An FNet sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: list of [input IDs](../glossary#input-ids) with the appropriate special tokens. N) sep_token_id cls_token_idr#r.r/sepclsr'r'r( build_inputs_with_special_tokensxs  z2FNetTokenizerFast.build_inputs_with_special_tokenscCsV|jg}|jg}|durt|||dgSt|||dgt||dgS)a Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An FNet sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). Nrr )r0r1lenr2r'r'r($create_token_type_ids_from_sequencess (z6FNetTokenizerFast.create_token_type_ids_from_sequencessave_directoryfilename_prefixcCsptj|std|ddStj||r|dndtd}tj|jtj|kr5t |j||fS)NzVocabulary path (z) should be a directory-r ) r*r+isdirloggererrorjoinVOCAB_FILES_NAMESabspathr r)r#r8r9Zout_vocab_filer'r'r(save_vocabularys  z!FNetTokenizerFast.save_vocabulary) NNFTTrrrrr)N)__name__ __module__ __qualname____doc__r@Zvocab_files_namesZmodel_input_namesr Zslow_tokenizer_classr"propertyboolr-rintrr5r7r rrB __classcell__r'r'r%r(r&sF"'    (r)rFr*shutilrtypingrrrZtokenization_utilsrZtokenization_utils_fastrutilsr r Ztokenization_fnetr Z get_loggerrCr=r@ZSPIECE_UNDERLINEr__all__r'r'r'r(s