o
Zh @ s d Z ddlZddlmZ ddlmZmZmZ ddlm Z ddl
m Z m Z e r.dd l
mZ ndZe eZd
d d ZG d
d de ZdgZdS )z Tokenization classes for XGLM. N)copyfile)ListOptionalTuple )PreTrainedTokenizerFast)is_sentencepiece_availablelogging )
XGLMTokenizerzsentencepiece.bpe.modelztokenizer.json)
vocab_filetokenizer_filec s e Zd ZdZeZddgZeZ d fd d
Z e
d e fd d
Z dde
e dee
e d e
e fddZ dde
e dee
e d e
e fddZddedee d ee fddZ ZS )XGLMTokenizerFasta{
Construct a "fast" XGLM tokenizer (backed by HuggingFace's *tokenizers* library). Adapted from [`RobertaTokenizer`]
and [`XLNetTokenizer`]. Based on
[BPE](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models).
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
additional_special_tokens (`List[str]`, *optional*, defaults to `["NOTUSED", "NOTUSED"]`):
Additional special tokens used by the tokenizer.
Z input_idsZattention_maskN X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
N) sep_token_idr( r0 r1 sepr r r build_inputs_with_special_tokens s z2XGLMTokenizerFast.build_inputs_with_special_tokensc C s@ | j g}|du rt|| dg S t|| | | | dg S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. XLM-RoBERTa does
not make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
Nr )r2 lenr3 r r r $create_token_type_ids_from_sequences s z6XGLMTokenizerFast.create_token_type_ids_from_sequencessave_directoryfilename_prefixc C s~ | j stdtj|std| d d S tj||r"|d ndtd }tj | j
tj |kr rA r __all__r r r r s
!