fTh31 S r SSKrSSKJr SSKJrJrJrJrJ r SSK
r SSK J
r
SSKJr SSKJr \R$ " \5 rS rS
S 0r\" S S
9 " S S\
5 5 rS/rg)zTokenization classes for . N)copyfile)AnyDictListOptionalTuple )PreTrainedTokenizer)logging)requiresu ▁
vocab_filezsentencepiece.bpe.model)
sentencepiece)backendsc
~ ^ \ rS rSrSr\rSS/r S S\\ \
\ 4 SS4U 4S jjjr S r
S
r S S \\ S \\\ S\\ 4S
jjr S S \\ S \\\ S\S\\ 4U 4S jjjr S S \\ S \\\ S\\ 4S jjr\S 5 rS rS\
S\\
4S jrS rS rS rS S\
S\\
S\ \
4S jjr Sr U =r $ )
XGLMTokenizer# a
Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on
[SentencePiece](https://github.com/google/sentencepiece).
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
sp_model_kwargs (`dict`, *optional*):
Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
to set:
- `enable_sampling`: Enable subword regularization.
- `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
- `nbest_size = {0,1}`: No sampling is performed.
- `nbest_size > 1`: samples from the nbest_size results.
- `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
- `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
Attributes:
sp_model (`SentencePieceProcessor`):
The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
input_idsattention_maskNsp_model_kwargsreturnc v > Uc 0 OUU l SU l [ U R 5 V
s/ s H n
SU
S3PM n n
U R S/ 5 =( d / U S' U S== U V s/ s H oU S ; d M
U PM sn -
ss' [ R
" S 0 U R D6U l U R R [ U5 5 Xl SU l
SSSSS .U l [ U R 5 n
[ U R 5 V
s0 s H n
SU
S3X- U R - _M n n
U R R U 5 U R R 5 VVs0 s H u pX_M snnU l [ TU ]D " S UUUUUUU R S
.U D6 g s sn
f s sn f s sn
f s snnf ) N z additional_special_tokens r r )`
- pair of sequences: ` A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) sep_token_idr8 rR rS seps rA build_inputs_with_special_tokens.XGLMTokenizer.build_inputs_with_special_tokens sD ( %%&4 4 ! 3&,{::rK already_has_special_tokensc > U( a [ TU ] XSS9$ Uc S/S/[ U5 - - $ S/S/[ U5 - - SS/- S/[ U5 - - $ )ad
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rR rS rZ r r )r6 get_special_tokens_maskr2 )r8 rR rS rZ r@ s rA r\ %XGLMTokenizer.get_special_tokens_mask sy & &72 ']a 3
31#K 001 1sqcC ,,-A61#K@P:PQQrK c x U R /nUc [ X1- 5 S/- $ [ X1- U- U- U- 5 S/- $ )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. XLM-RoBERTa does
not make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
r )rU r2 rV s rA $create_token_type_ids_from_sequences2XGLMTokenizer.create_token_type_ids_from_sequences sQ $ ! s()QC/ /3$s*S0;>?1#EErK c ` [ U R 5 U R - U R - $ N)r2 r- r0 r( )r8 s rA
vocab_sizeXGLMTokenizer.vocab_size s' 4==!D$7$77$:O:OOOrK c [ U R 5 Vs0 s H oR U5 U_M nnUR U R 5 U$ s snf rb )r) rc convert_ids_to_tokensr3 added_tokens_encoder)r8 r: vocabs rA get_vocabXGLMTokenizer.get_vocab sL ;@ ;QR;Qa++A.1;QR
T../ Ss Atextc > U R R U[ S9$ )N)out_type)r- encoder/ )r8 rk s rA _tokenizeXGLMTokenizer._tokenize s }}##D3#77rK c XR ; a U R U $ U R R U5 nU( a X R - $ U R $ )z0Converts a token (str) in an id using the vocab.)r1 r- PieceToIdr0 unk_token_id)r8 tokenspm_ids rA _convert_token_to_id"XGLMTokenizer._convert_token_to_id sQ .. .--e4 4 ((/ 06v+++L4;L;LLrK c XR ; a U R U $ U R R XR -
5 $ )z=Converts an index (integer) in a token (str) using the vocab.)r5 r- IdToPiecer0 )r8 indexs rA _convert_id_to_token"XGLMTokenizer._convert_id_to_token s= .. .--e4 4}}&&u/B/B'BCCrK c l SR U5 R [ S5 R 5 nU$ )zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r8 tokens
out_strings rA convert_tokens_to_string&XGLMTokenizer.convert_tokens_to_string s, WWV_,,-=sCIIK
rK save_directoryfilename_prefixc [ R R U5 ( d [ R SU S35 g [ R R X( a US- OS[ S - 5 n[ R R U R 5 [ R R U5 :w aG [ R R U R 5 ( a [ U R U5 U4$ [ R R U R 5 ( dC [ US5 nU R R 5 nUR U5 S S S 5 U4$ U4$ ! , ( d f U4$ = f)NzVocabulary path (z) should be a directory-r~ r
wb)ospathisdirloggererrorr VOCAB_FILES_NAMESabspathr
isfiler openr- rG write)r8 r r out_vocab_fileficontent_spiece_models rA save_vocabulary XGLMTokenizer.save_vocabulary s, ww}}^,, LL,^,<r sv ! 3 3 5 *
H % !#<=
%&G!' G! 'G!T
rK