fThi V S r SSKrSSKrSSKrSSKJr SSKJrJrJ r SSK
r
SSK J r SSK
Jr \R " \5 rSS S
.rS r " S S
\ 5 r SrSr\S\SSSSSS4 r\
R0 " SSR3 \5 - \
R4 \
R6 - \
R8 - 5 r \
R0 " S5 r \
R0 " \\
R4 \
R6 - \
R8 - 5 r \
R0 " S5 r S"S jr!S#S jr" " S S 5 r#S r$S r%S$S! jr&S
/r'g)%z!Tokenization classes for BERTweet N)copyfile)ListOptionalTuple )PreTrainedTokenizer)loggingz vocab.txtz bpe.codes)
vocab_file merges_filec z [ 5 nU S nU SS H nUR X#45 UnM [ U5 nU$ )zy
Return set of symbol pairs in a word.
Word is represented as tuple of symbols (symbols being variable-length strings).
r N)setadd)wordpairs prev_charchars j/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/bertweet/tokenization_bertweet.py get_pairsr & sH
EEQIQR
9#$
JE L c
V ^ \ rS rSrSr\r SU 4S jjr SS\\ S\
\\ S\\ 4S jjr SS\\ S\
\\ S\ S\\ 4U 4S jjjr
SS\\ S\
\\ S\\ 4S
jjr\S 5 rS rS
rS rS rS rS rS rS rSS\S\
\ S\\ 4S jjrS r Sr U =r $ ) BertweetTokenizer6 a8
Constructs a BERTweet tokenizer, using Byte-Pair-Encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
normalization (`bool`, *optional*, defaults to `False`):
Whether or not to apply a normalization preprocess.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
mask_token (`str`, *optional*, defaults to `""`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
c > SSK Jn Xl Xl X l 0 U l SU R [ U5 ' SU R [ U 5 ' SU R [ U5 ' SU R [ U5 ' U R U5 U R R 5 V
Vs0 s H u pX_M snn
U l [ USS9 nUR 5 R S 5 S S
nS S S 5 W Vs/ s H n[! UR 5 S S
5 PM! nn[# [% U[' [) U5 5 5 5 U l 0 U l X0l [1 5 U l S S S
.U l [6 TU ]p " SUUUUUUU U
S.U D6 g ! [ a [ R S5 S U l GNf = fs snn
f ! , ( d f N= fs snf )Nr )demojizezsemoji is not installed, thus not converting emoticons or emojis into text. Install emoji: pip3 install emoji==0.6.0r
r utf-8encoding
'z...)u ’u …)
normalization bos_token eos_token sep_token cls_token unk_token pad_token
mask_token ) emojir demojizer ImportErrorloggerwarningr
r encoderstr
add_from_fileitemsdecoderopenreadsplittupledictziprangelen bpe_rankscacher# TweetTokenizertweetPreprocessorspecial_punctssuper__init__)selfr
r r# r$ r% r& r' r( r) r* kwargsr kv
merges_handlemergesmerge __class__s r rD BertweetTokenizer.__init__k s " & %N %& '( S^$'( S^$'( S^$'( S^$ :&)- ););)=>)=)=>
+
0M"'')//5cr:F 19?@%
cr*+@ c&%F *< =>
*!/!1 &)% 8
' !
= " NN(
"DN "$ ?
0
0@s) F 0G#G &G&F>=F>
G token_ids_0 token_ids_1returnc Uc U R /U- U R /- $ U R /nU R /nX1- U- U- U- U- $ )a7
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A BERTweet sequence has the following format:
- single sequence: ` X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) cls_token_id sep_token_id)rE rN rO clsseps r build_inputs_with_special_tokens2BertweetTokenizer.build_inputs_with_special_tokens se ( %%&48I8I7JJ J ! ! 3&,{:S@@r already_has_special_tokensc > U( a [ TU ] XSS9$ Uc S/S/[ U5 - - S/- $ S/S/[ U5 - - SS/- S/[ U5 - - S/- $ )ad
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rN rO rX r
r )rC get_special_tokens_maskr= )rE rN rO rX rL s r rZ )BertweetTokenizer.get_special_tokens_mask s & &72 ']a 3
31#K 001QC7 7sqcC ,,-A61#K@P:PQUVTWWWr c U R /nU R /nUc [ XA- U- 5 S/- $ [ XA- U- U- U- U- 5 S/- $ )a}
Create a mask from the two sequences passed to be used in a sequence-pair classification task. BERTweet does
not make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
r )rS rR r= )rE rN rO rU rT s r $create_token_type_ids_from_sequences6BertweetTokenizer.create_token_type_ids_from_sequences si " ! ! s(3./1#5 53$s*S0;>DEKKr c , [ U R 5 $ N)r= r1 rE s r
vocab_size BertweetTokenizer.vocab_size s 4<< r c B [ U R 40 U R D6$ r` )r: r1 added_tokens_encoderra s r get_vocabBertweetTokenizer.get_vocab s DLL>D$=$=>>r c 6 ^ UT R ; a T R U $ [ U5 n[ [ US S 5 US S- /- 5 n[ U5 nU( d U$ [ UU 4S jS9nUT R
; a OUu pV/ nSnU[
U5 : a UR XX5 n UR X(U 5 U nX( U:X a6 U[
U5 S-
: a$ X(S- U:X a UR XV- 5 US-
nOUR X( 5 US-
nU[
U5 : a M [ U5 nUn[
U5 S:X a O
[ U5 nM SR U5 nUS S nUT R U' U$ ! [ a UR X(S 5 Mq f = f)
Nr! zc N > TR R U [ S5 5 $ )Ninf)r> getfloat)pairrE s r 'BertweetTokenizer.bpe.. s 1C1CD%PU,1Wr )keyr r
r @@ ) r? r9 listr minr> r= indexextend
ValueErrorappendjoin)
rE tokenr r bigramfirstsecondnew_wordijs
` r bpeBertweetTokenizer.bpe s DJJ ::e$ $U|T$s)_R6(9'::;$L$WXFT^^+ "MEHAc$i-
5,A
OOD1I.A7e#CIM(9dq5kV>S OOEN3FA OODG,FA c$i- XHD4yA~!$9 : zz$ CRy
5 - " OODH-s E7 7 FFc U R ( a U R U5 n/ n[ R " SU5 nU H; nUR [ U R
U5 R S5 5 5 M= U$ )zTokenize a string.z\S+\n? )r# normalizeTweetrefindallrv rs r r8 )rE text split_tokenswordsrz s r _tokenizeBertweetTokenizer._tokenize( sf &&t,D
9d+E TXXe_%:%:3%? @ A r c . U R H! nUR X R U 5 nM# U R R U5 nSR U Vs/ s H o@R U5 PM sn5 nUR SS5 R SS5 R SS5 R SS5 R S S
5 nUR S S 5 R S
S5 R SS5 R SS5 R SS5 R SS5 nUR SS5 R SS5 R SS 5 R S S 5 nSR UR
5 5 $ s snf ) z
Normalize a raw Tweet
r zcannot zcan not zn't z n't zn 't zca n'tzcan'tzai n'tzain'tz'm z 'm z're z 're z's z 's z'll z 'll z'd z 'd z've z 've z p . m .z p.m.z p . m z p.m z a . m .z a.m.z a . m z a.m )rB replacerA tokenizery normalizeTokenr8 )rE tweetpuncttokensrz normTweets r r BertweetTokenizer.normalizeTweet3 s^ ((EMM%)<) XR ; a U R U $ U R b U R U5 $ U$ U$ )z
Normalize tokens in a Tweet
@z@USERhttpwwwHTTPURLr
)lower
startswithr= rB r- )rE rz lowercased_tokens r r BertweetTokenizer.normalizeTokenU s !;;= C
(
(
0
04D4O4OPU4V4V
Z1_+++**511~~)~~e,, Lr c ~ U R R XR R U R 5 5 $ )z0Converts a token (str) in an id using the vocab.)r1 rk r( )rE rz s r _convert_token_to_id&BertweetTokenizer._convert_token_to_idh s* || ||'7'7'GHHr c L U R R XR 5 $ )z=Converts an index (integer) in a token (str) using the vocab.)r5 rk r( )rE ru s r _convert_id_to_token&BertweetTokenizer._convert_id_to_tokenl s || ~~66r c d SR U5 R SS5 R 5 nU$ )z:Converts a sequence of tokens (string) in a single string.r rq )ry r strip)rE r
out_strings r convert_tokens_to_string*BertweetTokenizer.convert_tokens_to_stringp s, XXf%--eR8>>@
r save_directoryfilename_prefixc 4 [ R R U5 ( d [ R SU S35 g [ R R X( a US- OS[ S - 5 n[ R R X( a US- OS[ S - 5 n[ R R U R 5 [ R R U5 :w aE [ R R U R 5 ( a [ U R U5 On[ R R U R 5 ( d@ [ US5 nU R R 5 nUR U5 S S S 5 [ R R U R 5 [ R R U5 :w a [ U R U5 X44$ ! , ( d f Nq= f)NzVocabulary path (z) should be a directory-r r
r wb)ospathisdirr/ errorry VOCAB_FILES_NAMESabspathr
isfiler r6 sp_modelserialized_model_protowriter )rE r r out_vocab_fileout_merge_fileficontent_spiece_models r save_vocabulary!BertweetTokenizer.save_vocabularyu sr ww}}^,, LL,^,< ')
isinstancer2 r6 r3 FileNotFoundError UnicodeError Exception readlinesr rfindrw r= r1 ) rE ffdfnfelineslineTmplineidxr s r r3 BertweetTokenizer.add_from_file s a
c!S73r&&r* 4
G==?D**S/Cby !XYY:D!$T\\!2DLL
43
%
c "A!D` abb
cs3 C B0C 0
B>:C >C
C* C
C*)
r> r? r5 r- r1 r r# rB rA r
)F