o Zh/@sdZddlZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl ZddlmZmZddlmZdd lmZmZd d d d dZddiZddZeeZddZidddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6id7d8d9d:d;d<d=d>d?d@dAdBdCdDdEdFdGdHdIdJdKdLdMdNdOdPdQdRdSdTdUdVdWdXidYdZd[d\d]d^d_d`dadbdcdddedfdgdhdidjdkdldmdndodpdqdrdsdtdudvdwdxdydzid{d|d}d~ddddddddddddddddddddddddddddddiddddddddddddddddddddddddddddddddddddddddddddddddddΜZiddЄeDdd+d-dddddGdGddddӜ Z ddgZ!GddׄdeZ"ddلZ#dddۄZ$dd݄Z%  ddee&dee'de'de'fddZ(dee&fddZ)dee&fddZ*ddZ+dgZ,dS)z!Tokenization classes for Whisper.N) lru_cache)ListOptionalTupleUnion) AddedTokenPreTrainedTokenizer)logging)BasicTextNormalizerEnglishTextNormalizerz vocab.jsonztokenizer.jsonz merges.txtznormalizer.json) vocab_fileZtokenizer_file merges_filenormalizer_filezopenai/whisper-baseicCstttdtddtttdtddtttdtdd}|dd}d }td D]}||vrI|||d ||d7}q3d d |D}tt||S) a8 Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. !~r ¡¬®ÿNrcSsg|]}t|qS)chr).0nrr_/var/www/auris/lib/python3.10/site-packages/transformers/models/whisper/tokenization_whisper.py Az$bytes_to_unicode..)listrangeordappenddictzip)bscsrbrrrbytes_to_unicode-sL    r(cCs6t}|d}|ddD] }|||f|}q |S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rr N)setadd)wordpairsZ prev_charcharrrr get_pairsIs r.enenglishzhchinesedegermanesspanishrurussiankokoreanfrfrenchjajapanesept portuguesetrturkishplpolishcacatalannldutchararabicsvswedishititalianidZ indonesianhiZhindififinnishviZ vietnamesehehebrewukZ ukrainianelgreekmsZmalayr&czechroromaniandadanishhu hungariantaZtamilno norwegianththaiurZurduhrcroatianbg bulgarianlt lithuanianZlalatinmiZmaorimlZ malayalamcyZwelshskslovakteZtelugufaZpersianlvZlatvianZbnZbengalisrZserbianazZ azerbaijanisl slovenianknZkannadaetestonianmkZ macedonianbrZbretoneuZbasqueis icelandichyZarmenianneZnepaliZmnZ mongolianr%ZbosniankkZkazakhsqZalbanianswZswahiliglgalicianmrZmarathipaZpunjabisisinhalakmZkhmerZsnZshonaZyoZyorubasoZsomaliafZ afrikaansocZoccitankaZgeorgianbeZ belarusiantgZtajiksdZsindhiguZgujaratiamZamharicyiZyiddishlolaouzZuzbekfoZfaroeseZhtzhaitian creoleZpsZpashtoZtkZturkmennnnynorskmtZmalteseZsanskritZ luxembourgishmyanmarZtibetanZtagalogZmalagasyZassameseZtatarZhawaiianZlingalaZhausaZbashkirZjavaneseZ sundanese cantonese)salbmyZbotlZmgasttZhawlnZhabaZjwZsuZyuecCi|]\}}||qSrr)rcodelanguagerrr rrr) ZburmeseZ valencianZflemishZhaitianZ letzeburgeschZpushtoZpanjabiZ moldavianZmoldovanZ sinhaleseZ castilianZmandarin translate transcribecsbeZdZdZeZddgZ          d]fdd Zed e fd d Z d dZ ddZ d^de ede ede efddZed ee fddZd_d ee fddZ d`dee de ee ded ee ffdd Zdd Zd!d"Zd#d$Zd%d&Zdad'd(Zd)d*Zedad+d,Z .dbd efd/d0Zdcd1d2Zeddd3d4Z dad5efd6d7Z!d8d9Z"    -    ded5ed:e ed;eded?ed@ed effdAdB Z$    dfdCe%e ee fd5ed>ed?ed@ed ef dDdEZ&dFdGZ'd_dHedIe ed e(efdJdKZ)dadLdMZ*dgdOdPZ+dQdRZ,dhdTefdUdVZ-dCee dWe dXe fdYdZZ.ed[d\Z/Z0S)iWhisperTokenizeraa Construct a Whisper tokenizer. This tokenizer inherits from [`PreTrainedTokenizer`] which contains some of the main methods. Users should refer to the superclass for more information regarding such methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. normalizer_file (`str`, *optional*): Path to the normalizer_file file. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. See [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The beginning of sequence token. The `decoder_start_token_id` is used to set the first token as `"<|startoftranscript|>"` when generating. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*): The token used for padding, for example when batching sequences of different lengths. add_prefix_space (`bool`, *optional*, defaults to `False`): Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. language (`str`, *optional*): The language of the transcription text. The corresponding language id token is appended to the start of the sequence for multilingual speech recognition and speech translation tasks, e.g. for Spanish the token `"<|es|>"` is appended to the start of sequence. This should be used for multilingual fine-tuning only. task (`str`, *optional*): Task identifier to append at the start of sequence (if any). This should be used for mulitlingual fine-tuning, with `"transcribe"` for speech recognition and `"translate"` for speech translation. predict_timestamps (`bool`, *optional*, defaults to `False`): Whether to omit the `<|notimestamps|>` token at the start of the sequence. input_idsZattention_maskNreplace <|endoftext|>Fc  st|trt|dddddn|}t|trt|dddddn|}t|tr.t|dddddn|}t|tr>t|dddddn|}t|dd}t||_Wdn1sWwYdd|jD|_||_ t |_ dd|j D|_ t|dd}| d d d }Wdn1swYd d |D}tt|tt||_i|_| |_|durt|dd}t||_Wdn1swYnd|_td|_td|_| |_tjd|||||| d| | |_| |_dS)NFT)lstriprstrip normalizedspecialutf-8encodingcSrrrrkvrrrr%rz-WhisperTokenizer.__init__..cSrrrrrrrr(r r cSsg|]}t|qSr)tuplesplit)rmergerrrr+z-WhisperTokenizer.__init__..zJ's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+z<\|(\d+\.\d+)\|>)errors unk_token bos_token eos_token pad_tokenadd_prefix_spacer) isinstancestrropenjsonloadencoderitemsdecoderrr( byte_encoder byte_decoderreadrr#r$r len bpe_rankscacherenglish_spelling_normalizerrecompilepat timestamp_patrsuper__init__taskpredict_timestamps)selfrrrrrrrrrrrrkwargsZ vocab_handleZ merges_handleZ bpe_merges __class__rrrsf   zWhisperTokenizer.__init__returncCs t|jSN)rrrrrr vocab_sizeHs zWhisperTokenizer.vocab_sizecs(fddtjD}|j|S)Ncsi|]}||qSr)convert_ids_to_tokensrirrrrMrz.WhisperTokenizer.get_vocab..)r rupdateadded_tokens_encoder)rZvocabrrr get_vocabLs zWhisperTokenizer.get_vocabc sX|jvr j|St|}t|}|s|S t|fddd}|jvr'ny|\}}g}d}|t|krz|||} WntyO|||dYn?w|||| | }|||kr}|t|dkr}||d|kr}| |||d7}n | |||d7}|t|ks5t|}|}t|dkrnt|}qd |}|j|<|S) NTcsj|tdS)Ninf)rgetfloat)pairrrr\sz&WhisperTokenizer.bpe..keyrr  ) rrr.minrrindex ValueErrorextendr"join) rtokenr+r,ZbigramfirstsecondZnew_wordrjrrrbpeRsJ     ,     zWhisperTokenizer.bperrrcCsF|dur|n|j|_|dur|n|j|_|dur||_dS|j|_dS)a Override the prefix tokens appended to the start of the label sequence. This method can be used standalone to update the prefix tokens as required when fine-tuning. Example: ```python >>> # instantiate the tokenizer and set the prefix token to Spanish >>> tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-tiny", language="spanish") >>> # now switch the prefix token from Spanish to French >>> tokenizer.set_prefix_tokens(language="french") ``` Args: language (`str`, *optional*, defaults to `None`): The language of the transcription text. task (`str`, *optional*, defaults to `None`): Task identifier to append at the start of sequence (if any). predict_timestamps (`bool`, *optional*, defaults to `None`): Whether to omit the `<|notimestamps|>` token at the start of the sequence. N)rrr)rrrrrrrset_prefix_tokens|sz"WhisperTokenizer.set_prefix_tokensc Cs<|d}|d}|d}|d}tt}|jdurZ|j|_|jtvr0t|j}n*|jtvr;|j}nt|jdk}t d|jd|rPt tnt td|j duro|j t vrot d |j d t |g}|jdur| |d |||j dur| |j d kr|n||js| ||S) N<|startoftranscript|>z <|translate|>z<|transcribe|><|notimestamps|>rzUnsupported language: z. Language should be one of: .zUnsupported task: z. Task should be in: r r)convert_tokens_to_idsr LANGUAGESkeysrlowerTO_LANGUAGE_CODEvaluesrrrrTASK_IDSr"rr) rZ bos_token_idZtranslate_token_idZtranscribe_token_idZnotimestamps_token_idZlangsZ language_idZis_language_codeZ bos_sequencerrr prefix_tokenss8               zWhisperTokenizer.prefix_tokenscCs0|dur |j||jgS|j|||jgS)z=Build model inputs from a sequence by appending eos_token_id.N)r  eos_token_id)r token_ids_0 token_ids_1rrr build_inputs_with_special_tokenssz1WhisperTokenizer.build_inputs_with_special_tokensrralready_has_special_tokenscsn|r tj||ddSdgt|j}dg}|dur%|dgt||S|dgt|dgt||S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrr Nr)rget_special_tokens_maskrr )rrrrZ prefix_onesZ suffix_onesrrrrs$z(WhisperTokenizer.get_special_tokens_maskcsZg}tj|D]!}dfdd|dD}|dd|dDq |S)zTokenize a string.c3s|]}j|VqdSr)r)rr'rrr s  z-WhisperTokenizer._tokenize..rcs|]}|VqdSrr)rZ bpe_tokenrrrrr)rfindallrrencoderrr)rtext bpe_tokensrrrr _tokenizes"zWhisperTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)rrr)rrrrr_convert_token_to_idsz%WhisperTokenizer._convert_token_to_idcCs|j|dS)z Converts an index (integer) in a token (str) using the vocab. Whisper's base tokenizer always decodes OOV tokens as "", thus we do not use the `unk_token` here. r)rr)rrrrr_convert_id_to_tokensz%WhisperTokenizer._convert_id_to_tokencCstd||S)NzThe private method `_normalize` is deprecated and will be removed in v5 of Transformers.You can normalize an input string using the Whisper English normalizer using the `normalize` method.)warningswarn normalize)rrrrr _normalizes zWhisperTokenizer._normalizecCstd|j||dS)NzThe private method `_basic_normalize` is deprecated and will be removed in v5 of Transformers.You can normalize an input string using the Whisper basic normalizer using the `basic_normalize` method.remove_diacritics)rr basic_normalize)rrr$rrr_basic_normalizesz!WhisperTokenizer._basic_normalizecCst|j}||S)z Normalize a given string using the `EnglishTextNormalizer` class, which performs commons transformation on english text. )r r)rr normalizerrrrr!s zWhisperTokenizer.normalizecCst|d}||S)z Normalize a given string using the `BasicTextNormalizer` class, which performs commons transformation on multilingual text. r#)r )rr$r'rrrr% s z WhisperTokenizer.basic_normalize{Gz?csjdd}gg}d}d}d} t|D][\} } | |krht| ||} | |krR| dko<|| d|ko;|| d|k } | rF|||7}n | }|| 7}|dd}|} | }|d| |dd |gq|d| qfd d |D}d |S) z Timestamp tokens are above the special tokens' id range and are ignored by `decode()`. This method decodes given tokens with timestamps tokens annotated, e.g. "<|1.08|>". rr rNz<|z.2fz|>cs(g|]}t|tr |nj|dqS)skip_special_tokens)rrdecode)rsrr-rrr:sz.r)all_special_ids enumeraterr"r)r token_idsr-time_precision segment_sizetimestamp_beginZoutputscur_max_timestampprev_segments_lenZpenultimate_timestamprr timestampZlast_was_single_endingrr0r_decode_with_timestampss4    z(WhisperTokenizer._decode_with_timestampscCsg}dtt|vrt|drt|jr|}t|}|jddkr/t|jdkr/t d|j dd}||k}t |dd|dd@dd}|jddkr\| dkr\gSt |ddd|vrxt |t |ddd}t |dd}d} d} |D]k} ||| } t| dkr| d|} | d|}| | kr|dko||d|ko||d|k }|r| |7} n| | 7} |} || } || }||}| || || |||| |fd | }q|S) a Compute offsets for a given tokenized input Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. segment_size (`int`, *optional*, defaults to 1500): The number of features in the input mel spectrogram. torchcpurr z)Can only process a single input at a timerNrrr9)rtypehasattrcallabler<nparrayshaperrr1wheresumr"item_preprocess_token_ids_decode_filter_timestamp_ids)rr3r4r5offsetsr6Ztimestamp_tokensZ consecutiveZ last_slicer7r8Z current_sliceZ sliced_tokensZstart_timestamp_positionZend_timestamp_positionZis_single_endingrrrr_compute_offsets?sR $ &       z!WhisperTokenizer._compute_offsetscs|fddtdDS)a Compute the timestamp token ids for a given precision and save to least-recently used (LRU) cache. Args: time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. csg|]}d|qS)z<|%.2f|>rrr4rrrrz2WhisperTokenizer.timestamp_ids..i)rr )rr4rrLr timestamp_idss zWhisperTokenizer.timestamp_idsr-cCs*|r|d}|d}||||}|S)a Pre-process the token ids for decoding by removing the prompt tokens ids and timestamp token ids. Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Typically, obtained using the `__call__` method of the tokenizer. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens from the token ids. If `True`, the prompt token ids will be removed. <|startofprev|>r)r _strip_prompt)rr3r-prompt_token_iddecoder_start_token_idrrrrGs   z&WhisperTokenizer._preprocess_token_idscCst|jd|S)Nr)rsubr)rr3rrrrIz&WhisperTokenizer._filter_timestamp_idsclean_up_tokenization_spacesoutput_offsetsr4decode_with_timestampsr!r%r$c  sn|j||d} tj| f||||| d| } |r"|j| ||d} n|| } |r5|j||d} | | dS| S)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Will remove the previous tokens (pre-prompt) if present. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). output_offsets (`bool`, *optional*, defaults to `False`): Whether or not to output the offsets of the tokens. This should only be set if the model predicted timestamps. If there are previous tokens (pre-prompt) to decode, they will only appear in the decoded text if they contain timestamp tokens. time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. decode_with_timestamps (`bool`, *optional*, defaults to `False`): Whether or not to decode with timestamps included in the raw text. normalize (`bool`, *optional*, defaults to `False`): Whether or not to apply the English text normalizer to the decoded text. Only applicable when the target text is in English. Otherwise, the basic text normalizer should be applied. basic_normalize (`bool`, *optional*, defaults to `False`): Whether or not to apply the Basic text normalizer to the decoded text. Applicable to multilingual target text. remove_diacritics (`bool`, *optional*, defaults to `False`): Whether or not to remove diacritics when applying the Basic text normalizer. Removing diacritics may destroy information in the decoded text, hence it should be used with caution. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. r,)r-rTr!r%r$)r4r-rL)rrJ)rGrr.r:rIrK)rr3r-rTrUr4rVr!r%r$rZ filtered_idsrrJrrrr.s02   zWhisperTokenizer.decoder3c Ks|dd|_|j||d}g}g} |D]&} |r| |jvrq| |jvr5| r/||| g} || q| | q| rE||| d|} |rS|| } | S|r^|j | |d} | S| S)NZuse_source_tokenizerFr,rr#) popZ_decode_use_source_tokenizerrr1rr"convert_tokens_to_stringrr!r%) rr3r-r!r%r$rZfiltered_tokensZ sub_textsZcurrent_sub_textrrZ clean_textrrrrHs.      zWhisperTokenizer._decodecs0d|}tfdd|Djdjd}|S)z:Converts a sequence of tokens (string) in a single string.rcsg|]}j|qSr)r)rcrrrrrz=WhisperTokenizer.convert_tokens_to_string..r)r)r bytearrayr.r)rtokensrrrrrXs "z)WhisperTokenizer.convert_tokens_to_stringsave_directoryfilename_prefixc Cstj|std|ddStj||r|dndtd}tj||r,|dndtd}tj||r=|dndtd}t|dd d }|t j |j d d d ddWdn1sewYd}t|dd d :}|dt |j dddD]!\} } || krtd|d| }|d| d|d7}qWdn1swY|jdurt|dd d }|t j |jd d d ddWdn1swY|||fS)NzVocabulary path (z) should be a directory-rrrrwrrrTF)indent sort_keys ensure_asciirrz#version: 0.2 cSs|dS)Nr r)kvrrrr5sz2WhisperTokenizer.save_vocabulary..rzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rr )ospathisdirloggererrorrVOCAB_FILES_NAMESrwriterdumpsrsortedrrwarningr) rr\r]rZ merge_filerfrwriterrZ token_indexrrrsave_vocabulary!sF       z WhisperTokenizer.save_vocabularycKs&|d|j}|s |rd|}||fS)Nrr)rWr)rrZis_split_into_wordsrrrrrprepare_for_tokenizationHsz)WhisperTokenizer.prepare_for_tokenizationTcCs6|j||| d|jdd}ddt|D}|S)N)rrrr cSsg|] \}}|d|fqS)r r)rZrankrrrrrUsz;WhisperTokenizer.get_decoder_prompt_ids..)rr r2)rrrZ no_timestampsZ forced_tokensZforced_decoder_idsrrrget_decoder_prompt_idsNsz'WhisperTokenizer.get_decoder_prompt_idscCst|||||dS)N)return_timestampsreturn_languager4) _decode_asr)r model_outputsrsrtr4rrrruXszWhisperTokenizer._decode_asrrArcstdd|dd}|ddd}tfdd |Dd}|dur0|}td |d |j|d |dS) z`Converts prompt text to IDs that can be passed to [`~WhisperForConditionalGeneration.generate`].rNrF)Zadd_special_tokensrr Nc3s"|] }|jdkr|VqdS)rN)r1)rxrrrrgs z2WhisperTokenizer.get_prompt_ids..zJEncountered text in the prompt corresponding to disallowed special token: r)Z tensor_type)stripnextrrZconvert_to_tensors)rrZreturn_tensorsZbatch_encodingZprompt_text_idsZspecial_token_idrrrrget_prompt_idsas  zWhisperTokenizer.get_prompt_idsrPrQcCsNt|ts ||}|s|S|d|k}|r%||vr#|||dSgS|S)Nr)rr_convert_to_listr)rr3rPrQZ has_promptrrrrOos   zWhisperTokenizer._strip_promptcCsrt|dr!dtt|vr|}ndtt|vr |}n dtt|vr-|}t|tjr7|}|S)Nnumpyr;Z tensorflowZjaxlib) r?rr>r<r|tolistrrAZndarray)r3rrrr{s  z!WhisperTokenizer._convert_to_list) NrrrrNFNNF)NNNr)NF)F)Fr(r))r(r))r()FNFr(FFFF)FFFF)NNT)rA)1__name__ __module__ __qualname____doc__riZvocab_files_namesZmodel_input_namesrpropertyintrrrrrboolrrr rrrrrr"r&r! staticmethodr%r:rKrrMrGrIrr.rrHrXrrprqrrrurzrOr{ __classcell__rrrrrs(J+ "       *D     Q *  '   rc)sdfdd}g}|}d}|dd} g} g} d} d} t|j}|d}|d }t|D]\}}|d d }||||}|d krQ|d d }d}| }d|vr|d\}}}||8}||} |rp||| }|rt|D]}|| kr|dur|| || krn|}qvg}g}t|D]\}}||vr||g}|dd}t |d}|durr|kr|s| |t | } || }!|!|d<| |g} g}|}||d<|q q|| krk|| ||}"t |"d}"|r||krd} q| s | r ||kr d} q|dd dur|"|dd <q|"|dd kr&q|"|dd<| ||d kr;| |t | | \} }#|| }!|!|d<|d krZt || |#||d<| |g} g}g} g}|}q| ||d krt |||d}$|dt|krt ||d|d}%nd}%| |$|%fqd|vr|||7}|r| ||d kr| |q1tdd| Ds|}g} g}g} g}q1| r|rtdt | | \} }#|| }!|!|d<|d krt || |#||d<| |ddd|D}&|s|rS|D]}|s|dnt|d|d<|s-|dq|d krKg}'|D] }|'|dq8d|'i}(|&|(fSd|i}(|&|(fSi}(|&|(fS)z Internal method meant to only be used by asr pipeline. Handles all the little quirks specific to whisper to handle the various options not allowed in other seq2seq models NcsddgddS)Nr)rr9rrrZ last_languagerr new_chunkrSz_decode_asr..new_chunkr*rr FrNrr[rr+token_timestampsZstriderr+rrTr9wordscsrrr)rprrrrFrz_decode_asr..zWhisper did not predict an ending timestamp, which can happen if audio is cut off in the middle of a word. Also make sure WhisperTimeStampLogitsProcessor was used during generation.rcss|]}|dVqdS)rNr)rchunkrrrr`schunks)rr)r1r2r}rOreversedr.rrr"_find_longest_common_sequenceround_collate_word_timestampsranyrgrmrrWrr)) tokenizerrvrsrtr4rrrZ time_offsetr6Zprevious_tokensZprevious_token_timestampsskipZright_stride_startr1rPrQZchunk_idoutputr3rZlast_timestampZfirst_timestampZ chunk_lenZ stride_leftZ stride_rightrcurrent_tokensZcurrent_token_timestampsrrrZresolved_tokensZ resolved_texttimeZresolved_token_timestamps start_timeend_time full_textZ new_chunksoptionalrrrrus                                      ruc s|d}t|}g}rdg}t|ddD]\}d}||ddf}t|} td|| D]r} | d} td|| t||| | } t|| } td| |t| | }t||t| tkrqtdrtfddt| D}nt| k}|| | }|dkr||kr|}| |f}q1|\} }| d}|d}| |d|||d}t|}r| d|d|dq| |dur|Stdkr| ||fS|gfS) Nrr r*g@ziThere is a bug within whisper `decode_asr` function, please report it. Dropping to prevent bad inference.c3sD|]\}}||kr|d|krdVqdS)r Nr)ridxelemZ left_startZleft_token_timestamp_sequencerightZ right_startZseq_idxtoken_timestamp_sequencesrrrs  z0_find_longest_common_sequence..r) rr2r maxrrArB RuntimeErrorrEr) sequencesrZ left_sequenceZ left_lengthZtotal_sequenceZtotal_token_timestamp_sequenceZright_sequenceZmax_Z max_indicesZ right_lengthrZepsZ left_stopleftZ right_stopmatchesZmatchingZleft_midZ right_midrrrrvsb            rc s@t|||\}}}|rd|inifddt||D}|S)Nrcs:g|]\}}||dd|ddfdqS)rrr r=r)rr+indicesZoptional_language_fieldrrrrsz,_collate_word_timestamps..)_combine_tokens_into_wordsr$) rr[rrrtr_ token_indicesZtimingsrrrrs  r "'“¡¿([{-"'.。,，!！?？:：”)]}、r[rprepend_punctuationsappend_punctuationscCs^|dur|j}|dur d}|dvrt||\}}}nt||\}}}t||||||||fS)z Groups tokens by word. Returns a tuple containing a list of strings with the words, and a list of `token_id` sequences with the tokens making up each word. Nr0>rr>rrerr2)r_split_tokens_on_unicode_split_tokens_on_spaces_merge_punctuations)rr[rrrr word_tokensrrrrrs  rc Cs|j|dd}d}g}g}g}g}g}d} t|D]=\} } || || |j|dd} || vs=|| | ||krV|| ||||g}g}| t| 7} q|||fS)zlCombine tokens into words by splitting at any position where the tokens are decoded as valid unicode points.T)rVu�r)r.r2r"rr) rr[Z decoded_fullreplacement_charrrrrZcurrent_indicesZunicode_offsetZ token_idxrdecodedrrrrs,       rcCst||\}}}g}g}g}t|||D]I\}} } | d|jk} |d} |dv} | s7| s7| s7t|dkrG|||| || q|d||d<|d| |d| q|||fS)zLCombine tokens into words by splitting at whitespace and punctuation tokens.rrz !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~r)rr$r startswithrxrr"r)rr[ZsubwordsZsubword_tokens_listZsubword_indices_listrrrZsubwordZsubword_tokensZsubword_indicesrZ with_space punctuationrrrr7s       rcCst|d}t|d}|dkrT||drJ|||vrJ||||||<||||||<||||||<d||<g||<g||<n|}|d8}|dksd}d}|t|kr||ds|||vr||||7<||||7<||||7<d||<g||<g||<n|}|d7}|t|ks^dd|D|dd<d d|D|dd<d d|D|dd<dS) z1Merges punctuation tokens with neighboring words.rr rrrcSg|]}|r|qSrr)rr+rrrrprz'_merge_punctuations..NcSrrr)rrrrrrqrcSrrr)rrrrrrrr)rrrxendswith)rr[rZ prependedZappendedrrrrrrOs:        rr)Nrr)-rrrdr functoolsrtypingrrrrr|rAregexrZtokenization_utilsrr utilsr Zenglish_normalizerr r riZMAX_MODEL_INPUT_SIZESr(Z get_loggerr~rgr.rrr r rrurrrrrrrr__all__rrrrs          !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVhD fx  &