o Zhe @sddlZddlZddlmZmZmZmZmZddlZ ddl Z ddl m Z ddl mZddlmZer9ddlmZddlmZddl mZd d lmZmZmZercdd lmZdd lmZd d lmZddl m!Z!GdddZ"Gddde"Z#Gddde#Z$Gddde j%Z&Gddde j%Z'GdddZ(GdddZ)Gddde$Z*Gdd d e"Z+Gd!d"d"e#Z,d#d$Z-d%ee.efd&e/d'e0d(ee.effd)d*Z1d%ee.efd&e/d(ee.effd+d,Z2dS)-N) TYPE_CHECKINGAnyDictOptionalTuple)prune_linear_layer)is_sklearn_available) roc_curve)Cache)isin_mps_friendly)LogitsProcessorListMinLengthLogitsProcessorSuppressTokensLogitsProcessor)PreTrainedModel)PreTrainedTokenizerBase)GenerationConfig)deprecate_kwargc@sNeZdZdZdejdeejeejffddZ dejdejde fdd Z d S) CandidateGeneratorz`Abstract base class for all candidate generators that can be applied during assisted generation. input_idsreturncCt|jd)ag Fetches the candidates to be tried for the current input. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) Return: `torch.LongTensor` of shape `(batch_size, candidate_length)` containing the candidate sequences to be assessed by the model and, optionally, a `torch.FloatTensor` of shape `(batch_size, candidate_length, vocabulary_size)` containing the logits associated to each candidate. zT is an abstract class. Only classes inheriting this class can call `get_candidates`.NotImplementedError __class__)selfrrZ/var/www/auris/lib/python3.10/site-packages/transformers/generation/candidate_generator.pyget_candidates/ z!CandidateGenerator.get_candidatesscores num_matchescCr) Updates the candidate generation strategy based on the outcomes. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) scores (`torch.FloatTensor` of shape `(batch_size, candidate_length, config.vocab_size)`): Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam search or log softmax for each vocabulary token when using beam search num_matches (`int`): The number of matches between the candidate sequences and the model predictions. z_ is an abstract class. Only classes inheriting this class can call `update_candidate_strategy`.rrrr!r"rrrupdate_candidate_strategy@r z,CandidateGenerator.update_candidate_strategyN) __name__ __module__ __qualname____doc__torch LongTensorrr FloatTensorrintr%rrrrr,s$rc @seZdZdZ  d$dejddddded eejd d f d d Z dejde ejeej ffddZ dejdej de fddZdejde e e ffddZ d%dejde de defddZdejde de defdd Zd!ede ejeej ffd"d#ZdS)&AssistedCandidateGeneratorar `CandidateGenerator` class to be used for assisted generation and speculative decoding. This class generates candidates through the use of a smaller model. Read the following blog post for more information: https://huggingface.co/blog/assisted-generation Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) assistant_model (`PreTrainedModel`): The model to be used for generating candidates. This model should be smaller than the main model. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. model_kwargs (`Dict`): The keyword arguments that will be passed to the main model, and are used as base inputs for the assistant model as well. inputs_tensor (`torch.Tensor`, *optional*): The model input tensor. In encoder-decoder models, this is the encoder input. Nrassistant_modelrgeneration_configr model_kwargs inputs_tensorlogits_processorrc Cs|j}||}|dur||}||_|jj|_|jj|_|j|jj_i}|D]\} } | dvrFt| t j r?| |nt | || <q*d|vrT|sT|d=|jjrn|||jj|\}} }|||| |j}n d|vrx|d|d<||_|jjrd|_n"d|vrd|_|jdt j|jddf|jt jd |jd <nd|_|dur|nt|_t ||_d |j_d |j_|j|j_d |j_|jj |_!d|j_ d|j_"|jD] } t| t#rt$d qd|j_%t&r|jjjrt'|t(urg|_)g|_*dSdSdSdS) N)encoder_outputspast_key_valuesZlogits_to_keepr4Zdecoder_input_idsrdecoder_attention_maskrr )devicedtypeattention_maskTzPassing `MinLengthLogitsProcessor` when using `assisted_generation is disabled. Please pass in `min_length` into `.generate()` instead)+r7tor/r0num_assistant_tokensassistant_confidence_threshold eos_token_iditems isinstancer*TensordetachcopydeepcopykeysZ_supports_logits_to_keepconfigis_encoder_decoderZ_prepare_model_inputsZ bos_token_idZ._prepare_encoder_decoder_kwargs_for_generationassistant_kwargs input_ids_keygetZonesshapelongrr3return_dict_in_generate output_scoresZ is_assistantZ min_lengthmain_model_min_lengthmin_new_tokensr ValueErrorZcache_implementationr typer.probsmatches) rrr/r0r1r2r3r7rGkeyvalueZmodel_input_name processorrrr__init__jsx     "           z#AssistedCandidateGenerator.__init__rcCsZ||jj}||\}}|dkr|dfS||||||}||\}}||fS)Z Fetches the candidates to be tried for the current input. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) Return: `torch.LongTensor` of shape `(batch_size, candidate_length)` containing the candidate sequences to be assessed by the model and a `torch.FloatTensor` of shape `(batch_size, candidate_length, vocabulary_size)` containing the logits associated to each candidate. rN)r:r/r7_calculate_new_tokens_update_past_and_masks_prepare_generation_args_generate_candidates)rrrOmax_new_tokensgeneration_args candidate_idscandidate_logitsrrrrs  z)AssistedCandidateGenerator.get_candidatesr!r"c Cs6|jjjdvr"|t|ddkr|jd7_n td|jd|_tr|jjjrt|t ur|j dg|t|j t|j krI|j dt|j t|j }|dkr^|j | d=t|j dkrddh|j rt|j |j \}}}d|}|d|} t| } || } | |jj_dSdSdSdSdSdS) r#> heuristicZheuristic_transientrr g@g?N)r/r0Znum_assistant_tokens_schedulelenr;maxr r<rQr.rSextendrRappendissubsetr npZargmin) rrr!r"Z excess_lengthZfprZtprZ thresholdsZfnrZcostsZoptimal_threshold_indexZbest_thresholdrrrr%s2     z4AssistedCandidateGenerator.update_candidate_strategycCsD|jd}tt|j|jj|d}tt||j|d}||fS)zCCalculate the minimum and maximum number of new tokens to generate.r r)rJminr-r;r0 max_lengthrerN)rrZ new_cur_lenr]rOrrrrYs z0AssistedCandidateGenerator._calculate_new_tokensrr remove_from_pkvnum_added_tokenscCsz|jdddu}|r;|jdd|}t|j|jd|||jd<t|j|jd|jjj|_t|j|jd|_|S)zLUpdate past key values and attention masks for subsequent generation rounds.r5Nrjr ) rGrIrJ_crop_past_key_valuesr/_prepare_attention_maskrErF_prepare_token_type_ids)rrrmrnZhas_past_key_valuesZnew_cache_sizerrrrZ#s z1AssistedCandidateGenerator._update_past_and_masksrOr]c Cs|j|d|d|d|jd|jiS)z*Prepare arguments for the generation call.rOr]r0r3)rHr0r3)rrrOr]rrrr[4s z3AssistedCandidateGenerator._prepare_generation_argsr^c Cs|jjdi||j}|j|jd<trN|jjjrNt|turNt j |j dd}t j |dd}|j dt|j  df}|tt||f}|j|t j|j dd}|j }||fS)z7Generate candidate sequences using the assistant model.r5rdimrjNr r)r/generaterGr5r r0r<rQr.r*catr!Zsoftmax sequencesrdrangerRrftoliststack) rr^assistant_outputZ scores_tensorZscores_softmaxZidspr`r_rrrr\>s  z/AssistedCandidateGenerator._generate_candidatesNN)rr )r&r'r(r)r*r+rrr@rWrr,rr-r%rYboolrZr[r\rrrrr.Ss@ $_: & r.cseZdZdZ  d"dejddddddd d d ed eejd dffdd Z e ddZ e ddZ e ddZ ddZdejdeejeejffddZdejdeejeffddZdejdejdejdejfd d!ZZS)#-AssistedCandidateGeneratorDifferentTokenizersa `CandidateGenerator` class to be used for Universal Assisted Generation (UAD): assisted generation with different tokenizers for the assistant and main models. This class generates candidates through the use of a smaller model. The main model input tokens are re-encoded into assistant model tokens, then candidate tokens are generated in the assistant encoding, which are in turn re-encoded into main model candidate tokens. Validation then proceeds as explained above. The re-encoding steps involve decoding token ids into text and then encoding the text using a different tokenizer. Since re-encoding the tokens may result in tokenization discrepancies, UAD finds the longest common subsequence between the source and target encodings, to ensure the new tokens include the correct prompt suffix. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) assistant_model (`PreTrainedModel`): The model to be used for generating candidates. This model should be smaller than the main model. target_tokenizer (`PreTrainedTokenizerBase`): The tokenizer used for the target model. assistant_tokenizer (`PreTrainedTokenizerBase`): The tokenizer used for the assistant model. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. model_kwargs (`Dict`): The keyword arguments that will be passed to the main model, and are used as base inputs for the assistant model as well. inputs_tensor (`torch.Tensor`, *optional*): The model input tensor. In encoder-decoder models, this is the encoder input. Nrr/rtarget_tokenizerrassistant_tokenizerr0rr1r2r3rc sFt||||||||_||_d|_d|_|jj|_|jj|_dSN) superrWrrprev_target_ids_lenprev_assistant_idsr0target_lookbehindassistant_lookbehind) rrr/rrr0r1r2r3rrrrWrs  z6AssistedCandidateGeneratorDifferentTokenizers.__init__cCst}i}|D]f}t|}t|}||vrq||d}|d7}|d|jdkri|d|jdkrit|}||||d|dfdkrV|d7}|d7}nn|d|jdkri|d|jdks6|||<q|S)a Calculates the length of the longest diagonal sequence in a given matrix. Args: input_matrix (torch.Tensor): The input matrix. nonzero_idx (torch.Tensor): The indices of the non-zero elements in the matrix. Returns: dict: A dictionary where the keys are the indices of the non-zero elements and the values are the lengths of the longest diagonal sequences starting from those indices. r r)setr*clonetuplerxaddrJ) input_matrixZ nonzero_idxvisiteddiagsidx start_idxZtuple_start_idxZ cur_diag_lenrrr_get_longest_diag_dicts(    $   $ zDAssistedCandidateGeneratorDifferentTokenizers._get_longest_diag_dictcCsJt||}t|}t|}t|}||}||}||fS)a Returns the start index and length of the longest diagonal in the given input. Args: input_matrix (numpy.ndarray): The input matrix. Returns: tuple: A tuple containing the start index and length of the longest diagonal. )r~rnonzerolistvaluesrDriZargmax)rrZ diags_valuesZ diags_keysZ best_diagZdiag_start_indexZdiag_start_lengthrrr_get_longest_diag_indexs    zEAssistedCandidateGeneratorDifferentTokenizers._get_longest_diag_indexc Cs|j|k}t|st|}|t}|sdSt |\}}|d|}|d|}|j d|}|dd||df} |dd|||f} || | fS)a Input: prompt: 2D array of shape (batch_size, prompt_length), represents the original prompt tokens prompt_plus_new_tokens: 2D array of shape (batch_size, prompt_length), represents the suffix of the original prompt, with additional new tokens. Output: discrepancy_length: int, represents the number of tokens that need to be replaced from prompt new_tokens_only: 2D array of shape (batch_size, new_token_length), represents the new tokens that are not in prompt discrepancy_only: 2D array of shape (batch_size, discrepancy_length), represents the new tokens that are in prompt but not in prompt_plus_new_tokens )NNNrr N) Tr*Z is_tensortensorr:r-anyitemr~rrJ) promptZprompt_plus_new_tokensZ compare_matZcompare_mat_intZlongest_locationZlongest_diag_lengthZnew_token_start_indexZdiscrepancy_with_olddiscrepancy_lengthnew_tokens_onlydiscrepancy_onlyrrr_get_tokens_diags"       z>AssistedCandidateGeneratorDifferentTokenizers._get_tokens_diagcCs.|j|ddd}||dddd}||jS)a$ Convert token IDs from one tokenizer to another. Args: input_ids: The input token IDs. source_tokenizer: The source tokenizer. destination_tokenizer: The destination tokenizer. Returns: The converted token IDs. TZskip_special_tokensZclean_up_tokenization_spacesptadd_special_tokensZreturn_tensorsr) batch_decoder:r7)rrsource_tokenizerdestination_tokenizertextZdest_idsrrr&convert_source_tokens_to_target_tokenss zTAssistedCandidateGeneratorDifferentTokenizers.convert_source_tokens_to_target_tokensrc Cst|j}|dkr |dfS||jj}d}||\}}||_tt||j |j dd}| ||| |||}|j dd|jjdi||j }|||j|}|j d|_|j|j d<|j|_|j|j dkrr|dfS|dfS)rXrNrjr9r r5r)r-r;r:r/r7_prepare_assistant_input_idsrrerkrNrJrZr[rGpoprt_process_assistant_outputsrvrr5) rrr]rmassistant_input_idsrOr^rznew_target_idsrrrrs&    zszRAssistantToTargetTranslator._get_assistant_to_target_input_ids..r r8)rZ get_vocabrrdZconvert_ids_to_tokensr>rerr*fullrr-rIr:r) rZ target_vocabZassistant_vocabZ space_strZtarget_space_idsZassistant_space_idsZmax_assistant_indexZassistant_to_target_input_idsrrZ assistant_idZ target_idrrrrs.       z>AssistantToTargetTranslator._get_assistant_to_target_input_idsrcCst|j|jkdS)z` Get the input ids that are in the assistant vocab but not in the target vocab. r)r*whererrrrrrrsz3AssistantToTargetTranslator._get_suppress_input_idsassistant_candidate_idscCsft|d|jd}|dkr|S|d| df}|jr"|j|}|j|}tj||dfddS)aS Return the target candidate ids that correspond to the assistant candidate ids. Note that we have already the target ids for the prompt and we only need to find the target ids for the new tokens. Moreover, assistant ids of the original prompt does not necessarily appear in _assistant_to_target_input_ids. rr Nrr)rdrJrrrr*rur)rrtarget_input_idsrZnum_new_tokensZlast_candidate_idsZtransformed_slicerrrget_target_idss   z*AssistantToTargetTranslator.get_target_idsassistant_logitscCsg|jdd|jR}tj||j|jd}|j|jk}|j|}|jr-||d|f<|S|dd|jjdf}|d|f|d|f<|S)zS Return the target logits that correspond to the assistant logits. Nrjr7.r) rJrr*rrrrrr)rrZ target_shapeZ target_logitsZassistant_indices_maskZtarget_logits_supported_indicesZvalid_assistant_logitsrrrget_target_logits4s    z-AssistantToTargetTranslator.get_target_logitsrNF)r&r'r(r)floatr__annotations__rr-rstrrr}rWrrrrr*r+rr,rrrrrrs<    , # rc@sfeZdZdZeZeeddd   ddd d d d e de d e d de de fddZeddZdS)AssistantVocabTranslatorCachez Cache for `AssistantToTargetTranslator` instances. The instances are computed at pre-processing time, and this cache allows us to avoid recomputing them. rrrrNFrrrrr/rrrc CsV|j|}|durt}||j|<||}|dur)t||||||}|||<|Sr)_cacherIweakrefWeakKeyDictionaryr) clsrrrrr/rassistant_dictmappingrrrget_translatorRs   z,AssistantVocabTranslatorCache.get_translatorcCsTdd|jD}|D]}|j|=q |jD]}dd|D}|D]}||=q!qdS)z Clean up dead references in the cache. This removes entries where either the target_tokenizer or assistant_tokenizer has been garbage collected. cSg|]}|dur|qSrrrrTrrr xz9AssistantVocabTranslatorCache.cleanup..cSrrrrrrrr~rN)rr)rZ dead_keysrTrrrrcleanupps z%AssistantVocabTranslatorCache.cleanupr)r&r'r(r)rrr classmethodrr-rrr}rrrrrrrrJs2 rcseZdZdZ  ddejddddddd d d ed ed eej ddffdd Z dejde ejeej ffddZ ddejdedeffdd ZdejdejfddZZS)%UniversalSpeculativeDecodingGeneratorz `CandidateGenerator` class to be used for Universal Speculative Decoding (USD): speculative decoding with different tokenizers for the assistant and main models. This class generates candidates through the use of a smaller model. Nrr/rrrrr0rr1atm_translatorr2r3rc s0||_t|||||||| d|_d|_dS)Nr)_atm_translatorrrW_target_seq_len_with_candidates_prev_assistant_ids) rrr/rrr0r1rr2r3rrrrWs  z.UniversalSpeculativeDecodingGenerator.__init__rc Cs||jj}||\}}||\}}|dkr|dfS|j||d||||}d|d_d|d_|j j durA|j j |d<| |\|_ }|j |||j } | jd|_|j |} | | fS)zk Simplified version of get_candidates that uses the translator cache for token conversion. rNrnTr0r3rj)r:r/r7rrYrZr[rMrLrrr\rrrJrr) rrrrrnrOr]r^Zassistant_candidate_logitsZtarget_candidate_idsZtarget_candidate_logitsrrrrs$      z4UniversalSpeculativeDecodingGenerator.get_candidatesr rrncs6|jdurt|j|jd|jjj|_tj||dS)Nrjr) rrprGrJr/rErFrrZ)rrrnrrrrZs zsF          '"'9tp8*)*$