o Zh @sxddlZddlmZddlmZddlmZddlmZm Z e e Z er*ddl Z ddZ   d1d eed ed d eed edeffddZ   d1d eed ed d eed edeffddZ   d1d eed ed d eed edeffddZ d2d ed d d eed edeffddZ d2d ed d d eed edeffddZ d2d ed d d eed edeffddZeeeeeedZ  d3dedededeed eef d!d"Zd2d ed eefd#d$Zd2d ed eefd%d&Zd2d ed eefd'd(Zd2d ed eefd)d*Zd2d ed eefd+d,Z d2d ed eefd-d.Z!eeeee e!dZ"d2d ed eefd/d0Z#dS)4Nwraps)Optional)PretrainedConfig)is_torch_availableloggingcs,ddddtfdd}|S)ad Decorator function to update the RoPE parameters in the forward pass, if the model is using a dynamic RoPE (i.e. a RoPE implementation that may recompute its frequencies in the forward pass). Args: rope_forward (Callable): The forward pass of the RoPE implementation. Returns: The decorated forward pass. cSst|d}t|jdr|jj}n|jj}||kr8t|ds-|j|j||dd\|_}|jd|jdddS|j ||_ |jd|j dddS) zbLongrope uses long factor if sequence is larger than original pretraining length, short otherwise.r original_max_position_embeddings long_inv_freqseq_leninv_freqF persistentN) torchmaxhasattrconfigr max_position_embeddings rope_init_fnr register_bufferoriginal_inv_freqto)self position_idsdevicer r _rO/var/www/auris/lib/python3.10/site-packages/transformers/modeling_rope_utils.pylongrope_frequency_update+s     z6dynamic_rope_update..longrope_frequency_updatecSst|d}||jkr#|j|j||d\}|_|jd|dd||_||jkrD|j|jkrF|j ||_|jd|jdd|j|_dSdSdS)a dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow scaling) 2 - the current sequence length is in the original scale (avoid losing precision with small sequences) rr r FrN) rrZmax_seq_len_cachedrrZattention_scalingrZoriginal_max_seq_lenrr)rrrr r rrrdynamic_frequency_update>s  z5dynamic_rope_update..dynamic_frequency_updatecsBd|jvr|||jdn |jdkr|||jd|||S)Ndynamic)rlongrope) rope_typer)rxrr r rope_forwardrrwrapperQs   z$dynamic_rope_update..wrapperr)r&r'rr%rdynamic_rope_updates  r(rrz torch.devicer returnz torch.Tensorc Ks|durt|dkrtd|d|t|dkr#|d}|d}n#|durF|j}t|dr2|jnd}t|d dp?|j|j}t||}d}d|t j d|d t j d j |t j d |} | |fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). NrzUnexpected arguments: `**rope_kwargs` and `config` are mutually exclusive in `_compute_default_rope_parameters`, got `rope_kwargs`= and `config`=basedimpartial_rotary_factor?head_dimdtyperr2)len ValueError rope_thetarr-getattr hidden_sizenum_attention_headsintrarangeint64rfloat) rrr rope_kwargsr+r,r-r/attention_factorr rrr _compute_default_rope_parameters\s&   ,r@cKsx|durt|dkrtd|d|t|dkr|d}n |dur(|jd}t|||fi|\}}||}||fS)a Computes the inverse frequencies with linear scaling. Credits to the Reddit user /u/kaiokendev Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). NrzUnexpected arguments: `**rope_kwargs` and `config` are mutually exclusive in `_compute_linear_scaling_rope_parameters`, got `rope_kwargs`=r*factor)r4r5 rope_scalingr@)rrr r>rAr r?rrr'_compute_linear_scaling_rope_parameterss   rCc Ks|durt|dkrtd|d|t|dkr+|d}|d}|d}|d}n)|durT|j}t|d r:|jnd }t|d |j|j} t| |}|j }|j d}d } |dur`||kr`|n|}|||||d ||d }d |t j d|d t j dj|t jd|} | | fS)a4 Computes the inverse frequencies with NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length, used to update the dynamic RoPE at inference time. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). NrzUnexpected arguments: `**rope_kwargs` and `config` are mutually exclusive in `_compute_dynamic_ntk_parameters`, got `rope_kwargs`=r*r+r,rrAr-r.r/rr0r1r3)r4r5r6rr-r7r8r9r:rrBrr;r<rr=) rrr r>r+r,rrAr-r/r?r rrr_compute_dynamic_ntk_parameterss2    $,rDc st|dkr td||j}t|dr|jnd}t|d|j|j}t||}|j d}|j d} |j d} |j d } d |j vrQ|j d } |j | }n|j } dd d } | durr| rn| rnt | || | || } n| |} |j dpyd}|j dpd }ddfdd}dd}|t d|dj|t j d|}d|}d||}|||||| \}}d ||||dj|t j d}|d |||}|| fS)a Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://arxiv.org/abs/2309.00071) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rzYUnexpected arguments: `**rope_kwargs` should be unset in `_compute_yarn_parameters`, got r-r.r/rAr?mscalemscale_all_dimr rcSs"|dkrdSd|t|dS)Nrr.g?)mathlog)scalerErrr get_mscale sz,_compute_yarn_parameters..get_mscaleN beta_fast beta_slowcSs*|t||dtjdt|S)zPInverse dimension formula to find the dimension based on the number of rotationsr0)rGrHpi)Z num_rotationsr,r+rrrrfind_correction_dims*z5_compute_yarn_parameters..find_correction_dimcs@t||||}t||||}t|dt||dfS)z.Find dimension range bounds based on rotationsrr)rGfloorceilrmin)Zlow_rotZhigh_rotr,r+rlowhighrOrrfind_correction_range#sz7_compute_yarn_parameters..find_correction_rangecSs>||kr|d7}tj|tjd|||}t|dd}|S)NgMbP?r1rr)rr;float32clamp)rRrr,Z linear_funcZ ramp_funcrrrlinear_ramp_factor)s z4_compute_yarn_parameters..linear_ramp_factorr0r3)r)r4r5r6rr-r7r8r9r:rBgetrr=rr;r)rrr r>r+r-r/r,rAr?rErFr rJrKrMrVrYZ pos_freqsZinv_freq_extrapolationZinv_freq_interpolationrSrTZinv_freq_extrapolation_factorr rrUr_compute_yarn_parameterssH           "    r[cKs@t|dkr td||j}t|dr|jnd}t|d|j|j}t||}|j d}|j d} |j d} |j d } t|d rO|j } |j |j } n|j } | d url| dkr]d} nt d t | t | } |r||| kr|tj|tj|d } n tj| tj|d } tjd|dtj|d |}d| ||}|| fS)a  Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rz]Unexpected arguments: `**rope_kwargs` should be unset in `_compute_longrope_parameters`, got r-r.r/ long_factor short_factorrAr?r Nr)r2rr0)r4r5r6rr-r7r8r9r:rBrZr rrGsqrtrHrZtensorrWr;r<r=)rrr r>r+r-r/r,r\r]rAr?r Z ext_factorsZinv_freq_shaper rrr_compute_longrope_parametersBs8        r_cKst|||fi|\}}|jd}|jd}|jd}|jd} | |} | |} dtj|} t| | k|||} | | |||}d|| ||| }| | k| | k}t||| } | |fS)a Computes the inverse frequencies for llama 3.1. Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rAlow_freq_factorhigh_freq_factorr r0r)r@rBrGrNrwhere)rrr r>r r?rAr`raZold_context_lenZlow_freq_wavelenZhigh_freq_wavelenZwavelenZinv_freq_llamaZ smooth_factorZsmoothed_inv_freqZis_medium_freqrrr_compute_llama3_parameterss    rc)defaultZlinearr!yarnr"Zllama3r# received_keys required_keys optional_keys ignore_keyscCsd|vr|dh8}|d|dur||8}||}|r&td|d||dur1|||}n||}|rDtd|d|dSdS)zYCompare the received keys in `config.rope_scaling` against the expected and optional keystyper#Nz9Missing required keys in `rope_scaling` for 'rope_type'='z': z5Unrecognized keys in `rope_scaling` for 'rope_type'=')addKeyErrorloggerwarning)r#rfrgrhriZ missing_keysZ unused_keysrrr_check_received_keyss  rocCs@|j}|d|dd}dh}t|}t||||ddS)Nr#rjri)rBrZsetkeysro)rrirBr#rgrfrrr!_validate_default_rope_parameterss  rscCsx|j}|d|dd}ddh}t|}t||||d|d}|dus0t|tr0|dkr:td|dSdS)Nr#rjrArpr.8`rope_scaling`'s factor field must be a float >= 1, got rBrZrqrrro isinstancer=rmrn)rrirBr#rgrfrArrr(_validate_linear_scaling_rope_parameterss rwcCs|j}|d|dd}ddh}dh}t|}t|||||d|d}|dus4t|tr4|dkr>td|dSdS)Nr#rjrAr rpr.rtru)rrirBr#rgrhrfrArrr)_validate_dynamic_scaling_rope_parameterss rxc Cs6|j}|d|dd}ddh}hd}t|}t|||||d|d}|dus5t|tr5|dkr=td||d}|durWt|trO|d krWtd ||d } | durmt| tsmtd | |d } | durt| tstd| | pd| pdkrtd| d| ddSdS)Nr#rjrA>rFrMr?rKr rErpr.rtr?rL`rope_scaling`'s attention_factor field must be a float greater than 0, got rKz6`rope_scaling`'s beta_fast field must be a float, got rMz6`rope_scaling`'s beta_slow field must be a float, got rLrzO`rope_scaling`'s beta_fast field must be greater than beta_slow, got beta_fast=z( (defaults to 32 if None) and beta_slow=z (defaults to 1 if None)ru) rrirBr#rgrhrfrAr?rKrMrrr_validate_yarn_parameterss6    rzcCs|j}|d|dd}hd}hd}t|}t|||||dt|dr,|jnd}t|d|j|j }t ||} |d } t | t sYt d d | DrYtd | t| | d ksptd| d dt| |d} t | t st dd | Drtd| t| | d kstd| d dt| t|drtddS|d} | durtdnt | tr| dkrtd| |d} | durt | tr| dkrtd| dSdSdS)Nr#rj>r#r]r\>rAr?r rpr-r.r/r]cs|] }t|ttfVqdSNrvr:r=.0r$rrr ,z0_validate_longrope_parameters..zC`rope_scaling`'s short_factor field must be a list of numbers, got r0z5`rope_scaling`'s short_factor field must have length z, got r\csr{r|r}r~rrrr2rzB`rope_scaling`'s long_factor field must be a list of numbers, got z4`rope_scaling`'s long_factor field must have length r aYThis model has set a `original_max_position_embeddings` field, to be used together with `max_position_embeddings` to determine a scaling factor. Please set the `factor` field of `rope_scaling`with this ratio instead -- we recommend the use of this field over `original_max_position_embeddings`, as it is compatible with most model architectures.rAz1Missing required keys in `rope_scaling`: 'factor'rtr?gry)rBrZrqrrrorr-r7r8r9r:rvlistallrmrnr4Z warning_oncer=)rrirBr#rgrhrfr-r/r,r]r\rAr?rrr_validate_longrope_parameterssH        rc Cs6|j}|d|dd}hd}t|}t||||d|d}|dus0t|tr0|dkr8td||d}|d }|dusIt|tsQtd ||dusZt|tsbtd |||krqtd |d ||d} | dus~t| t std| | |j krtd| d|j dSdS)Nr#rj>rAr`r#r rarprAr.rtr`raz<`rope_scaling`'s low_freq_factor field must be a float, got z=`rope_scaling`'s high_freq_factor field must be a float, got zc`rope_scaling`'s high_freq_factor field must be greater than low_freq_factor, got high_freq_factor=z and low_freq_factor=r zP`rope_scaling`'s original_max_position_embeddings field must be an integer, got zg`rope_scaling`'s original_max_position_embeddings field must be less than max_position_embeddings, got z and max_position_embeddings=) rBrZrqrrrorvr=rmrnr:r) rrirBr#rgrfrAr`rar rrr_validate_llama3_parametersPsL  rcCsdt|dd}|dur dS|d|dd}t|}|dur'|||ddStd|ddS) zO Validate the RoPE config arguments, given a `PretrainedConfig` object rBNr#rjrdrpzTMissing validation function mapping in `ROPE_VALIDATION_FUNCTIONS` for 'rope_type'='')r7rZROPE_VALIDATION_FUNCTIONSrmrn)rrirBr#Z validation_fnrrrrope_config_validations   r)NNNr|)NN)$rG functoolsrtypingrZconfiguration_utilsrutilsrrZ get_logger__name__rmrr(r:tupler=r@rCrDr[r_rcZROPE_INIT_FUNCTIONSstrrqrorsrwrxrzrrrrrrrrs    ?  +  *  4  a  @  /  &2&