eTh SSKrSSKJr SSKJr SSKJr SSKJrJ r \ R"\ 5r \"5(aSSK r SrS"S\\S \S S \\S \S \44SjjrS"S\\S \S S \\S \S \44SjjrS"S\\S \S S \\S \S \44SjjrS#S\S S S \\S \S \44SjjrS#S\S S S \\S \S \44SjjrS#S\S S S \\S \S \44Sjjr\\\\\\S.rS$S\S\S\S\\S\\4 SjjrS#S\S\\4SjjrS#S\S\\4SjjrS#S\S\\4SjjrS#S\S\\4SjjrS#S\S\\4Sjjr S#S\S\\4S jjr!\\\\\ \!S.r"S#S\S\\4S!jjr#g)%Nwraps)Optional)PretrainedConfig)is_torch_availableloggingcD^^^SmSm[T5UUU4Sj5nU$)aD Decorator function to update the RoPE parameters in the forward pass, if the model is using a dynamic RoPE (i.e. a RoPE implementation that may recompute its frequencies in the forward pass). Args: rope_forward (Callable): The forward pass of the RoPE implementation. Returns: The decorated forward pass. c[R"U5S-n[URS5(aURRnOURR nX4:aR[US5(d%UR URX$S-S9uUlnURSURSS9 gURRU5Ul URSURSS9 g) zbLongrope uses long factor if sequence is larger than original pretraining length, short otherwise.r original_max_position_embeddings long_inv_freqseq_leninv_freqF persistentN) torchmaxhasattrconfigr max_position_embeddings rope_init_fnr register_bufferoriginal_inv_freqto)self position_idsdevicerr _s X/var/www/auris/envauris/lib/python3.13/site-packages/transformers/modeling_rope_utils.pylongrope_frequency_update6dynamic_rope_update..longrope_frequency_update+s))L)A- 4;; B C C/3{{/[/[ ,/3{{/R/R ,  5411(,(9(9KKTU1U):)%"A  T-?-?E R&*%;%;%>%>v%FD "  T-C-CPU Vc[R"U5S-nX0R:a8URURX#S9uo@lUR SUSS9 X0lX0R:ahURUR:aMURRU5UlUR SURSS9 URUlggg)z dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow scaling) 2 - the current sequence length is in the original scale (avoid losing precision with small sequences) rrrFrN) rrmax_seq_len_cachedrrattention_scalingroriginal_max_seq_lenrr)rrrrrs r dynamic_frequency_update5dynamic_rope_update..dynamic_frequency_update>s ))L)A- ,, ,/3/@/@f/@/f ,H,  X% H&- # .. .43J3JTMfMf3f&*%;%;%>%>v%FD "  T-C-CPU V&*&?&?D # 4g .r#c>SUR;aT"XURS9 O!URS:XaT"XURS9 T"XU5$)Ndynamic)rlongrope) rope_typer)rxrr(r! rope_forwards r wrapper$dynamic_rope_update..wrapperQsD  & $T I ^^z ) %d JD\22r#r)r/r0r(r!s` @@r dynamic_rope_updater2s/W&@& <33 Nr#rrz torch.devicerreturnz torch.Tensorc Ub [U5S:a[SUSU35e[U5S:a USnUSnOhUbeURn[US5(a UROSn[ US S5=(d UR UR-n[Xv-5nSnSW[R"SWS [RS 9RU[RS 9U- -- n X4$) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). NrzUnexpected arguments: `**rope_kwargs` and `config` are mutually exclusive in `_compute_default_rope_parameters`, got `rope_kwargs`= and `config`=basedimpartial_rotary_factor?head_dimdtyperr=)len ValueError rope_thetarr8getattr hidden_sizenum_attention_headsintrarangeint64rfloat) rrr rope_kwargsr6r7r8r:attention_factorrs r _compute_default_rope_parametersrK\s*c+.2 EEPMQ_`f_g i   ;!6"%     @GPg@h@h < )r?r@rArr8rBrCrDrErrNrrFrGrrH) rrrrIr6r7rrMr8r:rJrs r _compute_dynamic_ntk_parametersrQs,c+.2 DDO=P^_e^f h   ;!6"% "-.G"HX&    @GPg@h@h < >6A:NTW[^ab[bTcd dDdu||AsAU[[ILLTZbgbmbmLnqttuvH  %%r#c j^[U5S:a[SU35eURn[US5(a UROSn[ USUR UR-5n[Xe-5nURSnURRS5n URRS5n URRS 5n S UR;aURS n URU - nO URn SS jn U c1U (a"U (a[U "X5U "X5- 5n OU "U5n URRS 5=(d SnURRS5=(d S nSmU4SjnSnU[R"SUS5RU[RS9U- -nSU- nSUU-- nU"XXtU 5unnS U"UUUS-5RU[RS9- nUS U- -UU--nUU 4$)a Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://arxiv.org/abs/2309.00071) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rzYUnexpected arguments: `**rope_kwargs` should be unset in `_compute_yarn_parameters`, got r8r9r:rMrJmscalemscale_all_dimr rcNUS::agSU-[R"U5-S-$)Nrr9g?)mathlog)scalerSs r get_mscale,_compute_yarn_parameters..get_mscale s( A:V|dhhuo-33r# beta_fast beta_slowcU[R"X0S-[R-- 5-S[R"U5-- $)zPInverse dimension formula to find the dimension based on the number of rotationsr;)rVrWpi) num_rotationsr7r6rs r find_correction_dim5_compute_yarn_parameters..find_correction_dims@dhh6!:Kdgg:UVWW\]`d`h`him`n\noor#c>[R"T"XX455n[R"T"XX455n[US5[ XbS- 54$)z.Find dimension range bounds based on rotationsrr)rVfloorceilrmin)low_rothigh_rotr7r6rlowhighras r find_correction_range7_compute_yarn_parameters..find_correction_range#sIjj,W4YZyy,XDZ[3{CAg...r#cX:XaUS- n[R"U[RS9U- X- - n[R"USS5nU$)NgMbP?r<rr)rrFfloat32clamp)rfrr7 linear_func ramp_funcs r linear_ramp_factor4_compute_yarn_parameters..linear_ramp_factor)sH : 5LC||Cu}}=C R KK Q2 r#r;r>)r)r?r@rArr8rBrCrDrErNgetrrHrrFr)rrrrIr6r8r:r7rMrJrSrTr rYr[r]rkrr pos_freqsinv_freq_extrapolationinv_freq_interpolationrirjinv_freq_extrapolation_factorrras @r _compute_yarn_parametersrysV( ;!ghsgt u     DN *V-@-@@+1+>+>?a+b(//2RR+1+I+I(4  n$Z%?*VBd%de )&1 ##'' 4:I##'' 49Ip/ aa03363UX[[\I 9_ FY$67%iCGghIC%&(:3cQh(O(R(RZ`hmhshs(R(t$t!!&C"CD #@ @ A  % %%r#c [U5S:a[SU35eURn[US5(a UROSn[ USUR UR-5n[Xe-5nURSnURSn URRS5n URRS 5n [US 5(a&URn URUR- n O URn U cMU S::aSn OD[R"S [R"U 5[R"U 5- -5n U(a*X,:a%[ R""U[ R$US 9n O$[ R""U [ R$US 9n [ R&"SUS [ R(US 9R+5U- nSXU--- nX4$)a Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rz]Unexpected arguments: `**rope_kwargs` should be unset in `_compute_longrope_parameters`, got r8r9r: long_factor short_factorrMrJr r)r=rr;)r?r@rArr8rBrCrDrErNrtr rrVsqrtrWrtensorrnrFrGrH)rrrrIr6r8r:r7r{r|rMrJr ext_factorsinv_freq_shapers r _compute_longrope_parametersrBs* ;! km     D))*LMO&8':$''kH$G[[!;X=NPXYN$.@EUEghM]*n= 1, got rNrtrrr isinstancerHrr)rrrNr-rrrMs r (_validate_linear_scaling_rope_parametersrs&&L  l.>.>vt.LMI (+M ))+,M=Z ( #F ~Z66&3,QRXQYZ[;Gr#c.URnURSURSS55nSS1nS1n[UR55n[ X6XEUS9 USnUb[ U[ 5(aUS:a[RSU35 gg)Nr-rrMr rr9rr)rrrNr-rrrrMs r )_validate_dynamic_scaling_rope_parametersrs&&L  l.>.>vt.LMI (+M78M ))+,M=]hi ( #F ~Z66&3,QRXQYZ[;Gr#c*URnURSURSS55nSS1n1Skn[UR55n[ X6XEUS9 USnUb[ U[ 5(aUS:a[RSU35 URS5nUb3[ U[ 5(aUS :a[RS U35 URS 5n U b-[ U [ 5(d[RS U 35 URS 5n U b-[ U [ 5(d[RSU 35 U =(d SU =(d S:a[RSU SU S35 gg)Nr-rrM>rSr[r]rTrJr rr9rrJrL`rope_scaling`'s attention_factor field must be a float greater than 0, got r[z6`rope_scaling`'s beta_fast field must be a float, got r]z6`rope_scaling`'s beta_slow field must be a float, got r\rzO`rope_scaling`'s beta_fast field must be greater than beta_slow, got beta_fast=z( (defaults to 32 if None) and beta_slow=z (defaults to 1 if None)r) rrrNr-rrrrMrJr[r]s r _validate_yarn_parametersrs&&L  l.>.>vt.LMI (+MM ))+,M=]hi ( #F ~Z66&3,QRXQYZ[#''(:;#Z8H%-P-PTdghThZ[kZl m   -IZ 5%A%AOPY{[\  -IZ 5%A%AOPY{[\RIN+]^g]hi66?[@X Z ,r#c*URnURSURSS55n1Skn1Skn[UR55n[ X6XEUS9 [ US5(a UR OSn[USURUR-5n[X-5n URS 5n [U [5(d/[S U 55(a[RS U 35 [!U 5U S -:Xd'[RS U S -S[!U 535 URS5n [U [5(d/[SU 55(a[RSU 35 [!U 5U S -:Xd'[RSU S -S[!U 535 [ US5(a[R#S5 gURS5n U c[RS5 O3[U [$5(aU S:a[RSU 35 URS5n U b5[U [$5(aU S:a[RSU 35 ggg)Nr-r>r-r{r|>rMrJr rr8r9r:r|c3N# UHn[U[[45v M g7fNrrErH.0r.s r 0_validate_longrope_parameters..,s!1dWcRS*Qe 2M2MWc#%zC`rope_scaling`'s short_factor field must be a list of numbers, got r;z5`rope_scaling`'s short_factor field must have length z, got r{c3N# UHn[U[[45v M g7frrrs r rr2s!0bVaQRAU|1L1LVarzB`rope_scaling`'s long_factor field must be a list of numbers, got z4`rope_scaling`'s long_factor field must have length r aYThis model has set a `original_max_position_embeddings` field, to be used together with `max_position_embeddings` to determine a scaling factor. Please set the `factor` field of `rope_scaling`with this ratio instead -- we recommend the use of this field over `original_max_position_embeddings`, as it is compatible with most model architectures.rMz1Missing required keys in `rope_scaling`: 'factor'rrJgr)rNrtrrrrr8rBrCrDrErlistallrrr? warning_oncerH)rrrNr-rrrr8r:r7r|r{rMrJs r _validate_longrope_parametersrsZ&&L  l.>.>vt.LMI@MVM ))+,M=]hi NNN OFE**fsl NNUV\U]^ _'++,>?  '.66:JS:Pbcsbtu;Q (r#c URnURSURSS55n1Skn[UR55n[ X5XAS9 USnUb[ U[ 5(aUS:a[RSU35 USnUS nUb[ U[ 5(d[RS U35 Ub[ U[ 5(d[RS U35 X::a[RS US U35 USn U b[ U [5(d[RSU 35 XR:a&[RSU SUR35 gg)Nr-r>rMr-rrr rrMr9rrrz<`rope_scaling`'s low_freq_factor field must be a float, got z=`rope_scaling`'s high_freq_factor field must be a float, got zc`rope_scaling`'s high_freq_factor field must be greater than low_freq_factor, got high_freq_factor=z and low_freq_factor=r zP`rope_scaling`'s original_max_position_embeddings field must be an integer, got zg`rope_scaling`'s original_max_position_embeddings field must be less than max_position_embeddings, got z and max_position_embeddings=) rNrtrrrrrHrrrEr) rrrNr-rrrMrrr s r _validate_llama3_parametersrPs&&L  l.>.>vt.LMIvM ))+,M=Z ( #F ~Z66&3,QRXQYZ["#45O#$67j%&H&HUVeUfghz2BE'J'JVWgVhij* q 5o5F H (44V'W$'/zBbdg7h7h ^/0 2 (+I+II u/00MfNlNlMm o Jr#c[USS5nUcgURSURSS55n[RU5nUbU"XS9 g[R SUS35 g) zG Validate the RoPE config arguments, given a `PretrainedConfig` object rNNr-rrrzTMissing validation function mapping in `ROPE_VALIDATION_FUNCTIONS` for 'rope_type'='')rBrtROPE_VALIDATION_FUNCTIONSrr)rrrNr- validation_fns r rope_config_validationrsu6>48L  l.>.>vy.QRI-11)rs 1.   H %;~*.'+!'& % &'& ^ $'&c]'&  >5 ! '&V*.'+!&& % &&& ^ $&&c]&&  >5 ! &&T*.'+!0& % &0& ^ $0&c]0&  >5 ! 0&hPT]& ]&&4]&?G}]& >5 !]&BPT<& <&&4<&?G}<& >5 !<&@PT(, (,&4(,?G}(, >5 !(,^05. $,( $(!% llllC= l # l:[.>[XVY][ \5E \T\]`Ta \ \6F \U]^aUb \# &6# Xc]# L/*:/RU/d! (8! xPS}! L168 %-)  #3 (3- r#