eThrASrSSKrSSKJrJrJr SSKrSSKJ r SSK Jr \"\R R"S5(a!\R R"R$rO\R R"r"SS \R&5rSS \S \S \S \S\S\S\S\\S\\S\S\S\\\4Sjjr"SS\ 5r"SS5rg!\ \ 4a SSK J r Nf=f)z?Functions and classes related to optimization (weight updates).N)CallableOptionalUnion)Adam)keraslearning_rate_schedulec ^^\rSrSrSrS S\S\S\S\S\\ 4 U4Sjjjr S r S r S r U=r$) WarmUp&a Applies a warmup schedule on a given learning rate decay schedule. Args: initial_learning_rate (`float`): The initial learning rate for the schedule after the warmup (so this will be the learning rate at the end of the warmup). decay_schedule_fn (`Callable`): The schedule function to apply after the warmup for the rest of training. warmup_steps (`int`): The number of steps for the warmup part of training. power (`float`, *optional*, defaults to 1.0): The power to use for the polynomial warmup (defaults is a linear warmup). name (`str`, *optional*): Optional name prefix for the returned tensors during the schedule. initial_learning_ratedecay_schedule_fn warmup_stepspowernamec^>[TU]5 XlX0lX@lX lXPlgN)super__init__r rrrr)selfr rrrr __class__s T/var/www/auris/envauris/lib/python3.13/site-packages/transformers/optimization_tf.pyrWarmUp.__init__8s, %:"( !2 c^^^[R"TR=(d S5n[R"T[R5n[R"TR [R5nX4- nTR [RRUTR5-m[R"X4:U4SjUU4SjUS9sSSS5 $!,(df  g=f)Nr c>T$r)warmup_learning_ratesr!WarmUp.__call__..Qs,rc@>TRTTR- 5$r)rr)rstepsrrr Rs..td6G6G/GHrr) tf name_scopercastfloat32rr mathpowrcond)rr"rglobal_step_floatwarmup_steps_floatwarmup_percent_doners`` @r__call__WarmUp.__call__Gs ]]4990 1T!#bjj 9 !#):):BJJ!G "3"H #'#=#= L_aeakak@l#l 77!6,H 2 1 1s B1C(( C6cvURURURURURS.$)Nr rrrrr1rs r get_configWarmUp.get_configVs5%)%?%?!%!7!7 --ZZII   r)rr rrr)?N)__name__ __module__ __qualname____firstlineno____doc__floatrintrstrrr.r3__static_attributes__ __classcell__rs@rr r &s]," $ $     sm     rr init_lrnum_train_stepsnum_warmup_steps min_lr_ratio adam_beta1 adam_beta2 adam_epsilon adam_clipnormadam_global_clipnormweight_decay_raterinclude_in_weight_decayc [RUX- X-U S9n U(a [UU US9n U S:a[U U UUUUU/SQU S9 n X4$[R R U UUUUUS9n X4$)a9 Creates an optimizer with a learning rate schedule using a warmup phase followed by a linear decay. Args: init_lr (`float`): The desired learning rate at the end of the warmup phase. num_train_steps (`int`): The total number of training steps. num_warmup_steps (`int`): The number of warmup steps. min_lr_ratio (`float`, *optional*, defaults to 0): The final learning rate at the end of the linear decay will be `init_lr * min_lr_ratio`. adam_beta1 (`float`, *optional*, defaults to 0.9): The beta1 to use in Adam. adam_beta2 (`float`, *optional*, defaults to 0.999): The beta2 to use in Adam. adam_epsilon (`float`, *optional*, defaults to 1e-8): The epsilon to use in Adam. adam_clipnorm (`float`, *optional*, defaults to `None`): If not `None`, clip the gradient norm for each weight tensor to this value. adam_global_clipnorm (`float`, *optional*, defaults to `None`) If not `None`, clip gradient norm to this value. When using this argument, the norm is computed over all weight tensors, as if they were concatenated into a single vector. weight_decay_rate (`float`, *optional*, defaults to 0): The weight decay to use. power (`float`, *optional*, defaults to 1.0): The power to use for PolynomialDecay. include_in_weight_decay (`List[str]`, *optional*): List of the parameter names (or re patterns) to apply weight decay to. If none is passed, weight decay is applied to all parameters except bias and layer norm parameters. )r decay_stepsend_learning_rater)r rr) LayerNorm layer_normbias) learning_raterJbeta_1beta_2epsilonclipnormglobal_clipnormexclude_from_weight_decayrK)rSrTrUrVrWrX) schedulesPolynomialDecayr AdamWeightDecayr optimizersr)rArBrCrDrErFrGrHrIrJrrK lr_schedule optimizers rcreate_optimizerr``s\++%#6!0 ,K "))) 3#%/ "0&I$;  ,  !!$$))% "0 *   !!rc^\rSrSrSrSS\\\R4S\S\S\S\ S\S \ \ \ S \ \ \ S \ 4U4S jjjr \U4S j5rU4SjrSrSU4SjjrSrSU4SjjrSU4SjjrU4SjrSrSrU=r$)r\a Adam enables L2 weight decay and clip_by_global_norm on gradients. Just adding the square of the weights to the loss function is *not* the correct way of using L2 regularization/weight decay with Adam, since that will interact with the m and v parameters in strange ways as shown in [Decoupled Weight Decay Regularization](https://arxiv.org/abs/1711.05101). Instead we want to decay the weights in a manner that doesn't interact with the m/v parameters. This is equivalent to adding the square of the weights to the loss with plain (non-momentum) SGD. Args: learning_rate (`Union[float, LearningRateSchedule]`, *optional*, defaults to 0.001): The learning rate to use or a schedule. beta_1 (`float`, *optional*, defaults to 0.9): The beta1 parameter in Adam, which is the exponential decay rate for the 1st momentum estimates. beta_2 (`float`, *optional*, defaults to 0.999): The beta2 parameter in Adam, which is the exponential decay rate for the 2nd momentum estimates. epsilon (`float`, *optional*, defaults to 1e-07): The epsilon parameter in Adam, which is a small constant for numerical stability. amsgrad (`bool`, *optional*, defaults to `False`): Whether to apply AMSGrad variant of this algorithm or not, see [On the Convergence of Adam and Beyond](https://arxiv.org/abs/1904.09237). weight_decay_rate (`float`, *optional*, defaults to 0.0): The weight decay to apply. include_in_weight_decay (`List[str]`, *optional*): List of the parameter names (or re patterns) to apply weight decay to. If none is passed, weight decay is applied to all parameters by default (unless they are in `exclude_from_weight_decay`). exclude_from_weight_decay (`List[str]`, *optional*): List of the parameter names (or re patterns) to exclude from applying weight decay to. If a `include_in_weight_decay` is passed, the names in it will supersede this list. name (`str`, *optional*, defaults to `"AdamWeightDecay"`): Optional name for the operations created when applying gradients. kwargs (`Dict[str, Any]`, *optional*): Keyword arguments. Allowed to be {`clipnorm`, `clipvalue`, `lr`, `decay`}. `clipnorm` is clip gradients by norm; `clipvalue` is clip gradients by value, `decay` is included for backward compatibility to allow time inverse decay of learning rate. `lr` is included for backward compatibility, recommended to use `learning_rate` instead. rSrTrUrVamsgradrJrKrYrc  P>[T U]"XX4XY40U D6 X`lXplXlgr)rrrJ_include_in_weight_decay_exclude_from_weight_decay) rrSrTrUrVrcrJrKrYrkwargsrs rrAdamWeightDecay.__init__s. YRXY!2(?%*C'rc.>S[0n[TU] XS9$)z?Creates an optimizer from its config with WarmUp custom object.r )custom_objects)r r from_config)clsconfigrjrs rrkAdamWeightDecay.from_configs"#F+w"6"IIrcr>[TU]XU5 [R"URSS9X1U4S'g)Nadam_weight_decay_rater#rJ)r_prepare_localr$constantrJ)r var_device var_dtype apply_staters rrqAdamWeightDecay._prepare_locals; zkBDFKK  " ")AE +,-@ArcURUR5nU(aDURX!-X1RURR 4S-UR S9$[R"5$)NrJ) use_locking) _do_use_weight_decayr assign_subdevicedtype base_dtype _use_lockingr$no_op)rvarrSrudo_decays r_decay_weights_op!AdamWeightDecay._decay_weights_opsm,,SXX6 >>#k::syy?S?S2T&UVi&jj --" xxzrc b>[[U65upE[TU] "[XE54SU0UD6$)Nr)listziprapply_gradients)rgrads_and_varsrrggradstvarsrs rrAdamWeightDecay.apply_gradientss3C01 w&s5'8NtNvNNrcUcURU04$U=(d 0nURX45nUcURX5nXCX4'USSU04$)z1Retrieves the learning rate with the given state.lr_tru) _decayed_lr_tget_fallback_apply_state)rrsrtru coefficientss r_get_lrAdamWeightDecay._get_lrso  %%i0"4 4!'R " '>?  55jLL3?/ 0F#m[%AAArc>URURURRU5upEUR X$U5n[ R "U/5 [TU] "X40UD6sSSS5 $!,(df  g=fr) rr{r|r}rr$control_dependenciesr_resource_apply_dense)rgradrrurrgdecayrs rr%AdamWeightDecay._resource_apply_densese||CJJ 0D0DkR &&s+>  $ $eW -70EfE. - -s A99 Bc>URURURRU5upVUR X%U5n[ R "U/5 [TU] "XU40UD6sSSS5 $!,(df  g=fr) rr{r|r}rr$rr_resource_apply_sparse) rrrindicesrurrgrrs rr&AdamWeightDecay._resource_apply_sparsesg||CJJ 0D0DkR &&s+>  $ $eW -71$WOO. - -s A:: Bc^>[TU]5nURSUR05 U$)NrJ)rr3updaterJ)rrmrs rr3AdamWeightDecay.get_configs-#% *D,B,BCD rcURS:XagUR(a,URHn[R"X!5cM g UR(a,URHn[R"X!5cM g g)z0Whether to use L2 weight decay for `param_name`.rFT)rJreresearchrf)r param_namers rry$AdamWeightDecay._do_use_weight_decay$sp  ! !Q &  ( (2299Q+73  * *4499Q+7 5r)rfrerJ) gMbP??+?gHz>FrONNr\r)r6r7r8r9r:rr;rZLearningRateScheduleboolrrr=r classmethodrkrqrrrrrr3ryr>r?r@s@rr\r\s$PGL#&7;9=%DUI$B$BBCDD D  D  D!D"*$s)!4D$,DI#6DDD$JJ  O BF P  rr\cJ\rSrSrSrSr\S5r\S5rSr Sr Sr g ) GradientAccumulatori6aB Gradient accumulation utility. When used with a distribution strategy, the accumulator should be called in a replica context. Gradients will be accumulated locally on each replica and without synchronization. Users should then call `.gradients`, scale the gradients if required, and pass the result to `apply_gradients`. c /UlSUlg)zInitializes the accumulator.N) _gradients _accum_stepsr2s rrGradientAccumulator.__init__As rc*URcm[R"[R"S[RS9S[R R [RRS9UlURR5$)zNumber of accumulated steps.r)r|F trainablesynchronization aggregation) rr$Variablerrint64VariableSynchronizationON_READVariableAggregationONLY_FIRST_REPLICAvaluer2s rr"GradientAccumulator.stepFsk    $ "  ARXX. " : : B B22EE !D   &&((rcUR(d [S5eURVs/sHobUR5OUPM sn$s snf)z1The accumulated gradients on the current replica.zBThe accumulator should be called first to initialize the gradients)r ValueErrorrrgradients r gradientsGradientAccumulator.gradientsSsFab bW[WfWfgWf8$8 hFWfgggsA c UR(dURnURRUVs/sHcnUb[[R"[R "U5S[R R[RRS9OUPMe sn5 [U5[UR5:wa-[S[UR5S[U535e[URU5H upCUcM UcMURU5 M" URRS5 gs snf)z/Accumulates `gradients` on the current replica.NFrz Expected z gradients, but got r)rr"extendr$r zeros_likerrrrlenrr assign_addr)rr_raccum_gradients rr.GradientAccumulator.__call__Zs A OO " "%. %. + KK h/"'(*(B(B(J(J$&$:$:$M$M ""%.   y>S1 1yT__)=(>>RSVW`SaRbcd d(+DOOY(G $N)h.B))(3)H $$Q'' sA*E cUR(dgURRS5 URH-nUcMUR[R"U55 M/ g)z8Resets the accumulated gradients on the current replica.Nr)rrassignr$rrs rresetGradientAccumulator.resettsI    #H# h 78(r)rrN) r6r7r8r9r:rpropertyr"rr.rr>rrrrr6s@!  ) )hh (49rr) rOrrg:0yE>NNrOr5N)r:rtypingrrr tensorflowr$tf_keras.optimizers.legacyr ImportErrorModuleNotFoundError"tensorflow.keras.optimizers.legacymodeling_tf_utilsrhasattrr]rZr rr r;r<rr=r`r\rrrrrsfF ,,8/% 5   % %'?@@  **AAI  **I7 Y + +7 |%),0"37Q" Q"Q"Q" Q"  Q"  Q"Q"E?Q"#5/Q"Q" Q"&d3i0Q"h~d~DE9E9{ ()878sCC+*C+