a hO@sdZddlmZmZmZddlZddlmZddlmZm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZddgZGd ddeZd d ed ed e d ed e d e_eeeeeeeeeeeeeeeeeeeeeedddZeeeeeeeeeeeeeeeeeeeeeedddZe eddeeeeeeeeeeeeeeeeeeeeeeeedddZdS)z)Implementation for the RMSprop algorithm.)castOptionalUnionN)Tensor)_capturable_doc_default_to_fused_or_foreach_differentiable_doc_disable_dynamo_if_unsupported _foreach_doc!_get_capturable_supported_devices_get_scalar_dtype _maximize_doc _params_doc _to_scalar_use_grad_for_differentiable _view_as_real OptimizerParamsTRMSproprmspropc sfeZdZdeeeefeeeeeeeeeed fdd Z fd d Z d d Z e dddZ ZS)r{Gz?Gz?:0yE>rFN) paramslralphaeps weight_decaymomentumcentered capturableforeachmaximizedifferentiablec st|tr|dkrtdd|ks4td|d|ksJtd|d|ks`td|d|ksvtd|d|kstd|t|||||||| | | d } t|| dS) NrzTensor lr must be 1-elementgzInvalid learning rate: zInvalid epsilon value: zInvalid momentum value: zInvalid weight_decay value: zInvalid alpha value: ) rrrrr rr!r"r#r$) isinstancerZnumel ValueErrordictsuper__init__) selfrrrrrrr r!r"r#r$defaults __class__A/var/www/auris/lib/python3.9/site-packages/torch/optim/rmsprop.pyr)s2 zRMSprop.__init__cst||jD]}|dd|dd|dd|dd|dd|dd|d D]h}|j|g}t|dkrft|d sft |d }|drtj |t |j d ntj |t d |d <qfqdS) Nrrr Fr"r#r$r!rstepdtypedevicer2) r( __setstate__ param_groups setdefaultstategetlentorchZ is_tensorfloattensorr r3)r*r8grouppZp_stateZstep_valr,r.r/r5Hs$           zRMSprop.__setstate__c Cs<d}|dD](} | jdurq |t| O}|| | jjrFtd|| j|j| } t| dkr|drtjdt | j dntjdt d| d <tj | tj d | d <|d dkrtj | tj d | d <|drtj | tj d | d<|| d || d |d dkr || d |dr || dq |S)NFrz)RMSprop does not support sparse gradientsrr!r.r1r4r0)Z memory_format square_avgrZmomentum_bufferr grad_avg) gradr; is_complexappendZ is_sparse RuntimeErrorr8r:zerosr r3Z zeros_likeZpreserve_format) r*r>params_with_gradgrads square_avgsmomentum_buffer_list grad_avgs state_steps has_complexr?r8r.r.r/ _init_group]sB          zRMSprop._init_groupc Cs|d}|durBt|}Wdn1s80Y|jD]}g}g}g}g}g}g} |||||||| } t|||||| |d|d|d|d|d|d|d|d |d |d | d qH|S) zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrrr r"r#r$r!) rrrrrr r"r#r$r!rM)Z _cuda_graph_capture_health_checkr;Z enable_gradr6rNr) r*closureZlossr>rGrHrIrKrJrLrMr.r.r/r0sR $  z RMSprop.step) rrrrrFFNFF)N)__name__ __module__ __qualname__rrr<rboolrr)r5rNrr0 __classcell__r.r.r,r/rs6 ) 3aj Implements RMSprop algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \alpha \text{ (alpha)}, \: \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)},\: \mu \text{ (momentum)}, \: centered, \: \epsilon \text{ (epsilon)} \\ &\textbf{initialize} : v_0 \leftarrow 0 \text{ (square average)}, \: \textbf{b}_0 \leftarrow 0 \text{ (buffer)}, \: g^{ave}_0 \leftarrow 0 \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}if \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}v_t \leftarrow \alpha v_{t-1} + (1 - \alpha) g^2_t \hspace{8mm} \\ &\hspace{5mm} \tilde{v_t} \leftarrow v_t \\ &\hspace{5mm}if \: centered \\ &\hspace{10mm} g^{ave}_t \leftarrow g^{ave}_{t-1} \alpha + (1-\alpha) g_t \\ &\hspace{10mm} \tilde{v_t} \leftarrow \tilde{v_t} - \big(g^{ave}_{t} \big)^2 \\ &\hspace{5mm}if \: \mu > 0 \\ &\hspace{10mm} \textbf{b}_t\leftarrow \mu \textbf{b}_{t-1} + g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \\ &\hspace{10mm} \theta_t \leftarrow \theta_{t-1} - \gamma \textbf{b}_t \\ &\hspace{5mm} else \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \hspace{3mm} \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `lecture notes `_ by G. Hinton. and centered version `Generating Sequences With Recurrent Neural Networks `_. The implementation here takes the square root of the gradient average before adding epsilon (note that TensorFlow interchanges these two operations). The effective learning rate is thus :math:`\gamma/(\sqrt{v} + \epsilon)` where :math:`\gamma` is the scheduled learning rate and :math:`v` is the weighted moving average of the squared gradient. z Args: a0 lr (float, Tensor, optional): learning rate (default: 1e-2) alpha (float, optional): smoothing constant (default: 0.99) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) momentum (float, optional): momentum factor (default: 0) centered (bool, optional) : if ``True``, compute the centered RMSProp, the gradient is normalized by an estimation of its variance z z )rrHrIrKrJrLrrrrrr r#r$r!rMc Cstjst|}t|D]\}}||}tjsl|rlt}|jj |jj kr\|jj |vslJd|d||}| s||n| }||}|d7}| dkr|j || d}t |}|rt |}t |}t |}| |j||d|d| r2||}|r t |}||d||j||dd}n|}| rL| |}n ||}| dkr||}|rxt |}| | |||j|| dq|j||| dqdS)NIIf capturable=True, params and state_steps must be on supported devices: .rrrvalue)r;jit is_scriptingr enumeratecompiler is_compilingr r3typeaddrCZ view_as_realZmul_Zaddcmul_Zlerp_ZaddcmulZsqrt_sqrtZadd_Zaddcdiv_)rrHrIrKrJrLrrrrrr r#r$r!rMiparamr0capturable_supported_devicesrBr@Zis_complex_paramrAavgbufr.r.r/_single_tensor_rmsprop sR            rhc !st|dkrdS| rJdtjs\|r\ttfddt||Ds\Jddt|}t ||||||g}| D]n\\}}}}}}}t t t |}t t t |}t t t |}t t t |}|r,||g}| dkrt t t |}||| rt t t |}||t|g|R| rpsz(_multi_tensor_rmsprop..rUrVg?cpu)r3rWrrXrZ)r:r;r^r_r allziprrZ"_group_tensors_by_device_and_dtypevaluesrlistrrDrZ _foreach_negZis_cpuZ _foreach_add_r=Z _foreach_addZ _foreach_mul_Z_foreach_addcmul_Z_foreach_lerp_Z_foreach_addcmulZ_foreach_sqrt_Z _foreach_sqrtZ_foreach_addcdiv_r%Z _foreach_mulZ _foreach_div_)!rrHrIrKrJrLrrrrrr r#r$r!rMZgrouped_tensorsZgrouped_params_Zgrouped_grads_Zgrouped_square_avgs_Zgrouped_grad_avgs_Zgrouped_momentum_buffer_list_Zgrouped_state_steps__Zgrouped_paramsZ grouped_gradsZgrouped_square_avgsZgrouped_state_stepsZstate_and_gradsZgrouped_momentum_buffer_listZgrouped_grad_avgsrfZ momentum_lrr.rkr/_multi_tensor_rmspropUs                  rs)Zsingle_tensor_fnF)rrHrIrKrJrLr"r#r$r!rMrrrrrr c Cstjs$tdd|Ds$td|dur>t||dd\}}|rTtjrTtd|rhtjsht}nt }|||||||| | | ||||| || ddS) ztFunctional API that performs rmsprop algorithm computation. See :class:`~torch.optim.RMSProp` for details. css|]}t|tjVqdSri)r%r;r)rjtr.r.r/rlszrmsprop..zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsNF)Z use_fusedz6torch.jit.script not supported with foreach optimizers) rrrrrr r#r!r$rM) r;r^r_rnrErr[r\rsrh)rrHrIrKrJrLr"r#r$r!rMrrrrrr rrfuncr.r.r/rsB )NFFFF)__doc__typingrrrr;rZ optimizerrrr r r r r rrrrrrr__all__rrqr<rSrhrsrr.r.r.r/s @,,   B M