[Thv6%SSKrSSKrSSKJrJr SSKJr SSKJrJ r J r J r J r SSK r SSKJr SSK Jr SSKJr SSKJr /r\\\S'\R4"\5r"S S \R:5rS \\S \4S jrg)N) CollectionMapping)deepcopy)AnyCallableOptionaloverloadUnion)optim) ShardedTensor)FullyShardedDataParallel__all__c \rSrSrSrSS\\\\R\ 44S\ RS\ \\\\4S\ \R"SS4 S jjrS rS\\\44S jr\SSS jj5r\S \/\4S\4Sj5rSS \ \/\4S\ \4Sjjr\S\\R\44Sj5rS\\\4SS4SjrS\\\4SS4SjrSSjrS\\\44SjrS\\\44Sjr Sr!g)_NamedOptimizera\ ``_NamedOptimizer`` takes a dict of parameters and exposes ``state_dict`` by parameter key. We replace the original key (number) in an optim to the fully qualified name (FQN) string. User can initialize the optim as they initialize a PyTorch optim, the only difference is that they also need to pass in the FQN of each parameters. Args: named_parameters (Mapping[str, Union[torch.Tensor, ShardedTensor]]): Mapping from FQN to parameter. optimizer_class (optim.Optimizer): The class of optimizer to instantiate. param_groups (Collection[Mapping[str, Any]]): `param_groups` to pass to optimizer if specified. The key of the inner map needs to be FQNs. Default: None module (nn.Module): the module whose parameters to updated by the optimizer. args: arguments to pass to the optimizer constructor. kwargs: arguments to pass to the optimizer constructor. Example:: >>> # xdoctest: +SKIP("distributed") >>> from torch import optim >>> from torch.distributed.optim import _NamedOptimizer >>> >>> # Define the named optimizer. >>> m = Model(...) >>> named_optim = _NamedOptimizer(m.named_parameters(), optim.SGD) >>> # Forward pass + backward pass. >>> named_optim.step() >>> ... >>> # Call state_dict for the named optimizer returns a FQN state_dict. >>> named_optim.state_dict() Warning: This API is still in development and subject to change. TODO: Add tutorial for _NamedOptimizer. TODO: Add documentation in the docstring for the public attributes like self.param_groups and self.named_parameters. Nnamed_parametersoptimizer_class param_groupsmodulereturnc[RRS5 X0lUR 5 [ U5UlUcUR R5OUnU"U/UQ70UD6UlX@l Uc)[UR R55Ul O[R"S5 UR R5VV s0sHupX_M n nn /n UH6n U SH*n X;a[!SU S35eU R#X5 M, M8 Xl URRUlgs sn nf)Nz'torch.distributed.optim._NamedOptimizerzvSince we pass in param_groups, we will use param_groups to initialize the optimizer, not all parameters of the module.paramszExpect param name z% found in param group but is missing.)torch_C_log_api_usage_oncer_param_groups_checkdictrvalues _optimizerrlistkeysordered_param_keyswarningswarnitems ValueErrorappend) selfrrrrargskwargsparams_for_optimizerkeyparam param_to_keyr"groups _/var/www/auris/envauris/lib/python3.13/site-packages/torch/distributed/optim/named_optimizer.py__init___NamedOptimizer.__init__@sO $$%NO;G   " $%5 6.:.BD ! ! ( ( *  *       &*4+@+@+E+E+G&HD # MMN :>9N9N9T9T9VW9V:3EJ9VLW!# %"8_E0(07\]'--l.AB -&'9 # OO88XsE cURbURHn[U[5(dS5eSU;dS5eUSn[U[R5(aU/n[ U5nUHDn[U[R5(aM$[ S[R"U5-5e X!S'M gg)Nparam group must be a dictrz#param group must contain key paramsz>optimizer can only optimize Tensors, but one of the params is )r isinstancerrTensorr TypeErrortypename)r( param_grouprr-s r0r#_NamedOptimizer._param_groups_checkjs    (#00 !+t44R6RR4;.U0UU.$X.fell33$XFf#E%eU\\::'8:?..:OP$ )/H% 1 )cURR5nUSnUSR5VVs0sHup4URUU_M nnn/nUHpnUSVs/sHoRUPM n nS[ U 50n UR5HupU S:wdM [ U 5X'M UR U 5 Mr URXVS.5$s snnfs snf)z Return the ``state_dict`` of the optimizer. Instead of using number to index parameters, we will use module fully qualified name (FQN) as the key. rstater)r=r)r state_dictr%r"sortedrr'_post_state_dict) r(r>rst_key state_val ret_state ret_groupsr/r- param_keys ret_groupkvs r0r>_NamedOptimizer.state_dict{s__//1 !.1 &0%8%>%>%@ %@!  # #F +Y 6%@   !EFKHoVoU11%8oJV!6*#56I =#+A;IL&   i ( "$$y%UVV Ws C #C&closurecgNr(rJs r0step_NamedOptimizer.steps14r;cgrLrMrNs r0rOrPs;>r;c4URRUS9$)zr Perform a single optimization step. This will call :meth:`torch.optim.Optimizer.step` on the wrapped optimizer. rJ)rrOrNs r0rOrPs##G#44r;c.URR$rL)rr=)r(s r0r=_NamedOptimizer.states$$$r;r>c URR5nURU5nUSnUSn[U5S:Xa [ S5e[ UR 5GHupVXcR5;aM[X65[XE5:wa*[ S[XE5SUS[X6535eXER5GHzupxXsU;a[ SUSUS35eX6Un [U[5(a[U [5(de[UR55n [U R55n X:wa[ S U S U S US U35e[UR5U R55H8upU RR5RU R5 M: GM[U[ R"5(aC[U [ R"5(deUR5RU 5 GMj[%U 5XEU'GM} GM US nUS n0nUHn['US5nUU[)U5'M! 0nUH=n/nUSH!nUR+UR U5 M# UU[)U5'M? UR5HunnUU;aMUUn[U5[U5:wa'[ S[U5SUS [U5S35eUH4nUU;a[ SUSUS35eUS:wdM#[%UU5UU'M6 M URR-U5 g)a@ Define the default behavior to load a state_dict for ``_NamedOptimizer``. Sample Code ``` my_model = MyModule() optimizer = _NamedOptimizer(my_model.named_parameters(), Adagrad) ... optim_state_dict = optimizer.state_dict() ... ... optimizer.load_state_dict(optim_state_dict) ... ``` Args: state_dict (Dict[str, Any]) : A ``state_dict`` to load into the optimizer. Note that this state dict update is performed in place. .. note:: PyTorch is using lazy init to initialize the optim states. So it is possible that there is no optim state when user call ``load_state_dict`` and for ``_NamedOptimizer`` we make it stricter that users can only call ``load_state_dict`` after the state is initialized. By doing this, we can validate the optim ``state_dict`` to be loaded. r=rzJExpects the optim to be initialized before load but found not initialized.zExpects equal length as z for parameter z but found: zExpects state z but not found.z"Expects equal number of shards as z but found z for /rrz"Expects equal param_group size as z for group .zExpects group key z to be in group z in `state_dict` but is missing.N)rr>_pre_load_state_dictlenr& enumerater"r!r%r5r local_shardsziptensordetachcopy_rr6rr _gen_param_group_keyr'load_state_dict)r(r>new_state_dictr= new_stateidx param_key state_keyrB src_state_val num_shardsnum_new_shardsshard src_shardsrc_param_groupsnew_param_groups src_group_mapr/rE new_group_map new_group group_key src_grouprGs r0rb_NamedOptimizer.load_state_dicts6335..z: 7#"7+ y>Q \ ((?(?@NC ,5#$IN(;; .s9>/B.C?S\R]]ijmnsn~jjAB)2(<(<(>$ )$44$( ?9+_]!& 0 ; i77%m]CCCC!$Y%;%;%=!>J%()C)C)E%FN!3(@@PP[\f[gglmvlwwxzCyDE-0!..0-2L2L2N-( ++-33I4D4DE- 5<<88%mU\\BBBB$$&,,];080GIN9-/)?AF&n5).9 %EeHo.J>CM.z: ;& )IJ&x0 !!$"9"9)"DE1>GM.z: ; * %2$7$7$9 Iy -%i0I9~Y/ 8Y8H T]S^^ijmnwjxiyyz{I%$,QC/? {Jjk=#+IaL#9IaL %:$ ''7r;r9c[U[5(dS5eUSn[U[R5(aU/US'O[ U5US'UR R 5VVs0sHup4XC_M nnnUSH0nXE;a [S5eURRXT5 M2 URRU5 URRUl gs snnf)z Add a param group to the :class:`_NamedOptimizer` s `param_groups`. Warning: This API is still in development and subject to change. r4rz%some parameters are not in the moduleN) r5rrr6r rr%r&r"r'radd_param_groupr)r(r9rr,r-r.s r0rv_NamedOptimizer.add_param_groups +t,,J.JJ,X& fell + +%+HK !$(LK !595J5J5P5P5RS5Rzs 5R S *E( !HII  # # * *<+> ?+ '' 4 OO88Ts3C4cURR5HPnUR(dM[R"U5n[R R U5UlMR URSS9 g)z Run a dummy optimizer step, which allows to initialize optimizer state because we do lazy init for most optimizers. This allows doing in-place loading of optimizer state from a checkpoint. NrS) rr requires_gradr zeros_likeautogradVariablegradrO)r(r-ts r0 init_state_NamedOptimizer.init_state(s_ **113E"""$$U+"^^44Q7 4 $ r;c[UR[5(a+[R"URURUSS9$U$)NT)is_named_optimizer)r5rFSDPoptim_state_dict_to_loadrr(r>s r0rY$_NamedOptimizer._pre_load_state_dict5sA dkk4 ( (00 T__jT r;c[UR[5(a,[R"URURU5 U$rL)r5rroptim_state_dictrrs r0r@ _NamedOptimizer._post_state_dict>s5 dkk4 ( (  ! !$++t Kr;)rrrr"r)NN).)rJNrNrL)rN)"__name__ __module__ __qualname____firstlineno____doc__rstrr rr6r r OptimizerrrrnnModuler1rrr>r rOrfloatpropertyr=rbrvrrYr@__static_attributes__rMr;r0rrs)^AE&* (9!#uU\\=-H'I"IJ(9(9z'#s(*;<= (9 # (9 (9T/"WDcNW444 >HRY/>E>>5HXb%i%895Xe_5%wu||S01%%f8'#s(*;f8f8P9738+<990  $sCx.d38nr;rrErc6SR[U55$)zGConcatenate all param keys as a unique indentifier for one param group.rW)joinr?)rEs r0raraFs 88F:& ''r;) loggingr#collections.abcrrcopyrtypingrrrr r rtorch.nnrr 'torch.distributed._shard.sharded_tensorr torch.distributed.fsdpr rrr r__annotations__ getLoggerrloggerrrrarMr;r0rsp/;; ACc   8 $oeoood (T#Y(3(r;