o Zhv6@sUddlZddlZddlmZmZddlmZddlmZm Z m Z m Z m Z ddl Z ddlmZddl mZddlmZddlmZgZeeed<eeZGd d d ejZd eed efd dZdS)N) CollectionMapping)deepcopy)AnyCallableOptionaloverloadUnion)optim) ShardedTensor)FullyShardedDataParallel__all__c @sTeZdZdZ  d"deeeeje ffde j de e eeefde ejddf dd Zd d Zdeeeffd d Zed#d$ddZedegefdefddZd%de egefde efddZedeejeffddZdeeefddfddZdeeefddfddZd&ddZdeeeffddZdeeeffd d!ZdS)'_NamedOptimizera ``_NamedOptimizer`` takes a dict of parameters and exposes ``state_dict`` by parameter key. We replace the original key (number) in an optim to the fully qualified name (FQN) string. User can initialize the optim as they initialize a PyTorch optim, the only difference is that they also need to pass in the FQN of each parameters. Args: named_parameters (Mapping[str, Union[torch.Tensor, ShardedTensor]]): Mapping from FQN to parameter. optimizer_class (optim.Optimizer): The class of optimizer to instantiate. param_groups (Collection[Mapping[str, Any]]): `param_groups` to pass to optimizer if specified. The key of the inner map needs to be FQNs. Default: None module (nn.Module): the module whose parameters to updated by the optimizer. args: arguments to pass to the optimizer constructor. kwargs: arguments to pass to the optimizer constructor. Example:: >>> # xdoctest: +SKIP("distributed") >>> from torch import optim >>> from torch.distributed.optim import _NamedOptimizer >>> >>> # Define the named optimizer. >>> m = Model(...) >>> named_optim = _NamedOptimizer(m.named_parameters(), optim.SGD) >>> # Forward pass + backward pass. >>> named_optim.step() >>> ... >>> # Call state_dict for the named optimizer returns a FQN state_dict. >>> named_optim.state_dict() Warning: This API is still in development and subject to change. TODO: Add tutorial for _NamedOptimizer. TODO: Add documentation in the docstring for the public attributes like self.param_groups and self.named_parameters. Nnamed_parametersoptimizer_class param_groupsmodulereturnc Ostjd||_|t||_|dur|jn|}||g|Ri||_||_ |dur9t |j |_ n3t ddd|jD}g} |D]} | dD]} | |vr`td| d| || qRqL| |_ |jj|_dS)Nz'torch.distributed.optim._NamedOptimizerzvSince we pass in param_groups, we will use param_groups to initialize the optimizer, not all parameters of the module.cSi|]\}}||qSr.0keyparamrrV/var/www/auris/lib/python3.10/site-packages/torch/distributed/optim/named_optimizer.py ]z,_NamedOptimizer.__init__..paramszExpect param name z% found in param group but is missing.)torchZ_CZ_log_api_usage_oncer_param_groups_checkdictrvalues _optimizerrlistkeysordered_param_keyswarningswarnitems ValueErrorappend) selfrrrrargskwargsZparams_for_optimizer param_to_keyr%grouprrrr__init__@s>    z_NamedOptimizer.__init__cCs|jdurE|jD]>}t|tsJdd|vsJd|d}t|tjr(|g}t|}|D]}t|tjs?tdt|q.||d<qdSdS)Nparam group must be a dictrz#param group must contain key paramsz>optimizer can only optimize Tensors, but one of the params is )r isinstancer rTensorr# TypeErrortypename)r+ param_grouprrrrrrjs&     z#_NamedOptimizer._param_groups_checkc sj}|d}fdd|dD}g}|D]+}fdd|dD}dt|i}|D]\}} |dkr?t| ||<q1||q||dS) z Return the ``state_dict`` of the optimizer. Instead of using number to index parameters, we will use module fully qualified name (FQN) as the key. rcsi|] \}}j||qSrr%)rZst_key state_valr+rrrs z._NamedOptimizer.state_dict..statecsg|]}j|qSrr7)rrr9rr rz._NamedOptimizer.state_dict..r)r:r)r" state_dictr(sortedrr*_post_state_dict) r+r<rZ ret_stateZ ret_groupsr/ param_keysZ ret_groupkvrr9rr<{s      z_NamedOptimizer.state_dict.closurecCdSNrr+rBrrrstepz_NamedOptimizer.stepcCrCrDrrErrrrFrGcCs|jj|dS)z Perform a single optimization step. This will call :meth:`torch.optim.Optimizer.step` on the wrapped optimizer. rB)r"rFrErrrrFscCs|jjSrD)r"r:r9rrrr:sz_NamedOptimizer.stater<c Cs|j}||}|d}|d}t|dkrtdt|jD]\}}||vr,q!t||t||krMtdt||d|dt||||D]}\}}|||vrhtd|d|d|||} t |t rt | t szJt| } t| } | | krtd | d | d |d |t | | D] \} } | j | j qqSt |tjrt | tjsJ|| qSt| |||<qSq!|d }|d }i}|D]}t|d}||t|<qi}|D]}g}|dD] }||j|q||t|<q|D]N\}}||vrq||}t|t|kr9tdt|d|d t|d|D] }||vrMtd|d|d|dkrZt||||<q;q|j|dS)a Define the default behavior to load a state_dict for ``_NamedOptimizer``. Sample Code ``` my_model = MyModule() optimizer = _NamedOptimizer(my_model.named_parameters(), Adagrad) ... optim_state_dict = optimizer.state_dict() ... ... optimizer.load_state_dict(optim_state_dict) ... ``` Args: state_dict (Dict[str, Any]) : A ``state_dict`` to load into the optimizer. Note that this state dict update is performed in place. .. note:: PyTorch is using lazy init to initialize the optim states. So it is possible that there is no optim state when user call ``load_state_dict`` and for ``_NamedOptimizer`` we make it stricter that users can only call ``load_state_dict`` after the state is initialized. By doing this, we can validate the optim ``state_dict`` to be loaded. r:rzJExpects the optim to be initialized before load but found not initialized.zExpects equal length as z for parameter z but found: zExpects state z but not found.z"Expects equal number of shards as z but found z for /rrz"Expects equal param_group size as z for group .zExpects group key z to be in group z in `state_dict` but is missing.N)r"r<_pre_load_state_dictlenr) enumerater%r$r(r2r Z local_shardszipZtensordetachZcopy_rr3rr#_gen_param_group_keyr*load_state_dict)r+r<Znew_state_dictr: new_stateidxZ param_keyZ state_keyr8Z src_state_valZ num_shardsZnum_new_shardsZshardZ src_shardZsrc_param_groupsZnew_param_groupsZ src_group_mapr/r?Z new_group_mapZ new_groupZ group_keyZ src_groupr@rrrrQs    $             z_NamedOptimizer.load_state_dictr6cCst|ts Jd|d}t|tjr|g|d<nt||d<dd|jD}|dD]}||vr7td|j ||q-|j ||j j |_ dS)z Add a param group to the :class:`_NamedOptimizer` s `param_groups`. Warning: This API is still in development and subject to change. r1rcSrrrrrrrrrz3_NamedOptimizer.add_param_group..z%some parameters are not in the moduleN) r2r rr3r#rr(r)r%r*r"add_param_groupr)r+r6rr.rrrrrTs     z_NamedOptimizer.add_param_groupcCs>|jD]}|jrt|}tj||_q|jdddS)z Run a dummy optimizer step, which allows to initialize optimizer state because we do lazy init for most optimizers. This allows doing in-place loading of optimizer state from a checkpoint. NrH) rr!Z requires_gradrZ zeros_likeZautogradVariableZgradrF)r+rtrrr init_state(s  z_NamedOptimizer.init_statecCs&t|jtrtj|j|j|ddS|S)NT)Zis_named_optimizer)r2rFSDPZoptim_state_dict_to_loadr"r+r<rrrrK5s  z$_NamedOptimizer._pre_load_state_dictcCs"t|jtrt|j|j||SrD)r2rrXZoptim_state_dictr"rYrrrr>>s z _NamedOptimizer._post_state_dict)NN).)rBNrNrD)rN) __name__ __module__ __qualname____doc__rstrr rr3r r OptimizerrrrnnModuler0rr r<rrFrfloatpropertyr:rQrTrWrKr>rrrrrs:/ *$ h   rr?rcCsdt|S)zGConcatenate all param keys as a unique indentifier for one param group.rI)joinr=)r?rrrrPFsrP) loggingr&collections.abcrrcopyrtypingrrrrr rZtorch.nnr`r Z'torch.distributed._shard.sharded_tensorr Ztorch.distributed.fsdpr rXr r#r^__annotations__ getLoggerrZloggerr_rrPrrrrs       4