a h @sUddlZddlZddlZddlZddlmZmZddlmZm Z m Z ddl m Z ddl mZmZmZmZmZmZddlZddlmZddlmZddlmZddlmZmZmZm Z m!Z!m"Z"ddl#m$Z$dd l%m&Z&m'Z'm(Z)m*Z*m+Z+m,Z,m-Z-m.Z.dd l/m0Z0m1Z1dd l2m3Z3dd l4m5Z5dd l6m7Z8ddl9m:Z:gdZ;dZdZ?e@eAZBee3eejCeDeEeAfZFeeFeGeFeHeFeIeAdffZJeIeAeJfZKeGeKZLeIeAeeKeLffZMe@ZNe@eeOd<ejPddZQe GdddZRe GdddeRZSdfejTeAeAeUeUeBddd ZVGd!d"d"ZWdgd#d$ZXddd%ejTeHejYjZd&feUee@ejTeeReSd'd(d)Z[eIeAeJfeMeSdd*d+d,Z\eejTejYjZfeAed-d.d/Z]eIeAefeSeIeAefd0d1d2Z^e_ejTeSeIeAeJfd3d4d5Z`e_ejTeIeAeJfeSe5d6d7d8ZaejYjZdd9d:d;ZbeMeIeAeJfd<d=d>ZcejYjZeIeAeJfeSeMd?d@dAZde_ejTeHejYjZd&feSeMdBdCdDZeejTejYjZeMeSeMdEdFdGZfe_ejTeHejYjZd&feMeSddHdIdJZgddd%ejTee@ejTeeReIeAeJfdKdLdMZhddd%ejTeejYjZeejYjZfee@ejTeeReMdNdOdPZiddd%ejTeejYjZeejYjZfee@ejTeeReHeIeAeJfeMfdNdQdRZjejTeeIejTeIeAeJffeIeAeJffeIeAeJfdSdTdUZkddVejTeIeAeJfeeRe5dWdXdYZlddVejTeejYjZeejYjZfeMeeRddZd[d\ZmddVejTeejYjZeejYjZfeIeAeJfeMeeRe5d]d^d_ZneddVejTeeRdd`dadbZoeddVejTeHejYjZd&feeRddcdddeZpdS)hN) GeneratorIterable)asdict dataclassfield)chain)AnyCallablecast no_type_checkOptionalUnion) ShardedTensor)_broadcast_state_dict_distribute_state_dict_flatten_state_dict_gather_state_dict_offload_state_dict_to_cpu_unflatten_state_dict)_CHECKPOINT_PREFIX)FullOptimStateDictConfigFullStateDictConfigFullyShardedDataParallelOptimStateDictConfigShardedOptimStateDictConfigShardedStateDictConfigStateDictConfig StateDictType)._get_module_fsdp_state_if_fully_sharded_moduleFSDP_WRAPPED_MODULE)DTensor)_IncompatibleKeys)DistributedDataParallel) tree_map_only) FQNS_T PrimitiveType ValueType DictValueTypeListDictValueTypeOptimizerStateTypeStateDictOptionsget_model_state_dictget_optimizer_state_dictget_state_dictset_model_state_dictset_optimizer_state_dictset_state_dictZ _flat_param param_groupsparamsstater&_patched_state_dictccs:t}tzdVW|r6tn|r4t0dSN)gc isenableddisableenable) is_enabledr;U/var/www/auris/lib/python3.9/site-packages/torch/distributed/checkpoint/state_dict.py _gc_contextQs r=c@sreZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<dZ eed <dZ eed <d Z eed <d S)r*ap This dataclass specifies how get_state_dict/set_state_dict will work. - ``full_state_dict``: if this is set to True, all the tensors in the returned state_dict will be gathered. No ShardedTensor and DTensor will be in the returned state_dict. - ``cpu_offload``: offload all the tensors to cpu. To prevent CPU OOM, if ``full_state_dict`` is also true, then only the rank0 will get the state_dict and all other ranks will get empty state_dict. - ``ignore_frozen_params``: if the value is True, the returned state_dict won't contain any frozen parameters -- the ``requires_grad`` is False. The default value is False. - ``keep_submodule_prefixes`` (deprecated): when ``submodules`` is not None, this option indicates whether to keep the submodule prefixes from the state_dict keys. or example, if the submodule is ``module.pretrain`` and the full FQN of the parameter is ``pretrain.layer1.weight`` of the param. When this option is True, the parameter's key in the returned state_dict will be ``pretrain.layer1.weight``. If the options is False, the key will be ``layer1.weight``. Note that if ``keep_submodule_prefixes`` is False, there may be conflicted FQNs, hence there should be only one submodule in ``submodules``. - ``strict``: the ``strict`` option when ``set_state_dict`` calls model.load_state_dict(). - ``broadcast_from_rank0``: when the option is True, rank0 should receive a full state_dict and will broadcast the tensors in the state_dict/ optim_state_dict one by one to other ranks. Other ranks will receive the tensors and shard according to the local shards in the model and optimizer. ``full_state_dict`` must be set to True when using this option. This option currently only supports DTensor, not the legacy ShardedTensor. Ffull_state_dict cpu_offloadignore_frozen_paramsTkeep_submodule_prefixesstrictbroadcast_from_rank0flatten_optimizer_state_dict_fqn_modifiersdsd_fqn_modifiersN)__name__ __module__ __qualname____doc__r>bool__annotations__r?r@rArBrCrDrFstrr;r;r;r<r*\s $       r*c@seZdZUeedZeeeej fee ej ffe d<eedZ eeeej fee ej ffe d<ee dZe ee d<dZee d<dZee d<ejZee d<eedZeeje d <d S) _StateDictInfo)default_factoryfqn_param_mappingshared_params_mappingsubmodule_prefixesT handle_model handle_optim fsdp_context fsdp_modulesN)rGrHrIrdictrPr rMtorchTensorr$rLrQsetrRrSrKrT contextlib nullcontextrUr listrVnnModuler;r;r;r<rNs"       rNrET)modelnamerFskip_ddp_prefixskip_compiler_prefixreturnc s|td}d|vr|hS|d}g}|}t|D]r\}} t|trj| dksTJ|j}|sh|| q4t|tr|t |dkr||dt krd |t |t } rdfdd| j DSt |t}| tkr|| t || }q4t|tjjjr.| dksJ|j}|s|| q4t||rht ||| } rht|| rht || }|| | tjjjkr|t |dkrtdq4t || }q4d |tdhS) a This API is used to convert the name of a parameter to the FQNs. For FSDP without `use_orig_params`, the name of FlatParameter can be mapped to multiple original parameters. As a result, the return type of this function is `set[str]`. Args: module (nn.Module): the root model. name (str): the name skip_ddp_prefix (bool): whether to skip DDP's `module` prefix Returns: The canonical FQNs based on the model traversal. .modulecsh|]}|qSr;r;.0fqnprefixr;r< z_get_fqns.. _orig_modz-Expect `_extra_state` to be the last obj name)replacersplit enumerate isinstanceDDPrgappendFSDPlen _FLAT_PARAMjoingetattrZ_fqnsrrXZ_dynamoZ eval_frameZOptimizedModulerphasattrgetr^modules_EXTRA_STATE_KEY_SUFFIX RuntimeError) r`rarFrbrcZ obj_namesZ fqn_obj_namesZcurr_objiZ curr_obj_nameZ flat_paramZ removed_fqnr;rlr< _get_fqnssP                      rc@s eZdZdS) _EXTRA_STATEN)rGrHrIr;r;r;r<rsrc#s6ttjttdfdd |dEdHdS)N)rgcurr_fqnrdc3s||r|dnd}|D]Z\}}|vr6q$t|rb|t|vrb|dd}n ||}||EdHq$t|jdd|jddD]*\}}||jvrq||}||fVqt|j dt j j t j j kr|t j jj}|tfVdS)NrfreF)recurseget_extra_state)addZnamed_childrenr|r{valuesrZ named_buffersnamed_parametersZ_non_persistent_buffers_set __class__r^r_rr~rgrr)rgrra submodulenew_fqnobjrFrZvisited_modulesr;r<rs2      z+_iterate_valid_model_state..recursere)rZr^r_rMr)r`rFr;rr<_iterate_valid_model_states"r) submodulesoptions.)r`optims optim_onlyrrrdc Cs:|rtdt|r |s td|p(t}i}i}t|D]\}}t|trNq:t||} | |d} | durt t t || | ||||<n | ||<| D]} t|ts||| <qq:t|D]"\} } | D]} t tj| || <qqt } |rVt |}|D]L\}}||vrqt||} t| dks>Jd| dd| Dq|jrn|jsntdt|}|r|jrt|j|jd }t|j|jp|jd }tj}nt|jd }t |jd }tj!}t"j#d d }t$j%|||||d }nt"j&}t'fit(|||| |t tt)j*|| t|dkdS)zW Verify the model and options passed by the user and generates _StateDictInfo. zGetting submodules only model/optim state_dict is deprecated and will be removed in 2.5. This feature can be achieved by manually filtering out the state_dict returned from get_state_dict.z;Optimizers are not passed in but optim_only is set to True.Nrhz)Submodule FQN should only have 1 instancecss|]}|dVqdS)rfNr;rir;r;r< Kroz"_verify_options..z?full_state_dict must be True when broadcast_from_rank0 is True.)offload_to_cpuZ rank0_only)rc ssttXtjddtdtj||||ddVWdn1sH0YWdn1sf0YdS)NignorezFSDP.state_dict_type)messagecategoryrgstate_dict_typestate_dict_configoptim_state_dict_config)warningscatch_warningsfilterwarnings FutureWarningrwrrr;r;r<$fsdp_state_dict_type_without_warningis z=_verify_options..fsdp_state_dict_type_without_warningrr)rPrQrRrUrVrSrT)+rwarnrrr*rrtrrr}r rZrMupdatecopyr]itemsrXrY named_modulesrxrCr> ValueErrorrwrVrr?rrZFULL_STATE_DICTrrZSHARDED_STATE_DICTr[contextmanager functoolspartialr\rNrr^r_)r`rrrrrPrQraparamfqnsrkZparam_Zfqns_rRrgrVrrrrrUr;r;r<_verify_optionss              r)model_state_dictoptim_state_dictinfordcCs|jD]}t|}|dusJdq|jrf|sf|jsf|jsf|jrF|jsf|jrf|jsft dt d|j r|s|jr||js|jst d|| D] }t|vrt |dtdqdS)Nz)Expected a fsdp_state with a fsdp module.z}The option indicates that model state_dict is required to save or load, but model state_dict is empty.rank = dist.get_rank()=rfzgThe option indicates that model state_dict is required to save, or load but optim state_dict is empty. z contains z6. This can happen if the model is not the root module.)rVrrSrRr@r?r>rBrCrdistget_rankrTkeysry)rrrrgZ fsdp_statekeyr;r;r<_verify_state_dictsV   r)rapirdcCs,t||}|tvr(tjt|j||d}|S)N)self)r{r4rrr)rrcallr;r;r<_state_dict_fns r) state_dictrrdcCsD|jr.|jrtjsdnd}t||j|dS|jrr?rX distributedZis_initializedrr)rrrr;r;r<_maybe_full_or_cpu_state_dictsr)r`rrdc Csz|js iS|t|d}Wdn1s40Yt|D]t}t||}t|dkspJ||ftt|}||krJt ddd}|||st d|d|| |||<qJ|j r(i}|D]N}|j D]B}| |sq|jr||||<q|t|d} |||| <qq|}|jrp|D]6\}} | jrLq8t||}|D]}| |qZq8t||S)Nrrh)rdcSst|t|krdS|d}|d}d}t|D]P\}}|||krr|d7}|t|kr|t|dkSq4|dvr~q4q4dSq4dS)NFrfrrh)rgrpT)rxrrrs)rrkZ fqn_splitZ key_splitZfqn_idxZkey_idxZkey_namer;r;r<verifys    z%_get_model_state_dict..verifyzAn unexpected key, z, exists. FQN is )rSrUrr]rrrxnextiterrKrpoprR startswithrAr@r requires_gradr) r`rrrrrkrnew_state_dictrmrrr;r;r<_get_model_state_dicts@ *       r)r`rrrdc Cs|jr|s|jstiiSi}t||jD]\}}t|||j}t|||jddd}t||D]^\}} |jrxtdkr|| kr| |d} | dur|j rt d|dn| || <||| <q^q*d} |js|j rt } |D]*\}}t|r|dkr| |jqtd| vr0| tdd} t| dkrP| tjnt| dkrftd |jrt||| |j |jd n|j rt||| d |||,ttt|d ||j | d WdS1s0YdS)NF)rbrcrz Missing key: rfmetaTrhzMultiple devices found)devicerBr?rload_state_dict)rrBassign) rSrCr!rrFrziprrrrBrr>rZrrXZ is_tensordimrrremoverxZdistributed_c10dZ_get_pg_default_devicerrr?rrrUr r) r`rrlocal_state_dictrvaluerZfqns_with_prefixrkZfqn_with_prefix load_valuerZdevicesr;r;r<_load_model_state_dictsp      r)optimrdcCs|jr dS|jD]$}|tD]}|jdurdSqq|jD]$}|tD]}|jrHt||_qHq._raise_if_type_not_supportedrf) r r'_STATErr(_PGrrr]rM) rrretrkr3krrrr;r;r<_flatten_optim_state_dicts* r)rrrrdc Csi}g}t|t|i}|jD]}|tgi|tD]}|j|D]}||jvrd} |D]2} | tkrlq^td|d| } | |vrd} qq^nd} | sqD|dt} t| t sJ| ||j sqDi||<|j |D]*} |td|d| t t ||| <qqDq6t t t|dtd}|D]} | tkrDq2|td|d| }| |dvrx||d| <n<|d| |kr2td|d| d|d|d| d q2q|S) z This API unflattens the state_dict generated by _flatten_optim_state_dict(). See the docstring of _flatten_optim_state_dict() for more detail. FrfTrrzaAll the parameters in the same parameter group should have the same saved param_group value. But z is z while other(s) is )rrr1rvrrPrQrrtr]rr3r r'rMr)rrrr3pg_state return_osdrrrk in_paramsrZ flatten_keyr2Z state_nameZfirst_param_fqnrr;r;r<_unflatten_optim_state_dictsb            r)r` optimizersrrdc s,|js iStitgi}|D]}t|t|d}|jr|t|||}Wdn1sf0Y|svqt |t D]*}d|vr|t ||t| dd<q|tD]}dd|t D}||t <qnt tdd|jD}tt|tt|} i|D]X\} } t|| } t| d ks!roz)_get_optim_state_dict..css|]}|tVqdSr5)r)rjgr;r;r<r$roz(_get_optim_state_dict..rhcsg|] }|qSr;r;)rjpidZfqn_pid_mappingr;r<r6ro)"rTrrrrrVrUrwrr]rrrqrr from_iterabler1rWrrangerxrrrrr r'rr(extendrDr)rr)r`rrrrZosdrrr2Zparam_pid_mappingrrrrkrgroupr;rr<_get_optim_state_dictsV    ,        r)r`rrrrdc CsLi}g}t|t|i}i}tddtt|tDr<|S|jD]}|tgi|tD]} |j | D]} | |j vrd} tt |tD]"} | tt t | tvrd} qqnd} | sql|dt} t| t sJ| | | jrtt|t| || <tt |tD]6} | tt t | tvr t|td|t| <q qlq^t|tdkrBg}tt |tD],} ttt t | tdkrh|| qht|dkrtdt|tt|jkrtd t|td|t| <qBtt |tD]R}|t|d}|dkrq|D]$\}}|tkr4q||||<qq|S) a Extract the corresponding optim state_dict from ``optim_state_dict`` for ``optim`` and return the result optim state_dict. Args: model (nn.Module): the root model. optim (torch.optim.Optimizer): the optimizer. optim_state_dict (Dict[str, ValueType]): the superset optim state_dict that contains the optim state_dict of ``optim``. info (_StateDictInfo): state dict information. Returns: The optim state_dict of ``optim``. css|]}t|tVqdSr5)rtrrr;r;r<r`sz*_split_optim_state_dict..FTrrhrzThere are param groups that have zero parameters. In such a case, DSD only support exactly one param group with zero parameters.But the loaded state_dict has zero or more than one param groups that have zero parameters.z`When there is a parameter group that has zero parameters, multiple optimizers are not supported.)rrallr r'rr1rvrrPrQr(r]rMrtrrxidrr}r)r`rrrr3rrZ pg_mappingrrrkrZloaded_param_groupr2rZpg_idxrrr;r;r<_split_optim_state_dictFsp         r)r`rrrrdc s|js dS|D]x}t||rTt|vr8t||||}qXt|ttttf||}ni}|j rr| D]\}}t ||}t ||dd} || krqht |dksJ| | |tD]6} ttttf| } fdd| tD} | | t<qtt|t} t| D]&}|vr | || |<q qh|t|||}Wdn1sd0Yn|jrxd|_t||f|}d|_dfdd}ttj||}dusJt|\}}t|\}}|jrt||d nt ||d |D]6}||vr||vs"J||||<||||<qt!||}|tD]&}t|vrPgttttf|t<qPt"|d |d qdS) NF)rcrhcsg|]}|qSr;r)rjr)rkfqn_with_compilerr;r<rsz*_load_optim_state_dict..Tcs2|dkr.dur|jn|jkr.td|S)NrzDevice mismatch)rrr)trr;r<_devices   z'_load_optim_state_dict.._devicerrr)#rTrrrrr rWrMr&rVrrrxrrrrr'r]rrqrUrwZoptim_state_dict_to_loadr>rr#rXrYrrCrrrr)r`rrrrrZ original_fqn_rZfqns_with_compilerrvalr2Z osd_staterrrZ flatten_osdZ osd_mappingZflatten_local_osdZlocal_osd_mappingZ optim_keyZpgr;)rrkrr<_load_optim_state_dicts~       (        r)r`rrrdcCsVt<t|dd||d}t||}t|i||WdS1sH0YdS)aH Return the model state_dict of ``model``. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. submodules (deprecated): Optional[set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``model``. :rtype: typing.Dict[str, ValueType] r;FrrrN)r=rrr)r`rrrrr;r;r<r+s  r+)r`rrrrdcCsttZt|tjjr|fnt|}t||d||d}t|||}ti|||WdS1sf0YdS)a Return the combined state_dict for optimizers. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``optimizers``. :rtype: OptimizerStateType TrN) r=rtrXr Optimizertuplerrr)r`rrrrrr;r;r<r,*s   r,cCstht|tjjr|fnt|}t||d||d}t||}t|||}t |||||fWdS1st0YdS)a Return the model state_dict and optimizers state_dict. ``get_state_dict`` can process any module that is parallelized by PyTorch FSDP/fully_shard, DDP/replicate, tensor_parallel/parallelize_module, and any combination of these parallelisms. The main functions of ``get_state_dict`` are: 1.) returning a model and optimizer state_dict that can be resharded with a different number of trainers and/or different parallelisms. 2.) hiding the parallelism-specific state_dict APIs. Users don't have to call these APIs. 3.) sanity checking the result state_dict. The keys of the result state dictionary are the canonical FQNs (Fully Qualified Names). A canonical FQN refers to the FQN based on a parameter's position in an nn.Module hierarchy. More specifically, a canonical FQN to a parameter is the FQN returned by ``module.named_parameters()`` or ``module.named_buffers()`` when the module is not distributed by any parallelisms. Since the optimizer internally uses parameter IDs to represent a parameter, there will be a conversion from the parameter IDs to the canonical FQNs when calling this API. ``get_state_dict`` can also process a module that is not parallelized. In such a case, ``get_state_dict`` only performs one function -- converting the optimizer parameter IDs to the canonical FQNs. Example: >>> # xdoctest: +SKIP >>> import torch >>> from torch.distributed.fsdp import FullyShardedDataParallel as FSDP >>> from torch.nn.parallel import DistributedDataParallel as DDP >>> from torch.distributed.checkpoint.state_dict import get_state_dict >>> fsdp_model = FSDP(copy.deepcopy(model)) >>> fsdp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_model = DDP(copy.deepcopy(model)) >>> ddp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_state_dict, ddp_optim_state_dict = get_state_dict(ddp_model, ddp_optim) >>> fsdp_state_dict, fsdp_optim_state_dict = get_state_dict( ... fsdp_model, fsdp_optim ... ) >>> # if we simply call ddp_model.state_dict() and fsdp_model.state_dict(), >>> # the asserts will fail. >>> assert ddp_state_dict == fsdp_state_dict >>> assert ddp_optim_state == fsdp_optim_state_dict Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: ``Tuple`` that contain model state_dict and optimizer state_dict. :rtype: typing.Tuple[typing.Dict[str, ValueType], OptimizerStateType] FrN) r=rtrXrrrrrrr)r`rrrrrrr;r;r<r-Ws H    r-)r`rrdc s|siSttt|tjrtdtt t tjt t t ff|}i}| D]p\}}|D]^\}}||krvqdt||}t|dksJdtt|d|fdd| DqdqT|St t t t f|SdS)NzPassing model_state_dict as a ``Dict[nn.Module, Dict[str, Any]]``is deprecated and will be removed in 2.5. If you need this feature, please preprocessing the model_state_dict to achieve the same functionality.rhz/FQNs for a submodule should only have 1 elementrfcsi|]\}}||qSr;r;)rjZsubfqnrrlr;r< roz/_unflatten_model_state_dict..)rtrrrr^r_rrrr rWrMr&rrrrxr) r`rZcast_state_dictrrZsub_state_dictramrr;rlr<_unflatten_model_state_dicts* r)r)r`rrrdcCs\t||}t8t|dd|d}t|i|t|||WdS1sN0YdS)a=Load the model state_dict. The counterpart of ``get_model_state_dict`` to set the state_dict to the model. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. model_state_dict: (Dict[str, ValueType]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys * **unexpected_keys** is a list of str containing the unexpected keys :type model_state_dict: typing.Dict[str, ValueType] r;FrrN)rr=rrr)r`rrrr;r;r<r.s r.)r`rrrrdcCsptVt|tjjr|fnt|}t||d|d}ti||t||||Wdn1sb0YdS)aLoad the optimizers state_dict. The counterpart of ``get_optimizer_state_dict`` to set the state_dict to the optimizers. See ``set_state_dict`` for the detail usage. WARN: ``set_optimizer_state_dict`` can only be called before ``backward()`` or after ``step()`` is called on the optimizers. Otherwise, the optimizer states won't be initialized correctly. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None :type optim_state_dict: typing.OptimizerStateType TrN) r=rtrXrrrrrr)r`rrrrr;r;r<r/s  r/)r`rrrrrdcCst||}tdt|tjjr&|fnt|}t||| |d}t|||t ||||t |||WdS1sz0YdS)aLoad the model state_dict and optimizers state_dict. The counterpart of ``get_state_dict`` to set the state_dict to the model and optimizers. The given ``model_state_dict`` and ``optim_state_dict`` do not have to be returned by ``get_state_dict`` but must meet the following requirements: 1) all FQNs are canonical FQNs as defined in ``get_state_dict``, 2) if a tensor is sharded, it must be either a ShardedTensor or DTensor, 3) optimizer state_dict cannot contain the parameter IDs; the keys should be the canonical FQNs. WARN: ``set_state_dict`` can only be called before ``backward()`` or after ``step()`` is called on the optimizers. Otherwise, the optimizer states won't be initialized correctly. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. model_state_dict: (Union[Dict[nn.Module, Dict[str, ValueType]], Dict[str, ValueType]]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys of the model state_dict. * **unexpected_keys** is a list of str containing the unexpected keys of the model state_dict. :type model_state_dict: typing.Dict[str, ValueType] :type optim_state_dict: typing.OptimizerStateType rN) rr=rtrXrrrrrrr)r`rrrrrr;r;r<r0%s.   r0)r`rrdcsjtjt||dfdd}||_tjt||dtttfdfdd }||_t |t |dS)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``model``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``model`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )r`rcsSr5r;r;_state_dict_callr;r<state_dict_callsz0_patch_model_state_dict..state_dict_callrcs|ddS)N)rr;r_load_state_dict_callr;r<load_state_dict_callsz5_patch_model_state_dict..load_state_dict_callN) rrr+rr.rWrMrrr4r)r`rr r r;r rr<_patch_model_state_dictgs   r)r`rrrdcstjt|||dfdd}tjt|||dtttfdfdd }t|t|t |t j j rr|fnt |}|D]}||_||_q~dS)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Note that if there are multiple optimizers, all of the optimizers will be patched. So users only need to call one of the state_dict() to get the full result. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )r`rrcsSr5r;r;rr;r<r sz4_patch_optimizer_state_dict..state_dict_callrcs|ddS)N)rr;rr r;r<r sz9_patch_optimizer_state_dict..load_state_dict_callN)rrr,r/rWrMrr4rrtrXrrrrr)r`rrr r rr;r r<_patch_optimizer_state_dicts.    r)rETT)rE)qr[rr6rcollections.abcrrZ dataclassesrrr itertoolsrtypingrr r r r r rXZtorch.distributedrrZtorch.nnr^Z'torch.distributed._shard.sharded_tensorrZ#torch.distributed._state_dict_utilsrrrrrrZ;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrZtorch.distributed.fsdprrrrwrrrrrZ$torch.distributed.fsdp._common_utilsrrZtorch.distributed.tensorr Ztorch.nn.modules.moduler!Ztorch.nn.parallelr"ruZtorch.utils._pytreer#__all__ryrrrrZrMr$rYrrr%r]rrWr&r'r(r)r4rLrr=r*rNr_rKrrrrrrrrrZno_gradrrrrrrrrr+r,r-rr.r/r0rrr;r;r;r<sb        (       / G -    -    ? D*A  ?@ ^c  )  1  \$ %  , 1  B6