a hV$@sddlZddlmZddlmZmZmZddlZddlm Z ddl m Z ddl m Z ddlmZmZdZGd d d e Zeeed d d ZGdddZeedde jeeej je jdddZe jedddZdS)N)Iterable)AnyNoReturnOptional)_State)DistributedDataParallel) _get_registrycontractcseZdZUejed<ddfdd Zefej e ej e ej e ddddZ ddd d Zej e ej dd d d ZddddZddddZej eedfee efedddZej eejejejdddZZS)_ReplicateState _ddp_weakrefNreturncsNtt|_d|_t|_|j|_g|_d|_ d|_ i|_ g|_ dS)NF) super__init__nnZ ParameterListmodulehas_initialized _param_listZ _orig_module _param_names_no_sync _init_args _init_kwargs_comm_hook_argsself __class__U/var/www/auris/lib/python3.9/site-packages/torch/distributed/_composable/replicate.pyrs   z_ReplicateState.__init__)rignored_modulesignored_paramsprefixrc Cst|r dS||vrdS|tkr*|dnt}|jddD]0\}}||vr:|j||j||q:|D]"\}} |j| ||||dqtdS)N.F)Zrecurse)r#)_is_fully_sharded_ROOT_MODULE_PREFIXZnamed_parametersrappendrZnamed_children_collect_params) rrr!r"r#Zrecurse_prefixnpnameZ child_modulerrr r("s"  z_ReplicateState._collect_paramscs"tjddfdd}|dS)NT) recursivecs:jdusJjjijd_i_dS)Nr)rinitrregister_comm_hookrrrr _lazy_initCs z-_ReplicateState.lazy_init.._lazy_init)torchZ_disable_dynamo)rr/rrr lazy_initBs z_ReplicateState.lazy_initrr!rcKs|jr dSd|_||_dd|D}|D]}t|r,||q,ddlm}|||d||||d|vr|ddur|d}t |t j r|j dkrd|d <q|g|d <nd|d <| dt|jfi||_t|jt|j_dS) NTcSsh|]}|D]}|qqSr) parameters).0mr*rrr Xz'_ReplicateState.init..r)_localize_dtensor)r" device_idcpuZ device_ids)rrmodulesr%updater3%torch.distributed.tensor.parallel.ddpr8r( isinstancer0devicetypepoprr_ddpweakrefref replicatestater )rrr!kwargsr" submoduler8r9rrr r-Ms*       z_ReplicateState.initcCs0|jD]\}}|jj|i|q|jdSN)rrBr.clear)rZ comm_argsZ comm_kwargsrrr r.wsz"_ReplicateState.register_comm_hookcOs||_||_dSrI)rrrargsrGrrr record_init_args|sz _ReplicateState.record_init_args.)rrLrGrcCs2|js |jr||j |j_|jj|i|SrI)rrr1rrBZrequire_backward_grad_syncZ _pre_forward)rrrLrGrrr forward_pre_hooks  z _ReplicateState.forward_pre_hook)rinputoutputrcCs |j|SrI)rBZ _post_forward)rrrOrPrrr forward_post_hooksz!_ReplicateState.forward_post_hook)__name__ __module__ __qualname__rCrD__annotations__rr&rModuleset Parameterstrr(r1r-r.rMtuplerdictrNr0ZTensorrQ __classcell__rrrr r s2     * r )rLrGrcOs tddS)NzGDDP does not support deepcopy. Please use state dict for serialization.)AssertionError)rLrGrrr unimplemented_deepcopysr^c@s2eZdZddZeddddZdddd ZdS) DDPcOs"|jd}|j|g|Ri|S)z Override ``__new__`` to remove the DDP class and directly construct the original class for cases like indexing into a container module. )__mro____new__)clsrLrGZorig_clsrrr rbs z DDP.__new__N)requires_gradient_syncrcCs| t|_dS)a Sets if the module should sync gradients. This can be used to implement gradient accumulation without communication. Args: requires_gradient_sync (bool): Whether to reduce gradients for the module's parameters. N)rErFr)rrdrrr set_requires_gradient_syncs zDDP.set_requires_gradient_syncrcOst|j||fdSrI)rErFrr'rKrrr r.szDDP.register_comm_hook)rRrSrTrbboolrer.rrrr r_s  r_)Z state_clsr2c Ks$tjdd|vr>t|dttjfs>tdt|dt|rNtd|dur\i}nt |}t |}|j |j dd|dd}|durd d lm}||}||krd d lm}m}| |||||j|j||fi||j} d ti} td | jt| f| } | |_|S)zReplicates a module Args: module (torch.nn.Module): module to replicate Example:: >>> # xdoctest: +REQUIRES(module:torch._C._distributed_c10d) >>> module = nn.Linear(3, 3) >>> replicate(module) ztorch.distributed.replicater9z6Expected device_id to be int or torch.device, but got zGCannot apply `replicate()` on a Module already managed by `fully_shard`NT)Z with_kwargs device_meshr)_mesh_resources)r8_reconstruct_dtensor __deepcopy__r_)r0Z_CZ_log_api_usage_oncer>intr? RuntimeErrorr@r%rWrErFZregister_forward_pre_hookrNgetZtorch.distributed.device_meshrhZ get_root_meshr=r8riZregister_forward_hookrQrMrr^rRr_) rr!rGrFrgrhZ root_meshr8rircdctZnew_clsrrr rEs@         rE)rrcCst|}|durdSd|vS)z+Check if module is marked with fully_shard.NFZ fully_shard)r )rregistryrrr r%sr%)N)rCcollections.abcrtypingrrrr0Ztorch.nnrZ#torch.distributed._composable_staterZtorch.nn.parallelrr r r&r r^r_rVrErfr%rrrr s(    H