[ThL&SSKrSSKJr SSKJr SSKJr SSKrSSKJ s J r SSK J s J r SSKJr SSKJr SSKJr SSK Jr SSKJr S /r\R."\5r\R4"S S 55r"S S \R85r"SS5rSrSr Sr!\RDS\\S\#SS4Sj5r$Sr%"SS 5r&g)N) defaultdict)Lock)Optional)Tensor)RRef)functional_optim_mapDistributedOptimizerc&\rSrSrS\SS4SjrSrg)_ScriptLocalOptimizerInterface"autograd_ctx_idreturnNcgN)selfrs Y/var/www/auris/envauris/lib/python3.13/site-packages/torch/distributed/optim/optimizer.pystep#_ScriptLocalOptimizerInterface.step$s r)__name__ __module__ __qualname____firstlineno__intr__static_attributes__rrrr r "s C D rr cb^\rSrSr\"5rU4Sjr\RS\ 4Sj5r Sr U=r $)_ScriptLocalOptimizer(c>[TU]5 UVs/sHoUR5PM snUlU"UR/UQ70UD6Ulgs snfr)super__init__ local_value _local_paramsoptim)r optim_clslocal_params_rrefargskwargsrref __class__s rr#_ScriptLocalOptimizer.__init__.sP =NO=NT..0=NOt11CDCFC PsArc[R"U5nURVs/sHnX2;aX#OSPM nnURR U5 gs snfr) dist_autograd get_gradientsr%r&r)rrall_local_gradspgradss rr_ScriptLocalOptimizer.step3s^'55oF'') '#$"6O D @' )  ) sAr%r&) rrrrr compile_lockr#jitexportrrr __classcell__)r,s@rrr(s16LD  ZZCrrc.\rSrSr\"5rSrSrSrg)_LocalOptimizerAcUVs/sHoUR5PM snUlU"UR/UQ70UD6Ulgs snfr)r$r%r&)rr'r(r)r*r+s rr#_LocalOptimizer.__init__KsD=NO=NT..0=NOt11CDCFC PsAc[R"U5n[R UR 5H up4XClM UR R5 SSS5 g!,(df  g=fr)r/r0r; global_lockitemsgradr&r)rrr1paramrBs rr_LocalOptimizer.stepOsO'55oF  ( (.446 !  7 JJOO ) ( (s :A** A8r5N) rrrrrr@r#rrrrrr;r;As&KDrr;cH[R"[X/UQ70UD65$r)rpcrr;)r'r(r)r*s r_new_local_optimizerrGXs 88OIR4R6R SSrcFUR5nURU5 grr$rlocal_optim_rrefr local_optims r_local_optimizer_steprM\s"..0K_%rc[X/UQ70UD6n[R [R"U5n[R "U[ 5sSSS5 $!,(df  g=fr)rr6r7scriptrFrr )r'r(r)r*r& script_optims r_new_script_local_optimizerrQbsH !) P P PE  + +zz%( xx &DE , + +s 1A A)rKrrcFUR5nURU5 grrIrJs r_script_local_optimizer_steprSjs #..0K_%rcSn/nUH#nURUR55 M% UbUeU$![anURU5 UnSnAMQSnAff=fr)appendwait Exception)rpc_futs exceptionresultsfutes r _wait_for_allr]rsbIG  NN388: &  N   NN1 I s5 AAAc$\rSrSrSrSrSrSrg)r a DistributedOptimizer takes remote references to parameters scattered across workers and applies the given optimizer locally for each parameter. This class uses :meth:`~torch.distributed.autograd.get_gradients` in order to retrieve the gradients for specific parameters. Concurrent calls to :meth:`~torch.distributed.optim.DistributedOptimizer.step`, either from the same or different clients, will be serialized on each worker -- as each worker's optimizer can only work on one set of gradients at a time. However, there is no guarantee that the full forward-backward-optimizer sequence will execute for one client at a time. This means that the gradients being applied may not correspond to the latest forward pass executed on a given worker. Also, there is no guaranteed ordering across workers. `DistributedOptimizer` creates the local optimizer with TorchScript enabled by default, so that optimizer updates are not blocked by the Python Global Interpreter Lock (GIL) in the case of multithreaded training (e.g. Distributed Model Parallel). This feature is currently enabled for most optimizers. You can also follow `the recipe`__ in PyTorch tutorials to enable TorchScript support for your own custom optimizers. Args: optimizer_class (optim.Optimizer): the class of optimizer to instantiate on each worker. params_rref (list[RRef]): list of RRefs to local or remote parameters to optimize. args: arguments to pass to the optimizer constructor on each worker. kwargs: arguments to pass to the optimizer constructor on each worker. Example:: >>> # xdoctest: +SKIP("distributed") >>> import torch.distributed.autograd as dist_autograd >>> import torch.distributed.rpc as rpc >>> from torch import optim >>> from torch.distributed.optim import DistributedOptimizer >>> >>> with dist_autograd.context() as context_id: >>> # Forward pass. >>> rref1 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 3)) >>> rref2 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 1)) >>> loss = rref1.to_here() + rref2.to_here() >>> >>> # Backward pass. >>> dist_autograd.backward(context_id, [loss.sum()]) >>> >>> # Optimizer. >>> dist_optim = DistributedOptimizer( >>> optim.SGD, >>> [rref1, rref2], >>> lr=0.05, >>> ) >>> dist_optim.step(context_id) __ https://github.com/pytorch/tutorials/pull/1465 c`[RRS5 [[5nUH$nXVR 5R U5 M& U[;a6[RR(a[R"U5nOUnXq:gUl UR(a[nO[RSU5 [ n/n UR#5H1up[$R&"U UX{4U-US9n U R U 5 M3 [)U 5Ulg)Nz,torch.distributed.optim.DistributedOptimizera$Creating the optimizer %s without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. )r)r*)torch_C_log_api_usage_oncerlistownerrUr r7_state_enabledgetis_functional_optimrQloggerwarningrGrArF rpc_asyncr]remote_optimizers) roptimizer_class params_rrefr)r*per_worker_params_rrefrC optim_ctoroptimizer_new_funcremote_optim_futsworker param_rrefsremote_optim_rref_futs rr#DistributedOptimizer.__init__s $$%ST!,T!2 E ";;= 1 8 8 ?! 2 2szz7J7J-11/BJ(J#-#@  # #!<  NN-   "6 #9#?#?#A F$'MM" .5 % !  $ $%: ;$B"//@!Arc [R"U5 UR(a[O[nUR Vs/sH(n[ R"UR5UX14S9PM* nn[U5 gs snf)a Performs a single optimization step. This will call :meth:`torch.optim.Optimizer.step` on each worker containing parameters to be optimized, and will block until all workers return. The provided ``context_id`` will be used to retrieve the corresponding :class:`~torch.distributed.autograd.context` that contains the gradients that should be applied to the parameters. Args: context_id: the autograd context id for which we should run the optimizer step. )r)N) r/_is_valid_contextrirSrMrmrFrlrer])r context_idoptimizer_step_func optimizerrXs rrDistributedOptimizer.steps '' 3'' )& "33  4 MM!#,  4   h s/A?)rirmN)rrrr__doc__r#rrrrrr r s9v#BJ r)'logging collectionsr threadingrtypingrratorch.distributed.autograd distributedautogradr/torch.distributed.rpcrF torch.jitr7torch.nnnnrrutilsr __all__ getLoggerrrj interfacer Modulerr;rGrMrQrOrrSr]r rrrrs# 22##&' " "   8 $   BII2.T& F&9:&MP& & &   r