a h,@s`ddlZddlZddlmZmZddlmZmZmZm Z ddl m m m mZddlmZmZddlmZddlmZddlmZmZmZddlmZdd lmZdd lm Z dd l!m"Z"gd Z#e"e$Z%eGd ddZ&GdddZ'e ee(dfe)ee(dddZ*ee+ee(ee,fdddZ-e&e ee(dfe)ee.e,efdddZ/dS)N) dataclassfield)AnyCallableOptionalUnion)eventsmetrics) WorkerSpec)LocalElasticAgent)DefaultLogsSpecs LogsSpecsSignalException)ChildFailedError)RendezvousParameters)parse_rendezvous_endpoint) get_logger) LaunchConfigelastic_launch launch_agentc@seZdZUdZeed<eed<eed<dZeeed<dZ e ed<d Z e ed <dZ e ed <d Z e ed <eedZee efed<dZeed<dZeed<dZeed<dZe ed<dZee ed<eedZee e fed<dZee ed<dZe ed<ddZdS)ra Creates a rendezvous config. Args: min_nodes: Minimum amount of nodes that the user function will be launched on. Elastic agent ensures that the user function start only when the min_nodes amount enters the rendezvous. max_nodes: Maximum amount of nodes that the user function will be launched on. nproc_per_node: On each node the elastic agent will launch this amount of workers that will execute user defined function. rdzv_backend: rdzv_backend to use in the rendezvous (zeus-adapter, etcd). rdzv_endpoint: The endpoint of the rdzv sync. storage. rdzv_configs: Key, value pair that specifies rendezvous specific configuration. rdzv_timeout: Legacy argument that specifies timeout for the rendezvous. It is going to be removed in future versions, see the note below. The default timeout is 900 seconds. run_id: The unique run id of the job (if not passed a unique one will be deduced from run environment - flow workflow id in flow - or auto generated). role: User defined role of the worker (defaults to "trainer"). max_restarts: The maximum amount of restarts that elastic agent will conduct on workers before failure. monitor_interval: The interval in seconds that is used by the elastic_agent as a period of monitoring workers. start_method: The method is used by the elastic agent to start the workers (spawn, fork, forkserver). metrics_cfg: configuration to initialize metrics. local_addr: address of the local node if any. If not set, a lookup on the local machine's FQDN will be performed. local_ranks_filter: ranks for which to show logs in console. If not set, show from all. event_log_handler: name of the event logging handler as registered in `elastic/events/handlers.py `_. .. note:: `rdzv_timeout` is a legacy argument that will be removed in future. Set the timeout via `rdzv_configs['timeout']` min_nodes max_nodesnproc_per_nodeN logs_specsrun_idZ default_rolerole rdzv_endpointZetcd rdzv_backend)default_factory rdzv_configs rdzv_timeout max_restartsg?monitor_intervalspawn start_methodlog_line_prefix_template metrics_cfg local_addrnullevent_log_handlercCsFd}|jdkr|j|jd<nd|jvr0||jd<|jdurBt|_dS)Nir!timeout)r"r rr )selfZdefault_timeoutr/L/var/www/auris/lib/python3.9/site-packages/torch/distributed/launcher/api.py __post_init___s    zLaunchConfig.__post_init__)__name__ __module__ __qualname____doc__int__annotations__rrr rstrrrrrdictr rr"r$r%floatr'r(r)r*r,r1r/r/r/r0r"s& )         rc@s2eZdZdZeeeedfdddZddZ dS)ra Launches an torchelastic agent on the container that invoked the entrypoint. 1. Pass the ``entrypoint`` arguments as non ``kwargs`` (e.g. no named parameters)/ ``entrypoint`` can be a function or a command. 2. The return value is a map of each worker's output mapped by their respective global rank. Usage :: def worker_fn(foo): # ... def main(): # entrypoint is a function. outputs = elastic_launch(LaunchConfig, worker_fn)(foo) # return rank 0's output return outputs[0] # entrypoint is a command and ``script.py`` is the python module. outputs = elastic_launch(LaunchConfig, "script.py")(args) outputs = elastic_launch(LaunchConfig, "python")("script.py") N)config entrypointcCs||_||_dSN)_config _entrypoint)r.r;r<r/r/r0__init__szelastic_launch.__init__cGst|j|jt|Sr=)rr>r?list)r.argsr/r/r0__call__szelastic_launch.__call__) r2r3r4r5rrrr8r@rCr/r/r/r0rks  r)r<rBreturncCsFt|tr|jSt|tr>|tjkr8tdd|DdS|SndSdS)aRetrieve entrypoint name with the rule: 1. If entrypoint is a function, use ``entrypoint.__qualname__``. 2. If entrypoint is a string, check its value: 2.1 if entrypoint equals to ``sys.executable`` (like "python"), use the first element from ``args`` which does not start with hifen letter (for example, "-u" will be skipped). 2.2 otherwise, use ``entrypoint`` value. 3. Otherwise, return empty string. css|]}|ddkr|VqdS)r-Nr/).0argr/r/r0 z'_get_entrypoint_name..rN) isinstancerr2r8sys executablenext)r<rBr/r/r0_get_entrypoint_names   rN)rdzv_parametersrDcCsX|jdkrdS|j}|}|s(tdt|dd\}}|dkrPtd|d||fS)NZstatic)NNzKEndpoint is missing in endpoint. Try to add --master-addr and --master-portr!) default_portzport is missing in endpoint: z. Try to specify --master-port)backendendpointstrip ValueErrorr)rOrR master_addr master_portr/r/r0_get_addr_and_ports  rW)r;r<rBrDc Cs|js&ttj}td|||_t||}td||j |j |j |j|j |j |j|j|j|jj|j|jd tf|j |j |j|j |j |jd|j}t|\}}t|j|j |t|t||j|j|||j|jd }t||j|j|jd} d} zz^t !t "|j| #} t$%| &|j| 'r>t(|| j)d| j*WW| rV|j+,St(ylYnRt-yd } t$%| .|jYn(t/yt$%| .|jYn0W| r|j+,n| r|j+,0dS) Nz3config has no run_id, generated a random run_id: %sa/Starting elastic_operator with launch configs: entrypoint : %(entrypoint)s min_nodes : %(min_nodes)s max_nodes : %(max_nodes)s nproc_per_node : %(nproc_per_node)s run_id : %(run_id)s rdzv_backend : %(rdzv_backend)s rdzv_endpoint : %(rdzv_endpoint)s rdzv_configs : %(rdzv_configs)s max_restarts : %(max_restarts)s monitor_interval : %(monitor_interval)s log_dir : %(log_dir)s metrics_cfg : %(metrics_cfg)s event_log_handler : %(event_log_handler)s ) r<rrrrrrr r$r%Zlog_dirr)r,)rQrRrrrr*) rZlocal_world_sizer<rB rdzv_handlerr$r%rUrVr*r,)specrr'r(T)namefailuresF)0rr8uuiduuid4r6loggerwarningrNinforrrrrr r$r%rZ root_log_dirr)r,rr*rWr rtuple rdzv_registryZget_rendezvous_handlerr r'r(r Zinitialize_metricsZ MetricsConfigrunrrecordZget_event_succeededZ is_failedrr[Z return_valuesrXshutdownrZget_event_failed Exception) r;r<rBrZentrypoint_namerOrUrVrYZagentZ shutdown_rdzvresultr/r/r0rs        r)0rKr\Z dataclassesrrtypingrrrrZ-torch.distributed.elastic.rendezvous.registryZ distributedZelasticZ rendezvousregistryrbZtorch.distributed.elasticrr Z*torch.distributed.elastic.agent.server.apir Z:torch.distributed.elastic.agent.server.local_elastic_agentr Z)torch.distributed.elastic.multiprocessingr r rZ0torch.distributed.elastic.multiprocessing.errorsrZ$torch.distributed.elastic.rendezvousrZ*torch.distributed.elastic.rendezvous.utilsrZ'torch.distributed.elastic.utils.loggingr__all__r2r^rrr8rArNrar6rWr9rr/r/r/r0 s8      H(