o Zh5 @sUddlZddlZddlmZmZmZddlmZddlZddl m Z ddl m Z ddlmZgZeeed<dZefded ejfd d Zefded eejeffd d Zead eeeffddZedZedZdeeefd eeeffddZdeeefd eeeffddZ dS)N)AnyCallableTypeVar) ParamSpec) _log_handlers) _WaitCounter__all__default destinationreturncCsPt|\}}td|}|tjtd}||d|_|||S)Nzc10d-za%(asctime)s %(filename)s:%(lineno)s %(levelname)s p:%(processName)s t:%(threadName)s: %(message)sF) _get_logging_handlerlogging getLoggersetLevelDEBUG Formatter setFormatter propagate addHandler)r Zlogging_handlerlog_handler_namelogger formatterrL/var/www/auris/lib/python3.10/site-packages/torch/distributed/c10d_logger.py_get_or_create_loggers    rcCs$t|}t|jd|}||fS)N-)rtype__name__)r Z log_handlerrrrrr 'sr c OstrL|dp |d}|t|dt|tt|tt|d}|ddkrJtjj }d dd |D|d <|Sd |i}|S) NgroupZ process_groupZpg) func_nameZpg_namebackendZ world_sizeZ group_sizeZ global_rankZ local_rankr nccl.css|]}t|VqdS)N)str).0vrrr Asz _get_msg_dict.. nccl_versionr) distZis_initializedgetZ_get_process_group_name get_backendZget_world_sizeZget_ranktorchcudar!versionjoin)rargskwargsrmsg_dictr'rrr _get_msg_dict3s"    r2_T_Pfuncc,tdtjdtjdtffdd }|S)Nr/r0r c sXz|i|WSty+}ztjg|Ri|}||d<t|d}~ww)Nerror) Exceptionr2r _c10d_loggerdebug)r/r0r7r1r5rrwrapperNs  z"_exception_logger..wrapper functoolswrapsr4r/r0r3r5r<rr;r_exception_loggerMs  rAcr6)Nr/r0r csHtdj|i|}Wd|S1swY|S)Nzpytorch.wait_counter.c10d.)rrguard)r/r0Z func_returnr;rrr<\s  z_time_logger..wrapperr=r@rr;r _time_logger[s rC)!r>r typingrrrZtyping_extensionsrr+Ztorch.distributed distributedr(Z"torch.distributed.logging_handlersrZ torch.monitorrrlistr#__annotations__Z_DEFAULT_DESTINATIONLoggerrtupleHandlerr r9dictr2r3r4rArCrrrrs.        "&