o Zh^ @sPddlZddlZddlmZddlZddlZddlmZddlm Z m Z ddl m Z GdddeZ Gd d d eZejd efd d Zdejd e fddZdejd efddZdejd efddZGdddeZGdddeZGdddeZdgdggZdgdggdgdggdgd gggZgd!gd"gd"gZdejd efd#d$ZdS)%N)IntEnum)ir)get_dtype_size sympy_product)Vc@eZdZdZdZdZdS) NCCL_COLLrrN)__name__ __module__ __qualname__ ALL_REDUCE ALL_GATHERREDUCE_SCATTERrrL/var/www/auris/lib/python3.10/site-packages/torch/_inductor/comm_analysis.pyr r c@r)NVIDIA_GPU_TYPErrr N)r r r VOLTAAMPEREHOPPERrrrrrrrreturncCsHtjjtjjjp d}d|vrtjSd|vrtjSd|vr!tjStjS)NZV100ZA100ZH100) torchutilsZ collect_envZ get_gpu_inforunrrrr)Zgpu_inforrr get_gpu_typesrnodecCsdt|tjs td||j}|dusJd|vrtjSd|vr$tjSd|vr+tjStd|)Nz!node is not a collective kernel: Z all_reduceZ all_gatherZreduce_scatterzUnsupported collective kernel: ) isinstancer_CollectiveKernel ValueErrorZpython_kernel_namer rrr)rZ kernel_namerrrget_collective_type(s  r"cCs\d}|jD]&}t|jj}t|tjrt|}n tj j j |dd}||t |jj 7}q|S)Nr)fallback)ZinputsrZlayoutsizersympyIntegerintrgraphZsizevarsZ size_hintrZdtype)rZsz_bytesinpZnumelrrrget_collective_input_size_bytes8s    r*cCs6t|tjkrddlm}||jdStd|)Nr)_get_group_size_by_namezUnsupported collective type: )typerr Z"torch.distributed.distributed_c10dr+Z constant_args TypeError)rr+rrrget_collective_group_sizeEs r/c@r)NCCL_HWrrr N)r r r NVLINKZPCINETrrrrr0Srr0c@seZdZdZdZdS) NCCL_ALGOrrN)r r r ZTREERINGrrrrr3Ysr3c@seZdZdZdS) NCCL_PROTOrN)r r r LLrrrrr5^sr5g333333@gffffff@g333333??g@g@)C@r8gffffff4@)gU@g6@g3@cCst|}|ddd}d}t|}t||}|}|dkr!dStj}tj}t|} t j j j } t j j j } t} |dkr@|dnd} |dkrH| nd}t|| }|dkrV| n| }d}||}t|||dksj| tjkrldnd}| tjkr|d|d}n | tjtjfvr|d}d||}||}|d }tj}| tjkr|dkrd|}nd}n | tjtjfvr|d}t||}t|||}ttj||}d }|dkrd}t||}||||||7}|d }||}||S) a9 Returns estimated NCCL collective runtime in nanoseconds (ns). The following heuristics are copied from https://github.com/NVIDIA/nccl/blob/master/src/graph/tuning.cc. We aim to estimate the runtime as accurately as possible. Assumptions: - only ring algorithm (NCCL_ALGO_RING) is used - only Low-Latency protocol (NCCL_PROTO_LL) is used, i.e. Simple or LL128 is not used - 8 gpus per node # TODO: Need to find a way to get accurate "gpus per node" and "# nodes" info. - collective is one of: allreduce, reducescatter, allgather irrr g?gUUUUUU?r7geAgg@@)r*r/mathceilr3r4r5r6r"rZ _inductorconfigZ intra_node_bwZ inter_node_bwrllMaxBwsminr rrrr0r1baseLathwLatr2max)rZtensor_storage_size_bytesZtensor_storage_size_GBZnum_gpus_per_nodeZ group_sizeZnNodesZnRanksZ nccl_algoZ nccl_protoZcollZbwIntraZbwInterZ compCapIndexindex2Zindex1ZllMaxBwbwZ nChannelsZbusBwZnstepsratio bandwidthZbandwidth_GB_per_nsZintraHwZ nInterStepsZlatencyZintraLatZinterLatZ netOverheadZ latency_nsZ transport_nsrrr estimate_nccl_collective_runtimesf          rF) functoolsr:enumrr%rrrrrrZ virtualizedrr r lru_cacherZIRNoder"r'r*r/r0r3r5r?r@r=floatrFrrrrsL