a h @sTddlZddlZddlmZddlmZmZddlZgdZ dZ ddZ ddZ d d Z d d Zeejeejfdd ddZde ddfddZdde ddfddeejeeejeejfeeeeejjeeejddddZdeejeddddZd eejeejddddZe ddfeejeejeddddZdS)!N)Sequence)OptionalUnion) all_reducereduce broadcast all_gatherreduce_scattercCstttjdstddSt}|D]J}|jr4dS|sBdS|jsNdS| }||vrddS| |q$dS)N_nccl_all_reducez)PyTorch is not compiled with NCCL supportFT) hasattrtorch_CwarningswarnsetZ is_sparseZ is_contiguousZis_cudaZ get_deviceadd)ZtensorsZdevicesZtensorZdevicer=/var/www/auris/lib/python3.9/site-packages/torch/cuda/nccl.py is_availables    rcCsXtj}|d?}|d?d@}|d@}tjd}|dkrH|||fS||||fSdS)a Returns the version of the NCCL. This function returns a tuple containing the major, minor, and patch version numbers of the NCCL. The suffix is also included in the tuple if a version suffix exists. Returns: tuple: The version information of the NCCL. izutf-8N)r r Z _nccl_versionZ_nccl_version_suffixdecode)vermajorminorpatchsuffixrrrversion$s   rcCs tjSN)r r Z_nccl_unique_idrrrr unique_id9sr cCstj|||Sr)r r Z_nccl_init_rank)Z num_ranksuidZrankrrr init_rank=sr")inputsreturncCs&t|tjjrt|tjr"tddS)Nz(Inputs should be a collection of tensors) isinstance collectionsabc Containerr Tensor TypeError)r#rrr_check_sequence_typeAsr+cCs4t||dur|}t|tj|||||dSr)r+r r r r#outputsopstreamscommsrrrrHs r)r-)r#outputrootr.r/r-r$cCst||dur<|dur"tdqtjdtdd||}nHt|tjspt|tj j rptjdtdd||}n|dur||n|}tj ||||||dS)Nz'output' and 'outputs' can not be both specified. 'outputs' is deprecated in favor of 'output', taking in a single output tensor. The signature of reduce is: reduce(inputs, output=None, root=0, op=SUM, streams=None, comms=None).z`nccl.reduce` with an output tensor list is deprecated. Please specify a single output tensor with argument 'output' instead instead.) stacklevelz\nccl.reduce with an output tensor list is deprecated. Please specify a single output tensor.) r+ ValueErrorrr FutureWarningr%r r)r&r'rr Z _nccl_reduce)r#r1r2r.r/r0r-_outputrrrrRs.   r)r#r2r$cCst|tj||||dSr)r+r r Z_nccl_broadcast)r#r2r/r0rrrr}sr)r#r-r$cCs&t|t|tj||||dSr)r+r r Z_nccl_all_gather)r#r-r/r0rrrrsr)r#r-r.r$cCs(t|t|tj|||||dSr)r+r r Z_nccl_reduce_scatterr,rrrr sr )rNN)NN)r&rcollections.abcrtypingrrZ torch.cudar __all__ZSUMrrr r"r)r+rintZcudaZStreamrrrr rrrrsb     ,