JTh*SSKrSSKrSSKJrJrJrJrJrJr SSK J r S SS.Sjjr S Sjr S Sjr S SjrSSS.S jjrSSS.S jjrg)N)_flatten_dense_tensors_get_device_index_handle_complex_reorder_tensors_as _take_tensors_unflatten_dense_tensors)nccl)outc[U5nUSLUSL- (d[SUSU35eUb:UVs/sHn[U5PM nn[RR X5$[RR X5$s snf)aBroadcasts a tensor to specified GPU devices. Args: tensor (Tensor): tensor to broadcast. Can be on CPU or GPU. devices (Iterable[torch.device, str or int], optional): an iterable of GPU devices, among which to broadcast. out (Sequence[Tensor], optional, keyword-only): the GPU tensors to store output results. .. note:: Exactly one of :attr:`devices` and :attr:`out` must be specified. Returns: - If :attr:`devices` is specified, a tuple containing copies of :attr:`tensor`, placed on :attr:`devices`. - If :attr:`out` is specified, a tuple containing :attr:`out` tensors, each containing a copy of :attr:`tensor`. NzFExactly one of 'devices' and 'out' must be specified, but got devices=z and out=)r RuntimeErrorrtorch_C _broadcast_broadcast_out)tensordevicesr ds N/var/www/auris/envauris/lib/python3.13/site-packages/torch/nn/parallel/comm.py broadcastrs*V $F _ -TU\T]]fgjfk l  189A$Q'9xx""633xx&&v33:sBcUVs/sHn[U5PM nnUVs/sHn[U5PM nn[RR XU5$s snfs snf)aBroadcast a sequence of tensors to the specified GPUs. Small tensors are first coalesced into a buffer to reduce the number of synchronizations. Args: tensors (sequence): tensors to broadcast. Must be on the same device, either CPU or GPU. devices (Iterable[torch.device, str or int]): an iterable of GPU devices, among which to broadcast. buffer_size (int): maximum size of the buffer used for coalescing Returns: A tuple containing copies of :attr:`tensor`, placed on :attr:`devices`. )rrr r_broadcast_coalesced)tensorsr buffer_sizerts rbroadcast_coalescedr1sV.55W #WG5+237aq!7G3 88 ( (; GG63s AAc [USS9nUSR5nSn[U5HupEURRS:wdS5eUR 5U:XaUnUR5U:wdMRSR SUR555nSR S U55n[S US US U35e Uc [S 5e[U5S:XaUS$[R"U5(a/[R"X5n[R"XUS9 U$[R"XRRU5n [U5VV s/sHupJXC:wdM U PM n nn XU SRU SS9-nU SSH"n UR!U RU SS95 M$ U$s sn nf)aSum tensors from multiple GPUs. All inputs should have matching shapes, dtype, and layout. The output tensor will be of the same shape, dtype, and layout. Args: inputs (Iterable[Tensor]): an iterable of tensors to add. destination (int, optional): a device on which the output will be placed (default: current device). Returns: A tensor containing an elementwise sum of all inputs, placed on the :attr:`destination` device. T)optionalrNcpuz+reduce_add expects all inputs to be on GPUsxc38# UHn[U5v M g7fNstr.0rs r reduce_add..\s6:a3q66:c38# UHn[U5v M g7fr!r"r$s rr&r']s; 1A r(zinput z has invalid size: got z, but expected zLreduce_add expects destination to be on the same GPU with one of the tensors)outputroot)device non_blocking)rsize enumerater-type get_devicejoin ValueErrorr lenr is_availabler empty_likereducetoadd_) inputs destination input_size root_indexiinpgotexpectedresultdestination_devicernonrootothers r reduce_addrGEs$K$?K!JJF#zz%'V)VV' >> { *J 88: #((6388:66Cxx; ;;H23%xjQ $ Z   6{aay   !!&"45 F ; M#\\&*<*C*C*H*H+V!*6!2F!2ao1!2F#gajmm%D'4'  QR[E KK(:N O! MGs 8 GGcUVs/sHn/PM nn/n/n[U6Hn[SU55(a2[Xq5nURU5 URUS5 ML[XG5H7upU RU R(aU R 5OU 5 M9 URUSS5 M UV s/sHn [ X5PM n n [U 6HZn U Vs/sHn[U5PM nn[X5n[UU S5Hn URU R5 M M\ [[XV55$s snfs sn fs snf)a,Sum tensors from multiple GPUs. Small tensors are first coalesced into a buffer to reduce the number of synchronizations. Args: inputs (Iterable[Iterable[Tensor]]): iterable of iterables that contain tensors from a single device. destination (int, optional): a device on which the output will be placed (default: current device). buffer_size (int): maximum size of the buffer used for coalescing Returns: A tuple of tensors containing an elementwise sum of each group of inputs, placed on the ``destination`` device. c38# UHoRv M g7fr!) is_sparse)r%rs rr&'reduce_add_coalesced..s3Nq{{Nr(r) zipallrGappendrJto_denserrrdatatupler)r;r<r_ dense_tensorsr+ ref_ordertensor_at_gpusrCcollrritrschunkschunk flat_tensors flat_results rreduce_add_coalescedr]xsQ&.4 4VVM 4 FIv, 3N3 3 3   ]1-b1 2'@M M}GM' /}D Mt*7= 7=e "5 )v  !; )+vayAA MM!&& ! B  $V7 883!5 N s E!E&6E+c H[U5nUcEUVs/sHn[U5PM nn[[RR XX#U55$Ub[ SU35eUb[ SU35e[[RRXX455$s snf)aScatters tensor across multiple GPUs. Args: tensor (Tensor): tensor to scatter. Can be on CPU or GPU. devices (Iterable[torch.device, str or int], optional): an iterable of GPU devices, among which to scatter. chunk_sizes (Iterable[int], optional): sizes of chunks to be placed on each device. It should match :attr:`devices` in length and sums to ``tensor.size(dim)``. If not specified, :attr:`tensor` will be divided into equal chunks. dim (int, optional): A dimension along which to chunk :attr:`tensor`. Default: ``0``. streams (Iterable[torch.cuda.Stream], optional): an iterable of Streams, among which to execute the scatter. If not specified, the default stream will be utilized. out (Sequence[Tensor], optional, keyword-only): the GPU tensors to store output results. Sizes of these tensors must match that of :attr:`tensor`, except for :attr:`dim`, where the total size must sum to ``tensor.size(dim)``. .. note:: Exactly one of :attr:`devices` and :attr:`out` must be specified. When :attr:`out` is specified, :attr:`chunk_sizes` must not be specified and will be inferred from sizes of :attr:`out`. Returns: - If :attr:`devices` is specified, a tuple containing chunks of :attr:`tensor`, placed on :attr:`devices`. - If :attr:`out` is specified, a tuple containing :attr:`out` tensors, each containing a chunk of :attr:`tensor`. zI'devices' must not be specified when 'out' is specified, but got devices=zQ'chunk_sizes' must not be specified when 'out' is specified, but got chunk_sizes=)rrrRr r_scatterr _scatter_out)rr chunk_sizesdimstreamsr rs rscatterrdsDV $F {189A$Q'9UXX&&v 'RSS  [\c[de   "cdocpq UXX**6EFF:sBc@UVs/sHn[U5PM nnUcKUS:Xa[R"S[SS9 [ USSS9n[ R RXU5$Ub[SU35e[ R RXU5$s snf)a^Gathers tensors from multiple GPU devices. Args: tensors (Iterable[Tensor]): an iterable of tensors to gather. Tensor sizes in all dimensions other than :attr:`dim` have to match. dim (int, optional): a dimension along which the tensors will be concatenated. Default: ``0``. destination (torch.device, str, or int, optional): the output device. Can be CPU or CUDA. Default: the current CUDA device. out (Tensor, optional, keyword-only): the tensor to store gather result. Its sizes must match those of :attr:`tensors`, except for :attr:`dim`, where the size must equal ``sum(tensor.size(dim) for tensor in tensors)``. Can be on CPU or CUDA. .. note:: :attr:`destination` must not be specified when :attr:`out` is specified. Returns: - If :attr:`destination` is specified, a tensor located on :attr:`destination` device, that is a result of concatenating :attr:`tensors` along :attr:`dim`. - If :attr:`out` is specified, the :attr:`out` tensor, now containing results of concatenating :attr:`tensors` along :attr:`dim`. rLzjUsing -1 to represent CPU tensor is deprecated. Please use a device object or string instead, e.g., "cpu".) stacklevelT) allow_cpurzQ'destination' must not be specified when 'out' is specified, but got destination=) rwarningswarn FutureWarningrr r_gatherr _gather_out)rrbr<r rs rgatherrns4,337aq!7G3 { "  MM@   ( tdS xxk::  "cdocpq xx##G#66!4sBr!))Nro)NNrN)rN)rir torch._utilsrrrrrr torch.cudar rrrGr]rdrnrrtsU 444BH(0f,9^/GPT/Gd*7D*7rs