JTh-SSKrSSKrSSKJr SSKJr SSKJrJrJ r J r J r SSK r SSK JrJrJrJr SSKJr SSKJr SSKJr SS KJrJr S S /rS \\ \\ R:4S S4Sjr\ "S\S9r"SS \\\5r SS\S\S \ \\ \\ R:4S\ \ \\ R:4S\S\ \S \ RB4Sjjr"g)N)Sequence)chain)AnyGenericOptionalTypeVarUnion)_get_all_device_indices_get_available_device_type_get_device_index_get_devices_properties)Module)parallel_apply) replicate)gatherscatter_kwargs DataParallel data_parallel device_idsreturnc^^^SmTVs/sHn[US5PM snm[T5mUUU4SjnU"S5(agU"S5(aggs snf)Na There is an imbalance between your GPUs. You may want to exclude GPU {} which has less than 75% of the memory or cores of GPU {}. You can do so by setting the device_ids argument to DataParallel, or by setting the CUDA_VISIBLE_DEVICES environment variable.TcL>TVs/sH o"U5PM nn[[U5[R"S5S9up4[ [U5[R"S5S9upVXF- S:a-[ R "T RTUTU55 ggs snf)N)keyg?TF)min enumerateoperator itemgettermaxwarningswarnformat) get_proppropsvaluesmin_posmin_valmax_posmax_val dev_propsrimbalance_warns W/var/www/auris/envauris/lib/python3.13/site-packages/torch/nn/parallel/data_parallel.pywarn_imbalance&_check_balance..warn_imbalance!s/89ye(5/y9y0h6I6I!6LMy0h6I6I!6LM  t # MM%%j&9:g;NO :sB!cUR$N) total_memoryr$s r, _check_balance..,s E$6$6cUR$r0)multi_processor_countr2s r,r3r4.s E$?$?r5)r r )rxr-r*r+s` @@r,_check_balancer9sdN 7AAj#At,jAJ' 3I 677?@@A!BsAT)boundc ^\rSrSrSrSS\S\\\\ \ R4S\\\ \ R4S\ SS4 U4S jjjr S \ S \ S\ 4S jrS\S\\\ \ R4S\\4S jrS \\ S4S \\\\ 4S\\\ \ R4S\ 4SjrS\\S \\ S \ S\\ 4SjrS\ S\\ \ R4S\ 4SjrSrU=r$)r5aImplements data parallelism at the module level. This container parallelizes the application of the given :attr:`module` by splitting the input across the specified devices by chunking in the batch dimension (other objects will be copied once per device). In the forward pass, the module is replicated on each device, and each replica handles a portion of the input. During the backwards pass, gradients from each replica are summed into the original module. The batch size should be larger than the number of GPUs used. .. warning:: It is recommended to use :class:`~torch.nn.parallel.DistributedDataParallel`, instead of this class, to do multi-GPU training, even if there is only a single node. See: :ref:`cuda-nn-ddp-instead` and :ref:`ddp`. Arbitrary positional and keyword inputs are allowed to be passed into DataParallel but some types are specially handled. tensors will be **scattered** on dim specified (default 0). tuple, list and dict types will be shallow copied. The other types will be shared among different threads and can be corrupted if written to in the model's forward pass. The parallelized :attr:`module` must have its parameters and buffers on ``device_ids[0]`` before running this :class:`~torch.nn.DataParallel` module. .. warning:: In each forward, :attr:`module` is **replicated** on each device, so any updates to the running module in ``forward`` will be lost. For example, if :attr:`module` has a counter attribute that is incremented in each ``forward``, it will always stay at the initial value because the update is done on the replicas which are destroyed after ``forward``. However, :class:`~torch.nn.DataParallel` guarantees that the replica on ``device[0]`` will have its parameters and buffers sharing storage with the base parallelized :attr:`module`. So **in-place** updates to the parameters or buffers on ``device[0]`` will be recorded. E.g., :class:`~torch.nn.BatchNorm2d` and :func:`~torch.nn.utils.spectral_norm` rely on this behavior to update the buffers. .. warning:: Forward and backward hooks defined on :attr:`module` and its submodules will be invoked ``len(device_ids)`` times, each with inputs located on a particular device. Particularly, the hooks are only guaranteed to be executed in correct order with respect to operations on corresponding devices. For example, it is not guaranteed that hooks set via :meth:`~torch.nn.Module.register_forward_pre_hook` be executed before `all` ``len(device_ids)`` :meth:`~torch.nn.Module.forward` calls, but that each such hook be executed before the corresponding :meth:`~torch.nn.Module.forward` call of that device. .. warning:: When :attr:`module` returns a scalar (i.e., 0-dimensional tensor) in :func:`forward`, this wrapper will return a vector of length equal to number of devices used in data parallelism, containing the result from each device. .. note:: There is a subtlety in using the ``pack sequence -> recurrent network -> unpack sequence`` pattern in a :class:`~torch.nn.Module` wrapped in :class:`~torch.nn.DataParallel`. See :ref:`pack-rnn-unpack-with-data-parallelism` section in FAQ for details. Args: module (Module): module to be parallelized device_ids (list of int or torch.device): CUDA devices (default: all devices) output_device (int or torch.device): device location of output (default: device_ids[0]) Attributes: module (Module): the module to be parallelized Example:: >>> # xdoctest: +SKIP >>> net = torch.nn.DataParallel(model, device_ids=[0, 1, 2]) >>> output = net(input_var) # input_var can be on any device, including CPU Nmoduler output_devicedimrct>[TU]5 [RR S5 [ 5nUbUS:XaXl/UlgUc [5nUc [S5eUcUSnX@l XlUVs/sHn[US5PM snUl[US5Ul [R"XPRS5UlUS:Xa[UR5 [!UR5S:Xa&UR R#UR5 ggs snf)Nztorch.nn.parallel.DataParallelmpsno available devices were foundrTcudar)super__init__torch_C_log_api_usage_oncer r>rr RuntimeErrorr@r r?devicesrc_device_objr9lento)selfr>rr?r@ device_typer8 __class__s r,rFDataParallel.__init__s   $$%EF02  +"6 K DO   02J  @A A  &qMM ?IJz!,Q5zJ.}dC#ll;8JK & 4?? + t 1 $ KKNN4.. / %KsD5inputskwargsc z[RRRS5 UR(dUR "U0UD6sSSS5 $[ UR R5UR R55HBnURUR:wdM[SURSUR35e URXUR5upU(d U(dSn04n[UR5S:Xa!UR "US0USD6sSSS5 $URUR URS[U55nURXQU5nUR!X`R"5sSSS5 $!,(df  g=f)NzDataParallel.forward6module must have its parameters and buffers on device 2 (device_ids[0]) but found one of them on device: rr)rGautogradprofilerrecord_functionrr>r parametersbuffersrKrLrJscatterrMrrrr?)rOrSrTt module_kwargsreplicasoutputss r,forwardDataParallel.forwardsc ^^ $ $ 4 45K L??{{F5f5M L4;;113T[[5H5H5JK88t222&%%)%8%8$9:++,88*6L%)LL$Q !F-!# 4??#q({{F1IBq1AB)M L*~~dkk4??=S[3QRH))(MJG;;w(:(:;/M L Ls #F,AF,2BF,AF,, F:cJ[X[R"5(+5$r0)rrGis_grad_enabled)rOr>rs r,rDataParallel.replicates1F1F1H-HIIr5.c*[XX0RS9$N)r@)rr@)rOrSrTrs r,r_DataParallel.scatters fjhhGGr5rbc F[XX0RS[U55$r0)rrrM)rOrbrSrTs r,rDataParallel.parallel_applys% foooH &F  r5rcc*[XURS9$rj)rr@)rOrcr?s r,rDataParallel.gathersg$((;;r5)rr@r>r?rL)NNr)__name__ __module__ __qualname____firstlineno____doc__r:rrr intrGrKrFrrdlistrtupledictstrr_rr__static_attributes__ __classcell__)rQs@r,rr5sMhDH<@ "0"0XeC,=&>?@"0 c5<<&7 89 "0  "0  "0"0HrSr?r@rac[U[5(dUbU4OSn[5nUc [S5eUc [ 5nUc [S5eUcUSnUVs/sHn[ US5PM nn[ US5n[ R"XbS5n[UR5UR55H.n U RU:wdM[SUSU R35e [XX$5upU(d U(dS n04nUce[U5S :XaU"US0USD6$US[U5n [X 5n [XXZ5n [XU5$s snf) a$Evaluate module(input) in parallel across the GPUs given in device_ids. This is the functional version of the DataParallel module. Args: module (Module): the module to evaluate in parallel inputs (Tensor): inputs to the module device_ids (list of int or torch.device): GPU ids on which to replicate module output_device (list of int or torch.device): GPU location of the output Use -1 to indicate the CPU. (default: device_ids[0]) Returns: a Tensor containing the result of module(input) located on output_device NrYz#device type could not be determinedrCrTrVrWrXr) isinstancerwr rJr r rGrKrr]r^rrMrrr) r>rSrr?r@rarPr8rLr`used_device_idsrbrcs r,rrs, fe $ $$0&b,.K@AA,. <=="1 6@Aj#At,jJA%mT:M\\+!}=N 6$$&(8 9 88~ %+,-##$88*. :+6*RF -  $$ $ :!vay5M!$455 3v;/O1HX}NG '# ..5BsE()NNrN)#rr collections.abcr itertoolsrtypingrrrrr rG torch._utilsr r r r torch.nn.modulesr torch.nn.parallel.parallel_applyrtorch.nn.parallel.replicater torch.nn.parallel.scatter_gatherrr__all__rurKr9r:rTensorrrYr5r,rs$99  $;1C ? +xc5<<.?(@Ad4 Cve<671:e