[Th>> # xdoctest: +SKIP >>> sum_reducer = _CustomReducer( >>> torch.tensor(0.0), >>> lambda a, b: a + b >>> ) cXlX lgN init_value reduce_fn)selfrrs _/var/www/auris/envauris/lib/python3.13/site-packages/torch/distributed/pipelining/microbatch.py__init___CustomReducer.__init__(s $"rN)__name__ __module__ __qualname____firstlineno____doc__r__static_attributes__rrr r s  #rr c\rSrSrSrg) _LossReducer-rNrrrrrrrrr!r!-rr!gc X-$rr)abs rr(1s15rcz\rSrSr%SrSr\\S'SrSr \ S\ \S44S j5r \ S\ \\44S j5rS rg ) r 8z* Class used to specify chunking of inputs cXlgr split_dim)rr-s rrTensorChunkSpec.__init__=s"rr-c|URRSURRSURS3$)N.()) __class__rrr-rs r__repr__TensorChunkSpec.__repr__Bs9~~(()4>>+B+B*C1T^^DTTU V rc"SURS3$)NzTensorChunkSpec(r2r,r4s r__str__TensorChunkSpec.__str__Gs!$..!133r chunk_dims.c [US5nU$)aJ A helper for creating a tuple of `TensorChunkSpec` from a tuple of chunk dimensions (int's). Example: >>> # xdoctest: +SKIP >>> # There are three positional arguments to the model, and >>> # we are chunking them along dimension 0, 0 and 1, respectively >>> args_chunk_spec = TensorChunkSpec.from_tuple((0, 0, 1)) c[U5$rr dims rr(,TensorChunkSpec.from_tuple..Y ,rr)r:args_chunk_specs r from_tupleTensorChunkSpec.from_tupleJs(  , rc [US5nU$)a$ A helper for creating a dictionary of `TensorChunkSpec` from a dictionary of chunk dimensions (int's). Example: >>> # xdoctest: +SKIP >>> # Chunk dimension 0 for the "id" argument, 1 for the "mask" argument >>> kwargs_chunk_spec = TensorChunkSpec.from_dict({"id": 0, "mask": 1}) c[U5$rr=r>s rr(+TensorChunkSpec.from_dict..krArr)r:kwargs_chunk_specs r from_dictTensorChunkSpec.from_dict]s*  , ! rr,N)rrrrrrint__annotations__r5r8 staticmethodtuplerCdictstrrIrrrrr r 8se#N 4#s(O$!cN!!rr c\rSrSrSrg) _ReplicateqrNr#rrrrRrRqr$rrRc L0n/nUnSn[U5[U5:Xd;S[UR55S[UR5535eUR5GHeupx[ U5upUR U 5 Xn U ce[ U 5up[U 5[U 5:wa[ SUSU 35e/n[X5GHunnU[Ld[U[R5(dUR U/U-5 MF[U[5(Ga[U[R5(d US35eURUR5nUU:a>U(a"[R!SUS US US 35 UnO[#S US USUS35e[R$"XUR5n[&(a/nSnUHn[R("U5nUURUR5-n[+SSS5/UR,-n[+UU5UUR'UUU'UR U5 UURUR5- nM UR U5 OUR U5 SnGM[/SU35e XU'GMh /n[1U5HJn0nUR5H unnUVs/sHnUUPM nnUUU'M" UR U5 ML /nUHfn 0n![U5[U 5:Xde[U R5U5Huunnn"[3UU"5U!U'M UR U!5 Mh U$s snf)a3 Given a dictionary of args, and a dictionary of chunking specs, shard the args according to the chunking specs. Args: args_dict: Dictionary of args args_chunk_spec: Dictionary of chunking specs num_chunks: Number of chunks to shard the args into Returns: args_split: List of sharded args Tzargs_dict.keys() = z args_chunk_spec.keys() = NzArgument value z9 did not have the same number of values as as chunk spec z is not a tensorz%Tensor size on chunking dimension is z', downsizing the number of chunks from z to r0zArg z% on chunking dimension has a size of z$, smaller than the number of chunks z. PiPPy cannot reduce the number of chunks because other arguments have bigger chunk-dimension sizes. Please adjust your num_chunks setting.rFzUnrecognized chunk spec: )lenlistkeysitemsrappend ValueErrorziprR isinstancetorchTensorr sizer-loggerwarning RuntimeError tensor_split_debug_mask_minibatches zeros_likeslicendim TypeErrorranger)# args_dictrB num_chunksargs_sharded_replicated arg_specsreal_num_chunks first_tensorarg_keyargflatspec chunk_specchunk_spec_flat_sharded_arg_flatvchunk_vv_split_dim_size chunk_tensorsexpanded_chunks split_dim_idx chunk_tensornew_val upper_idx slice_indices chunks_flat chunk_idx chunk_argskeyv_flatarg_single_chunk args_splitchunkper_chunk_argsarg_specs# r_shard_dict_of_argsrus(!I OL y>S1 1 d9>>#3455OPTUdUiUiUkPlOmn 1") !#& $- %%%)*5 t9O, ,!#'++5,8  d4JAw*$Jq%,,,G,G ''o(=>G_55"!U\\22Jqc9I4JJ2#$66'*;*;#< #o5#CDTCUVDDN .'..w7%):):7;L;L)MM )6%++O<$++M:$ ";G9 EFFq5t,<(S*XK?+  /557HC@CDfy 1 D.JsO8 :& ,J9~U+++$' y$A JS#"0h"?N3 %B.)   EsN!args.kwargschunksrBrHreturnc ^Uc0nUc[[54[U5-nUc#[R U[[55n[ [[ U55[[ U55U5n[U5n[ UUU5n[U5U:a<[U5n[ [[ U55[[ U55U5n[U5[U5:wa#[S[U5S[U535eUV^s/sH*m[U4Sj[[T5555PM, n nX4$s snf)a Given a sequence of args and kwargs, split them into a number of chunks according to their respective chunking specs. Args: args: Tuple of args kwargs: Dict of kwargs chunks: Number of chunks to split the args and kwargs into args_chunk_spec: chunking specs for args, in same shape as args kwargs_chunk_spec: chunking specs for kwargs, in same shape as kwargs Returns: args_split: List of sharded args kwargs_split: List of sharded kwargs z;args and kwargs are split into different number of chunks: z, c3.># UH nTUv M g7frr).0irs r 0split_args_kwargs_into_chunks..Vs <%;jm%;s) r DEFAULT_CHUNK_DIMrUrOfromkeysr enumeraterbrNri) rrrrBrHargs_split_dictrn kwargs_splitrrs ` rr r sXp~*+<=?#d)K  MM&/BS2TU) Yt_ Y '(O /*O&L  <?*l+- 4 ! ?+ ,    ?s<00 I?#$Bs<'8&9 ;  *)J  )rr rrUrZrYrr\rirdshaper]rcemptyr-r[r_rfrgcatr rrr)rrtspec_flattened flatten_spec chunk0_flatchunks_flattenedrchunk_flattenedrvargs_flattenedarg_idxrqrpartial_values overall_shapeval meta_chunks values_to_catchunk_start_idx partial_value meta_chunk chunk_end_idxrsliced reduced_valvalues rr r ]sZ'3J'?$ %1$;! )*;<=K@PP)%0  3~#6 6veW,FzlST T0 N!.1  c? + +"'s+;'v> 0  !# "#>*c+.>>>>14^[1Q-M:$3joocmm6T$TM%*4t%<$= @R@R$RM38-3XM#--0*=9F!((0&3O2R!/  ! !%))Ms}}"M N ^ , ,..K"3'7#89 !mm!1!rs4 '<    8 $  ##$ > 5<<,.@A 5!5!r  |F>B>B f$ S/f$ T#s(^ $f$ f$eOS$89: f$  S/%9 :; f$  4;T "# f$Rw8 Iw8r