eTh zSSKJr SSKrSSKJrJr SSKJrJrJ r J r SSK r SSK J r SSK JrJr \ R /r\R$"\5r\ R*R-5r\"S5(a\(a SS KJrJrJrJr S(S jrS)S jr\ R>\ R@\ RB\ RD\ RF\ RH\ RJ\ RL\ RN\ RP\ RRS . r*S r+S*S+Sjjr,Sr-S,S-Sjjr."SS5r/"SS\/5r0"SS\/5r1"SS\/5r2"SS\25r3"SS\/5r4"SS\45r5"SS \/5r61S!kr7\S.S"j5r8S/S#jr9S0S$jr:S%r;S&rF""3'J!!#J(JqIKNL! %3'q,,% 4l6IJJ" """$Ki . ax+, SBY>3./ SBY^+,+C50RSTT 99' 4 55r*cUS:wa [S5eUS:aUO XR-nURUnXS-nXb-nURSUnURUS-Sn UR"/UQUPUPUPU Q76n [ U5n [ U5S-n [ [ U R55n XXsX'X'U R"U 6nURU5nU$)a? Reorders a tensor that was reconstructed from sharded packed weights into its canonical packed format. For example, if a weight was packed (e.g., gate_proj and up_proj) and then sharded, DTensor.full_tensor() might produce an interleaved layout like [G0, U0, G1, U1, ...] along the sharded dimension. This function reorders it to [G0, G1, ..., U0, U1, ...]. This is an inverse operation to get_packed_weights. Args: reconstructed_tensor: The tensor reconstructed from DTensor (e.g., via .full_tensor().contiguous()). sharded_dim: The dimension index in the reconstructed_tensor that was originally sharded. world_size: The tensor parallel world size. num_packed_projs: The number of projections that were packed together (e.g., 2 for gate_up_proj). Returns: The reordered tensor in canonical packed format. r zNum blocks different from 2 is not supported yet. This is most likely a bug in your implementation as we only pack gate and up projections together.rNr") rBndimr;viewlenrr=permute reshape_as)packed_parameter sharded_dimrJ num_blocksactual_sharded_dimtotal_size_on_sharded_dimoriginal_block_size_on_dimshard_chunk_size prefix_shape suffix_shape tensor_view axis_ws_abs axis_npp_abs permute_ordertensor_permutedfinal_ordered_tensors rrepack_weightsrjs:0Q c  )4q(8kLaLa>a 0 6 67I J!:!H1?#))*=+=>L#))*K>Y[h[u;M ;!))=9O+556FG r*cUS:XaBURSnXXRR5--US-XRR5--2S4nU$US:XdUS:XaFURSnUSX5UR5--US-XRR5--2SS24nU$US:XdUS:XaCURSnUSX5UR5--US-XRR5--24nU$[SUS35e) Nrr".r7r r8r9r:)r;r<rB)rDrErFrGrHsize_s rget_tensor_shardrms6 ax!!!$e'7'7'99:dQh5TdTdTfKf=ggillm L SBY!!"%c4K,<,<,>#>?4!8PUYiYiYkPkBllnoop L SBY!!"%c4K,<,<,>#>?4!8PUYiYiYkPkBlllm L+C50RSTTr*c^^^[UR5S:Xa2TbURUU4Sj5 TbURUU4Sj5 U$)z Copy pasted from torch's function but we remove the communications (partitioning) as well as buffer registering that is similarly not efficient. rc>T"XT5$N)modinputsrFinput_fns r#distribute_module..s #WbAcr*c>T"XT5$rprq)rrrsoutputsrF output_fns rrurvsiPS^iFjr*)rX_forward_pre_hooksregister_forward_pre_hookregister_forward_hook)modulerFrtrys ```rdistribute_moduler~sG 6 $ $%*    , ,-c d   ( ()j k Mr*cL\rSrSrSrSr\S5r\S5rSr S Sjr Sr g ) TensorParallelLayer1 General tensor parallel layer for transformers. Tcgrprq input_layoutsdesired_input_layoutsrrrsrFs r_prepare_input_fn%TensorParallelLayer._prepare_input_fns[^r*cgrprqoutput_layoutsuse_local_outputrrrxrFs r_prepare_output_fn&TensorParallelLayer._prepare_output_fn sY\r*c[erp)NotImplementedError)selfrDrE param_typeparam_casting_dtype to_contiguousrGrFs rpartition_tensor$TensorParallelLayer.partition_tensor s!!r*c UR(aa[UU[URURUR 5[UR URUR55 ggrp) use_dtensorr~rrrrrrrrr}rFs rprepare_module_tp%TensorParallelLayer.prepare_module_tpsZ    ..0B0BDD^D^_//1D1DdF[F[\   r*rqNr} nn.Modulereturnr) __name__ __module__ __qualname____firstlineno____doc__r staticmethodrrrr__static_attributes__rqr*rrrs4K^^\\"r*rcf^\rSrSrSrSSSS.S U4Sjjjr\S5r\S5rS r U=r $) GatherParalleliza Simple class used to define the hooks to add to a layer when we just want to gather the outputs NT)rrrc>[TU]5 U=(d [54UlX l[54UlX0lgrp)super__init__rrrrr)rrrr __class__s rrGatherParallel.__init__ s: +:y{<,&/k^" 0r*cjU(a+[US[5(aUSR5nU$Nrrrto_localrs rr GatherParallel._prepare_input_fn-s, jG44AY'')F r*c[RRUS[RRRSS9 U$)NrF)opasync_op)r@ distributed all_reduceReduceOpSUMrs rr!GatherParallel._prepare_output_fn3s; $$WQZE4E4E4N4N4R4R]b$cr*)rrrrrOptional[Placement]rrrbool) rrrrrrrrrr __classcell__rs@rrrsc.2.2!% 1+ 1, 1  1 1 r*rcJ\rSrSrSr\SSj5r\SSj5rS SjrSr g) IsolatedParalleli:z This class is used to isolate computation in a TP layer from the rest of the world. Parameters need to be LOCAL, so not dtensors NcZUSn[U[5(aUR5nU$rrrrrrrsrF input_tensors rr"IsolatedParallel._prepare_input_fn@s.ay lG , ,'002Lr*cU$rprqrs rr#IsolatedParallel._prepare_output_fnHs r*c t[UU[URSS5[URSS55 grp)r~rrrrs rr"IsolatedParallel.prepare_module_tpMs4   D**D$ 7 D++T4 8  r*rqrpr) rrrrrrrrrrrqr*rrr:s4  r*rcn^\rSrSrSrSSSSS.S U4Sjjjr\S5rSr\S 5r S r U=r $) ColwiseParalleliVrNTrrrrc>[TU]5 U=(d [54UlU=(d [ S54Ul[54UlX0lX@lgNr8) rrrrrrrrrrrrrrrs rrColwiseParallel.__init__[sN +:y{<-:r<&/k^" 0&r*cUSn[U[5(d[R"XTUSS9nX:waURUSS9nU$)NrF run_check placementsrrr from_local redistributers rr!ColwiseParallel._prepare_input_fnjsTay ,00"--lbghL  1'44@U`e4fLr*cVUS:Xa[XXvS5n[S5/n O[S5/n [XXvS5nURU5nU(aUR5nUR(a[ R "XU SS9n[R"XR5S9$)Nbiasr8r7Fr requires_grad) rmrr? contiguousrrrr Parameteris_floating_point rrDrErrrrGrF parametershards rr ColwiseParallel.partition_tensorws  ([PRSI2YKE2YKE([PRSILL!45 !,,.I   **95TYZI||I5P5P5RSSr*ctURU:waURUSS9nU(aUR5$U$)NFr)rrrrs rr"ColwiseParallel._prepare_output_fns=    /**nu*UG%5w!B7Br*rrrrrr) rrrrrrrrrrrrrs@rrrVsn.2.2!% '+ ', '  ' '  T$CCr*rc\rSrSrSrSrg)PackedColwiseParallelic[XXvS5nURU5nU(aUR5nUR(a [R "X[ S5/SS9n[R"XR5S9$)Nr7Frr rTr?rrrrrr rr rrDrErrrrGrFrs rr&PackedColwiseParallel.partition_tensorn'u;bQ LL!45 !,,.I   **9E"I;Z_`I||I5P5P5RSSr*rqNrrrrrrrqr*rrr Tr*rcx^\rSrSrSrSSSSS.S U4SjjjrSr\S5r\S 5r S S jr S r U=r $)RowwiseParallelia Partition a compatible nn.Module in a row-wise fashion. Currently supports nn.Linear and nn.Embedding. Users can compose it with ColwiseParallel to achieve the sharding of more complicated modules. (i.e. MLP, Attention) Keyword Args: input_layouts (Placement, optional): The DTensor layout of input tensor for the nn.Module, this is used to annotate the input tensor to become a DTensor. If not specified, we assume the input tensor to be sharded on the last dimension. output_layouts (Placement, optional): The DTensor layout of the output for the nn.Module, this is used to ensure the output of the nn.Module with the user desired layout. If not specified, the output tensor is replicated. use_local_output (bool, optional): Whether to use local :class:`torch.Tensor` instead of :class:`DTensor` for the module output, default: True. Returns: A :class:`ParallelStyle` object that represents Rowwise sharding of the nn.Module. NTrc>[TU]5 U=(d [S54UlU=(d [ 54UlX0lX@lgr)rrrrrrrrrs rrRowwiseParallel.__init__sB +8uRy:-<> 0&r*cDUS:wa[XXvS5n[S5/n O[5/n USSnURU5nU(aUR 5nUR (a[ R"XU SS9n[R"XR5S9$)Nrr8Frr) rmrrr?rrrrr rrrs rr RowwiseParallel.partition_tensors  ([PRSI2YKE[MEaILL!45 !,,.I   **95TYZI||I5P5P5RSSr*c[US5(a%URbURUlSUlUSn[U[5(d[R "XTUSS9nX:waUR USS9nU$)NrrFrTr)hasattrr_biasrrrrrs rr!RowwiseParallel._prepare_input_fnsx 3  CHH$8CICHay ,00"--lbghL  1'44@U`d4eLr*cURU:waURUSS9n[US5(aX2R- nU(aUR 5$U$)NTrr)rrrrrrs rr"RowwiseParallel._prepare_output_fnsV    /**nt*TG 3 yy G%5w!B7Br*c 0SUlUR(a[U[R5(a[ S54UlOl[U[R5(a[54UlO<[U[R5(a[ S54UlO [S5e[UU[URURUR 5[URUR UR"55 gg)NTr8zBRowwiseParallel currently only support nn.Linear and nn.Embedding!)_distribute_module_appliedrrr Linearrr Embeddingrrrr~rrrrrrrs rr!RowwiseParallel.prepare_module_tps,0)   &")),,EJ2YL*FBLL11.7k^*FBLL11.3Bi\*)*noo ..0B0BDD^D^_//1D1DdF[F[\   r*rrr) rrrrrrrrrrrrrrs@rrrsx*.2.2!% '+ ', '  ' 'T$   C Cr*rc\rSrSrSrSrg)PackedRowwiseParallelic[XXvS5nURU5nU(aUR5nUR(a [R "X[ S5/SS9n[R"XR5S9$)Nr8Frrrrs rr&PackedRowwiseParallel.partition_tensorrr*rqNrrqr*rrrrr*rcb^\rSrSrSrSSSS.S U4Sjjjr\S5r\S5rS r S r U=r $) SequenceParallelia SequenceParallel replicates a compatible ``nn.Module`` parameters and runs the sharded computation with input sharded on the sequence dimension. This currently supports ``nn.LayerNorm``, ``nn.Dropout``, and the `RMSNorm python implementation `__ This style implements the operation that is described in the paper `Reducing Activation Recomputation in Large Transformer Models `__ If the input passed in to this ``nn.Module`` is a :class:`torch.Tensor`, it assumes that the input is already sharded on the sequence dimension and converts the input to a :class:`DTensor` sharded on the sequence dimension. If the input passed in to this ``nn.Module`` is already a :class:`DTensor` but is not sharded on the sequence dimension, it would redistribute the input to be sharded on the sequence dimension. The output of the ``nn.Module`` will be sharded on the sequence dimension. Keyword Args: sequence_dim (int, optional): The sequence dimension of the input tensor for the ``nn.Module``, this is used to annotate the input tensor to become a DTensor that is sharded on the sequence dimension, default: 1. use_local_output (bool, optional): Whether to use local :class:`torch.Tensor` instead of :class:`DTensor` for the module output, default: False. Returns: A :class:`ParallelStyle` object that represents Sequence Parallel of the ``nn.Module``. Example:: >>> # xdoctest: +SKIP(failing) >>> from torch.distributed.tensor.parallel import parallelize_module, SequenceParallel >>> from torch.distributed.device_mesh import init_device_mesh >>> ... >>> m = Model(...) # m is a nn.Module that contains a "norm" nn.LayerNorm submodule >>> tp_mesh = init_device_mesh("cuda", (8,)) >>> >>> # By default, the input of the "norm" will be converted to DTensor that shards on the sequence dim >>> # and the output of "norm" will return a sharded on sequence dimension :class:`DTensor`. >>> >>> sharded_mod = parallelize_module(m, tp_mesh, {"norm": SequenceParallel()}), >>> ... .. note:: SequenceParallel style assumes ones initialization if there are weights in the nn.Module (i.e. ``nn.LayerNorm`` or ``RMSNorm``, and they by default have ones initialization). If you have custom inits for the weights on those modules, you need to broadcast the weights before/after parallelizing to ensure that they are replicated. r"F) sequence_dimrrc>[TU]5 [54Ul[ S54Ul[54UlX lSUl[ U54Ul X lg)Nr"T) rrrrrrrrrsequence_sharding)rrrrrs rrSequenceParallel.__init__?sX 'k^&+Ah["({n 0"' "5!7 0r*cUSn[U[5(d[R"XTUSS9nX:waURUSS9nU$)NrFrTrrrs rr"SequenceParallel._prepare_input_fnIsPay ,00"--lbghL  1'44@U`d4eLr*cTUR[54SS9nUR5$)NTr)rrrrs rr#SequenceParallel._prepare_output_fnRs1&&! ~' !!r*cUSnURU5nU(aUR5nUR(a[R"X[ 5/SS9n[ R"XR5S9$)N.Frr) r?rrrrrr rrrs rr!SequenceParallel.partition_tensorYsg#J LL!45 !,,.I   **9IK=\abI||I5P5P5RSSr*)rrrr rr)rintrr) rrrrrrrrrrrrrs@rrrsQ*X/0%]b11"" T Tr*r> localgathercolwiserowwise colwise_rep rowwise_rep local_colwise local_rowwisesequence_parallellocal_packed_rowwisec[U[5(d[S[U5S35eUS:Xa [ 5$US:Xa [ 5$US:Xa[ [ 5S9$US:Xa[ [ 5S9$US :Xa [ S S 9$US :Xa [ S S 9$US :Xa [5$US:Xa [5$US:Xa [S S 9$US:Xa [5$[SU35e)z In model configurations, we use a neutral type (string) to specify parallel styles, here we translate them into torch.distributed tensor-parallel types. z Unsupported parallel style type z, expected strrrr)rr)rrF)rrrrrrz"Unsupported parallel style value: ) rstrrBtyperrrrrrr)styles r!translate_to_torch_parallel_styler ts eS ! !;DK=WXX   )   - ik:: - Y[99 / !511 / !511 ' !! (  ( ($77 % %!!=eWEFFr*cXSU;aURSS5OUupE[X5nU(dU$US;aU$US:Xa [S5/nOJUS:XaUS:Xa [5/nO2[S5/nO%US:XaUS:Xa [S5/nO [S 5/n[R "XWS S 9$) z Converts a local variant of weights to a DTensor with corresponding placements. Shouldn't be done ever except of before saving the model. r!r")rrrrr8rrrr7Fr)r%r)rrrr)rr&rFr'_rtp_stylers rconvert_local_tensor_to_dtensorr$s 69N5JN))#q1P^MA%n>H QQ))Bi[ _ $  #+J)J _ $  )J)J   ijE RRr*cUR5HKup4[U[R5(dM&[U[5(aM=[ XCX!5X'MM U$)z} Replaces all tensors that were sharded with `local_*` strategy with DTensor to make determining their proper size possible. )itemsrr@Tensorrr$) state_dictr'rFkeyvalues r%replace_state_dict_local_with_dtensorr+sL!&&(  eU\\ * *:eW3M3M=e+_JO) r*c Ub[U5nURX5 SU;auUR SS5Sn[ R "SS U5n URU S 5=n (a.[U 5nURU5n URX5 ggg![an[SUSUSU35 SnANSnAff=f) ag Add hooks to the module holding the layer. Meaning: ``` class MyModel(nn.Module): def __init__(self): self.layer = nn.Linear(10, 10) ``` has state_dict like: ``` { "layer.weight": torch.Tensor, "layer.bias": torch.Tensor } ``` we add hooks to `MyModel` as well as `layer` to make sure that the tensors are correctly sharded and gathered. NTrying to prepare 0, but it's not supported. Corresponding module: z Fix it's TP plan: r!r"rrr F) r rrprintr%r#r$get get_submodule) modelr}r' layer_namecurrent_module_planrFtp_layereparent_layer_name generic_name module_plan module_to_tp_s r#add_tensor_parallel_hooks_to_moduler;s&&45HI   & &v ; j&--c15a8vvfc+<= !++lE: :; :8EH!//0ABM  & &} B ;  #  $ZL0`ag`hh{|}{~   sB C'CCc SU;aURSS5OUupURn URU5n [U5n[ X:5n [ U SS5(d[ X XX5 SU lU b![U 5n U RXXXVU5nO+US RU5nU(aUR5n[U[R R"5(d+[R R#XR%5S 9n['XU5 U$![a!n[SUSU S W S U35 SnANSnAff=f) a Main uses cases: - column / rowise parallelism, you just shard all the weights of the layer (weight and bias) - packed layers: you slice the weights, then shard like above - custom operation: - you want to add an all-gather at the end of a local layer. - you want to have a layer that is isolated from the rest of the world (because torch.DTensor does not work well with `.view` for instance) r!r" _is_hookedFTNr-r.z" Fix it's TP plan, current layer: z : .r)r%_tp_planr1rr)getattrr;r=r rrr/r?rrr@r rrsetattr)r2rDrEr&r is_contiguousrGrF param_namerr' module_to_tpr4r5r6s rshard_and_distribute_modulerDsu?B^>S^223:YgJnnG&&z2L t9D0I <u 5 5+EViw"& & 89LMH--J]ZeEc 12 $$&E eUXX// 0 0""58O8O8Q"R Le, L##  $^$44deqdrsUV^U__bcdbef   s:D'' E1E  EcUcgUVs1sHn[R"SSU5iM nn[U5nUnUHnSU;aURSS5OUupg[R"SSU5nX;a$UR U5 UR U5 M`SU;a>URSS5S=o;a$UR U 5 UR U5 MM [ U5S:a[RSU35 [ U5S:a([RSS RU535 ggs snf) zy Verify the TP plan of the model, log a warning if the layers that were not sharded and the rules that were not applied. Nrr r!r"rz>The following TP rules were not applied on any of the layers: z'The following layers were not sharded: z, ) r#r$setr%popdiscardrXloggerwarningjoin) expected_keysr'r) generic_keysunsharded_layers unused_rulesrBr"r(parent_param_names rverify_tp_planrQs> 8EF BFF63, LF<(L.1Sj 3*c VVFC<  (   / 0  $ $S ) & &ASAZAZ[^`aAbcdAe,e,=+q   . /  $ $S )  <1WXdWefg q @K[A\@]^_!'Gs"E)rrrzUnion[int, List[int]]rz List[int])r&rr'dict[str, str]rz Optional[str])r ) r[ torch.Tensorr\rrJrr]rrrS)NNNr)rr)rrSr&rr'rRrr)r(dict[str, torch.Tensor]r'rRrrT)rLz list[str]r'zOptional[dict[str, str]])> __future__rr# functoolsrrtypingrrrr r@r utilsr r LayerNormALL_LAYERNORM_LAYERS get_loggerrrIr is_available_torch_distributed_availabletorch.distributed.tensorrrrrrr)ruint8int8int16rAbfloat16int32float32float64int64 float8_e4m3fnrCrTrjrmr~rrrrrrrrSUPPORTED_TP_STYLESr r$r+r;rDrQrqr*rris # (// 6 ~   H % %00==?U##(DMM&4 " JJ ++ ** ;; == NN ;; == == ;;"" ;6D > "> > >  >  > B "     $8(> * 89C)9Cx TO Ta)aH TO TQT*QTh  G GBSS-0SHVS S: '    $CN0f`r*