[Thl0$%SSKJr SSKJr SSKrSSKJr SSKJ r SSKJ r SSK J r SSK Jr SSKJr SS KJrJr SS \R(S \S \ 4S jjrSS\RR,S\S \4SjjrSq\\R4\S'\S5rSrS\RR,S\S \RR,4SjrS\RR,S \RR,4SjrSS\ R,S\4Sjjr g))contextmanager)OptionalN)distributed_c10d) ShardedTensor)Sharder) ShardingPlan)ChunkShardingSpec ShardingSpectensor sharding_specreturnc UR5(d [S5eUbUO[R"5n[R "U5n[R "U5nS/U-n[R"XrU4US9 [U5HIupX)S:wa[SUSUSU SSU35eXS:wdM1[S USUS U SSU35e URXUS 9n U $) a Given a :class:`torch.Tensor`, it shards that tensor according to the provided ``sharding_spec``. ``src_rank`` denotes the source rank which would be used as the ground truth of the data which would be scattered as shards across the rest of the ranks. Args: tensor (:class:`torch.Tensor`): Tensor needs to be sharded. sharding_spec (:class:`torch.distributed._shard.sharding_spec.ShardingSpec`): The specification describing how to shard the Tensor. Keyword args: src_rank (int, optional): The source rank which is used as the ground truth of the data for the parameter that would be sharded and scattered across the rest of the ranks. Default: 0. process_group (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. Returns: A :class:`ShardedTensor` sharded from the given tensor. .. warning:: Only :class:`torch.distributed._shard.sharding_spec.ChunkShardingSpec` is currently supported as the ``sharding_spec``. z'input tensor is not a contiguous TensorN)grouprz src_rank=z on rank: z does not match with src_rank=rzsharding_spec=z# does not match with sharding_spec=src_rank process_group) is_contiguous ValueErrorr_get_default_groupdistget_world_sizeget_rankall_gather_object enumerateshard) r r rrpg world_size current_rank gathered_listidxentrysts T/var/www/auris/envauris/lib/python3.13/site-packages/torch/distributed/_shard/api.py _shard_tensorr%s,:    ! !BCC  $   0 0 2 $$R(J==$LFZ'M=]*C2N .  Qx H:Z ~>'',Qxj 3%A  !H $ z,H,,1!H:ZuF /   Vb  IB Imodule param_namec [X5(d[UR5SUS35e[X5n[ U[ R 5(d:[S[U5RSUS[U5R35eUR5(d[SUS35e[XRX45nURU[R"U55 g) a Given a :class:`torch.nn.Module`, a ``param_name`` for a parameter in that module, it shards that parameter according to the provided ``sharding_spec``. ``src_rank`` denotes the source rank which would be used as the ground truth of the data which would be scattered as shards across the rest of the ranks. This method replaces ``module.param_name`` with a :class:`torch.distributed._sharded_tensor.ShardedTensor` Args: module (:class:`torch.nn.Module`): Module whose parameter needs to be sharded. param_name (str): Name of the parameter of ``module`` that needs to be sharded. sharding_spec (:class:`torch.distributed._shard.sharding_spec.ShardingSpec`): The specification describing how to shard the Tensor. Keyword args: src_rank (int, optional): The source rank which is used as the ground truth of the data for the parameter that would be sharded and scattered across the rest of the ranks. Default: 0. process_group (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. .. warning:: Only :class:`torch.distributed._shard.sharding_spec.ChunkShardingSpec` is currently supported as the ``sharding_spec``. z has no attribute ``z Expected .z to be a Tensor, but found zparam: z is not a contiguous TensorN)hasattrAttributeError _get_namegetattr isinstancetorchTensorrtype__name__rr%register_parameternn Parameter)r'r(r rrr r#s r$shard_parameterr8MsH 6 & & 0 0 233FzlRSTUU V (F fell + +V --.a |;VW[\bWcWlWlVm n      ! !7:,.IJKK vh FB j",,r*:;r&_CURRENT_PROCESS_GROUPc#P# [b [S5eUqUv Sqg!Sqf=f7f)zN Context manager to set the process group with which to load a ShardedTensor. NzNProcessGroup already set by previous "load_with_process_group" context manager)r9 RuntimeError)rs r$load_with_process_groupr<s> )   +&!%s&&#&cF[c[R"5$[$)zx Retrieves the current process group set by ``load_with_process_group``. If not set, it just returns the default group. )r9rrr&r$_get_current_process_groupr?s %2244%%r&resharding_specc6^U4SjnURU5 U$)a Hook a module with output resharding in the forward pass according to the given ``resharding_spec``. Args: module (:class:`torch.nn.Module`): Module whose output needs to be resharded. resharding_spec (:class:`torch.distributed._shard.sharding_spec.ShardingSpec`): The specification describing how the output of the module will be resharded. Returns: A :class:`torch.nn.Module` object with reshard API hooked. cT>[U[5(aURT5$U$)N)r0rreshard)_module_inputoutputr@s r$ hook_func"_reshard_output..hook_funcs$ fm , ,>>/2 2 r&register_forward_hook)r'r@rGs ` r$_reshard_outputrKs     + Mr&c.SnURU5 U$)a Hook a module with local shards collection in the forward pass. This API is typically used to convert a sharded representation back to data parallel representation. In particular, it returns the local tensor for this Shard. If the size along the sharding dimension for the local tensor is 1, this dimension is removed from the final result. For example a [4, 16] ShardedTensor across 4 ranks is typically a local Tensor of size [16] across each rank and not [1, 16] across each rank. Args: module (:class:`torch.nn.Module`): Module whose output is ShardedTensor and the local tensor value needs to be returned. Returns: A :class:`torch.nn.Module` object with collection API hooked. c[U[5(awUR5nURn[U[5(aDUR UR 5S:Xa%URURR 5nU$g)Nr)r0r local_tensor_sharding_specr sizedimsqueeze)rDrErFrNr s r$rG'_collect_local_shard..hook_funcs{ fm , ,!..0L"11M=*;<< %%m&7&78A=+33))--    -r&rI)r'rGs r$_collect_local_shardrTs$     + Mr&planc 6/nURR5H-upV[U[5(dMUR U5 M/ URR5HupV[U[ 5(acUR S5upxn UH+n URU 5(dM[SUSU S35e URU5n [XXbUS9 M}[U[5(a`UR S5upn US:Xa [S5eURU5n URU 5nURU 5nXl M[SUS 35e URbeURR5HGunn[U[ 5(aURU5n [!U U5 M:[S US 35e UR"b0UR"HnURU5n [%U 5 M! g g ) a Shards a given module according to the provided sharding `plan`. This method first shards all the parameters according to the given sharding `plan`. Then if `output_plan` and `return_local_tensor` are specified in the sharding `plan`, it will tag the output of modules according `output_plan`, convert the module's output back to data parallel according to `return_local_tensor`. Needs to be called on all ranks in an SPMD fashion. Args: module (:class:`torch.nn.Module`): The module to apply sharding to plan (:class:`torch.distributed._shard.sharding_plan.ShardingPlan`): The ShardingPlan which specified param name to ShardingSpec to apply to each parameter. Keyword args: src_rank (int, optional): The source rank which is used as the ground truth of the data for the module that would be sharded and scattered across the rest of the ranks. Default: 0. process_group (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. r+z7ShardingPlan is in-valid, trying to shard a parameter: z1, but there's already a Sharder entry for module z`, parameter sharding should not conflict with the submodule tree that a Sharder is working with!rz1Module path must not be empty for custom sharder!z:Only `ShardingSpec` and `Sharder` are supported to shard ''Nz5Only `ShardingSpec` is supported as output_plan for ')rUitemsr0rappendr rpartition startswithr; get_submoduler8KeyErrorrmod_name TypeError output_planrKreturn_local_tensorrT)r'rUrr sharder_pathsnamespec module_path_r( sharder_pathmodparent_mod_path _mod_name parent_mod sharded_mod output_specs r$ shard_moduleros 4Miioo'  dG $ $   &( iioo'  dL ) ))-)= &KJ - )),77&QRVQWXKKW.Y;<!.&&{3C   g & &,0OOC,@ )O rzRSS&&t,C--o>J**S/K"- LTFRST 9(B #(,(8(8(>(>(@ $K+|44**;7[1KK=XYZ )A +33K&&{3C  %4,r&)rN)! contextlibrtypingrr1torch.distributed distributedrtorch.nnr6r'torch.distributed._shard.sharded_tensorrsharderr sharding_planr r r r r2r%Modulestrr8r9 ProcessGroup__annotations__r<r?rKrTror>r&r$r|s% .A':RV: LL:)5::B 3< HHOO3<3< 3