[Th>pSrSSKrSSKrSSKJr SSKrSSKJr SSKJ r J r SSK J r SSK Jr SS KJr SS KJrJr SS KJr SS KJr S SKJr S SKJrJr \(aSSKJr "SS\5rSr Sr!"SS\5r""SS\5r#"SS\5r$"SS\5r%"SS\5r&"SS \5r'g)!a Distributed computing variable tracking classes for PyTorch Dynamo. This module implements variable tracking for distributed computing components: - Process Groups (for collective communication) - Device Meshes (for distributed tensor sharding) - Placement Types (for specifying distribution strategies) - Distributed Tensors and their operations - Backward hooks for distributed module operations These classes are responsible for tracking distributed operations during graph compilation while maintaining proper guards and handling distributed-specific behaviors. They ensure correct handling of distributed components like process groups, device meshes, and placement strategies while preserving proper semantics for distributed tensor operations in the compiled code. The implementation provides special handling for distributed package availability checks and proper tracking of distributed state and operations across processes. N) TYPE_CHECKING) BackwardState)compiled_autograd variables) trace_wrapped)unimplemented_v2)%call_module_hooks_from_backward_state) GuardBuilder install_guard) AttrSource)istype)VariableTracker)ConstantVariable EnumVariable)InstructionTranslatorcF^\rSrSrSrSU4SjjrSr\S5rSr U=r $)DistributedVariable-a_ The base distributed variable that encapsulates common methods for the distributed objects (i.e. ProcessGroup, DeviceMesh, etc.). Concrete distributed objects could inherit this class and add object specific logic. i.e. It provides the check on the distributed package existance and hold the tracking value for the corresponding distributed object. c ~>[TU]"S0UD6 [R5(d [ SSSS/S9 Xlg)Nz+torch.distributed package is not available!zOThe PyTorch package doesn't include torch.distributed when builing from source.zESet USE_DISTRIBUTED=1 to enable it when building PyTorch from source.gb_typecontext explanationhints)super__init__r is_availabler value)selfr"kwargs __class__s [/var/www/auris/envauris/lib/python3.13/site-packages/torch/_dynamo/variables/distributed.pyr DistributedVariable.__init__8sB "6""//11 Em[   c,[UR5$N)typer"r#s r& python_typeDistributedVariable.python_typeEsDJJr(c>[RR5$r*)torch distributedr!rr(r&r! DistributedVariable.is_availableHs  --//r(r")returnN) __name__ __module__ __qualname____firstlineno____doc__r r- staticmethodr!__static_attributes__ __classcell__r%s@r&rr-s&  00r(rc[R5(dgSSKJn [R "U5=(a XR L$)NFr)DTensor)rr!torch.distributed.tensorr?inspect isfunction from_local)r"r?s r& is_from_localrDNs6  + + - -0   e $ D2D2D)DDr(c[R5(dgSSKJnJnJnJnJn UUUUU/n[R"U5=(a X;$)NFr)_get_group_size_by_name_get_group_tag_rank_not_in_group$_resolve_group_name_by_ranks_and_tagget_process_group_ranks) rr!"torch.distributed.distributed_c10drFrGrHrIrJrArB)r"rFrGrHrIrJconstant_processgroup_functionss r&is_constant_pg_functionsrMVsR  + + - - , '#   e $ Q)QQr(cL^\rSrSrSr\S5rSSS\S\4U4Sjjr S r U=r $) WorldMetaClassVariablemzt Tracks torch.distributed.GroupMember and torch.distributed.group, which are instances of the metaclass _WorldMeta. cTUR5(dgSSKJn [U5UL$)NFr) _WorldMeta)r!rKrRr+)clsr"rRs r&is_group_member_type+WorldMetaClassVariable.is_group_member_typess&!!AE{j((r(txrnamer4c>US:Xa[[URSS9n[UR[R 55 [ URR5$US:Xa[[URSS9n[UR[R 55 [URR5$[TU]1X5$)NWORLD)basememberNON_GROUP_MEMBER) r sourcer make_guardr ID_MATCHProcessGroupVariabler"rYrr\r var_getattr)r#rVrWr]r%s r&ra"WorldMetaClassVariable.var_getattr|s 7?T[[AF &++L,A,AB C' (8(89 9 ' 'T[[9KLF &++L,A,AB C ; ;< <w"2,,r(r) r5r6r7r8r9 classmethodrTstrrrar;r<r=s@r&rOrOms< )) -5 -S -_ - -r(rOcR^\rSrSr\S5rSrSU4SjjrSrU=r $)PlacementClassVariablec[R5(dgSSKJn [ U5[L=(a [ X5$NFr) Placement)rr!(torch.distributed.tensor.placement_typesrjr+ issubclassr"rjs r&is_placement_type(PlacementClassVariable.is_placement_types2#//11FE{d"Cz%'CCr(cUR$r*r3r,s r&as_python_constant)PlacementClassVariable.as_python_constant zzr(c|>[R"URSS5[R4;awUR (af[R UR5n[ U5n[R"URSS5(aURUSX#5 U$[TU]%XU5$)N__new__r ) rAgetattr_staticr"objectrur]PlacementVariable call_methodr call_function)r#rVargsr$new_objvarr%s r&rz$PlacementClassVariable.call_functions  " "4::y$ ?FNNCT T nnTZZ0G#G,C%%djj*dCCJ= w$Rv66r(r)rVrr{list[VariableTracker]r$dict[str, VariableTracker]r4r) r5r6r7r8r:rnrqrzr;r<r=s@r&rfrfsLDD7 #7&7- 7  77r(rfcj^\rSrSr\S5rSrSSS\S\4U4Sjjr S U4S jjr S r U=r $) rxcX[R5(dgSSKJn [ X5$ri)rr!rkrj isinstancerms r& is_placementPlacementVariable.is_placements$#//11F%++r(cUR$r*r3r,s r&rq$PlacementVariable.as_python_constantrsr(rVrrWr4c>US:Xa*[R"URR5$[TU]X5$)Ndim)rcreater"rrrar#rVrWr%s r&raPlacementVariable.var_getattrs3 5=#**4::>>: :w"2,,r(c>SSKJn /SQnX&;Ga[UR5n[R "USS5bS5e[R "Xr5nU[RLaUR"S5$UV s/sHoR5PM nn UR5V V s0sHupXR5_M nn n US:XaU"UR/UQ70UD6 U$U"UR/UQ70UD6n UR"U 5$[T U]5XX45$![ a SnNf=fs sn fs sn n f)Nr)r)r __setattr__is_shard is_partial is_replicate __getattr__zno custom getattr allowed!r)rrr+r"rArvAttributeErrorrwr rrqitemsrry)r#rVrWr{r$rconstant_fold_functions value_typemethodxkv constant_valr%s r&ryPlacementVariable.call_methods= '#   * !$**- **:}dKS0/0S // A('..t44489Dq((*DD9t>v>L#**<8 8w"2T::"   :KsA D*D<5E* D98D9rr{rr$rr4r) r5r6r7r8r:rrqrdrraryr;r<r=s@r&rxrxs],,-5-S-_- *;& *; - *;  *;*;r(rxcj^\rSrSr\S5rSrSSS\S\4U4Sjjr S U4S jjr S r U=r $) DeviceMeshVariablecX[R5(dgSSKJn [ X5$)NFr) DeviceMesh)rr!torch.distributed.device_meshrr)r"rs r&is_device_mesh!DeviceMeshVariable.is_device_meshs$#//11<e((r(cUR$r*r3r,s r&rq%DeviceMeshVariable.as_python_constantrsr(rVrrWr4c>US:Xa*[R"URR5$US:Xa*[R"URR5$[ TU]X5$)Nndim device_type)rrr"rrrrars r&raDeviceMeshVariable.var_getattrsV 6>#**4::??; ; = #**4::+A+AB Bw"2,,r(c>US:XaUVs/sHoUR5PM nnUR5VVs0sHupxXxR5_M n nn[R"URR "U0U D65$US:Xa.[R"URR 55$US:XavUVs/sHoUR5PM nnUR5VVs0sHupxXxR5_M n nn[URR"U0U D65$US:Xa#[URR55$[T U]-XX45$s snfs snnfs snfs snnf)Nsizeget_coordinate get_group_get_or_create_default_group) rqrrrr"rrr`rrrry) r#rVrWr{r$r const_argsrr const_kwargsr%s r&ryDeviceMeshVariable.call_methodsO 6>:>?$Q..0$J?BH,,.Q.$!A3355.LQ#**4::??J+W,+WX X # ##**4::+D+D+FG G ; :>?$Q..0$J?BH,,.Q.$!A3355.LQ' $$jALA  1 1' (O(O(QR Rw"2T::@Q @QsE2E7E=5Frr) r5r6r7r8r:rrqrdrraryr;r<r=s@r&rrs]))-5-S-_-;& ; - ;  ;;r(rcb^\rSrSrSrSrSU4SjjrS U4Sjjr\S5r Sr U=r $) r`iao We don't want a ProcessGroup object to end up in our output graph. But it's common for dynamo to intercept a PG that is then used to get info like rank() or world_size(), as well as passed to utility functions in distributed_c10d which desugar it into plain types like a ranklist and tag. For convenience and proper guarding, we construct a variable type. TODO: make it possible to use ProcessGroupVariable as input to simple functions like _expand_group without dynamo complaining about making a proxy for it. It is not a tensor-like type, and we don't want a proxy- but dynamo assumes torch library functions are dealing with tensor-like types and would have proxies for their args. TODO: should we make this inherit VT instead of UDOV? Do we want any of the default behaviors or just graph-break whenever one of our special cases is not hit? cUR$r*r3r,s r&rq'ProcessGroupVariable.as_python_constant+rsr(c>US:Xa7[RRURR 55$US:Xa7[RRURR 55$US:Xa7[RRURR 55$[TU]!XX45$)Nrankr_get_backend_name) rrrr"rrrrryr#rVrWr{r$r%s r&ry ProcessGroupVariable.call_method.s 6>--44TZZ__5FG G 6>--44TZZ__5FG G & &--44TZZ5Q5Q5ST Tw"2T::r(c>^^^TS:Xa3[RRTRR5$TS;a[R "UUU4Sj5$[ TT]TT5$)N group_name)rrc*>TRTTX5$r*)ry)r{r$rWr#rVs r&2ProcessGroupVariable.var_getattr..Cs(8(8T4(Pr()rrrr"rLambdaVariablerrars```r&ra ProcessGroupVariable.var_getattr>s_ < --44TZZ5J5JK K # #++P w"2t,,r(ch[R5(dgSSKJn SSKJn [ XU45$)NFr) ProcessGroup)FakeProcessGroup)rr!torch._C._distributed_c10dr+torch.testing._internal.distributed.fake_pgrr)r"rrs r&is_process_group%ProcessGroupVariable.is_process_groupHs-#//11;Pe,<=>>r(rrrVr) r5r6r7r8r9rqryrar:rr;r<r=s@r&r`r`sI$;& ; - ;  ; -??r(r`c ^\rSrSrSr\S\S\S\4Sj5rS\RRS\S\S\SS 4 U4S jjr S r S \ \S \\\4S\4U4SjjrSSjrSrU=r$)BackwardHookVariableiSzI Handles torch.utils.hooks.BackwardHook for module-level backward hooks. module user_hooksuser_pre_hooksc^^ ^ [R(d [SSSS/S9 S[4UU U 4SjjnURR US5umnURR U5um nURR U5um nURR SUU405n[RRRSS S 5URRS '[XqX#5$) Nz7Module-level backwards hooks require compiled autograd.rzREnable compiled autograd by setting torch._dynamo.config.compiled_autograd = True.rbw_statec >[RRRS[R "[ [UTTS94[R "[ [UTTS945$)z Rather than installing the user hooks in the graph (which don't survive AotAutograd), we install hooks that will call trace_wrapped in the backward pass that CompiledAutograd can turn into actual hook calls. N)fnr hooks_name module_name)r0utilshooks BackwardHook functoolspartialrr )rruser_hooks_nameuser_pre_hooks_names r&_in_graph_bw_hooks7BackwardHookVariable.create.._in_graph_bw_hooksjsk;;$$11%%%@!)#2$/ %%%@!)#6$/  r(modrzr example_value)rcompiled_autograd_enabledr routputadd_backward_state_hook create_proxyr0rrrnodemetar) rVrrrrbw_state_proxy_proxyrrrs @@@r&rBackwardHookVariable.createYs!:: Qh     :')ii&G&GPU&V# ^!#!B!B>!RQYY>>zJ &&        ,1;;+<+<+I+I$PRTV+W (#E:NNr(rr4Nc V>[TU]"S0UD6 XlX lX0lX@lg)Nr)rr rrrr)r#rrrroptionsr%s r&r BackwardHookVariable.__init__s* #7#  $,r(cUR$r*)rr,s r&as_proxyBackwardHookVariable.as_proxyrsr(r{r$c\>US;aUR"X/UQ70UD6$[TU] XX45$)N)setup_input_hooksetup_output_hook) _setup_hookrryrs r&ry BackwardHookVariable.call_methods; < <##B>t>v> >w"2T::r(c SSKJn U"UURRSUUR 5UR 54055$)Nr) wrap_fx_proxyry)builderrrrr)r#rVhook_method_namer{rs r&r BackwardHookVariable._setup_hooksD*  II " " $--/2    r()rrrrr)r5r6r7r8r9r:rrr0fxProxyr rlistdictrdryrr;r<r=s@r&rrSs 7O7O$7O( 7O7Or -xx~~ -  -$ - ( -  - ;?# ; S/)* ;  ;   r(r)(r9rrAtypingrr0%torch.fx.experimental._backward_staterrrr_trace_wrapped_higher_order_oprexcr external_utilsr guardsr r r]r rrrZrconstantrrtorch._dynamo.symbolic_convertrrrDrMrOrfrxrr`rrr(r&rs( ?+:"B0!4D0/0BER.-0-6 70 7F=;+=;@*;,*;Z8?.8?vg ?g r(