a h@sddlZddlmZddlmZddlZddlmZeeZ ddZ ddZ Gd d d e Z d d Zeeejeejd ffeeejeejd ffdddZdeeeeeefdddZeGdddZdS)N) dataclass)Union)fxcs&gfdd}tj||}|fS)z\ Flatten the args into a list form and detach the tensors from computational graph. cs<t|tjr*||j}||S||SdSN) isinstancetorchTensordetachZrequires_grad_Z requires_gradappend)avalZflat_detached_argsQ/var/www/auris/lib/python3.9/site-packages/torch/distributed/pipelining/_utils.pyextract_tensor_argss    z0flatten_args_detach..extract_tensor_argsrnodeZ map_aggregate)argsrnew_argsrr rflatten_args_detachs rcs"gfdd}tj||S)z, Flatten the args into a list form. cs||Sr)r )r Z flat_argsrrr,s z)flatten_args..extract_tensor_argsr)rrrrr flatten_args&s rc@seZdZdZdS)PipeliningShapeErrorz5Shape mismatch between configured and runtime values.N)__name__ __module__ __qualname____doc__rrrrr9srcCs|j|jks(t|d|jd|j|j|jksPt|d|jd|j||kst|d|d|dS)Nz has a shape mismatch: expected z actual z has a dtype mismatch: expected z! has a stride mismatch: expected )shaperZdtypeZstride)descexpectedgivenrrrvalidate_tensor_metadata=s  r!.)expected_tensorsactual_tensorscCsft|t|kr2t|dt|dt|dtt|D]"}t|d|||||q>dS)Nz: Number of values (z") does not match expected number ()z: value )lenrranger!)rr"r#irrrvalidate_tensors_metadataLsr(loop)pp_size num_stagesstylereturncCsi}|dkr(t|D]}||||<qn|dkr||dkrRtd|d|dd}t|D]@}|||<|d|dkr|q^||ddkr|d7}q^|d8}q^ntd |d |S) z Compute the stage id to rank mapping for either a looped or V-style schedule. Most commonly num_stages == pp_size * 2, but this function can be used to compute the mapping for any number of stages per rank. r)vrz num_stages z% must be evenly divisible by pp_size z for V scheduleszStyle z is not supported.)r& ValueError)r*r+r,mappingZ stage_indexZ rank_indexrrrgenerate_stage_to_rank_mapping[s&      r3c@s,eZdZUdZejed<eed<eed<dS)PipeInfoz> Captures information for a pipeline (`Pipe` object). graphr+Zhas_loss_and_backwardN) rrrrrZGraph__annotations__intboolrrrrr4}s  r4)r))loggingZ dataclassesrtypingrrr getLoggerrloggerrr RuntimeErrorrr!listrtupler(r7strdictr3r4rrrrs&      "