o Zh @sddlZddlmZddlmZddlZddlmZeeZ ddZ ddZ Gd d d e Z d d Zd eeejeejdffdeeejeejdfffddZ ddedededeeeffddZeGdddZdS)N) dataclass)Union)fxcs&gfdd}tj||}|fS)z\ Flatten the args into a list form and detach the tensors from computational graph. cs8t|tjr||j}||S||SN) isinstancetorchTensordetachZrequires_grad_Z requires_gradappend)avalZflat_detached_argsR/var/www/auris/lib/python3.10/site-packages/torch/distributed/pipelining/_utils.pyextract_tensor_argss   z0flatten_args_detach..extract_tensor_argsrnodeZ map_aggregate)argsrnew_argsrr rflatten_args_detachs  rcs"gfdd}tj||S)z, Flatten the args into a list form. cs||Sr)r )r Z flat_argsrrr,s z)flatten_args..extract_tensor_argsr)rrrrr flatten_args&s rc@seZdZdZdS)PipeliningShapeErrorz5Shape mismatch between configured and runtime values.N)__name__ __module__ __qualname____doc__rrrrr9srcCs|j|jkst|d|jd|j|j|jks(t|d|jd|j||ks@t|d|d|dS)Nz has a shape mismatch: expected z actual z has a dtype mismatch: expected z! has a stride mismatch: expected )shaperZdtypeZstride)descexpectedgivenrrrvalidate_tensor_metadata=s  r!expected_tensors.actual_tensorscCsft|t|krt|dt|dt|dtt|D]}t|d|||||qdS)Nz: Number of values (z") does not match expected number ()z: value )lenrranger!)rr"r#irrrvalidate_tensors_metadataLsr(looppp_size num_stagesstylereturncCsi}|dkrt|D]}||||<q |S|dkrS||dkr*td|d|dd}t|D] }|||<|d|dkr?q0||ddkrL|d7}q0|d8}q0|Std |d ) z Compute the stage id to rank mapping for either a looped or V-style schedule. Most commonly num_stages == pp_size * 2, but this function can be used to compute the mapping for any number of stages per rank. r)vrz num_stages z% must be evenly divisible by pp_size z for V scheduleszStyle z is not supported.)r& ValueError)r*r+r,mappingZ stage_indexZ rank_indexrrrgenerate_stage_to_rank_mapping[s(      r3c@s,eZdZUdZejed<eed<eed<dS)PipeInfoz> Captures information for a pipeline (`Pipe` object). graphr+Zhas_loss_and_backwardN) rrrrrZGraph__annotations__intboolrrrrr4}s   r4)r))logging dataclassesrtypingrrr getLoggerrloggerrr RuntimeErrorrr!listrtupler(r7strdictr3r4rrrrs6       "