o Zh4@sddlmZddlZddlmZmZddlZddlmZddlm Z ddl m Z m Z ddl mZmZdd lmZmZmZdd lmZerZddlZdd lmZdd lmZdd lmZGdddejZGdddZdS)) annotationsN)Any TYPE_CHECKING)config) write_text)get_metric_tableis_metric_table_enabled)DeviceProperties ReductionHint)BaseSchedulerNode Scheduler WhyNoFuse)V) OrderedSet)SIMDKernelFeatures) TritonKernelc@seZdZdZd ddZdS) Sortablez>Anything that can be used as a list.sort() key (int/tuple/etc)other typing.SelfreturnboolcCsdSN)selfrrrF/var/www/auris/lib/python3.10/site-packages/torch/_inductor/choices.py__lt__szSortable.__lt__N)rrrr)__name__ __module__ __qualname____doc__rrrrrrsrc@seZdZdZd-d d Zed.ddZed/ddZed.ddZed0ddZ ed1d#d$Z ed1d%d&Z ed1d'd(Z ed2d*d+Z d,S)3InductorChoicesax This class contains a collection of default heuristics that effect performance of our generated code. We try to not put correctness requirements in this file. You can override the choices made here by doing: class MyHeuristics(InductorChoices): ... torch._inductor.virtualized.V.set_choices_handler(MyHeuristics()) kernel_clstype[TritonKernel]featuresrgroupslist[sympy.Expr] kernel_kwargsdict[str, Any]rcCs|S)zTHook to change the kwargs passed to TritonKernel, used to apply fixed configurationsr)rr"r$r%r'rrrtriton_kernel_kwargs+z$InductorChoices.triton_kernel_kwargsrcCsrtjjrdStjjrtjjdkrdStjjj |j dd}|dkr'd|}n |dkr.d }ndStjj |j |S) z>Heuristic to decide if a cooperative reduction should be used.TcpuF)fallbackii ) rtritonZforce_cooperative_reductionsZcooperative_reductionsrgraphZget_current_device_or_throwtypesizevars size_hintnumelstatically_known_geqreduction_numel)r$Zxhint thresholdrrr should_use_cooperative_reduction5s z0InductorChoices.should_use_cooperative_reductioncooperative_reductioncCs~tjjsdStjdi|d}|r.z|dttj j |j d9}Wn t y-Ynwtjjr6|d9}tj j |j|S)zO Heuristic to decide if a persistent reduction should be used. Fi@ r/)rr0Zpersistent_reductionsr INNERgetget_reduction_hintminrr1r3r4r5 ValueErrorZ multi_kernelZstatically_known_leqr7)r$r:r8rrrshould_use_persistent_reductionLs" " z/InductorChoices.should_use_persistent_reductioncCs |tjkotjj|jdS)a Heuristic to decide if we should drop the X dimension from a persistent reduction kernel. So the [XBLOCK, RBLOCK] block becomes a [RBLOCK] block and XBLOCK is forced to be always 1. Strangely this is faster than a [1, RBLOCK] block in some cases. )r?r r=rr1r3r6r7)r$rrr want_no_x_dimjszInductorChoices.want_no_x_dimdevice torch.devicereduction_numel_hintint numel_hintinner_reductioncsBt|}|j}d}dd}|||}||} d} d| } |r|d|kr*dS|dkr0dS|||kr9|} n^||| kr{||d| } | |d|}|| |d| |t|}t|fdd d }t|d krxt||} n} nt|}t|fd d d }t|d kr|} n} || | d| | Sd}d}||d|}|||kr|} n]||| kr||| } | |d|} ||| d|| t|}t|fdd d }t|dkrt||} n } nt|}t|fdd d }t|d kr|} n} ||| d|| S)zHeuristic to decide the RSPLIT used for split reductions. When a reduction has a small number of outputs there is not enough parallelism, so we will do the reduction in two phases.r<iir.r,ri c t|Srabsxtmp_split_sizerr z8InductorChoices.reduction_split_factor..)keycrKrrLrNmax_elements_per_threadrrrRrS2crKrrLrNrPrrrRrScrKrrLrNrVrrrRrS)r createZmulti_processor_countsympydivisorsr@rMmax)rErGrIrJpropsZnum_smZmin_elements_per_threadZthreads_per_smZmin_elements_per_deviceZmax_elements_per_deviceZ num_warpsZ num_threadsZ split_sizeZ target_blocksZblocks_per_outputr^ZclosestZrvals_per_threadZxvals_per_blockZxblocksr)rWrQrreduction_split_factorvsv               z&InductorChoices.reduction_split_factor schedulerr node1r node2shared_data_scorecs|dkrCtjrsrCtdr:jj@tdkr:tdfddt ddSt ddS sc sct t tj krct ddS rrt d dSd S) a Heuristics to prevent fusion applied to both horizontal and vertical fusions. Heuristics here should not be needed for correctness and tweaking them may yield additional performance. See also some related heuristics that can be changed via config: - config.triton.tiling_prevents_pointwise_fusion - config.triton.tiling_prevents_reduction_fusion - config.aggressive_fusion (will cause this function to be called more times) rZ'fusion_failure_due_to_indexing_mismatchc sDtjjtjjtttdS)N)Zpre_grad_graph_idpost_grad_graph_idZ node1_nameZ node2_nameZnode1_debug_strZnode2_debug_strZcommon_buffer_namesZfailure_reason) rr1Zgraph_idrfget_namerZ debug_strlistZdecide_fusion_fail_reasonrZcommon_buf_namesrcrdrbrrrRs  z*InductorChoices.can_fuse..z'no shared data due to indexing mismatchFzno shared datazexceeds max fusionz Fusion will increase peak memoryT)rZaggressive_fusion is_reductionr Z read_writesZ buffer_nameslenradd_rowrZ is_foreachZ get_nodesZmax_fusion_sizeZcan_fusion_increase_peak_memoryrbrcrdrerrircan_fuses:  zInductorChoices.can_fusecCsdS)zCHook for heuristics to prevent vertical (producer/consumer) fusionsTrrmrrrcan_fuse_verticalr*z!InductorChoices.can_fuse_verticalcCs>|tjkrt||ddS|||rt||ddSdS)zEHook for heuristics to prevent horizontal (consumer/consumer) fusionsscore_fusion_memory_thresholdFz=Nodes are too far away. Fusing them may increase peak memory.T)rrprZare_long_distant_nodesrmrrrcan_fuse_horizontals  z#InductorChoices.can_fuse_horizontalrcCsx|||}tt|j|jt|j|j }|rd}n d|tjko*|dk}|||ko8|dk||fS)a Assign a score (higher comes first) to the fusion of node1 and node2. When different fusions conflict with each other, this is the way we decide what order to run them in. Our current score is based on: - The type of fusion (template/reduction/etc) - Estimate of the saved memory operations - Fusions closer together in original graph order rr) Zscore_fusion_memoryr_rMZ min_orderZ max_orderZ is_templaterZepilogue_fusion_firstrj)rbrcrdZ memory_scoreZproximity_scoreZtemplate_scorerrr score_fusion"s zInductorChoices.score_fusionN) r"r#r$rr%r&r'r(rr()r$rrr)r$rr:rrr) rErFrGrHrIrHrJrrrH) rbr rcr rdr rerHrr)rbr rcr rdr rr)rrrr r) staticmethodr9rBrDrarnrorqrrrrrrr!s&     U 9  r!) __future__rtypingrrr]rZ codecacherZmetricsrr Z runtime.hintsr r rbr r rZ virtualizedrZtorchZtorch.utils._ordered_setrZcodegen.simd_kernel_featuresrZcodegen.tritonrProtocolrr!rrrrs"