eThn2SSKrSSKJrJrJr \"5(aSSKr\R "\5r"SS5r Sr Sr "SS \5r g) N) ExplicitEnumis_torch_availableloggingcz\rSrSrSrS/S4SjrSSjrSrSrS r S r S r S r S r SrSrSrSrSrSrg)DebugUnderflowOverflowa This debug class helps detect and understand where the model starts getting very large or very small, and more importantly `nan` or `inf` weight and activation elements. There are 2 working modes: 1. Underflow/overflow detection (default) 2. Specific batch absolute min/max tracing without detection Mode 1: Underflow/overflow detection To activate the underflow/overflow detection, initialize the object with the model : ```python debug_overflow = DebugUnderflowOverflow(model) ``` then run the training as normal and if `nan` or `inf` gets detected in at least one of the weight, input or output elements this module will throw an exception and will print `max_frames_to_save` frames that lead to this event, each frame reporting 1. the fully qualified module name plus the class name whose `forward` was run 2. the absolute min and max value of all elements for each module weights, and the inputs and output For example, here is the header and the last few frames in detection report for `google/mt5-small` run in fp16 mixed precision : ``` Detected inf/nan during batch_number=0 Last 21 forward frames: abs min abs max metadata [...] encoder.block.2.layer.1.DenseReluDense.wi_0 Linear 2.17e-07 4.50e+00 weight 1.79e-06 4.65e+00 input[0] 2.68e-06 3.70e+01 output encoder.block.2.layer.1.DenseReluDense.wi_1 Linear 8.08e-07 2.66e+01 weight 1.79e-06 4.65e+00 input[0] 1.27e-04 2.37e+02 output encoder.block.2.layer.1.DenseReluDense.wo Linear 1.01e-06 6.44e+00 weight 0.00e+00 9.74e+03 input[0] 3.18e-04 6.27e+04 output encoder.block.2.layer.1.DenseReluDense T5DenseGatedGeluDense 1.79e-06 4.65e+00 input[0] 3.18e-04 6.27e+04 output encoder.block.2.layer.1.dropout Dropout 3.18e-04 6.27e+04 input[0] 0.00e+00 inf output ``` You can see here, that `T5DenseGatedGeluDense.forward` resulted in output activations, whose absolute max value was around 62.7K, which is very close to fp16's top limit of 64K. In the next frame we have `Dropout` which renormalizes the weights, after it zeroed some of the elements, which pushes the absolute max value to more than 64K, and we get an overflow. As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers. The tracking is done in a forward hook, which gets invoked immediately after `forward` has completed. By default the last 21 frames are printed. You can change the default to adjust for your needs. For example : ```python debug_overflow = DebugUnderflowOverflow(model, max_frames_to_save=100) ``` To validate that you have set up this debugging feature correctly, and you intend to use it in a training that may take hours to complete, first run it with normal tracing enabled for one of a few batches as explained in the next section. Mode 2. Specific batch absolute min/max tracing without detection The second work mode is per-batch tracing with the underflow/overflow detection feature turned off. Let's say you want to watch the absolute min and max values for all the ingredients of each `forward` call of a given batch, and only do that for batches 1 and 3. Then you instantiate this class as : ```python debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3]) ``` And now full batches 1 and 3 will be traced using the same format as explained above. Batches are 0-indexed. This is helpful if you know that the program starts misbehaving after a certain batch number, so you can fast-forward right to that area. Early stopping: You can also specify the batch number after which to stop the training, with : ```python debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3], abort_after_batch_num=3) ``` This feature is mainly useful in the tracing mode, but you can use it for any mode. **Performance**: As this module measures absolute `min`/``max` of each weight of the model on every forward it'll slow the training down. Therefore remember to turn it off once the debugging needs have been met. Args: model (`nn.Module`): The model to debug. max_frames_to_save (`int`, *optional*, defaults to 21): How many frames back to record trace_batch_nums(`List[int]`, *optional*, defaults to `[]`): Which batch numbers to trace (turns detection off) abort_after_batch_num (`int``, *optional*): Whether to abort after a certain batch number has finished NcXlX0lX@l[R"/U5Ul/UlSUlSUlSUl SUl UR5 UR5 g)NrFz ) modeltrace_batch_numsabort_after_batch_num collectionsdequeframesframe batch_number total_callsdetected_overflowprefix analyse_modelregister_forward_hook)selfr max_frames_to_saver rs P/var/www/auris/envauris/lib/python3.13/site-packages/transformers/debug_utils.py__init__DebugUnderflowOverflow.__init__sh 0%:""'',>?  !&)   ""$cUbURU5 URRSRUR55 /UlgN ) expand_framerappendjoinr)rrs r save_frame!DebugUnderflowOverflow.save_frames<     e $ 499TZZ01 rc:URRU5 gN)rr#)rlines rr"#DebugUnderflowOverflow.expand_frames $rcZ[SRUR55 /Ulgr )printr$rrs r trace_frames#DebugUnderflowOverflow.trace_framess dii $% rc/Ulgr()rr-s rreset_saved_frames)DebugUnderflowOverflow.reset_saved_framess  rc [SUR35 [S[UR5S35 [SSSSSS35 [S R UR55 [S 5 /Ulg) Nz& Detected inf/nan during batch_number=zLast z forward frames:abs min8 abs max metadatar! )r,rlenrr$r-s rdump_saved_frames(DebugUnderflowOverflow.dump_saved_framessq 78I8I7JKL c$++&''789 1 Qym956 dii $% f  rcxURR5VVs0sHupX!_M snnUlgs snnfr()r named_modules module_names)rnamems rr$DebugUnderflowOverflow.analyse_models1 59JJ4L4L4NO4NQW4NOOs6c[R"U5(a3UR[X55 [ X5(aSUlggUcURSSSU35 gURSSSU35 g)NTNonez>17r6z not a tensor)torch is_tensorr"get_abs_min_maxdetect_overflowr)rvarctxs ranalyse_variable'DebugUnderflowOverflow.analyse_variablesx ??3     oc7 8s(()-&) [    AcU3 4    4AcU;  eU # #!%(%%a6!A7)  ! !% 1 fe $ $!&)a'' )! --a71#Rs!1DE!-))!wqc^< *  ! !&( 3 rcNURRUR5 gr()r apply_register_forward_hookr-s rr,DebugUnderflowOverflow.register_forward_hooks 445rc:URUR5 gr()r forward_hook)rr_s rri-DebugUnderflowOverflow._register_forward_hooks$$T%6%67rcSnURUR;aSOSnU(aUR5 URS:XaUR 5 U=RS- slXR :XaU=RS- slSnUR XU5 U(aUR5 U(aUR 5 UR(a"U(dUR5 [S5eURbAURUR:a&[SURSURS35egg) NFTrrzDebugUnderflowOverflow: inf/nan detected, aborting as there is no point running further. Please scroll up above this traceback to see the activation values prior to this event.z'DebugUnderflowOverflow: aborting after z' batches due to `abort_after_batch_num=z` arg) rr r1rrOr rer.rr; ValueErrorr)rr_rWrXlast_frame_of_batch trace_modes rrl#DebugUnderflowOverflow.forward_hooks<$!..$2G2GGTU   # # %   q  " " $ A ZZ     " "&  &0        " " $  ! !*  " " $j   % % 1d6G6G$JdJd6d9$:K:K9LM++/+E+E*FeM 7e 1r) rrrrrr r?rrr r()rZ __module__ __qualname____firstlineno____doc__rr%r"r.r1r;rrKrOrRrerrirl__static_attributes__rrrrs\sj24b`d%"  P=Cf868-rrcrUR5nUR5SSUR5SSU3$)Nz8.2er6)absminmax)rIrJabs_vars rrGrG%s6ggiGkkmD !7;;="6au ==rc,Sn[R"U5R5R5(aSn[ US35 [R "U5R5R5(aSn[ US35 U$)a Report whether the tensor contains any `nan` or `inf` entries. This is useful for detecting overflows/underflows and best to call right after the function that did some math that modified the tensor in question. This function contains a few other helper features that you can enable and tweak directly if you want to track various other things. Args: var: the tensor variable to check ctx: the message to print as a context Return: `True` if `inf` or `nan` was detected, `False` otherwise FTz has nansz has infs) rEisnananyitemr,isinfgerznumelr{r|rImean)rIrJdetectedn100n1000n10000s rrHrH*s"H {{3""$$ Y  {{3""$$ Y     Orc\rSrSrSrSrSrg) DebugOptioniXunderflow_overflowtpu_metrics_debugrxN)rZrsrtruUNDERFLOW_OVERFLOWTPU_METRICS_DEBUGrwrxrrrrXs-+rr) rutilsrrrrE get_loggerrZloggerrrGrHrrxrrrsT<<   H %GGT> +\,,,r