eTh5FSrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK J r J r J r Jr SSKrSSKJrJrJrJrJrJrJrJrJrJrJrJrJr \"5(aSSKrS\ S\ S\ 4S jr!SDS \ S \"4S jjr#SDS \ S \"4Sjjr$Sr%"SS5r&"SS\ 5r'"SS\ 5r("SS\ 5r)Sr*\RV"S\*-S-5r,Sr-"SS\5r."SS\5r/"S S!\5r0"S"S#\5r1"S$S%\ 5r2S&\3\4\54S'\54S(jr6S'\3\4\544S)jr7S'\3\4\544S*jr8S+r9S'\3\4\544S,jr:"S-S.\5r;S/r"S2S3\5r?"S4S55r@S6rAS7rBS8rCSFS9\ \DS:\ S;\"4S<jjrE"S=S>\5rF"S?S@5rGSDSA\4SB\"4SCjjrHg)Gz6 PyTorch-independent utilities for the Trainer class. N)Any NamedTupleOptionalUnion) ExplicitEnumis_psutil_availableis_tf_availableis_torch_availableis_torch_cuda_availableis_torch_hpu_availableis_torch_mlu_availableis_torch_mps_availableis_torch_musa_availableis_torch_npu_availableis_torch_xla_availableis_torch_xpu_availablerequires_backends worker_id num_workersrankcX[R"5S-nX-U-n[U5 g)zF Helper function to set worker seed during Dataloader initialization. lN)torch initial_seedset_seed)rrr init_seed worker_seeds R/var/www/auris/envauris/lib/python3.13/site-packages/transformers/trainer_utils.py seed_workerr4s,""$u,I$y0K [seed warn_onlyc [U5 [5(aS[RS'S[RS'S[RS'S[RS'S[RS'[R "SUS 9 S[R RlS [R Rl [5(a)S S K nURRR5 g g ) z Helper function for reproducible behavior during distributed training. See - https://pytorch.org/docs/stable/notes/randomness.html for pytorch - https://www.tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism for tensorflow 1CUDA_LAUNCH_BLOCKINGz:16:8CUBLAS_WORKSPACE_CONFIGASCEND_LAUNCH_BLOCKINGHCCL_DETERMINISTICFLASH_ATTENTION_DETERMINISTICT)r"FrN)rr osenvironruse_deterministic_algorithmsbackendscudnn deterministic benchmarkr tensorflowconfig experimentalenable_op_determinism)r!r"tfs renable_full_determinismr6=s TN.1 )*07 ,-/2 +,+. '(69 23 **49E.2*).& 446r r/c[R"U5 [RRU5 [5(aR[R "U5 [R RU5 U(a[R"S5 [5(a[RRU5 [5(a[RRU5 [5(a[RRU5 [5(a[R RU5 [#5(a[R$RU5 ['5(aLSSKnURR+U5 U(a%UR,R.R15 ggg)a= Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch` and/or `tf` (if installed). Args: seed (`int`): The seed to set. deterministic (`bool`, *optional*, defaults to `False`): Whether to use deterministic algorithms where available. Can slow down training. TrN)randomr!npr r manual_seedcudamanual_seed_allr,rmlurmusarnpur hpurxpur r1rr2r3r4)r!r/r5s rrr]s" KKIINN4 $ ""4(   . .t 4 !!$'   ""4( !!$' !!$' !!$' 4  II " " 8 8 :  r c0UR(a[R"URS5URS5-5nUR[R "U5- nU[R "U5RU*U5-nU$)a Implements the NEFTune forward pass for the model using forward hooks. Note this works only for torch.nn.Embedding layers. This method is slightly adapted from the original source code that can be found here: https://github.com/neelsjain/NEFTune Simply add it to your model as follows: ```python model = ... model.embed_tokens.neftune_noise_alpha = 0.1 model.embed_tokens.register_forward_hook(neftune_post_forward_hook) ``` Args: module (`torch.nn.Module`): The embedding module where the hook is attached. Note that you need to set `module.neftune_noise_alpha` to the desired noise alpha value. input (`torch.Tensor`): The input tensor to the model. output (`torch.Tensor`): The output tensor of the model (i.e. the embeddings). r)trainingrtensorsizeneftune_noise_alphasqrt zeros_likeuniform_)moduleinputoutputdimsmag_norms rneftune_post_forward_hookrPsp&||FKKNV[[^;<-- 40@@%**62;;XIxPP Mr c 2\rSrSrSrS S\\R\\R4S\\R\\R4S\ \\R\\R4S\ \\R\\R44Sjjr S r S r S r g) EvalPredictiona^ Evaluation output (always contains labels), to be used to compute metrics. Parameters: predictions (`np.ndarray`): Predictions of the model. label_ids (`np.ndarray`): Targets to be matched. inputs (`np.ndarray`, *optional*): Input data passed to the model. losses (`np.ndarray`, *optional*): Loss values computed during evaluation. N predictions label_idsinputslossesc$XlX lX0lX@lURUR4UlURb U=RUR4- slURb!U=RUR4- slggN)rTrUrVrWelements)selfrTrUrVrWs r__init__EvalPrediction.__init__sp'"  ))4>>: ;; " MMdkk^ +M ;; " MMdkk^ +M #r c,[UR5$rY)iterrZr[s r__iter__EvalPrediction.__iter__sDMM""r ctUS:dU[UR5:a [S5eURU$)Nrztuple index out of range)lenrZ IndexError)r[idxs r __getitem__EvalPrediction.__getitem__s4 7cS//78 8}}S!!r )rZrVrUrWrT)NN)__name__ __module__ __qualname____firstlineno____doc__rr9ndarraytuplerr\rarg__static_attributes__r rrRrRsBFAE ,2::uRZZ'889,U2::%667,rzz5+<<=> , rzz5+<<=> ,"#"r rRc\rSrSr%\\R \\R 4\S'\ \\R \\R 4\S'\ \ \ \ 4\S'\ \ \S'Srg)EvalLoopOutputrTrUmetrics num_samplesrqN)rirjrkrlrr9rnro__annotations__rdictstrfloatintrprqr rrsrss_rzz5#4455bjj% *;;<== d3:& ''#r rsc\rSrSr%\\R \\R 4\S'\ \\R \\R 4\S'\ \ \ \ 4\S'Sr g)PredictionOutputrTrUrurqN)rirjrkrlrr9rnrorwrrxryrzrprqr rr}r}sUrzz5#4455bjj% *;;<== d3:& ''r r}c>\rSrSr%\\S'\\S'\\\4\S'Sr g) TrainOutput global_step training_lossrurqN) rirjrkrlr{rwrzrxryrprqr rrrs #u* r r checkpoint^z\-(\d+)$c [R"U5nUVs/sH`n[RU5cMR [RR X55(dM^UPMb nn[U5S:Xag[RR U[USS95$s snf)Nrc`[[RU5R5S5$)Nr)r{_re_checkpointsearchgroups)xs r%get_last_checkpoint..s$s>CXCXYZC[CbCbCdefCg?hr )key) r*listdirrrpathisdirjoinrdmax)foldercontentr checkpointss rget_last_checkpointrsjj GD   & 79ww}}RWW\\RXE_7`   ;1 77<<K5h i jjsB=?B=<B=c \rSrSrSrSrSrSrg)IntervalStrategynostepsepochrqNrirjrkrlNOSTEPSEPOCHrprqr rrr B E Er rc$\rSrSrSrSrSrSrSrg) SaveStrategyrrrbestrqN) rirjrkrlrrrBESTrprqr rrrs B E E Dr rc \rSrSrSrSrSrSrg)EvaluationStrategyrrrrqNrrqr rrrrr rc$\rSrSrSrSrSrSrSrg) HubStrategyend every_saverall_checkpointsrqN) rirjrkrlEND EVERY_SAVE CHECKPOINTALL_CHECKPOINTSrprqr rrrs CJJ'Or rcf\rSrSr%Sr\\S'\\\ \4\S'\ \\ 4\S'Sr \ \ \S'Srg) BestRuna3 The best run found by a hyperparameter search (see [`~Trainer.hyperparameter_search`]). Parameters: run_id (`str`): The id of the best run (if models were saved, the corresponding checkpoint will be in the folder ending with run-{run_id}). objective (`float`): The objective that was obtained for this run. hyperparameters (`Dict[str, Any]`): The hyperparameters picked to get this run. run_summary (`Optional[Any]`): A summary of tuning experiments. `ray.tune.ExperimentAnalysis` object for Ray backend. run_id objectivehyperparametersN run_summaryrq)rirjrkrlrmryrwrrzlistrxrrrrprqr rrrs>  KUDK'((#s(^#!%K#%r rrureturnc[R"U5nURSS5nURSS5nUR5Vs/sHInUR S5(d.UR S5(dUR S5(dMGUPMK nnUHnURUS5nM [ U5S:XaU$[ UR55$s snf)aN The default objective to maximize/minimize when doing an hyperparameter search. It is the evaluation loss if no metrics are provided to the [`Trainer`], the sum of all metrics otherwise. Args: metrics (`Dict[str, float]`): The metrics returned by the evaluate method. Return: `float`: The objective to minimize or maximize eval_lossNr_runtime _per_second_compilation_timer)copydeepcopypopkeysendswithrdsumvalues)ruloss_m speed_metricssms rdefault_compute_objectiversmmG$G ;;{D )D GT"AA ::j ! !QZZ %>%>!**M`Ba   KKD !w<1$4?#gnn.>*??s AC$C$cSSKJn U"5(dS5eURSSSSS9URS SS 5URS SS 5UR S /SQ5S.$)Nr)is_optuna_availablez:This function needs Optuna installed: `pip install optuna` learning_rateư>-C6?T)lognum_train_epochsr!(per_device_train_batch_size @rrr!r) integrationsr suggest_float suggest_intsuggest_categorical)trialrs rdefault_hp_space_optunar,sr1  ^"^^ ,,_dDd,S!--.@!QG!!&!R0','@'@A^`r's  r c SSKJn U"5(dS5eSSKJn UR SS5UR [ [SS555URSS 5UR /S Q5S .$) Nr)is_ray_tune_availablez:This function needs ray installed: `pip install ray[tune]`r)tunerrrrr) rrrayr loguniformchoicerrangeuniform)rrrs rdefault_hp_space_rayr8si3 " "`$`` "t4 KKU1a[(9: Q#'+{{3E'F  r cFSSS.SSSS.SS S.S S S .SS S.SS S ./SQSSS./$)Nrr)minrrdoubler)boundsnametypetransformationrrrr{)rrrrr!)48163264r categorical)categorical_valuesrrrq)rs rdefault_hp_space_sigoptrFsL-PXlqrQ'1CUSR(&%H">1!  r cxSSKJn U"5(d [S5eSSSS.SS S S .S SS S .S SSS .S/SQ0S.S.$)Nr)is_wandb_availablez8This function needs wandb installed: `pip install wandb`r8rminimize)rgoalrrr) distributionrr int_uniformrrrrr)methodmetric parameters)rr ImportError)rrs rdefault_hp_space_wandbr Ss^0   TUU& ;.7TR1>qQR S%21RH,46H+I    r c$\rSrSrSrSrSrSrSrg)HPSearchBackendieoptunarsigoptwandbrqN) rirjrkrlOPTUNARAYSIGOPTWANDBrprqr rr r es F C F Er r c\[5(aSSKJn UR5S:H$US;$)z Whether or not the current process is the local process, based on `xr.global_ordinal()` (for TPUs) first, then on `local_rank`. rN)r)rtorch_xla.runtimeruntimeglobal_ordinal) local_rankxrs ris_main_processrls/ &  "a''   r c[5(aSSKJn UR5$US:wa-[ 5(aSSKnUR R5$g)z_ Return the number of processes launched in parallel. Works with `torch.distributed` and TPUs. rNrr)rrr world_sizer r distributedget_world_size)rrrs rtotal_processes_numberr xsH&}} r 022  //11 r c[R"5U- nUS3[US50nUS:XaU$UbX%- n[US5X`S3'UbX5- n[US5X`S3'UbXE- n [U S5X`S3'U$)a Measure and return speed performance metrics. This function requires a time snapshot `start_time` before the operation to be measured starts and this function should be run immediately after the operation to be measured has completed. Args: - split: name to prefix metric (like train, eval, test...) - start_time: operation start time - num_samples: number of samples processed - num_steps: number of steps processed - num_tokens: number of tokens processed rrr_samples_per_second_steps_per_second_tokens_per_second)timeround) split start_timerv num_steps num_tokensrresultsamples_per_secondsteps_per_secondtokens_per_seconds rrrsiikJ&Gx %"3 4F!| (2056H!0L+,-$..34Da.H)*+&0/45F/J*+, Mr c@\rSrSrSrSrSrSrSrSr Sr S r S r S r S rS rg) SchedulerTypeia Scheduler names for the parameter `lr_scheduler_type` in [`TrainingArguments`]. By default, it uses "linear". Internally, this retrieves `get_linear_schedule_with_warmup` scheduler from [`Trainer`]. Scheduler types: - "linear" = get_linear_schedule_with_warmup - "cosine" = get_cosine_schedule_with_warmup - "cosine_with_restarts" = get_cosine_with_hard_restarts_schedule_with_warmup - "polynomial" = get_polynomial_decay_schedule_with_warmup - "constant" = get_constant_schedule - "constant_with_warmup" = get_constant_schedule_with_warmup - "inverse_sqrt" = get_inverse_sqrt_schedule - "reduce_lr_on_plateau" = get_reduce_on_plateau_schedule - "cosine_with_min_lr" = get_cosine_with_min_lr_schedule_with_warmup - "warmup_stable_decay" = get_wsd_schedule linearcosinecosine_with_restarts polynomialconstantconstant_with_warmup inverse_sqrtreduce_lr_on_plateaucosine_with_min_lrwarmup_stable_decayrqN)rirjrkrlrmLINEARCOSINECOSINE_WITH_RESTARTS POLYNOMIALCONSTANTCONSTANT_WITH_WARMUP INVERSE_SQRTREDUCE_ON_PLATEAUCOSINE_WITH_MIN_LRWARMUP_STABLE_DECAYrprqr rr1r1s@ F F1JH1!L.-/r r1c`\rSrSrSrSSSSSS.rSSjrS rS rS r S r S r Sr SSjr Srg)TrainerMemoryTrackeria A helper class that tracks cpu and gpu memory. This class will silently skip unless `psutil` is available. Install with `pip install psutil`. When a stage completes, it can pass metrics dict to update with the memory metrics gathered during this stage. Example : ```python self._memory_tracker = TrainerMemoryTracker(self.args.skip_memory_metrics) self._memory_tracker.start() # code ... metrics = {"train_runtime": 10.5} self._memory_tracker.stop_and_update_metrics(metrics) ``` At the moment GPU tracking is only for `pytorch`, but can be extended to support `tensorflow`. To understand this class' intricacies please read the documentation of [`~Trainer.log_metrics`]. inittrainevaltest)r\rI_inner_training_loopevaluatepredictcPXl[5(dSUlUR(agSSKn[5(d[ 5(d[ 5(aSSKnX0l0UlO[5(aSSKnX0l0UlOj[5(aSSKnX0l0UlOI[5(aSSKnX0l0UlO([5(aSSKnX0l0UlOSUlUR5Ul SUl0UlSUlg)NTrF)skip_memory_metricsr psutilr rrrgpurrrr Processprocess cur_stagecpu init_reported)r[rPrQrs rr\TrainerMemoryTracker.__init__s#6 "$$'+D $  # #  " $ $(>(@(@D[D]D] JDH # % % JDH # % % JDH # % % JDH # % % JDHDJ~~' "r c [R"5RRRRnXR ;aUR U$[ SUSUR R535e)z+derives the stage/caller name automaticallyzwas called from z+, but only expect to be called from one of )inspect currentframef_backf_codeco_namestages ValueErrorr)r[callers r derive_stage!TrainerMemoryTracker.derive_stagesp%%'..55<<DD [[ ;;v& &"6(*UVZVaVaVfVfVhUij r cJURR5R$)z4get resident set size memory for the current process)rT memory_inforssr`s r cpu_mem_used!TrainerMemoryTracker.cpu_mem_useds||'')---r cSUl[UR5UR5UlUR(dgM=)Nr)cpu_mem_used_peakrrgpeak_monitoringr`s rpeak_monitor_func&TrainerMemoryTracker.peak_monitor_funcs>!#%():):)T>T%UD " ''r cH UR(agUR5nURbURU:wagXl[R"5 UR Gb6[ R R5(aJUR R R5 UR R R5 GO[5(aJUR RR5 UR RR5 GOp[5(aJUR RR5 UR RR5 GO[5(aIUR RR5 UR RR5 O[!5(aIUR R"R5 UR R"R5 Og[%5(a%UR R&R5 O3[)5(a$UR R*R5 UR Gb[ R R5(a+UR R R-5UlGOV[5(a+UR RR-5UlGO[5(a*UR RR-5UlO[5(a*UR RR-5UlO[!5(a*UR R"R-5UlOq[%5(a*UR R&R-5UlO8[)5(a)UR R*R15UlUR35UlSUl:"UR<S9nSUlURA5 g)z%start tracking for the caller's stageNT)target)!rPrbrUgccollectrr; is_availablereset_peak_memory_stats empty_cacherr=rr>rrArr?r r@rmpsmemory_allocatedgpu_mem_used_at_startcurrent_allocated_memoryrgcpu_mem_used_at_startrk threadingThreadrldaemonstart)r[stagepeak_monitor_threads rr}TrainerMemoryTracker.start*s  # # !!# >> %$..E*A  :: !zz&&(( 779 ++-')) 668 **,(** 779 ++-')) 668 **,')) 668 **,')) 668()) **, :: !zz&&((-1ZZ__-M-M-O*'))-1ZZ^^-L-L-N*(**-1ZZ__-M-M-O*'))-1ZZ^^-L-L-N*'))-1ZZ^^-L-L-N*'))-1ZZ^^-L-L-N*'))-1ZZ^^-T-T-V*&*%6%6%8"#'..d6L6LM%)"!!#r c0 URbURU:wagSUl[R"5 URGb\[R R 5(a&URR R5 GO[5(a%URRR5 O[5(a%URRR5 O[5(a%URRR5 Ow[5(a%URRR5 OC[!5(aO3[#5(a$URR$R5 URGb^[R R 5(aTURR R'5UlURR R+5UlGO8[5(aTURRR'5UlURRR+5UlGO[5(aTURRR'5UlURRR+5UlGOr[5(aTURRR'5UlURRR+5UlGO[5(aSURRR'5UlURRR+5UlO[!5(aSURR.R'5UlURR.R+5UlOK[#5(a1URR$R15UlSUlO [3S5eUR4UR(UR(UR4- S.UR6UR'UR,b>[9SUR,UR(- 5UR6URS'OSUR6URS'UR;5UlUR>UR<UR<UR>- [9SUR@UR<- 5S.URBUR'SUlg) z"stop tracking for the passed stageNFzNo available GPU device found!)beginrallocrpeakedz Not available)rrrr)"rUrkrprqrr;rrrtrr=rr>rrArr?r rrurvgpu_mem_used_nowmax_memory_allocatedgpu_mem_used_peakr@rxr`rwrRrrgcpu_mem_used_nowryrjrV)r[r~s rstopTrainerMemoryTracker.stophs >> %$..E*A  % :: !zz&&(( ++-')) **,(** ++-')) **,')) **,'))')) **, :: !zz&&(((, (H(H(J%)-)M)M)O&'))(, (G(G(I%)-)L)L)N&(**(, (H(H(J%)-)M)M)O&'))(, (G(G(I%)-)L)L)N&'))(, (G(G(I%)-)L)L)N&'))(, (G(G(I%)-)L)L)N&'))(, (O(O(Q%)-&!!ABB33,,//$2L2LL(DHHT^^ $ %%158D> %$..E*A !! MM!V $!%D E(HH$hhuo)=<@HHUOA[U5"UR 5VVs0sHupU[ U5_M snn5$[U[R5(aUR5$[5(aC[U[R5(a$UR5S:XaUR5$U$s snnf)zE Recursively calls `.item()` on the element of the dictionary passed c38# UHn[U5v M g7frY)denumpify_detensorize).0rs r (denumpify_detensorize..sGw!2155wsr) isinstancerrorrxitemsrr9genericitemr rTensornumel)rukvs rrrs'D%=))G}GwGGG GT " "G}gmmoVodaa!6q!99oVWW GRZZ ( (||~   *Well"C"C [\H\||~ N Ws'D cT[U[R5(aa[[R "UR 5R5nU[UR5- [UR5- $[[R "U5R5$)zY Return the number of arguments of the passed function, even if it's a partial function. ) r functoolspartialrdrZ signaturefuncrargskeywords)r total_argss rnumber_of_argumentsrss$ ))****4995@@A C N*S-??? w  &11 22r functionstarting_batch_sizeauto_find_batch_sizecUc[R"[UUS9$U(a[[S5 SSKJn U"XS9$[R"XS9$)a Args: A basic decorator that will try to execute `function`. If it fails from exceptions related to out-of-memory or CUDNN, the batch size is cut in half and passed to `function`. `function` must take in a `batch_size` parameter as its first argument. function (`callable`, *optional*) A function to wrap starting_batch_size (`int`, *optional*) The batch size to try and fit into memory auto_find_batch_size (`bool`, *optional*) If False, will just execute `function` )rr accelerater)find_executable_batch_size)rr) batch_size)rrrraccelerate.utils)rrr%accelerate_find_executable_batch_sizes rrrsU  & 3!5  4lCh4hpp   X FFr c0\rSrSrSrSrSrSrSrSr Sr S r g ) FSDPOptioni: full_shard shard_grad_opno_shard hybrid_shardhybrid_shard_zero2offload auto_wraprqN) rirjrkrl FULL_SHARD SHARD_GRAD_OPNO_SHARD HYBRID_SHARDHYBRID_SHARD_ZERO2OFFLOAD AUTO_WRAPrprqr rrr:s&J#MH!L-GIr rcf\rSrSrSrS S\\S\\4SjjrS\S\4S jr S \ \4S jr S r g)RemoveColumnsCollatoriDzWWrap the data collator to remove unused columns before they are passed to the collator.N model_name descriptioncNXlX lX0lXPlX@lSUlg)NF) data_collatorsignature_columnsloggerrrmessage_logged)r[rrrrrs rr\RemoveColumnsCollator.__init__Gs(+!2 &$#r featurerc [U[5(dU$UR(dUR(aUR(a[ [ UR55[ UR5- 5n[U5S:aURcSOSURS3nURRSUSURSSRU5S SRU5S URS 3 5 S UlUR5VVs0sHupEX@R;dMXE_M snn$s snnf) Nrzin the z setzThe following columns z) don't have a corresponding argument in `z!.forward` and have been ignored: z, z. If z are not expected by `z/.forward`, you can safely ignore this message.T)rrxrrrrsetrrrdrinforr)r[rignored_columnsdset_descriptionrrs r_remove_columns%RemoveColumnsCollator._remove_columnsVs!'4((N""t{{t"3w||~#6T=S=S9T#TUO?#a')-)9)9)A2QUQaQaPbbfGg    ,-=,>?((I$))TcJdIef99_566LT__L]^;; '+#!(PA9O9O4OPPPs (E E featurescnUVs/sHo RU5PM nnURU5$s snfrY)rr)r[rrs r__call__RemoveColumnsCollator.__call__fs6AIJg((1J!!(++Ks2)rrrrrrNNN) rirjrkrlrmrryr\rxrrrrprqr rrrDsWa $(%) $ SM $ c] $QtQQ ,d,r rrreturn_is_regexc,^SnSn[U[5(a+[[R"UT55nUT:XdSOSnODTU;aSnO;[ U4SjU55(aSnO[ U4SjU55(aSnSnU(aX44$U$)aA helper method to check if the passed module's key name matches any of the target modules in the optim_target_modules. Args: optim_target_modules (`Union[str, List[str]]`): A list of strings to try to match. Can be also a full string. key (`str`): A key to search any matches in optim_target_modules return_is_regex (`bool`): If set to `True`, the method will return whether the passed `optim_target_modules` is a regex or not. Returns: `bool` : True of match object if key matches any target modules from config, False or None if no match found `bool` : If the matched target module is a regex to silence out the warnings in Trainer for extra modules being found (only if `target_module_found=True` for an array of regex). FTc3,># UH oT;v M g7frYrq)r target_keyrs rr-check_target_module_exists..s F1E:3 1Esc3d># UH%n[[R"UT55v M' g7frY)boolre fullmatch)roptim_target_modulers rrrs( jUi>QT",,2C8 9 9Uis-0)rryrrrany)optim_target_modulesrrtarget_module_foundis_regexs ` rcheck_target_module_existsrks$ H&,,"2<<0Dc#JK3s:4 $ $" F1E F F F" jUi j j j"",, r rr)NF)IrmrrrprZr*r8rrzr&typingrrrrnumpyr9utilsrr r r r r rrrrrrrrr{rrr6rrPrRrsr}rPREFIX_CHECKPOINT_DIRcompilerrrrrrrrxryrzrrrrr r rr rr1rGrrrcallablerrrrrqr rrs)    33"3S7#7$7@!;3!;t!;H4""""JZ(z( * %D#88;FG k| < (,(&j&,@tCJ'7@E@2 d3:&6  4U #3  T#u*%5$l !  <0L0:o0o0d  3glGx G>AG_cG>$,$,N$#$PT$r