eThrSrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK r SSK r SSK r SSK r SSKrSSKrSSKrSSKrSSKJr SSKJr SSKJr SSKJrJrJrJrJr SSKJr SSKJ r! SSK"r#SSK$r$SSK%J&r' SS K(J)r)J*r*J+r+ SS K,J-r- SS K$J.r. SS K/J0r0J1r1J2r2J3r3J4r4 SS K5J6r6 SSK7J8r8 SSK9J:r:J;r;Jr>J?r? SSK@JArA SSKBJCrC SSKDJErEJFrF SSKGJHrH SSKIJJrJJKrKJLrL SSKMJNrN SSKOJPrP SSKQJRrRJSrSJTrT SSKUJVrVJWrW SSKXJYrYJZrZ SSK[J\r\ SSK]J^r^J_r_ SSK`Jara SSKbJcrcJdrdJereJfrfJgrgJhrhJiriJjrj SSKkJlrlJmrmJnrnJoroJprpJqrqJrrrJsrsJtrtJuruJvrvJwrwJxrxJyryJzrzJ{r{J|r|J}r}J~r~Jr SS KJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJr SS!KJrJrJr SS"K JrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJrJr SS#KJr SS$KJr SS%KJr \d/r\gr\"5(aSS&KJr \r\"5(aSS'KJr \"5(aSSKr\"5(aVSSKJs Jr SSKJs Jr SSKJr SS KJ6r \-GR"\5\-GR"\5:r\(a SSKJ&s Jr OS(r\"5(aCSSKJs J$r SS KJ6r \-GR"\5\-GR"S)5:rSS*KkJrJrJrJr OS(r\"5(aSSKr\"5(aSS+KJr \"5(aSS,KJrJr SS KJ6r SS-KJr SS.KJrJrJrJrJrJrJr \3/Gr\-GR"\5\-GR"S/5:aSS0KGJGr \-GR"\5\-GR"S15:aSS2GKGJGr G\G\/- Gr\L"5(aSS3KGJGr \"S45(aSS5KGJGr S6GrS7GrS8Gr\(aSSGK Gr \"5(aSSKr\GR"G\ 5Gr S9Gr S:GrS;GrSGrS?Gr\"S@SA9"SBSC55Grg)Duc The Trainer class, to easily train a 🤗 Transformers from scratch or finetune it on a new task. N)Mappingpartial)Path) TYPE_CHECKINGAnyCallableOptionalUnion)#get_reporting_integration_callbacks) ModelCard create_repo upload_folder)version)nn) DataLoaderDatasetIterableDataset RandomSamplerSequentialSampler) __version__)PretrainedConfig) DataCollatorDataCollatorWithPaddingdefault_data_collator) DebugOptionDebugUnderflowOverflow)SequenceFeatureExtractor)FeatureExtractionMixin)"ALL_HYPERPARAMETER_SEARCH_BACKENDSdefault_hp_search_backend)BaseImageProcessor)deepspeed_initdeepspeed_load_checkpointis_deepspeed_available)tpu_spmd_dataloader)TrainingSummary)PreTrainedModelload_sharded_checkpoint unwrap_model)!MODEL_FOR_CAUSAL_LM_MAPPING_NAMESMODEL_MAPPING_NAMES) Adafactor get_scheduler)ProcessorMixin)ALL_LAYERNORM_LAYERS"is_torch_greater_or_equal_than_2_3)PreTrainedTokenizerBase)CallbackHandlerDefaultFlowCallbackExportableStatePrinterCallbackProgressCallbackTrainerCallbackTrainerControl TrainerState)DistributedTensorGathererEvalLoopContainerIterableDatasetShard LabelSmootherLayerWiseDummyOptimizerLengthGroupedSamplerSequentialDistributedSamplerdistributed_broadcast_scalarsdistributed_concatfind_batch_sizeget_model_param_countget_module_class_from_nameget_parameter_names nested_concat nested_detachnested_numpifynested_xla_mesh_reducereissue_pt_warningsremove_dummy_checkpointset_rng_state_for_device)PREFIX_CHECKPOINT_DIRBestRunEvalLoopOutputEvalPredictionHPSearchBackend HubStrategyPredictionOutputRemoveColumnsCollator SaveStrategyTrainerMemoryTracker TrainOutputcheck_target_module_existsdefault_compute_objectivedenumpify_detensorizeenable_full_determinismfind_executable_batch_sizeget_last_checkpoint has_lengthneftune_post_forward_hooknumber_of_arguments seed_workerset_seed speed_metrics)OptimizerNames ParallelModeTrainingArguments))ADAPTER_CONFIG_NAMEADAPTER_SAFE_WEIGHTS_NAMEADAPTER_WEIGHTS_NAME CONFIG_NAMESAFE_WEIGHTS_INDEX_NAMESAFE_WEIGHTS_NAMEWEIGHTS_INDEX_NAME WEIGHTS_NAMEXLA_FSDPV2_MIN_VERSIONPushInProgressPushToHubMixincan_return_losscheck_torch_load_is_safe find_labelsis_accelerate_availableis_apex_availableis_apollo_torch_availableis_bitsandbytes_availableis_datasets_availableis_galore_torch_availableis_grokadamw_availableis_in_notebookis_ipex_availableis_liger_kernel_availableis_lomo_availableis_peft_availableis_safetensors_availableis_sagemaker_dp_enabledis_sagemaker_mp_enabledis_schedulefree_availableis_torch_hpu_availableis_torch_mlu_availableis_torch_mps_availableis_torch_musa_availableis_torch_neuroncore_availableis_torch_npu_availableis_torch_xla_availableis_torch_xpu_availableis_torchao_availablelogging strtobool)deprecate_kwarg)requires)QuantizationMethod)NotebookProgressCallback)ampFz1.10)smp_forward_backwardsmp_forward_only smp_gathersmp_nested_concat) PeftModel) Acceleratorskip_first_batches)AcceleratorState)AutocastKwargsDistributedDataParallelKwargsDistributedTypeload_fsdp_modelload_fsdp_optimizersave_fsdp_modelsave_fsdp_optimizer1.3.0)TorchTensorParallelPluginz0.23.0)SeedableRandomSampler)DeepSpeedSchedulerWrapper0.28.0)DataLoaderConfigurationc[5(aj[4n[R"[R RS55[R"S5:a SSKJn /UQUP7n[X5$g)Npeftz0.7.0r)PeftMixedModelF) rrrparse importlibmetadatarr isinstance)modelclasses_to_checkrs L/var/www/auris/envauris/lib/python3.13/site-packages/transformers/trainer.py_is_peft_modelrs`%< ==++33F; < g@V V +B!1B>B %22 c[5(a5S[[R"[5R 5;aSS0$0$)N adapter_onlyT)rxlistinspect signaturer parametersrr_get_fsdp_ckpt_kwargsrs8  ^tGSS\\ \\4S\;4S1jjr?S2\S\;4S3jr@S4\94S5jrAS\\4S6jrBS7rCS8rDS9rESS:\\ \\ RRR44S;jjrH\ISS \S \\ S\\&\&44S<jj5rJSS4\9S=\ RBRD4S>jjrKS?\;S\94S@jrL\ISSA\;SB\\9S\94SCjj5rMSD\ SE\\\&444SFjrNSD\ SE\\\&44SG\9SH\\\O44SIjrPSJ\4SKjrQSSLjrRSSMjrSS)\ R4SNjrUSOrVSSPjrWSSQ\\ \\:4SD\ SE\\\&4S 4SR\\\4SSjjrXSSTjrYSUrZSSVjr[SWr\SXr]SSYjr^SSZjr_S[r`S\raS]rbS^rcS_rdS`reSarfSbrgScrhSSd\\SE/\\\O44Se\\\\\O4/\O4Sf\9Sg\ \\\4Sh\\ Si\i4Sj\\SE/\4S\ \j\\j44SkjjrkSSl\\\O4Sm\\OSS 4SnjjrlSo\ \ RN\&4S\ \ RN\&44SpjrmSq\\\ \ RN\&44S\\\ \ RN\&444SrjrnSsroSSt\\:4SujjrpSS \RSq\\\ \ RN\&44S\ RN4SvjjrqSSwjrrS\:4SxjrsS\:4SyjrtSSz\\S{\:4S|jjruSSz\\4S}jjrvS Sz\\4S~jjrwSrxS \yS)4S\\4SjjrzSSSjjr{SS\\ \\\\44S\\\S\S\\\O44Sjjr|SS2\S\\\S\S\}4Sjjr~SS?\;S%\S\\:S\\\S\S\4 SjjrSSjrSS \RSq\\\ \ RN\&44S\:S\\\S\\\ RN\\ RN\\ RN44 SjjrSq\\\ \ RN\&444SjrSS\\4SjjrSS\\S\\S\ \\\S 4S\\S\\S\ \\\S 4S\ \\\S 4S$\ \\\S 4S\ \\\S 44SjjrSrSrSS\\S\:S\\S\\S\4 SjjrSS?\;S%\S\\:S\\\S\S\4 SjjrSrSSjrSrSSjrSrSrS \S?\;S\94SjrSrg )Traineri>u Trainer is a simple but feature-complete training and eval loop for PyTorch, optimized for 🤗 Transformers. Args: model ([`PreTrainedModel`] or `torch.nn.Module`, *optional*): The model to train, evaluate or use for predictions. If not provided, a `model_init` must be passed. [`Trainer`] is optimized to work with the [`PreTrainedModel`] provided by the library. You can still use your own models defined as `torch.nn.Module` as long as they work the same way as the 🤗 Transformers models. args ([`TrainingArguments`], *optional*): The arguments to tweak for training. Will default to a basic instance of [`TrainingArguments`] with the `output_dir` set to a directory named *tmp_trainer* in the current directory if not provided. data_collator (`DataCollator`, *optional*): The function to use to form a batch from a list of elements of `train_dataset` or `eval_dataset`. Will default to [`default_data_collator`] if no `processing_class` is provided, an instance of [`DataCollatorWithPadding`] otherwise if the processing_class is a feature extractor or tokenizer. train_dataset (Union[`torch.utils.data.Dataset`, `torch.utils.data.IterableDataset`, `datasets.Dataset`], *optional*): The dataset to use for training. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. Note that if it's a `torch.utils.data.IterableDataset` with some randomization and you are training in a distributed fashion, your iterable dataset should either use a internal attribute `generator` that is a `torch.Generator` for the randomization that must be identical on all processes (and the Trainer will manually set the seed of this `generator` at each epoch) or have a `set_epoch()` method that internally sets the seed of the RNGs used. eval_dataset (Union[`torch.utils.data.Dataset`, Dict[str, `torch.utils.data.Dataset`, `datasets.Dataset`]), *optional*): The dataset to use for evaluation. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. If it is a dictionary, it will evaluate on each dataset prepending the dictionary key to the metric name. processing_class (`PreTrainedTokenizerBase` or `BaseImageProcessor` or `FeatureExtractionMixin` or `ProcessorMixin`, *optional*): Processing class used to process the data. If provided, will be used to automatically process the inputs for the model, and it will be saved along the model to make it easier to rerun an interrupted training or reuse the fine-tuned model. This supersedes the `tokenizer` argument, which is now deprecated. model_init (`Callable[[], PreTrainedModel]`, *optional*): A function that instantiates the model to be used. If provided, each call to [`~Trainer.train`] will start from a new instance of the model as given by this function. The function may have zero argument, or a single one containing the optuna/Ray Tune/SigOpt trial object, to be able to choose different architectures according to hyper parameters (such as layer count, sizes of inner layers, dropout probabilities etc). compute_loss_func (`Callable`, *optional*): A function that accepts the raw model outputs, labels, and the number of items in the entire accumulated batch (batch_size * gradient_accumulation_steps) and returns the loss. For example, see the default [loss function](https://github.com/huggingface/transformers/blob/052e652d6d53c2b26ffde87e039b723949a53493/src/transformers/trainer.py#L3618) used by [`Trainer`]. compute_metrics (`Callable[[EvalPrediction], Dict]`, *optional*): The function that will be used to compute metrics at evaluation. Must take a [`EvalPrediction`] and return a dictionary string to metric values. *Note* When passing TrainingArgs with `batch_eval_metrics` set to `True`, your compute_metrics function must take a boolean `compute_result` argument. This will be triggered after the last eval batch to signal that the function needs to calculate and return the global summary statistics rather than accumulating the batch-level statistics callbacks (List of [`TrainerCallback`], *optional*): A list of callbacks to customize the training loop. Will add those to the list of default callbacks detailed in [here](callback). If you want to remove one of the default callbacks used, use the [`Trainer.remove_callback`] method. optimizers (`Tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`, *optional*, defaults to `(None, None)`): A tuple containing the optimizer and the scheduler to use. Will default to an instance of [`AdamW`] on your model and a scheduler given by [`get_linear_schedule_with_warmup`] controlled by `args`. optimizer_cls_and_kwargs (`Tuple[Type[torch.optim.Optimizer], Dict[str, Any]]`, *optional*): A tuple containing the optimizer class and keyword arguments to use. Overrides `optim` and `optim_args` in `args`. Incompatible with the `optimizers` argument. Unlike `optimizers`, this argument avoids the need to place model parameters on the correct devices before initializing the Trainer. preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`, *optional*): A function that preprocess the logits right before caching them at each evaluation step. Must take two tensors, the logits and the labels, and return the logits once processed as desired. The modifications made by this function will be reflected in the predictions received by `compute_metrics`. Note that the labels (second parameter) will be `None` if the dataset does not have them. Important attributes: - **model** -- Always points to the core model. If using a transformers model, it will be a [`PreTrainedModel`] subclass. - **model_wrapped** -- Always points to the most external model in case one or more other modules wrap the original model. This is the model that should be used for the forward pass. For example, under `DeepSpeed`, the inner model is wrapped in `DeepSpeed` and then again in `torch.nn.DistributedDataParallel`. If the inner model hasn't been wrapped, then `self.model_wrapped` is the same as `self.model`. - **is_model_parallel** -- Whether or not a model has been switched to a model parallel mode (different from data parallelism, this means some of the model layers are split on different GPUs). - **place_model_on_device** -- Whether or not to automatically place the model on the device - it will be set to `False` if model parallel or deepspeed is used, or if the default `TrainingArguments.place_model_on_device` is overridden to return `False` . - **is_in_train** -- Whether or not a model is currently running `train` (e.g. when `evaluate` is called while in `train`) r )_get_learning_rate log_metricsmetrics_format save_metrics save_state tokenizerprocessing_classz5.0.0T)new_namerraise_if_both_namesNNNrargs data_collator train_datasetzdatasets.Dataset eval_dataset model_initcompute_loss_funccompute_metrics callbacks optimizersoptimizer_cls_and_kwargspreprocess_logits_for_metricsc #Uc$Sn[RSUS35 [US9nUR(a@U b=S[R "U 5R R5;a [S5eURb,URS:waUc[SURS 35eUR[R:XdUR(aURc [S 5eX lXlURR"(a[%URR&5O[)URR&5 SUlSUlS UlXlUR35 [5URR65UlUR8R;5 UR=5n[>R@"U5 URB Uc%UbXpl"URG5nO/[IS 5eUbRL"S [N5 Xpl"URPRR[T;a#[SURPRRS35e[WUSS 5(a[WUSS 5(aSUl,OS Ul,[WUSS5b[[UR\R_55Vs/sH nUS;dM UPM nn[aU5S:aSUl,OL[aU5S:Xa6URRb[dRb"US5:gUl,OS Ul,URX(a[RS5 URRf(a[i5(aSSK5J6n [oU[p5(aU"US9 Oj[sUS5(a8[oURu5[p5(aU"URu5S9 O![RwS5 O [yS5e[WUSS 5=(a [WUSS 5(+n[WUSS5SL=(a URzR|n[WUSS5SL=(a [WURzS S 5nU(a[sUS!5(a [S"5eU(a"[U5(dU(d [S#5eU(aTU(dM[S$URzRRS%URzRR35eURS&UlC[aUR5S:aYUR(a [S'5eURS&(d)UR[R:wa [S(5eURUlIURX(dfUR(dUUR(dUR(aUR(a"UR(dUR(aS UlIUb&[oU[[45(a [U5O[nUbUOUUlRX@lSXPlTX`lUUR(a:[WUS)S5[R:XdURXRb5 URX(aSURlYXlZXlURRU5n[U5(d UROURu5Rn[R "U5R n[sUS*5(aURUl_O%[S+UR_555Ul_URUlaXlbXlcU uUldUleXlfURbURb [IS,5eUb%URc URb [IS-5e[5(aURbUR0R 5HnURbn O URRH)n[aUS.5S:dMUS.SRbn O WW:wa [S/5eUR(dUR(a%URc URb [IS05e[[URR5-nU cUOUU -n [XR0URURUR5UlmURURR(a[O[5 S UlrSUlsURR(aUR5 URR(a)[R"URRSS19 [UR5(d0[[WURS2S55(a [S35eURS:a%URS:a[RS45 Ub+[U5(dURS::a [S55eUbQ[oU[dRRGR5(aUGR(a [S65eSUlS UlS UlG[ 5(GanUGR (a [S75eG[(aUGRG[GRGRGR:wa[RwS8G[GRGRGRS9UGRS:G[GRGRGR35 G[GRGRGRUlOk[sG[GRGRS;5(a?[RwS8G[GRGRGRS<35 UGR(dUGR (aUGRS=:XayURb[dRb"S>5:Xa1UGR(aG[(d [S?5eOS@Ul[RSAUGRSB35 UGR(dUGR (aUR(drG[ 5(dbUGRS@:XaSUl[dGRUlO3UGRSC:Xa"G[!5(d [ySD5eSUlURGR"S:wa%G[%URGR"SE9UlOSUlG[)5UlG[-UGR/5UGR15URGR2UGR*/-Vs/sHn[oUG[45(dMUPM snSF9UlSUlSUl[UR05(aOURGR:c7[RwSGUR0RPRRSH35 G[=UR0RP5n URGR:cU OURGR:UlG[?UR0RP5UlURGRAURUGRUGR*5UlUGRBUlS UlUR8GRI5 URGRKSIS 5UlUGRL(a~G[N(d [SJ5eG[PGRR"5n!G[TGRV"G[TGRX"G[ZGR\"G[_U!55U!S4SKSL95 UR=(a UGRL(+Ulgs snfs snf)MN tmp_trainerz1No `TrainingArguments` passed, using `output_dir=z`.) output_dircompute_resultzWhen using `batch_eval_metrics`, your `compute_metrics` function must take a `compute_result` boolean argument which will be triggered after the last batch of the eval set to signal that the summary statistics should be returned by the function.noz%You have set `args.eval_strategy` to zx but you didn't pass an `eval_dataset` to `Trainer`. Either set `args.eval_strategy` to `no` or pass an `eval_dataset`. z`args.metric_for_best_model` must be provided when using 'best' save_strategy or if `args.load_best_model_at_end` is set to `True`.Fz<`Trainer` requires either a `model` or `model_init` argumentz`Trainer` requires either a `model` or `model_init` argument, but not both. `model_init` will overwrite your model when calling the `train` method. This will become a fatal error in the next release.zThe model you have picked (a) cannot be used as is for training: it only computes hidden states and does not accept any labels. You should choose a model with a head suitable for your task like any of the `AutoModelForXxx` listed at https://huggingface.co/docs/transformers/model_doc/autois_parallelizablemodel_parallelT hf_device_map)cpudiskr rzYou have loaded a model on multiple GPUs. `is_model_parallel` attribute will be force-set to `True` to avoid any unexpected behavior such as device placement mismatching.)_apply_liger_kernel_to_instancerget_base_modelzRThe model is not an instance of PreTrainedModel. No liger kernels will be applied.zYou have set `use_liger_kernel` to `True` but liger-kernel >= 0.3.0 is not available. Please install it with `pip install liger-kernel` is_quantized_hf_peft_config_loaded hf_quantizeris_qat_trainable _orig_modzYou cannot fine-tune quantized model with `torch.compile()` make sure to pass a non-compiled model when fine-tuning a quantized model with PEFTzYou cannot perform fine-tuning on purely quantized models. Please attach trainable adapters on top of the quantized model to correctly perform fine-tuning. Please see: https://huggingface.co/docs/transformers/peft for more detailsz8The model you are trying to fine-tune is quantized with z but that quantization method do not support training. Please open an issue on GitHub: https://github.com/huggingface/transformers to request the support for training support for xlazZUsing --fsdp xxx together with --deepspeed is not possible, deactivate one of those flags.z.Using fsdp only works in distributed training.quantization_methodaccepts_loss_kwargsc3n# UH+oR[RR:Hv M- g7fN)kindr Parameter VAR_KEYWORD).0ks r #Trainer.__init__..s'1AXA'++777AXs35zSPassing both `optimizers` and `optimizer_cls_and_kwargs` arguments is incompatible.zPassing a `model_init` is incompatible with providing the `optimizers` argument. You should subclass `Trainer` and override the `create_optimizer_and_scheduler` method.paramsa[The model and the optimizer parameters are not on the same device, which probably means you created an optimizer around your model **before** putting on the device and passing it to the `Trainer`. Make sure the lines `import torch_xla.core.xla_model as xm` and `model.to(xm.xla_device())` is performed before the optimizer creation in your script.zPassing `optimizers` is not allowed if PyTorch FSDP is enabled. You should subclass `Trainer` and override the `create_optimizer_and_scheduler` method.exist_ok collate_batchzRThe `data_collator` should be a simple callable (function, class with `__call__`).zHmax_steps is given, it will override any value given in num_train_epochszThe train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.zTthe `--group_by_length` option is only available for `Dataset`, not `IterableDatasetzOSageMaker Model Parallelism does not support BF16 yet. Please use FP16 instead z(FP16 provided in SM_HP_MP_PARAMETERS is z+, but FP16 provided in trainer argument is z , setting to fp16zJ, but SageMaker Model Parallelism < 1.10 does not support FP16 in trainer.autorz2Tried to use `fp16` but it is not supported on cpucpu_ampzUsing z half precision backendapexzcUsing FP16 with APEX but APEX is not installed, please refer to https://www.github.com/nvidia/apex.)epsilon)is_local_process_zerois_world_process_zerostateful_callbacksz)No label_names provided for model class `z`. Since `PeftModel` hides base models input arguments, if label_names is not given, label_names can't be set automatically within `Trainer`. Note that empty label_names list will be used instead. xla_fsdp_v2z*FSDPv2 requires `torch_xla` 2.2 or higher.)fsdptensor) axis_names)loggerinforibatch_eval_metricsrrrkeys ValueError eval_strategy save_strategyrXBESTload_best_model_at_endmetric_for_best_modelrrfull_determinismr^seedrehp_name deepspeed is_in_trainr"create_accelerator_and_postprocessrYskip_memory_metrics_memory_trackerstartget_process_log_levelr set_verbosity_setup_devicesrcall_model_init RuntimeErrorwarningswarn FutureWarning __class____name__r-getattris_model_parallelsetrvalueslendeviceruse_liger_kernelrliger_kernel.transformersrrr)hasattrrwarning ImportErrorr is_trainablerquantization_config quant_method fsdp_configis_fsdp_xla_enabledris_deepspeed_enabled parallel_moderh DISTRIBUTEDplace_model_on_devicefp16_full_evalbf16_full_evaldo_trainis_fsdp_enabledr3rrrrrrrrBITS_AND_BYTES_move_model_to_device_n_gpu model_wrapped acceleratorr+forwardrmodel_accepts_loss_kwargsanyneftune_noise_alpharr optimizer lr_schedulerrr param_groupsDEFAULT_CALLBACKSr report_tor4callback_handler add_callback disable_tqdmr7DEFAULT_PROGRESS_CALLBACK_loggers_initialized hub_model_id push_to_hub init_hf_repo should_saveosmakedirsrcallable max_stepsnum_train_epochsrautilsdatargroup_by_length_signature_columnsuse_apex use_cpu_amprbf16IS_SAGEMAKER_MP_POST_1_10rsmpstatecfghalf_precision_backendr2bfloat16 amp_dtyperylabel_smoothing_factorr?label_smootherr:controlr;rrrr6 current_floshp_search_backend label_namesrwru on_init_endtrain_batch_size_train_batch_size_created_lr_schedulerstop_and_update_metricsgetis_fsdp_xla_v2_enabledIS_XLA_FSDPV2_POST_2_2xrglobal_runtime_device_countxsset_global_meshMeshrarrayrangeis_fsdp_xla_v1_enabled)"selfrrrrrrrrrrrrrr log_levelr=devicesr_is_quantized_and_base_model&_quantization_method_supports_training%_is_model_quantized_and_qat_trainabledefault_collatorunwrapped_model model_forwardforward_paramsparam model_device param_groupoptimizer_devicedefault_callbackscbdefault_label_names num_devicess" r__init__Trainer.__init__s& <&J KKKJ4 0 < `ASASA`A` /18~t0T\`0`1 el    2Ef - (GE;,G,Gb  (u0E0ENs$  *2XJ5K]K]KqKqK~K~JDDIDVDVDjDjDwDwCxz  $(#3#3E#: tyy>A (( p##E*t/A/A\E]E]/] !QRR&*%?%?"  " "(($$(;(;T]]''##).D & ++.EG_-`aa $$4 5'  /<.G]M]*( 0  & &E#8$?CUCdCdd  & &ukk :  ! ! DII # **77>"/22  # # //199  !**=9DD 5/ 0 0-2-F-FD *-01AOAVAVAX1.D *$(#;#; .-J*,6))(@%  ( ( 49Stu u  !t~~'ATEVEVEbj  " # #(B..0$|| 1 $~~:: {8,-1'28'>A $"7"7!"; KKb c  $Z -F-F4>>]^K^d   %=%++*:*:*J*JKK$$$st t"&   # $ $yyy !rss((99 2 22NNB399==CUCUBVWDDHII;O&&)iimm&8&8%9; !$ 2 2DI399==&11NNB399==CUCUBVWcc III(C(Cv(M{{ell511999==()]^^>3"&"<"<">!22<< ~M MrQ[\^`oQpM   !% $** % %$))*?*?*G NN;DJJ*>?26))2G2G2O.UYU^U^UjUj.tzz/C/CD,,88DJJPTP\P\] "&!6!6%*" 446&*&6&6&:&:=%&P#  & & &)) !MNN88:K   rwwrxxk0B'CkSTEUbtu v&*&>&>&btGbGbCb#] pX s, AE?:AE? AF )AF returncD[RS5 UR$)NzUTrainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.rrArrs rrTrainer.tokenizer2sno$$$rc:[RS5 Xlg)NzjTrainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead.r)rrs rrr7s x !1rc URRU5n[U5(a%URRR 5nOUR 5nAUR UlUR[5nX@l U$)z Activates the neftune as presented in this code: https://github.com/neelsjain/NEFTune and paper: https://arxiv.org/abs/2310.05914 ) rTr+r base_modelrget_input_embeddingsrXregister_forward_hookrbneftune_hook_handle)rrr embeddings hook_handles r_activate_neftuneTrainer._activate_neftune>sz **77> / * *(3399NNPJ(==?J )-)A)A & 667PQ #.  rc6[US5(d [S5eURRU5n[ U5(a%UR R R5nOUR5nURR5 U? Ag)zN Deactivates the neftune method. Make sure to call `_activate_neftune` first. rzNNeftune is not activated make sure to call `trainer._activate_neftune()` firstN) r@rrTr+rrrrrremoverX)rrrrs r_deactivate_neftuneTrainer._deactivate_neftuneQst233mn n**77> / * *(3399NNPJ(==?J   '')  *Orc:URRU5 g)a7 Add a callback to the current list of [`~transformers.TrainerCallback`]. Args: callback (`type` or [`~transformers.TrainerCallback]`): A [`~transformers.TrainerCallback`] class or an instance of a [`~transformers.TrainerCallback`]. In the first case, will instantiate a member of that class. N)r^r_rcallbacks rr_Trainer.add_callbackbs **84rc8URRU5$)a Remove a callback from the current list of [`~transformers.TrainerCallback`] and returns it. If the callback is not found, returns `None` (and no error is raised). Args: callback (`type` or [`~transformers.TrainerCallback]`): A [`~transformers.TrainerCallback`] class or an instance of a [`~transformers.TrainerCallback`]. In the first case, will pop the first member of that class found in the list of callbacks. Returns: [`~transformers.TrainerCallback`]: The callback removed, if found. )r^ pop_callbackrs rrTrainer.pop_callbackms$$11(;;rc:URRU5 g)a^ Remove a callback from the current list of [`~transformers.TrainerCallback`]. Args: callback (`type` or [`~transformers.TrainerCallback]`): A [`~transformers.TrainerCallback`] class or an instance of a [`~transformers.TrainerCallback`]. In the first case, will remove the first member of that class found in the list of callbacks. N)r^remove_callbackrs rrTrainer.remove_callback}s --h7rcURU5nURR[R:Xa#[ US5(aUR 5 ggg)N tie_weights)torrIrhTPUr@r)rrr=s rrQTrainer._move_model_to_devicesI  99 " "l&6&6 675-;X;X    Y  KK()9(:;JJ((1122STXT]T]^mTnSopyy122HI]I]IfIfHgh77 0M/8L8L3L1/M w<1 ==AYY=W">? >B>S>S>_%%77:ei ( **TYY-R-RR%!1  !/ /rF batch_size sampler_fn is_trainingdataloader_keycURn[5(a/[U[R5(aUR XS9nOUR URUS9nUUURRURRURRS.n[U[RRR5(dUb U"U5US'URRUS'URR US'U(a:[#[$URRURR&S9US'[)U40UD6n UbCURR(a([+US5(aXR,U'OXi0UlUR.R1U 5$) zACreate a [`~torch.utils.data.DataLoader`] from the given dataset.)r)r collate_fn num_workers pin_memorypersistent_workerssampler drop_lastprefetch_factor)rrankworker_init_fn_eval_dataloaders)rr|rrrrrrdataloader_num_workersdataloader_pin_memorydataloader_persistent_workersrrlrmrdataloader_drop_lastdataloader_prefetch_factorrrd process_indexrr@rrTprepare) rrrrrrrrdataloader_params dataloaders r_get_dataloaderTrainer._get_dataloaders** " "z'8;K;K'L'L11'1SG CCDDVDVdoCpM%'99;;))99"&))"I"I  '5;;#3#3#C#CDD%/9'/B!),-1YY-K-K k *37993W3W / 06=TYY-M-MTXT]T]TkTk7!"23 =+<=   %$))*Q*Qt0119C&&~6*8)E&'' 33rcURc [S5eURURSURURSS9$)a Returns the training [`~torch.utils.data.DataLoader`]. Will use no sampler if `train_dataset` does not implement `__len__`, a random sampler (adapted to distributed training if necessary) otherwise. Subclass and override this method if you want to inject some custom behavior. z+Trainer: training requires a train_dataset.TrainingT)rrrrr)rrrrrrs rget_train_dataloaderTrainer.get_train_dataloadersU    %JK K##&&"--.. $  rcUb[U5(dgURR(a[5(a1[ U[ R "5[ R"5S9$[5(aF[ U[R"5[R"5URRS9$[U5$URR(a[5(a^[!U["R$5(a?URR&UR(;aXRR&OSnOSnUR*bUR*R,SOSn[/URR0UUUS9$URR S::a [U5$g)N) num_replicasr)r rrrrr )raruse_legacy_prediction_looprrBr world_sizeglobal_ordinalrrtdp_sizedp_rankper_device_eval_batch_sizerrnr|rrrrrrrrAeval_batch_size)rrrrs r_get_eval_samplerTrainer._get_eval_sampler/sh  z,'?'? 99 / /%''3 r}}REVEVEX)**3 !$#yyCC )66 99 $ $$&&:lHDTDT+U+Uyy33|7P7PP!!=!=> >B>S>S>_%%77:ei ( ))$!1   99  1 $$\2 2rcUcURc [S5e[U[5(aUOSn[ US5(aRX R ;aCUR R(a(URRUR U5$[U[5(aURUOUbUO URnURUSUR RURUS9$)a Returns the evaluation [`~torch.utils.data.DataLoader`]. Subclass and override this method if you want to inject some custom behavior. Args: eval_dataset (`str` or `torch.utils.data.Dataset`, *optional*): If a `str`, will use `self.eval_dataset[eval_dataset]` as the evaluation dataset. If a `Dataset`, will override `self.eval_dataset` and must implement `__len__`. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. z-Trainer: evaluation requires an eval_dataset.evalr Evaluation)rrrrr) rrrstrr@rrrrTrrrr)rrrs rget_eval_dataloaderTrainer.get_eval_dataloader\s  D$5$5$=LM M*4L#)F)FF D- . ."8"88 77##++D,B,B>,RS S,,,   l +'"" ## $yy00--) $  r test_datasetcbURUSURRURS9$)a} Returns the test [`~torch.utils.data.DataLoader`]. Subclass and override this method if you want to inject some custom behavior. Args: test_dataset (`torch.utils.data.Dataset`, *optional*): The test dataset to use. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. It must implement `__len__`. test)rrrr)rrrr)rrs rget_test_dataloaderTrainer.get_test_dataloaders8## yy00-- $  rnum_training_stepscUR5 [(a@[RRR (aUR R nO UR nURXS9 g)a2 Setup the optimizer and the learning rate scheduler. We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the Trainer's init through `optimizers`, or subclass and override this method (or `create_optimizer` and/or `create_scheduler`) in a subclass. )r"rYN)create_optimizerrsrtrurvrrYcreate_schedulerrr"rYs rcreate_optimizer_and_scheduler&Trainer.create_optimizer_and_schedulersP  $ $););00II 1CYrc,[U[/SQ5nU$)a Get all parameter names that weight decay will be applied to. This function filters out parameters in two ways: 1. By layer type (instances of layers specified in ALL_LAYERNORM_LAYERS) 2. By parameter name patterns (containing 'bias', 'layernorm', or 'rmsnorm') )bias layernormrmsnorm)rHr1)rrdecay_parameterss rget_decay_parameter_names!Trainer.get_decay_parameter_namess/u6JLlmrc  [5(a URO URnURGcUR U5nUR 5VVs/sH!up4X2;dM UR (dMUPM# snnURRS.UR 5VVs/sH!up4X2;dM UR (dMUPM# snnSS./nURbURupgOURURU5upgSU;aURS5nSU;aURS5nSU;aURS5nU"U40UD6UlS[U5;Ga9URSS5S :XGa"S SKnURR R#5n S n UR%5Hn ['U [(R*5(dM$U [-U R/5Vs0sH!oDR15UR35_M# snR555- n [6R9S U S U S - S35 U R;U SSS05 [6R=SU S35 M [6R9SU S - S35 [5(a%[>R@"UR5UlUR$s snnfs snnfs snf)z Setup the optimizer. We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the Trainer's init through `optimizers`, or subclass and override this method in a subclass. N)r  weight_decayr roptimizer_dict bitsandbytes optim_bitsrzskipped : izM paramsweight zbitsandbytes: will optimize z in fp32z skipped: )!rrSrrYr.named_parameters requires_gradrr1rget_optimizer_cls_and_kwargspoprrr4optimGlobalOptimManager get_instancemodulesrr Embeddingsumrdata_ptrnumelr;rrregister_module_overridedebugrtDistributedOptimizer) r opt_modelr-npoptimizer_grouped_parameters optimizer_clsoptimizer_kwargsr4managerskippedmodules rr$Trainer.create_optimizers+B*C*CD&& >> !#==iH '0&@&@&B&BdaqG\abapap&B%)II$:$: '0&@&@&B&BdaqG`efetet&B%(  , (,,8262O2O/ /262S2STXT]T]_h2i/ ++/?/C/CH/M,**/?/C/CG/L, #33/?/C/CDT/U,*+G\K[\DN]!338H8L8L\[_8`de8e#&,,??LLN'//1F!&",,773IZIZI\']I\A aggi(?I\']'d'd'f#gg hvhb58I$RS88LZ\K]^ 'CF88%TU 2  i%'8AB " $ $ 55dnnEDN~~i J(^s* K5)K5<K5/ K;>K;K; (L cV[SURR555$)z) Get the number of trainable parameters. c3f# UH'oR(dMUR5v M) g7fr)r;rE)rrKs rr 7Trainer.get_num_trainable_parameters..s Q&=917799&=s11)rCrrrs rget_num_trainable_parameters$Trainer.get_num_trainable_parameterss"Qdjj&;&;&=QQQrcURc [S5eURRVs/sHoSPM sn$s snf)zB Returns the learning rate of each parameter from self.optimizer. PTrainer optimizer is None, please make sure you have setup the optimizer before.lrrYrr[)rgroups rget_learning_ratesTrainer.get_learning_ratess@ >> !op p)-)D)DE)Dd )DEEEsArcURc [S5eUb)URRHnXS;dM Us $ URRVs/sHo"SPM sn$s snf)z Returns optimizer group for a parameter if given, else returns all optimizer groups for params. Args: param (`str` or `torch.nn.parameter.Parameter`, *optional*): The parameter for which optimizer group needs to be returned. rYr r[)rrr\s rget_optimizer_groupTrainer.get_optimizer_groupsk >> !op p  44(O+ L5.2^^-H-HI-HEh-HIIIsA.c^^^#^$0m#TR(aJTRRSS5RS5HnURS5up4UT#U'M STR0m$TRTR 4TR S.nS|S[S [[[4S [[[4S [S [[[44 UUU#U$4S jjjnTR[R:Xa[nT$R!SSS.5 UT$4$TR[R"[R$4;aNSSKJn UnT$R!U5 TR[R$:XaT$R!SS05 UT$4$TR[R*:XaSSKJn UnT$R!U5 UT$4$TR[R2:XaSSKJn U nT$R!U5 UT$4$TR[R8:XaSSKJn U nT$R!U5 UT$4$TR[R>[R@[RB[RD[RF[RH[RJ[RL[RN[RP[RR[RT[RV[RX[RZ4;GaSSK.JnJ/n J0n Sn SnSnUnSTR;aSn STR;aSnSTR;aUnGOSTR;aU nS TRTR 40nGOfS!TR;aU nT#nGOPS"TR;Ga?[c5(aV[dRf"[hRjReS#55[dRf"S$5:a [1S%5eSS&K.J6n Un[oT#RqS'TR55[oT#RqS(TR 55[oT#RqS)S*554[oT#RqS+S,55[oT#RqS-TR 55S..nS/T#;a[sT#S/5US/'S0T#;a[sT#S05US0'S1U0nS!TR;aU US2'T$R!U5 T$R!U5 [c5(a`[dRf"[hRjReS#55[dRf"S45:a[tRwS55 UT$4$TR[Rx:XaSS6K=J>n UnT$R!U5 T$R![T#RqS7S855[[T#RqS9S:55[[T#RqS;S:55[[T#RqS.5 UT$4$TR[R:Xa[RRnUT$4$TR[R:Xa[RRnUT$4$TR[R:Xa[RR`nUT$4$TR[R[R[R[R[R[R4;Ga5[5(d [/S@5eSSAKMJNnJOnJPn [RU[RU[RU[RU[RU[RU0n[sT#RSBSC55[sT#RSDSE55[oT#RSFSG55T#RSHSI5SJ.nU"TRUU5unm$TR[R:XaT$R!SSS.5 UT$4$TR[R[R4;a[5(d [/SK5eSSLKUJVn [RU[RU0n[sT#RSBSC55T#RSMSN5T#RSOSP5[sT#RSDSE55[oT#RSFSQ55T#RSHSI5SR.nUR!U5 U"TRUU5unm$UT$4$TR[R[R4;aw[5(d [/SS5e[ST5(d [/SU5eTc [1SV5eSSWK[J\nJ]n SXTR;aUnOUnT$R!SYT05 UT$4$TR[R:Xa[5(d [1SZ5eSS[K`Jan UnT$R![oT#RqS\S]55[oT#RqS^S_55[oT#RqS`Sa55[oT#RqSbSa55[oT#RqScSQ55Sd.5 UT$4$TR[R[R4;Gay[5(aK[dRf"[hRjReSe55[dRf"Sf5:a [/Sg5e[dRf"[hRjReSh55[dRf"Si5::a [/Sj5e[dRf"[hRjReSe55[dRf"Sk5:a SSlKeJfnJgn OSSlKhJfnJgn TR[R:XaUnO,TR[R:XaUnO [1Sm5eT$R!U5 UT$4$TR[R[R[R4;GaV[5(d [/Sn5e[ST5(d [/So5eSSpKmJnnJon 0nSn!TR[R:Xa([Sq5(d [/Sr5eSSsKmJpn" U"nUnSn!OOTR[R:XaUnUnO,TR[R:XaU nO [1St5eTRUSu'U!(aTRUSv'UR![oT#RqSwS_55[oT#RqSxSy55Sz.5 T$R!U5 UT$4$[1S{TR35e![.a [1S5ef=f![.a [1S5ef=f![.a [1S5ef=f![.a [1S35ef=f![.a [1S?5ef=f)}z Returns the optimizer class and optimizer parameters based on the training arguments. Args: args (`transformers.training_args.TrainingArguments`): The training arguments for the training session.  r,=rZ)betasepsToptimizer_nameoptimizer_mapping optim_kwargsis_layerwise_supportedrcB>^UR5RS5nU(a4TR[R:XaU(a[ SUS35eXnTR c[SUS35e[TR [[45(d[STR 35eTc[SUS35e[TR [5=(a TR RS S 5S :Hn/nTR5Hup[TR US S 9up[U [R5(d+U (a"U (d[ R#USUS35 MfU (d U(dMvUR%US-5 M ['U5S:Xa[SUSTR S35eTR)5V V s/sHupX;dM U PM nn n TR)5V V s/sHupX;dM U PM nn n UR+T5 SU0SU0UE/nU(aTR,S:wa[SUS35e0mUHnU"SU/0/40TD6TU'M UHnU"SU/0UE/40TD6TU'M U4SjnTR/5H'nUR0(dMUR3U5 M) [4nTR+ST05 TR+SU05 UT4$s sn n fs sn n f)a Helper function to set up low-rank optimizers like GaLore and Apollo. Args: optimizer_name (str): Name of the optimizer. optimizer_mapping (dict): Mapping of optimizer names to their classes. optim_kwargs (dict): Keyword arguments for the optimizer. is_layerwise_supported (bool): Whether layerwise optimization is supported. Returns: Tuple[Any, Any]: Optimizer class and updated optimizer kwargs. layerwisez Layer-wise z" does not support DDP at this timez1You need to define `optim_target_modules` to use z optimizerszX`optim_target_modules` must be a list of strings, a regex string, or 'all-linear'. Got: z'You need to pass a model to initialize z optimizer._-z all-linearT)return_is_regexz matched but ignored. z only supports linear layers.z.weightrzNo target modules found for z ().r r z Layerwise z( does not support gradient accumulation!cn>URb'TUR5 TUR5 ggr)gradstep zero_grad)rr3s roptimizer_hook^Trainer.get_optimizer_cls_and_kwargs..setup_low_rank_optimizer..optimizer_hook|s4zz-&u-224&u-779.rr3)lowerendswithrIrhrJNotImplementedErroroptim_target_modulesrrrrreplace named_modulesr[rLinearrrAappendr<r:updaterrr;"register_post_accumulate_grad_hookr@)rhrirjrk is_layerwiserM all_lineartarget_params_names module_namerQtarget_module_existsis_regexrJrK target_paramsnon_target_paramsr[rrvr3rr optim_argsrNs @rsetup_low_rank_optimizerFTrainer.get_optimizer_cls_and_kwargs..setup_low_rank_optimizer,sQ$*//1::;GL 2 2l6N6N NSi)K7GGi*jkk-=M((0 #TUcTddo!pqqd77$EE nospIpIoJK} #J>JZZe!fgg444c:P--55c3?<O  #% ',':':'<# 1K--{D2.$"&"))44+H*m+A.AQQno+J#**;+BC(="&'1, #??OrRVRkRkQlln!opp+0+A+A+C`+C41qG_Q+CM`/4/E/E/G h/Gtq1Kg/G  h    +,-=9L9L 33q8$z.1AAi%jkk!#.E,9Hug;N:O,dSc,dN5)/*E,9Hug;^Q];^:_,tcs,tN5)+: #--/E***@@P0!8  '')9>(JK  # #X|$< = "22 2Aa hs* L9L L$LF)scale_parameter relative_stepr)AdamWfusedz7Trainer failed to import syncfree AdamW from torch_xla.) NpuFusedAdamWz3Trainer failed to import FusedAdamW from torch_npu.) FusedAdamzFTrainer tried to instantiate apex FusedAdam but apex is not installed!)rLionRMSpropr9Npaged8bitr6adamlionrfrmspropademamixr4z0.44.0z{The AdEMAMix optimizer is not supported by your current version of `bitsandbytes`. Please install `bitsandbytes` >= 0.44.0.)AdEMAMixbeta1beta2beta3gH.?alphag@rg)rfrrgt_alphat_beta3r5is_pagedzOTrainer tried to instantiate bnb optimizer but `bitsandbytes` is not installed!z0.41.1zYou are using 8-bit optimizers with a version of `bitsandbytes` < 0.41.1. It is recommended to update your version as a major bug has been fixed in 8-bit optimizers.)AnyPrecisionAdamWuse_kahan_summationFalsemomentum_dtypefloat32variance_dtypecompensation_buffer_dtyperx)rrrrz4Please install https://github.com/pytorch/torchdistxzYou need to install `galore_torch` in order to use GaLore optimizers install it with `pip install git+https://github.com/jiaweizzhao/GaLore`)GaLoreAdafactor GaLoreAdamWGaLoreAdamW8bitrupdate_proj_gapscaleg? proj_typestd)rrrrzYou need to install `apollo_torch` in order to use APOLLO optimizers install it with `pip install git+https://github.com/zhuhanqing/APOLLO`) APOLLOAdamWprojrandom scale_typechannel?)rrrrrrziYou need to install `lomo_optim` in order to use LOMO optimizers install it with `pip install lomo-optim`0.30.0zGYou need to have `accelerate>=0.30.0` to be able to use LOMO optimizerszMYou need to pass a `model` in order to correctly initialize a LOMO optimizer.)AdaLomoLomoadarz5Please install grokadamw with `pip install grokadamw`) GrokAdamW alpha_initg\(\?lambg@gammag?grokking_signal_decay_rategradient_clipping)rrrrrtorchaoz0.4.0zYou need to have `torchao>=0.4.0` in order to use torch 4-bit optimizers.Install it with `pip install torchao` or follow the instructions here: https://github.com/pytorch/aorz2.4zYou need to have `torch>2.4` in order to use torch 4-bit optimizers. Install it with `pip install --upgrade torch` it is available on pipy. Otherwise, you need to install torch nightly.z0.11.0) AdamW4bit AdamW8bitzInvalid optimizerzwYou need to install `schedulefree` in order to use schedulefree optimizers. Install it with `pip install schedulefree.`zOYou need to have `accelerate>=0.30.0` to be able to use schedulefree optimizers)AdamWScheduleFreeSGDScheduleFreerzYou need to install `schedulefree>=1.4.0` in order to use RAdamScheduleFree optimizer. Install it with `pip install schedulefree.`)RAdamScheduleFreezInvalid schedulefree optimizerr1 warmup_stepsweight_lr_powerrr2)rrz2Trainer cannot instantiate unsupported optimizer: T)srr|split learning_rate adam_beta1 adam_beta2 adam_epsilonrdictrbooltupler>rg ADAFACTORr.r ADAMW_TORCHADAMW_TORCH_FUSED torch.optimrADAMW_TORCH_XLAtorch_xla.amp.syncfreerBrADAMW_TORCH_NPU_FUSEDtorch_npu.optimrADAMW_APEX_FUSEDapex.optimizersr ADAMW_BNB ADAMW_8BIT PAGED_ADAMWPAGED_ADAMW_8BITADEMAMIX ADEMAMIX_8BITPAGED_ADEMAMIXPAGED_ADEMAMIX_8BITLION LION_8BIT PAGED_LIONPAGED_LION_8BIT RMSPROP_BNB RMSPROP_8BIT RMSPROP_32BITbitsandbytes.optimrrr{rrrrrfloatrintrrAADAMW_ANYPRECISIONtorchdistx.optimizersrrr8rSGDADAGRADAdagradRMSPROP GALORE_ADAMWGALORE_ADAMW_8BITGALORE_ADAFACTORGALORE_ADAMW_LAYERWISEGALORE_ADAMW_8BIT_LAYERWISEGALORE_ADAFACTOR_LAYERWISEr} galore_torchrrrr= APOLLO_ADAMWAPOLLO_ADAMW_LAYERWISErz apollo_torchrLOMOADALOMOrrx lomo_optimrr GROKADAMWr~ grokadamwrADAMW_TORCH_4BITADAMW_TORCH_8BITr torchao.optimrrtorchao.prototype.low_bit_optimSCHEDULE_FREE_RADAMSCHEDULE_FREE_ADAMWSCHEDULE_FREE_SGDr schedulefreerrrr1r)%rrmappingkeyvalue adam_kwargsrrMrrrrrrr5additional_optim_kwargsr bnb_kwargsrrrrrigalore_optim_kwargsrapollo_optim_kwargsrrrrrrrrequire_warmuprrrNs%`` @@rr<$Trainer.get_optimizer_cls_and_kwargss  ????223;AA#F$]]3/ "' 3G!$"4"45oot7$$ ,0 ] 3] 3#CH~] 3sCx.] 3%) ] 3 38_ ] 3] 3~ ::11 1%M  # #PU$V Wx ...w ZZN668X8XY Y )!M  # #K 0zz^=== ''$8j ...i ZZ>99 9 \8 %  '' 4^ ...Y ZZ>?? ? X9 -  '' 4N ...I ZZ>:: : k5 )  '' 4~...yZZ  $ $  % %  & &  + +  # #  ( (  ) )  . .     $ $  % %  * *  & &  ' '  ( (  "8 tCC  $ *5'djj(#HTZZ'!"JTZZ'$)Mtzz)$(M/6$//8Z.[+$**,$+M/9+4::-022w}}!**22>B8 h/80)G <$,M"*..$//"JK!*..$//"JK!*..&"AB" "'z~~gs'C!D$Z^^E4;L;L%MN/+!J.=@IAV=W/ : J.=@IAV=W/ :*J7 DJJ.-5Jz* ''(?@ '' 3)**w}}""**>:0 h'0(r^...WZZ><< < YC 1  '' 4!''/8H]_f9g/h*1%HXZc9d*e*1%HXZc9d*e5<!:>>2Mz#Z6  F...oZZ>-- -!KKOOMl...kZZ>11 1!KK//Mh...gZZ>11 1!KK//Md...cZZ  ' '  , ,  + +  1 1  6 6  5 5   -..!_ S R++[00///55{::O99? ! JNN6378#&z~~6G'M#Nz~~gt<='^^K? # /G -/B/ +M+zz^<<< ''ETY(Z[\...[ZZ  ' '  1 1  -..!^ 1++[55{!  JNN6378"vx8(nn\9E#&z~~6G'M#Nz~~gs;<'^^K? #  & &{ 3.F -/B/ +M+f...aZZN//1G1GH H$&&!@+844!"kll} !pqq 0 " ' $  # #We$4 5z...yZZ>33 3)++ !XYY +%M  # #"' |T(J"K!*.."=>":>>'3#?@27 Gceh8i2j).z~~>QSV/W)X  j...YZZ  + +  + +  ())W]]9;M;M;U;UV_;`-adkdqdqe."{}}Y//77@AW]]SXEYY!K}}Y//77 BCw}}U]G^^>>Pzz^<<< ) ~>>> )  !455  # #K 0b...aZZ  . .  . .  , ,  -..!B+844!"stt G&( #!Nzz^???099%F; 1 *5'!&~AAA 1 *5'~??? /  !ABB6:6G6G #N 3:>:K:K'7 # * *',Z^^?{;{67|H|(2B"}{36| |%(|>}rYcURci[URRUc UROUURR U5UURR S9UlSUlUR$)z Setup the scheduler. The optimizer of the trainer must have been set up either before this method is called or passed as an argument. Args: num_training_steps (int): The number of training steps to do. )rYnum_warmup_stepsr"scheduler_specific_kwargsT)rZr/rlr_scheduler_typerYget_warmup_stepslr_scheduler_kwargsrr&s rr%Trainer.create_schedulersq    $ - ++,5,=$..9!%!;!;r< NameErrorAttributeError TypeErrorrper_device_train_batch_size)rrrs r num_examplesTrainer.num_examplessz  K ((G'#788:--5566z))* *>95 Kz?TYY%J%JJ J Ks?AA7BBtrain_dlrjcSnUH#nUSR5nUbXA-s $X$- nM% U$![a [RS5 U$f=f)za Helper to get number of tokens in a [`~torch.utils.data.DataLoader`] by enumerating dataloader. r input_idsz%Cannot get num_tokens from dataloader)rEKeyErrorrrA)r&rj train_tokensbatchtokenss r num_tokensTrainer.num_tokenssq   D!{+113(!--& " D NNB C Ds //AAtrialz optuna.Trialc XlURbUcgUR[R:XaUR U5nOUR[R :XaUnUR SS5 OUR[R:XaPURR5VVs0sH(up4U[U[5(a [U5OU_M* nnnO UR[R:XaUnWR5H|upV[URU5(d[ R#SUS35 M;[%URUS5nUb['U5"U5n[)URXV5 M~ UR[R:Xa"[ R+SUR,35 UR[R:Xa"[ R+SUR35 UR[R:Xa[ R+SU35 UR.(aURR0c [3S5eUR4R75 S S KJn S S KJn U "URR05URl URR@RCUR5 U"URR@S 9URl"[G5RI5 URK5 gs snnf) zHP search setup codeNwandbzTrying to set zY in the hyperparameter search but there is no corresponding field in `TrainingArguments`.zTrial: zSigOpt Assignments: zW&B Sweep parameters: z7For sweeps with deepspeed, `args.deepspeed` must be setr)DeepSpeedPluginHfTrainerDeepSpeedConfig) hf_ds_config)&_trialr~rTOPTUNAhp_spaceRAYr=SIGOPT assignmentsitemsrrrWANDBr@rrrAr8rsetattrrr rHr(rrT free_memoryaccelerate.utilsr2#transformers.integrations.deepspeedr4hf_deepspeed_configtrainer_config_processdeepspeed_pluginr _reset_stater*) rr/r rvrrold_attrr2r4s r_hp_search_setupTrainer._hp_search_setups  ! ! )U]   ! !_%;%; ;]]5)F  # #':': :F JJw %  # #'='= =INIZIZI`I`IbcIba:a#5#5Q1Q.R!$!5!5!D!DEX!YJ'+~~ $    @ /.;/.s A:G GrMc[RRU[SURR 35nUR USS9 URR(GaURR 5URRS'URR[RRU[55 [R"URR!5[RRU["55 [R"UR$R!5[RRU[&55 gg)NroT_internal_callr:)rgpathrrPru global_step save_modelrrfr|r save_to_jsonTRAINER_STATE_NAMErsaverY state_dictOPTIMIZER_NAMErZSCHEDULER_NAME)rrMrs rr[Trainer._tune_save_checkpointLsWW\\.5J4K1TZZMcMcLd2ef   48 99 >Bll>P>P>RDJJ ) )*: ; JJ # #BGGLL=O$P Q JJt~~002BGGLL^4\ ] JJt((335rww||JP^7_ ` !rc[UR5nUS:XaUR5nO#US:XaURU5nO [S5eUc [S5eU$)Nrr z'model_init should have 0 or 1 argument.z"model_init should not return None.)rcrr2)rr/model_init_argcountrs rr1Trainer.call_model_initVs]1$//B ! #OO%E A %OOE*EHI I =CD D rc >U(GdUc[RS5 U$[[U55nUR U5n[ R "U5nUR 5 URRSS5nU(aXel [SS9nURRUS9 [R"5 [U[ 5(a[R"R%XTSS9nO2[R"R%UUVs0sHoXH_M snSS9nSSS5 SSS5 [R"R'U5n[R"5 U"S 0UD6 U"S 0UD6 SSS5 UnSUlU$U$s snf!,(df  Nu=f!,(df  N~=f!,(df  NB=f![*[,[.[0[24a$n [RSU S35 Sn A U$Sn A ff=f) NzAfailed to use PyTorch jit mode due to current dataloader is none._original_forwardF) cache_enabled)autocast_handler)example_kwarg_inputsstrictz'failed to use PyTorch jit mode due to: .r)rrAnextiter_prepare_inputsrPr__dict__r=rUrrTautocastrno_gradrrjittracefreezerqr2r"rr  IndexError) rrrtraining example_batch jit_modeloriginal_forwardrvres rtorch_jit_model_evalTrainer.torch_jit_model_evalds!bc  j!12M 00?M O IIe,  #,#5#5#9#9:Mt#T #(8%#1#F %%..@P.QSXS`S`Sb!-66$)IIOOIjoO$p $)IIOO%Ub1cUbc}7I2IUb1c#(%4% TcQ"II,,Y7 ]]_. .. .%"#(  u 2d TcSbQQ%_ !)ZJO O!H1MNN  Ost A1G:F7AF&#F!1F&7F7?> $(MMdnndRV%2% !E> rc |SSSS.nSnSnUR5H?upg[XS5n[X'S5n UcM"U cM'X:wdM.USUSUS U S 3- nS nMA URn UR[ S UR 5-n X:waUS U S U S 3- nS nU(a[ RU5 gg)N logging_steps eval_steps save_steps)rrrFztWarning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: z r7z (from args) != z (from trainer_state.json)Tr z per_device_train_batch_size: )r<r8r#rmaxn_gpur warning_once) r training_args trainer_stateattributes_map has_warning warning_strarg_attr state_attr arg_value state_value train_bs_argstrain_bs_states r#compare_trainer_and_checkpoint_args+Trainer.compare_trainer_and_checkpoint_argss,&&   M $2$8$8$: H >I!-TBK$)@YE]hZr)}oM]^l]mnHI IKK     , rc ^^^^TRR(aATR(a[RO[R nTR XUS9n[5(ah[TR[RR5(a TR$[R"UTRRS9$TRRUSS9ULaU$TR (aCU(a<["R$"UTR&TRR(S9unTlTRR*S:a([-USS5(d[.R0"U5nTRR2(aN[4R4"5nTR7XU5n[9[4R4"5U- S5TlU(dU$TR<(GaS S KJ m S S KJ!m S S K"J#nJ$n TRJ(aS S K&J'm SnSn [-USS5n TRRRRUSU 5n TRRRSS :a-[VRX"UTRRRSS9nOUU bR[[5n U H-n []X5nUc [_S5eU RaU5 M/ [VRX"UU S9nTRRbnTRRRS(aJURdRf(a&[hRkS5 SURdl3UUUU4Sjn TRJ(aSnT"UUUU S9=Tl nOT"U4UU S.UD6=Tl nS04SjnU[ll7U$[q5(a>[.RrRuU[w[xRz"S55/S9nU$TRR|[~R:Xa[5(aU$0nTRRbTRRUS'O/[U[5(aUR(+US'OSUS'TRRbTRRUS 'TRRbTRRUS!'[S"0UD6TRlHU$![Pa [QS5ef=f)#N)r)backward_passes_per_stepFkeep_torch_compile) opt_levelr is_loaded_in_8bitr)XlaFullyShardedDataParallel)checkpoint_module)size_based_auto_wrap_policytransformer_auto_wrap_policy)SpmdFullyShardedDataParallelzJMissing XLA FSDP related module; please make sure to use torch-xla >= 2.0._no_split_modulestransformer_layer_cls_to_wrapmin_num_params)rz@Could not find the transformer layer class to wrap in the model.)transformer_layer_clsxla_fsdp_grad_ckptzX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.cR>TR(dTOTnU"T"U5/UQ70UD6$r)r)mrkwargs target_clsFSDPFSDPv2rrs rauto_wrapper_callable2Trainer._wrap_model..auto_wrapper_callables--1-H-HfJ%&7&:LTLVLLrcSSKJn Sn[U[R5(aUnO7[U[ 5(aUSnO[X5(a UR nUc [S5e[R"X1S5 g)Nr )CausalLMOutputWithPastrzASomething went wrong, the output of the model shouldn't be `None`)rNN) modeling_outputsrrrTensorrlogitsrr mark_sharding)outputmeshr real_outputs r shard_output)Trainer._wrap_model..shard_outputsmH"&K!&%,,77&, #FE22&,Qi #FCC&,mm "*()lmm$$[8LMr)rauto_wrap_policyr)rrcbUR"S0UD6nU(a[R"5 U$)Nr)rtxm mark_step)rYbarrieroptimizer_argslosss rpatched_optimizer_step3Trainer._wrap_model..patched_optimizer_step4s$ ~~77LLN rSMDATAPARALLEL_LOCAL_RANK) device_idsfind_unused_parametersT bucket_cap_mbbroadcast_buffersr)Iruse_ipexrqrrxrrrrrSrtrDistributedModelrrTr+rpr initializerYfp16_opt_levelrr8r DataParallel jit_mode_evaltimerroundjit_compilation_timerGtorch_xla.distributed.fsdprrtorch_xla.distributed.fsdp.wraprrr7torch_xla.experimental.spmd_fully_sharded_data_parallelrrBrFr functoolsrr:rG Exceptionaddxla_fsdp_configconfig use_cacherrroptimizer_steprparallelDistributedDataParallelrrggetenvrIrhrJrddp_find_unused_parametersr)is_gradient_checkpointingddp_bucket_cap_mbddp_broadcast_buffersr ddp_handler)rrrrr start_timerrrr%default_transformer_cls_names_to_wrap"fsdp_transformer_layer_cls_to_wraptransformer_cls_to_wrap layer_classtransformer_cls fsdp_kwargsrrrrrrs` @@@r _wrap_modelTrainer._wrap_models 99  &*&6&6ENNEMME,,UE,JE " $ $$,,cii.H.HII)))'' HmHmn n    ( (5 ( IQV VL ==X$'NN5$..TXT]T]TlTl$m !E4> 99??Q wu6I5'Q'QOOE*E 99 " "J--eJE(-diikJ.F(JD %L  # # # pZH .. $ $( !4;ECVX\4] 1151F1F1J1J/1V2 .yy$$%56:#,#4#4/ @U@UVf@g$ 4?*-%'#EK&@&TO&.'(jkk/33OD $F$-#4#40*A$ ))33Kyy$$%9:<<))''r.3ELL*MM ** N&,!-%5*? & U&*&%5*?&" & U;@PR  !7B 4 3% & &KK773ryy1L'M#N"O8E0 +YY $ $ (@(@ @,.. Fyy33?37993W3W/0E?338=7V7V3V/037/0yy**6*.))*E*E'yy..:.2ii.M.M*++H+R6+RD   ( U p!"noo ps +UUresume_from_checkpointignore_keys_for_evalc USLaSnURR5 URnSUlURb UR UR 5UlUR(dUR(aUUR(dDUR(d3URc&URUR UR5 SU;a,URS5n[R "S["5 [%U5S:a5['SSR)[+UR-555S 35eUR/U5 URR0UlSnURbURR4(a[7URR85O[;URR85 UR=U5UlSnS uUlUl [CU[D5(a8U(a1[GURH5nUc[KS URHS 35eUb[M5(d3URN(d"URP(dURSU5 [TRV"[XRZR)U[\55nUR0bUR0UlU(aHUR^(a&URUR UR5 UR Ul0[cURdUR2URf5nURh(a4[jRl"5 U"UUUUS 9[jRn"5 $U"UUUUS 9$![jRn"5 f=f)a Main training entry point. Args: resume_from_checkpoint (`str` or `bool`, *optional*): If a `str`, local path to a saved checkpoint as saved by a previous instance of [`Trainer`]. If a `bool` and equals `True`, load the last checkpoint in *args.output_dir* as saved by a previous instance of [`Trainer`]. If present, training will resume from the model/optimizer/scheduler states loaded here. trial (`optuna.Trial` or `Dict[str, Any]`, *optional*): The trial run or the hyperparameter dictionary for hyperparameter search. ignore_keys_for_eval (`List[str]`, *optional*) A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions for evaluation during the training. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments used to hide deprecated arguments FNT model_pathzi`model_path` is deprecated and will be removed in a future version. Use `resume_from_checkpoint` instead.rz*train() got unexpected keyword arguments: rryrz/No valid checkpoint found in output directory ())rrr/r)8r,r-rr)rXrrrLrMrNr9rrQr=r=r3r4r5r<r"rrrrHrrr%r^r&rer1rYrZrrr`rrrrHrO_load_from_checkpointr;load_from_jsonrgrfrjrKrSr__inner_training_loopauto_find_batch_sizerd hf_hub_utilsdisable_progress_barsenable_progress_bars) rrr/rrrmodel_reloadedruinner_training_loops rr\ Trainer.trainVs. "U *%) " ""$yy  # # /// ;DJ  D$7$7MM**'  & &tzz4;; ? 6 !%+ZZ %= " MM  v;?HSWX^XcXcXeSfIgHhhijk k e$!%!;!; ?? &7;yy7Q7Q #DIINN 3W_`d`i`i`n`nWo--e4DJ!N0: -DND- ,d 3 38N%8%I "%- #RSWSbSbRccd!eff ! -*,,T5N5NW[WkWk**+AB // =SUg0hiE%%1).)?)?& ))**4::t{{C!%D 8  % %t'='=t?X?X     4224*+A)= 113&'=%9  113s >N::Oc2URR5 XlURR(aUR R UR:waSSKJn U"UR5uUl URUl UR(asURRnUR[SURR5-URl URS5 XpRl URUR l[ R#SUR35 UR%5nUR&(a [)U5nURUR*-UR,-n UR/X(U 5un n n n nnnSnURR0(aCUR3X(aSOU5nUbU(aUUR4-nOUUR*-n[6R8URR";a:URRS:a [;S5e[=UR5n[?5=(d UR@=(d URBnURB=(a. [EURR RFSS5S:HnU(aS nURH(aSUl%S Ul$UR(a[MUUS 9uUl'Ul%U(dURQUS 9 [SURTRVURX/-Vs/sHn[[U[\5(dMUPM snS 9UlUSLUR l/URUR lUR RaUU5 URb(a#URReURfS 9 URiUR5nUURLaSOS nU(a*URB(a[kURSS 9Ul U(ajU(aTURm5 URRnS:wa*URRqUR5Ul URQUS 9 U(aURRs5 [uURJS5(apURv(a&URRqUR5nOURRqURURN5unUl'OURRqURURNURJ5unUl'Ul%ObURRx[zR|[zR~4;a*URRqURN5Ul'URB(aU=Ul Ul UURLaUUl UR(aURUl@UbzUR(a.[URU[UR5(+S9 O;[?5(dURB(aURX0R5 URU5 URU5 [ RS5 [ RSU S35 [ RSU S35 [ RSURRS35 URRUR:wa#[ RSURS35 [ RSU S35 [ RSUR*35 [ RSUS35 [ RS[USS9S35 SUR lH[R"5nSnSnSnUGb[RR[RRU[55(GaW[RR"[RRU[55UlURURUR 5 UR5 [UR RU -5nUR(d)UR RU -nUUR*-nOSn[ RS5 [ RSU35 [ RSUR R35 UR(d[ RSUS US!35 S"H$n[URTU[EUU55 M& XRTlVUR RUUX5 [R"S#URS$9nS#Ul[UR RUl\UR5 SnSnURTRX R URX5Ul,UR(aURXESS%9 [UU 5G H}n Un![uU!S&5(aU!RU 5 URS:aSUldUb [U!5OURUR*-n"URTRX R URX5Ul,U U:XaUbUS:XaURU5 S n#Sn$US:a[U!U5n!Un$SnSn#S'n%[U!5n&U"UR*-n'U'S:Xa UR*n'S'n(U"UR*-[U'UR*:5-n)[U)5GHn*U(S- n(U(U)S- :wa UR*OU'n+URU&U+UR5un,n-[U,5GH_un.n/U%S- n%U%S-UR*-S:H=(d U%S-U":Hn0URRRU05 URR(a[EURS(S)5n1U1U/;a[ RS*5 OU/U1R5n2[R"U2URR[RS+9n2UR =RURRU25R5R5- slsU#(aURU5 S n#US:a3US-nUbURS5 US:XaURU5 GMUbUR5 SnU%UR*-S:Xa5URTRX R URX5Ul,U.[U,5S- :waQURR[R:wa)[R"URRUS,9OG[GRn3U3"5 UGRUU/U-5n4SSS5 UGR(auG[ 5(de[GR "W45(d[GR "U45(a-UUSUR R-UR- - -nODURW4R:wa%[;S-URS.U4R35eUU4-nU=GRG[UGRU/55- slU0(Ga:URRRS5 UGRGb_UGRS:GaM[?5(a:UGR(a(URNGRUGR5n5OURv(aOG[GRGRG[ GR""URN5UGR5n5O7URGRUGR%5UGR5n5G['5(a[URR[R:Xa3UGR)5n[uUS/5(aUR5nOU5nURTGR+X R URX5Ul,URNGR-5 URTGR/X R URX5Ul,UGR15nURGR2(dY[[URJ[RxRJGR45(dURJGR-5 UR5 UR =RS- slSU U%S-U$-U"- -UR lHURTGR7X R URX5Ul,UGR9UUUUU UUUS09 O6URTGR;X R URX5Ul,URXGR<(dURXGR>(dGM9G[ 5(aG[@GRB"5 O URXGR<(dURXGR>(dGMG[ 5(aG[@GRB"5 O U%S:aA[ RS1UR RS2US335 SURXlURTGREX R URX5Ul,UGR9UUUUU UUUS09 [6GRFURR";aSG[ 5(a.G[@GRH"G[JGRL"55 O[ RS45 URXGR>(dG M~ O UR(a[uUS55(a G[OUS55 [ RS65 UGRP(aUR GRRbG[ 5(aG[@GRT"S75 O_UGRVG[XGRZ:XaG[\GR^"5 O&[?5(aG[`GR^"5 UGRc5 U=RUR5- sl[[UR RS85n6URU6- n7G[eS9UU UR RUS:9n8UGRg5 UR GRhU8S;'U7U8S<'S UlUGRlGRoU85 UGRqU85 UGRsU5n9UGRuS U9S=9n:URGRv(aUR GRRbURGRxS:XavU:Hpn;[RGR{U;UR GRR5(aM@[ RS>U;S?35 G[|GR~"U;SS@9 Mr URTGRX R URX5Ul,UGR5 UGRbUGRUR5 G[UR RU7U85$s snf!,(df  G N=f)ANr)release_memoryr Tz)Currently training with a batch size of: zCurrently --debug underflow_overflow is not supported under DP. Please use DDP (torchrun or torch.distributed.launch (deprecated)). fsdp_versionF)r")r)gradient_checkpointing_kwargs) recursivefp8rtload_module_strictz***** Running training ***** Num examples = rdz Num Epochs = z( Instantaneous batch size per device = zA Training with DataParallel so batch size has been adjusted to: zE Total train batch size (w. parallel, distributed & accumulation) = z Gradient Accumulation steps = z Total optimization steps = z# Number of trainable parameters = )trainable_onlyzE Continuing training from checkpoint, will skip to saved global_stepz! Continuing training from epoch z' Continuing training from global step z Will skip the first z epochs then the first z batches in the first epoch.)rrYrZr2r=)skip_scheduler set_epochmain_input_namer(zTried to track the number of tokens seen, however the current model is not configured properly to know what item is the input. To fix this, add a `main_input_name` attribute to the model class you are using.)r=rrz0Calculated loss must be on the original device: z but device in use is item)rzXThere seems not to be a single sample in your epoch_iterator, stopping training at step zI! This is expected if you're using an IterableDataset and set num_steps (z.) higher than the number of available samples.zYou enabled PyTorch/XLA debug metrics but you don't have a TPU configured. Check your training configuration if this is unexpected._pastzU Training completed. Do not forget to share your model on huggingface.co/models =) r#gMbP?r\) num_samples num_stepsr- total_flos train_loss use_mtimerDeleting older checkpoint [] due to args.save_total_limit ignore_errors)rTr?rrrrurr@rrSrrHr#rrpropagate_args_to_deepspeedrrGr rr'rrset_initial_training_valuesinclude_tokens_per_secondr-rkrUNDERFLOW_OVERFLOWrrrrGrOr8 fsdp_pluginrrZr$rYr'r;r^rr|rr6is_hyper_param_search compute_stepsgradient_checkpointinggradient_checkpointing_enablerrr+_fsdp_qlora_plugin_updatesmixed_precisionrr\r@rpr>rgrrr(r%rr_load_optimizer_and_scheduler _load_scalerrrFepochrrgrfisfilerrjrr_load_callback_staterrgignore_data_skipr>train_dataloaderinit_training_referencesrrr=_total_loss_scalar_globalstep_last_loggedruon_train_begin eval_on_start _evaluaterr past_indexrr<rjon_epoch_begin_load_rng_staterr{get_batch_samples enumerategradient_state_set_sync_gradientsinclude_num_input_tokens_seenrArEint64num_input_tokens_seengatherrCrrclose on_step_begindistributed_typer DEEPSPEEDrrno_syncrr training_steplogging_nan_inf_filterrisnanisinfr}rfloating_point_ops max_grad_normrclip_master_gradsrrlclip_grad_norm_r master_paramsrrxget_global_grad_normon_pre_optimizer_steprton_optimizer_steproptimizer_step_was_skippedReduceLROnPlateau on_step_end_maybe_log_save_evaluateon_substep_endshould_epoch_stopshould_training_stoprr on_epoch_endTPU_METRICS_DEBUG master_printmetmetrics_reportdelattrr#best_model_checkpoint rendezvousrIrhrJdistrrt_load_best_modelrf store_flosr"r)r,rlog_get_output_dir_sorted_checkpointsrfsave_total_limitsamefileshutilrmtreerV_finish_current_pushrXrrZ)tXa0b -DND-'  / /9 / M!!22<< ~M MrQ[\^`oQpM   ,1+< (&*&<&< #   y1  & & JJ 4 4SWSuSu 4 v  !3!34 +04::*=$5 "t';';&djjDADJ #&//1##33u<@;K;K;S;SJJ0A0A<8t~t'8YY__!4!4n6L6L M M!--55dnnEDN   .3 3DJ+  "!&D   $ $!//DN " -(()&&(>WefjfpfpWqSq)**d.B.B**+ACUCUV **+AB 01  23 ' Q'789 o&6q%9:; >tyy?d?def>ghi 99 0 0D4J4J J KK[\`\r\rst[uv w [\rst[uvw 6t7W7W6XYZ 3Ia=AB 9:OPUfj:klm9nop YY[ )*&%)" " -"''.. GGLL/1C D3 3 &44RWW\\BXZl5mnDJ  4 4TYY K  % % ' !7!7;U!UVN((151G1GKe1f..$2R2RR.12. KK_ ` KK;N;KL M KKA$**BXBXAYZ [(( ,^,<=677SU ;D D))4t1D E;1A. ++D)=MU,,s4;;7"%'+zz'='=$ %)  ,,;;D**dll[    NN5tN L>+; 12." D!"23N&)I)IIIA~ << K*d.N.NNQTD<<<RM=)q BMR_bcRcBdd>>js 484J4J>[fhlhshs4t1 1!*=!9IAvAID$(1H0P0P#PTU#U#uZ^abZbguYuL$$33GG Uyy>>*1$**>OQ\*]*&8"NN!b ,2/+B+H+H+JL+0<< TYYM]M]ejepep+qL JJ<<@P@P@W@WXd@e@i@i@k@p@p@rr<",,-CD&+ 696!;65A6==a@9Q> 001GH 3?288:592d>>>!C'+'<'<'J'J4Q[Q[]a]i]i'j M 2Q 66 ,,==AZAZZ"))$*:*:*B*B%P(33  !'+'9'9%I['\ #333 6 8 8"[[66%++l:S:S#*Gq4::;Q;Q7QTXTpTp7p,q"q">>\-@-@@","RSZSaSaRbbxzFzMzMyN!O##*L"8%%t/F/Fv/N)OO%#((77KKDQ --9d>P>PST>T688TYYY-1^^-M-MdN`N`-a !%-/XX-E-E$'$5$5dnn$E$($6$6." .2-=-=-M-M$)$4$4$6$($6$6." !8 9 9$($4$4$E$EIbIb$b,1,F,F,H #*9f#=#=090@I,6 '+'<'<'R'RSWYcYceieqeq'r ++-'+'<'<'N'NtU_U_aeamam'n )-(?(?(A #//JJJ#-d.?.?AYAYAkAk#l#l $ 1 1 6 6 8) ..!3.+0D1H}4LP^3^+^ ('+'<'<'H'Hzz[_[g[g'h 55#%!!!0&*76 (,'<'<'K'KDR\R\^b^j^j'k ||5559Z9Z9Z133LLNG":J<<111T\\5V5V5V-// Y*Zax ../0##,+-[] 59 100==dJJPTP\P\]DL  ) )E5%9Mziv * ,, ?)++OOC$6$6$89NN_||000S=V ??wtW55 D' " op  & & &4::+K+K+W%'' 67##|'?'?? (**  ! ! # 7<<>1 #DJJ$:$:E B,,/DD   )jj**'    $ 5 5  *   44W= &&u-!55RY5Z 99 TZZ%E%E%QVZV_V_VpVptuVu0 ww'' DJJ4T4TUUKK"=j\Ig hiMM*DA1 ,,99$ DLLY  !!#  # # /  $ $TZZ 04::11:wGGe  D#sAd4Ad4 Ad9 d9 Ae cURGb6UGb2UR[R:Xa URnOUR[R:Xa-SSKnUR R5R5nOcUR[R:Xa URnO8UR[R:XaSSK nURRnURbURU5OSW3n[R R#UR$R&U5nU$UR$R&nU$)Nrzrun-)r~rTr7numberr9rXr\ get_context get_trial_idr:idr=r1runr'rgrfrrr)rr/run_idr_r1run_namers rrqTrainer._get_output_dir s  ! ! -%2C%%)?)??''?+>+>> ..0==?''?+A+AA''?+@+@@.2ll.Ft||E*dSYRZOHggll499#7#7BGii**Grc ^Uc URn[RRT[5n[RRT[ 5n[RRT[ 5n[RRT[5n[RRT[5n[RRT[5n[RRT[5n [RRT5=(at [U4Sj[R"T555=(dD [RR[RRT[S355n [RRT5(a[R"T5V s/sHn [RR[RRTU 55(dMG[RR[RRTU [ 55(dI[RR[RRTU [ 55(dMU PM sn O/n U (a UR (d[#STS35e[SUUUU UU455(dU (dU (d[#ST35e[$R'STS35 [RRU5(aO[(R*"U5n U R,nUb*U[.:wa [$R1S US [.S 35 [RRU5(d,[RRU5(dU (Ga[35(a[RR[RRTS 55(a[4R6"T[S S S9 g[9UR:S5(a.UR:R<SLa[$R1S5 [?5 [@RB"USSS9nS US'UREUSS9nAgUR (a@[GURHRJRLURHUT40[O5D6 gUR:RP(aC[RRU5(a[RR@RUUSS9nO [?5 [@RB"USSS9nUREUS 5nAURWU5 g[YU5(GaG[9US5(d[9US5(Ga[9US5(a[RR[T5(a[9US5(a6UR\n[_U5S:a[$R1S5 USnO UR`nU (aOU H7n[RRTU5nURcUUUU:HS9 M9 UReU5 gURcTUSS9 g[$R1S[ S35 g[$R1S 5 g[gUT[35UR:RPS!9n[35(dURWU5 ggs sn f)"Nc3># UHRn[RR[RRTU55(dMG[U;v MT g7fr)rgrfisdirrFSDP_MODEL_NAME)r folder_namers rr 0Trainer._load_from_checkpoint.. s?#EK77==.Dk!RS/;.#Es AA Az.binzCheckpoint found at z* is only supported when using PyTorch FSDPc3`# UH$n[RRU5v M& g7fr)rgrfr8)rfs rr r s+ Aq!!s,.z!Can't find a valid checkpoint at zLoading model from ryz9You are resuming training from a checkpoint trained with z- of Transformers but your current version is zJ. This is not recommended and could yield to errors or unwanted behaviors.user_content.ptFrftagrload_optimizerrTzOEnabling FP16 and loading from smp < 1.10 checkpoint together is not supported.r map_location weights_only_smp_is_partialrxractive_adapteractive_adapters load_adapterr zFMultiple active adapters detected will only consider the first adapterr)rCjThe intermediate checkpoints of PEFT may not be saved correctly, consider using a custom callback to save i in corresponding saving folders. Check some examples here: https://github.com/huggingface/peft/issues/96GCould not load adapter model, make sure to have `peft>=0.3.0` installed)rx prefer_safe)4rrgrfrrmrlrkrqrprornrrWlistdirr8rrOrrrrfrom_json_filetransformers_versionrrArrtrr@rrrvrloadload_state_dictrrTrur.rsave_safetensors safetensors load_file_issue_warnings_after_loadrexistsrr<rr set_adapterr*)rrr config_fileadapter_weights_fileadapter_safe_weights_file weights_fileweights_index_filesafe_weights_filesafe_weights_index_file is_fsdp_ckptradapter_subdirsrcheckpoint_versionrl load_resultrr subdir_namepeft_ids ` rrTrainer._load_from_checkpoint s =JJEggll#9;G !ww||,BDXY$&GGLL1GIb$c!ww||$:LI WW\\*@BTUGGLL)?ARS"$'',,/EG^"_ww}}%;<  #%::.D#E  ^ ww~~bggll+AoEVVZC[\] *ww}}344$&::.D#E #EK77==.Dk!RSGGNN277<<0F Ui#jkkww~~bggll3I;Xq&rs #E    4 434J3KKuvw w  !%&+(-    @AW@XYZ Z )*@)ACD 77>>+ & &%44[AF!'!>, ' '277>>:K+L+LP\&((77>>"'',,/EGX"YZZ..3uej tyy&11diinn6Lm-.!&Lu[_!`J49J01"'"7"7 4"7"PK"%%$$**66$$*  ,- 99--"''..AR2S2S!,!2!2!>"899u&788*/*?*?/!3"NN+st)8);).)=)=&+:K&(ggll3I;&WG!..w S^bpSp.r,;)).9**+A>`d*eNNDDXCYZbb hi2-6M6O]a]f]f]w]wK+,,// <-A s#A^-B ^=^c[RSURRSURRS35 [ R RURR[5n[ R RURR[5n[ R RURR[5n[ R RURR[5n[5(a URO URnUR(aB[!URURR[#UR5(+S9 gUR$(aT['UR(RR*UR(UURR40[-5D6ng[ R R/U5(dm[ R R/U5(dI[ R R/U5(d%[ R R/U5(Ga=Sn[5(Ga[ R R1[ R RURRS55(a0[2R4"URR[SSS9 gUR6R8(aC[ R R1U5(a[:R<R?US S 9nO [A5 [<RB"US SS 9nSUS 'UREUSS 9ng[#U5(GaF[GUS5(d[GUS5(Ga [GUS5(a[GUS5(a>URHSn [KURH5S:a[RMS5 O URNn [ R R/U5(d$[ R R/U5(a7URQURRU 5 SSK.J/n U "//5nO[RMS[S35 SnO[RMS5 SnOUR6R8(aC[ R R1U5(a[:R<R?US S 9nO [A5 [<RB"US SS 9nUREUS5n[5(dU(aURaW5 ggg[ R R/[ R RURR[b55(dZ[ R R/[ R RURR[d55(aH[gXPRR[5S 9n[5(dURaU5 gg[RMSUS35 g![RaWn URTU RV(a3SURTU RXRZS3n [SU 5U eeSn A ff=f)NzLoading best model from z (score: rqrTrFrrrrrrrrrrr zCDetected multiple active adapters, will only consider the first onez0When using prompt learning PEFT methods such as z, setting load_best_model_at_end=True can lead to errors, it is recommended to set this to False and to load the model manually from the checkpoint directory using PeftModel.from_pretrained(base_model, ) after training has finished.)_IncompatibleKeysrrrz#Could not locate the best model at zi, if you are running a distributed training on multiple nodes, you should activate `--save_on_each_node`.)4rrrurk best_metricrgrfrrqrorlrkrrSrrHr%rrOrrTr.rrr8rtrrrrrrrvrrr@rr<rArrr2 peft_configis_prompt_learning peft_typertorch.nn.modules.modulerrrnrpr*) rbest_model_pathbest_safe_model_pathbest_adapter_model_pathbest_safe_adapter_model_pathrrhas_been_loadedrlrexcmsgrs rrnTrainer._load_best_modelZ s^ .tzz/O/O.PPYZ^ZdZdZpZpYqqstu'',,tzz'G'GV!ww||DJJ,L,LN_`"$'',,tzz/O/OQe"f')ww||DJJ4T4TVo'p$&=&?&?""TZZ  $ $ %"" 00'5djj'A#A   ! !)  &&22   00  () K GGNN? + +ww~~233ww~~566ww~~:;;"O&((77>>"'',,tzz/O/OQb"cdd..!ZZ==( %', yy11bggnnEY6Z6Z%0%6%6%@%@AU^c%@%d 02%*ZZebf%g 49J01"'"7"7 4"7"PK!%(( '788GEK\<]<]cj~dd#5*;<<-2-B-B1-EN"5#8#89A= &/t u-2-A-AN77>>*ABBbggnnUqFrFr* % 2 24::3S3SUc d R*;B*CK"NN!LL`Kabj!j /4O'pq*/yy11bggnnEY6Z6Z%0%6%6%@%@AU^c%@%d 02%*ZZebf%g #("7"7 E"JK.00_33K@6E0 WW^^BGGLL)I)IKbc d dhjhohohvhv GGLL99;M Ni i 2zz77@W@YK+,,// <- NN5o5FGPP g$0 *#(#4#4^#D#W#W)[+0+<+<^+L+V+V+\+\*]^8)8%(+7s*;$D$) *s*&[(( ] 2A]] c[UR5S:waURRbQ[ UR5[ URR5:XaURR 5 O#[ RSURS35 [UR5S:wa$[ RSURS35 gg)Nrz8There were missing keys in the checkpoint model loaded: ryz;There were unexpected keys in the checkpoint model loaded: ) r< missing_keysr_keys_to_ignore_on_saver:rrrAunexpected_keys)rrs rr"Trainer._issue_warnings_after_load s {'' (A -zz11=#kF^F^B_cf 22dC &&(!YZeZrZrYsstuv {** +q 0 NNMkNiNiMjjkl  1rc URUS9nURXRRU5 [ UR [ RR R5(aXU(dQURRnURS5(dSU3nUR RXE5 U$U$![a2n[SUS[UR55SUS35UeSnAff=f)N ignore_keyseval_9The `metric_for_best_model` training argument is set to 'W', which is not found in the evaluation metrics. The available evaluation metrics are: zX. Please ensure that the `compute_metrics` function returns a dictionary that includes 'zM' or consider changing the `metric_for_best_model` via the TrainingArguments.)evaluaterarurgrrZrr>r_rr$ startswithrtr)rr)rr/rrrJmetric_to_checkrs rrATrainer._evaluate s--,@-A !!%)?)?I d'')A)A)S)S T T]k"ii==O"--g66$)/):"; !!&&w'?@w OP_O`a==A',,.=Qrrr_nested_gathermeanrrrrrrr=rorpshould_evaluaterA_determine_best_metricrr!rXr"rf_save_checkpointr^on_save) rrrrr/r7rrrlogstr_loss_scalarrJis_new_best_metrics rra Trainer._maybe_log_save_evaluate s << " " "tzz'='=@\@\'\%'' %'D"009>>@EEGN  G ::3I3IDLhLh3h!iklmDL$8B9ell8[8[INN$4aj[!((5_%(,(?(?(A_%  # #~ 5 #+/::+A+AD ( OO  HHT & << ' 'nnUAG!%!>(++ :=/Jcc ,ww||J@H77>>(++ B ^#(::h#7 ,X67 09: ""#7#>? ! # #   1%8 900L4L4LL :: " " $ $ $VUZZ9M ^ ! # # $UEII7K \ ! # # $UEII7K \ ! # # $UEII7K \ " $ $ $VUZZ9M ^ %#^s <J// J>c bSnURRGbXURRnURS5(dSU3nXnURR (a[RO[RnURRc@URR (a [S5O [S5URl U"XPRR5(aoXPRl URR[R[R 4;a%URR"URlS nU$![a/n[SUS[ UR 55S35UeSnAff=f) z Determine if the model should be saved based on the evaluation metrics. Returns: bool: True if a new best metric was found, else False FNrrrzJ. Consider changing the `metric_for_best_model` via the TrainingArguments.z-infinfT)rr$rr)rrgreater_is_betterrgreaterlessrurrr!rXSTEPSEPOCHrgbest_global_step)rrJr/rr metric_valueroperators rrTrainer._determine_best_metricL s[# 99 * * 6"ii==O"--g66$)/):"; &7 &*YY%@%@rzzbggHzz%%-:>)):U:Uv[`af[g & jj&<&<==)5 &99**|/A/A5(a*URR<U RAU 5 MXRR<U 'M URRC[ RRU[D55 URRF(aURIU5 URR,(aURKSUS9 ggs snf)Nro)r/Trdr$)&rPrurgr~rorqrgrfrrhrr!rXr r!r"rrksave_only_model_save_optimizer_and_scheduler _save_scaler_save_rng_staterfr^rr|rr6r6r7rrrrirjrd_push_from_checkpoint_rotate_checkpoints) rrr/checkpoint_folderrrbest_checkpoint_folderbest_checkpoint_dirrcb_namecb_states rrTrainer._save_checkpointr sO 55Qtzz7M7M6NO  ! ! )em OO &&U&3WW\\'=   48 99 " "|'9'9<;M;M&N NSWS]S]SnSn(='>a @[@[?\%] ""$'',,w"O ww~~1223F 0yy((  . .z :   j )   , 99 "22<< ~MMrQ[\^`oQpM,,//88:djj;;GDdKKJJ11':AA(K=EJJ11': JJ # #BGGLL=O$P Q 99  & &z 2 99  $ $t $ H !s L*0L*c [R"5[RR5[RR 5S.n[R R5(aURR[R:Xa,[R RR5US'O+[R RR 5US'[5(a[R "5US'[5(aURR[R:Xa,[R RR5US'O+[R RR 5US'[#5(aURR[R:Xa,[R$RR5US'O+[R$RR 5US'['5(aURR[R:Xa,[R(RR5US'O+[R(RR 5US'[+5(akURR[R:Xa"[R,R5US'O![R,R 5US'[.R0"USS 9 URR2S ::a6[R4"U[.R6R9US 55 g[R4"U[.R6R9US URR:S 355 g)N)rrrrrrrrrTr r rrr)rgetstater get_stater get_rng_staterrrrIrhrJget_rng_state_allrrrrrrrrrrrgrhrrkrfrr)rr rng_statess rr*Trainer._save_rng_state soo'YY((*<<--/ :: " " $ $yy&&,*B*BB%*ZZ%6%6%H%H%J 6"%*ZZ%6%6%D%D%F 6" ! # # " 0 0 2Ju  ! # #yy&&,*B*BB$)II$4$4$F$F$H 5!$)II$4$4$B$B$D 5! ! # #yy&&,*B*BB$)II$4$4$F$F$H 5!$)II$4$4$B$B$D 5! ! # #yy&&,*B*BB$)II$4$4$F$F$H 5!$)II$4$4$B$B$D 5! " $ $yy&&,*B*BB%*ZZ%A%A%C 6"%*ZZ%=%=%? 6" J. 99  1 $ JJz277<< O#L M JJz277<< jI`I`HaaeR@"URBRD5RFRI55;nU(a6[KUR 5(aURBREUSS9 GO/URBREU5 GOURL(a[OURPR4RRURPUR U40[U5D6 [WURPR4RRURPURUR U5 OlURRX(aQ[ZR"URR 5[RRU[55 UR:=(a []UR$[^5(+nURRX(aUR:(aU(a[5(dz[ R""SS 9n[ZR"UR$R 5[RRU[&55 SSS5 [)W5 gggg!,(df  GN=f!,(df  N/=f)Nsaving_optimizer_states)rYshard_metadatar-of-roF master_onlyTrecord)gather_if_shardr)rv3exclude_frozen_parameters)rD)0rrrlrrYrlrget_shard_metadatarkrgrfrrrrrmr3catch_warningsrZrnrMrlocal_state_dictrtrrdp_rankrurvshard_optimizer_staterHr:rrrSsave_checkpointrrrrOrrTr.rrrfrrr)rroptmcaught_warningsopt_state_dict accept_exclude_frozen_parametersis_deepspeed_custom_schedulers rr(%Trainer._save_optimizer_and_scheduler s ! # # MM3 4**!%!:!:!<&*jj&C&C&EGGLL"d499+B+B*C4 H\H\G]]^_m^n$o!& 113RWW\\*n5]^((5))446 ZQ_8`a#O465% & &!^^<  ! !   &&22D4D4DdjjR\ `u`w    &&22D4D4DdnnVZV`V`bl YY " " JJt~~002BGGLL^4\ ])-(A(A) *   8K G % II ! !..2O*,,((5 4,,779277<< Tb;cd6  0 -3P "Q65X65sRU5 URRU5 g[5(ak[RR%[RRTS55(aU4SjnOU4SjnUR@RCU5 OUR0R2S:aUR0R<OS nURD(aU[GURHRJRLURHUR>URNT40[Q5D6 O[[ 5 UR>R[R"[RRT["5USS 95 [R "SS9n[ 5 URR[R"[RRT[5SS95 SSS5 [W5 ggg!,(df  GN=f!,(df  GN=f!,(df  N@=f)z3If optimizer and scheduler states exist, load them.NTr@r_*c3># UHdn[RR[RRTU55(dMG[R S5SU;v Mf g7f)ryrN)rgrfrrOPTIMIZER_NAME_BINr)rrrOs rr 8Trainer._load_optimizer_and_scheduler..0 sP+AK77==j+)NOH*005a8KG+As AA/ "A/z rank*-of-rorr=rrrYrc>UR[R"[RR T[ 5SS95 g)NTr)rrtrrgrfrrmmodoptrOs r opt_load_hook.opt_load_hooka s-//jR`9ako0pqrc8>[(aHUR[R"[R R T[5SSS95 gUR[R"[R R T[5SS95 g)NT)r back_compatr)rsrrtrrgrfrrmrXs rr[r\f sd88 # 3 3$'HHRWW\\*n-U_cqu$v!"!$ 3 3CHHRWW\\*Vd=eos4t urr ))rHrrZrr3rFrvrrrrgrfrrnrMrglobrmr8rUrrWrrrrrrrsend_cpu_data_to_devicer=rYrSregister_post_step_hookrOrrTrur.rr)rrOrLcheckpoint_file_existsoptimizer_statelr_scheduler_stater[rs ` rr5%Trainer._load_optimizer_and_scheduler sq     $ $d//1JKK,,D9_,.%%55 277<< N#KZ^_: $O4 '(( IIbggll:~>E Frww||JGH 77>>"'',,z;M"NO GGMM*-+-::j+A $** IIbggll:499;O;O:PPQR`Qa/bc d'  "bggnnRWW\\*n5]&^&^%''..,.&+jj &$tyy/F/F.GtDIIL`L`Kaabcqbr(s&+%) 'O'6k&BO,.&+jj Z@ucg'O,,D9_,.). Z@ucg*&: $O4**?II>}M 8> ,, NN JJ&  45 1266!JJ " Z HWcrv ,,D9_,.%%55 277<< N#KZ^_: $O4Y'_ !;:9b:9f:9s' AYAY,)AY> Y), Y;> Z cLURRnUcg[5(a[R "S5 [ R"SS9n[R"URRR5[RRU[55 [U5 SSS5 URR (a[5(d[ R"SS9n["R"URRR5[RRU[55 SSS5 [W5 ggg![a gf=f!,(df  N=f!,(df  N==f)Nsaving_scaler_stateTr@)rTscalerr!rrrlr3rFrkrlrgrfr SCALER_NAMErMrrfr)rrrhrLs rr)Trainer._save_scaler s %%,,F >  ! # # MM/ 0((5((//::cd#O46 99 )?)A)A((5 4++22==?jZeAfg6  0*B     65 65s+E4A'FAF4 FF F F#c Ucg[RR[RRU[55nU(GaW[ 5(a[ R"SS9n[5 [R"[RRU[5SSS9nSSS5 [W5 [R"WURR5 UR R"R%U5 g[ R"SS9n[5 UR R"R%[R"[RRU[5SS95 SSS5 [W5 gg!,(df  N=f!,(df  N,=f)z If scaler state exists, load it.NTr@rrrR)rgrfr8rrirr3rFrvrrrMrr`rr=rTrhr)rrOrbrL scaler_states rr6Trainer._load_scaler s-   !# Z0U!V !&'',,D9_,.#(:: Z=E`d$L: $O4**<9I9IJ  ''77 E,,D9_,.$$++;; 277<< K#HW[\: $O4% ":9:9s1AF&,A%F7& F47 Gc>^ URR(dg/n/nURRUR/-nUR R R5GHOum n[U[5(dU/n[U 4SjU55(GaUVs/sH oURRT :XdMUPM" nn[Xd5HupWURS05nURS05n [U5"S 0UD6n U R5Hup[!XU 5 M [U["5(aXlOUR%U 5 URR'[U 55 M [(R+S5 GM>UR%T 5 GMR [-U5S:a([(R/SSR1U5S 35 UHnURR3U5 M gs snf) zLIf callback states exist and were passed in, restore their states if enabledNc3T># UHoRRT:Hv M g7fr)r6r7)rrstored_callbacks rr /Trainer._load_callback_state.. s"eRdh%%../ARds%(r attributeszPContinuing training from checkpoint, restoring any callbacks that were passed inrzPCheckpoint included callbacks not included in current configuration. Ignoring. (rrr)r'restore_callback_states_from_checkpointr^rr|rurr<rrrWr6r7ziprrr>r:rrrrr<rArr_)r not_found new_callbacksoriginal_callbacksrmr duplicates callback_datarrr new_callback attributerrps @rr9Trainer._load_callback_state syy@@   !22<< ~M%)ZZ%B%B%H%H%J !OTdD))veRdeee.@-?CUCUC^C^bqCqH-?03:/D+H(,,VR8D!.!2!2<!DJ#'>#9D#9L,6,<,<,>(  ?-?!(N;;'3 %,,\:))99$|:LM0E no  1/&K0 y>A  NNbcgclclmvcwbxxyz &H  ! ! . .x 8&/s (H Hr8rQn_trials directionbackendrr'c <Uc [5n[U5n[U"5nUR5 XPlUR c [ S5eUc UROUUlX`l Uc[OUUl UR"XU40UD6n SUlU $)a Launch an hyperparameter search using `optuna` or `Ray Tune` or `SigOpt`. The optimized quantity is determined by `compute_objective`, which defaults to a function returning the evaluation loss when no metric is provided, the sum of all metrics otherwise. To use this method, you need to have provided a `model_init` when initializing your [`Trainer`]: we need to reinitialize the model at each new run. This is incompatible with the `optimizers` argument, so you need to subclass [`Trainer`] and override the method [`~Trainer.create_optimizer_and_scheduler`] for custom optimizer/scheduler. Args: hp_space (`Callable[["optuna.Trial"], Dict[str, float]]`, *optional*): A function that defines the hyperparameter search space. Will default to [`~trainer_utils.default_hp_space_optuna`] or [`~trainer_utils.default_hp_space_ray`] or [`~trainer_utils.default_hp_space_sigopt`] depending on your backend. compute_objective (`Callable[[Dict[str, float]], float]`, *optional*): A function computing the objective to minimize or maximize from the metrics returned by the `evaluate` method. Will default to [`~trainer_utils.default_compute_objective`]. n_trials (`int`, *optional*, defaults to 100): The number of trial runs to test. direction (`str` or `List[str]`, *optional*, defaults to `"minimize"`): If it's single objective optimization, direction is `str`, can be `"minimize"` or `"maximize"`, you should pick `"minimize"` when optimizing the validation loss, `"maximize"` when optimizing one or several metrics. If it's multi objectives optimization, direction is `List[str]`, can be List of `"minimize"` and `"maximize"`, you should pick `"minimize"` when optimizing the validation loss, `"maximize"` when optimizing one or several metrics. backend (`str` or [`~training_utils.HPSearchBackend`], *optional*): The backend to use for hyperparameter search. Will default to optuna or Ray Tune or SigOpt, depending on which one is installed. If all are installed, will default to optuna. hp_name (`Callable[["optuna.Trial"], str]]`, *optional*): A function that defines the trial/run name. Will default to None. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments for each backend: - `optuna`: parameters from [optuna.study.create_study](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.create_study.html) and also the parameters `timeout`, `n_jobs` and `gc_after_trial` from [optuna.study.Study.optimize](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.Study.html#optuna.study.Study.optimize) - `ray`: parameters from [tune.run](https://docs.ray.io/en/latest/tune/api_docs/execution.html#tune-run). If `resources_per_trial` is not set in the `kwargs`, it defaults to 1 CPU core and 1 GPU (if available). If `progress_reporter` is not set in the `kwargs`, [ray.tune.CLIReporter](https://docs.ray.io/en/latest/tune/api/doc/ray.tune.CLIReporter.html) is used. - `sigopt`: the parameter `proxies` from [sigopt.Connection.set_proxies](https://docs.sigopt.com/support/faq#how-do-i-use-sigopt-with-a-proxy). Returns: [`trainer_utils.BestRun` or `List[trainer_utils.BestRun]`]: All the information about the best run or best runs for multi-objective optimization. Experiment summary can be found in `run_summary` attribute for Ray backend. NzXTo use hyperparameter search, you need to pass your model through a model_init function.) r"rTr!ensure_availabler~rr2default_hp_spacer8r'r\rQr) rr8rQr}r~rr'r backend_objbest_runs rhyperparameter_searchTrainer.hyperparameter_search s@ ?/1G!'*8AC $$&!( ?? "j 9A8H 44h  >O>W!:]n??49GG!%rrrc0URRbURRUS'URR(aIURRUS'Ub-UR [ SX RRS95 0UESURR0EnURRRU5 URRURURURU5Ul g)z Log `logs` on the various objects watching training. Subclass and override this method to inject custom behavior. Args: logs (`Dict[str, float]`): The values to log. start_time (`Optional[float]`): The start of training. Nr7rKr\)r-rt) rur7rrIrKrrfrg log_historyrr^on_logr|)rrrrs rrp Trainer.log5s ::   ' JJ,,DM 99 2 2,0JJ,L,LD( )% M':**JjJjkl=D=VTZZ%;%;<= %%f-,,33DIItzz4<# UHnTRU5v M g7fr)_prepare_input)rrFrs rr )Trainer._prepare_input..SsCdd11!44d!r=rr)rrrr<rrrrrrr=rHis_floating_point is_complexrrTrurDr5rr)rrmrrFrs` rrTrainer._prepare_inputLs dG $ $:TZZ\R\TQq$"5"5a"88\RS S udm , ,:CdCC C ell + + 0 01F((e.E.Ed.K.KuO_O_`dOeOe w(8(8(>(>(O(O(\(\(b(b(def77$V$ $ SsE, rcURU5n[U5S:Xa([SSRUR5S35eUR R S:aURbURUS'U$)z Prepare `inputs` before feeding them to the model, converting them to tensors if they are not already and handling potential state. rzThe batch received was empty, your model won't be able to train on it. Double-check that your training dataset contains keys expected by the model: rdrymems)rr<rrrorrBrrrs rr|Trainer._prepare_inputs^s $$V, v;! IILRVRiRiIjHkkln  99  1 $)?!ZZF6N rc"UR5$)z6 A helper wrapper to group together context managers. )autocast_smart_context_managerrs rcompute_loss_context_manager$Trainer.compute_loss_context_managerns2244rrucUR(a3[RRR XR S9nU$[ R"5nU$)z A helper wrapper that creates an appropriate context manager for `autocast` while feeding it the desired arguments, depending on the situation. )rur)rqrrrr~ryrr)rru ctx_managers rr&Trainer.autocast_smart_context_managertsN    ))--00}TbTb0cK%002KrcUR5 [URS5(a>[URR5(aURR5 UR U5n[ 5(aa[ XURR5nUR5R5RURR5$UR5 URXUS9nSSS5 AURRGb[UR R"URR-S:XGa)[%5(a[&R(R+5 O[-5(a[&R.R+5 O[15(a[&R2R+5 O[55(a[&R6R+5 Oq[95(a[&R:R+5 OC[=5(a[>RAS5 O[&RBR+5 0nURRD[FRH[FRJ4;aURM5US'URRNS:aWRQ5nURR(a;[TRV"WUR5nURY5 SSS5 gURZ(d&UR\cWURR- nUR^R`[bRd:XaSUS 'UR^RX"W40UD6 UR5$!,(df  GN=f!,(df  g=f) a Perform a training step on a batch of inputs. Subclass and override to inject custom behavior. Args: model (`nn.Module`): The model to train. inputs (`Dict[str, Union[torch.Tensor, Any]]`): The inputs and targets of the model. The dictionary will be unpacked before being fed to the model. Most models expect the targets under the argument `labels`. Check your model's documentation for all accepted arguments. Return: `torch.Tensor`: The tensor with training loss on this batch. r\rNrzW`torch_empty_cache_steps` is set but HPU device/backend does not support empty_cache().rr F scale_wrt_gas)3r\r@rYrir|rrrr reduce_meandetachrr=r compute_losstorch_empty_cache_stepsrurgrrxpu empty_cacherrrrrrrmpsrrrArr>rgrrrrrrpr scale_lossbackwardrVrrTrOrrP)rrrrloss_mbrr scaled_losss rrRTrainer.training_steps(  4>>7 + +9M9M0N0N NN "%%f- " $ $*5$)):_:_`G&&(//144TYY5E5EF F  . . 0$$UGY$ZD1  II - - 9 &&)J)JJaO%'' %%'')) %%'(** &&(')) %%'')) %%''))m &&( 99??~22N4J4JK K&*&=&=&?F? # 99??Q 99;D ==dnn5$$&6511d6L6L6TdiiCCC00O4M4MM*/'    % %d 5f 5;;= c1 0F65s;O1O. O+. O<c vURc URbSU;aURS5nOSnUR(a0nUbXFS'0UEUEnU"S 0UD6nURR S:aXpRR UlUbURRU5n[U5(a%URRR5n OUR5n URbURXuUS9n OU [R"5;aURXuSS9n OURXu5n O[U[ 5(aRSU;aL[#S S R%UR'55S S R%UR'55S 35e[U[ 5(aUSOUSn URR((a=UR(dUR(aUbXRR*-n U(aX4$U $)z How the loss is computed by Trainer. By default, all models return the loss in the first element. Subclass and override for custom behavior. NlabelsrrrT) shift_labelsrzJThe model did not return a loss from the inputs, only the following keys: rdz,. For reference, the inputs it received are ryr)r{rr=rVrrBrrTr+rrr _get_namer,r;rrrrraverage_tokens_across_devices num_processes) rrrreturn_outputsrr loss_kwargsoutputsrrrs rrTrainer.compute_losss    +t/E/E/QW_ciWiZZ)FF  ) )K!-4F01..+.F/&/ 99  1 $ !5!56DJ  "..;;EBOo..,77==GGI ,668 %%1--gRd-e@GGII**7*N**7;'4((V7-B `xx /00\]`]e]eflfqfqfs]t\uuvx '1$&?&?76?WQZD II 3 3//43I3I". $$22 2D"0:d:rc4URRS:H$)z Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on several machines) main process. r)rlocal_process_indexrs rrTrainer.is_local_process_zeros yy,,11rc[5(a[R"5S:H$URRS:H$)z Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only going to be `True` for one process). r)rrtrrrrs rrTrainer.is_world_process_zeros2 # $ $88:? "99**a/ /rrrecUcURRn[5(aURU5 GO[ 5(a[ R "USS9 URR5nURR(aURXS9 [(a7[[ RRUS55R5 GOXUR (aS[#UR$R&R(R*5;a[,R."[05[,R."S5:aOUR$R3UR45nURR(aURXS9 OUR6(aQUR$R3UR85nURR(aURXS9 O,URR(aURU5 URRH(aU(dURIS S 9 ggg![:a [<R?S5 URR(aURU0S9 [AURRU[B[D/5 URRGU5 Nf=f) zm Will save the model, so you can reload it using `from_pretrained()`. Will only save from the main process. NTr )rlrFULL_STATE_DICTz0.24.1z| stage3_gather_16bit_weights_on_model_save=false. Saving the full checkpoint instead, use zero_to_fp32.py to recover weightsz Model save)commit_message)%rrr _save_tpurrgrhrSrlrf_saversrrfrtouchrOrrTrur.state_dict_typerraccelerate_versionget_state_dictrrHr(rrrArNrqrorJrd)rrrerls rrhTrainer.save_models  --J ! # # NN: & $ & & KK T 2++668Jyy$$ : =((RWW\\*.?@AGGI  ! !!S)9)9)?)?)K)K)[)[%\\ 01GMM(4KK!--< ?sAI44BLLc " UbUOURRn[RSU35 URn[ R "5 [ R"SS9(aX[R"USS9 [R"UR[RRU[55 [4n[ R "S5 UR"(GaUR%5UR'5S.n[RRUSURR(S URR*S [,35n[ R"XESS 9 [ R "S 5 URR.(aS SKJn U"[RRUS5S[,3SS9upxUR4R4nUR6R9U5n [;X5(a6U R=UU[ RURR>S9 GO[RS5 [ R"U[RRU[,55 GO[;X#5(Gd"[;UR6R9U5U5(aUR6R9U5R=UURR.[ R@"UR%55[ RURR>S9 O[RS5 [ R@"UR%55n [ R"U [RRU[,55 OkUR=UURR.[ RURR>[ R@"UR%55S9 URBb8URR.(aURBR=U5 ggg)NSaving model checkpoint to F)localTr saving_checkpoint)rr<rr=ror>save_full_checkpointsr)%consolidate_sharded_model_checkpointsrz rank*-of-*-) ckpt_prefix ckpt_suffixrh)rl save_functionsafe_serializationETrainer.model is not a `PreTrainedModel`, only saving its state dict.)is_main_processrlrr)rrrrl)"rrrrrrris_master_ordinalrgrhrrkrfrTRAINING_ARGS_NAMErtrlrrlrErrrqrfrrrQrTr+rsave_pretrainedr_maybe_convert_to_cpur) rrrsupported_classesckpt ckpt_pathrfull_state_dictrnrrls rrTrainer._save_tpuMs=#-#9Ztyy?S?S  1*>?    e , KK T 2 JJtyy"'',,z;M"N O,- )*  & & &))+"'":":"<D d499#:#:";4 @T@T?UUVWcVdeI GGD 7 MM1 2yy$$\%J " Z <"-l^ <$&"  ++"&"2"2"?"?"FoAA#33"#2&(gg+/99+E+E 4KK ghGGORWW\\*l-STE55$**77>@QRR  --e4DD$(II$9$9!778H8H8JK"$'''+yy'A'A E cd55e6F6F6HI  BGGLL\$JK  ! ! $ 5 5 gg#'99#=#=33E4D4D4FG "   ,1F1F  ! ! 1 1* =2G ,rcUbUOURRn[R"USS9 [R SU35 [ 5(d[4O [[4n[URU5(GdEUcURR5n[URRURSS9U5(aGURRURSS9RXURRS9 O[R S5 URR(aD[ R"R%U[R&R)U[*5SS 0S 9 Oh["R,"U[R&R)U[.55 O.URRXURRS9 UR0bUR0RU5 OyUR2bl[5UR2S 5(aQUR2R6b:[R S 5 UR2R6RU5 ["R,"UR[R&R)U[855 g) NTr rFr)rlrrrpt)rrzWSaving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None`)rrrgrhrrrr)rrrrlrTr+rrrr save_filerfrrorkrqrrr@rr)rrrlrs rr Trainer._saves#-#9Ztyy?S?S  J. 1*>?6G6I6I_.P_ajOk$**&788!!ZZ224 $**77 W\7]_pqq  --djjU-Scc$))JdJdd cd99--%%//"BGGLL=N$O[ceiZj0JJz277<< L+QR JJ & &diiF`F` '   ,  ! ! 1 1* =    ***K88"",,8 KKq r    ( ( 8 8 D  499bggll:7IJKrcURR[R:XajUR=R [ UR/URRS9R5R5- slSUlgUR=R UR- slSUlg)Nrr) rrIrhrJrur"rCr}r=rCrrs rroTrainer.store_floss 99 " "l&>&> > JJ ! !-t/@/@.A$))JZJZ[__affh  !!"D  JJ ! !T%6%6 6 ! !D rcV/n[U5RUS35Vs/sH4n[RR U5(dM)[ U5PM6 nnUHnU(a2UR [RRU5U45 M<[R"SUS3U5nUcM\UR5cMoUR [UR5S5U45 M [U5n U(aL[U5S:a=U SSU SS- n U S:a&[R"S5 UR!S XS 9$U V s/sHoSPM n n UR"R$b[ [UR"R$55U ;aiU R'[ [UR"R$555n [)U [U 5S - 5Hn XS-XsX'XS-'M U $s snfs sn f) N-*z.*z -([0-9]+)rr rrzPmtime may not be reliable on this filesystem, falling back to numerical orderingF)r%rcheckpoint_prefixr)rr_rgrfrrrgetmtimerematchgroupsrsortedr<r3r4rrrurkindexr)rrrr%ordering_and_checkpoint_pathxglob_checkpointsrf regex_matchr mtime_diffrObest_model_indexrs rrrTrainer._sorted_checkpointss(*$,0,<,A,AEVDWWYBZ,[p,[q_a_f_f_l_lmn_oFCF,[p$D,33RWW5E5Ed5KT4RS hh,=+>i'H$O *{/A/A/C/O077[=O=O=QRS=T9UW[8\] %$$@A 9:Q>+B/25G5J15MMJC pq//# 0?QQ>P m>PQ JJ , , 8D99:;?QQ177DAaAa"55 UR@RCURURDURFU R&5Ul#UR RIU R&5 U R&$) a Run evaluation and returns metrics. The calling script will be responsible for providing a method to compute metrics, as they are task-dependent (pass it to the init `compute_metrics` argument). You can also subclass and override this method to inject custom behavior. Args: eval_dataset (Union[`Dataset`, Dict[str, `Dataset`]), *optional*): Pass a dataset if you wish to override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. If it is a dictionary, it will evaluate on each dataset, prepending the dictionary key to the metric name. Datasets must implement the `__len__` method. If you pass a dictionary with names of datasets as keys and datasets as values, evaluate will run separate evaluations on each dataset. This can be useful to monitor how training affects other datasets or simply to get a more fine-grained evaluation. When used with `load_best_model_at_end`, make sure `metric_for_best_model` references exactly one of the datasets. If you, for example, pass in `{"data1": data1, "data2": data2}` for two datasets `data1` and `data2`, you could specify `metric_for_best_model="eval_data1_loss"` for using the loss on `data1` and `metric_for_best_model="eval_data2_loss"` for the loss on `data2`. ignore_keys (`List[str]`, *optional*): A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions. metric_key_prefix (`str`, *optional*, defaults to `"eval"`): An optional prefix to be used as the metrics key prefix. For example the metrics "bleu" will be named "eval_bleu" if the prefix is "eval" (default) Returns: A dictionary containing the evaluation loss and the potential metrics computed from the predictions. The dictionary also contains the epoch number which comes from the training state. Nrn)rrrrT)rprediction_loss_onlyrr_jit_compilation_time_model_preparation_timer r!)%rrrr<rrr,r-rrr'rrrprediction_loopevaluation_looprrrrJrfr mathceilrprrfrGrrgrhrir^ on_evaluaterur|r)rrrroverriderJeval_dataset_name _eval_datasetdataset_metricseval_dataloaderr eval_looprtotal_batch_sizes rrTrainer.evaluate skZ t+'/|T5F5F lD ) )G4@4F4F4H0!"&--2:@Q +):(;1=N>S)TU UJ 7 8FNN J ..,=+>>U)VW WJ !"..))F$6$69I$IJ      ( (DIIOO ; OOC..0 1,,88DJJPTP\P\^d^l^lm  44V^^D~~rc URR5 URU5n[R"5nURR (a UR O URnU"USX#S9nURRURR-nUS3UR;aXWRUS3- nUS3UR;aXWRUS3- nURR[UUUR[R"URU- 5S95 UR R#URUR$UR&UR5UlURR)UR5 [+UR,UR.URS9$)a Run prediction and returns predictions and potential metrics. Depending on the dataset and your use case, your test dataset may contain labels. In that case, this method will also return metrics, like in `evaluate()`. Args: test_dataset (`Dataset`): Dataset to run the predictions on. If it is an `datasets.Dataset`, columns not accepted by the `model.forward()` method are automatically removed. Has to implement the method `__len__` ignore_keys (`List[str]`, *optional*): A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions. metric_key_prefix (`str`, *optional*, defaults to `"test"`): An optional prefix to be used as the metrics key prefix. For example the metrics "bleu" will be named "test_bleu" if the prefix is "test" (default) If your predictions or labels have different sequence length (for instance because you're doing dynamic padding in a token classification task) the predictions will be padded (on the right) to allow for concatenation into one array. The padding index is -100. Returns: *NamedTuple* A namedtuple with the following keys: - predictions (`np.ndarray`): The predictions on `test_dataset`. - label_ids (`np.ndarray`, *optional*): The labels (if the dataset contained some). - metrics (`Dict[str, float]`, *optional*): The potential dictionary of metrics (if the dataset contained labels). Prediction)rrrrrr) predictionsrrJ)r,r-r rrrrrrrrJrrfr rrr^ on_predictrur|rrVr r) rrrrtest_dataloaderrrrrs rpredictTrainer.predictqsH ""$22<@YY[ ,0II,P,PD((VZVjVj  ;  9944tyy7K7KK 5 6&.. H ..,=+>>S)TU UJ 7 8FNN J ..,=+>>U)VW WJ !"..))F$6$69I$IJ   ,,77 4::t||]c]k]kl  44V^^DF,>,>&JZJZdjdrdrssrrc URnUbUO URnUR(aURc[ USSS9u nUR UR SUS9n[URR5S:XGa$XR LGa[R"5n UR(dFUR(aPURRS:wa6URR(dURRU5OURRUSS9n[![R"5U - S 5UlUR(aXlXR LaXlUR(aUR$UlUR&(dsUR((a)UR+[,R.UR0S 9nO9UR2(a(UR+[,R4UR0S 9nURR6n [8R;S US 35 [=U5(a([8R;S UR?U535 O[8R;S5 [8R;SU 35 URA5 [CURDS5(a>[GURDR@5(aURDRA5 XRHl%[MUSS5n URNS:aSUl([SURRTSS9n [SURRTSS9n [SURRTSS9n[SURRTSS9nSn0nSn[WU5GHunn[YU5nUb UU- nU cUn UR[UUX4S9unnn[MUR SS5nSUR\;aUR_UU5OSn[a5(a[bRd"5 Ub1URgURiU 55nU RkU5 Ub^URRmUSSS9nURgU5nURRn(aUS:XaURkU5 UbURRmUSSS9nUb}URRmUSSS9nURpbURqUU5nURgU5nURRn(aUS:XaU RkU5 UbCURgU5nURRn(aUS:XaURkU5 URHRsX`RtURv5Ul;URRn(aURxbrUboUblURRzR|n0nSUR\;aUOSUS'SUR\;aUOSUS'URy[S(UUS.UD6US9nAAAA[,RR5 GMURcGMUS-UR-S:XdGM+U R5 U R5 UR5 UR5 AAAA[,RR5 GM URRUl3URN(a[CUS5(a [US5 U R5n U R5n UR5nUR5n[=U 5(a [U 5nOW[U [5(a[MU S S5S:a U R>nO$[=U5(aUR?U5nOUnUS:XaUS:aUnURxblU biUbfURRn(dKSUR\;aU OSUS'SUR\;aUOSUS'URy[S(XS.UD65nOUc0n[U5n[U [5(a@U (a9[R"U 5R5R5UUS!3'OC[U [R5(a$U R5R5UUS!3'[CUS"5(aURUUS#3'[CUS$5(aUR"UUS%3'[UR55H7nURUS&35(aMURU5UUS&U3'M9 [XUUS'9$))y Prediction/evaluation loop, shared by `Trainer.evaluate()` and `Trainer.predict()`. Works both with or without labels. NrTr" inferenceFrrrevaluation_moderrr= ***** Running  *****rz Num examples: Unknown Batch size = rr padding_indexrrr(rr )dim pad_indexr rlossesr rrrr$_lossrrmodel_preparation_timerrnr rrJr r)UrrrHr(r$rrr<rT_modelsrrOr4 torch_compiler prepare_modelrr#rSr)rLrrfloat16r=rMrxrrrrar$rr@rYrir^rr8rBrr=eval_do_concat_batchesrFrEprediction_stepinclude_for_metricsrrrrgather_functionrepeatrpad_across_processesrron_prediction_steprur|rrGend_of_dataloaderrSrreval_accumulation_stepsto_cpu_and_numpygather_for_metricsrj get_arraysrr>r]rr concatenaterrrrrrr=rR)rrrrrrrrnrrrr all_losses all_preds all_labels all_inputsrJeval_set_kwargsobserved_num_examplesrtrobserved_batch_sizerrrr inputs_decode is_last_step batch_kwargsr rs rrTrainer.evaluation_loopsyy7K7W3]a]v]v  $ $)?!$1MDAq  e  S t'' (A -%::2EJ,,((T-=-=-M-MQV-V_c_h_h_v_v  ((/%%33E43P  +0 j0H!*LD '##" JJ&%*"((!%!3!3""u}}T[[I$$u~~dkkJYY..  &{m6:; j ! ! KK+D,=,=j,I+JK L KK1 2 oj\23 4>>6 * *x8K8K/L/L NN   !0:-z9d; ??a DJ'tyy'G'GW[\ %dii&F&FVZ[ &tyy'G'GW[\ &tyy'G'GW[\ !"&j1LD&"1&"9 ".%)<<%%!4J&*%9%9%I]%9%w "FFF%djj2C[QO@HDLdLd@d##F?$;>v1X\>]!))>>v1X\>]55A!??OF--f5yy33{l7RMM&)!--f5yy33{l7RNN6*00CCD**VZVbVbcDLyy++''38JvOa#'#3#3#B#B#T#TL#%L7=AYAY7YV_cL*7?4C[C[7[VaeL*"22&\6V\|\'33G FFF &&(--9tax4KgKg>gkl>l++-**,++-++-FFF &&(G2L $//BB ??wtW55 D' " **, ((* **, **,  l # #l+K &: ; ; Vdfg@hkl@l&33K*%%"// ; 3 !  5 9/K  ,%&II006<@X@X6X ^bOH %6>$BZBZ6Z `dOH %**^9^o^G_G(0 j$ ' 'J35>>*3M3R3R3T3Y3Y3[G()/ 0  BJJ / /3=??3D3I3I3KG()/ 0 4/ 0 0CGC\C\G())>? @ 41 2 2EIE`E`G())@A B 'C>>%6$7q"9::8? C8H,-Qse45()SZhsttrcvUcg[5(aUcSn[X5nU$[5(a [U5nU$URR b$URR R S:wd1URR c%URRS:wa [U5nU$) Gather value of `tensors` (tensor or list/tuple of nested tensors) and convert them to numpy before concatenating them to `gathered` N nested_gatherNOr) rrLrrrdistributed_staterO local_rankrDrtensorsnames rrTrainer._nested_gathers ?  ! # #|&,W;G % & & )G  ii))5$)):U:U:f:fjn:n II ' ' /DII4H4HB4N(1Grc^^[UR5S:XaSO[U4SjUR55nTRSS5nUc URn[UR5S:Xa U(aSOSnUR T5mTc@[ URS5(a#[URRSS /5mO/mU(dU(a=[[U4S jUR555n[U5S :XaUSnOSn[R"5 [5(a[UT5n U(dU(a[!U ["5(a)U S n [U4S jU R%555n O U Sn U S Sn U R'5R)5R+5n [-U 5n GOySn [!U ["5(a$[U4SjU R%555n OU n [-U 5n GO/U(dU(aUR/5 UR1UTSS9upSSS5 W R)5R35n [!W["5(a$[U4SjUR%555n OUS Sn OSn UR/5 U"S0TD6nSSS5 [!W["5(a$[U4SjUR%555n OUn UR4R6S:a XR4R6S - UlSSS5 U(aW SS4$[W 5n [U 5S :XaU Sn W X4$!,(df  GN;=f!,(df  N=f!,(df  N`=f)av Perform an evaluation step on `model` using `inputs`. Subclass and override to inject custom behavior. Args: model (`nn.Module`): The model to evaluate. inputs (`Dict[str, Union[torch.Tensor, Any]]`): The inputs and targets of the model. The dictionary will be unpacked before being fed to the model. Most models expect the targets under the argument `labels`. Check your model's documentation for all accepted arguments. prediction_loss_only (`bool`): Whether or not to return the loss only. ignore_keys (`List[str]`, *optional*): A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions. Return: Tuple[Optional[torch.Tensor], Optional[torch.Tensor], Optional[torch.Tensor]]: A tuple with the loss, logits and labels (each being optional). rFc3J># UHnTRU5SLv M g7frr)rrrs rr *Trainer.prediction_step..s"Aveu`a&**Q-W[B[eus # return_lossNTrkeys_to_ignore_at_inferencepast_key_valuesc3F># UHnTRU5v M g7frrM)rrIrs rr rNs(WFVdD)9)9FVrr rc3D># UHupUTS/-;dMUv M g7frNrrrrFrs rr rNs()o8KqXcgmfnXnOn!!8K  c3<># UHupUT;dM Uv M g7frrrUs rr rNs)d8KqXcOc!!8K  )rc3D># UHupUTS/-;dMUv M g7frTrrUs rr rNs&&h_TQQ\`f_gQgHgqq_rVc3<># UHupUT;dM Uv M g7frrrUs rr rNs&]_TQQ\H\qq_rXr)r<rallrrur|r@rr8rrJrrrrrrrr<rrrrrrrrrBr)rrrrr has_labelsrOloss_without_labelsr raw_outputsr logits_mbrrrs ` ` rr*Trainer.prediction_steps<"$"2"23q8UcAveieueuAv>v jj5  ..K&)$*:*:&;q&@[dV[%%f-  tzz8,,%djj&7&79VYjXkl   ,"5(WdFVFV(W#WXF6{aF ]]_&((.uf= !4!+t44"-f"5$))o 8I8I8K)o$o "-a.$/O "..0779==?D.y9FD!+t44$))d 8I8I8K)d$d $/ .y9F!4::<(,(9(9%X\(9(] =;;=--/D!'400!&&hW]]_&h!h!(D::<"'/&/=!'400!&&]W]]_&]!]!(yy++q0%,YY-A-AA-E%F QT $% %v& v;! AYFf%%7=<=>- . .$>>*=>CCGGWL*f4O&NN+>?DDIIMM$=dC(( 6D(C C()+77 !)%%  &335 %s +q GGJ ,     ) )$** 5 Q QRVR[R[RfRf g , +s F22 Gc  UR5(a(URR[R:XagURR (d-UR b UR R5(dgURRn[[[/n[[4Hn[RR!X5n[RR#U5(dMHUR%U5 ['U5n[(R*"UR-55nSSS5 [/[1WSR3555nUR5U5 M [75(a UR5[8[:[</5 UHn [RR#[RR!X55(dMF[>R@"[RR!X5[RR!X)55 M URBbURBREU5 [FRH"UR[RR!U[J55 URRL[NRP:XaSURRRT3n O"S[WURRRX53n [[UR\UU URR^SS[`S3/S9n U /n URR[Rb[Rd4;aURR[Rb:XaSO[gU5Rhn [[UR\UU U S -URR^SS 9nU R%U5 UR bUR R5(a[kU 5UlgUR RlR5U 5 g!,(df  GN6=f) N weight_mapzTraining in progress, step zTraining in progress, epoch TrSr)ri folder_pathrrc run_as_futureignore_patternszlast-checkpointz , checkpoint)rir path_in_reporrcr)7rr hub_strategyrUENDhub_always_pushrjis_donerrmrqrorprnrgrfrr8rrzjsonloadsreadrr:r;extendrrjrlrkrurPrrrrkrr!rXr rurgrr7rrcrgrP CHECKPOINTALL_CHECKPOINTSrrIrsjobs)rr-rmodeling_files index_file index_pathrr shard_files modeling_filermodel_push_job push_jobsrcheckpoint_pushs rr+Trainer._push_from_checkpointrs9))++tyy/E/E/X yy((T-B-B-NW[WlWlWtWtWvWv YY)) %|5FG-/FGJ&7DJww~~j))%%j1*% JJqvvx0E&"3u\':'A'A'C#DE %%k2H     ! !#68LNg"h i+Mww~~bggll+<LMM BGGLL):JBGGLLYcLst,  ,  ! ! 1 1* = 499bggll:7IJK 99 " "l&8&8 8:4::;Q;Q:RSN;C @P@PY>Y%Z Z%)YY%;%;{?U?U%U![_`q[r[w[w ,))-)->ii))" O   _ -  (D,A,A,I,I,K,K$29$=D !  ! ! & & - -i 8a&%s %Q22 R c[US5(dgURbPURR5(d0[R S5 URR 5 ggg)Nrjz\Waiting for the current checkpoint push to be finished, this might take a couple of minutes.)r@rjrrrwait_until_doners rrwTrainer._finish_current_pushs[t/00   ,T5J5J5R5R5T5T KKv w  ! ! 1 1 36U ,rrblockingrevisionc URSS5nUcURR(aiURRc*[ URR 5R nO(URRRS5SnUbUOURRnURcURUS9 URSS9 UR5(dg[URSS5bgS U;a/US '[US [5(a US /US 'URR H!nXuS ;dM US R#U5 M# UR$"S SU0UD6 UR'5 [)URURR UUU(+S [*S 3/US 9$)u Upload `self.model` and `self.processing_class` to the 🤗 model hub on the repo `self.args.hub_model_id`. Parameters: commit_message (`str`, *optional*, defaults to `"End of training"`): Message to commit while pushing. blocking (`bool`, *optional*, defaults to `True`): Whether the function should return only when the `git push` has finished. token (`str`, *optional*, defaults to `None`): Token with write permission to overwrite Trainer's original args. revision (`str`, *optional*): The git revision to commit from. Defaults to the head of the "main" branch. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to [`~Trainer.create_model_card`]. Returns: The URL of the repository where the model was pushed if `blocking=False`, or a `Future` object tracking the progress of the commit if `blocking=True`. rN/r)rcTrd model_tagsrprSr)rirrrcrrrr)r=rrfrcrrrIrrgrerhrr8rrrrrrrwrrP)rrrrcrrr model_tags rrdTrainer.push_to_hubs6ZZ d3  $))"7"7yy%%-!$))"6"67<< !YY3399#>rB * 0C0C    $   E  * t,))++  4::|T 2 >V#!#v&.#.."(.!1v!ZZ22 6N26N)))43 ?*?? !!#%% ,,)&,!&;%5n[AUXS9nU(diSn[CUS5(a5[EURF[H5(aURFR4n[AUU US9n[AUU US9n[AUU US9nURK5 [CURLS5(a>[OURLRJ5(aURLRK5 URPS:aSUl)XRTl+[YU5GHunnUR[UUX4S9unnnRSS5nSUR\;aUR_UU5OSnUb-URaU 5nU cUO[&Rb"U U4SS9n UbU cUO [eU USS9n UbU cUO [eU USS9n UbUcUO [eUUSS9nURTRgX`RhURj5Ul5URRl(a~URnbqU bnU bkURRpRrn0n SUR\;aU OSU S'SUR\;aUOSU S'URo[uS(XS.U D6US9nURRl(d)URvcGMUS-URv-S:XdGMURyUR{U S55 U(dcWRyUR{U S 55 WRyUR{U S!55 WRyUR{US"55 A A A A[&R|R5 S#uppGM URP(a[CUS$5(a [US$5 URyUR{U S55 U(dcWRyUR{U S 55 WRyUR{U S!55 WRyUR{US"55 UR5n!U(dWR5OSn"U(dWR5OSn#U(dWR5OSn$URnbmU"bjU#bgURRl(dLSUR\;aU!OSUS'SUR\;aU$OSUS'URo[uS(U"U#S.UD65nOUc0n[U5nU!b#U!R5R5XS%3'[UR55H6n%U%RUS&35(aMURU%5XS&U%3'M8 [U"U#XS'9$))rz+dataloader must implement a working __len__NrTrFrrr_is_accelerate_preparedz\Batch size cannot be None. Ensure the dataloader has a valid batch_size or total_batch_size.rrrrr )make_multiple_ofrrrrr(r)rrrrrr r! eval_losses eval_predseval_label_idseval_inputs_ids)NNNNrr"rnr$r)JrrarrrHr(r$rrr<rTr%rOrr'rSr)rLrrr(r=rMrxr8rrr$rrrrr<r@rrrBrrYrirBrr^rrFr*r+rr-catrIr/rur|rrrGr0rSr1 add_arrays_gather_and_numpifyrrrjfinalizer]rrrrrr=rR)&rrrrrrrrnrrr$ losses_host preds_host labels_host inputs_hostrJr:reval_losses_gathererrpreds_gathererlabels_gathererinputs_gathererrtrrrrrr=rr>r? eval_losspredsr inputs_idsrs& rrTrainer.prediction_loop syy*%%JK K7K7W3]a]v]v  $ $)?!$1MDAq  e  S t'' (A -%::2E,,0D0D  ((/%%33E43P  ##" JJ&%*"((!%!3!3""u}}T[[I$$u~~dkkJz#>6 * *x8K8K/L/L NN   ! ??a DJ0:-%j1LD&#'#7#7vG[#7#u D&&%djj2C[QO@HDLdLd@d##F?$;&>z<&X Y  & &t'?'? M]'^ _  & &t'?'? M^'_ `(113 1E'')46JO,,.PT 7K_--/QU   ,!%II005;t?W?W5W ]aOH %6>$BZBZ6Z `dOH %**>+teW`+tds+tuG _G(0  3<>>3C3H3H3JG)/ 0 'C>>%6$7q"9::8? C8H-Qse45(%9gpprcUcg[5(a [X5nON[5(a [U5nO3URR [ R:Xa [U5n[U5$)rBN) rrLrrrrIrhrJrDrKrGs rrTrainer._gather_and_numpifys_ ?  ! # #,W;G $ & & )G YY $ $ (@(@ @(1Gg&&rc>UR5(dgSS/n[RR[RR UR R S55(aW[[RR UR R S55nUR5nSSS5 OSnWnUH.nXT;dM URS5(aXE- nM&USU3- nM0 XC:wap[[RR UR R S5S5n[RSU35 URU5 SSS5 UR RS5 [R"S 5 UR R!5(d6UR R#S 5 UR R%5 gg!,(df  GNA=f!,(df  N=f) z8Add SageMaker Checkpointing patterns to .gitignore file.Nz*.sagemaker-uploadingz*.sagemaker-uploadedz .gitignorer rwz"Writing .gitignore file. Content: g?z'Add *.sagemaker patterns to .gitignore.)rrgrfrrrepo local_dirrzrryrrGr{git_addrsleep is_repo_clean git_commitgit_push)rpatternsrcurrent_contentcontentpatterns r_add_sm_patterns_to_gitignore%Trainer._add_sm_patterns_to_gitignores))++ +-CD 77>>"'',,tyy':':LI J Jbggll499#6#6 EF!"#&&(GF!O"G%##D))&GG9~-G    %bggll499#6#6 EsKq A'KL L ,' 3yy&&(( II !J K II   )3GF LKs.G<*H< H  Hc  0n[S5(aAURRRb URRRnSU;a9URRS:a [ S5eUSURlURRR 5n[S5(aZ/SQn[S!0UVs0sHoDURU5_M snD6n[S5(aURRUl URS5n[S5(dU(a [S 5eO>U(a0URR(d[RS 5 UWlURS 5 S URR0n[S5(aWUS 'OUR!U5 [#UR$S5(aUR$R&bUR$R&S:aeSUl[*R,"[.5[*R,"S5:a![1UR$R&S9US'O [ S5e[3S!0UD6UlUR4R6UlS[:R<"UR85R>RA5;a8[BRD"UR8URRFS9Ul[IUR4RJS S5SLUl&[IUR4RJSS5SLUl'[IUR4RJSS5SLUlURN(aUR4RJRPnSH<n[SXURRTRWU[IX555 M> URX(a&URRZ(a [ S5eURL(a([IURSS5cUR]5 URR^(a`URL(dURN(a>URR`(a#URL(aSOSn [ U S35eURL(aTUR4RJRRbS:Xa&URRd(a [ S5eURR^(aUURN(aCS[gUR4RJRPRh5;a [ S 5egggs snf)"Nrr!r zThe `AcceleratorConfig`'s `num_steps` is set but `gradient_accumulation_steps` is greater than 1 in the passed `TrainingArguments`If using the passed `AcceleratorConfig` is desired, do not set the `TrainingArguments` `gradient_accumulation_steps`.) split_batchesdispatch_batches even_batchesuse_seedable_samplerz1.1.0 non_blockingrzp`non_blocking` is only supported in accelerate v0.30.0 and above. Please upgrade accelerate to use this feature.zx`non_blocking` is enabled but `dataloader_pin_memory` is not. For the best performance, it's recommended to enable both.gradient_accumulation_kwargsrDdataloader_configtp_sizeTr)rtorch_tp_pluginz4Requires accelerate>1.3.0 to use Tensor Parallelism.use_gather_object)rr.)limit_all_gathersactivation_checkpointingzThe activation_checkpointing in FSDP config and the gradient_checkpointing in training arg can't be set to True simultaneously. Please use FSDP's activation_checkpointing logic when using FSDP.rB DeepSpeedrzJ can't be used with `save_only_model` along with `load_best_model_at_end`.zo`auto_find_batch_size` isn't supported yet with DeepSpeed Zero-3. Please consider using Zero-2, Zero-1, or FSDPSHARDED_STATE_DICTzWsave_only_model option is not compatible with FSDP state dict type 'SHARDED_STATE_DICT'r)5rxraccelerator_configrrrto_dictrr= data_seedrBrrrArrDrr@rr is_tp_enabledrrrrrrTr3r,rrrrrreval_use_gather_objectr8rurHrOr.r>rFrrr1r*r'r# zero_stagerrr) rgrad_acc_kwargsrrrrrrr.wrappers rr**Trainer.create_accelerator_and_postprocesss "8 , ,1M1M1j1j1v"ii::WWO / )yy44q8 L 9H 8T 5!YY99AAC "8 , , m  7!EVWEVE,0077EVW! 'w//.2ii.A.A!+)--n= &x00!G DII$C$CO.:  *=>  : :  #8 , ,(9D$ % KK* + 4::y ) )djj.@.@.LQUQ[Q[QcQcfgQg!%D }}/07==3II*CDJJL^L^*_&' !WXX'..#//BB '"3"3D4H4H"I"T"T"Y"Y"[ [#,#4#4$$ 8X8X$D %,D,<,<,B,BDVX\$]ei$i!&t'7'7'='=}dS[__$T%5%5%;%;=NPTU]aa   **00<%>kFGy(rst t  % %  &&77BBaG ..B  II % %$$$D,<,<,B,B,N,N,^,^(__vw w`% &iXsW cSSKJn URRRnU"UR R 5UlUR R UlUR RURU5 g)z? Sets values in the deepspeed plugin based on the Trainer args rr3N) rAr4rTrurDr5rdeepspeed_configrCr)rrr4 ds_plugins rr*#Trainer.propagate_args_to_deepspeedmsg Q$$**;; !9):P:P:W:W!X %.%;%;%B%B "55diiAUVrcUR(Gap[UR5(GaTSSKJn SSKJn [URRU5(a5U"UR5URRRl [URSS5[R:XaURR R"R$R&(a[(R*"[,5[(R*"S5:aWURRRR/URR R"R$SS9 gggggg)Nr) PeftConfig)fsdp_auto_wrap_policyrz0.27.0T)r)rOrrrrpeft.utils.otherrractive_peft_configrTrur.rr8rrPrrDbnb_4bit_quant_storagerrrrset_mixed_precision)rrrs rr3"Trainer._fsdp_qlora_plugin_updatesys    N4::$>$> ' >$**77DDF[\`\f\fFg  &&22C $94@DVDeDeeJJ++??VVhhMM"45 h8OO  &&22FFJJ++??VVaeGPif%? rc:/nSn[U5HnUR[U55 M [ U5S:=(a0 SUS;=(a! UR =(d UR SLnU(a<[UVs/sH$oSRS5R5PM& sn5nUbURR(a)URRU5R5n[R "U5(aPUR#U5nURR$S:a%UR'5S:XaUR)S5nXE4$![a  GMLf=fs snf![[4a Nf=f)Nrrrr )rrrz StopIterationr<rVrrCner"r!rrrTrLr is_tensorrrr unsqueeze) rrrr=rrrncount_num_items_in_batchr+s rrETrainer.get_batch_sampless !{#A $$T.%9:$   " M!,,  ..6))5 ! $ %(Xe)fXeu?+=+=d+C*H*H*JXe)f%g"  )yy66%)%5%5%<%<=O%P%T%T%V"122%7%:%:6%B"99??Q&+=+A+A+Cq+H);)E)Ea)H&00I!  **g~.  s5E/? F+F3F/ E?>E?FFFrycURnUS:n[U5(a [U5OSnUb][XaR-[ XaR-S:5-S5nU(a#[ R"URU-5nU(a{URU5nURS:aUW-[ XG-S:5-n XC-n O[ R"UR5n URU5UR-n OXURS:a0[Rn UnX1R-nURU-n O[SUR35eU WUU UUU4$)z Calculates and returns the following values: - `num_train_epochs` - `num_update_steps_per_epoch` - `num_examples` - `num_train_samples` - `epoch_based` - `len_dataloader` - `max_steps` rNr zYargs.max_steps must be set to a positive value if dataloader does not have a length, was ) rjrar<rrrrrrkr$sysmaxsizer) rrrryrjr|r}rzr$rkr{s rr+#Trainer.set_initial_training_valuessNN !m ,6z,B,BZ  %),"B"BBn'G'GG!KLM* &  IId&;&;>X&XY  ,,Z8L~~!#,0J#JS:Q>N$ %.$F!#'99T-B-B#C $($5$5j$ADDYDY$Y! ^^a "{{ )2 &1NNBL $1G G NN#%   &       r):rrr>rbr,rror=rr6rTryrr^rurrrQr|r}rr(rr,r'r~r8rcrHrOrGrrr)r9rrrr{rZrrVrr#rSrrXrNrYrrKrrrjrurrprq) NNNNNNNNNNrNN)rNr)NFN)F)TN)NNN)NNNNN)NNminimizeNNr)FN)NF)NNr)Nr) NNNNNNNNN)zEnd of trainingTNN)r7 __module__ __qualname____firstlineno____doc__trainer_pt_utilsrrrrrrr r)rModulerir rrrrrr3r#r r0r rSrr9rrr> OptimizerrZLambdaLRrrrrpropertyrsetterrrr_rrrQrrrrlrmSamplerrrrrrr rrr r'r.r$rVr] parameterrr` staticmethodr<r%r$r-rHrrar[r1rrrrrr\rrqrrnrrArarDrrr*r(r5r)r6r9rTrQrrprr|rrrRrrrrhrrrorPrrr,rrVr rRrrr*rVrerr+rwrdrrrr*r*r3rEr+__static_attributes__rrrrr>sC \~lk[+=wdhi:>"&04W[Y] >B04FJ59jvaehl!Ic_bii56Ic Ic - Ic  g@R&R ST Ic uWd3<.@BT%TUV Ic# )+=?UWee f IcXb/&9:;Ic$H-Ic"(N+;T+A"BCIcD12Ic(5;;#8#898EKKD\D\DeDe;ffgIc#+5ekk6K6K1LdSVX[S[n1\+]"^Ic (0%,, 9UW\WcWc9c0d'e!IcjIcV %8$;<%%11 &<" 5< 8 \;.@;xX[};BEI'%'4B=A48 o (sDy)9 :o^T#s(^T9:o'tCy1 odimcHJ(X=tzx ,hl'^R)_V$"L1If0id?1Bq5f1(56%9RLPKO+5;?=AR8^$4d3:6F$FGHR$Hd3:.>-?-F$GHR R d3i( R % 678 R(N#3S#89:R wW % &Rh_S%Z(_huo_QU_.5s):#;ellTWFW@X$d3ellC6G0H+H&IdSVX]^c^j^jlo^oXpSpNq 5 HTN  aeN!YYN!(,S% c8I2J-J(KN! N!`2;h2t2 0t 01:Xc]1:41:fF>HSMF>P)L )LV "1FRW#" c#"J:6FJ+/!' fuWd3<.@%@ABfd3i(f f c5j  fRhn>t#>t2:492E>tad>t >tH04+/!' UuUuUu'tn Uu d3i( Uu  Uu Uun0,0 g&yyg&S% c 1223g&# g& d3i( g& x %x '=x ?UU V g&Rc5s9J3K.K)L$%(3-%(#'!%,0$((,-148/348Hh3-Hh#HhCcD() Hh SM Hh ! HhS$s)T)*HhCcD01HhsDIt+,HhCcD01HhT@9D4):#"& J   J J } J 3- J  J h04+/!' xqxqxq'tn xq d3i( xq  xq xqt' %!Noxb W +1Z> %> 3=> WZ> rr(rrrPrr_importlib.metadatarrrrrgrrrurrYrr3collections.abcrrpathlibrtypingrrr r r integrationsr huggingface_hub.utilsrlrrrrtorch.distributed distributedrmhuggingface_hubrrr packagingrrtorch.utils.datarrrrrrrconfiguration_utilsrdata.data_collatorrrr debug_utilsrr!feature_extraction_sequence_utilsrfeature_extraction_utilsr rr!r"image_processing_utilsr#integrations.deepspeedr$r%r&integrations.tpur' modelcardr(modeling_utilsr)r*r+models.auto.modeling_autor,r- optimizationr.r/processing_utilsr0 pytorch_utilsr1r2tokenization_utils_baser3trainer_callbackr4r5r6r7r8r9r:r;rr<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrO trainer_utilsrPrQrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxryrzr{r|r}r~rrrrrrrrrrrrrrrrrrrrutils.deprecationrutils.import_utilsrutils.quantization_configrr\rautils.notebookrrrrtorch_xla.core.xla_modelr xla_modelrtorch_xla.debug.metricsrGrJrhtorch_xla.runtimeruntimer torch_xla XLA_VERSIONrrtorch_xla.distributed.spmdspmdr!smdistributed.modelparallel.torch modelparallelrtsmdistributed.modelparallel SMP_VERSIONrsrrrrsafetensors.torchrrrrrrraccelerate.staterr@rrrrrrr DATA_SAMPLERSraccelerate.data_loaderrrrrrrrR get_loggerr7rrrjrmrirUrnrrrrrrBsB    #@@  - AAcc1\\<G<`6ee1&RR3,=   ,2KJ***********V/(9)),8 8))))"4$]];77==I_;``//"33F ' k :gmmF>S Sgg %:<1#OM}}'(7==+AA>}}'(7==+BB@@/00 >8$$8 7&   H %)) $&  pN pN  pN r