o ZhT@sdZddlZddlmZddlZddlZddlm Z ddl m Z ddl m Z mZmZer7ddlZddlmZeeZdd Ze rMerMdd lmZndd lmZGd d d eZGdddeZdaddZddZddZddZ ddZ!ddZ"d"ddZ#d#d d!Z$dS)$z Integration with Deepspeed N) partialmethod)dep_version_check)is_accelerate_availableis_torch_availablelogging)nncCs@tjddu}|rztd}WdStjyYdSwdS)N deepspeedTF) importlibutil find_specimportlib_metadatametadataZPackageNotFoundError)Zpackage_exists_rR/var/www/auris/lib/python3.10/site-packages/transformers/integrations/deepspeed.pyis_deepspeed_available$s r)HfDeepSpeedConfig)objectcs eZdZdZfddZZS)raJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. cs(t|tdtdt|dS)NZ accelerater )set_hf_deepspeed_configrsuper__init__selfZconfig_file_or_dict __class__rrrJszHfDeepSpeedConfig.__init__)__name__ __module__ __qualname____doc__r __classcell__rrrrr9srcsXeZdZdZfddZddZddZdd d Zeed d Z dddZ ddZ Z S)HfTrainerDeepSpeedConfigz The `HfTrainerDeepSpeedConfig` object is meant to be created during `TrainingArguments` object creation and has the same lifespan as the latter. cst|d|_g|_dSN)rr_dtype mismatchesrrrrrXs  z!HfTrainerDeepSpeedConfig.__init__cCs|jdur td|jS)Nz8trainer_config_process() wasn't called yet to tell dtype)r# ValueErrorrrrrdtype]s zHfTrainerDeepSpeedConfig.dtypecCs||}|dur dS|dkS)NFauto) get_value)r ds_key_longvalrrris_autobs z HfTrainerDeepSpeedConfig.is_autoNTc Cs||\}}|dur dS||dkr|||<dS|sdS||}|dur?||krA|jd|d|d|d|dSdSdS)a A utility method that massages the config file and can optionally verify that the values match. 1. Replace "auto" values with `TrainingArguments` value. 2. If it wasn't "auto" and `must_match` is true, then check that DS config matches Trainer config values and if mismatched add the entry to `self.mismatched` - will assert during `trainer_config_finalize` for one or more mismatches. Nr(z- ds =z vs hf )Zfind_config_nodegetr$append)rr*Zhf_valZhf_key must_matchconfigZds_keyZds_valrrr fill_matchis  (z#HfTrainerDeepSpeedConfig.fill_matchF)r0cCs|j|j|j}|d|jd| |d|jd|d|d| |d|jd|d|jd |d |j|jgd |d |jd |d|j d| dd|d|jd |j sc|j rm|j dkrjdnd}nd}|jr|jdi|jd<|j|jdd<|d|j s|j o|dkd|d|dkd|d|jd|d|jp|jd|drtj|_dS|drtj|_dStj|_dS) z Adjust the config with `TrainingArguments` values. This stage is run during `TrainingArguments` object creation. Ztrain_micro_batch_size_per_gpuper_device_train_batch_sizegradient_accumulation_stepstrain_batch_sizeztrain_batch_size (calculated)Zgradient_clipping max_grad_normzoptimizer.params.lr learning_ratezoptimizer.params.betaszadam_beta1+adam_beta2zoptimizer.params.eps adam_epsilonzoptimizer.params.weight_decay weight_decayzscheduler.params.warmup_min_lrrzscheduler.params.warmup_max_lrZapexampN checkpointZuse_node_local_storagez fp16.enabledz%fp16|fp16_full_eval+fp16_backend(amp)z amp.enabledzfp16+fp16_backend(apex)z amp.opt_levelfp16_opt_levelz bf16.enabledzbf16|bf16_full_eval)Z world_sizer3r4r2r6r7Z adam_beta1Z adam_beta2r8r9 fill_onlyZfp16Zfp16_full_eval fp16_backendZsave_on_each_noder1r.r<Zbf16Zbf16_full_evalZis_truetorchZbfloat16r#Zis_falseZfloat32Zfloat16)rargsZauto_find_batch_sizer5r>rrrtrainer_config_processs`        z/HfTrainerDeepSpeedConfig.trainer_config_processcsFgd}fdd|D}t|dkr{t|jdr|jj}n=t|jdr+t|jj}n0t|jdr>t|jjdr>|jjj}nt|jdrSt|jjdrSt|jjj}ntd|d d || r{d t d ||d d| d|d d| |dtj dkrdj }td|ddS)z This stage is run after we have the model and know num_training_steps. Now we can complete the configuration process. )$zero_optimization.reduce_bucket_size-zero_optimization.stage3_prefetch_bucket_size4zero_optimization.stage3_param_persistence_thresholdcsg|] }|r|qSr)r,).0xr&rr zDHfTrainerDeepSpeedConfig.trainer_config_finalize..r hidden_size hidden_sizes text_configzThe model's config file has neither `hidden_size` nor `hidden_sizes` entry, therefore it's not possible to automatically fill out the following `auto` entries in the DeepSpeed config file: zb. You can fix that by replacing `auto` values for these keys with an integer value of your choice.rBrCg?rD z scheduler.params.total_num_stepsznum_training_steps (calculated)z!scheduler.params.warmup_num_stepsZ warmup_steps z]Please correct the following DeepSpeed config values that mismatch TrainingArguments values: zF The easiest method is to set these DeepSpeed config values to 'auto'.N)lenhasattrr1rImaxrJrKr%r=is_zero3intr2Zget_warmup_stepsr$join)rr@modelnum_training_stepsZhidden_size_based_keysZhidden_size_auto_keysrIr$rr&rtrainer_config_finalizesX       z0HfTrainerDeepSpeedConfig.trainer_config_finalize)NTF) rrrrrr'r,r2rr=rArVr rrrrr!Rs    Jr!cCst|adSr")weakrefref_hf_deepspeed_config_weak_ref)Zhf_deepspeed_config_objrrrrsrcCsdadSr")rZrrrrunset_hf_deepspeed_configsr[cCs tdurtdurtSdS)NF)rZrQrrrris_deepspeed_zero3_enabled$s r\cCstdur tdur tjSdSr")rZr1rrrrdeepspeed_config+sr]csRt|dd|}dur|_gd dtjffdd ||ddS) z Loads state dict into a model specifically for Zero3, since DeepSpeed does not support the `transformers` tensor parallelism API. Nearly identical code to PyTorch's `_load_from_state_dict` _metadataNFmodulec sdurin ddi}||d<||dggf}trstfdd|Ddkrsddl}t|jddddfd d|D}t|dkrs|jj|dd t j dkrd|j |Wdn1snwY|j D]\}} | dur| ||d |qxdS) Nassign_to_params_buffersTcsg|] }|r|qSr) startswith)rEkey)prefixrrrGJrHzC_load_state_dict_into_zero3_model..load..rF)rerecursecsg|] }|vr|qSrr)rEk)named_parametersrrrGPs)Z modifier_rank.)r.r\rNr dictrhkeyszeroZGatheredParametersr? distributedZget_rankZ_load_from_state_dictZ_modulesitems) r` state_dictrerbZlocal_metadatar@r Zparams_to_gathernamechild error_msgsloadr)rhrerrtCs$    z/_load_state_dict_into_zero3_model..load)rb)r_F)getattrcopyr^rModule)Z model_to_loadrorrrr!_load_state_dict_into_zero3_model2s rxc sddlm}m}|j}d}d|vr|jrtd||d}n|r'td }d|d <d} d |vr=||} || fSt ||rSfd d } ||| d } || fSj |d} || fS)zY A convenience wrapper that deals with optimizer and lr scheduler configuration. r) DummyOptimDummySchedulerN optimizerz|--adafactor was passed, but also found `optimizer` configured in the DeepSpeed config. Only one optimizer can be configured.)paramszDetected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)TZzero_allow_untested_optimizerZ schedulercs"t}d|_|j|d}|S)NrUr{)rv lr_schedulercreate_scheduler)r{Z trainer_copyr~rUtrainerrr_lr_scheduler_callables z5deepspeed_optim_sched.._lr_scheduler_callable)Zlr_scheduler_callabler}) Zaccelerate.utilsryrzr1Z adafactorr%Z is_offloadloggerinfoZcreate_optimizer isinstancer) rhf_deepspeed_configr@rUmodel_parametersryrzr1r{r~rrrrdeepspeed_optim_schedbs2    rFc Csddlm}|j}|j}|jjjj}||||| | |r>| s*t d| d| dd\}}d} ||fSd|_|jdid d} | d kr^ddl} | j|| |d }ttd d |} t||||| \}}||fS)a Init DeepSpeed, after updating the DeepSpeed configuration with any relevant Trainer's args. If `resume_from_checkpoint` was passed then an attempt to resume from a previously saved checkpoint will be made. Args: trainer: Trainer object num_training_steps: per single gpu resume_from_checkpoint: path to a checkpoint if to resume from after normal DeepSpeedEngine load inference: launch in inference mode (no optimizer and no lr scheduler) auto_find_batch_size: whether to ignore the `train_micro_batch_size_per_gpu` argument as it's being set automatically by the auto batch size finder Returns: optimizer, lr_scheduler We may use `deepspeed_init` more than once during the life of Trainer, when we do - it's a temp hack based on: https://github.com/deepspeedai/DeepSpeed/issues/1394#issuecomment-937405374 until Deepspeed fixes a bug where it can't resume from a checkpoint after it did some stepping https://github.com/deepspeedai/DeepSpeed/issues/1612 r)rzMZeRO inference only makes sense with ZeRO Stage 3 - please adjust your configr{r~)NNNZtensor_parallelZ autotp_size)rTtp_sizer'cSs|jSr")Z requires_grad)prrrsz deepspeed_init..)Zdeepspeed.utilsrrTr@Z acceleratorstateZdeepspeed_pluginZ hf_ds_configrVsetLevelZget_process_log_levelrQr%Zdel_config_sub_treer{r1r.r Z tp_model_initr'listfilter parametersr) rrUZ inferenceZ ds_loggerrTr@rr{r~rrr rrrdeepspeed_inits0     rTcCsvddl}t||d}t|dkr4td||j||ddd\}}|dur2td|dStd|)Nrz /global_step*zAttempting to resume from T)load_module_strictZload_optimizer_statesZload_lr_scheduler_statesz-[deepspeed] failed to resume from checkpoint z!Can't find a valid checkpoint at )globsortedrNrrZload_checkpointr%)Zdeepspeed_engineZcheckpoint_pathrrZdeepspeed_checkpoint_dirsZ load_pathrrrrdeepspeed_load_checkpoints  rrW)T)%rrvimportlib.metadatarr importlib.utilr rX functoolsrZdependency_versions_checkrutilsrrrr?rZ get_loggerrrrZaccelerate.utils.deepspeedrZDeepSpeedConfigbuiltinsrr!rZrr[r\r]rxrrrrrrrs8        B0 =>