o ZhT @s&dZddlZddlmZddlmZmZddlZddlm Z ddl Z ddl m Z ddlmZmZddlZddlmZmZdd lmZmZerKddlZerZdd lmZdd lmZeeZ dd dZ!dee"de j#dee"e j#fde"dee"e j#ff ddZ$ddZ%ddZ&ddZ'ddZ(dS)z!PyTorch - Flax general utilities.N)UnpicklingError)DictTuple) from_bytes) flatten_dictunflatten_dict)is_safetensors_availableis_torch_available)check_torch_load_is_safelogging) safe_open) load_fileFc Cs|swtj|}td||drLsz=load_pytorch_checkpoint_in_flax_state_dict..,z parameters.)ospathabspathloggerinfoendswithr keysZ get_tensortorch ImportErrorModuleNotFoundErrorerrorr loadsumvalues"convert_pytorch_state_dict_to_flax*convert_pytorch_sharded_state_dict_to_flax) flax_modelZpytorch_checkpoint_pathZ is_shardedZallow_missing_keysZpt_path pt_state_dictfkr flax_state_dictrrr*load_pytorch_checkpoint_in_flax_state_dict1s4    &  r. pt_tuple_key pt_tensorrandom_flax_state_dict model_prefixreturncsdttdtffdd }|ddd}|ddvr$||r$||fS|ddd }|dd kr:||s:||fS|ddd }|dd krP||sP||fS|ddd }|ddkrf||rf||fS|ddd}|ddkr|jdkr||s|dddd}||fS|ddd}|ddkr||s|j}||fS|ddd}|ddkr||fS|ddd}|ddkr||fSd}|ddddkr|dd}n|ddddkr|dd}|dur|dd|f}||fS||fS)zYRename PT weight names to corresponding Flax weight names and reshape tensor if necessarykeyr3cstt|f|h@dkS)zAChecks if `key` of `(prefix,) + key` is in random_flax_state_dictr)lenset)r4r2r1rris_key_or_prefix_key_in_dict]szCrename_key_and_reshape_tensor..is_key_or_prefix_key_in_dictN)scale)weightgamma)mean running_mean)var running_var) embeddingr;)kernelrrr;r<)Zbiasbeta)parametrizations original0_g)rI original1_v)rstrboolndim transposeT)r/r0r1r2r8Zrenamed_pt_tuple_keynamerr7rrename_key_and_reshape_tensorUsH   rUc Csftottt|tj}|rtjnd}dd|D}|r;|D]\}}|j |kr2| }| ||<q%|j }d|jvrI|jd}n|j}t|} d|jvrat|jd} | | i} ||voq|dd|Dv} ||vo|dd|Dv} |D]\}}t|d }|||k}|d |k}| r|r|d d}t||| |\}}|f|| v}| r|r|f|}|| vr|j| |jkrtd |d | |jd|jd d|jvrd|dvsd|dvrt|| d|<qd|dvr| |dq|st|ntj|tjd| d|<q|s$t|ntj|tjd| |<qt| S)Nbfloat16cSi|]\}}||jqSrdtyperr,vrrr z6convert_pytorch_state_dict_to_flax..params batch_statscSh|] }|ddqS.rsplitrr,rrr z5convert_pytorch_state_dict_to_flax..cSr`rarcrerrrrfrgrbrr1PyTorch checkpoint seems to be incorrect. Weight  was expected to be of shape , but is r=r9r?r_num_batches_trackedrXr^)r isinstancenextiterr&r ZTensorrVitemsrYfloatrnumpybase_model_prefixr^rupdatertuplerdrUshape ValueErrorjnpasarraypopr)r*r)Zfrom_binrV weight_dtypesr,r[r2flax_model_paramsr1Zflax_batch_statsr-$load_model_with_head_into_base_model$load_base_model_into_model_with_headpt_keyr0r/ is_bfloat_16has_base_model_prefixflax_key flax_tensorrequire_base_model_prefixrrrr'sp            r'c sNddli}|D]}tj|dd}dd|D}fdd|D}|j}d|jvrE|jd}t|}|t|jdn|j}t|}||voZ|d d |Dv} ||voi|d d |Dv} |D]\} } t | d } || j k}| d|k}| r|r| d d} t | | ||\}}|f||v}| r|r|f|}||vr|j ||j krtd| d||j d|j d d|jvrd|dvrt||d|<qnd|dvrt||d|<qnd|dvr||dqn|st|ntj|tj d|d|<qn|st|ntj|tj d||<qnqt|S)NrT)rcSrWrrXrZrrrr\r]z>convert_pytorch_sharded_state_dict_to_flax..cs2i|]\}}||jjkr|n|qSr)rYrVrsrrrZr rrr\s&r_r^cSr`rarcrerrrrfrgz=convert_pytorch_sharded_state_dict_to_flax..cSr`rarcrerrrrfrgrbrrhrirjr=r9rkr?rlrXrm)r r r$rqrtr^rrurrvrdrVrUrwrxryrzr{r)Zshard_filenamesr)r-Z shard_filer*r|r2r}r1r~rrr0r/rrrrrrrrr(sv             0r(c Cstj|}td|ttd|jj}| dr't |}t |dd}n.t |d!}z t ||}WntyEtd|dwWd n1sPwYt||S) (Load flax checkpoints in a PyTorch modelzLoading Flax weights from ZFlaxrrb)seprbzUnable to convert z to Flax deserializable object. N)rrrrrgetattr transformers __class____name__rsafe_load_fileropenrreadrEnvironmentError"load_flax_weights_in_pytorch_model)modelZflax_checkpoint_pathZflax_clsr-Zstate_frrr%load_flax_checkpoint_in_pytorch_modelMs     rc Cszddl}Wnttfytdwttjdd| }t |r5t dtjdd|}t|}| }|j |voM|j dd |Dv}|j |vo^|j d d |Dv}g}t|} |D]9\} } | d|j k} d |j f| |v} |r| r| d d} n |r| r|j f| } | d dkr| jdkrd | |vr| dd d} t| d} nE| d dkrd | |vr| dd d} | j} n,| d dvr| dd d} nd| d vr| dd d} nd| d vr| dd d} d|vr d | d d}nd | }i}|D]F}|d }d}|dddddgkr1|dd}n|dddddgkrC|dd}|durZ|dd|g}d |}|||<q||vre||}||vr| j||jkrtd | d!||jd"| jd t| tjst| n| } || ||<| |qk||qk| |t!| } t"|dkrt d#|j#j$d$|d%|j#j$d&|j#j$d' n t d(|j#j$d)t"| dkrt d*|j#j$d+| d,|St d-|j#j$d.|j#j$d/|S)0rrNzLoading a Flax weights in PyTorch, requires both PyTorch and Flax to be installed. Please see https://pytorch.org/ and https://flax.readthedocs.io/en/latest/index.html#installation for installation instructions.cSs |jtjkSr)rYryrV)xrrrqs z4load_flax_weights_in_pytorch_model..zFound ``bfloat16`` weights in Flax model. Casting all ``bfloat16`` weights to ``float32`` before loading those in PyTorch model.cSs|jtjkr |tjS|Sr)rYryrVZastypenpZfloat32rmrrrrzscSr`rarcrerrrrfrgz5load_flax_weights_in_pytorch_model..cSr`rarcrerrrrfrgrbrr9rBrCrF)rErDrr)r:rAr=)r>r?)r@r_rHrDrIrJrKrLrMrNz.Flax checkpoint seems to be incorrect. Weight rirjzQSome weights of the Flax model were not used when initializing the PyTorch model z: z, - This IS expected if you are initializing z from a Flax model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a FlaxBertForPreTraining model). - This IS NOT expected if you are initializing z from a Flax model that you expect to be exactly identical (e.g. initializing a BertForSequenceClassification model from a FlaxBertForSequenceClassification model).z3All Flax model weights were used when initializing z. zSome weights of zE were not initialized from the Flax model and are newly initialized: zo You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.zAll the weights of z were initialized from the Flax model. If your task is similar to the task the model of the checkpoint was trained on, you can already use z* for predictions without further training.)%r r!r"rr#rjaxZ tree_utilZtree_mapr&anywarningZ state_dictrtrr6rqjoinrQryrRrSrdrwrxrnrndarrayrzZ from_numpyremoveappendZload_state_dictlistr5rr)Zpt_modelZ flax_stater Z is_type_bf16r-Z pt_model_dictr~rZunexpected_keysZ missing_keysZflax_key_tuplerrrrZspecial_pt_namesr4Zkey_componentsrTZ key_to_checkrrrrcs      $                  r)F))__doc__rpicklertypingrrrZ jax.numpyrsryrZflax.serializationrZflax.traverse_utilrrrr r utilsr r r Z safetensorsr Zsafetensors.flaxrrZ get_loggerrrr.rOrrUr'r(rrrrrrsF       $   C\Y