o Zh-@s(dZddlZddlZddlZddlZddlmZddlmZddl m Z m Z ddl m Z mZmZmZmZmZmZmZmZdd lmZmZmZerOdd lmZeeZd Zd Z d Z!dZ"dZ#ddZ$GdddZ%Gddde%Z&d#ddZ'd$ddZ(ddZ)ddZ*dd Z+Gd!d"d"eZ,dgZ-dS)%z-Factory function to build auto-model classes.N) OrderedDict)PretrainedConfig)get_class_from_dynamic_moduleresolve_trust_remote_code) CONFIG_NAME cached_file copy_funcextract_commit_hashfind_adapter_config_fileis_peft_availableis_torch_availableloggingrequires_backends) AutoConfigmodel_type_to_module_name!replace_list_option_in_docstrings)GenerationMixinaJ This is a generic model class that will be instantiated as one of the model classes of the library when created with the [`~BaseAutoModelClass.from_pretrained`] class method or the [`~BaseAutoModelClass.from_config`] class method. This class cannot be instantiated directly using `__init__()` (throws an error). ax Instantiates one of the model classes of the library from a configuration. Note: Loading a model from its configuration file does **not** load the model weights. It only affects the model's configuration. Use [`~BaseAutoModelClass.from_pretrained`] to load the model weights. Args: config ([`PretrainedConfig`]): The model class to instantiate is selected based on the configuration class: List options attn_implementation (`str`, *optional*): The attention implementation to use in the model (if relevant). Can be any of `"eager"` (manual implementation of the attention), `"sdpa"` (using [`F.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html)), or `"flash_attention_2"` (using [Dao-AILab/flash-attention](https://github.com/Dao-AILab/flash-attention)). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual `"eager"` implementation. Examples: ```python >>> from transformers import AutoConfig, BaseAutoModelClass >>> # Download configuration from huggingface.co and cache. >>> config = AutoConfig.from_pretrained("checkpoint_placeholder") >>> model = BaseAutoModelClass.from_config(config) ``` ac Instantiate one of the model classes of the library from a pretrained model. The model class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it's missing, by falling back to using pattern matching on `pretrained_model_name_or_path`: List options The model is set in evaluation mode by default using `model.eval()` (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with `model.train()` Args: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. - A path or url to a *tensorflow index checkpoint file* (e.g, `./tf_model/model.ckpt.index`). In this case, `from_tf` should be set to `True` and a configuration object should be provided as `config` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards. model_args (additional positional arguments, *optional*): Will be passed along to the underlying model `__init__()` method. config ([`PretrainedConfig`], *optional*): Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when: - The model is a model provided by the library (loaded with the *model id* string of a pretrained model). - The model was saved using [`~PreTrainedModel.save_pretrained`] and is reloaded by supplying the save directory. - The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a configuration JSON file named *config.json* is found in the directory. state_dict (*Dict[str, torch.Tensor]*, *optional*): A state dictionary to use instead of a state dictionary loaded from saved weights file. This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using [`~PreTrainedModel.save_pretrained`] and [`~PreTrainedModel.from_pretrained`] is not a simpler option. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. from_tf (`bool`, *optional*, defaults to `False`): Load the model weights from a TensorFlow checkpoint save file (see docstring of `pretrained_model_name_or_path` argument). force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. output_loading_info(`bool`, *optional*, defaults to `False`): Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (e.g., not try downloading the model). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. code_revision (`str`, *optional*, defaults to `"main"`): The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. kwargs (additional keyword arguments, *optional*): Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). Behaves differently depending on whether a `config` is provided or automatically loaded: - If a configuration is provided with `config`, `**kwargs` will be directly passed to the underlying model's `__init__` method (we assume all relevant updates to the configuration have already been done) - If a configuration is not provided, `kwargs` will be first passed to the configuration class initialization function ([`~PretrainedConfig.from_pretrained`]). Each key of `kwargs` that corresponds to a configuration attribute will be used to override said attribute with the supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model's `__init__` function. Examples: ```python >>> from transformers import AutoConfig, BaseAutoModelClass >>> # Download model and configuration from huggingface.co and cache. >>> model = BaseAutoModelClass.from_pretrained("checkpoint_placeholder") >>> # Update configuration during loading >>> model = BaseAutoModelClass.from_pretrained("checkpoint_placeholder", output_attentions=True) >>> model.config.output_attentions True >>> # Loading from a TF checkpoint file instead of a PyTorch model (slower) >>> config = AutoConfig.from_pretrained("./tf_model/shortcut_placeholder_tf_model_config.json") >>> model = BaseAutoModelClass.from_pretrained( ... "./tf_model/shortcut_placeholder_tf_checkpoint.ckpt.index", from_tf=True, config=config ... ) ``` a Instantiate one of the model classes of the library from a pretrained model. The model class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it's missing, by falling back to using pattern matching on `pretrained_model_name_or_path`: List options Args: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. - A path or url to a *PyTorch state_dict save file* (e.g, `./pt_model/pytorch_model.bin`). In this case, `from_pt` should be set to `True` and a configuration object should be provided as `config` argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards. model_args (additional positional arguments, *optional*): Will be passed along to the underlying model `__init__()` method. config ([`PretrainedConfig`], *optional*): Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when: - The model is a model provided by the library (loaded with the *model id* string of a pretrained model). - The model was saved using [`~PreTrainedModel.save_pretrained`] and is reloaded by supplying the save directory. - The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a configuration JSON file named *config.json* is found in the directory. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. from_pt (`bool`, *optional*, defaults to `False`): Load the model weights from a PyTorch checkpoint save file (see docstring of `pretrained_model_name_or_path` argument). force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. output_loading_info(`bool`, *optional*, defaults to `False`): Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (e.g., not try downloading the model). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. code_revision (`str`, *optional*, defaults to `"main"`): The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. kwargs (additional keyword arguments, *optional*): Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). Behaves differently depending on whether a `config` is provided or automatically loaded: - If a configuration is provided with `config`, `**kwargs` will be directly passed to the underlying model's `__init__` method (we assume all relevant updates to the configuration have already been done) - If a configuration is not provided, `kwargs` will be first passed to the configuration class initialization function ([`~PretrainedConfig.from_pretrained`]). Each key of `kwargs` that corresponds to a configuration attribute will be used to override said attribute with the supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model's `__init__` function. Examples: ```python >>> from transformers import AutoConfig, BaseAutoModelClass >>> # Download model and configuration from huggingface.co and cache. >>> model = BaseAutoModelClass.from_pretrained("checkpoint_placeholder") >>> # Update configuration during loading >>> model = BaseAutoModelClass.from_pretrained("checkpoint_placeholder", output_attentions=True) >>> model.config.output_attentions True >>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower) >>> config = AutoConfig.from_pretrained("./pt_model/shortcut_placeholder_pt_model_config.json") >>> model = BaseAutoModelClass.from_pretrained( ... "./pt_model/shortcut_placeholder_pytorch_model.bin", from_pt=True, config=config ... ) ``` cCs|t|}t|ttfs|Sdd|D}t|dg}|D],}||vr*||Sd||vr:|d|Sd||vrJ|d|Sq|dS)NcSsi|]}|j|qS__name__).0modelrrT/var/www/auris/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py sz$_get_model_class.. architecturesTFFlaxr)type isinstancelisttuplegetattr)config model_mappingZsupported_modelsZ name_to_modelrarchrrr_get_model_classs   r'c@sTeZdZdZddZeddZededefdd Zed d Z edd dZ dS)_BaseAutoModelClassNcOs&t|jjd|jjd|jjd)Nz+ is designed to be instantiated using the `z5.from_pretrained(pretrained_model_name_or_path)` or `z.from_config(config)` methods.)EnvironmentError __class__r)selfargskwargsrrr__init__s z_BaseAutoModelClass.__init__c Ks&|dd}t|do|j|jv}t||jv}t||j||}|r`|r`|j|j}d|vr8| d\}}n|j }t ||fi|}|j |j |dd|dd} t|}|j|fi|St||jvrxt||j}|j|fi|Std|j d|jd d d d |jDd )Ntrust_remote_codeauto_mapz--Texist_ok code_revision!Unrecognized configuration class  for this kind of AutoModel: . Model type should be one of , cs|]}|jVqdSNrrcrrr z2_BaseAutoModelClass.from_config...)pophasattrrr0r_model_mappingkeysrZ _name_or_pathsplitZ name_or_pathrregisterr*$add_generation_mixin_to_remote_modelZ _from_configr' ValueErrorjoin) clsr$r-r/has_remote_codehas_local_code class_refZrepo_id model_class_rrr from_configs0     z_BaseAutoModelClass.from_configr$returncCs|S)z`Additional autoclass-specific config post-loading manipulation. May be overridden in subclasses.r)rHr$rrr_prepare_config_for_auto_classsz2_BaseAutoModelClass._prepare_config_for_auto_classc sndd}dd}dd<gd}fdd|D}dd}d d} d d} |d d} |d d} | durOtd t| durMtd| } | durW| |d <| duryt|tsst|t fdddd|} t | | } nt |d d} t r| duri} | dur| | d <t |fd | i| }|durt|ddd}t|}|| d<|d}Wdn1swYt|ts t}dddkrԈd}dddurd}tj|fd|| d|\}|dddkrdd<|dddur |dd<t|do|j|jv}t||jv}t||||}|d<| d <|rm|rm|j|j}t||fd|i|}|dd}|j|j|ddt|}|j|g|Rd|i|St||jvrt ||j}|j!|j"ddkr|#}|j|g|Rd|i|Std|jd|jdd $d!d"|jDd#)$Nr$r/TZ _from_auto) cache_dirZforce_downloadZlocal_files_onlyproxiesZresume_downloadrevisionZ subfolderuse_auth_tokentokencs i|] }|vr||qSr)r?)rnamer-rrrs z7_BaseAutoModelClass.from_pretrained..r3 _commit_hashadapter_kwargsrUrTzrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.zV`token` and `use_auth_token` are both specified. Please set only the argument `token`.F)Z _raise_exceptions_for_gated_repoZ%_raise_exceptions_for_missing_entriesZ'_raise_exceptions_for_connection_errorsrzutf-8)encodingZ_adapter_model_pathZbase_model_name_or_pathZ torch_dtypeautoZquantization_config)Zreturn_unused_kwargsr3rXr0r1Z text_configr4r5r6r7csr8r9rr:rrrr<@r=z6_BaseAutoModelClass.from_pretrained..r>)%r?getwarningswarn FutureWarningrFr rrrr r#r r openjsonloadcopydeepcopyrfrom_pretrainedr@rr0rrArBrrrDr*rEr' config_classZ sub_configsZget_text_configrG)rHpretrained_model_name_or_path model_argsr-r$r/Zhub_kwargs_namesZ hub_kwargsr3Z commit_hashrYrUrTZresolved_config_fileZmaybe_adapter_pathfZadapter_configZ kwargs_origrMrIrJrKrLrrWrrfs                       z#_BaseAutoModelClass.from_pretrainedFcCsFt|dr|jj|jkrtd|jd|d|jj|||ddS)a Register a new model for this class. Args: config_class ([`PretrainedConfig`]): The configuration corresponding to the model to register. model_class ([`PreTrainedModel`]): The model to register. rgzThe model class you are passing has a `config_class` attribute that is not consistent with the config class you passed (model has z and you passed z!. Fix one of those so they match!r1N)r@rgrrFrArD)rHrgrLr2rrrrDCs z_BaseAutoModelClass.registerF) r __module__ __qualname__rAr. classmethodrNrrPrfrDrrrrr(s  ~r(cs4eZdZdZefddZefddZZS)_BaseAutoBackboneClassNc st|ddgddlm}|d|}|dddur td|dd r*td |d |j}|d |j}|d |j}|d|j } |||||| d}t j |fi|S)NZvisionZtimmr)TimmBackboneConfigr$Z out_featuresz0Cannot specify `out_features` for timm backbonesZoutput_loading_infoFz@Cannot specify `output_loading_info=True` when loading from timm num_channels features_onlyuse_pretrained_backbone out_indices)Zbackbonerqrrrsrt) rZmodels.timm_backbonerpr?r]rFrqrrrsrtsuperrN) rHrhrir-rpr$rqrrrsrtr*rr#_load_timm_backbone_from_pretrained[s&  z:_BaseAutoBackboneClass._load_timm_backbone_from_pretrainedcsB|dd}|r|j|g|Ri|Stj|g|Ri|S)Nuse_timm_backboneF)r?rwrurf)rHrhrir-rxrvrrrfus z&_BaseAutoBackboneClass.from_pretrained)rrlrmrArnrwrf __classcell__rrrvrroWs rocCs,t|dkr|dd|dS|ddS)Nrz(one of the model classes of the library z0one of the model classes of the library (with a z head) z-one of the base model classes of the library )lenreplace) docstringhead_docrrrinsert_head_doc~s  rgoogle-bert/bert-base-casedc Cs|j}|j}tt|d}|d||_ttj}tt |d}|d|}|d|}||_t |jdd|}t ||_| drCt }n | drKt}nt}ttj} t||d}|d|}|d|}|dd d d } |d | }|| _t |j| } t | |_|S) N)r~ZBaseAutoModelClassZcheckpoint_placeholderF)Zuse_model_typesrr/-rZshortcut_placeholder)rArrCLASS_DOCSTRINGr|__doc__r r(rNFROM_CONFIG_DOCSTRINGrrn startswithFROM_PRETRAINED_TF_DOCSTRINGFROM_PRETRAINED_FLAX_DOCSTRINGFROM_PRETRAINED_TORCH_DOCSTRINGrfrC) rHZcheckpoint_for_exampler~r%rVZclass_docstringrNZfrom_config_docstringZfrom_pretrained_docstringrfZshortcutrrrauto_class_updates4              rcCs<g}|D]}t|ttfr|t|7}q||q|Sr9)valuesr r!r"append)r%resultrrrr get_valuess   rc s|durdSt|trtfdd|DSt|r t|Std}|krEzt||WStyDtd|dd|dwtd|d|d) Nc3s|]}t|VqdSr9)getattribute_from_module)ramodulerrr<sz+getattribute_from_module..Z transformerszCould not find z neither in z nor in !z in )r r"r@r# importlib import_modulerrF)rattrZtransformers_modulerrrrs      rcCsdt|jvr |Sdt|jvr|St|dodtt|dv}t|do-dtt|dv}|s2|r@t|j|tfi|j}|S|S)a Adds `GenerationMixin` to the inheritance of `model_class`, if `model_class` is a PyTorch model. This function is used for backwards compatibility purposes: in v4.45, we've started a deprecation cycle to make `PreTrainedModel` stop inheriting from `GenerationMixin`. Without this function, older models dynamically loaded from the Hub may not have the `generate` method after we remove the inheritance. ztorch.nn.modules.module.ModulergenerateZprepare_inputs_for_generation) str__mro__ __bases__r@r#rrr__dict__)rLZhas_custom_generate_in_classZhas_custom_prepare_inputsZ!model_class_with_generation_mixinrrrrEs  rEc@sreZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ ddZ ddZdddZdS)_LazyAutoMappinga " A mapping config to object (model or tokenizer for instance) that will load keys and values when it is accessed. Args: - config_mapping: The map model type to config class - model_mapping: The map model type to model (or tokenizer) class cCs8||_dd|D|_||_||j_i|_i|_dS)NcSsi|]\}}||qSrrrkvrrrrsz-_LazyAutoMapping.__init__..)_config_mappingitems_reverse_config_mappingrA_extra_content_modules)r+Zconfig_mappingr%rrrr.s  z_LazyAutoMapping.__init__cCs,t|j|j}t|t|jSr9)setrrB intersectionrAr{r)r+Z common_keysrrr__len__sz_LazyAutoMapping.__len__cs|jvr |jS|jj}||jvr |j|}|||Sfdd|jD}|D]}||jvrB|j|}|||Sq.t)Ncsg|] \}}|jkr|qSrrrkeyrr sz0_LazyAutoMapping.__getitem__..)rrrrA_load_attr_from_modulerrKeyError)r+r model_typeZ model_nameZ model_typesmtyperrr __getitem__s        z_LazyAutoMapping.__getitem__cCs:t|}||jvrtd|d|j|<t|j||S)Nr>ztransformers.models)rrrrr)r+rr module_namerrrrs z'_LazyAutoMapping._load_attr_from_modulec*fddjD}|tjS)Nc*g|]\}}|jvr||qSr)rArBrrrrVr+rrr  z)_LazyAutoMapping.keys..)rrr!rrB)r+Z mapping_keysrrrrB z_LazyAutoMapping.keyscCs$z||WSty|YSwr9)rr)r+rdefaultrrrr]s   z_LazyAutoMapping.getcC t|Sr9)boolrBrrrr__bool__$ z_LazyAutoMapping.__bool__cr)Ncrr)rrBrrrrrr(rz+_LazyAutoMapping.values..)rArr!rr)r+Zmapping_valuesrrrr'rz_LazyAutoMapping.valuescr)Ncs>g|]}|jvr|j||j|fqSr)rrBrrA)rrrrrr0sz*_LazyAutoMapping.items..)rArBr!rr)r+Z mapping_itemsrrrr/s z_LazyAutoMapping.itemscCrr9)iterrBrrrr__iter__:rz_LazyAutoMapping.__iter__cCs>||jvrdSt|dr|j|jvrdS|j|j}||jvS)NTrF)rr@rrrA)r+itemrrrr __contains__=s   z_LazyAutoMapping.__contains__FcCsRt|dr"|j|jvr"|j|j}||jvr"|s"td|d||j|<dS)z7 Register a new model in this mapping. r'z*' is already used by a Transformers model.N)r@rrrArBrFr)r+rvaluer2rrrrrDEs  z_LazyAutoMapping.registerNrk)rrlrmrr.rrrrBr]rrrrrrDrrrrrs r)rz)rrz).rrdrrbr^ collectionsrZconfiguration_utilsrZdynamic_module_utilsrrutilsrrr r r r r rrZconfiguration_autorrrZ generationrZ get_loggerrloggerrrrrrr'r(rorrrrrEr__all__rrrrs:  ,   lcdA ' #  f