o ZhB@sjdZddlmZddlmZddlmZddlmZe e Z GdddeZ Gd d d eZ dd gZd S) zmT5 model configuration)Mapping)PretrainedConfig)OnnxSeq2SeqConfigWithPast)loggingcsdeZdZdZdZdgZdddddZ                dfdd ZZS) MT5Configa7 This is the configuration class to store the configuration of a [`MT5Model`] or a [`TFMT5Model`]. It is used to instantiate a mT5 model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the mT5 [google/mt5-small](https://huggingface.co/google/mt5-small) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Arguments: vocab_size (`int`, *optional*, defaults to 250112): Vocabulary size of the T5 model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`T5Model`] or [`TFT5Model`]. d_model (`int`, *optional*, defaults to 512): Size of the encoder layers and the pooler layer. d_kv (`int`, *optional*, defaults to 64): Size of the key, query, value projections per attention head. In the conventional context, it is typically expected that `d_kv` has to be equal to `d_model // num_heads`. But in the architecture of mt5-small, `d_kv` is not equal to `d_model //num_heads`. The `inner_dim` of the projection layer will be defined as `num_heads * d_kv`. d_ff (`int`, *optional*, defaults to 1024): Size of the intermediate feed forward layer in each `T5Block`. num_layers (`int`, *optional*, defaults to 8): Number of hidden layers in the Transformer encoder. num_decoder_layers (`int`, *optional*): Number of hidden layers in the Transformer decoder. Will use the same value as `num_layers` if not set. num_heads (`int`, *optional*, defaults to 6): Number of attention heads for each attention layer in the Transformer encoder. relative_attention_num_buckets (`int`, *optional*, defaults to 32): The number of buckets to use for each attention layer. relative_attention_max_distance (`int`, *optional*, defaults to 128): The maximum distance of the longer sequences for the bucket separation. dropout_rate (`float`, *optional*, defaults to 0.1): The ratio for all dropout layers. classifier_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for classifier. layer_norm_eps (`float`, *optional*, defaults to 1e-6): The epsilon used by the layer normalization layers. initializer_factor (`float`, *optional*, defaults to 1): A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing). feed_forward_proj (`string`, *optional*, defaults to `"gated-gelu"`): Type of feed forward layer to be used. Should be one of `"relu"` or `"gated-gelu"`. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Zmt5Zpast_key_valuesd_model num_heads num_layersd_kv)Z hidden_sizeZnum_attention_headsZnum_hidden_layersZhead_dim@N 皙?ư>? gated-geluT T5TokenizerFrc s||_||_||_||_||_|dur|n|j|_||_||_| |_| |_ ||_ | |_ | |_ | |_ ||_|j d}|d|_|ddk|_t|dkrR|ddksXt|dkr`td| d| d krgd |_tjd ||||||d |dS) N-rZgatedrz`feed_forward_proj`: z is not a valid activation function of the dense layer. Please make sure `feed_forward_proj` is of the format `gated-{ACT_FN}` or `{ACT_FN}`, e.g. 'gated-gelu' or 'relu'rZgelu_new)is_encoder_decodertokenizer_classtie_word_embeddings pad_token_id eos_token_iddecoder_start_token_id) vocab_sizerr d_ffr num_decoder_layersr relative_attention_num_bucketsrelative_attention_max_distance dropout_rateclassifier_dropoutlayer_norm_epsiloninitializer_factorfeed_forward_proj use_cachesplitZ dense_act_fnZ is_gated_actlen ValueErrorsuper__init__)selfr%rr r&r r'r r(r)r*r,r-r.rr/rr r!r"r#r+kwargsZact_info __class__r$X/var/www/auris/lib/python3.10/site-packages/transformers/models/mt5/configuration_mt5.pyr4RsF  $  zMT5Config.__init__)r r rrrNrrrrrrrTTrFrrrr) __name__ __module__ __qualname____doc__Z model_typeZkeys_to_ignore_at_inferenceZ attribute_mapr4 __classcell__r$r$r7r9rs>- rc@sReZdZedeeeeefffddZedefddZede fddZ dS) MT5OnnxConfigreturncCsxddddddd}|jr"d|dd<ddi|d <dd d|d <ndd d|d <dd d|d <|jr:|j|d d|S)NbatchZencoder_sequence)rr)Z input_idsattention_maskz past_encoder_sequence + sequencerBrrZdecoder_input_idsz past_decoder_sequence + sequenceZdecoder_attention_maskZdecoder_sequenceinputs) direction)Zuse_pastZfill_with_past_key_values_)r5Z common_inputsr$r$r9rCs  zMT5OnnxConfig.inputscCdS)N r$r5r$r$r9default_onnx_opsetsz MT5OnnxConfig.default_onnx_opsetcCrE)NgMb@?r$rGr$r$r9atol_for_validationsz!MT5OnnxConfig.atol_for_validationN) r:r;r<propertyrstrintrCrHfloatrIr$r$r$r9r?s r?N)r=typingrZconfiguration_utilsrZonnxrutilsrZ get_loggerr:loggerrr?__all__r$r$r$r9s     |